Python urllib

The Python urllib library is used to operate on web URLs and scrape web content.

This article mainly introduces Python3's urllib.

The urllib package contains the following modules:

• urllib.request- Open and read URLs.
• urllib.error- Contains exceptions raised by urllib.request.
• urllib.parse- Parse URLs.
• urllib.robotparser- Parse robots.txt files.

urllib.request

urllib.request defines some functions and classes for opening URLs, including authorization verification, redirects, browser cookies, etc.

urllib.request can simulate a browser's request initiation process.

We can use the urlopen method of urllib.request to open a URL, the syntax is as follows:

urllib.request.urlopen(url, data=None, [timeout, ]*, cafile=None, capath=None, cadefault=False, context=None)

• url: url address.
• data: Other data objects sent to the server, default is None.
• timeout: Set access timeout.
• cafile and capath: cafile is the CA certificate, capath is the path to the CA certificate, required for HTTPS.
• cadefault: Deprecated.
• context: ssl.SSLContext type, used to specify SSL settings.

Example is as follows:

The above code uses urlopen to open a URL, then uses the read() function to get the HTML source code of the webpage.

read() reads the entire webpage content; we can specify the length to read:

In addition to the read() function, it also includes the following two functions for reading webpage content:

• readline()- Read one line of the file

  from urllib.request import urlopen
  
  myURL = urlopen("https://www.example.com/")
  print(myURL.readline()) #读取一行内容

• readlines()- Read all content of the file, it assigns the read content to a list variable.

  from urllib.request import urlopen
  
  myURL = urlopen("https://www.example.com/")
  lines = myURL.readlines()
  for line in lines:
      print(line) 

When scraping webpages, we often need to determine whether the webpage can be accessed normally. Here we can use the getcode() function to get the webpage status code. Returning 200 means the webpage is normal, returning 404 means the webpage does not exist:

For more webpage status codes, refer to:https://www.example.com/http/http-status-codes.html。

If you want to save the scraped webpage locally, you can usePython3 File write() methodfunction:

After executing the above code, a example_urllib_test.html file will be generated locally, containing the content of the https://www.example.com/ webpage.

For more Python File handling, refer to:https://www.example.com/python3/python3-file-methods.html

URL encoding and decoding can useurllib.request.quote()andurllib.request.unquote()methods:

The output result is:

https%3A//www.example.com/
https://www.example.com/

Simulate Headers

When scraping webpages, we generally need to simulate headers (webpage header information). At this time, we need to use the urllib.request.Request class:

class urllib.request.Request(url, data=None, headers={}, origin_req_host=None, unverifiable=False, method=None)

• url: url address.
• data: Other data objects sent to the server, default is None.
• headers: HTTP request header information, in dictionary format.
• origin_req_host: The host address of the request, IP or domain name.
• unverifiable: This parameter is rarely used. It is used to set whether the webpage requires verification. Default is False.
• method: Request method, such as GET, POST, DELETE, PUT, etc.

After executing the above Python code, an urllib_test_example_search.html file will be generated in the current directory. Open the urllib_test_example_search.html file (you can open it with a browser), the content is as follows:

For form POST data transmission, let's first create a form. The code is as follows. Here I use PHP code to get the form data:

Executing the above code will submit form data to the py3_urllib_test.php file, and write the output result to the urllib_test_post_example.html file.

Open the urllib_test_post_example.html file (you can open it with a browser), the displayed result is as follows:

urllib.error

The urllib.error module defines exception classes for exceptions raised by urllib.request, and the base exception class is URLError.

urllib.error contains two exception classes, URLError and HTTPError.

URLError is a subclass of OSError, used to handle exceptions (or derived exceptions) that the program raises when encountering problems. The attribute reason it contains is the cause of the exception.

HTTPError is a subclass of URLError, used to handle special HTTP errors such as authentication requests. The attributes it contains are:codecode: the HTTP status code,reasonreason: the cause of the exception,headersheaders: the HTTP response headers of the specific HTTP request that caused the HTTPError.

Scrape a non-existent webpage and handle exceptions:

urllib.parse

urllib.parse is used to parse URLs, the format is as follows:

urllib.parse.urlparse(urlstring, scheme='', allow_fragments=True)

urlstring is the string url address, scheme is the protocol type,

If the allow_fragments parameter is false, fragment identifiers cannot be recognized. Instead, they are parsed as part of the path, parameters, or query components, and fragment is set to an empty string in the return value.

The output result of the above example is:

ParseResult(scheme='https', netloc='www.example.com', path='/', params='', query='s=python+%E6%95%99%E7%A8%8B', fragment='')

From the result, it can be seen that the content is a tuple containing 6 strings: protocol, location, path, parameters, query, and fragment.

We can directly read the protocol content:

The output result of the above example is:

https

The complete content is as follows:

urllib.robotparser

urllib.robotparser is used to parse robots.txt files.

robots.txt (all lowercase) is a robots protocol stored in the root directory of a website. It is usually used to tell search engines the crawling rules for the website.

urllib.robotparser provides the RobotFileParser class, the syntax is as follows:

class urllib.robotparser.RobotFileParser(url='')

This class provides methods that can read and parse robots.txt files:

• set_url(url) - Sets the URL of the robots.txt file.

• read() - Reads the robots.txt URL and feeds it to the parser.

• parse(lines) - Parses the lines parameter.

• can_fetch(useragent, url) - Returns True if the useragent is allowed to fetch the url according to the rules in the parsed robots.txt file.

• mtime() - Returns the time the robots.txt file was last fetched. This is useful for long-running web crawlers that need to periodically check for robots.txt updates.

• modified() - Sets the time the robots.txt file was last fetched to the current time.

• crawl_delay(useragent) - Returns the Crawl-delay parameter from robots.txt for the specified useragent. Returns None if this parameter does not exist, does not apply to the specified useragent, or if the robots.txt entry for this parameter has a syntax error.

• request_rate(useragent) - Returns the content of the Request-rate parameter from robots.txt as a named tuple RequestRate(requests, seconds). Returns None if this parameter does not exist, does not apply to the specified useragent, or if the robots.txt entry for this parameter has a syntax error.

• site_maps() - Returns the content of the Sitemap parameter from robots.txt as a list(). Returns None if this parameter does not exist or if the robots.txt entry for this parameter has a syntax error.