Python Web Scraping - BeautifulSoup

Python web scraping refers to the process of automatically extracting information from the internet by writing Python programs.

The basic web scraping process usually includes sending HTTP requests to fetch web page content, parsing the web pages and extracting data, and then storing the data.

Python's rich ecosystem makes it a popular language for developing web scrapers, especially because of its powerful library support.

Generally speaking, the web scraping process can be divided into the following steps:

• Send HTTP requests: The scraper obtains HTML pages from the target website via HTTP requests. Commonly used libraries includerequests。
• Parse HTML content: After obtaining the HTML page, the scraper needs to parse the content and extract data. Common libraries includeBeautifulSoup、lxml、Scrapy, etc.
• Extract data: Extract the required data by locating HTML elements (such as tags, attributes, class names, etc.).
• Store data: Store the extracted data in formats such as databases, CSV files, and JSON files for later use or analysis.

This chapter mainly introduces BeautifulSoup, which is a Python library for parsing HTML and XML documents. It can extract data from web pages and is commonly used for web scraping and data mining.

BeautifulSoup

BeautifulSoup is a Python library for extracting data from web pages, especially suitable for parsing HTML and XML files.

BeautifulSoup can extract and manipulate web page content by providing a simple API, making it very suitable for web scraping and data extraction tasks.

Installing BeautifulSoup

To use BeautifulSoup, you need to install beautifulsoup4 and lxml or html.parser (an HTML parser).

We can use pip to install these dependencies:

pip install beautifulsoup4
pip install lxml  # 推荐使用 lxml 作为解析器（速度更快）

If you don't have lxml, you can use Python's built-in html.parser as the parser.

Basic Usage

BeautifulSoup is used to parse HTML or XML data and provides methods to navigate, search, and modify the parse tree.

Common BeautifulSoup operations include finding tags, getting tag attributes, extracting text, etc.

To use BeautifulSoup, you need to first import BeautifulSoup and load the HTML page into a BeautifulSoup object.

Usually, you will first use a web scraping library (such asrequests) to obtain the web page content:

Get the web page title:

After running the above code, the output title is:

搜索 - Microsoft 必应

Chinese Garbled Text Issue

When using the requests library to scrape Chinese web pages, encoding issues may occur, causing Chinese content to not display correctly. To ensure that Chinese web pages are correctly scraped and displayed, you usually need to handle the character encoding of the web page.

Automatic encoding detection: requests usually automatically infers the web page encoding based on the Content-Type in the response headers, but this may sometimes be inaccurate. In such cases, you can use chardet to automatically detect the encoding.

After running the above code, the output is:

utf-8

If you know the encoding of the web page (e.g., utf-8 or gbk), you can directly setresponse.encoding:

response.encoding = 'utf-8'  # 或者 'gbk'，根据实际情况选择

Finding Tags

BeautifulSoup provides multiple methods for finding tags in a web page. The most commonly used ones includefind()andfind_all()。

• find()Returns the first matching tag
• find_all()Returns all matching tags

The output result is similar to the following:

<a class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻</a>
----------------------------
http://news.baidu.com
----------------------------
[<a class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻</a>, <a class="mnav" href="https://www.hao123.com" name="tj_trhao123">hao123</a>, <a class="mnav" href="http://map.baidu.com" name="tj_trmap">地图</a>, <a class="mnav" href="http://v.baidu.com" name="tj_trvideo">视频</a>,

Getting Tag Text

Throughget_text()method, you can extract the text content within a tag:

The output result is similar to the following:

Baidu it, and you'll know.

Finding Child and Parent Tags

# 获取当前标签的父标签
parent_tag = first_link.parent

# 获取当前标签的所有子标签
children = first_link.children

The output result is similar to the following:

<a class="mnav" href="http://news.baidu.com" name="tj_trnews">新闻</a>
----------------------------
 新闻 hao123 地图 视频 贴吧  登录   更多产品 

Finding Tags with Specific Attributes

You can find tags with specific attributes by passing attribute parameters.

For example, find all div tags with the class name example-class:

# 查找所有 class="example-class" 的 <div> 标签
divs_with_class = soup.find_all('div', class_='example-class')

# 查找具有 id="unique-id" 的 <p> 标签
unique_paragraph = soup.find('p', id='unique-id')

Get the search button with the id su:

The output result is:

Baidu it.

Advanced Usage

CSS Selectors

BeautifulSoup also supports finding tags using CSS selectors.

The select() method allows you to use jQuery-like selector syntax to find tags:

# 使用 CSS 选择器查找所有 class 为 'example' 的 <div> 标签
example_divs = soup.select('div.example')

# 查找所有 <a> 标签中的 href 属性
links = soup.select('a[href]')

Handling Nested Tags

BeautifulSoup supports deeply nested HTML structures. You can handle these structures by recursively finding child tags:

# 查找嵌套的 <div> 标签
nested_divs = soup.find_all('div', class_='nested')
for div in nested_divs:
    print(div.get_text())

Modifying Web Page Content

BeautifulSoup allows you to modify HTML content.

We can modify tag attributes, text, or delete tags:

Converting to String

You can convert the parsed BeautifulSoup object back to an HTML string:

# 转换为字符串
html_str = str(soup)

BeautifulSoup Attributes and Methods

The following are commonly used attributes and methods in BeautifulSoup:

Other Attributes

Other