Python selenium library

Selenium is a powerful tool for automating web browser operations, widely used in scenarios such as web application testing, web data scraping, and task automation.

Selenium provides APIs for various programming languages for testing. The current official API documentation includes C#, JavaScript, Java, Python, and Ruby.

Selenium Tutorial:https://www.example.com/selenium/

Install Selenium and WebDriver

Install Selenium

To start using Selenium, you first need to install the selenium library and download the WebDriver for your browser.

Install Selenium using pip:

pip install selenium

After installation, you can use the following command to check the version information of selenium:

pip show selenium

You can also check it using Python code:

import selenium
print(selenium.__version__)

Download WebDriver

Selenium requires a WebDriver to interact with the browser.

Different browsers require different WebDrivers. For example, Chrome browser requires ChromeDriver. You need to download the corresponding WebDriver for the browser you are using and ensure it is in your system's PATH.

Choose a browser and initialize the WebDriver:

Example

from selenium import webdriver

# Use Chrome browser
driver = webdriver.Chrome(executable_path='/path/to/chromedriver')

# Or use Firefox browser
# driver = webdriver.Firefox(executable_path='/path/to/geckodriver')

# Or use Edge browser
# driver = webdriver.Edge(executable_path='/path/to/msedgedriver')

Starting with Selenium 4, there has been a change in how browser drivers are managed: Selenium 4 attempts to automatically detect the browser version installed on the system and download the corresponding driver. This means users no longer need to manually download and set the driver path, unless they need a specific version of the driver.

Example

from selenium import webdriver

driver = webdriver.Chrome()  # If using another browser, such as Firefox, modify accordingly

In domestic network environments, automatically detecting and downloading the driver requires a different network environment, so it is recommended to manually download the driver and then specify the driver path.

In Selenium 4, instead of directly setting the driver path in webdriver.Chrome, you set it by introducing a Service object. This avoids deprecation warnings and ensures the driver loads correctly. For example:

Example

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService

service = ChromeService(executable_path="PATH_TO_DRIVER")
options = webdriver.ChromeOptions()
driver = webdriver.Chrome(service=service, options=options)

The following code specifies the driver file path to get the page title:

Example

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService

# Set the correct driver path
service = ChromeService(executable_path="/EXAMPLE/Downloads/chromedriver-mac-arm64/chromedriver")
options = webdriver.ChromeOptions()
driver = webdriver.Chrome(service=service, options=options)


# Open a website
driver.get("https://cn.bing.com")

# Get the page title
print(driver.title)

# Close the browser
driver.quit()

Basic Usage

Initialize WebDriver

Choose a browser and initialize the WebDriver:

Example

from selenium import webdriver

# Use Chrome browser
driver = webdriver.Chrome()

# Or use Firefox browser
# driver = webdriver.Firefox()

# Or use Edge browser
# driver = webdriver.Edge()

Open a webpage

Use the get() method to open a webpage:

driver.get("https://www.baidu.com")

Find page elements

Page elements can be found in various ways, such as by ID, class name, tag name, etc.:

Example

# Find element by ID
search_box = driver.find_element("id", "kw")

# Find element by class name
search_button = driver.find_element("class name", "s_ipt")

# Find element by tag name
links = driver.find_elements("tag name", "a")

Simulate user operations

Selenium can simulate user operations in the browser, such as clicking, entering text, etc.:

Example

# Enter text in the search box
search_box.send_keys("Selenium Python")

# Click the search button
search_button.click()

Get element attributes and text

You can get the attribute values or text content of page elements:

Example

# Get the text of the element
element_text = search_box.text

# Get the attribute value of the element
element_attribute = search_box.get_attribute("placeholder")

Waits

Sometimes the page needs time to load. You can use explicit wait or implicit wait to ensure the element is actionable:

Example

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Explicit wait
element = WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((By.ID, "kw"))
)

# Implicit wait
driver.implicitly_wait(10)

Close the browser

After completing the operations, remember to close the browser:

driver.quit()

Simple web automation

Below is a simple Selenium project example used to automate keyword search and get the title of the result page.

Example

from selenium import webdriver
from selenium.webdriver.common.by import By  # Import the By module
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Set the driver path and start the browser
service = ChromeService(executable_path="/EXAMPLE/Downloads/chromedriver-mac-arm64/chromedriver")
options = webdriver.ChromeOptions()
driver = webdriver.Chrome(service=service, options=options)


try:
    # Open the Baidu homepage
    driver.get("https://www.baidu.com")

    # Find the search box element
    search_box = driver.find_element(By.ID, "kw")

    # Enter search content
    search_box.send_keys("Selenium Python")

    # Submit the search form
    search_box.send_keys(Keys.RETURN)

    # Wait for search results to load
    WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "content_left"))
    )

    # Print the page title
    print("The page title is:", driver.title)

finally:
    # Close the browser
    driver.quit()

Selenium common methods

The following table lists the common methods of the selenium library:

MethodDescriptionExample code
webdriver.Chrome()Initialize a Chrome browser instance.driver = webdriver.Chrome()
driver.get(url)Visit the specified URL address.driver.get("https://example.com")
driver.find_element(By, value)Find the first matching element.element = driver.find_element(By.ID, "id")
driver.find_elements(By, value)Find all matching elements.elements = driver.find_elements(By.CLASS_NAME, "class")
element.click()Click an element.element.click()
element.send_keys(value)Send keyboard input to an input box.element.send_keys("text")
element.textGet the text content of an element.text = element.text
driver.back()Browser back.driver.back()
driver.forward()Browser forward.driver.forward()
driver.refresh()Refresh the current page.driver.refresh()
driver.execute_script(script, *args)Execute JavaScript script.driver.execute_script("alert('Hello!')")
driver.switch_to.frame(frame_reference)Switch to the specified iframe.driver.switch_to.frame("frame_id")
driver.switch_to.default_content()Switch back to the main document.driver.switch_to.default_content()
driver.quit()Close the browser and quit the driver.driver.quit()
driver.close()Close the current window.driver.close()
Other extensions