Make Selenium Wait in Python

Nowadays, the majority of online applications use the AJAX framework. The elements in AJAX pages may load at different time intervals. This means that selenium can attempt to locate an element that is not yet available, leading to an error.

We can fix this problem by implementing waits. These delays give the site elements time to load fully. In this article, we will discuss three kinds of waits (with examples):

Hard wait – using time.sleep(),
Implicit wait, and,
Explicit wait.

At the end of the post, you should have a solid understanding of these waits and when to use them.

Hard wait – using time.sleep()

Python time.sleep(<secs>) function can be used to halt the execution of a Python script for a given number of seconds. With this wait, we hope the website we are trying to access with selenium will finish loading.

In the following example, we use time.sleep() to halt execution for 10 seconds, locate web element with ID=”Culture”, stop the program execution for another 10 seconds, then attempt to locate another element with ID=”Element22″.

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.common.exceptions import NoSuchElementException

import time

driver = webdriver.Firefox()

driver.get("https://en.wikipedia.org/wiki/Wikipedia:Contents")

################################################################

# Find an element that exists, issuing hard sleep at the start.

################################################################

start = time.time()

time.sleep(10)

try:

# Element with Culture ID is available.

element = driver.find_element(By.ID, "Culture")

except NoSuchElementException:

print("Element count not be found. Timed out.")

end = time.time()

print(f"It took {end-start} seconds.")

################################################################

# Find element that does not exist issuing hard sleep at the start.

################################################################

start = time.time()

time.sleep(10)

try:

# Element22 does not exist on the source therefore

# NoSuchElementException is raised.

element = driver.find_element(By.ID, "Element22")

except NoSuchElementException:

# Catch the exception here.

print("Element count not be found. Timed out.")

end = time.time()

print(f"It took {end-start} seconds.")

Output:

It took 10.048056840896606 seconds.
Element count not be found. Timed out.
It took 10.01723027229309 seconds.

Unlike the first element, the second element does not exist on the web page (NoSuchElementException was raised and handled using a try-except block); therefore, waiting will not help us here.

Assuming that the first element became available after 2 seconds of loading the site, time.sleep(10) will still wait for 10 seconds. That means 8 seconds of unnecessary wait. Is there something to do about this? Yes, by using explicit or implicit waits. Let’s discuss them next.

Implicit wait

When searching for any element (or elements) that aren’t immediately available, WebDriver is instructed via an implicit delay to poll the DOM for a pre-determined period.

The implicit wait is set once and remains set for as long as the WebDriver object remains alive.

By default, selenium does not implement implicit wait.

Unlike time.sleep(), implicit wait proceeds with program execution as soon as the sought element is found. For example,

driver.implicit_wait(10) polls the DOM for 10 seconds when finding any element below it on the script. If a given element is located within 10 seconds, execution continues; otherwise, NoSuchElementException is raised.

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.common.exceptions import NoSuchElementException

import time

driver = webdriver.Firefox()

driver.get("https://en.wikipedia.org/wiki/Wikipedia:Contents")

# A wait of 20 seconds will be implemented on any element below this line

driver.implicitly_wait(20)

start = time.time()

# This element is available on the webpage therefore no need to wait.

a = driver.find_element(By.ID, "Culture")

end = time.time()

print("Available element (Culture): ", end - start)

start = time.time()

try:

# This element is not present therefore NoSuchElementException is

# raised

a = driver.find_element(By.ID, "Cultur")

except NoSuchElementException:

print("Element not found. Quitting...")

end = time.time()

print("Element not present (Cultur): ", end - start)

start = time.time()

# Element available.

a = driver.find_element(By.ID, "Geography")

end = time.time()

print("Another present element: ", end - start)

# Quit the WebDriver.

driver.quit()

Output:

Available element (Culture):  0.04906463623046875
Element not found.
Element not present (Cultur):  20.023759365081787
Another present element:  0.0120849609375

driver.implicitly_wait(20) implemented 20 seconds wait for each element after it was defined, and the execution is resumed as soon as the element is available.

If the element is not found after the pre-determined time (like for Cultur), NoSuchElementException is raised.

Explicit Wait

An explicit wait is a code you define for a specific condition to occur before continuing the code execution.

In most cases, this involves using two functions:

selenium.webdriver.support.ui.WebDriverWait() to implement the wait on the Web Driver object, and,
Selenium.webdriver.support.expected_conditions() to set the conditions.

WebDriverWait(driver, timeout, poll_frequency= 0.5).until(method) – where the expected_conditions are defined on the until method.

By default, WebDriverWait calls the until method every half a second (500 milliseconds) until it returns success. The expected_conditons will return True in case of success or not null if it fails to locate an element.

If expected_conditions fails to locate the specified element in the allocated time, it throws a TimeoutException.

from selenium import webdriver

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

from selenium.common.exceptions import TimeoutException

import time

driver = webdriver.Firefox()

driver.get("https://en.wikipedia.org/wiki/Wikipedia:Contents")

start = time.time()

try:

# Issuing a 20 seconds wait on SomeIDElement.

# This ID doesn't exist and therefore WebDriverWait

# will wait for 20 seconds and raise TimeoutException.

element = WebDriverWait(driver, 20).until(

EC.presence_of_element_located((By.ID, "SomeIDElement"))

)

except TimeoutException:

print("Element count not be found. Timed out.")

end = time.time()

print(f"It took {end-start} seconds.")

start = time.time()

try:

# Implement a 20 seconds wait for Culture element

# This element is available

element = WebDriverWait(driver, 20).until(

EC.presence_of_element_located((By.ID, "Culture"))

)

except TimeoutException:

print("Element count not be found. Timed out.")

finally:

driver.quit()

end = time.time()

print(f"It took {end-start} seconds.")

# Close the webdriver object.

driver.quit()

Output:

Element count not be found. Timed out.
It took 20.400476694107056 seconds.
It took 0.4250028133392334 seconds.

WebDriverWait was timed out in the first case because the element didn’t exist. After that, TimeoutException was raised.

In the second case, the WebDriverWait can wait for 20 seconds, but the element was found sooner (0.42 seconds), reducing the waiting time.

Expected Conditions

In the example above, we used the presence_of_element_located() function to define the expected conditions. Other common expected conditions include:

element_to_be_clickable(mark),
element_to_be_selected(element),
staleness_of(element),
title_contains(title), and,
text_to_be_present_in_element(locator, text)

You can see the complete list on the documentation of webdriver.support.expected_conditions.

Conclusion

We have discussed three kinds of waits in this article. As shown, using time.sleep() is not a good practice in most cases because it halts the execution of the code for a specified time without checking if the page is loaded.

To avoid this, use explicit or implicit waits. The implicit wait can be defined on top of the script (after initializing the WebDriver object) so that the pre-determined delay can be implemented for every element in the code. Alternatively, you can define an explicit wait for the specific elements of your choice.

Codeigo

Just programming

Make Selenium Wait in Python

Hard wait – using time.sleep()

Implicit wait

Explicit Wait

Expected Conditions

Conclusion