This article discusses how to pass proxies to the Python requests library. Before doing that, let” s discuss why we need proxies.
Why do we Need Proxies?
Most sites prohibit users from making so many requests to their servers. We create a lot of traffic to the site when we send so many requests. This can make the site slow for other users. To mitigate this potential problem, the sites may block a client if so many requests are made from that client. The sites usually identify the clients based on their IP addresses. This is where the use of proxies comes in.
Proxies can be used to bypass restrictions (like requests limit) or even just to keep our requests anonymous. A proxy is a server that exists as an intermediary between the client (your browser) and the server (the website we are visiting). When proxies are used, proxy IP will be used instead of sharing our IP address directly.
Using Proxies with Python requests Library
Proxies can be configured in the Python requests library using the “proxies” parameter in requests.get() and requests.post() functions. Let” s go through the steps you should follow.
Step 0: Install the requests package
The requests package does not come preinstalled with Python. You can install it using pip by executing the command
|
1 |
pip install requests |
Step 1: Find the proxy server(s) to use
There are many publicly available proxy servers on the internet. Some are free, and others are paid. In this post, we will use the following site https://free-proxy-list.net/ (we need the IP address and the port).
Step 2: Specify the proxy by setting the “proxies” parameter of your request
The proxies parameters accept the proxies as a dictionary with the following format.
|
1 2 3 4 |
proxies = { "http": "http://[proxy_IP_address]:[proxy_port]", "https": "https://[proxy_IP_address]:[proxy_port]" } |
Then we can pass the proxies to the requests library, as shown below.
|
1 2 3 4 5 6 7 8 |
import requests proxies = { "http": "http://8.219.176.202:8080", "https": "http://8.219.176.202:8080", } response = requests.get("http://www.example.com", proxies=proxies) print(response.text) |
In the code above, we created a dictionary with proxies that specifies the proxy server” s IP address and port number. The details were obtained from the URL given in Step 1.
Passing Proxies to requests Session
The Session object in requests allows you to persist specific parameters across requests. In our cases, we want to set proxies and use them across all requests made from the Session instance we create.
Here is an example.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
import requests from requests.adapters import HTTPAdapter from urllib3.util.retry import Retry # Initialize requests.Session() session = requests.Session() # Pass proxies to our session session.proxies = { "http": "http://137.184.100.135:80" } # Use Retry() class to allow you to retry connections. # You can read more from this link: https://urllib3.readthedocs.io/en/stable/reference/urllib3.util.html retry = Retry(connect=4, backoff_factor=0.5) adapter = HTTPAdapter(max_retries=retry) # Mount HTTP and HTTPS protocol. session.mount("http://", adapter) session.mount("https://", adapter) response = session.get("http://www.example.com") print(response.text) |
Note: some proxy servers are publicly available and require no authentication, while others are private and require authentication (password, API keys, etc.). If you are using a protected proxy provider, ensure you read their documentation to get the form of authentication they use. You can read more here.
Conclusion
This article discussed how to pass proxies in Python requests to keep our connection anonymous or bypass restrictions.
First, we need to find proxy servers (some are publicly available, and others are private), then use the IP address and the port numbers for those proxies to establish a connection using Python requests. The proxies are passed into the requests functions like requests.get() and requests.post() using the “proxies” parameter.