Set HTTP proxy with Python Requests

This article discusses how to pass proxies to the Python requests library. Before doing that, let” s discuss why we need proxies.

Why do we Need Proxies?

Most sites prohibit users from making so many requests to their servers. We create a lot of traffic to the site when we send so many requests. This can make the site slow for other users. To mitigate this potential problem, the sites may block a client if so many requests are made from that client. The sites usually identify the clients based on their IP addresses. This is where the use of proxies comes in.

Proxies can be used to bypass restrictions (like requests limit) or even just to keep our requests anonymous. A proxy is a server that exists as an intermediary between the client (your browser) and the server (the website we are visiting). When proxies are used, proxy IP will be used instead of sharing our IP address directly.

Using Proxies with Python requests Library

Proxies can be configured in the Python requests library using the “proxies” parameter in requests.get() and requests.post() functions. Let” s go through the steps you should follow.

Step 0: Install the requests package

The requests package does not come preinstalled with Python. You can install it using pip by executing the command

Step 1: Find the proxy server(s) to use

There are many publicly available proxy servers on the internet. Some are free, and others are paid. In this post, we will use the following site https://free-proxy-list.net/ (we need the IP address and the port).

Step 2: Specify the proxy by setting the “proxies” parameter of your request

The proxies parameters accept the proxies as a dictionary with the following format.

Then we can pass the proxies to the requests library, as shown below.

In the code above, we created a dictionary with proxies that specifies the proxy server” s IP address and port number. The details were obtained from the URL given in Step 1.

Passing Proxies to requests Session

The Session object in requests allows you to persist specific parameters across requests. In our cases, we want to set proxies and use them across all requests made from the Session instance we create.

Here is an example.

Note: some proxy servers are publicly available and require no authentication, while others are private and require authentication (password, API keys, etc.). If you are using a protected proxy provider, ensure you read their documentation to get the form of authentication they use. You can read more here.

Conclusion

This article discussed how to pass proxies in Python requests to keep our connection anonymous or bypass restrictions.

First, we need to find proxy servers (some are publicly available, and others are private), then use the IP address and the port numbers for those proxies to establish a connection using Python requests. The proxies are passed into the requests functions like requests.get() and requests.post() using the “proxies” parameter.