This article will discuss three methods that can be used to print HTTP requests sent using Python.
- Turning verbosity on when sending requests,
- Using the attributes in PreparedRequest of requests library, and,
- Using requests_toolbelt library
Method 1: Logging and Turning request Verbosity on
In this method, we need requests, logging, and http_client() to enable the logging of the output of a request in debugging mode. The log will show the HTTP request, including headers and data, response with headers but without the data.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
import requests import logging try: # For Python 3 import http.client as http_client except ImportError: # For Python 2 users import httplib as http_client # Turn verbosity on with debugging http_client.HTTPConnection.debuglevel = 1 # You must initialize logging, otherwise you'll not see debug output. logging.basicConfig() logging.getLogger().setLevel(logging.DEBUG) requests_log = logging.getLogger("requests.packages.urllib3") requests_log.setLevel(logging.DEBUG) requests_log.propagate = True headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.0.1) Gecko/20100101 Firefox/1.0.1", "Custom": "Testing Request"} # Calling a get request on a link that doesn't exist. response = requests.get('http://api.github.com/ussfsdsers', headers=headers) |
Output (truncated):
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): api.github.com:80 send: b'GET /ussfsdsers HTTP/1.1\r\nHost: api.github.com\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.0.1) Gecko/20100101 Firefox/1.0.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nCustom: Testing Request\r\n\r\n' reply: 'HTTP/1.1 301 Moved Permanently\r\n' header: Location: https://api.github.com/ussfsdsers header: Content-Length: 0 DEBUG:urllib3.connectionpool:http://api.github.com:80 "GET /ussfsdsers HTTP/1.1" 301 0 DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.github.com:443 send: b'GET /ussfsdsers HTTP/1.1\r\nHost: api.github.com\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.0.1) Gecko/20100101 Firefox/1.0.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nCustom: Testing Request\r\n\r\n' reply: 'HTTP/1.1 404 Not Found\r\n' header: Server: GitHub.com <output truncated> header: Content-Length: 98 header: X-GitHub-Request-Id: D3EA:12B09:9DE5DA6:A072E38:632D9481 DEBUG:urllib3.connectionpool:https://api.github.com:443 "GET /ussfsdsers HTTP/1.1" 404 98
The output log is informative. It shows the actual HTTP request containing the headers and the data. The debugger also details all the events happening under the hoods. The final debug statement shows that the URL we passed does not exist (resulting in a 404).
Method 2: Using PreparedRequests class of requests
The class has a method called prepare() that allows us to pass request parameters to the object before sending the actual request. The object “contains the exact bytes that will be sent to the server.“
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
import requests # Defining some headers for the request headers = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36", "Custom-headers": "Test request"} # Calling the PreparedReqest class req = requests.Request('GET','http://api.github.com/users', headers=headers) # prepare the request parameters for the GET request prepared = req.prepare() print(prepared.headers) print(prepared.body) print(prepared.method) print(prepared.url) |
Output:
{'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36', 'Custom-headers': 'Test request'}
None
GET
http://api.github.com/users
The output shows the user-agent (my browser) on the headers, the request method (GET), and the body, which is None because we are making a GET request, not POST. Once you are satisfied with the HTTP request parameters, you can now make the actual request on a requests Session as follows:
|
1 2 3 |
sess = requests.Session() response = sess.send(prepared) print(response) |
Output:
<Response [200]>
Method 3: Using the requests_toolbelt library
The requests_toolbelt.dump() class has a method called dump_all() that gives not only the response but also the HTTP parameters of the request. This method may provide a more comprehensive HTTP request than using PreparedRequest() class. Here is an example of a used case.
|
1 2 3 4 5 6 7 |
import requests from requests_toolbelt.utils import dump # pip install requests_toolbelt if the package is not installed resp = requests.get('http://api.github.com/users') data = dump.dump_all(resp) print(data.decode('utf-8')) |
Output:
< GET /users HTTP/1.1 < Host: api.github.com < User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 < Accept-Encoding: gzip, deflate < Accept: */* < Connection: keep-alive < Custom-headers: Test request < > HTTP/1.1 301 Moved Permanently > Location: https://api.github.com/users > Content-Length: 0 > <output truncated> > HTTP/1.1 200 OK > Server: GitHub.com <output truncated> > X-GitHub-Request-Id: 9B2C:0BB5:DC30C6B:DF7E54B:632D9236 > [<Actual response data - omitted>]
The output shows that two requests were made. The first request returned 301, and therefore the request was redirected. At the bottom of the output, the actual response is printed.
Conclusion
In this article, we discussed three methods that can be used to view and analyze our HTTP requests. The first and the third methods give more information on HTTP request parameters, but the actual request is to be made. The second method gives precise request parameters we need without us sending the actual request.