Print the Entire Http Request (Raw) in Python

This article will discuss three methods that can be used to print HTTP requests sent using Python.

  1. Turning verbosity on when sending requests,
  2. Using the attributes in PreparedRequest of requests library, and,
  3. Using requests_toolbelt library

Method 1: Logging and Turning request Verbosity on

In this method, we need requests, logging, and http_client() to enable the logging of the output of a request in debugging mode. The log will show the HTTP request, including headers and data, response with headers but without the data.

import requests
import logging
try:
	# For Python 3
	import http.client as http_client
except ImportError:
	# For Python 2 users
	import httplib as http_client
# Turn verbosity on with debugging
http_client.HTTPConnection.debuglevel = 1
# You must initialize logging, otherwise you'll not see debug output.
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True
headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.0.1) Gecko/20100101 Firefox/1.0.1", "Custom": "Testing Request"}
# Calling a get request on a link that doesn't exist.
response = requests.get('http://api.github.com/ussfsdsers', headers=headers)

Output (truncated):

DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): api.github.com:80
send: b'GET /ussfsdsers HTTP/1.1\r\nHost: api.github.com\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.0.1) Gecko/20100101 Firefox/1.0.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nCustom: Testing Request\r\n\r\n'
reply: 'HTTP/1.1 301 Moved Permanently\r\n'
header: Location: https://api.github.com/ussfsdsers
header: Content-Length: 0
DEBUG:urllib3.connectionpool:http://api.github.com:80 "GET /ussfsdsers HTTP/1.1" 301 0
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.github.com:443
send: b'GET /ussfsdsers HTTP/1.1\r\nHost: api.github.com\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; en-US; rv:1.0.1) Gecko/20100101 Firefox/1.0.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nCustom: Testing Request\r\n\r\n'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Server: GitHub.com
<output truncated>
header: Content-Length: 98
header: X-GitHub-Request-Id: D3EA:12B09:9DE5DA6:A072E38:632D9481
DEBUG:urllib3.connectionpool:https://api.github.com:443 "GET /ussfsdsers HTTP/1.1" 404 98

The output log is informative. It shows the actual HTTP request containing the headers and the data. The debugger also details all the events happening under the hoods. The final debug statement shows that the URL we passed does not exist (resulting in a 404).

Method 2: Using PreparedRequests class of requests

The class has a method called prepare() that allows us to pass request parameters to the object before sending the actual request. The object “contains the exact bytes that will be sent to the server.

import requests
# Defining some headers for the request
headers = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36",
"Custom-headers": "Test request"}
# Calling the PreparedReqest class
req = requests.Request('GET','http://api.github.com/users', headers=headers)
# prepare the request parameters for the GET request
prepared = req.prepare()
print(prepared.headers)
print(prepared.body)
print(prepared.method)
print(prepared.url)

Output:

{'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36', 'Custom-headers': 'Test request'}
None
GET
http://api.github.com/users

The output shows the user-agent (my browser) on the headers, the request method (GET), and the body, which is None because we are making a GET request, not POST. Once you are satisfied with the HTTP request parameters, you can now make the actual request on a requests Session as follows:

sess = requests.Session()
response = sess.send(prepared)
print(response)

Output:

<Response [200]>

Method 3: Using the requests_toolbelt library

The requests_toolbelt.dump() class has a method called dump_all() that gives not only the response but also the HTTP parameters of the request. This method may provide a more comprehensive HTTP request than using PreparedRequest() class. Here is an example of a used case.

import requests
from requests_toolbelt.utils import dump
# pip install requests_toolbelt if the package is not installed
resp = requests.get('http://api.github.com/users')
data = dump.dump_all(resp)
print(data.decode('utf-8'))

Output:

< GET /users HTTP/1.1
< Host: api.github.com
< User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36
< Accept-Encoding: gzip, deflate
< Accept: */*
< Connection: keep-alive
< Custom-headers: Test request
<
> HTTP/1.1 301 Moved Permanently
> Location: https://api.github.com/users
> Content-Length: 0
>
<output truncated>
> HTTP/1.1 200 OK
> Server: GitHub.com
<output truncated>
> X-GitHub-Request-Id: 9B2C:0BB5:DC30C6B:DF7E54B:632D9236
>
[<Actual response data - omitted>]

The output shows that two requests were made. The first request returned 301, and therefore the request was redirected. At the bottom of the output, the actual response is printed.

Conclusion

In this article, we discussed three methods that can be used to view and analyze our HTTP requests. The first and the third methods give more information on HTTP request parameters, but the actual request is to be made. The second method gives precise request parameters we need without us sending the actual request.