This article will discuss how to convert a sequence of bytes into a Python dictionary. But before that, we need to understand the difference between strings and bytes in Python.
Strings vs Bytes
Python string (also termed literal string) is a sequence of literal characters, whereas bytes (also byte string) is a sequence of bytes. In Python, strings are enclosed by single or double quotes, while bytes are surrounded by quotes and preceded by the letter “b.” Here are two more points to explain the difference between strings and bytes.
- Bytes are machine-readable, whereas bytes are human-readable,
- Strings must be encoded to bytes before being stored on a disk and decoded into human-readable form (strings).
Figure : A Figure showing differences between Python strings and bytes. Encoding and decoding and inverse operations in this sense.
|
1 2 3 4 5 6 7 |
string1 ="Fort-Ternan" print(string1) print(type(string1)) bytes1 = "Fort-Ternan".encode("utf-8") print(bytes1) print(type(bytes1)) |
Output:
Fort-Ternan <class 'str'> b'Fort-Ternan' <class 'bytes'>
The above example shows a string and a byte (UTF-8 encoded string). When printing the bytes, Python represented bytes as b’Fort-Ternan’.
Under the hoods, Python decoded the bytes to string before printing so that we can read it; otherwise, by nature, they are not human-readable.
Converting Bytes to Dictionary in Python
There are two libraries to use when converting bytes into a dictionary: json and ast. Let’s discuss how to use each one of them.
Method 1: Using Json.loads() function
The json.loads() method converts an object (string, bytes, or bytearray) to the Python dictionary. The input should be encoded using UTF-8, UTF-16, or UTF-32. Here is an example of how to use the function to convert bytes to strings.
Note: The ability of the function to convert bytes was added in Python 3.6. Before then, string data types were the only accepted input.
|
1 2 3 4 5 6 7 8 |
import json # Byte string representation of data a= b'{"Name": "Smith", "Reg": 206}' # Converting a to a dictionary b = json.loads(a) print(b) print(type(b)) |
Output:
{'Name': 'Smith', 'Reg': 206}
<class 'dict'>
If you are running Python older than 3.6, you must pass string (not bytes) into json.loads(). To do that, we need to decode the input to the function. We will change line 5 to b = json.loads(.decode(‘utf-8’)) assuming that UTF-8 encoding was used.
Note: The json.loads() method expects the byte string to be enclosed on single quotes and all other strings inside to be enclosed using double quotes (keys and values to be surrounded by double quotes). The code below, therefore, will lead to an error.
|
1 2 3 4 |
import json a = b"{'Name': 'Smith', 'Reg': 206}" b = json.loads(a.decode('utf-8')) |
Output:
json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)
If you find yourself in a situation like that, you need to replace single quotes with double ones, as shown below.
|
1 2 3 4 5 6 7 8 |
import json a = b"{'Name': 'Smith', 'Reg': 206}" # Decode a to a string, then replace single # quotes with double quotes. a = a.decode("utf-8").replace("'",'\"') b = json.loads(a) |
Method 2: Using ast module
The ast library has a function called literal_eval that can be repurposed to convert byte strings into a Python dictionary. Unlike json.loads(), ast.literal_eval() expects the input to be a string (we need to decode bytes to string), and it does not throw errors based on the usage of single or double quotes.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# Load literal_eval function from ast from ast import literal_eval # Our byte string a = b'{"Name": "Smith", "Reg": 206}' # Convert byte (a) to string by decoding it a = a.decode("utf-8") # Convert the string (a) to a dictionary. b = literal_eval(a) print(b) |
Output:
{'Name': 'Smith', 'Reg': 206}
Conclusion
In this article, we have seen how to use json.loads() and ast.literal_eval() functions to convert bytes into a Python dictionary.
We noted that json must use double quotes inside the byte string and that it takes any of the following as input: string, bytes, or bytearray.
On the other hand, ast.literal_eval() takes a string as input, and single or double quotes can be used inside the bytes string. To convert a byte string to a literal string, we need to decode it.
Lastly, note that the built-in json and ast modules can only handle Python primitives types with a direct dictionary equivalent.