This article discusses how to use struct and Numpy packages in Python to convert bytes to floats and vice-versa. Before we do that, however, let’s briefly discuss the difference between bytes and floating-point numbers.
Difference Between Bytes and Floats
Human beings and computers understand numbers differently. Whereas human beings understand numbers in base 10 (0 to 9) – the floating numbers, computers represent numbers in binary/bit form (sequences of 0s and 1s).
A byte is a data unit consisting of binary digits (usually 8).
In most cases, interpreters like Python keep converting bytes of data to numbers/characters we can understand before printing them out.
Now that we understand the difference between floats and bytes, let us discuss how to use struct and Numpy to convert floats to bytes and vice-versa.
Struct Packing and Unpacking
The struct module in Python is motivated by the struct data type in C programming.
Using struct to convert floats into bytes
This is done using the struct.pack() method, which has the following syntax:
struct.pack(<format>, val1, val2, …)
Where <format> is the formatting string for packing the floats – val1, val2, …
Let’s see an example.
|
1 2 3 4 5 6 7 8 |
import struct # "f" means converting a float into bytes result1 = struct.pack("f", 3.1415927410125732) print(result1) # "i" stands for integer - we are converting an integer to bytes result2 = struct.pack("i", 65) print(result2) |
Output:
b'\xdb\x0fI@' b'A\x00\x00\x00'
The above examples show how to convert single values (float and integer) into bytes.
The formatting string “f” means we want to convert a float, and “i” means passing an integer to the struct.pack() function.
The syntax shows that we can also convert multiple values to bytes. For example,
|
1 2 3 |
import struct struct.pack("ffi", 2.718, 3.142, 65) |
This means we are converting a float 2.718, another float 3.142, and an integer 65 into bytes. As said before, “f” stands for float, and “i” represents float.
Note that you can use repeating data types with numerical. For example, the “fff” formatting string can be mapped to “3f” – for converting 3 floating numbers.
Also, you can represent mixed datatypes in the format string. For example, “3f i” or “3fi” is another representation of “fffi”.
Here are more examples.
|
1 2 3 4 5 6 7 8 9 10 11 |
import struct # convert 3 values -float, float and int - to bytes result3 = struct.pack("ffi", 2.718, 3.142, 65) print(result3) # Converting 3 values - 3 floats - to bytes result3 = struct.pack("3f", 2.718, 3.142, 14.1) print(result3) # Converting 4 values - 3 integers and a float - to bytes result3a = struct.pack("3i f", 56, -14, 16, 2.89) print(result3a) |
Output:
b'\xb6\xf3-@\x87\x16I@A\x00\x00\x00' b'\xb6\xf3-@\x87\x16I@\x9a\x99aA' b'8\x00\x00\x00\xf2\xff\xff\xff\x10\x00\x00\x00\xc3\xf58@'
We can also convert numbers in a tuple into bytes, as shown below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
import struct # Take a tuple of values and convert data in it record = (-234, 0.98726, -13.56) # Unpack the values from the tuple using *, then convert them to bytes. result4 = struct.pack(f"3f", *record) print(result4) # Convert several tuples to byes in a for-loop. records = [(2.3,), (-67, 0, 111), (3.24, 78)] for record in records: n_records = len(record) # Infer the format string using the len of the record result = struct.pack(f"{n_records}f", *record) print(f"{record} =", result) |
Output:
b'\x00\x00j\xc3\x12\xbd|?\xc3\xf5X\xc1' (2.3,) = b'33\x13@' (-67, 0, 111) = b'\x00\x00\x86\xc2\x00\x00\x00\x00\x00\x00\xdeB' (3.24, 78) = b')\\O@\x00\x00\x9cB'
Fun fact: (2.3)=2.3, which means Python doesn’t see (2.3) as a tuple; it sees it as a float. If you want to define a single-element tuple, add a comma after the element, that is, (2.3,).
Convert bytes to floats using struct.unpack()
The struct.unpack() for converting bytes to floats has the following syntax.
struct.unpack(<format>, buffer)
The function unpacks bytes on the buffer according to the <format> provided. The method presumes that the buffer was packed with the same <format>.
Let’s see some examples.
|
1 2 3 4 5 6 |
import struct result2b = struct.unpack("f", b"\xdb\x0fI@") print(result2b) result3b = struct.unpack("ffi", b"\xb6\xf3-@\x87\x16I@A\x00\x00\x00") print(result3b) |
Output:
(3.1415927410125732,) (2.7179999351501465, 3.1419999599456787, 65)
Note that the results match what we got in the conversion of floats to bytes in the previous subsection.
Bonus: struct.calcsize()
The struct.calcsize() function is used to check the format size.
|
1 2 3 4 5 6 7 8 9 10 11 |
import struct # Size of 3 intergers size1 = struct.calcsize("iii") print(size1) # Size of 4 intergers size1 = struct.calcsize("4f") print(size1) # Size of 2 characters size1 = struct.calcsize("cc") print(size1) |
Output:
12 16 2
The standard size for a float is 4 bytes, an integer is 4 bytes, and char is 1 byte. That is why “iii” is 12 bytes, “4f” is 16 bytes, and “cc” is 2 bytes.
Using NumPy to Convert Floats to Bytes and Vice-versa
NumPy and struct modules yield the same result. If you want to parse bytes to floats and don’t know the byte string format for the struct module, you can use NumPy.
You should also use NumPy if you want to control the size of the bytes and floats.
Converting floats into bytes in Numpy
This is done using the array.tobytes() function. For example
|
1 2 3 4 5 6 7 |
# Bytes to floats import numpy as np b1 = np.array(3.1415927410125732, dtype=np.float32).tobytes() print(b1) b2 = np.array([2.718, 3.142, 14.1], dtype=np.float32).tobytes() print(b2) |
Output:
b'\xdb\x0fI@' b'\xb6\xf3-@\x87\x16I@\x9a\x99aA'
Note that the output matches the results we got with the struct package, but we did not have to pass the format here.
The dtype=np.float32 means we are storing a 32-bit float (equal to 4 bytes) which matches the standard size used by struct for float.
If you use a different dtype, you will get a different result. For example,
|
1 2 3 4 5 |
import numpy as np # 64-bit = 8 bytes float. b3 = np.array(3.1415927410125732, dtype=np.float64).tobytes() print(b3) |
Output:
b'\x00\x00\x00`\xfb!\t@'
Converting bytes to float
The function np.frombuffer() serves the purpose in this case. For example
|
1 2 3 4 5 6 7 |
# Floats to bytes import numpy as np f1 = np.frombuffer(b"\x00\x00\x86\xc2\x00\x00\x00\x00\x00\x00\xdeB", dtype=np.float32) print(f1) f2 = np.frombuffer(b"\xdb\x0fI@", dtype=np.float32) print(f2) |
Output:
[-67. 0. 111.] [3.1415927]
Note that the dtype used to pack the bytes to floats must be used when unpacking. If this is violated, NumPy raises an error.
|
1 2 3 4 5 6 7 8 9 10 |
import numpy as np b5 = np.array([-25.6, 45, -3], dtype=np.float32).tobytes() print(b5) # Use the same dtype used to pack the array, i.e., float32. unpack_b5 = np.frombuffer(b5, dtype=np.float32) print(unpack_b5) # Using np.float64 will lead to an error. unpack_b5 = np.frombuffer(b5, dtype=np.float64) print(unpack_b5) |
Output:
b'\xcd\xcc\xcc\xc1\x00\x004B\x00\x00@\xc0' [-25.6 45. -3. ] ValueError: buffer size must be a multiple of element size
Conclusion
This article discussed how to use struct and NumPy to pack floats to bytes and unpack bytes to floating-point numbers. The two packages yield the same result but with slightly different ways of use. You can go through the examples given in the guide and pick your poison.