After running this snippet:
import numpy as np a = np.array([0.112233445566778899], dtype=np.float32) b = np.array([0.112233445566778899], dtype=np.float64) print(a, b)
It print out:
Why np.float32 and np.float64 have the same output? The answer is: displaying of numpy array need to set options.
Let’s set option before print:
import numpy as np a = np.array([0.112233445566778899], dtype=np.float32) b = np.array([0.112233445566778899], dtype=np.float64) np.set_printoptions(precision=18) print(a, b)
The result has became:
which looks much reasonable.
Furthermore, why it prints out ‘0.1122334455667789’ which has only ’16’ precision instead of ’18’? Because the float64 only support about 15~16 precisions, as this reference said.
There are two parquet files which look different after using ‘cksum’ to compare. But after we export them as CSV files:
import pandas as pd df = pd.read_parquet("my.parquet") df.to_csv("my.csv") ...
The two output CSV files are exactly the same.
Then what happened in those previous two parquet files? Dose parquet file have some hidden metadata in it?
As a matter of fact, parquet file will save the ‘index’ of a DataFrame of Pandas while CSV file will not. If we drop the index before writing out the parquet file:
df.reset_index(drop=True) df.to_parquet("my.parquet") ...
These two parquet files would become identical.