RobinDong

Be careful when you use “isin()” method in Pandas

import pandas as pd
df_excl = pd.DataFrame({"id": ["12345"]})
df = pd.DataFrame({"id": ["12345", "67890"]})
result = df[~df.id.isin(df_excl[["id"]])]
print(result)

Guess what’s the result of above snippet? Just a dataframe with “67890”? No, the result is

      id
0  12345
1  67890

Why the “12345” has not been excluded? The reason is quite tricky: df_excl[["id"]] is a DataFrame but what we need in isin() is Series! So we shouldn’t use [[]] here, but []

The correct code should use df_excl["id"], as below:

...
result = df[~df.id.isin(df_excl["id"])]
print(result)

An error about multiprocessing of Python

Our python program reported errors when running a new dataset:

[77 rows x 4 columns]]'. Reason: 'error("'i' format requires -2147483648 <= number <= 2147483647",)'
multiprocessing.pool.MaybeEncodingError: Error sending result: '[                          id  ... email_send_date
    raise self._value
  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 644, in get
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/local/lib/python3.6/multiprocessing/pool.py", line 266, in map
    dfs = pool.map(partial(pd.read_parquet, **kwargs), file_list)

Then I found this issue in the python community quickly https://bugs.python.org/issue17560 https://bugs.python.org/issue17560. Seems the reason is the multiprocessing mechanism in python only support 32bit to encode object length. And this problem existed even up to Python-3.8

The solution is just using multithreads instead of multiprocessing

Previous code:

with Pool(processes=n_jobs) as pool:
  pool.map(...)

Solution code:

with ThreadPoolExecutor(max_workers=n_jobs) as pool:
  pool.map(...)

Source code reading of LightGBM

Finally I get a few hours to look into the code of LightGBM.

I used to have some questions about LighGBM, and now fortunately I can answer some of them by myself. Even some answers may be wrong, that is still better than no answer at all 🙂

Q: Will LightGBM contruct a couple of trees as one model?

A: No. It will only contruct one tree as a model for a dataset

Q: How would LightGBM choose the feature that has the highest gain in entropy?

A: It will simply iterate all features (with a loop, in code) and try to find the best split for all of them. After that, it will pick the feature and the split with the highest gain

"src/treelearner/serial_tree_learner.cpp"
158 Tree* SerialTreeLearner::Train(const score_t* gradients, const score_t *hessians) {
...
185   int init_splits = ForceSplits(tree_ptr, &left_leaf, &right_leaf, &cur_depth);
186
187   for (int split = init_splits; split < config_->num_leaves - 1; ++split) {
188     // some initial works before finding best split
189     if (BeforeFindBestSplit(tree_ptr, left_leaf, right_leaf)) {
190       // find best threshold for every feature
191       FindBestSplits(tree_ptr);
192     }
...

"src/treelearner/serial_tree_learner.cpp"
322 void SerialTreeLearner::FindBestSplits(const Tree* tree) {
323   std::vector<int8_t> is_feature_used(num_features_, 0);
324   #pragma omp parallel for schedule(static, 256) if (num_features_ >= 512)
325   for (int feature_index = 0; feature_index < num_features_; ++feature_index) {
326     if (!col_sampler_.is_feature_used_bytree()[feature_index]) continue;
327     if (parent_leaf_histogram_array_ != nullptr
328         && !parent_leaf_histogram_array_[feature_index].is_splittable()) {
329       smaller_leaf_histogram_array_[feature_index].set_is_splittable(false);
330       continue;
331     }
332     is_feature_used[feature_index] = 1;
333   }
...

Q: In the model file of LightGBM, it shows “num_leaves=63” in every iteration. Shouldn’t it change the depth and leaves of model for every iteration?

A: Can’t answer yet. Still need to look into the code to see why…

Accelerate reading of NumPy array from files

In the training process, I need to read array data from .npy file and get a part of it:

import numpy as np
data = np.load("sample1.npy")
sound1 = data[start1: end1]
sound2 = data[start2: end2]

Since the .npy files are large, it became slowly to read a large file but only get some small parts of it. Is there a simple way to let us only read these small parts?

Yes, it’s just a simple line change mmap_mode:

import numpy as np
data = np.load("sample1.npy", mmap_mode="r")  # just change this line
sound1 = data[start1: end1]
sound2 = data[start2: end2]

Now, the load() function will not generate any IO for disk. But when we process the segment (like sound1 or sound2), it will load and only load the pages that contains the segment, which decrease the total IO tremendously.

After I change this line of code, the reading bandwidth dropped from 300MB/s to less than 100MB/s

Change the schema of BigQuery tables

We can easily add new column for a table in BigQuery:

ALTER TABLE mydataset.mytable
      ADD COLUMN new_col STRING

But when you want to delete or rename an existed column, there is no SQL to implement it. The only way to delete or rename an existed column is to use the bq command:

bq show --format=prettyjson mydataset.mytable > schema.json
# Edit the schema.json to only leave a list of columns
bq mk --table mydataset.new_mytable schema.json
# Export data from `mytable` to `new_mytable`
bq rm --table mydataset.mytable
bq cp --table mydataset.new_mytable mydataset.mytable

And remember to backup your data before operating!

Strange time output in a container of Kubernetes cluster

After running a workflow in Argo, I found out the output of the “date” command is totally wrong:

# date
Wed Mar 3 00:41:27 2021
# TZ='America/Los_Angeles' date
Wed Mar 3 00:41:36 2021
# TZ='America/New_York' date
Wed Mar 3 00:41:38 2021
# TZ='Australia/Sydney' date
Wed Mar 3 00:42:01 2021

No matter what timezones I used, the time from the “date” command looks didn’t change at all.

This phenomenon only existed in the container of our k8s cluster. In my personal VM, it works just fine.

Finally, my colleague Tianchu found out the reason: the docker image used by the container of the k8s cluster didn’t even install the timezone file! And the solution is simple: just install the timezone data

DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends tzdata

Some thoughts about cuDF and cuML

I just received an email from NVIDIA about their RAPIDS. Although the cuDF and cuML look fantastic for a data scientist. I am still doubtful about them.

In our daily work, we usually process small DataFrame by Pandas, so cuDF will be too expensive since it needs GPU. And even we need to join two large DataFrame, we tend to use BigQuery, for it’s distributed and relatively cheap. The only proper case for cuDF I think is some heavy operations on less than 8GB data. Who need so many heavy operations on a DataFrame? I don’t know.

For cuML, it’s more like a GPU version scikit-learn. Actually, for tabular data we use XGBoost/LightGBM, for non-structure-data we use PyTorch/Tensorflow. Who will even use scikit-learn? Not even mention the cuML.

To put Back-Quote in a string of Bash

It’s very simple to print a word “hello” in Bash:

echo "hello"

But how to print a word with Back-Quotes?

echo "`hello`"
# It will report error because Bash will try to run 'hello' as a command
bash: hello: command not found

There are two common solutions for this problem. One is using back-slash:

echo "\`hello\`"

Another is just using single quote for words:

echo '`hello`'

How to gracefully end a PySpark application

This article recommend using “return” to jump out of a PySpark application. But after I did by following what he said. It reports error:

  File "test.py", line 333
    return
    ^
SyntaxError: 'return' outside function

Seems it can’t work. After trying to run PySpark application on my own laptop, I finally got the correct answer:

import sys
if df.rdd.isEmpty():
  sys.exit(0)

An old bug about PyArrow

To save memory for my program using Pandas, I change types of some column from string to category as the reference.

df[["os_type", "cpu_type", "chip_brand"]] =
	df[["os_type", "cpu_type", "chip_brand"]].astype("category")

It could save at least half memory in my case. But when I use pyarrow to store the dataframe to parquet

df.to_parquet("my.parquet")

it reports errors:

Invalid: BinaryArray cannot contain more than 2147483646 bytes, have 2147483647

It’s a bug from old version pyarrow and had been fixed in Sep 2019. Then I tried to upgrade my pyarrow-0.12.1 to pyarrow-0.17.1 and it fixed this error.

But the story hasn’t ended up here.

For pyarrow-0.12.1, the below snippet will return a class of type <pyarrow.lib.Column>

import pyarrow.parquet as pq
table = pq.read_table(path)
table.column(0)

and this class will also contain a attribute “Column name”

But for pyarrow-0.17.1, the same code will return a class of type <pyarrow.lib.ChunkedArray> which doesn’t have a “Column name”.

This difference will make some code fail (actually, our program). Beware of this: after you upgrade pyarrow (or any other library in Python), run the test to make sure all the legacy code work properly.