Robin on Linux – Page 15 – All about technology

Experiments on Bayesian Optimization

Bayesian Optimization is a popular searching algorithm for hyper-parameters in the machine learning area.
There are also two popular Python libraries for this algorithm: Hyperopt and Optuna. So I have done some experiments on these two libraries. The trial is using LightGBM to classify tabular data, and the hyper-parameters and their ranges are:

n_estimators (10~100000)
learning_rate (0.0001~1.0)
num_leaves (2, 2000)
max_depth (2, 200)
min_child_samples (1, 10000)
reg_alpha (0.001, 10.0)
reg_lambda (0.001, 10.0)

I run 10 trials each time for 10 times, by using different algorithms in Hyperopt and Optuna. The result for Hyperopt is:

And the result for Optuna is:

Seems that for my tabular data, TPE and Annealing works best, and Hyperopt works better than Optuna

Failed to establish pod watch in Argo

After creating a brand new Kubernetes cluster in GKE, I launched an Argo workflow but saw these errors:

Argo will create two containers for a step: ‘main’ container and ‘wait’ container. But why the creation of ‘wait’ failed? The reason is the default service account for Argo haven’t permission for `get pod`, as this comment said.
So I just run

kubectl create rolebinding default-admin --clusterrole=admin --serviceaccount=default:default

to give default service account ‘admin’ privilege. And everything goes successfully now.

A problem about running Argo

After I launched an Argo workflow, it just hanged on ContainerCreating stage. After waiting for more than 10 minutes, it hasn’t changed at all. Then I found this article.
After using

kubectl describe pods

I finally found out the problem in Message column:

MountVolume.SetUp failed for volume "gcp-key" : secret "service-key" not found

That’s because I forgot to add secret to this Kubernetes cluster…

Some tips about BigQuery on GCP

Migrate SQL script from AWS Redshift to BigQuery

CONVERT_TIMEZONE('AEDT',getdate())::DATE

in Redshift should be changed to

current_date("Australia/Sydney")

in BigQuery.
Since BigQuery doesn’t force type conversion, some NULL value in Redshift could be a NULL value or a ‘NULL’ string in BigQuery. Make sure you use both

column is NULL

and

column = 'NULL'

for checking.
In BigQuery, we can also use UDF like this:

create temp function change_date(the_date DATE, offset_day INT64) AS (
    DATE_ADD(the_date, INTERVAL offset_day DAY)
);
create temp function next_thursday(the_date DATE) AS (
    change_date(DATE_TRUNC(the_date, WEEK(THURSDAY)), 7)
);

Performance improvement of BigQuery SQL
Remove ‘DISTINCT’ in SQL and de-dup data later in Pandas could boost whole performance for data processing. Even ‘CAST’ in BigQuery would hurt the performance. The best way to find the bottlenecks for your SQL is by looking at the ‘Execution details‘ in GUI.
Loading speed
For pandas-gbq, we can accelerate the speed of reading BigQuery table by adding argument ‘use_bqstorage_api=True’ in ‘read_gbq()’ function:

df = pandas_gbq.read_gbq(bqsqlfile, project_id='myproject', use_bqstorage_api=True)

The nn.Sigmoid() of PyTorch on Android device

I have trained an EfficientNet model to classify more than ten thousand different categories of birds, by using PyTorch. To run this model on the mobile device, I built a program by learning the PyTorch iOS demo at first, make sure it runs well, and then try to build another Android program by learning the PyTorch Android demo. But after replacing the demo model with my model, the Android program prints out the result as all ‘NaN’.
To figure out the reason step by step. I tested a few widely used deep learning models:

Model Name	Result on Android demo
EfficientNet	Nan
ResNet-50	Normal
ResNet-101	Normal
RegNet+SE	NaN
RegNet	Normal

Seems that the only difference between ‘Nan’ and ‘Normal’ is the Squeeze-Excitation module. But Squeeze-Excitation module is quite simple. It’s just:

        self.f_ex = nn.Sequential(
            nn.Conv2d(w_in, w_se, 1, bias=True),
            nn.ReLU(inplace=cfg.MEM.RELU_INPLACE),
            nn.Conv2d(w_se, w_in, 1, bias=True),
            nn.Sigmoid(),
        )

Aha, I saw the nn.Sigmoid. That’s the only layer which haven’t been used in ResNet.
Let’s dive into Sigmoid function

Looks the $e^{-x}$ is the problem: it will became underflow if the X is too big!
Until now, I still have another question: why the ‘NaN’ appears in the Android program but not the iOS program. Perhaps it’s about JVM or the emulator that Android Studio used.

Use `psql` to download data as CSV file

Although SQL WorkBench is a handy tool for querying AWS Redshift, we still need to CLI tool for automation.
To install psql on MacOS, we need to

brew install postgresql

Then we could download data without using \copy but only –csv

echo "select * from example" | psql -h my.endpoint.com -p 5439 --csv mydatabase myuser > output.csv

Recent learned tips abou Numpy and Pandas

Precision
After running this snippet:

import numpy as np
a = np.array([0.112233445566778899], dtype=np.float32)
b = np.array([0.112233445566778899], dtype=np.float64)
print(a, b)

It print out:

[0.11223345] [0.11223345]

Why np.float32 and np.float64 have the same output? The answer is: displaying of numpy array need to set options.
Let’s set option before print:

import numpy as np
a = np.array([0.112233445566778899], dtype=np.float32)
b = np.array([0.112233445566778899], dtype=np.float64)
np.set_printoptions(precision=18)
print(a, b)

The result has became:

[0.112233445] [0.1122334455667789]

which looks much reasonable.
Furthermore, why it prints out ‘0.1122334455667789’ which has only ’16’ precision instead of ’18’? Because the float64 only support about 15~16 precisions, as this reference said.

Hidden metadata
There are two parquet files which look different after using ‘cksum’ to compare. But after we export them as CSV files:

import pandas as pd
df = pd.read_parquet("my.parquet")
df.to_csv("my.csv")
...

The two output CSV files are exactly the same.
Then what happened in those previous two parquet files? Dose parquet file have some hidden metadata in it?
As a matter of fact, parquet file will save the ‘index’ of a DataFrame of Pandas while CSV file will not. If we drop the index before writing out the parquet file:

df.reset_index(drop=True)
df.to_parquet("my.parquet")
...

These two parquet files would become identical.

Problems about using treelite in Kubernetes

treelite is an easy-to-use tool to accelerate prediction speed of XGBoost/LightGBM models. Three days ago, I tested it in my 4-CPU-cores virtual machine and found out that it could reduce the running time half. But after I deployed it into our Kubernetes cluster, it ran even slower than LightGBM library!
There are some strange phenomenons I noticed in the Kubernetes environment:

If I run only one pod, it will run as fastly as previous test in my virtual machine. But if I run 100 pods simutaneously, they will all run very slow.
After login to a pod for profiling, I found out that CPU usage of the application would start from 400% (for I set the cpu request and limit to "4") and gradually descend to less than 10%
There was no memory-swapping happend in any node
I used sleep 3600 to hang all the pods, login to one pod with kubectl exec -ti <pod_name> /bin/bash and run my application manually. It could still run very fast.

This weird problem haunted me, dejected me, and frustrated me for two days and almost ruined one night of my sleeping. Only until I turned to look into the code of treelite, the answer jumped out:

# src/thread_pool/thread_pool.h
    for (int i = 0; i < num_worker_; ++i) {
      thread_[i] = std::thread(task_, incoming_queue_[i].get(),
                                      outgoing_queue_[i].get(),
                                      context_);
    }
    /* bind threads to cores */
    SetAffinity();

The treelite wil pin its processes to each CPU-core by default!
This explained all the thing: when we start 100 pods, they were all trying to ping their 4 processes to CPU-core 0-3. But in the container environment, every pod could see all the CPU-cores of the node (Unlike a virtual machine, in which every VM could only see 4 CPU-cores of itself). Hence they all pin their processes to first 4 CPU-cores of that node. Using only 4 CPU-cores to run all 100 pods? That’s a terrible disaster!

The conclusion is, some acceleration tricks that work very well in classic-computer-era couldn’t be used in container-era.

Use both ‘withParam’ and ‘when’ in Argo Workflows (on Kubernetes)

In Argo, we can use ‘withParam’ to create loop logic:

    - - name: generate
        template: gen-number-list
    # Iterate over the list of numbers generated by the generate step above
    - - name: sleep
        template: sleep-n-sec
        arguments:
          parameters:
          - name: seconds
            value: "{{item}}"
        withParam: "{{steps.generate.outputs.result}}"

But in my YAML, it also use when in Argo:

    - - name: generate
        template: gen-number-list
        when: "{{workflow.parameters.NEED_RUN}} == 1"
    # Iterate over the list of numbers generated by the generate step above
    - - name: sleep
        template: sleep-n-sec
        arguments:
          parameters:
          - name: seconds
            value: "{{item}}"
        withParam: "{{steps.generate.outputs.result}}"
        when: "{{workflow.parameters.NEED_RUN}} == 1"

When the NEED_RUN is 0, the Argo will report error since it can’t find the {{steps.generate.outputs.result}}. Seems the YAML parser of Argo will try to parse withParam before when phrase.
Fortunately we don’t need to modify Argo or Kubernetes to solve this problem — we just need to let template gen-number-list generate a fake output (empty array):

      script:
        image: "{{workflow.parameters.BASEIMAGE}}"
        command: [bash]
        source: |
            if [ $NEED_RUN -eq 1 ]; then
                python3 -u output.py
            else
                echo "[]"
            fi
        env:
          - name: NEED_RUN
            value: "{{workflow.parameters.NEED_RUN}}"

Image pull policy in Kubernetes

Recently, we use Kubernetes for our project. Yesterday, a problem haunted me severely: even I have pushed the docker image to the GCR (Goolge Container Registry), the pod in Kubernetes will still use the stale image.
I tried many ways to solve the problem: removing the image in GCR, removing the image in local laptop, rebuild the image again and again. And finally I have found the reason and also realised that I am still a stupid starter on Kubernetes.
The reason for pod to use stale docker image is: the Kubernetes will (and should, I think) cache the docker images it used before for speed. Hence if you want it to re-pull image forcedly. You should use configuration item imagePullPlicy(ref), like:

spec:
  containers:
    - name: uses-private-image
      image: gcr.io/my-project/my-feature/develop:1.0.1
      imagePullPolicy: Always

Fortunately I can debug my docker image correctly now…