machine learning

Some thoughts about cuDF and cuML

I just received an email from NVIDIA about their RAPIDS. Although the cuDF and cuML look fantastic for a data scientist. I am still doubtful about them.

In our daily work, we usually process small DataFrame by Pandas, so cuDF will be too expensive since it needs GPU. And even we need to join two large DataFrame, we tend to use BigQuery, for it’s distributed and relatively cheap. The only proper case for cuDF I think is some heavy operations on less than 8GB data. Who need so many heavy operations on a DataFrame? I don’t know.

For cuML, it’s more like a GPU version scikit-learn. Actually, for tabular data we use XGBoost/LightGBM, for non-structure-data we use PyTorch/Tensorflow. Who will even use scikit-learn? Not even mention the cuML.

A stupid mistake in the new deep learning experiment

After my old colleague, JianMei prepared about 1TB data of the birds’ sound records (every mp3 file will be transferred to an image by using spectrogram and split into chunks with each chunk 2.5 seconds period. After all, every file is a 1250×78 multi-dimension array), I started training with almost the same code using in bird image classification.

The train-accuracy rises very slowly so I add a line of code to normalize every input sample:

image = (image - image.mean()) / image.std()

After that, the train-accuracy could rise faster, but the eval-accuracy still quite low.

In order to find out the root of the problem, I started to train from only two classes: “Black-capped Donacobius” and “Blue-eared Barbet”

Eval accuracy:0.965278 | Train accuracy:1.000000

The result seems pretty good. So I increate the number of classes to 20

Eval accuracy:0.211102 | Train accuracy:0.840278
Eval accuracy:0.203834 | Train accuracy:0.885417
Eval accuracy:0.191514 | Train accuracy:0.904247
Eval accuracy:0.193245 | Train accuracy:0.916667
Eval accuracy:0.210894 | Train accuracy:0.916667
Eval accuracy:0.190269 | Train accuracy:0.932292

Are some types of bird hard to generalize in deep learning model? Then I began to consider how to find these “hard to train” bird type: maybe start from 2 classes and increase the number of classes step by step, and then draw a few curves about the train-accuracy and eval-accuracy…

Suddenly I realized that I just use normalization in the training sample but not evaluation sample!

What a stupid mistake. It wasted me a whole stuffy afternoon for nothing. I really should remember this lesson: do what you do in training samples to evaluation samples, except dropout.

A struggle to keep the accuracy

In this August, we have got 0.83 evaluation accuracy for DIB-10K dataset. But since last month, we have updated the dataset and the accuracy could only get to 0.82.

The first doubtful point is the Weight Standardization method we used for micro-batch (since the model is too big). So I turned to try gradient-accumulation and use this snippet as an example because it won’t need me to change my code heavily:

model.zero_grad()                                   # Reset gradients tensors
for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)                     # Forward pass
    loss = loss_function(predictions, labels)
    loss = loss / accumulation_steps
    loss.backward()                                 # Backward pass
    if (i+1) % accumulation_steps == 0:
        optimizer.step()
        model.zero_grad()
        if (i+1) % evaluation_steps == 0:
            evaluate_model()

But after changing my code and retrain the model, the accuracy still keep around 0.82:

Epoch     4: reducing learning rate of group 0 to 1.0000e-01.
[2020-12-16 05:53:29] Eval accuracy: 0.8283 | Train accuracy: 0.8187
[2020-12-16 10:01:40] Eval accuracy: 0.8284 | Train accuracy: 0.8938
[2020-12-16 14:11:35] Eval accuracy: 0.8284 | Train accuracy: 0.8313
Epoch     7: reducing learning rate of group 0 to 5.0000e-02.
[2020-12-16 18:21:47] Eval accuracy: 0.8285 | Train accuracy: 0.8750
[2020-12-16 22:31:19] Eval accuracy: 0.8285 | Train accuracy: 0.8313
[2020-12-17 02:41:37] Eval accuracy: 0.8284 | Train accuracy: 0.8625
Epoch    10: reducing learning rate of group 0 to 2.5000e-02.
[2020-12-17 06:52:05] Eval accuracy: 0.8286 | Train accuracy: 0.8500
[2020-12-17 11:02:11] Eval accuracy: 0.8285 | Train accuracy: 0.8063
[2020-12-17 15:12:23] Eval accuracy: 0.8286 | Train accuracy: 0.8375
Epoch    13: reducing learning rate of group 0 to 1.2500e-02.
[2020-12-17 19:22:04] Eval accuracy: 0.8285 | Train accuracy: 0.8313

This makes me really desperate. Maybe I should temporarily put this task aside and go on other works.

Export YOLOv5 models for mobile device

Somebody has finished the work about exporting YOLOv5 models to tflite model. To use it, we only need to:

git clone --single-branch --branch tf-export https://github.com/zldrobit/yolov5.git
cd yolov5
# it will download all pytorch models
sh -x weights/download_weights.sh
# export a tflite model from yolov5l
PYTHONPATH=. python3 models/tf.py --weights yolov5l.pt --cfg models/yolov5l.yaml --img 640
# there will be a tflite model file
ls yolov5l-fp16.tflite

The model file yolov5l-fp16.tflite is 91MB, which is a little too big but still could be put into a mobile phone.

The awesome YOLOv5

I just found a repository YOLOv5 from Github. It’s not just the models are accurate and fast but also easy to get and to use.

Just download the code, install some dependent libraries. And you can just run a simple command:

python3 detect.py --weights yolov5l.pt

Then it will automatically download the YOLO v5 Large model, process all images in inference/images, and put the annotated images into inference/output.

Let’s see some images predicted by YOLO v5 Large:

And every image cost no more than 2 seconds to predict on my laptop. Isn’t it awesome? At least it’s much better than my experiments…

TabNet: a new neural-network architecture for tabular data

The neural network seems mostly to be used on Computer Vision and Natural Language Processing scenarios, while tree-models like GBDT are mainly used for tabular data.

But why?

Although this article tries to give an explanation of this, it hasn’t been so promising to me. In my humble opinion, the neural network could finally surpass, or at least be competitive, to the GBDT model.

For example, the paper <TabNet: Attentive Interpretable Tabular Learning> describe a Transformer-like model to simulate the tree-model. The PyTorch implementation is here. I have used it on our own data and it finally reached 90% accuracy ( the accuracy of LightGBM is 93%). In spite of the lower accuracy, this is the first neural model reached 90% accuracy when used on our private data. The author has already done a great job.

Understanding Transformer

In the paper Attention Is All You Need, the Transformer neural network had been introduced for the first time in 2017. One year later, the BERT appeared. And last year I gave a simple presentation in my previous company about the Transformer and BERT. As showed below:

Transformer and BERT from Hao(Robin) Dong

A couple of days before I started to review the Transformer paper and found out that I need to recommend the article The Illustrated Transformer again. This article really helps me to understand a lot of details in the Transformer.

But there is still a question jump out of my brain: what’s the use of decoder in Transformer? How the information flows through encoder to decoder ? After thinking for quite a while, I figured it out: Transformer was used for Machine Translation task at the first place. The encoder is used to “transform” sentence of source language to a couple of Keys and Values; the decoder will “transform” a word of target language to a Query. By using a Query and a couple of Keys and Values, it could get a vector, which is actually the embedding of next word in target language.

Here is a digram draw by me. Hope it could explain my own confusion.

“Ich bin ein guter Kerl” in German means “I am a good guy”. By encoding all German words to a couple of Keys and Values, and decode “good” to a Query, the Transformer could finally output the embedding vector of “guy”.

Using GPU for LightGBM

One of my team members had accomplished some tests on using GPU for LightGBM training. The result is quite good that GPU could accelerate training speed to 2 times fast.

But this also rises up my interesting about how LightGBM uses GPU for training. Since the GBDT algorithm only use operations like condition checking, data sorting, split point searching etc, it doesn’t have matrix operations which is the strong point of GPU.

Then Jimmy (one of my colleague) send me a paper. This is exactly how LightGBM uses GPU — using GPU for histogram algorithm. The story is: to find the best split point for a feature (or a column of a dataset), LightGBM needs to collect them into bins with different value ranges. This process could be concurrently executed so it could be put into the GPU.

So, GPU could not only used for heavy matrix operation situation but also highly parallel case. Thanks to Jimmy, the paper explained my doubt.

The code for GPU kernel of LightGBM is in three files:

src/treelearner/ocl/histogram16.cl
src/treelearner/ocl/histogram64.cl
src/treelearner/ocl/histogram256.cl

Since it use OpenCL framework to implement, the LightGBM could use both Nvidia and AMD’s GPU to train.

The first trial for PyCaret

The code for using PyCaret is quite simple:

df = pd.read_csv(TRAIN_CSV_FILE)
setup(data=data, target="TARGET", session_id=1023)
compare_models(verbose = False)

But it reported error in the first run:

ValueError: array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size.

This is because it has a custom_id column in my data that have string value like “11100000033445566”. Seems PyCaret recognize it as integer type.

After drop that column, I got the result of models comparison:

Looks the GradientBoostingClassifier works as well as LightGBM.

Experiments on Bayesian Optimization

Bayesian Optimization is a popular searching algorithm for hyper-parameters in the machine learning area.
There are also two popular Python libraries for this algorithm: Hyperopt and Optuna. So I have done some experiments on these two libraries. The trial is using LightGBM to classify tabular data, and the hyper-parameters and their ranges are:

n_estimators (10~100000)
learning_rate (0.0001~1.0)
num_leaves (2, 2000)
max_depth (2, 200)
min_child_samples (1, 10000)
reg_alpha (0.001, 10.0)
reg_lambda (0.001, 10.0)

I run 10 trials each time for 10 times, by using different algorithms in Hyperopt and Optuna. The result for Hyperopt is:

And the result for Optuna is:

Seems that for my tabular data, TPE and Annealing works best, and Hyperopt works better than Optuna