Robin on Linux – Page 12 – All about technology

To solve the problem about pivot() of Pandas

Below is an example from pandas official document for pivot():

import pandas as pd
df = pd.DataFrame({'foo': ['one', 'one', 'one', 'two', 'two', 'two'],
                   'bar': ['A', 'B', 'C', 'A', 'B', 'C'],
                   'baz': [1, 2, 3, 4, 5, 6],
                   'zoo': ['x', 'y', 'z', 'q', 'w', 't']})
result = df.pivot(index='foo', columns='bar', values='baz')
print(result)

the result will be

bar  A  B  C
foo
one  1  2  3
two  4  5  6

But if there are duplicate rows in the Dataframe, it will report error:

ValueError: Index contains duplicate entries, cannot reshape

To fix this, we can just drop_duplicates() first and then pivot():

result = df.drop_duplicates().pivot(index='foo', columns='bar', values='baz')

As matter of fact, there are situations that drop_duplicates() couldn’t fix:

df = pd.DataFrame({'foo': ['one', 'one', 'one', 'two', 'two', 'two'],
                   'bar': ['A', 'A', 'C', 'A', 'B', 'C'],
                   'baz': [1, 2, 3, 4, 5, 6],
                   'zoo': ['x', 'y', 'z', 'q', 'w', 't']})

Now we will need to use groupby() and unstack() to replace pivot():

result = (df.groupby(["foo", "bar"])
    .baz
    .first()
    .unstack()
)

And the result is

bar    A    B    C
foo
one  1.0  NaN  3.0
two  4.0  5.0  6.0

Books I read in the year 2020

Since beginning my new job on 6th January, I start to learn Kubernetes for the first time. Frankly speaking, Kubernetes is very powerful and also easy to use (from my perspective). And the book “Kubernetes in action” is a really good book for beginners.

Bought seven books of “A song of ice and fire” by 35 dollars from eBay in last year, I finally start to read this epic novel. The world built by Martin is so cruel, so cold, and so attractive. The “A clash of kings” maybe the thickest single book I have read in my memory.

To learn more hardware knowledge, I have gone through the fifth version of “Computer Architecture“. Passed a lot of paragraphs, just got a preliminary view about modern CPU architecture. Hope to have the opportunity in future to read it more elaborately.

In spite of writing by the same author Wolfgang Faust, the “Tiger Tracks” is not as evocative as “The last panther“, which is the best war literature I read until today.

“Ensemble Methods” is written by a Chinese professor at Nanjing University, Zhi-Hua Zhou. I love his book because they are very easy to understand and also could give me a lot of new concepts and knowledge. Though this book is not very practical in my work, I still recommend it for expanding our view of the horizon.

Last but not least. I’d like to recommend two extra papers, which are very interesting and inspirational this year for both software and hardware areas.

Software: Dota 2 with Large Scale Deep Reinforcement Learning

Hardware: Fast Stencil-Code Computation on a Wafer-Scale Processor

Run cron jobs in Kubernetes

In the old age of a single machine, we use /etc/crontab to schedule the job. But in the new age of the distributed system, especially Kubernetes, what should we use?

The first choice in my mind is CronJob, like this one:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: hello
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: hello
            image: busybox
            imagePullPolicy: IfNotPresent
            args:
            - /bin/sh
            - -c
            - date; echo Hello from the Kubernetes cluster
          restartPolicy: OnFailure

Then my colleague recommends the CronWorkflow (need at least v2.5 of Argo) to me. Its configuration seems not very different:

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: test-cron-wf
spec:
  schedule: "* * * * *"
  concurrencyPolicy: "Replace"
  startingDeadlineSeconds: 0
  workflowSpec:
    entrypoint: whalesay
    templates:
    - name: whalesay
      container:
        image: alpine:3.6
        command: [sh, -c]
        args: ["date; sleep 90"]

But CronWorkflow have a advantage over CronJob: the UI. We can use Web UI to access and manage different types of workflows:

And the UI also makes the accessing of pod log very easy.

A struggle to keep the accuracy

In this August, we have got 0.83 evaluation accuracy for DIB-10K dataset. But since last month, we have updated the dataset and the accuracy could only get to 0.82.

The first doubtful point is the Weight Standardization method we used for micro-batch (since the model is too big). So I turned to try gradient-accumulation and use this snippet as an example because it won’t need me to change my code heavily:

model.zero_grad()                                   # Reset gradients tensors
for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)                     # Forward pass
    loss = loss_function(predictions, labels)
    loss = loss / accumulation_steps
    loss.backward()                                 # Backward pass
    if (i+1) % accumulation_steps == 0:
        optimizer.step()
        model.zero_grad()
        if (i+1) % evaluation_steps == 0:
            evaluate_model()

But after changing my code and retrain the model, the accuracy still keep around 0.82:

Epoch     4: reducing learning rate of group 0 to 1.0000e-01.
[2020-12-16 05:53:29] Eval accuracy: 0.8283 | Train accuracy: 0.8187
[2020-12-16 10:01:40] Eval accuracy: 0.8284 | Train accuracy: 0.8938
[2020-12-16 14:11:35] Eval accuracy: 0.8284 | Train accuracy: 0.8313
Epoch     7: reducing learning rate of group 0 to 5.0000e-02.
[2020-12-16 18:21:47] Eval accuracy: 0.8285 | Train accuracy: 0.8750
[2020-12-16 22:31:19] Eval accuracy: 0.8285 | Train accuracy: 0.8313
[2020-12-17 02:41:37] Eval accuracy: 0.8284 | Train accuracy: 0.8625
Epoch    10: reducing learning rate of group 0 to 2.5000e-02.
[2020-12-17 06:52:05] Eval accuracy: 0.8286 | Train accuracy: 0.8500
[2020-12-17 11:02:11] Eval accuracy: 0.8285 | Train accuracy: 0.8063
[2020-12-17 15:12:23] Eval accuracy: 0.8286 | Train accuracy: 0.8375
Epoch    13: reducing learning rate of group 0 to 1.2500e-02.
[2020-12-17 19:22:04] Eval accuracy: 0.8285 | Train accuracy: 0.8313

This makes me really desperate. Maybe I should temporarily put this task aside and go on other works.

Directly update git commit

Previously when I want to update my branch of git, I just commit new modifications again and again. This makes my branch log quite ugly.

After my colleague introduced git --amend to me, he really helps me a lot. Now, I just use

git commit -a --amend --no-edit

to update my modification into previous commit and

git push -f origin my_branch_name

to totally update my branch in remote

And, if I want to also change the message of my previous comment, I only need to

git commit -a --amend -m "Let's rock again"

Get the schema of a parquet file

Previously I just use this snippet to get all the column names of a parquet file:

import pandas as pd
df = pd.read_parquet("hello.parquet")
print(list(df.columns))

But if the parquet file is very large (maybe not very large, for example, 1GB), it will cause OOM in my small VM (about 4GB RAM).

Actually, what I want is just column names, not the whole data. Since parquet file has strongly designed format, there must be someway we can only get the schema instead of all data.

And, here it is:

import pyarrow.parquet as pq
schema = pq.read_schema("hello.parquet", memory_map=True)
print(list(schema.names))

Export YOLOv5 models for mobile device

Somebody has finished the work about exporting YOLOv5 models to tflite model. To use it, we only need to:

git clone --single-branch --branch tf-export https://github.com/zldrobit/yolov5.git
cd yolov5
# it will download all pytorch models
sh -x weights/download_weights.sh
# export a tflite model from yolov5l
PYTHONPATH=. python3 models/tf.py --weights yolov5l.pt --cfg models/yolov5l.yaml --img 640
# there will be a tflite model file
ls yolov5l-fp16.tflite

The model file yolov5l-fp16.tflite is 91MB, which is a little too big but still could be put into a mobile phone.

Attach references on arXiv.org

About one month ago I and my old colleague Jian Mei had published a paper about ornithology images on arXiv.org. But two days ago, Jian Mei found out that I have written the number of images and categories wrong.

To update the paper, I just packed my .tex and .bib files into a zip and uploaded this zip to arXiv.org to replace my old pdf. But strange things happened: all the references are lost.

Then I caught sight of the document. Since the arXiv.org only supports .bbl file instead of .bib, I need to compile the .bib to a .bbl file manually.

Could overleaf do it? Fortunately it could. Just upload the zip file and click the “Logs and output files” icon as below:

Then click the “Other logs & files” and choose “bbl file”. The bbl file will be downloaded. Finally, change the bbl file name to be the same with the .tex file (except the .bbl suffix) and pack them again to be a zip file.

This time, all references came back to the paper.

Separate pip environment

Days ago, I thought anaconda could separate different python environment, so I created a new conda environment to install new version pip package. And no doubt, it failed.

The fact is: conda could only separate different conda package environment, not pip package. To separate the pip environment, we need to use virtualenv.

# Create new env
virtualenv -p /home/my/anaconda3/bin/python3.6 venv_my
# Enter the new env
source venv_my/bin/activate

The awesome YOLOv5

I just found a repository YOLOv5 from Github. It’s not just the models are accurate and fast but also easy to get and to use.

Just download the code, install some dependent libraries. And you can just run a simple command:

python3 detect.py --weights yolov5l.pt

Then it will automatically download the YOLO v5 Large model, process all images in inference/images, and put the annotated images into inference/output.

Let’s see some images predicted by YOLO v5 Large:

And every image cost no more than 2 seconds to predict on my laptop. Isn’t it awesome? At least it’s much better than my experiments…