Uncategorized

Books I read in the year 2020

Since beginning my new job on 6th January, I start to learn Kubernetes for the first time. Frankly speaking, Kubernetes is very powerful and also easy to use (from my perspective). And the book “Kubernetes in action” is a really good book for beginners.

Bought seven books of “A song of ice and fire” by 35 dollars from eBay in last year, I finally start to read this epic novel. The world built by Martin is so cruel, so cold, and so attractive. The “A clash of kings” maybe the thickest single book I have read in my memory.

To learn more hardware knowledge, I have gone through the fifth version of “Computer Architecture“. Passed a lot of paragraphs, just got a preliminary view about modern CPU architecture. Hope to have the opportunity in future to read it more elaborately.

In spite of writing by the same author Wolfgang Faust, the “Tiger Tracks” is not as evocative as “The last panther“, which is the best war literature I read until today.

“Ensemble Methods” is written by a Chinese professor at Nanjing University, Zhi-Hua Zhou. I love his book because they are very easy to understand and also could give me a lot of new concepts and knowledge. Though this book is not very practical in my work, I still recommend it for expanding our view of the horizon.

Last but not least. I’d like to recommend two extra papers, which are very interesting and inspirational this year for both software and hardware areas.

Software: Dota 2 with Large Scale Deep Reinforcement Learning

Hardware: Fast Stencil-Code Computation on a Wafer-Scale Processor

Using keras.layers.Embedding instead of python dictionary

Firstly, I use a function to transform words into word-embedding:

def text_to_array(text, embeddings_index):
    empty_embed = np.zeros(EMBEDDING_LENGTH, dtype = np.float32)
    text = text[:-1].split()[:MAX_TEXT_LENGTH]
    embeds = []
    for x in text:
        em = embeddings_index.get(x)
        if em is not None:
            embeds.append(em)
    embeds += [empty_embed] * (MAX_TEXT_LENGTH - len(embeds))
    return np.array(embeds, dtype = np.float32)

But I noticed that it costs quite a few CPU resource while GPU usage is still low. The reason is simple: using single thread python to do search in dictionary is uneffective. We should use Embedding layer in Keras to put all word-embedding-table into GPU memory.
The code is not difficult to understand:

...
    # word_index = tokenizer.word_index
    nb_words = max(MAX_WORDS, len(word_index))
    embedding_matrix = np.random.normal(emb_mean, emb_std, (nb_words, embed_size))
    for word, i in tqdm(word_index.items()):
        if i >= nb_words: continue
        embedding_vector = embeddings_index.get(word)
        if embedding_vector is not None: embedding_matrix[i] = embedding_vector
...
    inp = Input(shape = (MAX_TEXT_LENGTH,))

    net = Embedding(embedding_matrix.shape[0], EMBEDDING_LENGTH, weights=[embedding_matrix], trainable = False)(inp)
...

This time, the program run two times faster than before. Using GPU memory (GDDR) to find word embedding is the right way.

A few other lessons from Kaggle’s competition ‘Human Protein Atlas Image Classification’

Practice makes progress. Therefore I continued to join Kaggle’s new competition ‘Human Protein Atlas Image Classification’ after the previous one.
I used think I could get a higher rating in image processing competition. But actually, I haven’t even entered the top half of rankings. After almost three month trials and errors, here are my rethinkings:

1. To solve the unbalanced data problem, we need to use ‘focal loss’ instead of normal cross entropy loss. I should be looking at other experts’ kernels earlier, then I could use new techniques as soon as possible.

2. To augment images, ‘lower resolution’ may be a better way than ‘mix up’

3. Try SGD and Cosine Decay, not only RMSProp

4. MobileNet may cause severe overfitting than Resnet

5. If dropout and weight-decay still can’t get better affection for regularization, what should we do? (An open question, feature engineering may be the answer)

6. Use more powerful DNN framework, such as Keras, so I can spend more time on the model itself