life

We are supposed to learn like the Large Language Model

When joined the Junior Middle school in a small town in west-south China in 1993, I met my first English book. Yes, it looks exactly like this:

Then, the terrible 6 years of Chinese-style-English-learning started. For normal kids in poor regions of China, the only way to learn a new language is to REMEMBER IT. For the vocabulary, I remembered all of them by writing them in the draft paper again and again. For the phonogram, I remembered all of them by writing them in the draft paper again and again. For the grammar, I remembered all of them by —- Wait a minute. Why did I need to remember the grammar? Because the English examination will test them. That’s the only purpose of learning English, not to read foreign stories or know the world, but to get a higher examination score.

For the six years of middle school, I spent more than 60 per cent of my hard work time on English (by hardly remembering phonograms and grammar) and still got inferior results: I couldn’t read a long English story, couldn’t recognize a lot of common English words, and couldn’t even write a decent article. All I learned was just some basic English words and some useless grammar.

Only when I went to the University and started to read a 300-page English-Reading-and-Understanding book. Yes, still for the examination, but at least no need to recite those stupid grammar or phonograms. I finally noticed that I could improve my English by just reading books.

The time zips by. I got my Kindle (yes, that electric paper device) in August 2011 (when I was 31 years old) and finished reading my long-desired but first long English story “Jurassic Park”. Since then, I started to read a lot of English books: “The Lost World”, “The Wild Wheel”, “The Swiss Family Robinson”, and all of the best “A Song of Ice and Fire”. I feel happy when reading English books, and my English skills improve as well. Happy learning, that’s the result of reading books.

So my conclusion is: if you want to learn English well, don’t try to remember those boring grammar, just read books, a lot of books.

Doesn’t this sound familiar? Yes, it sounds just like “The Bitter Lesson“, or the Scaling Law for machine learning. A Large Language Model doesn’t need to learn the grammar or go to school. It only needs to read a lot of books and articles (training on a large amount of corpus).

The LLM learns like a human, and I think I can also learn from it: reading a lot is already enough for learning.

Books I read in the year 2022

It does not mean I just read three books in the whole year of 2022 when I just showed three books above. Although reading some short history stories and books about financial knowledge, I still think those are not vital to my life experience.

As a software developer, why did I read a book about the semiconductor material: Silicon Carbide? Frankly speaking, just because of curiosity. In recent years, a lot of news and articles said SiC (Silicon Carbide) will be the future material of EVs (Electric Vehicles), and all the companies that produced SiC will become extremely popular and rich. But after skimming this book (or maybe a long paper), I realized that SiC has been found and produced for many years and at an early age, scientists only thought of it as a material for lighting and sensing. If even the most intelligent people consider SiC as “not very popular” about 10 years ago, why should I believe it will become “the future of EV” in the next 10 years only because some financial guys said this? Inflating a financial bubble is easy, but science is all about hard work and frequent failures.

“A Song of Ice and Fire” is a great novel. The only drawback after reading it (I mean, the first 5 volumes) is I couldn’t be interested in other novels for a long time. Even the “A Knight of the Seven Kingdoms”, written by George RR Martin himself, can’t match it. The first story for Dunk the Hedge Knight is wonderful, but the left two are normal.

The reason I put the algorithm book here (also in 2021) is that I am still trying to revisit and learn algorithms in 2022. After graduating from school, I spend quite a lot of time learning technology about machine learning, compilers, operating systems, and even semiconductors. But not the algorithms. Since learning new algorithms are terribly tedious, I avoid touching them for such a long time. How could a software engineer try to learn everything but algorithms? I felt a little regret. So, I will do it now.

Books I read in the year 2021

I know it looks too late that I wrote this article in the middle of the year. But, late is much better than none. Right?

I bought the seven books series of “A song of ice and fire” on 23rd November 2019 and finish reading them on 3rd August 2021. It is definitely the best fantasy novel I have read so far (sorry Salvatore, I just told you my true thought). The whole world and the whole story in it are cold and cruel, but unimaginably fascinating and attractive.

The first book of the series is published early in 1996 when I was just a stupid middle school student. I feel really regret I hadn’t started to learn English more intensive and started to read “A song of ice and fire” since then.

The currently last book of the series is published in 2011, a total of eleven years ago. Oh, George, please write the last two volumes as quickly as possible. Your fans had been waiting for 11 years. I don’t want to wait another 11 years more…

Fortunately, I also got some time to look into some area that I am really interesting in but couldn’t benifit my work, such as Semiconductor. The book “Introduction to Semiconductor Technology” really helps me a lot. It starts the introduction from basic phisicals and chemistry knowledge to how to make silicon wafer, to how to paste photoresist, to how to etch, to how lithography, to how to etch, to how to make backend wires. Although forgot almost all my chemistry knowledge before, I still could understand all the process in this industry. And it looks really cool! I can’t hold to show a image for a beautiful DRAM:

https://www.eetimes.com/dram-the-field-for-material-and-process-innovation/

In computer science, the algorithm is one of the most important, and also the most difficult part. As a career routine, I found this “The Algorithm Design Manual” and study the algorithms again, mainly focus on the dynamic-programming and just skim through the graph algorithm (they are far away from my daily work, and also far difficult…). I couldn’t say I understood this book very well but at least I take my share of it. And, hope I can revisit it again in the near future.

Books I read in the year 2020

Since beginning my new job on 6th January, I start to learn Kubernetes for the first time. Frankly speaking, Kubernetes is very powerful and also easy to use (from my perspective). And the book “Kubernetes in action” is a really good book for beginners.

Bought seven books of “A song of ice and fire” by 35 dollars from eBay in last year, I finally start to read this epic novel. The world built by Martin is so cruel, so cold, and so attractive. The “A clash of kings” maybe the thickest single book I have read in my memory.

To learn more hardware knowledge, I have gone through the fifth version of “Computer Architecture“. Passed a lot of paragraphs, just got a preliminary view about modern CPU architecture. Hope to have the opportunity in future to read it more elaborately.

In spite of writing by the same author Wolfgang Faust, the “Tiger Tracks” is not as evocative as “The last panther“, which is the best war literature I read until today.

“Ensemble Methods” is written by a Chinese professor at Nanjing University, Zhi-Hua Zhou. I love his book because they are very easy to understand and also could give me a lot of new concepts and knowledge. Though this book is not very practical in my work, I still recommend it for expanding our view of the horizon.

Last but not least. I’d like to recommend two extra papers, which are very interesting and inspirational this year for both software and hardware areas.

Software: Dota 2 with Large Scale Deep Reinforcement Learning

Hardware: Fast Stencil-Code Computation on a Wafer-Scale Processor

Books I read in year 2019

At the beginning of 2019, I finished the book “The Great Siege: Malta 1565”. The story about a few loyal knights protecting Europe from the Ottoman Empire is so extraordinary that it encouraged me to go on my learning and working about information technology.
To find a new job about Data Engineer or Data Scientist, I almost remembered the whole book of “Hundreds of interviews about machine learning” (Title translated from Chinese). Although I haven’t found a job about machine learning (actually, it’s a job about just damned PHP and Javascript), this book gave me confidence and direction before looking for a new job.
I bought the book “Rats of NIMH” at the end of 2016, and finished reading it after more than two years. In the period, life changed tremendously for me, though I hope the end of it would be as good as the Frisby family.
The most exciting new thing I learned is about NLP in deep learning. After reading the papers about Word2Vec, Transformer, Elmo, BERT, etc. I became very familiar and interesting about NLP.
After started my new job in June 2019, I read the book “Statistical Machine Learning” (Title translated from Chinese) on the commute bus. The bus was very vibrant so I have to read the book for a while and take some rest for my eyes and repeat them. Life is not easy, so I should insist further.

Books I read in year 2018

In the 2018 year, I continued to learn more knowledge about machine learning and deep Learning. “Deep Learning” is pretty suitable for me and “Hands-On Machine Learning with Scikit-Learn and TensorFlow” is also a wonderful supplement for programming practice. I also learned some basic knowledge about Reinforcement learning.
To teach my daughters programming, I read some books about Arduino. In the process of learning Arduino, I became more and more interested in electronics on myself! After reading more technical documents about electronics (diode, transistor, capacitor, relay, thyristor etc.), Microcontrollers (Atmega from Atmel, MSP430 from Texas Instruments, STM8 from ST and so on), I had opened my view to a new area.
History books are always my favorite type. The most astonishing history book I have read in 2018 is “The Last Panther”. This book tells us an extremely cruel but real story in WWII.
Kazuo Inamori is a famous entrepreneur in Japan. I read some books written by him at the end of this year. Surprisingly, his books definitely inspired me and even changed some parts of my mind. I really want to thank him for his teaching.

Books I read in year 2017

2017 is not an easy year for me, therefore I read so many books to comfort myself. The books show above are just the top-10 books I rated.
I can’t remember how many times I have read “The old man and the sea”. This time, I read it for my daughter because she wants to hear some “fantastic and powerful” stories. After I told her the whole story (of course, I changed some parts of the story for a child), she is a little puzzle. Hmm, it’s hard to understand for a girl, even the story is truly “powerful” 🙂
At the beginning of 2017 (about February), my old colleague Jian Mei asked my help for building a Bird-Classification-Application. I accept his requirement and start to learn knowledge of Deep Learning from the beginning. Actually, this task makes me “alive” again: I begin to learn new frameworks (MXNET and Tensorflow), read papers (I haven’t read papers for many years) and thick books (such as “Deep Learning“). Finally, we have completed the application for classifying Chinese birds, and I begin my new career in Deep Learning area. Thanks to my old colleague again.
When I was a child, I read a small comic book about a family living on an isolated island. In January, I found an English book in a bookshop, which named “The Swiss Family Robinson”. Suddenly I realized this must be the original version of that old comic book. So I bought it. It only cost me 20 RMB (about 3 dollars, books are desperately cheap in China). In the following weeks, I read this book chapter by chapter and tell them to my daughter. This time, she can understand the book and become interesting about this old story.
In 2015, I traveled to Boston to give my presentation on Linux Vault Conference with my colleague Coly Li. In MIT bookshop, I bought many books, including a history book named “Ancient Rome” (I have read a lot of books about ancient Rome, but none is the English version). But year 2015 and 2016 are very busy, and I only have enough time to read the book over in 2017. This book contains many pictures, which is good for children too. Maybe in the future, I could read it for my children.

Books I read in year 2016

Here comes the last day of 2016 year. And it is also the time for me to review my harvest about knowledge, or books.
Frankly speaking, the book “All hard thing about hard things” literally frighten me, and cause me to give up any idea about joining a startup company in China. Maybe this is the best consequence, for many startup companies failed in this end of year and I fortunately avoid this tempest.
Diving more deeper into the ocean of “Hadoop Ecosystem”, or “Big Data”, I find out Spark is really a convenient and powerful framework (compare to MapReduce) which could implement complicated algorithm or data-flow with a few lines of code. Surely, Scala is also a key element for Spark’s efficiency and concision.
Today, even normal person could imagine a sci-fi story about how modern people will fight with Alien invaders. But, what will happen if Aliens attacked the earth in the ancient time? What about Medieval age? Then comes the funny and bold sci-fi novel “The High Crusade”. A group of Medieval army defeat the invader of Alien， and did even more: occupied a frontline planet of a gigantic Alien Empire. It is really out of my imagination 🙂

Books I read in year 2015

The first book is about network hardware, like router, switcher. As a coder, I usually use servers on cloud, therefore haven’t see the real high performance routers (I have sought bare server, 1Gb switcher). This book open my eyes.
The second book is about how to build Datacenter. It’s really a work for architecture, not IT guys.
About two years ago, I worked with Mysql team in my company as a kernel developer. We have used PCIE-card of NAND and flashcache as our solution for Mysql to process hight throughput pressure. But util this year, I have read over the architecture of InnoDB Engine which is the most powerful and effective engine in Mysql. Actually, it’s not so difficult to have a overview of the InnoDB Engine in a book. But, it is still very hard to understand the code of it 🙂
I haven’t go to cinema to watch “The Martian” because I have read it in my Kindle on my commute everyday. It is really a sci-fi story for Geeks who like do research on Computer,Chemistry,Physics,etc. The only question I want to ask the author is:” How could you invent so much troubles on Mars to torture Mark Watney?”