Category Archives: develope

Wrong ‘struct timeval’ for setsockopt()

What if we deliberately use ‘struct timeval’ like this incorrect way to set timeout of receiving to 3 seconds:

the ‘setsockopt’ will return fail (-1). Let’s look up the linux kernel code for systemcall sys_setsockopt():

sock_setsockopt() will invoke sock_set_timeout() and sock_set_timeout() looks like:

That’s it. If ‘tv.tv_usec’… Read more »

Use zram to reduce the memory of program

zram is a driver in linux kernel. It compress the content in memory to reduce the pages used by application.

But that’s not the only way we could use zram. Furthermore, we could use zram with tcmalloc to reduce user application’s cost of memory.

Now we make redis-server… Read more »

Some problems and solutions when deploying and running Hadoop-2.7.2

1. If we see this error report:

The solution is here, the heap size of Java should not be bigger than map/reduce memory. The Cloudera recommends the head size prefer to be 0.8 of the map/reduce memory, such as:

2. The directory of “/tmp/” became full. This is… Read more »

Use type of “Any” carefully in Scala

Think about the code below:

I intent to see the compiler error for second ‘increment’ int the first place. But it don’t, the compiler report ok and the output of the program is:

The compiler recognize the three arguments “world”, “6”, “7” as a tuple of (“world”, 6,… Read more »

Usage limitations of HDFS’s C API

I have to change a program which is written by c language from writing local files to writing on HDFS. After learning the example of C API in libhdfs, I complete the modification of open()/write()/read() to hdfsOpenFile()/hdfsWriteFile()/hdfsReadFile() and so on. But when running the new program, many problems occured. The… Read more »

Network problem of installing docker-engine on Centos-7

After installing docker-engine on Centos-7, it failed to start by

After I use

it shows:

To see more details, I use

There were much more informations now:

The ethernet interface of docker0 can not be assigned IPv4 addresse. It seems many people meet this problem… Read more »

The CPU usage of soft-irq will be counted into a process.

Redis usually runs as a single process daemon that is the perfect sample of UNIX philosophy — one process just do one thing. But current servers have many CPUs(Cores) so we need to launch many Redis processes to provide service. After running multi-process redis-server in server, I found out there… Read more »

Avoid “page allocation failure” for linux kernel in big memory server

After adding pressure to a Key-Value cluster, I found many error in dmesg:

It’s hard to understand the “page allocation failure” error because the memory capacity is very big in our servers. By looking at the result “free” command, I noticed that a large mount of memory was used… Read more »

Too many “ext4-dio-unwrit” processes in system

After adding pressure to application which will write tremendous data into ext4 file system, we see many “ext4-dio-unwrit” kernel threads in “top” screen. Many guys say this is a normal phenomenon, so I check the source code of ext4 in 2.6.32 linux kernel. The beginning of writing a file in… Read more »

Why you should update your gcc (and c++ library)

Consider the code below:

It could be compiled and run on CentOS 5 (gcc-4.1.2), but will core dump at runtime.

The gdb stack shows the breakpoint is in string_hashfunc::operator():

Let’s see the source code of “ext/hash_map” in /usr/include/c++/4.1.2/ext/hashtable.h:

And in the implementation of _M_bkt_num():

It… Read more »