Category Archives: develope

The type of variables in Python

Haven’t written python code for more than one year, I met this simple problem:

Even the code have print out the value of “a” and “b” as 2 and 1, the condition check “if a >= b:” is false! Spending more than 10 minutes, I eventually get the reason:… Read more »

Using “sysbench” to test memory performance

Sysbench is a powerful testing tool for CPU / Memory / Mysql etc. Three years ago, I used to test performance of MYSQL by using it. Yesterday, I used Sysbench to test memory bandwidth of my server. By using command:

It reported the memory bandwidth could reach 8.4GB/s, which… Read more »

How to show the memory details of a process in Linux

Recently we are evaluating the memory consumption of redis process when it is rewriting of append-only-file. In this situation, redis will firstly fork(), and then the child process will write all the data from memory to new AOF file and the parent process will still provide service to custom. The… Read more »

Some problems about programming Mapreduce

1. After submitting job, the console report:

The reason is I forgot to setJarByClass():

2. When the job finished, I found the reducer haven’t run at all. The reason is I haven’t override the correct reduce() member function of Reducer so MapReduce Framework ignore it and didn’t report… Read more »

Use MapReduce to join two datasets

The two datasets are:

To join the two tables above by “student id”, we need to use MultipleInputs. The code is:

Compile and run it:

And the result in /my is:

Wrong ‘struct timeval’ for setsockopt()

What if we deliberately use ‘struct timeval’ like this incorrect way to set timeout of receiving to 3 seconds:

the ‘setsockopt’ will return fail (-1). Let’s look up the linux kernel code for systemcall sys_setsockopt():

sock_setsockopt() will invoke sock_set_timeout() and sock_set_timeout() looks like:

That’s it. If ‘tv.tv_usec’… Read more »

Use zram to reduce the memory of program

zram is a driver in linux kernel. It compress the content in memory to reduce the pages used by application.

But that’s not the only way we could use zram. Furthermore, we could use zram with tcmalloc to reduce user application’s cost of memory.

Now we make redis-server… Read more »

Some problems and solutions when deploying and running Hadoop-2.7.2

1. If we see this error report:

The solution is here, the heap size of Java should not be bigger than map/reduce memory. The Cloudera recommends the head size prefer to be 0.8 of the map/reduce memory, such as:

2. The directory of “/tmp/” became full. This is… Read more »