Author Archives: Robin Dong

Wrong ‘struct timeval’ for setsockopt()

What if we deliberately use ‘struct timeval’ like this incorrect way to set timeout of receiving to 3 seconds:

the ‘setsockopt’ will return fail (-1). Let’s look up the linux kernel code for systemcall sys_setsockopt():

sock_setsockopt() will invoke sock_set_timeout() and sock_set_timeout() looks like:

That’s it. If ‘tv.tv_usec’… Read more »

Some tips about Hive

      No Comments on Some tips about Hive

Found some tips about Hive in my learning progress: 1. When I start “bin/hive” at first time, these errors report:

The solution is simple:

Actually, we’d better use mysql instead of derby for multi-users environment. 2. Control the number of mappers for SQL jobs. If a SQL job… Read more »

Use zram to reduce the memory of program

zram is a driver in linux kernel. It compress the content in memory to reduce the pages used by application.

But that’s not the only way we could use zram. Furthermore, we could use zram with tcmalloc to reduce user application’s cost of memory.

Now we make redis-server… Read more »

Some problems and solutions when deploying and running Hadoop-2.7.2

1. If we see this error report:

The solution is here, the heap size of Java should not be bigger than map/reduce memory. The Cloudera recommends the head size prefer to be 0.8 of the map/reduce memory, such as:

2. The directory of “/tmp/” became full. This is… Read more »

Book notes about “Amazon Redshift Database Developer Guide”

Although be already familiar with Cloud Computing for may years, I haven’t look inside many services provided by Amazon Web Service. Because my company (Alibaba) has it’s own cloud platform: Aliyun, so we are only allowed to use home-made cloud products, such as ECS(like EC2 in AWS), RDS(like RDS in… Read more »

Use type of “Any” carefully in Scala

Think about the code below:

I intent to see the compiler error for second ‘increment’ int the first place. But it don’t, the compiler report ok and the output of the program is:

The compiler recognize the three arguments “world”, “6”, “7” as a tuple of (“world”, 6,… Read more »

Usage limitations of HDFS’s C API

I have to change a program which is written by c language from writing local files to writing on HDFS. After learning the example of C API in libhdfs, I complete the modification of open()/write()/read() to hdfsOpenFile()/hdfsWriteFile()/hdfsReadFile() and so on. But when running the new program, many problems occured. The… Read more »