1. If we see this error report:

Container XXX is running beyond virtual memory limits

The solution is here, the heap size of Java should not be bigger than map/reduce memory. The Cloudera recommends the head size prefer to be 0.8 of the map/reduce memory, such as:

    mapreduce.map.memory.mb
    4096


    mapreduce.reduce.memory.mb
    8192


    mapreduce.map.java.opts
    -Xmx3276m


    mapreduce.map.java.opts
    -Xmx6553m

2. The directory of “/tmp/” became full.
This is usually caused by spilled data from map output. This article introduced the whole overview of Map/Reduce algorithm in Hadoop with a detailed and clear picture.
As a result, my solution is adding this configuration:

    hadoop.tmp.dir
    /data1/tmp/,/data2/tmp/,/data3/tmp/

into core-site.xml, so the inevitable spill data will be write into different disks for load balance.
3. Don’t use more than 0.8 of physical memory as “yarn.nodemanager.resource.memory-mb”, or it will cause unexpected fail for jobs.
4. If we launch too many map jobs or reduce jobs more than physical cores of servers, it may lead to tremendous timeouts for these jobs. Therefore, adjust the “mapreduce.map.memory.mb” and “mapreduce.reduce.memory.mb” carefully to limit the number of map/reduce jobs.
5. If you notice that all the CPU cores are full in Hadoop cluster, that does not mean we can’t do optimizations anymore. By using perf, I find out system waste too many times on launching and stopping java task (or containers):

So I change the value of “mapreduce.input.fileinputformat.split.minsize” to 8GB for reducing the number of mappers. After decrease the number of mappers from thousands to hundreds, the running time of Terasort program drop down more than 50% (Also the Context Switch of system fall from ten thousands per second to thousands). Therefore, adjust the number of java tasks close to the number of physical CPU cores is a better solution.