task java:26503 blocked for more than 120 seconds

案發原因：
JAVA process一直活著，但莫名其妙無法服務，檢查/var/log/messages，發現有下面訊息：

Jul 31 06:25:19 localhost kernel: INFO: task java:6498 blocked for more than 120 seconds.
Jul 31 06:25:19 localhost kernel:      Not tainted 2.6.32-504.el6.x86_64 #1
Jul 31 06:25:19 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 31 06:25:19 localhost kernel: java          D 0000000000000000     0  6498  26501 0x00000080
Jul 31 06:25:19 localhost kernel: ffff88003cd6be08 0000000000000082 ffff88003cd6bdc8 ffff88003cd6bdc8
Jul 31 06:25:19 localhost kernel: ffff88003cd6bdc8 ffff88000000070d 0000000000000001 00000009ffffffff
Jul 31 06:25:19 localhost kernel: ffff880000000000 ffffffffa0121e08 ffff88007c99d058 ffff88003cd6bfd8
Jul 31 06:25:19 localhost kernel: Call Trace:
Jul 31 06:25:19 localhost kernel: [<ffffffffa0121e08>] ? ext4_file_write+0x58/0x190 [ext4]
Jul 31 06:25:19 localhost kernel: [<ffffffff81236ba1>] ? file_has_perm+0xd1/0xe0
Jul 31 06:25:19 localhost kernel: [<ffffffff8152cae5>] rwsem_down_failed_common+0x95/0x1d0
Jul 31 06:25:19 localhost kernel: [<ffffffff8152cc43>] rwsem_down_write_failed+0x23/0x30
Jul 31 06:25:19 localhost kernel: [<ffffffff812989f3>] call_rwsem_down_write_failed+0x13/0x20
Jul 31 06:25:19 localhost kernel: [<ffffffff8152c142>] ? down_write+0x32/0x40
Jul 31 06:25:19 localhost kernel: [<ffffffff8114545c>] sys_mmap_pgoff+0x5c/0x2d0
Jul 31 06:25:19 localhost kernel: [<ffffffff81012499>] sys_mmap+0x29/0x30
Jul 31 06:25:19 localhost kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

google大神告訴我：
http://www.blackmoreops.com/2014/09/22/linux-kernel-panic-issue-fix-hung_task_timeout_secs-blocked-120-seconds-problem/

Make it permanent

When the server seemed more stable and no Kernel/Swap/Memory Panic for a week, I edited /etc/sysctl.conf file to make these permanent after reboot.

someuser@servercore [/home/someuser]$ sudo vi /etc/sysctl.conf

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

 reboot，繼續觀察是否解決

補充：
另外搜尋到關鍵字：hung_task_timeout_secs
查詢到相關資訊：

http://blog.uouo123.com/post/700.html

解決辦法：
按照告警裡的提示將該提醒disable
echo 0 > /proc/sys/kernel/hung_task_timeout_secs

意思就是：

告知是linux會設置40%的可用內存用來做系統cache，當flush數據時這40%內存中的數據由於和IO同步問題導致超時（120s），所將40%減小到10%，避免超時。