ログを見ると
Feb 10 03:46:03 ssiscirine kernel: Out of memory: Kill process 1983 (clamd) score 35 or sacrifice child
Feb 10 03:47:47 ssiscirine kernel: Out of memory: Kill process 31369 (httpd) score 27 or sacrifice child
Feb 10 03:49:05 ssiscirine kernel: Out of memory: Kill process 21990 (httpd) score 27 or sacrifice child
Feb 10 03:53:27 ssiscirine kernel: Out of memory: Kill process 26774 (httpd) score 23 or sacrifice child
Feb 10 03:56:28 ssiscirine kernel: Out of memory: Kill process 24407 (httpd) score 22 or sacrifice child
Feb 10 04:15:37 ssiscirine kernel: Out of memory: Kill process 26782 (httpd) score 22 or sacrifice child
Feb 10 04:16:35 ssiscirine kernel: Out of memory: Kill process 31367 (httpd) score 22 or sacrifice child
Feb 10 04:17:59 ssiscirine kernel: Out of memory: Kill process 32525 (httpd) score 22 or sacrifice child
Feb 10 04:19:22 ssiscirine kernel: Out of memory: Kill process 20797 (httpd) score 22 or sacrifice child
Feb 10 04:19:46 ssiscirine kernel: Out of memory: Kill process 26759 (httpd) score 21 or sacrifice child
Feb 10 04:21:02 ssiscirine kernel: Out of memory: Kill process 27261 (httpd) score 21 or sacrifice child
Feb 10 04:22:36 ssiscirine kernel: Out of memory: Kill process 30144 (httpd) score 21 or sacrifice child
Feb 10 04:23:11 ssiscirine kernel: Out of memory: Kill process 31370 (httpd) score 21 or sacrifice child
Feb 10 04:24:25 ssiscirine kernel: Out of memory: Kill process 31371 (httpd) score 21 or sacrifice child
Feb 10 04:25:59 ssiscirine kernel: Out of memory: Kill process 31374 (httpd) score 21 or sacrifice child
Feb 10 04:26:00 ssiscirine kernel: Out of memory: Kill process 20798 (httpd) score 21 or sacrifice child
Feb 10 04:30:29 ssiscirine kernel: Out of memory: Kill process 20800 (httpd) score 21 or sacrifice child
Feb 10 04:32:54 ssiscirine kernel: Out of memory: Kill process 21312 (httpd) score 21 or sacrifice child
Feb 10 04:34:05 ssiscirine kernel: Out of memory: Kill process 21994 (httpd) score 21 or sacrifice child
Feb 10 04:42:06 ssiscirine kernel: Out of memory: Kill process 22164 (httpd) score 21 or sacrifice child
夜中に大量のアクセスで1GBのスワップを食いつぶしているらしい。
カーネルがhttpdをkillしてもinitデーモンがすぐhttpdを起動するから特に問題ない(ログは消えてそうだが)。
そのうちの一つをよく見ると、
Feb 10 03:47:47 ssiscirine kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Feb 10 03:47:47 ssiscirine kernel: [ 436] 0 436 665 31 0 -17 ****
Feb 10 03:47:47 ssiscirine kernel: [ 911] 0 911 664 32 0 -17 ****
Feb 10 03:47:47 ssiscirine kernel: [ 912] 0 912 664 32 0 -17 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1736] 0 1736 3487 35 0 -17 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1752] 0 1752 8884 81 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1804] 81 1804 800 34 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1814] 0 1814 2723 24 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1819] 0 1819 1142 24 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1826] 70 1826 773 34 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1827] 70 1827 773 15 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1837] 0 1837 1810 18 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1875] 0 1875 498 17 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1891] 68 1891 1539 116 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1892] 0 1892 939 19 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1920] 0 1920 958 19 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1929] 68 1929 869 22 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1952] 0 1952 4922 130 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1967] 0 1967 2113 23 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 1975] 38 1975 1277 32 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2020] 0 2020 1561 22 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2112] 27 2112 37491 1053 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2135] 0 2135 711 29 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2137] 97 2137 677 32 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2138] 0 2138 675 33 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2147] 0 2147 2162 16 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2148] 0 2148 2162 5 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2149] 0 2149 2162 5 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2150] 0 2150 2162 5 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2151] 0 2151 2162 5 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2227] 0 2227 3105 38 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2238] 89 2238 3164 50 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2251] 0 2251 1681 22 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2259] 0 2259 751 27 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2278] 498 2278 9284 79 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2315] 0 2315 1759 39 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2326] 0 2326 718 14 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2341] 0 2341 495 14 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2343] 0 2343 495 14 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2345] 0 2345 495 14 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2347] 0 2347 495 14 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2349] 0 2349 495 14 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 2449] 0 2449 5481 30 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [ 3750] 0 3750 8485 61 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [ 8549] 0 8549 495 22 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [25462] 48 25462 8485 44 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [24407] 48 24407 17062 1878 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [26759] 48 26759 16550 1493 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [26774] 48 26774 17318 1417 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [26782] 48 26782 16806 1697 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27261] 48 27261 16559 1394 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [30144] 48 30144 16550 1437 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [31367] 48 31367 16806 1313 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [31369] 48 31369 19622 1454 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [31370] 48 31370 16294 1287 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [31371] 48 31371 16563 1535 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [31374] 48 31374 16550 1363 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [32525] 48 32525 16550 1382 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [20797] 48 20797 17062 1310 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [20798] 48 20798 16550 1539 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [20800] 48 20800 16550 448 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [21312] 48 21312 16687 1500 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [21990] 48 21990 19626 1410 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [21993] 48 21993 16294 1386 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [21994] 48 21994 16550 1388 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [22164] 48 22164 16550 1316 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27243] 0 27243 716 32 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [27318] 0 27318 740 35 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [27323] 0 27323 741 36 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [27324] 0 27324 630 22 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [27331] 0 27331 497 15 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [27438] 89 27438 3124 51 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [27510] 48 27510 14384 1727 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27512] 48 27512 14760 1994 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27513] 48 27513 14397 1640 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27521] 48 27521 14760 2179 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27522] 48 27522 14760 1499 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27523] 48 27523 14760 1825 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27526] 48 27526 14760 1881 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27528] 48 27528 14760 1899 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27529] 48 27529 15014 1428 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27530] 48 27530 14760 1979 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27533] 48 27533 14760 1456 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27534] 48 27534 14760 1542 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27535] 48 27535 15014 1497 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27538] 48 27538 14760 1856 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27540] 48 27540 14760 1524 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27543] 48 27543 14760 1497 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27544] 48 27544 14760 1463 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27546] 48 27546 14760 1732 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27547] 48 27547 14760 1636 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27548] 48 27548 14760 2081 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27549] 48 27549 14760 1526 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27559] 48 27559 14760 2064 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27561] 48 27561 14760 2081 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27562] 48 27562 14760 1917 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27563] 48 27563 14760 1932 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27566] 48 27566 14769 2155 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27570] 48 27570 14760 2634 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27571] 48 27571 14760 3115 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27572] 48 27572 14769 2475 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27574] 48 27574 14769 2671 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27575] 48 27575 14760 2747 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27576] 48 27576 14633 3633 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27580] 48 27580 14760 3232 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27583] 48 27583 14633 3248 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27585] 48 27585 14760 3436 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27586] 48 27586 14760 3351 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27590] 48 27590 8741 53 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27591] 0 27591 1858 61 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [27592] 0 27592 1858 47 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [27595] 0 27595 1517 35 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [27596] 0 27596 5664 186 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [27597] 48 27597 8741 227 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27598] 0 27598 2815 642 0 0 0 ****
Feb 10 03:47:47 ssiscirine kernel: [27599] 48 27599 8741 79 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27600] 48 27600 8741 79 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27601] 48 27601 8741 63 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: [27606] 48 27606 8741 91 0 0 0 httpd
Feb 10 03:47:47 ssiscirine kernel: Out of memory: Kill process 31369 (httpd) score 27 or sacrifice child
。。。
同接数64件なんて、AMD E-350のVMwareの中のCentOS(1CPU:メモリ512MB)のヒトには酷な仕打ち。
でも、自分がブラウザで見ててもtopコマンドにhttpdが6くらい出るから、実質同接10でいっぱいなのかもしれない。
apacheの設定は変えてないから
MaxClients 256
のまま。
アクセスが集中するとclamdも耐えられないようなので、
メモリは512MBのままにして、スワップだけを4GBに増やしてみた。
そうなるとSSDの空きがキツイのでHDDへ移動、遅いから事前に4GB割り当てておいた。
JavaでWebサーバーを作ってたらこんな事態は想定外で復旧は大変だろうな。
IISはデフォの10アクセス制限なら死なないだろうけど、商用利用でアクセス枠広げたら、一気に逝くだろうな。