0
我在Gentoo上有一個生產LAMP服務器，它運行着一個個人內部網站。自從昨天下午以來，它在任何連接後不久就掛起;相對較小的登錄頁面加載正常，但在登錄時涉及到大量MySQL查詢和數據，它會無限期地掛起。Apache和其他服務Hung
SSH仍然能夠連接，但奇怪的是，傳輸幾頁的字符後突然掛起。因此，爲了獲得下面的信息，我必須保持重新登錄。我嘗試了/etc/init.d/apache2 restart和/etc/init.d/mysql restart，然後重新啓動了系統;但唉，這個問題依然存在。詳情如下。
上：
top - 12:23:52 up 1:34, 2 users, load average: 0.16, 0.09, 0.06 Tasks: 81 total, 1 running, 80 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si,
0.0%st Mem: 3920788k total, 123476k used, 3797312k free, 4676k buffers Swap: 1227772k total, 0k used, 1227772k free, 48524k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 447 root 20 0 0 0 0 S 0 0.0 0:00.16 khubd
1 root 20 0 2020 640 568 S 0 0.0 0:00.51 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0 0.0 0:00.00 ksoftirqd/0
5 root 20 0 0 0 0 S 0 0.0 0:00.00 kworker/u:0
6 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/0
7 root RT 0 0 0 0 S 0 0.0 0:00.00 migration/1
9 root 20 0 0 0 0 S 0 0.0 0:00.00 ksoftirqd/1
10 root 20 0 0 0 0 S 0 0.0 0:00.69 kworker/0:1
Apache日誌顯示一般的黑客攻擊：
# tail -50 /var/log/apache2/error_log
[Mon Mar 17 19:03:48 2014] [error] [client 116.58.240.169] File does not exist: /var/www/mysite/pma
[Mon Mar 17 19:03:48 2014] [error] [client 116.58.240.169] File does not exist: /var/www/mysite/myadmin
[Tue Mar 18 05:58:42 2014] [error] [client 202.53.8.82] File does not exist: /var/www/mysite/admin.cgi
[Tue Mar 18 07:19:42 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/phpTest
[Tue Mar 18 07:19:43 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/phpMyAdmin
[Tue Mar 18 07:19:43 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/pma
[Tue Mar 18 07:19:44 2014] [error] [client 74.63.220.132] File does not exist: /var/www/mysite/myadmin
[Tue Mar 18 08:24:16 2014] [error] [client 222.5.204.73] invalid request-URI \xcc\\\xa4/\x83\x8f\x90:\x84\x90\x0f\xc4\x8dfe\xecb\x94v\x1f[\xd7Z\x95$X\xaby\x13k\x88\xf2\xeb\xf7\x1b\xfc\xe8a\xff
[Tue Mar 18 08:29:49 2014] [error] [client 76.3.191.245] invalid request-URI
[Tue Mar 18 08:38:00 2014] [error] [client 35.2.240.149] invalid request-URI
[Tue Mar 18 08:50:52 2014] [error] [client 173.26.148.34] invalid request-URI
[Tue Mar 18 10:57:48 2014] [error] [client 110.175.79.216] invalid request-URI
[Tue Mar 18 10:57:53 2014] [error] [client 110.248.140.59] invalid request-URI D\xe8\x91a\xbc\xe5WZ\xd0C]\x9f~\xb5\x89\bd\x9e"[w,\xc6\xd9\xde\x8b]#JJ\xbf\x12
[Tue Mar 18 14:24:54 2014] [error] [client 108.14.2.113] invalid request-URI
[Tue Mar 18 14:40:08 2014] [error] [client 86.217.136.41] invalid request-URI \x94FI-\x02;4JVOV\x0f\xba\b
[Tue Mar 18 14:45:42 2014] [error] [client 98.119.127.76] invalid request-URI
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-precomposed.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-120x120-precomposed.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-120x120.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon-precomposed.png
[Tue Mar 18 15:25:11 2014] [error] [client 192.168.0.3] File does not exist: /var/www/mysite/apple-touch-icon.png
[Tue Mar 18 16:20:45 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/phpTest
[Tue Mar 18 16:20:46 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/phpMyAdmin
[Tue Mar 18 16:20:46 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/pma
[Tue Mar 18 16:20:46 2014] [error] [client 103.24.32.14] File does not exist: /var/www/mysite/myadmin
[Tue Mar 18 16:40:58 2014] [error] [client 122.170.93.35] invalid request-URI
[Tue Mar 18 16:57:54 2014] [error] [client 124.107.151.190] invalid request-URI
[Tue Mar 18 17:36:17 2014] [error] [client 68.147.250.90] invalid request-URI \x1d\x1e;&\x9e\xd2\xa8\xc2GNQ\\
[Tue Mar 18 23:38:20 2014] [error] [client 92.240.68.153] request failed: error reading the headers
[Wed Mar 19 02:52:43 2014] [error] [client 162.213.24.36] File does not exist: /var/www/mysite/CFIDE
[Wed Mar 19 06:26:06 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:07 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:07 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:09 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 06:26:15 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 07:48:28 2014] [error] [client 201.161.37.93] File does not exist: /var/www/crownware/manager
[Wed Mar 19 09:27:08 2014] [error] [client 113.184.228.73] invalid request-URI \xad_X\xdf\x9aIM6x\x01ti\xf6Ko\xebi
[Wed Mar 19 09:36:06 2014] [error] [client 162.213.24.36] File does not exist: /var/www/crownware/CFIDE
[Wed Mar 19 10:28:15 2014] [notice] caught SIGTERM, shutting down
[Wed Mar 19 10:28:17 2014] [notice] Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0j PHP/5.4.6--pl0-gentoo configured -- resuming normal operations
[Wed Mar 19 10:43:31 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:31 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:35 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:35 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:43:36 2014] [error] [client 5.249.137.202] script not found or unable to stat: /var/www/mysite/cgi-bin
[Wed Mar 19 10:47:16 2014] [notice] caught SIGTERM, shutting down
[Wed Mar 19 10:49:32 2014] [notice] Apache/2.2.23 (Unix) mod_ssl/2.2.23 OpenSSL/1.0.0j PHP/5.4.6--pl0-gentoo configured -- resuming normal operations
[Wed Mar 19 10:53:45 2014] [error] [client 65.60.209.141] Invalid URI in request \x13\xe0\x94\xc4\xa4o\xd1\xd3*\xe0\xe7\x1a\xce\xd9\xe8\t\xca\xc3k\x9f\xb0\x06\x13\xbcE\x17\xbb\x02\x9c:\xffD\x8d\x1f\x85Wv\x14\xfd\x8f\xe3k\xc6\xfe\xf7\x1bu
[Wed Mar 19 12:20:07 2014] [error] [client 173.24.52.209] invalid request-URI
從/var/log/mysql/mysqld.err感興趣最後消息（5天前）：
140314 9:56:02 InnoDB: ERROR: the age of the last checkpoint is 9448765,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.
版本：
# uname -a
Linux myhost 3.3.8-gentoo #1 SMP Fri Sep 28 09:34:42 MYT 2012 i686 Intel(R) Xeon(R) CPU E31220 @ 3.10GHz GenuineIntel GNU/Linux
# mysqld -V
140319 12:37:13 [Warning] '--default-character-set' is deprecated and will be removed in a future release. Please use '--character-set-server' instead.
140319 12:37:13 [Warning] '--default-collation' is deprecated and will be removed in a future release. Please use '--collation-server' instead.
mysqld Ver 5.1.62-log for pc-linux-gnu on i686 (Gentoo Linux mysql-5.1.62-r1)
# apache2 -V
Server version: Apache/2.2.23 (Unix)
Server built: Oct 27 2012 19:17:52
Server's Module Magic Number: 20051115:31
Server loaded: APR 1.4.5, APR-Util 1.3.12
Compiled using: APR 1.4.5, APR-Util 1.3.12
Architecture: 32-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APACHE_MPM_DIR="server/mpm/prefork"
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT="/usr"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="/var/run/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_LOCKFILE="/var/run/accept.lock"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
-D SERVER_CONFIG_FILE="/etc/apache2/httpd.conf"
# php -v
PHP 5.4.6--pl0-gentoo (cli) (built: Oct 27 2012 18:42:24)
Copyright (c) 1997-2012 The PHP Group
Zend Engine v2.4.0, Copyright (c) 1998-2012 Zend Technologies
磁盤似乎有足夠的空間依然：
# df
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 960125048 84604800 826748732 10%/
udev 10240 0 10240 0% /dev
/dev/sda3 960125048 84604800 826748732 10%/
tmpfs 1960392 220 1960172 1% /run
rc-svcdir 1024 64 960 7% /lib/rc/init.d
cgroup_root 10240 0 10240 0% /sys/fs/cgroup
shm 1960392 0 1960392 0% /dev/shm
Apache進程：
# ps -ef|grep -i apache
root 2060 1 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2062 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2066 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2067 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2068 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2069 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2070 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2123 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2124 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2125 2060 0 10:49 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2148 2060 0 10:50 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
apache 2149 2060 0 10:50 ? 00:00:00 /usr/sbin/apache2 -D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP5 -d /usr/lib/apache2 -f /etc/apache2/httpd.conf -k start
Stracing父（根）進程顯示反覆這一點，不知道這是否是正常的：
# strace -p 2060
Process 2060 attached
select(0, NULL, NULL, NULL, {0, 669445}) = 0 (Timeout)
waitpid(-1, 0xbffb4b6c, WNOHANG|WSTOPPED) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
waitpid(-1, 0xbffb4b6c, WNOHANG|WSTOPPED) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
waitpid(-1, 0xbffb4b6c, WNOHANG|WSTOPPED) = 0
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
SSH在幾千字節後也掛起的事實表明我應該看起來更寬韓Apache。下一步要進行診斷？