一、起因
某日发现smokeping访问失败，发布的apache 进程不存在；
在/usr/local 下找不到任何apache文件，怀疑apache是rpm安装，使用/etc/init.d/httpd启动后果真listen 8001端口···
二、排查
1、获取error日志的关键内容
启动后尝试刷新8001 下的smokeping页面，发现仍旧错误，哎，rpm装的apache error日志在哪来着？
想不起来了，没办法，迂回一下：
首先找到apache的进程号，需要work进程
[[email protected] smokeping]# ps -ef | grep http
root 4009 1 0 10:43 ? 00:00:00 /usr/sbin/httpd
apache 4012 4009 0 10:43 ? 00:00:00 /usr/sbin/httpd
apache 4013 4009 0 10:43 ? 00:00:00 /usr/sbin/httpd
apache 4014 4009 0 10:43 ? 00:00:00 /usr/sbin/httpd
apache 4015 4009 0 10:43 ? 00:00:00 /usr/sbin/httpd
apache 4016 4009 0 10:43 ? 00:00:00 /usr/sbin/httpd
apache 4017 4009 0 10:43 ? 00:00:00 /usr/sbin/httpd
apache 4018 4009 0 10:43 ? 00:00:00 /usr/sbin/httpd
apache 4019 4009 0 10:43 ? 00:00:00 /usr/sbin/httpd
root 6476 4195 0 11:16 pts/4 00:00:00 grep http
上lsof利器
[[email protected] smokeping]# lsof -n -p 4012
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
httpd 4012 apache cwd DIR 253,0 4096 2 /
httpd 4012 apache rtd DIR 253,0 4096 2 /
.......................
httpd 4012 apache 2w REG 253,0 5782 119768004 /var/log/httpd/error_log
httpd 4012 apache 3u IPv6 7688235 TCP *:vcom-tunnel (LISTEN)
httpd 4012 apache 4u sock 0,5 7688236 can‘t identify protocol
httpd 4012 apache 5r FIFO 0,6 7688250 pipe
httpd 4012 apache 6w FIFO 0,6 7688250 pipe
httpd 4012 apache 7w REG 253,0 5782 119768004 /var/log/httpd/error_log
httpd 4012 apache 8w REG 253,0 2876 119768000 /var/log/httpd/access_log
httpd 4012 apache 9r CHR 1,9 787 /dev/urandom
httpd 4012 apache 10r 0000 0,11 0 7688264 eventpoll
httpd 4012 apache 11w FIFO 0,6 7688261 pipe
ok，得到了，在/var/log/httpd/，查看error_log发现如下：
[Tue May 06 10:53:01 2014] [error] [client 101.71.249.170] (13)Permission denied: exec of ‘/usr/local/smokeping/htdocs/smokeping.fcgi‘ failed, referer: http://opsstat.vvku.com/sa_status/index/ops_monitor.htm
[Tue May 06 10:53:01 2014] [error] [client 101.71.249.170] Premature end of script headers: smokeping.fcgi, referer: http://opsstat.vvku.com/sa_status/index/ops_monitor.htm
[Tue May 06 10:55:00 2014] [error] [client 101.71.249.170] (13)Permission denied: exec of ‘/usr/local/smokeping/htdocs/smokeping.fcgi‘ failed, referer: http://opsstat.vvku.com/sa_status/index/ops_monitor.htm
[Tue May 06 10:55:00 2014] [error] [client 101.71.249.170] Premature end of script headers: smokeping.fcgi, referer: http://opsstat.vvku.com/sa_status/index/ops_monitor.htm
[Tue May 06 10:55:02 2014] [error] [client 101.71.249.170] (13)Permission denied: exec of ‘/usr/local/smokeping/htdocs/smokeping.fcgi‘ failed, referer: http://opsstat.vvku.com/sa_status/index/ops_monitor.htm
[Tue May 06 10:55:02 2014] [error] [client 101.71.249.170] Premature end of script headers: smokeping.fcgi, referer: http://opsstat.vvku.com/sa_status/index/ops_monitor.htm
[Tue May 06 10:57:08 2014] [error] [client 101.71.249.170] (13)Permission denied: exec of ‘/usr/local/smokeping/htdocs/smokeping.fcgi‘ failed, referer: http://opsstat.vvku.com/sa_status/index/ops_monitor.htm
[Tue May 06 10:57:08 2014] [error] [client 101.71.249.170] Premature end of script headers: smokeping.fcgi, referer: http://opsstat.vvku.com/sa_status/index/ops_monitor.htm
[Tue May 06 10:57:10 2014] [error] [client 101.71.249.170] (13)Permission denied: exec of ‘/usr/local/smokeping/htdocs/smokeping.fcgi‘ failed, referer: http://opsstat.vvku.com/sa_status/index/ops_monitor.htm
[Tue May 06 10:57:10 2014] [error] [client 101.71.249.170] Premature end of script headers: smokeping.fcgi, referer: http://opsstat.vvku.com/sa_status/index/ops_monitor.htm
从(13)Permission denied: exec of ‘/usr/local/smokeping/htdocs/smokeping.fcgi‘ failed看出，权限不够
2、依旧Permission denied查找权限问题
此时httpd由apache账户运行，而smokeping的账户是daemon，怀疑是此处，因此把smokeping的cache、data、htdocs、var的账户属主变更为apache；
很自信的一刷，结果错误依旧···
查看/usr/local/smokeping/htdocs/smokeping.fcgi内容：
/usr/local/smokeping/bin/smokeping_cgi /usr/local/smokeping/etc/config
发现依赖于两个文件，把这两个文件的权限属主也赋予apache，再很自信的一刷，结果错误依旧···
额，奇怪了，脾气不好，将smokeping目录进行了备份，将整个目录的属主赋予apache，结果错误依旧···
这这这，怒上777，结果很无奈的，依旧错误···
3、回归到apache的cgi配置
看来已经不是目录和执行权限的问题了，整理思绪，回归apache的配置，这里吐槽下，这种rpm装的apache conf真是难看，1000多行；
从配置中找到关键配置，AllowOverride已给，AddHandler已加
Alias /cache "/usr/local/smokeping/cache/"
Alias /cropper "/usr/local/smokeping/htdocs/cropper/"
Alias /smokeping "/usr/local/smokeping/htdocs/smokeping.fcgi"
<Directory "/usr/local/smokeping">
AllowOverride None
Options All
AddHandler cgi-script .fcgi .cgi
Order allow,deny
Allow from all
DirectoryIndex smokeping.fcgi
</Directory>
这下难办了，不知道了···
4、无奈的google
google (13)Permission denied: exec of，发现如下链接：
http://stackoverflow.com/questions/1599728/13permission-denied-access-to-cgi-bin-test-cgi-denied
老外的排错提醒：
Check your os permissions for test.cgi and be sure the user or group you are using to run your apache it has read access.
EDIT - The problem is with permissions, but not with read permissions, as you are using SELinux, you need to worry about your file context.
Check this thread at fedora forums, it explains quite a few options to solve your problem.
5、思路转向selinux
SELinux，记得早年学lamp的时候，记得要把他关掉，不然php的动态执行会不成功；
看一下配置，额，果真，赤裸裸的没关
[[email protected] smokeping]# cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - SELinux is fully disabled.
SELINUX=enforcing
# SELINUXTYPE= type of policy in use. Possible values are:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.
SELINUXTYPE=targeted
由于服务器在线上跑着其他服务，不能因为一个监控而重启，没办法再次google selinux的及时关闭方法：
查看SELinux状态：
1、/usr/sbin/sestatus -v ##如果SELinux status参数为enabled即为开启状态
SELinux status: enabled
2、getenforce ##也可以用这个命令检查
关闭SELinux：
1、临时关闭（不用重启机器）：
setenforce 0 ##设置SELinux 成为permissive模式
##setenforce 1 设置SELinux 成为enforcing模式
2、修改配置文件需要重启机器：
修改/etc/selinux/config 文件
将SELINUX=enforcing改为SELINUX=disabled
重启机器即可
6、尝试解决
参照如上，尝试如下：
查看selinux状态，是启动的
[[email protected] ~]# getenforce
Enforcing
立即关闭selinux
[[email protected] ~]# setenforce 0
再次查看，已经是宽容模式了
[[email protected] ~]# getenforce
Permissive
再次很自信的一刷，虽然已经错误，但已经没有权限的问题了，是smokeping的配置问题
[Tue May 06 11:12:07 2014] [error] [client 101.71.249.170] [Tue May 6 11:12:07 2014] smokeping_cgi: ERROR: /usr/local/smokeping/etc/config, line 125: File ‘/usr/local/smokeping/etc/smokeping_secrets.dist‘ is world-readable or writable, refusing it, referer: http://opsstat.vvku.com/sa_status/index/ops_monitor.htm
三、总结
坑多、人杂，问题不少，还需继续磨练基础知识，思路真心比敲点命令重要很多···
话说，老外在selinux不关的情况下，是怎么玩的？
selinux 限制 apache exec执行cgi,布布扣,bubuko.com