Our servers recently were crashed due to a large number connections to web (apache) services, we didn’t see anything abnormal on the web. The server will crash when there are over 150 connections, these connections could just come from one IP. After a long investigation, we realize that the attacker attack us by sending a lot of connections, but somehow they put all the TCP status in CLOSE_WAIT status, apache can’t release this.
The solution is , we write this script to kill all httpd process having CLOSE_WAIT status. This seems fixing our problems.
'-')
{
$arr3=explode('/',$arr2[6]);
$process_id=$arr3[0];
if($process_id==0)
continue;
$process_name=$arr3[1];
if(!in_array($process_name,$process_array))
continue;
debug("Found: $line: with process_id: $process_id");
$cmd="kill $process_id";
if(!$test)
{
debug("Kill $cmd");
shell_exec("$cmd");
$log=true;
}else
{
debug("Test - not kill : $cmd");
}
}
}
if($log)
{
$content="--------------------------------------------".date("Y-m-d H:i:s")."-------------------------------------------------------- \r\n $netstat";
write_file($file_log,$content);
}
function debug($s)
{
echo "\r\n $s";
}
function write_file($filename,$content)
{
$fp = fopen($filename, 'a+');
fwrite($fp, $content);
fclose($fp);
}
?>