Friday, September 26, 2014

transition from sysV to systemd, from chkconfig to systemctl

If you have just been installed CentOS 7.0 and as usual, command chkconfig is executed
to list what processes will be start on boot. As seen below:
[root@localhost ~]# chkconfig

Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
systemd configuration.

If you want to list systemd services use 'systemctl list-unit-files'.
To see services enabled on particular target use
'systemctl list-dependencies [target]'.

iprdump 0:off 1:off 2:on 3:on 4:on 5:on 6:off
iprinit 0:off 1:off 2:on 3:on 4:on 5:on 6:off
iprupdate 0:off 1:off 2:on 3:on 4:on 5:on 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
tomcat 0:off 1:off 2:off 3:off 4:off 5:off 6:off

That's odd, something has changed. For your information, sysV has been replaced in favor of systemd and today we are going to learn what is systemd is. So what is systemd ?

systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux cgroups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic. It can work as a drop-in replacement for sysvinit. 

That is a very lengthy definition. If you are still not so sure, perhaps take a moment to watch a video here.



Because there are a lot of documentations in the google to explain what is systemd in details, but this article will target busy people who need the solution right now. As such, if you want more details solutions, you should google or read a few helpful links below.

So why replace sysV with systemd? What have been improved?

Lennart Poettering and Kay Sievers, the software engineers who initially developed systemd,[1] sought to surpass the efficiency of the init daemon in several ways. They wanted to improve the software framework for expressing dependencies; to allow more processing to be done concurrently or in parallel during system booting; and to reduce the computational overhead of the shell.

Systemd's initialization instructions for each daemon are recorded in a declarative configuration file rather than a shell script. For inter-process communication, systemd makes Unix domain sockets and D-Bus available to the running daemons. Systemd is also capable of aggressive parallelization.

There are several tools to manage systemd.

  • systemctl:
    used to introspect and control the state of the systemd system and service manager

  • systemd-cgls:
    recursively shows the contents of the selected Linux control group hierarchy in a tree

  • systemadm:
    a graphical frontend for the systemd system and service manager that allows introspection and control of systemd. Part of the systemd-gtk package. This is an early version and needs more work. Do not use it for now unless you are a developer.


Below are a table to summarize what you usually done in chkconfig and in systemd, what command you can use as a replacement.











































































Sysvinit CommandSystemd CommandNotes
service frobozz startsystemctl start frobozz.serviceUsed to start a service (not reboot persistent)
service frobozz stopsystemctl stop frobozz.serviceUsed to stop a service (not reboot persistent)
service frobozz restartsystemctl restart frobozz.serviceUsed to stop and then start a service
service frobozz reloadsystemctl reload frobozz.serviceWhen supported, reloads the config file without interrupting pending operations.
service frobozz condrestartsystemctl condrestart frobozz.serviceRestarts if the service is already running.
service frobozz statussystemctl status frobozz.serviceTells whether a service is currently running.
ls /etc/rc.d/init.d/systemctl list-unit-files --type=service (preferred)
ls /lib/systemd/system/*.service /etc/systemd/system/*.service
Used to list the services that can be started or stopped Used to list all the services and other units
chkconfig frobozz onsystemctl enable frobozz.serviceTurn the service on, for start at next boot, or other trigger.
chkconfig frobozz offsystemctl disable frobozz.serviceTurn the service off for the next reboot, or any other trigger.
chkconfig frobozzsystemctl is-enabled frobozz.serviceUsed to check whether a service is configured to start or not in the current environment.
chkconfig --listsystemctl list-unit-files --type=service(preferred)
ls /etc/systemd/system/*.wants/
Print a table of services that lists which runlevels each is configured on or off
chkconfig frobozz --listls /etc/systemd/system/*.wants/frobozz.serviceUsed to list what levels this service is configured on or off
chkconfig frobozz --addsystemctl daemon-reloadUsed when you create a new service file or modify any configuration

Runlevels/targets

Systemd has a concept of targets which serve a similar purpose as runlevels but act a little different. Each target is named instead of numbered and is intended to serve a specific purpose.













































Sysvinit RunlevelSystemd TargetNotes
0runlevel0.target, poweroff.targetHalt the system.
1, s, singlerunlevel1.target, rescue.targetSingle user mode.
2, 4runlevel2.target, runlevel4.target, multi-user.targetUser-defined/Site-specific runlevels. By default, identical to 3.
3runlevel3.target, multi-user.targetMulti-user, non-graphical. Users can usually login via multiple consoles or via the network.
5runlevel5.target, graphical.targetMulti-user, graphical. Usually has all the services of runlevel 3 plus a graphical login.
6runlevel6.target, reboot.targetReboot
emergencyemergency.targetEmergency shell

Below are a summarize the command you will (hopefully) use.

  • systemctl isolate multi-user.target
    To change the target/runlevel, to switch to runlevel 3

  • systemctl set-default <name of target>.target
    graphical.target is the default. You might want multi-user.target for the equivalent of non graphical (runlevel 3) from sysv init.

  • systemctl get-default
    to show the currentl target/runlevel


Note, there are several changes you should keep in mind.
* systemd does not use /etc/inittab file.
* change number of gettys in /etc/systemd/logind.conf
* unit files are now store in /usr/lib/systemd/system/

That's it, I hope you get a basic understanding and will be able to start using systemd.

Sunday, September 14, 2014

How to convert java keystore to format apache httpd understand

If you received a java keystore file from a Certificate Authority and want to use this cert to setup in apache httpd ssl, you will meet failure, at least I did. So today, I will share my finding on how to convert java keystore file into PEM format which is understand by apache httpd.

So how do you know if a certificate signed by CA is of type java keystore? Simple, just check the content using keytool. Keytool is an app come together when you install java environment.
$ keytool -list -keystore abc.jks
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

ABC_Certificate, Aug 19, 2013, PrivateKeyEntry,
Certificate fingerprint (MD5): 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00

As you can read above, this is a valid java keystore file and we will now convert to a intermittent format, pkcs12 first. We will use keytool again to do the conversion.
$ keytool -importkeystore -srckeystore abc.jks -destkeystore abc.p12 -srcalias ABC_Certificate -srcstoretype jks -deststoretype pkcs12
Enter destination keystore password:
Re-enter new password:
Enter source keystore password:
$

the output abc.p12 is the certificate in pkcs12 and now we are ready to convert to pem format. We will use openssl to do this conversion.
$ openssl pkcs12 -in myapp.p12 -out myapp.pem
Enter Import Password:
MAC verified OK
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
$

You can basically use myapp.pem for the field in SSLCertificateFile and SSLCertificateKeyFile but unfortunately when apache httpd is restarted, it will ask for the private key passphrase. With the following steps, we will remove the passphrase from the private key.

Removed passphrase so when apache httpd instance is restarted, it will not ask for password.
$ openssl rsa -in abc.pem -out abc_private_key.pem
Enter pass phrase for abc.pem:
writing RSA key
$ openssl x509 -in abc.pem >>abc_cert.pem

As you noticed, right now you end up with the certificate private key and the certificate. Now move these two files, abc_private_key.pem and abc_cert.pem to a directory in the apache httpd server and change the ssl configuration in apache httpd.
SSLCertificateFile    /path/to/the/directory/contain/abc_cert.pem
SSLCertificateKeyFile /path/to/the/directory/contain/abc_private_key.pem

That's it, I hope it works for you too.

Saturday, September 13, 2014

TMnet UNIFI response on IPv6 subnet request

Recently, with the support of dual stack IPv4 and IPv6, it is pretty exciting for a small country like Malaysia be in the fore runner for new technology like supporting IPv6. As myself been in I.T. technology for years, enabling this and learn more about IPv6 is definitely a must. With the abundance of IPv6 address, perhaps the ISP will be generous and knowledgable on topic of IPv6 since they announced supported on year 2013, and statically allocate a static IPv6 subnet for its user. Here goes the inquiry.

Hi, with the abundance of ipv6 addresses, can you assign a static ipv6 subnet for my router? thank you.

and the response from isp...

Referring to your report no : 1-1111111111 regarding on your issue. Upon checking with our respective department, kindly be inform that we unable to provide IPV6 setting / assist third party service. Please be advice to get second party service vendor / email administrator opinion to assist regarding your third party service configuration.

We hope the explanation has reverted to your concern. Your understanding on this matter is highly appreciated.

Honestly, after read the response, mixture feeling of awkward and/or feeling retard of this isp staffs just arouse! I can't stop the urge to write an article to bashing.

As always, we think positive and we want to appreciate what can be salvage for this. Firstly, omg, finally the isp response! :) I have filed cases of ipv6 in the isp router, why it is not supported. Guess what, 5 reports get close with no response. Hey, this is the first time they response! YAY! (asshole ;-) )

Apart from that, I can't think of any reasons other than just bashing. I summarize my thoughts in points.

1. WHY would configuration of IPv6 managed by a third party, second party or EMAIL ADMINISTRATOR ?! oh my fucking jesus christ, email administrator for the configuration?! LOL..

2. "Upon checking with our respective department", so what department actually get check?! toy story department?!

3. I am asking for static IPv6 subnet for my router. I am NOT asking for configuration problem.. ffs, is this idiot support drone even understand what the user asking? From the situation, apparently no. They do not even bother to ask for clarification and answer with second party service vendor/ email administrator as a answer...ROFLMAO.

4. Now why would I ask any third party to configure a static IPv6 ?! This assignment of static IP is done by ISP network admin. Not any third party, period. Well hey, if any third party can configure this in the ISP, and as a third party, can I reconfigure your whole network ?! :D

This ISP support has always been "amazing". I don't supposed even if I reply the response with explanation, that it will get solve technically or god grace static IPv6 subnet will be granted, justified based on past experience and knowledge know how.

Writing it down here and hopefully and believing that someone knowledgeable in the ISP will get the chance to read user response and for instance, in this case, knowledge enough to assign a static IPv6 subnet for its user.

How to check if Debian Jessie, Ubuntu Trusty, Nokia N900 if it IPv6 ready?

With recent rise of IPv6 usage, it is imperative that we understand if our devices are ready for IPv6. Linux kernel supported IPv6 as early as year 1996! Chances are, all these distributions should be IPv6 ready. But for the sake to be sure and learning the basic, we will check these distribution to be sure.

To check, launch a terminal and execute this command as a user.
$ cat /proc/net/if_inet6
fe80000000000000022401fffed782ea 03 40 20 80 eth2
00000000000000000000000000000001 01 80 10 80 lo

You should see the above output and if you are not , maybe the kernel is not compile with ipv6 module. If so, you can enable it and check if it loaded.

# modprobe ipv6
# lsmod | grep ipv6
ipv6 237436 14

You can run the above commands for all the devices, they are all IPv6 ready.



There are many articles out there to disable IPv6 but with the depletion of IPv4 addresses, I think this practice should not continue but be ready and prepared for it. Of cause unless you got good reason not to use IPv6.

Friday, September 12, 2014

Understand basic network configuration in CentOS 7

With the recent release of CentOS7, today we are going to check out the basic network configuration. My usual quick command, ifconfig.
[root@localhost ~]# ifconfig
-bash: ifconfig: command not found

it seem like ifconfig is not longer there, note that if you do upgrade from centos 6.x , you should be aware of this. If you are going to configure network interface, start to get familiar to command ip. But if you want command ifconfig, you can still install the package net-tools.

Let's restart network interface.
[root@centos7-test1 network-scripts]# service network restart
Restarting network (via systemctl): [ OK ]
[root@centos7-test1 network-scripts]# service network status
Configured devices:
lo eth0
Currently active devices:
lo eth0
[root@centos7-test1 init.d]# systemctl restart network
[root@centos7-test1 init.d]# systemctl status network
network.service - LSB: Bring up/down networking
Loaded: loaded (/etc/rc.d/init.d/network)
Active: active (exited) since Tue 2014-07-15 14:33:28 CEST; 13s ago
Process: 11597 ExecStop=/etc/rc.d/init.d/network stop (code=exited, status=0/SUCCESS)
Process: 11753 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=0/SUCCESS)

Jul 15 14:33:27 centos7-test1 network[11753]: Bringing up loopback interface: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Jul 15 14:33:27 centos7-test1 network[11753]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Jul 15 14:33:27 centos7-test1 network[11753]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Jul 15 14:33:28 centos7-test1 network[11753]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Jul 15 14:33:28 centos7-test1 network[11753]: [ OK ]
Jul 15 14:33:28 centos7-test1 network[11753]: Bringing up interface eth0: Connection successfully activated (D-Bus active path: /org/...tion/3)
Jul 15 14:33:28 centos7-test1 network[11753]: [ OK ]
Jul 15 14:33:28 centos7-test1 systemd[1]: Started LSB: Bring up/down networking.
Hint: Some lines were ellipsized, use -l to show in full.

Noticed that service manager now is done via systemctl, C7 is using systemctl in replace of SysV.  Also notice configuration file for ifcfg-lo is not loadable? This issue has been file here.

Upstream has changed the default networking service is provided by NetworkManager, which is a dynamic network control and configuration daemon that attempts to keep network devices and connections up and active when they are available.

If it does not install for any reason (which it should not because it comes with predefault installation), you can follow these commands
# # install it
# yum install NetworkManager
# # ensure network manager service is started everything system boot up.
# systemctl enable NetworkManager
# # manual start for now.
# systemctl start NetworkManager
# # check the status.
[root@centos7-test1 ~]# systemctl status NetworkManager
NetworkManager.service - Network Manager
Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled)
Active: active (running) since Tue 2014-07-15 13:39:18 CEST; 3h 40min ago
Main PID: 679 (NetworkManager)
CGroup: /system.slice/NetworkManager.service
├─ 679 /usr/sbin/NetworkManager --no-daemon
└─11896 /sbin/dhclient -d -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-55911be2-9763-471f...

Jul 15 17:05:21 centos7-test1 NetworkManager[679]: bound to 192.168.0.116 -- renewal in 3581 seconds.
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info> (eth0): DHCPv4 state changed renew -> renew
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info> address 192.168.0.116
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info> plen 24 (255.255.255.0)
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info> gateway 192.168.0.1
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info> server identifier 192.168.0.1
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info> lease time 7200
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info> nameserver '192.168.0.1'
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info> nameserver '8.8.8.8'
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info> domain name 'PowerRanger'

If you are configuring manually remotely, you can use command nmtui. nmtui is a simple curses-based text user interface. But if you want to configure interface using script, better still to use command ip or nmcli. For more information, you can read here.

That's it for this article. I would like to thank my buddy for kind enough to let me ssh and study centos 7 in his host. :) you know who you are! dankeschon!

Sunday, August 31, 2014

How to setup pidgin WhatsApp using credential from Nokia n900

If you own a smart phone from Nokia, model n900, you are in luck to use WhatsApp
on your pc. The intention here is personal usage as sometime you ran out of
power in n900 whilst on a important conversation with friends. By setup up
whatsapp in pidgin messenger chatting software on linux, you can also save the
trouble of switch devices back and forth. This is intended for personal usage.

In this article, we are going to learn how to setup pidgin and so it can connect
to WhatsApp with the registration made in Nokia n900. Of cause, first, in n900,
you will need to install yappari, a whatsapp client for n900 and register yourself
an account in whatsapp. This article will not cover on how to install yappari
in n900 and getting an whatsapp account in yappari because it is very easy.

The official website of this plugin available here.  At the bottom of the site, there are several links to the operating system.

  • Windows/Linux: http://davidgf.net/nightly/whatsapp-purple/

  • Ubuntu/Debian: https://launchpad.net/~whatsapp-purple/+archive/ubuntu/ppa

  • Fedora: https://copr.fedoraproject.org/coprs/davidgf/whatsapp-purple/

  • ArchLinux: https://aur.archlinux.org/packages/purple-whatsapp/


If you do not want to go through the hassle of setting up apt repository , what you can do quickly is by

  1. go to this link

  2. depending on what cpu architecture, if it is 64bit cpu, click on x64/ http://davidgf.net/nightly/whatsapp-purple/x64/

  3. pick the latest version, that is last-whatsapp.so and download to your computer.

  4. then with root access, copy the lib to pidgin plugin directory.
    # cp last-whatsapp.so /usr/lib/purple-2

  5. restart your pidgin.


At this moment of writing, I'm using last-whatsapp.so on the server with date of
this file is 31-Jul-2014 01:02 of 313075 bytes. Meanwhile for pidgin, the version
I'm using in debian is Pidgin 2.10.9 (libpurple 2.10.9) and this works very well for
me.

Once pidgin restarted, go to Manage Accounts and then click on Add button. This is
to add the WhatsApp account that you have setup in yappari. In the pop up Add Account
window, under protocol field, there should be a new protocol WhatsApp available in the
drop down selection. Pick that.

For Username and Password is very tricky here.
Username will be the phone number that you registered in yappari and as for password, you will need some work to retrieve from yappari configuration file in n900. But we will goes through this step by step.

Let's start with the easy one. The username field. It will be your country code follow by your mobile number without the prefix plus sign. For instance, if your mobile sim card is malaysian registered, it will be something like.

Username: 60123456789

Because the password which I'm gonna show you later will be a difficult one, I suggest you check the button Remember password. Unless you are paranoid, you can try to remember your password. Your choice.

For the field Local alias, it will be your name, just put anything that you like to identify yourself.

Now onto the password field, if you notice during registration, there is no procedure nor password sent to you. The only verification WhatsApp need is to identify this is a valid registration when you register an account. Note that WhatsApp code that sent to your phone is not your password.

I have been following the tutorial like using wireshark and tcpdump to get the password, see the attached screen below. This is just not possible because the traffic is encrypted using ssl.
12:07:45.317453 IP (tos 0x0, ttl 64, id 29179, offset 0, flags [DF], proto TCP (6), length 60)
192.168.0.82.62751 > 208.43.122.151-static.reverse.softlayer.com.https: Flags [S], cksum 0x938f (correct), seq 416925910, win 5840, options [mss 1460,sackOK,TS val 2996526 ecr 0,nop,wscale 4], length 0
0x0000: 4500 003c 71fb 4000 4006 bd03 c0a8 0052 E..<q.@.@......R
0x0010: d02b 7a97 f51f 01bb 18d9 c8d6 0000 0000 .+z.............
0x0020: a002 16d0 938f 0000 0204 05b4 0402 080a ................
0x0030: 002d b92e 0000 0000 0103 0304 .-..........
12:07:46.135812 IP (tos 0x0, ttl 54, id 37650, offset 0, flags [DF], proto TCP (6), length 60)
208.43.122.151-static.reverse.softlayer.com.https > 192.168.0.82.62751: Flags [S.], cksum 0xb738 (correct), seq 2641574608, ack 416925911, win 65535, options [mss 1452,nop,wscale 9,sackOK,TS val 3690413789 ecr 2996526], length 0
0x0000: 4500 003c 9312 4000 3606 a5ec d02b 7a97 E..<..@.6....+z.
0x0010: c0a8 0052 01bb f51f 9d73 3ad0 18d9 c8d7 ...R.....s:.....
0x0020: a012 ffff b738 0000 0204 05ac 0103 0309 .....8..........
0x0030: 0402 080a dbf7 3edd 002d b92e ......>..-..
12:07:46.136301 IP (tos 0x0, ttl 64, id 29180, offset 0, flags [DF], proto TCP (6), length 52)
192.168.0.82.62751 > 208.43.122.151-static.reverse.softlayer.com.https: Flags [.], cksum 0xe429 (correct), seq 1, ack 1, win 365, options [nop,nop,TS val 2996630 ecr 3690413789], length 0
0x0000: 4500 0034 71fc 4000 4006 bd0a c0a8 0052 E..4q.@.@......R
0x0010: d02b 7a97 f51f 01bb 18d9 c8d7 9d73 3ad1 .+z..........s:.
0x0020: 8010 016d e429 0000 0101 080a 002d b996 ...m.).......-



That's impossible to decode the traffic if you do not have good knowledge on ssl but that's the whole point of ssl encrypt the message in the transport. So retrieving password via sniffing on the network packet will not work. We will now go to n900 and retrieve the password.

  1. open a X terminal in n900. (if you do not have, you should install now)

  2. change directory to .config/scorpius and check your current directory should be /home/user/.config/scorpius
    $ cd .config/scorpius
    $ pwd
    /home/user/.config/scorpius 

  3. check current directory content with ls
    $ ls
    counters.conf yappari.conf yappari.log

  4. what is important is yappari.conf where it contain the password that is needed. so cat yappari.conf
    $ cat yappari.conf
    [General]
    whatsnew=555555555
    imsi=502121212121212
    registered=true
    number=123456789
    cc=60
    phonenumber=60123456789
    password="ABCDEFGHIJKLMNOPQRSTUVWXYZ/="
    username=JohnSmith
    creation=1407484663
    expiration=1439020663
    kind=free
    accountstatus=active
    lastsync=1407484676392
    status=Available
    lastimagedir=/home/user/MyDocs/DCIM
    nextchallenge="APAPAPAPAPAPAPAPAPAPAPAPAPAP"


You will see similar content as of above, and you should copy and paste the password to your pidgin. Note, because it is long, you might want to copy this file out and copy and paste it.


Fill in the password from step 4 into pidgin password field. Note that below is just an example of demonstration, you should replace your own value.


Password: ABCDEFGHIJKLMNOPQRSTUVWXYZ/=





When you click the checkbox for 'Enabled' for your account, it should now connect.


WhatsApp has a smiley theme called emoji. So you might want to install that as WhatsApp users normally will send in emoji that pidgin will not able to decode and display as a rectangular box with hexadecimal. To install emoji for your what's app, you can follow these steps.



  1. read introduction at https://github.com/davidgfnet/whatsapp-purple/blob/master/README.md#how-do-i-get-graphical-whatsapp-smileys

  2. download the unicode-emoji and emoji-for-pidgin to your home directory.

  3. extract the zip files and copy the directory to your pidgin home.
    $ cp -R android apple symbola $HOME/.purple/smileys
    $ cp -R Emoji-for-Pidgin $HOME/.purple/smileys
    $ ls $HOME/.purple/smileys
    android apple Emoji-for-Pidgin symbola

  4. restart your pidgin and go to Tools -> Preferences -> Themes.

  5. under Smiley Theme, select the emoji you want. :)


That's it. Start sending WhatsApp message from your pc!




UPDATE 


If you have setup whatsapp on pidgin using this published article during the period on 31 August 2014 to 22 November 2014, you should really get the update again. Then in the setting for this whatsapp account in pidgin, under Advanced tab, in the resource field , change to Android-2.31.151-443. Restart pidgin and it should connect again.

Saturday, August 30, 2014

What should you do if the server you administered got hacked.

If you realized that your server has been compromised, this discovery will create confusion, reduce confidence and if the server is serving user requests, you have to declare down time. That's not good.

In order to restore service as quickly as possible, it is best if you have a server ready to replace instantly, that you can reduce the noise from the customers. But in order to prevent such attack coming in the future, you must at least identify how it happened and taking counter measurement.

In this article, we will learn how to discover, and then taking counter measure.

Quick solution.

Probably the quickest solution is to format and reinstall the operating system together with your applications that serve user requests. This probably is good if you do not have a backup server and you want to reinstate the server to serve user requests as soon as possible. But this does not solve the actual problem on how the hacked took place. Hence, it might happen again in the near future.

Long and workable solution.

  1. identify your own custom application deployed and start to investigate from there.

  2. update the system using package manager and restart system.

  3. tighten up security


identify your own custom application deployed and start to investigate from there.

Because open source are mostly tested well and updated often, the first place you are going to investigate mostly come from your own application. Hence, you must at least have good understanding about your app and so to quickly identify source of problem.

Following are a sets of commands which might able to help you in your investigation.

  • w
    who is on the server

  • sudo netstat -nalp | grep ":22"
    change 22 to your application listen to. check if there is any abnormally.

  • if you are using opensource for your custom applications, check the log as well. For which attacker will always find the exploit for the opensource softwares and started to target those.


update the system using package manager and restart system.

First you can start by checking.

  • last
    check when was invalid last access.

  • cat /var/log/secure* | grep Accept
    check invalid access.

  • ps -elf
    check if the malware is running and if you spot one, get the process where it run from and delete all malware files.

  • ls /tmp /var/tmp /dev/shm -la
    this directory normally allow process to write in, so you might want to check any fishy files here.

  • file <filename>
    check what type of the file.

  • cat /etc/passwd
    check if there is unknown entry which is not supposed to be there.

  • sudo netstat -plant |awk ' /^tcp/ {split($7, a, "/"); print $6, a[2]}' |sort | uniq -c | sort -n| tail
    4 ESTABLISHED java
    4 LISTEN kadmind
    5 LISTEN java
    5 LISTEN python
    6 ESTABLISHED python
    if your server has been turned into a trojan, the malware will probably launching a lot of ddos, with this command, you should be able to identify if the cp connection has been spike.

  • sudo netstat -plant | awk '$4 ~ /:22$/ {print $5}' | cut -f1 -d: | sort | uniq -c | sort -n
    1
    1 0.0.0.0
    2 192.168.0.2
    check total connection established to your server on port 22.

  • sudo netstat -plant | awk '/^tcp/ {print $6}' | sort | uniq -c | sort -n
    2 CLOSING
    4 SYN_RECV
    5 LAST_ACK
    6 FIN_WAIT1
    12 LISTEN
    13 FIN_WAIT2
    344 TIME_WAIT
    977 ESTABLISHED
    check network states, this is a good information should your server suddenly spike in the state established or state syn. if there is any spike, you will know something maybe gone fishy.

  • $HOME/.bash_history
    check every users bash_history to see if there is any suspect. If the server application run user a user id, especially check the bash_history in the user home directory.

  • find / -mtime 5
    find what files has been changes since 5 days ago.


If there is nothing found, just update the system packages using package manager and reboot the system.

tighten up security and monitor

if you have a loose firewall policy (iptables or some hardware firewall), you should review it.

Prevention in the future would probably notify when the count of TCP connection exceed or suddenly spike to a threshold.

 

whilst these steps are not exhaustive, as evil people always come with different type attacks, thus you should be prepare and be alert. Gather information using google as well.