Sunday, April 26, 2015

Benchmarking unigine heaven on debian

In this article, we are trying something different, a good buddy of mine asking me to do graphic benchmark on linux system. So let's roll. Start by, downloading the benchmark application here at

This benchmark application size about 290MB, so while waiting for the download to complete, you should probably check if your graphic card has 3d driver installed and enabled. You can check by running in the terminal with the command glxgears, see screenshot below.


So make sure the gearing windows pop up or you will need to solve the problem shown in the terminal if it is not. Once the benchmark application is downloaded, you need to unpack and run it. See below.
user@localhost:~$ sh 
Creating directory Unigine_Heaven-4.0
Verifying archive integrity... All good.
Uncompressing Unigine Heaven Benchmark.............................................................................
Unigine Heaven Benchmark installation is completed. Launch heaven to run it
user@localhost:~$ cd Unigine_Heaven-4.0
user@localhost:~/Unigine_Heaven-4.0$ ls
total 16K
-rwxr-xr-x 1 jason jason 278 Feb 13 2013 heaven
drwxr-xr-x 4 jason jason 4.0K Feb 13 2013 bin
drwxr-xr-x 2 jason jason 4.0K Feb 13 2013 documentation
drwxr-xr-x 3 jason jason 4.0K Feb 13 2013 data
user@localhost:~/Unigine_Heaven-4.0$ ./heaven
Loading "/home/user/Unigine_Heaven-4.0/bin/../data/heaven_4.0.cfg"...
Loading ""...
Loading ""...
Loading ""...
Set 1920x1080 fullscreen video mode
Set 1.00 gamma value
Unigine engine
Binary: Linux 64bit GCC 4.4.5 Release Feb 13 2013 r11274
Features: OpenGL OpenAL XPad360 Joystick Flash Editor
App path: /home/user/Unigine_Heaven-4.0/bin/
Data path: /home/user/Unigine_Heaven-4.0/data/
Save path: /home/user/.Heaven/

---- System ----
System: Linux 3.9-1-amd64 x86_64
CPU: Intel(R) Core(TM) i3 CPU 380 @ 2.53GHz 2526MHz MMX SSE SSE2 SSE3 SSSE3 SSE41 SSE42 HTT x4
GPU: Unknown GPU x1
System memory: 7869 MB
Video memory: 256 MB
Sync threads: 3
Async threads: 4

---- MathLib ----
Set SSE2 simd processor

---- Sound ----
Renderer: OpenAL Soft
OpenAL vendor: OpenAL Community
OpenAL renderer: OpenAL Soft
OpenAL version: 1.1 ALSOFT 1.15.1
Found EFX Filter
Found EFX Reverb
Found EAX Reverb
Found QUAD16 format
Found 51CHN16 format
Found 61CHN16 format
Found 71CHN16 format
Maximum sources: 256
Maximum effect slots: 4
Maximum auxiliary sends: 2

---- Render ----
GLRender::GLRender(): Unknown GPU
OpenGL vendor: X.Org
OpenGL renderer: Gallium 0.4 on AMD REDWOOD
OpenGL version: 3.2 (Core Profile) Mesa 10.2.8
OpenGL flags: Core Profile
Found required GL_ARB_map_buffer_range
Found required GL_ARB_vertex_array_object
Found required GL_ARB_draw_instanced
Found required GL_ARB_draw_elements_base_vertex
Found required GL_ARB_transform_feedback
Found required GL_ARB_half_float_vertex
Found required GL_ARB_half_float_pixel
Found required GL_ARB_framebuffer_object
Found required GL_ARB_texture_multisample
Found required GL_ARB_uniform_buffer_object
Found required GL_ARB_geometry_shader4
Found optional GL_EXT_texture_compression_s3tc
Found optional GL_ARB_texture_compression_rgtc
Shading language: 3.30
Maximum texture size: 16384
Maximum texture units: 48
Maximum texture renders: 8

---- Physics ----
Physics: Multi-threaded

---- PathFind ----
PathFind: Multi-threaded

GPUMonitorPlugin::init(): can't initialize GPUMonitor
EnginePlugins::init(): can't initialize "GPUMonitor" plugin
---- Interpreter ----
Version: 2.52

Loading "heaven/unigine.cpp" 60ms
Unigine~# render_restart
Loading "heaven/locale/unigine.en" dictionary
Loading "core/materials/default/unigine_post.mat" 23 materials 50 shaders 34ms
Loading "core/materials/default/unigine_render.mat" 47 materials 2368 shaders 17ms
Loading "core/materials/default/unigine_mesh.mat" 5 materials 3386 shaders 15ms
Loading "core/materials/default/unigine_mesh_lut.mat" 2 materials 1062 shaders 4ms
Loading "core/materials/default/unigine_mesh_paint.mat" 2 materials 1158 shaders 8ms
Loading "core/materials/default/unigine_mesh_tessellation.mat" 5 materials 3332 shaders 15ms
Loading "core/materials/default/unigine_mesh_tessellation_paint.mat" 2 materials 2276 shaders 9ms
Loading "core/materials/default/unigine_mesh_triplanar.mat" 1 material 112 shaders 2ms
Loading "core/materials/default/unigine_mesh_overlap.mat" 1 material 300 shaders 4ms
Loading "core/materials/default/unigine_mesh_terrain.mat" 1 material 813 shaders 5ms
Loading "core/materials/default/unigine_mesh_layer.mat" 1 material 84 shaders 1ms
Loading "core/materials/default/unigine_mesh_noise.mat" 1 material 106 shaders 2ms
Loading "core/materials/default/unigine_mesh_stem.mat" 2 materials 2180 shaders 16ms
Loading "core/materials/default/unigine_mesh_wire.mat" 1 material 45 shaders 1ms
Loading "core/materials/default/unigine_terrain.mat" 1 material 1980 shaders 9ms
Loading "core/materials/default/unigine_grass.mat" 2 materials 474 shaders 5ms
Loading "core/materials/default/unigine_particles.mat" 1 material 109 shaders 2ms
Loading "core/materials/default/unigine_billboard.mat" 1 material 51 shaders 1ms
Loading "core/materials/default/unigine_billboards.mat" 2 materials 840 shaders 4ms
Loading "core/materials/default/unigine_volume.mat" 6 materials 53 shaders 5ms
Loading "core/materials/default/unigine_gui.mat" 1 material 82 shaders 0ms
Loading "core/materials/default/unigine_water.mat" 1 material 533 shaders 24ms
Loading "core/materials/default/unigine_sky.mat" 1 material 21 shaders 16ms
Loading "core/materials/default/unigine_decal.mat" 1 material 99 shaders 1ms
Loading "core/properties/unigine.prop" 2 properties 0ms
Unigine Heaven Benchmark 4.0 (4.0)Unigine~# world_load heaven/heaven
Loading "heaven/heaven.cpp" 152ms
Loading "heaven/materials/heaven_base.mat" 7 materials 10ms
Loading "heaven/materials/heaven_environment.mat" 13 materials 838ms
Loading "heaven/materials/heaven_ruins.mat" 27 materials 2101ms
Loading "heaven/materials/heaven_buildings.mat" 58 materials 2116ms
Loading "heaven/materials/heaven_props.mat" 10 materials 412ms
Loading "heaven/materials/heaven_sfx.mat" 11 materials 8ms
Loading "heaven/materials/heaven_fort.mat" 15 materials 544ms
Loading "heaven/materials/heaven_airship.mat" 26 materials 5176ms
Loading "heaven/" 13817ms
Unigine~# render_restart
Unigine~# video_grab
Saving /home/user/.Heaven/screenshots/00000.tga
Unigine~# video_grab
Saving /home/user/.Heaven/screenshots/00001.tga
Unigine~# video_grab
Saving /home/user/.Heaven/screenshots/00002.tga
Unigine~# render_restart
Benchmark running
Benchmark results:
Time: 261.689
Frames: 1286
FPS: 4.91422
Min FPS: 3.69493
Max FPS: 13.4421
Score: 123.789
Unigine~# quit
Close ""
Close ""
Memory usage: none
Allocations: none

This benchmark was performed on an OLD system, hence, score was very low. Otherwise, it ran fine, screenshots were taken and since benchmark is available for linux, I'm pretty sure more games will develop on linux. Come onbard on linux for better gaming experience. :) Enjoy the screenshots below.

unigine_heaven_benchmark_screenshot_0 unigine_heaven_benchmark_screenshot_1

Saturday, April 25, 2015

My way of solving tomcat memory leaking issue

Recently, I did a mistake by accidentally commit a stupid static codes into a static method into production causing heap usage grow tremendously. Since the static method stay persisted with the object, tomcat has to restart often to free up the heap that get hold. So today, I will share my experience on how I solve it and I hope it will give you a way on how to solve this difficult problem.
First is the to end, I will summarize the sequence you need to investigate and find out the fix.

* learn on how to find the memory leak using google.
* one step at a time to trace until you successfully pin down the problem and fix it.

As you can read, only three general steps but for each step, I will talk more about it.

Always check your code by reading and tests! Best if you have someone experience and you can probably send your code for inspection. Remember, 4 eyes ball and 2 brains are better than 2 eyes ball and a brain. If you are using opensource project, most probably, the library are well tested and you should just spend time to investigate your codes. It's difficult especially for new programmer, but that should not stopped you to find out the problem. If you still cannot find out the problem, then you should start to search on search engine on how people solve it.
learn on how to find the memory leak using google.
Nobody is perfect and know everything, but if you are unsure, always google away. Google keyword such as java memory leak, tomcat memory leak or even best java coding practice. Pay attention on the first 10 links return by google and then read on blogging or even stackoverflow, it will give you knowledge that you never know of. Example of tools needed include jstat, jmap, jhat, and visualvm that can give you an idea what or even where might be the problem from. Remember, reading this material is a way of growing and it take times, so please be patience at this step and make sure u spend adequate amount of time and jot down important points mentioned and so you can use it on final step.

one step at a time to trace until you successfully pin down the problem and fix it.
Final step would probably repeating step 1 and step 2 slowly to determine the root cause. If you are using versoning system, you should really find out when was the last best working codes and start to check file by file where the problem was introduced. This is a TEDIOUS and DAUNTING process but this is effective to solving the root cause.
These steps were used by myself during determine the tomcat web application memory problem. Thank you and I hope you can benefit too.

Friday, April 24, 2015

Learning java jstat

Today, we will going to learn a java tool, which is incredibly useful if you are frequent coding for java application. This java tool is a monitoring tool known as jstat and it came with jdk. So you would ask why would I need to use jstat, my app run just fine. So for a simple java application, yes, you do not need to this monitoring tool. However if you have a long running application or big java codebase application, and sometime when your java application run midway hang (pause/freeze), then you should start to look into this tool really. In this article, I'm going to show you how I use it.

But first, let understand on what is jstat.
The jstat tool displays performance statistics for an instrumented HotSpot Java virtual machine (JVM).

As you aware, object that you wrote in the code will eventually get free from heap when it is not reference. If you has a lot of objects and heap usage grow, then you can use this monitoring tool to check out wassup of the heap allocation. Okay now, let's read into the command input.
jstat [ generalOption | outputOptions vmid [interval[s|ms] [count]] ]

so pretty simple, the commands jstat followed by a few parameters. The parameters can be explain below. You can find official documentation here.

A single general command-line option (-help or -options)

One or more output options, consisting of a single statOption, plus any of the -t, -h, and -J options.

Virtual machine identifier, a string indicating the target Java virtual machine (JVM). The general syntax is
The syntax of the vmid string largely corresponds to the syntax of a URI. The vmid can vary from a simple integer representing a local JVM to a more complex construction
specifying a communications protocol, port number, and other implementation-specific values. See Virtual Machine Identifier for details.

Sampling interval in the specified units, seconds (s) or milliseconds (ms). Default units are milliseconds. Must be a positive integer. If specified, jstat will produce its
output at each interval.

Number of samples to display. Default value is infinity; that is, jstat displays statistics until the target JVM terminates or the jstat command is terminated. Must be a
positive integer.

It should be very clear to you if you are season java coder and if you don't, take a look at an example below.
[iser@localhost ~]$ jstat -gcutil 12345 1s
10.08 0.00 70.70 69.22 59.49 122328 4380.327 355 43.146 4423.474
10.08 0.00 84.99 69.22 59.49 122328 4380.327 355 43.146 4423.474
0.00 15.62 0.00 69.24 59.49 122329 4380.351 355 43.146 4423.497

so jstat is instrument a local jvm with process id 12345 with an interval of 1 second and loop infinitely. There are different type of statistics can be shown and with the above example given, it show summary of garbage collection statistics. If you want to shown different types of gc statistics, you can use the command jstat -options and below is the table of summaries what these options display means.
Option 	                Displays...
class Statistics on the behavior of the class loader.
compiler Statistics of the behavior of the HotSpot Just-in-Time compiler.
gc Statistics of the behavior of the garbage collected heap.
gccapacity Statistics of the capacities of the generations and their corresponding spaces.
gccause Summary of garbage collection statistics (same as -gcutil), with the cause of the last and current (if applicable) garbage collection events.
gcnew Statistics of the behavior of the new generation.
gcnewcapacity Statistics of the sizes of the new generations and its corresponding spaces.
gcold Statistics of the behavior of the old and permanent generations.
gcoldcapacity Statistics of the sizes of the old generation.
gcpermcapacity Statistics of the sizes of the permanent generation.
gcutil Summary of garbage collection statistics.
printcompilation HotSpot compilation method statistics.

Out of all these options, probably the most frequently you will use is gcutil, gc and gccapacity. We will look at them with example. Please note that in order to protect the privacy of the user, there are some information is removed but what need to be presented in this article shall remained as is.

option gcutil


As can be read above, the command jstat with option gcutil on a java process id 23483. The statistics are generated with an interval at 1 second. It has 10 columns and these column can be explain below.
Column 	Description
S0 Survivor space 0 utilization as a percentage of the space's current capacity.
S1 Survivor space 1 utilization as a percentage of the space's current capacity.
E Eden space utilization as a percentage of the space's current capacity.
O Old space utilization as a percentage of the space's current capacity.
P Permanent space utilization as a percentage of the space's current capacity.
YGC Number of young generation GC events.
YGCT Young generation garbage collection time.
FGC Number of full GC events.
FGCT Full garbage collection time.
GCT Total garbage collection time.

First five columns depict space utilization in term of percentage. The next five depict amount of young generation collection and its time, full garbage collection and its time and last, total garbage collection time. With this screen capture, we see that the eden space is filling up quickly and promoted to either survivor space 0 or survivor space 1. At one instance, some object survived and eventually promoted to old space and increased the usage by 0.01% to 5.24%. Note that also YGC is increased by one as a result to 256. This young generation collection time took 13 milliseconds. Similar pattern happen again later and we see that, YGC is increased by oen to 257 with another 13 milliseconds of collection time. In this output, there is no change to full collection, which is good. It is only one full collection happened but with a pause of 94millseconds! You might want to keep an eye on the E column so it dont fill up quickly and adjust hte young gen in your java app accordingly. But for a long term solution, you might want to spend some time to find out which code take a lot of resources and improve it.

option gc

jstat-gcAs can be read above, the command jstat with option gc on a java process id 28276. The statistics are generated with an interval at 1 second. It has 15 columns and these column can be explain below.
Column 	Description
S0C Current survivor space 0 capacity (KB).
S1C Current survivor space 1 capacity (KB).
S0U Survivor space 0 utilization (KB).
S1U Survivor space 1 utilization (KB).
EC Current eden space capacity (KB).
EU Eden space utilization (KB).
OC Current old space capacity (KB).
OU Old space utilization (KB).
PC Current permanent space capacity (KB).
PU Permanent space utilization (KB).
YGC Number of young generation GC Events.
YGCT Young generation garbage collection time.
FGC Number of full GC events.
FGCT Full garbage collection time.
GCT Total garbage collection time.

The statistics shown the capacity in term of kilobytes. First ten columns are pretty easy, the space capacity and its current utilization. The last five columns are the same as gcutil last five columns. Notice that when the column EU value near to the column EC value, young generation collection happened. Object promoted to survivor spaces. Notice that column OU grow gradually. This statistics almost the same with gcutil except that the statistics shown here display in term of bytes whereas gcutil statistics display in term of percentage.

option gccapacity


As can be read above, the command jstat with option gccapacity on a java process id 13080. The statistics are generated with an interval at 1 second. It has 16 columns and these column can be explain below.
Column 	Description
NGCMN Minimum new generation capacity (KB).
NGCMX Maximum new generation capacity (KB).
NGC Current new generation capacity (KB).
S0C Current survivor space 0 capacity (KB).
S1C Current survivor space 1 capacity (KB).
EC Current eden space capacity (KB).
OGCMN Minimum old generation capacity (KB).
OGCMX Maximum old generation capacity (KB).
OGC Current old generation capacity (KB).
OC Current old space capacity (KB).
PGCMN Minimum permanent generation capacity (KB).
PGCMX Maximum Permanent generation capacity (KB).
PGC Current Permanent generation capacity (KB).
PC Current Permanent space capacity (KB).
YGC Number of Young generation GC Events.
FGC Number of Full GC Events.

These output is similar to the output of option gc but with minimum and maximum for the individual java heap.

That's it for this article and I will leave three links for your references.


Friday, April 10, 2015

A year of active blogging information technology articles

Time flies, it's been a year I started to blog actively and persistently. Today, we are not going to learn any information technology related article. Today, I will share my thoughts about my blogging experience and journey.

Reading through various people's blogs on information technology article all these while make me learn a lot. There is also time when I encounter questions in real life or working life and then started to research further on the topic in concern. So I asked myself, well, why not write out my research and typing into a blog? Not only that I could reference this writing in the future, but I can also share this knowledge to the people that are interested in. These are my main motivation to keep on writing and contributing back especially to the opensource community.

These blogging continue until today and it took a lot of efforts and time to research on the topic. Because there are many generous people also blogging and I thank all of them for which I reference in the past either quoted their write up, code snippets and/or links. As time progress, I thought of sharing my article by publishing the link into social media and so more people can read about it. As a programmer, I have google a lot for topic that interest me and this spark an idea to me to make this site searchable in google and so people who need help able to reach these article through search engine too.

There are times when I felt to give up writing either due to hectic work, already no time for family and/or for self, like a miracle, people across the world who read and favorited either in twitter, leave comment in the blog, +1 in google, people show appreciation for my blog that helped them in their life drive and motivate me to continue or even drive me higher to produce more quality blog.

In the future, I hope with more strength and more time for myself, I could have the opportunity to continue write information technology related blog. As writing require time and efforts, it would be great to receive donation either to acknowledge to the blogger or to my friend that host this site. It would means a lot to us now and into the future.

Last but not least, there maybe mistakes in the articles written and this is not intended but please comment in the article and so it can be correct. Any feedback on how to improve this site or article, please let me know either email me, message me or leave your comment below.

Till then for the next article,

Jason Wee

Sunday, March 29, 2015

My journey and experience on upgrading apache cassandra 1.0.8 to 1.0.12

Upon request of my blog reader, today I will share with you my experience on upgrading apache cassandra version 1.0.8 to 1.0.12 on a production live cluster. By sharing this information, I hope if you are also running and/or administer cassandra cluster, you can learn from my experience and ease your worry or pain.

First, let's lay out what's the current architecture in this environment.

  • java 6

  • 12 nodes cluster.

  • two spinning disk with raid 0, 32GB total system memory where 14GB allocated to the cassandra heap instance, with 800MB for young gen. quad core cpu.

  • pretty much stock cassandra.yaml configuration with the following different like concurrent_write to 64, flush_largest_memtables_at to 0.8, compaction_throughput_mb_per_sec to 64.

  • node load per node average at 500-550GB.

As you can see, this is pretty ancient cassandra we are using at of this time of writing but because cassandra has been rock solid serving read/write requests for years, it stays like this stable condition forever and we leverage on the benefit of scalling out like adding nodes from six to nine and eventually to twelve now. Realizing that the disk failure do happened in the nodes of the cluster, because of cassandra has a no single point of failure in mind, we can afford to loose a single node out of operation while replacing it. That were a few of the reasons we stayed with cassandra 1.0 for quite sometime.

Because we would like to probably goes to cassandra 2.0 and beyond, and java 6 has been EOL for quite sometime, it would be wise to upgrade java before cassandra. Because system are integrated like an ecosystem, it would be also wise to look at java used in the client system that read/write requests to the cassandra cluster. So make a checklist brainstorming what are clients that integrate into the cluster and then check out what are the current stable java 7 available. Example:

cassandra 1.0 cassandra-1.0.12 java miniumum 6 and above.

hector client using casandra 2.0.4 so java 7 minimum

datastax cql driver use cassandra 2.1.2 so java 7 minimum

java 7 update release note

features and enhancement

java 7 in wiki

before upgrading, check if cassandra using different unicode on the data
Early versions of the Java SE 7 release added support for Unicode 5.1.0. The final version of the Java SE 7 release supports Unicode 6.0.0. Unicode 6.0.0 is a major version of the Unicode Standard and adds support for over 2000 additional characters, as well as support for properties and data files.

As of the time of checking, we picked java 7 update 72. Upgrading java 6 to java 7 update 72 in the cassandra 1.0.8 is a painless process other than just time consuming. As load per node is huge and total number of nodes in cluster. I follow this steps for java upgrade in cassandra node.

upgrade java for all cassandra node
1. write a script to automatically install java7 on node, update java stacked size to 256k in set JAVA_HOME for file to java 7.
2. execute the script in rolling fashion for all the node with one upgrade at a time.
3. stop cassandra
4. execute the script.
5. start the cassandra instance
6.0 start the cassandra instance and monitor after the node is up and then check the monitoring system after node elapsed for 30minutes, 60minutes, 1hours and 2hours.
6.1 check your client can read/write to that one upgraded node.

By now, you can perform the next node in the ring, but you can skip step 6.0 as you are sure that it is going to work. One thing I observed is that, the gc duration for cassandra using java 6 and java 7 is it is down by half! That's could means faster gc means more cpu cycle to process other tasks and less stop of the world for cassandra instance.

Leave this cluster with java 7 upgraded run a day or two and if it is okay, continue to cassandra upgrade. So which cassandra version to upgrade to? There are several guidelines I followed.

1. choose ONLY STABLE release for production cluster. How to choose? You should read this link.
2. read NEWS.txt  and Changes.txt . As time to time, change to the code base may affect example, the sstable. So pay attention especially between cassandra major upgrade.
3. read the code difference between the version you decided to upgrade too, example for this upgrade.
4. read the datastax upgrading node for minor version.

I spent a lot of time doing step 3 and by reading the code diference, realize what has been change and/or added and consider it will impact your cassandra environment. In order for further upgrade to cassandra 1.1, you will need to upgrade to the latest version of the one currently deployed. Example here. Once read the above checkpoints, you may have a lot of questions and TODOs and that will give further works. In the next step, it is best if you find out the questions and TODOs you have and then verify in the test cluster before apply to a production cluster.

For me, I have written a few bash scripts example mentioned above, java upgrade. Also I have written install test cluster for cassandra upgrade. Remember to also write script to snapshot the data directory using nodetool and then also write script to automatically downgrade. When something goes wrong, you can revert using the automatic downgrade script and using the backup from nodetool dump. Then you will need to save the configurations example,,, cassandra.yaml or any other in your environment cluster.

With these scripts written and tested, it is best if you get and acknowledgements from the management if this is to be proceed and also, it would be best if you have someone who is also administer of cassandra cluster with you just for the good and bad moments. ;-) You can also reach me by my follow button in the home page. :)

upgrade cassandra from 1.0.8 to 1.0.12

  1. stop repair and cleanup in all nodes in the cluster.

  2. write a script to automatically upgrade it and so you dont panic, waste time and composed during node upgrade. Trust me, save you a lot of time and human error free. scripts content could be the following:
    - download cassandra 1.0.12 and extract, file permission ,etc
    - backup current cassandra 1.0.8 using nodetool snapshots. make sure you write the snapshot directory name like MyKeyspace-1.0.8-date
    - drain the node.
    - stop cassandra if it is not yet stopped.
    - update cassandra 1.0.12 with your cluster settings.

  3. check the configuration changed and then start cassandra 1.0.12 new instance.

  4. monitor after the node is up and then check the monitoring system after node elapsed for 30minutes, 60minutes, 1hours and 2hours.

  5. check your client can read/write to that one upgraded node.

By now, you can perform the next node in the ring, but you can skip step 4.0 as you are sure that it is going to work. As the version of the cassandra sstable change in 1.0.10, from hc to hd, it is best all sstables in all nodes, using the hd version before perform the next major upgrade.

That's it for this article and whilst this maybe not cover all, may contain mistake, and/or if you want to comment, please leave your comment below.

Saturday, March 28, 2015

Investigate into apache cassandra corrupt sstable exception

Today, we will take a look at another apache cassandra 1.0.8 exception. Example of stack trace below.
ERROR [SSTableBatchOpen:2] 2015-03-07 06:11:58,544 (line 228) Corrupt sstable /var/lib/cassandra/data/MySuperKeyspace/MyColumnFamily-hc-6681=[Index.db, Statistics.db, CompressionInfo.db, Filter.db, Data.db]; skipped Input/output error
at Method)
at java.util.concurrent.Executors$
at java.util.concurrent.ThreadPoolExecutor.runWorker(
at java.util.concurrent.ThreadPoolExecutor$

Before we go into the code base for this stacktrace, I have no idea what is this about and this one shown when the cassandra 1.0.12 instance is booting up. Last I remember I trigger user defined compaction twice in cassandra 1.0.8 using the same sstables and after first compaction is done, then this sstable stay forever... like for two weeks plus. Then we have upgrade for the cassandra.

Enough said, let's go into the code base and understand what is really mean by corrupt sstable. Bottom of the the stack trace pretty obvious, ThreadPoolExecutor execute a future task run method.Then it is now on apache cassandra namespace codebase, as can be read below class SSTableReader, method batchOpen(), code snippet
    public static Collection<SSTableReader> batchOpen(Set<Map.Entry<Descriptor, Set<Component>>> entries,
final Set<DecoratedKey> savedKeys,
final DataTracker tracker,
final CFMetaData metadata,
final IPartitioner partitioner)
final Collection<SSTableReader> sstables = new LinkedBlockingQueue<SSTableReader>();

ExecutorService executor = DebuggableThreadPoolExecutor.createWithPoolSize("SSTableBatchOpen", Runtime.getRuntime().availableProcessors());
for (final Map.Entry<Descriptor, Set<Component>> entry : entries)
Runnable runnable = new Runnable()
public void run()
SSTableReader sstable;
sstable = open(entry.getKey(), entry.getValue(), savedKeys, tracker, metadata, partitioner);
catch (IOException ex)
logger.error("Corrupt sstable " + entry + "; skipped", ex);

executor.awaitTermination(7, TimeUnit.DAYS);
catch (InterruptedException e)
throw new AssertionError(e);

return sstables;


As can be read above, somewhere within the method open() throw the IOException, hence the above exception was thrown. Two stack trace up, we read that, sstable load method execute and, method. With the method read from class ByteBufferUtil as shown below.
    public static ByteBuffer read(DataInput in, int length) throws IOException
if (in instanceof FileDataInput)
return ((FileDataInput) in).readBytes(length);

byte[] buff = new byte[length];
return ByteBuffer.wrap(buff);

We see that, the input in a instance of FileDataInput stream and read the bytes with length. Since FileDataInput is a interface, we read that, the class that implement this interface is RandomAccessReader class and method readBytes as the follow.
public ByteBuffer readBytes(int length) throws IOException
assert length >= 0 : "buffer length should not be negative: " + length;

byte[] buff = new byte[length];
readFully(buff); // reading data buffer

return ByteBuffer.wrap(buff);

to read bytes with length is actually to read fully on the length but started on the current file pointer pointing at. And a little bit way up in the stack trace, method reBuffer()
* Read data from file starting from current currentOffset to populate buffer.
* @throws IOException on any I/O error.
protected void reBuffer() throws IOException

if (bufferOffset >= channel.size())

channel.position(bufferOffset); // setting channel position

int read = 0;

while (read < buffer.length)
int n =, read, buffer.length - read);
if (n < 0)
read += n;

validBufferBytes = read;

bytesSinceCacheFlush += read;

if (skipIOCache && bytesSinceCacheFlush >= MAX_BYTES_IN_PAGE_CACHE)
// with random I/O we can't control what we are skipping so
// it will be more appropriate to just skip a whole file after
// we reach threshold
CLibrary.trySkipCache(this.fd, 0, 0);
bytesSinceCacheFlush = 0;

and this method call superclass to read another chunk into the buffer. The upper class RandomAccessFile , method readBytes()
* Reads a sub array as a sequence of bytes.
* @param b the buffer into which the data is read.
* @param off the start offset of the data.
* @param len the number of bytes to read.
* @exception IOException If an I/O error has occurred.
private int readBytes(byte b[], int off, int len) throws IOException {
Object traceContext = IoTrace.fileReadBegin(path);
int bytesRead = 0;
try {
bytesRead = readBytes0(b, off, len);
} finally {
IoTrace.fileReadEnd(traceContext, bytesRead == -1 ? 0 : bytesRead);
return bytesRead;

private native int readBytes0(byte b[], int off, int len) throws IOException;

.. and we are at the end of this path, it turn out that the call to readBytes0 thrown exception, the lower layer native non java call throwing the IO exception. You can use nodetool scrub to see if this fix the problem but what I do basically wipe the data directory for the cassandra and rebuild it. Then I don't see anymore of this message anymore.

That's it for this article and if you want to improve and/or comment, please leave your input below.

Friday, March 27, 2015

Investigate into apache cassandra get_slice assertion error

Today, we will investigate another error from apache cassandra. Error as shown below in cassandra log.
ERROR [Thrift:2] 2015-02-11 11:06:10,837 (line 3041) Internal error processing get_slice
at org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(
at org.apache.cassandra.locator.TokenMetadata.firstToken(
at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(
at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(
at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(
at org.apache.cassandra.service.StorageProxy.fetchRows(
at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(
at org.apache.cassandra.thrift.CassandraServer.getSlice(
at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(
at org.apache.cassandra.thrift.CassandraServer.get_slice(
at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(
at org.apache.cassandra.thrift.Cassandra$Processor.process(
at org.apache.cassandra.thrift.CustomTThreadPoolServer$
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(
at java.util.concurrent.ThreadPoolExecutor$

So bottom first three lines pretty easy, a thread is ran with the thread pool executor. As indicated by the code snipet below, that a worker process having trouble in processing a request.
processor = processorFactory_.getProcessor(client_);
inputTransport = inputTransportFactory_.getTransport(client_);
outputTransport = outputTransportFactory_.getTransport(client_);
inputProtocol = inputProtocolFactory_.getProtocol(inputTransport);
outputProtocol = outputProtocolFactory_.getProtocol(outputTransport);
// we check stopped_ first to make sure we're not supposed to be shutting
// down. this is necessary for graceful shutdown. (but not sufficient,
// since process() can take arbitrarily long waiting for client input.
// See comments at the end of serve().)
while (!stopped_ && processor.process(inputProtocol, outputProtocol))
inputProtocol = inputProtocolFactory_.getProtocol(inputTransport);
outputProtocol = outputProtocolFactory_.getProtocol(outputTransport);

Skipping a few low level byte stream processing, we arrived at the actual class which actually implement the method get_slice. Read code snippet below.
    public List<ColumnOrSuperColumn> get_slice(ByteBuffer key, ColumnParent column_parent, SlicePredicate predicate, ConsistencyLevel consistency_level)
throws InvalidRequestException, UnavailableException, TimedOutException

state().hasColumnFamilyAccess(column_parent.column_family, Permission.READ);
return multigetSliceInternal(state().getKeyspace(), Collections.singletonList(key), column_parent, predicate, consistency_level).get(key);

so we see another method is called, multigetSliceInternal. Read code snippet below where a few validations on the data.
    private Map<ByteBuffer, List<ColumnOrSuperColumn>> multigetSliceInternal(String keyspace, List<ByteBuffer> keys, ColumnParent column_parent, SlicePredicate predicate, ConsistencyLevel consistency_level)
throws InvalidRequestException, UnavailableException, TimedOutException
CFMetaData metadata = ThriftValidation.validateColumnFamily(keyspace, column_parent.column_family);
ThriftValidation.validateColumnParent(metadata, column_parent);
ThriftValidation.validatePredicate(metadata, column_parent, predicate);
ThriftValidation.validateConsistencyLevel(keyspace, consistency_level);

List<ReadCommand> commands = new ArrayList<ReadCommand>();
if (predicate.column_names != null)
for (ByteBuffer key: keys)
ThriftValidation.validateKey(metadata, key);
commands.add(new SliceByNamesReadCommand(keyspace, key, column_parent, predicate.column_names));
SliceRange range = predicate.slice_range;
for (ByteBuffer key: keys)
ThriftValidation.validateKey(metadata, key);
commands.add(new SliceFromReadCommand(keyspace, key, column_parent, range.start, range.finish, range.reversed, range.count));

return getSlice(commands, consistency_level);

then method getSlice is called,  and method readColumnFamily() is also called. As shown below, the code snippet
  protected Map<DecoratedKey, ColumnFamily> readColumnFamily(List<ReadCommand> commands, ConsistencyLevel consistency_level)
throws InvalidRequestException, UnavailableException, TimedOutException
// TODO - Support multiple column families per row, right now row only contains 1 column family
Map<DecoratedKey, ColumnFamily> columnFamilyKeyMap = new HashMap<DecoratedKey, ColumnFamily>();

if (consistency_level == ConsistencyLevel.ANY)
throw new InvalidRequestException("Consistency level any may not be applied to read operations");

List<Row> rows;
rows =, consistency_level);
catch (TimeoutException e)
logger.debug("... timed out");
throw new TimedOutException();
catch (IOException e)
throw new RuntimeException(e);

for (Row row: rows)
return columnFamilyKeyMap;

another class is called, StorageProxy to read the row in concern and the read method code snippet below.
* Performs the actual reading of a row out of the StorageService, fetching
* a specific set of column names from a given column family.
public static List<Row> read(List<ReadCommand> commands, ConsistencyLevel consistency_level)
throws IOException, UnavailableException, TimeoutException, InvalidRequestException
if (StorageService.instance.isBootstrapMode())
throw new UnavailableException();
long startTime = System.nanoTime();
List<Row> rows;
rows = fetchRows(commands, consistency_level);
readStats.addNano(System.nanoTime() - startTime);
return rows;

the exception lead this investigation to fetching the row and within the same class, for method fetchRows, code snippet below.
* This function executes local and remote reads, and blocks for the results:
* 1. Get the replica locations, sorted by response time according to the snitch
* 2. Send a data request to the closest replica, and digest requests to either
* a) all the replicas, if read repair is enabled
* b) the closest R-1 replicas, where R is the number required to satisfy the ConsistencyLevel
* 3. Wait for a response from R replicas
* 4. If the digests (if any) match the data return the data
* 5. else carry out read repair by getting data from all the nodes.
private static List<Row> fetchRows(List<ReadCommand> initialCommands, ConsistencyLevel consistency_level) throws IOException, UnavailableException, TimeoutException
List<Row> rows = new ArrayList<Row>(initialCommands.size());
List<ReadCommand> commandsToRetry = Collections.emptyList();

List<ReadCommand> commands = commandsToRetry.isEmpty() ? initialCommands : commandsToRetry;
ReadCallback<Row>[] readCallbacks = new ReadCallback[commands.size()];

if (!commandsToRetry.isEmpty())
logger.debug("Retrying {} commands", commandsToRetry.size());

// send out read requests
for (int i = 0; i < commands.size(); i++)
ReadCommand command = commands.get(i);
assert !command.isDigestQuery();
logger.debug("Command/ConsistencyLevel is {}/{}", command, consistency_level);

List<InetAddress> endpoints = StorageService.instance.getLiveNaturalEndpoints(command.table,
DatabaseDescriptor.getEndpointSnitch().sortByProximity(FBUtilities.getBroadcastAddress(), endpoints);

RowDigestResolver resolver = new RowDigestResolver(command.table, command.key);
ReadCallback<Row> handler = getReadCallback(resolver, command, consistency_level, endpoints);
assert !handler.endpoints.isEmpty();
readCallbacks[i] = handler;

// The data-request message is sent to dataPoint, the node that will actually get the data for us
InetAddress dataPoint = handler.endpoints.get(0);
if (dataPoint.equals(FBUtilities.getBroadcastAddress()) && OPTIMIZE_LOCAL_REQUESTS)
logger.debug("reading data locally");
StageManager.getStage(Stage.READ).execute(new LocalReadRunnable(command, handler));
logger.debug("reading data from {}", dataPoint);
MessagingService.instance().sendRR(command, dataPoint, handler);

if (handler.endpoints.size() == 1)

// send the other endpoints a digest request
ReadCommand digestCommand = command.copy();
MessageProducer producer = null;
for (InetAddress digestPoint : handler.endpoints.subList(1, handler.endpoints.size()))
if (digestPoint.equals(FBUtilities.getBroadcastAddress()))
logger.debug("reading digest locally");
StageManager.getStage(Stage.READ).execute(new LocalReadRunnable(digestCommand, handler));
logger.debug("reading digest from {}", digestPoint);
// (We lazy-construct the digest Message object since it may not be necessary if we
// are doing a local digest read, or no digest reads at all.)
if (producer == null)
producer = new CachingMessageProducer(digestCommand);
MessagingService.instance().sendRR(producer, digestPoint, handler);

// read results and make a second pass for any digest mismatches
List<ReadCommand> repairCommands = null;
List<RepairCallback> repairResponseHandlers = null;
for (int i = 0; i < commands.size(); i++)
ReadCallback<Row> handler = readCallbacks[i];
ReadCommand command = commands.get(i);
long startTime2 = System.currentTimeMillis();
Row row = handler.get();
if (row != null)

if (logger.isDebugEnabled())
logger.debug("Read: " + (System.currentTimeMillis() - startTime2) + " ms.");
catch (TimeoutException ex)
if (logger.isDebugEnabled())
logger.debug("Read timeout: {}", ex.toString());
throw ex;
catch (DigestMismatchException ex)
if (logger.isDebugEnabled())
logger.debug("Digest mismatch: {}", ex.toString());
RowRepairResolver resolver = new RowRepairResolver(command.table, command.key);
RepairCallback repairHandler = new RepairCallback(resolver, handler.endpoints);

if (repairCommands == null)
repairCommands = new ArrayList<ReadCommand>();
repairResponseHandlers = new ArrayList<RepairCallback>();

MessageProducer producer = new CachingMessageProducer(command);
for (InetAddress endpoint : handler.endpoints)
MessagingService.instance().sendRR(producer, endpoint, repairHandler);

if (commandsToRetry != Collections.EMPTY_LIST)

// read the results for the digest mismatch retries
if (repairResponseHandlers != null)
for (int i = 0; i < repairCommands.size(); i++)
ReadCommand command = repairCommands.get(i);
RepairCallback handler = repairResponseHandlers.get(i);
// wait for the repair writes to be acknowledged, to minimize impact on any replica that's
// behind on writes in case the out-of-sync row is read multiple times in quick succession
FBUtilities.waitOnFutures(handler.resolver.repairResults, DatabaseDescriptor.getRpcTimeout());

Row row;
row = handler.get();
catch (DigestMismatchException e)
throw new AssertionError(e); // full data requested from each node here, no digests should be sent

ReadCommand retryCommand = command.maybeGenerateRetryCommand(handler, row);
if (retryCommand != null)
logger.debug("issuing retry for read command");
if (commandsToRetry == Collections.EMPTY_LIST)
commandsToRetry = new ArrayList<ReadCommand>();

if (row != null)
} while (!commandsToRetry.isEmpty());

return rows;

As this point of investigation, this method, fetchRows documentation is pretty useful for us.
* This function executes local and remote reads, and blocks for the results:
* 1. Get the replica locations, sorted by response time according to the snitch
* 2. Send a data request to the closest replica, and digest requests to either
* a) all the replicas, if read repair is enabled
* b) the closest R-1 replicas, where R is the number required to satisfy the ConsistencyLevel

we see this method actually execute on local and remote node, and during getting the node who is responsible to keep the row, problem occur. Let's read on the method getLiveNaturalEndpoints() and as shown below.
* This method attempts to return N endpoints that are responsible for storing the
* specified key i.e for replication.
* @param key - key for which we need to find the endpoint return value -
* the endpoint responsible for this key
public List<InetAddress> getLiveNaturalEndpoints(String table, ByteBuffer key)
return getLiveNaturalEndpoints(table, partitioner.getToken(key));

public List<InetAddress> getLiveNaturalEndpoints(String table, Token token)
List<InetAddress> liveEps = new ArrayList<InetAddress>();
List<InetAddress> endpoints =;

for (InetAddress endpoint : endpoints)
if (FailureDetector.instance.isAlive(endpoint))

return liveEps;

a little upper in the stack trace, abstract class AbstractReplicationStrategy
* get the (possibly cached) endpoints that should store the given Token
* Note that while the endpoints are conceptually a Set (no duplicates will be included),
* we return a List to avoid an extra allocation when sorting by proximity later
* @param searchToken the token the natural endpoints are requested for
* @return a copy of the natural endpoints for the given token
public ArrayList<InetAddress> getNaturalEndpoints(Token searchToken)
Token keyToken = TokenMetadata.firstToken(tokenMetadata.sortedTokens(), searchToken);
ArrayList<InetAddress> endpoints = getCachedEndpoints(keyToken);
if (endpoints == null)
TokenMetadata tokenMetadataClone = tokenMetadata.cloneOnlyTokenMap();
keyToken = TokenMetadata.firstToken(tokenMetadataClone.sortedTokens(), searchToken);
endpoints = new ArrayList<InetAddress>(calculateNaturalEndpoints(searchToken, tokenMetadataClone));
cacheEndpoint(keyToken, endpoints);

return new ArrayList<InetAddress>(endpoints);

somehow the ring size is equal to 0 or less than 0. class and code snippet where the assertion thrown,
    public static int firstTokenIndex(final ArrayList ring, Token start, boolean insertMin)
assert ring.size() > 0;
// insert the minimum token (at index == -1) if we were asked to include it and it isn't a member of the ring
int i = Collections.binarySearch(ring, start);
if (i < 0)
i = (i + 1) * (-1);
if (i >= ring.size())
i = insertMin ? -1 : 0;
return i;

public static Token firstToken(final ArrayList<Token> ring, Token start)
return ring.get(firstTokenIndex(ring, start, false));

So something went during during reading a row's column and somehow the natural endpoint is either 0 or empty. My guess is that, it could be gossip is disable so the ring metadata is empty. The solution is to enable the gossip and then restart cassandra instance.

If you think this analysis is not accurate or want to provide more information, please do so by commenting below. Thank you.