Friday, November 6, 2015

Learning basic of apache karaf

Recently container is such a hot topic especially from docker and today, we will look into another container from apache. Today, we will take a look into Apache Karaf.

What is Apache Karaf?

Apache Karaf is a small OSGi based runtime which provides a lightweight container onto which various components and applications can be deployed.

If you have no idea what does that means, maybe a simple quick how to give some idea. First, let's download a copy of apache karaf and you can do that just here. At the time of this learning experience, I'm using Apache Karaf version 4.0.1. Then extract to a path so that will be karaf home directory.

 user@localhost:~/Desktop$ ll apache-karaf-4.0.1.tar.gz   
 -rw-r----- 1 user user 16M Oct 5 22:35 apache-karaf-4.0.1.tar.gz  
 user@localhost:~/Desktop$ tar -zxf apache-karaf-4.0.1.tar.gz   
 user@localhost:~/Desktop$ cd apache-karaf-4.0.1  
 apache-karaf-4.0.1/    apache-karaf-4.0.1.tar.gz   
 user@localhost:~/Desktop$ cd apache-karaf-4.0.1  
 user@localhost:~/Desktop/apache-karaf-4.0.1$ ls  
 bin data demos deploy etc lib LICENSE NOTICE README RELEASE-NOTES system  

So apache is about 16MB compressed and contain a few directories to work with.

The directory layout of a Karaf installation is as follows:
/bin: control scripts to start, stop, login.
/demos: contains some simple Karaf samples
/etc: configuration files
/data: working directory
/cache: OSGi framework bundle cache
/generated-bundles: temporary folder used by the deployers
/log: log files
/deploy: hot deploy directory
/instances: directory containing instances
/lib: contains libraries
/lib/boot: contains the system libraries used at Karaf bootstrap
/lib/endorsed: directory for endorsed libraries
/lib/ext: directory for JRE extensions
/system: OSGi bundles repository, laid out as a Maven 2 repository

Let's launch karaf, see screenshot below in the terminal. Let's add apache camel repository into apache karaf and then install on it. We will therefore using this as a sample for this learning experience.



 karaf@root()> feature:repo-add camel 2.15.3  
 Adding feature url mvn:org.apache.camel.karaf/apache-camel/2.15.3/xml/features  
 karaf@root()> feature:info camel  
 Feature camel 2.15.3  
 Feature has no configuration  
 Feature has no configuration files  
 Feature depends on:  
  camel-core 2.15.3  
  camel-spring 2.15.3  
  camel-blueprint 2.15.3  
 Feature has no bundles.  
 Feature has no conditionals.  
 karaf@root()> feature:install camel-spring  
 karaf@root()> bundle:install -s mvn:org.apache.camel/camel-example-osgi/2.15.3  
 Bundle ID: 82  
 karaf@root()> log:display  
 2015-10-05 22:41:37,872 | INFO | pool-17-thread-1 | core               | 17 - org.apache.aries.jmx.core - 1.1.3 | Registering org.osgi.jmx.framework.BundleStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@dd1e765 with name osgi.core:type=bundleState,version=1.7,framework=org.apache.felix.framework,uuid=e7d79bed-237a-4c4d-b912-920b57fef63b  
 2015-10-05 22:41:37,876 | INFO | pool-17-thread-1 | core               | 17 - org.apache.aries.jmx.core - 1.1.3 | Registering org.osgi.jmx.service.cm.ConfigurationAdminMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@dd1e765 with name osgi.compendium:service=cm,version=1.3,framework=org.apache.felix.framework,uuid=e7d79bed-237a-4c4d-b912-920b57fef63b  
 2015-10-05 22:41:37,876 | INFO | pool-17-thread-1 | core               | 17 - org.apache.aries.jmx.core - 1.1.3 | Registering org.osgi.jmx.framework.wiring.BundleWiringStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@dd1e765 with name osgi.core:type=wiringState,version=1.1,framework=org.apache.felix.framework,uuid=e7d79bed-237a-4c4d-b912-920b57fef63b  
 2015-10-05 22:41:37,877 | INFO | pool-17-thread-1 | core               | 17 - org.apache.aries.jmx.core - 1.1.3 | Registering org.osgi.jmx.framework.FrameworkMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@dd1e765 with name osgi.core:type=framework,version=1.7,framework=org.apache.felix.framework,uuid=e7d79bed-237a-4c4d-b912-920b57fef63b  
 2015-10-05 22:41:37,878 | INFO | pool-17-thread-1 | core               | 17 - org.apache.aries.jmx.core - 1.1.3 | Registering org.osgi.jmx.framework.PackageStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@dd1e765 with name osgi.core:type=packageState,version=1.5,framework=org.apache.felix.framework,uuid=e7d79bed-237a-4c4d-b912-920b57fef63b  
 2015-10-05 22:41:37,878 | INFO | pool-17-thread-1 | core               | 17 - org.apache.aries.jmx.core - 1.1.3 | Registering org.osgi.jmx.framework.ServiceStateMBean to MBeanServer com.sun.jmx.mbeanserver.JmxMBeanServer@dd1e765 with name osgi.core:type=serviceState,version=1.7,framework=org.apache.felix.framework,uuid=e7d79bed-237a-4c4d-b912-920b57fef63b  
 2015-10-05 22:41:38,145 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Registering commands for bundle org.apache.karaf.bundle.core/4.0.1  
 2015-10-05 22:41:38,168 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Registering commands for bundle org.apache.karaf.config.core/4.0.1  
 2015-10-05 22:41:38,178 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Registering commands for bundle org.apache.karaf.deployer.kar/4.0.1  
 2015-10-05 22:41:38,180 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Registering commands for bundle org.apache.karaf.diagnostic.core/4.0.1  
 2015-10-05 22:41:38,204 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Registering commands for bundle org.apache.karaf.features.command/4.0.1  
 2015-10-05 22:41:38,230 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Registering commands for bundle org.apache.karaf.instance.core/4.0.1  
 2015-10-05 22:41:38,256 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Registering commands for bundle org.apache.karaf.jaas.command/4.0.1  
 2015-10-05 22:41:38,259 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Updating commands for bundle org.apache.karaf.jaas.command/4.0.1  
 2015-10-05 22:41:38,260 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Updating commands for bundle org.apache.karaf.jaas.command/4.0.1  
 2015-10-05 22:41:38,266 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Registering commands for bundle org.apache.karaf.kar.core/4.0.1  
 2015-10-05 22:41:38,277 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Registering commands for bundle org.apache.karaf.log.core/4.0.1  
 2015-10-05 22:41:38,281 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Registering commands for bundle org.apache.karaf.package.core/4.0.1  
 2015-10-05 22:41:38,285 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Registering commands for bundle org.apache.karaf.service.core/4.0.1  
 2015-10-05 22:41:38,327 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Command registration delayed for bundle org.apache.karaf.shell.commands/4.0.1. Missing dependencies: [org.jledit.EditorFactory]  
 2015-10-05 22:41:38,581 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Command registration delayed for bundle org.apache.karaf.shell.ssh/4.0.1. Missing dependencies: [org.apache.sshd.SshServer]  
 2015-10-05 22:41:38,593 | INFO | pool-23-thread-1 | SecurityUtils          | 47 - org.apache.sshd.core - 0.14.0 | BouncyCastle not registered, using the default JCE provider  
 2015-10-05 22:41:38,624 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Registering commands for bundle org.apache.karaf.system.core/4.0.1  
 2015-10-05 22:41:38,631 | INFO | FelixStartLevel | CommandExtension         | 43 - org.apache.karaf.shell.core - 4.0.1 | Registering commands for bundle org.apache.karaf.shell.commands/4.0.1  
 2015-10-05 23:18:09,193 | INFO | nsole user karaf | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 | Adding features: camel-spring/[2.15.3,2.15.3]  
 2015-10-05 23:19:30,370 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 | Changes to perform:  
 2015-10-05 23:19:30,371 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  Region: root  
 2015-10-05 23:19:30,371 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |   Bundles to install:  
 2015-10-05 23:19:30,371 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.camel/camel-catalog/2.15.3  
 2015-10-05 23:19:30,371 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.camel/camel-commands-core/2.15.3  
 2015-10-05 23:19:30,372 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.camel/camel-core/2.15.3  
 2015-10-05 23:19:30,372 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.camel/camel-spring/2.15.3  
 2015-10-05 23:19:30,372 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.camel.karaf/camel-karaf-commands/2.15.3  
 2015-10-05 23:19:30,373 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.geronimo.specs/geronimo-jta_1.1_spec/1.1.1  
 2015-10-05 23:19:30,373 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.karaf.bundle/org.apache.karaf.bundle.springstate/4.0.1  
 2015-10-05 23:19:30,373 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.karaf.deployer/org.apache.karaf.deployer.spring/4.0.1  
 2015-10-05 23:19:30,373 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.aopalliance/1.0_6  
 2015-10-05 23:19:30,374 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.cglib/3.0_1  
 2015-10-05 23:19:30,374 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.jaxb-impl/2.2.6_1  
 2015-10-05 23:19:30,374 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-aop/3.2.14.RELEASE_1  
 2015-10-05 23:19:30,374 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-beans/3.2.14.RELEASE_1  
 2015-10-05 23:19:30,375 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-context/3.2.14.RELEASE_1  
 2015-10-05 23:19:30,375 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-context-support/3.2.14.RELEASE_1  
 2015-10-05 23:19:30,375 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-core/3.2.14.RELEASE_1  
 2015-10-05 23:19:30,375 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-expression/3.2.14.RELEASE_1  
 2015-10-05 23:19:30,376 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-tx/3.2.14.RELEASE_1  
 2015-10-05 23:19:30,376 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.springframework/spring-aop/3.1.4.RELEASE  
 2015-10-05 23:19:30,376 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.springframework/spring-asm/3.1.4.RELEASE  
 2015-10-05 23:19:30,377 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.springframework/spring-beans/3.1.4.RELEASE  
 2015-10-05 23:19:30,377 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.springframework/spring-context/3.1.4.RELEASE  
 2015-10-05 23:19:30,377 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.springframework/spring-context-support/3.1.4.RELEASE  
 2015-10-05 23:19:30,377 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.springframework/spring-core/3.1.4.RELEASE  
 2015-10-05 23:19:30,378 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.springframework/spring-expression/3.1.4.RELEASE  
 2015-10-05 23:19:30,378 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.springframework.osgi/spring-osgi-core/1.2.1  
 2015-10-05 23:19:30,378 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.springframework.osgi/spring-osgi-extender/1.2.1  
 2015-10-05 23:19:30,378 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.springframework.osgi/spring-osgi-annotation/1.2.1  
 2015-10-05 23:19:30,379 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.springframework.osgi/spring-osgi-io/1.2.1  
 2015-10-05 23:19:30,379 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.codehaus.woodstox/stax2-api/3.1.4  
 2015-10-05 23:19:30,380 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |    mvn:org.codehaus.woodstox/woodstox-core-asl/4.4.1  
 2015-10-05 23:19:30,383 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 | Installing bundles:  
 2015-10-05 23:19:30,383 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.camel/camel-catalog/2.15.3  
 2015-10-05 23:19:30,393 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.camel/camel-commands-core/2.15.3  
 2015-10-05 23:19:30,399 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.camel/camel-core/2.15.3  
 2015-10-05 23:19:30,436 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.camel/camel-spring/2.15.3  
 2015-10-05 23:19:30,447 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.camel.karaf/camel-karaf-commands/2.15.3  
 2015-10-05 23:19:30,451 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.geronimo.specs/geronimo-jta_1.1_spec/1.1.1  
 2015-10-05 23:19:30,453 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.karaf.bundle/org.apache.karaf.bundle.springstate/4.0.1  
 2015-10-05 23:19:30,458 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.karaf.deployer/org.apache.karaf.deployer.spring/4.0.1  
 2015-10-05 23:19:30,462 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.aopalliance/1.0_6  
 2015-10-05 23:19:30,465 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.cglib/3.0_1  
 2015-10-05 23:19:30,471 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.jaxb-impl/2.2.6_1  
 2015-10-05 23:19:30,485 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-aop/3.2.14.RELEASE_1  
 2015-10-05 23:19:30,498 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-beans/3.2.14.RELEASE_1  
 2015-10-05 23:19:30,510 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-context/3.2.14.RELEASE_1  
 2015-10-05 23:19:30,536 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-context-support/3.2.14.RELEASE_1  
 2015-10-05 23:19:30,545 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-core/3.2.14.RELEASE_1  
 2015-10-05 23:19:30,560 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-expression/3.2.14.RELEASE_1  
 2015-10-05 23:19:30,567 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.apache.servicemix.bundles/org.apache.servicemix.bundles.spring-tx/3.2.14.RELEASE_1  
 2015-10-05 23:19:30,576 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.springframework/spring-aop/3.1.4.RELEASE  
 2015-10-05 23:19:30,585 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.springframework/spring-asm/3.1.4.RELEASE  
 2015-10-05 23:19:30,590 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.springframework/spring-beans/3.1.4.RELEASE  
 2015-10-05 23:19:30,606 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.springframework/spring-context/3.1.4.RELEASE  
 2015-10-05 23:19:30,636 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.springframework/spring-context-support/3.1.4.RELEASE  
 2015-10-05 23:19:30,655 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.springframework/spring-core/3.1.4.RELEASE  
 2015-10-05 23:19:30,667 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.springframework/spring-expression/3.1.4.RELEASE  
 2015-10-05 23:19:30,673 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.springframework.osgi/spring-osgi-core/1.2.1  
 2015-10-05 23:19:30,684 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.springframework.osgi/spring-osgi-extender/1.2.1  
 2015-10-05 23:19:30,691 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.springframework.osgi/spring-osgi-annotation/1.2.1  
 2015-10-05 23:19:30,696 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.springframework.osgi/spring-osgi-io/1.2.1  
 2015-10-05 23:19:30,702 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.codehaus.woodstox/stax2-api/3.1.4  
 2015-10-05 23:19:30,708 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  mvn:org.codehaus.woodstox/woodstox-core-asl/4.4.1  
 2015-10-05 23:19:31,219 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 | Starting bundles:  
 2015-10-05 23:19:31,337 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.servicemix.bundles.aopalliance/1.0.0.6  
 2015-10-05 23:19:31,339 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.springframework.asm/3.1.4.RELEASE  
 2015-10-05 23:19:31,341 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.springframework.core/3.1.4.RELEASE  
 2015-10-05 23:19:31,343 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.springframework.beans/3.1.4.RELEASE  
 2015-10-05 23:19:31,344 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.springframework.aop/3.1.4.RELEASE  
 2015-10-05 23:19:31,346 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.springframework.expression/3.1.4.RELEASE  
 2015-10-05 23:19:31,347 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.springframework.context/3.1.4.RELEASE  
 2015-10-05 23:19:31,349 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.servicemix.bundles.cglib/3.0.0.1  
 2015-10-05 23:19:31,352 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.servicemix.bundles.spring-core/3.2.14.RELEASE_1  
 2015-10-05 23:19:31,354 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.servicemix.bundles.spring-beans/3.2.14.RELEASE_1  
 2015-10-05 23:19:31,356 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.servicemix.bundles.spring-aop/3.2.14.RELEASE_1  
 2015-10-05 23:19:31,357 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.springframework.osgi.io/1.2.1  
 2015-10-05 23:19:31,359 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.servicemix.bundles.spring-expression/3.2.14.RELEASE_1  
 2015-10-05 23:19:31,360 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.servicemix.bundles.spring-context/3.2.14.RELEASE_1  
 2015-10-05 23:19:31,362 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.springframework.osgi.core/1.2.1  
 2015-10-05 23:19:31,364 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.springframework.osgi.extensions.annotations/1.2.1  
 2015-10-05 23:19:31,366 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.springframework.osgi.extender/1.2.1  
 2015-10-05 23:19:31,393 | INFO | pool-25-thread-1 | ContextLoaderListener      | 77 - org.springframework.osgi.extender - 1.2.1 | Starting [org.springframework.osgi.extender] bundle v.[1.2.1]  
 2015-10-05 23:19:31,727 | INFO | pool-25-thread-1 | ExtenderConfiguration      | 77 - org.springframework.osgi.extender - 1.2.1 | No custom extender configuration detected; using defaults...  
 2015-10-05 23:19:31,740 | INFO | pool-25-thread-1 | TimerTaskExecutor        | 64 - org.apache.servicemix.bundles.spring-context - 3.2.14.RELEASE_1 | Initializing Timer  
 2015-10-05 23:19:31,839 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.springframework.context.support/3.1.4.RELEASE  
 2015-10-05 23:19:31,841 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.servicemix.bundles.spring-tx/3.2.14.RELEASE_1  
 2015-10-05 23:19:31,842 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.karaf.deployer.spring/4.0.1  
 2015-10-05 23:19:31,854 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.servicemix.bundles.spring-context-support/3.2.14.RELEASE_1  
 2015-10-05 23:19:31,856 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.karaf.bundle.springstate/4.0.1  
 2015-10-05 23:19:31,916 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.camel.camel-catalog/2.15.3  
 2015-10-05 23:19:31,922 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.camel.camel-spring/2.15.3  
 2015-10-05 23:19:31,939 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.camel.camel-core/2.15.3  
 2015-10-05 23:19:31,943 | INFO | pool-25-thread-1 | Activator            | 53 - org.apache.camel.camel-core - 2.15.3 | Camel activator starting  
 2015-10-05 23:19:31,982 | INFO | pool-25-thread-1 | Activator            | 53 - org.apache.camel.camel-core - 2.15.3 | Camel activator started  
 2015-10-05 23:19:32,007 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.camel.camel-commands-core/2.15.3  
 2015-10-05 23:19:32,012 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  stax2-api/3.1.4  
 2015-10-05 23:19:32,014 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  woodstox-core-asl/4.4.1  
 2015-10-05 23:19:32,051 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.geronimo.specs.geronimo-jta_1.1_spec/1.1.1  
 2015-10-05 23:19:32,054 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.servicemix.bundles.jaxb-impl/2.2.6.1  
 2015-10-05 23:19:32,056 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 |  org.apache.camel.karaf.camel-karaf-commands/2.15.3  
 2015-10-05 23:19:32,315 | INFO | pool-25-thread-1 | FeaturesServiceImpl       | 8 - org.apache.karaf.features.core - 4.0.1 | Done.  
 2015-10-05 23:20:53,179 | INFO | nsole user karaf | ultOsgiApplicationContextCreator | 77 - org.springframework.osgi.extender - 1.2.1 | Discovered configurations {osgibundle:/META-INF/spring/*.xml} in bundle [camel-example-osgi (camel-example-osgi)]  
 2015-10-05 23:20:53,219 | INFO | ExtenderThread-1 | OsgiBundleXmlApplicationContext | 64 - org.apache.servicemix.bundles.spring-context - 3.2.14.RELEASE_1 | Refreshing OsgiBundleXmlApplicationContext(bundle=camel-example-osgi, config=osgibundle:/META-INF/spring/*.xml): startup date [Mon Oct 05 23:20:53 MYT 2015]; root of context hierarchy  
 2015-10-05 23:20:53,262 | INFO | ExtenderThread-1 | OsgiBundleXmlApplicationContext | 64 - org.apache.servicemix.bundles.spring-context - 3.2.14.RELEASE_1 | Application Context service already unpublished  
 2015-10-05 23:20:53,325 | INFO | ExtenderThread-1 | XmlBeanDefinitionReader     | 63 - org.apache.servicemix.bundles.spring-beans - 3.2.14.RELEASE_1 | Loading XML bean definitions from URL [bundle://82.0:0/META-INF/spring/camelContext.xml]  
 2015-10-05 23:20:53,611 | INFO | ExtenderThread-1 | CamelNamespaceHandler      | 54 - org.apache.camel.camel-spring - 2.15.3 | OSGi environment detected.  
 2015-10-05 23:20:54,970 | INFO | ExtenderThread-1 | WaiterApplicationContextExecutor | 77 - org.springframework.osgi.extender - 1.2.1 | No outstanding OSGi service dependencies, completing initialization for OsgiBundleXmlApplicationContext(bundle=camel-example-osgi, config=osgibundle:/META-INF/spring/*.xml)  
 2015-10-05 23:20:55,034 | INFO | ExtenderThread-2 | DefaultListableBeanFactory    | 63 - org.apache.servicemix.bundles.spring-beans - 3.2.14.RELEASE_1 | Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@69f2fae4: defining beans [template,consumerTemplate,camel-1:beanPostProcessor,camel-1,myTransform]; root of factory hierarchy  
 2015-10-05 23:20:55,302 | INFO | ExtenderThread-2 | OsgiSpringCamelContext      | 53 - org.apache.camel.camel-core - 2.15.3 | Apache Camel 2.15.3 (CamelContext: camel-1) is starting  
 2015-10-05 23:20:55,304 | INFO | ExtenderThread-2 | ManagedManagementStrategy    | 53 - org.apache.camel.camel-core - 2.15.3 | JMX is enabled  
 2015-10-05 23:20:55,651 | INFO | ExtenderThread-2 | OsgiSpringCamelContext      | 53 - org.apache.camel.camel-core - 2.15.3 | AllowUseOriginalMessage is enabled. If access to the original message is not needed, then its recommended to turn this option off as it may improve performance.  
 2015-10-05 23:20:55,651 | INFO | ExtenderThread-2 | OsgiSpringCamelContext      | 53 - org.apache.camel.camel-core - 2.15.3 | StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html  
 2015-10-05 23:20:55,742 | INFO | ExtenderThread-2 | OsgiSpringCamelContext      | 53 - org.apache.camel.camel-core - 2.15.3 | Route: route1 started and consuming from: Endpoint[timer://myTimer?fixedRate=true&period=2000]  
 2015-10-05 23:20:55,745 | INFO | ExtenderThread-2 | OsgiSpringCamelContext      | 53 - org.apache.camel.camel-core - 2.15.3 | Total 1 routes, of which 1 is started.  
 2015-10-05 23:20:55,752 | INFO | ExtenderThread-2 | OsgiSpringCamelContext      | 53 - org.apache.camel.camel-core - 2.15.3 | Apache Camel 2.15.3 (CamelContext: camel-1) started in 0.444 seconds  
 2015-10-05 23:20:55,759 | INFO | ExtenderThread-2 | OsgiBundleXmlApplicationContext | 64 - org.apache.servicemix.bundles.spring-context - 3.2.14.RELEASE_1 | Publishing application context as OSGi service with properties {org.springframework.context.service.name=camel-example-osgi, Bundle-SymbolicName=camel-example-osgi, Bundle-Version=2.15.3}  
 2015-10-05 23:20:55,775 | INFO | ExtenderThread-2 | ContextLoaderListener      | 77 - org.springframework.osgi.extender - 1.2.1 | Application context successfully refreshed (OsgiBundleXmlApplicationContext(bundle=camel-example-osgi, config=osgibundle:/META-INF/spring/*.xml))  
 2015-10-05 23:20:56,759 | INFO | timer://myTimer | MyTransform           | 82 - camel-example-osgi - 2.15.3 | >>>> SpringDSL set body: Mon Oct 05 23:20:56 MYT 2015  
 2015-10-05 23:20:56,766 | INFO | timer://myTimer | ExampleRouter          | 53 - org.apache.camel.camel-core - 2.15.3 | Exchange[ExchangePattern: InOnly, BodyType: String, Body: SpringDSL set body: Mon Oct 05 23:20:56 MYT 2015]  
 2015-10-05 23:20:58,746 | INFO | timer://myTimer | MyTransform           | 82 - camel-example-osgi - 2.15.3 | >>>> SpringDSL set body: Mon Oct 05 23:20:58 MYT 2015  
 2015-10-05 23:20:58,747 | INFO | timer://myTimer | ExampleRouter          | 53 - org.apache.camel.camel-core - 2.15.3 | Exchange[ExchangePattern: InOnly, BodyType: String, Body: SpringDSL set body: Mon Oct 05 23:20:58 MYT 2015]  
 2015-10-05 23:21:00,746 | INFO | timer://myTimer | MyTransform           | 82 - camel-example-osgi - 2.15.3 | >>>> SpringDSL set body: Mon Oct 05 23:21:00 MYT 2015  
 2015-10-05 23:21:00,747 | INFO | timer://myTimer | ExampleRouter          | 53 - org.apache.camel.camel-core - 2.15.3 | Exchange[ExchangePattern: InOnly, BodyType: String, Body: SpringDSL set body: Mon Oct 05 23:21:00 MYT 2015]  
 2015-10-05 23:21:02,746 | INFO | timer://myTimer | MyTransform           | 82 - camel-example-osgi - 2.15.3 | >>>> SpringDSL set body: Mon Oct 05 23:21:02 MYT 2015  
 2015-10-05 23:21:02,747 | INFO | timer://myTimer | ExampleRouter          | 53 - org.apache.camel.camel-core - 2.15.3 | Exchange[ExchangePattern: InOnly, BodyType: String, Body: SpringDSL set body: Mon Oct 05 23:21:02 MYT 2015]  
 2015-10-05 23:21:04,745 | INFO | timer://myTimer | MyTransform           | 82 - camel-example-osgi - 2.15.3 | >>>> SpringDSL set body: Mon Oct 05 23:21:04 MYT 2015  
 2015-10-05 23:21:04,746 | INFO | timer://myTimer | ExampleRouter          | 53 - org.apache.camel.camel-core - 2.15.3 | Exchange[ExchangePattern: InOnly, BodyType: String, Body: SpringDSL set body: Mon Oct 05 23:21:04 MYT 2015]  
 2015-10-05 23:21:06,745 | INFO | timer://myTimer | MyTransform           | 82 - camel-example-osgi - 2.15.3 | >>>> SpringDSL set body: Mon Oct 05 23:21:06 MYT 2015  
 2015-10-05 23:21:06,746 | INFO | timer://myTimer | ExampleRouter          | 53 - org.apache.camel.camel-core - 2.15.3 | Exchange[ExchangePattern: InOnly, BodyType: String, Body: SpringDSL set body: Mon Oct 05 23:21:06 MYT 2015]  
   
 karaf@root()>  

and if you list again on the bundle, there are multiple instances running.

 karaf@root()> bundle:list  
 START LEVEL 100 , List Threshold: 50  
 ID | State | Lvl | Version | Name  
 -----------------------------------------------------------------------  
 51 | Active | 80 | 2.15.3 | camel-catalog  
 52 | Active | 80 | 2.15.3 | camel-commands-core  
 53 | Active | 80 | 2.15.3 | camel-core  
 54 | Active | 80 | 2.15.3 | camel-spring  
 55 | Active | 80 | 2.15.3 | camel-karaf-commands  
 56 | Active | 80 | 1.1.1  | geronimo-jta_1.1_spec  
 61 | Active | 80 | 2.2.6.1 | Apache ServiceMix :: Bundles :: jaxb-impl  
 80 | Active | 80 | 3.1.4  | Stax2 API  
 81 | Active | 80 | 4.4.1  | Woodstox XML-processor  
 82 | Active | 80 | 2.15.3 | camel-example-osgi  
 karaf@root()>   

Just like docker contaianer, you can stop and uninstall the bundle and then exit apache karaf.

 karaf@root()> bundle:stop camel-example-osgi  
 karaf@root()> bundle:uninstall camel-example-osgi  
 karaf@root()> bundle:list  
 START LEVEL 100 , List Threshold: 50  
 ID | State | Lvl | Version | Name  
 -----------------------------------------------------------------------  
 51 | Active | 80 | 2.15.3 | camel-catalog  
 52 | Active | 80 | 2.15.3 | camel-commands-core  
 53 | Active | 80 | 2.15.3 | camel-core  
 54 | Active | 80 | 2.15.3 | camel-spring  
 55 | Active | 80 | 2.15.3 | camel-karaf-commands  
 56 | Active | 80 | 1.1.1  | geronimo-jta_1.1_spec  
 61 | Active | 80 | 2.2.6.1 | Apache ServiceMix :: Bundles :: jaxb-impl  
 80 | Active | 80 | 3.1.4  | Stax2 API  
 81 | Active | 80 | 4.4.1  | Woodstox XML-processor  
 karaf@root()>   
 karaf@root()> system:shutdown  
 Confirm: halt instance root (yes/no): yes  
 karaf@root()>   
   

In this article, we are just going through the surface on what apache karaf can do and it certainly deliver! If you are looking for docker container alternative, apache karaf certainly worth the time to look into. Hence forth I think these links will provide you further learning experience.

http://liquid-reality.de/display/liquid/Karaf+Tutorials

https://karaf.apache.org/manual/latest/users-guide/

Sunday, October 25, 2015

Learning Java Eden Space


If you have been a java developer and you should came across java garbage collection that free the object created by your application from occupied all the java heap. In today article, we will look into java heap and particular into java eden space. First, let's look at the general java heap.

From this StackOverflow

Heap memory

The heap memory is the runtime data area from which the Java VM allocates memory for all class instances and arrays. The heap may be of a fixed or variable size. The garbage collector is an automatic memory management system that reclaims heap memory for objects.

Eden Space: The pool from which memory is initially allocated for most objects.

Survivor Space: The pool containing objects that have survived the garbage collection of the Eden space.

Tenured Generation: The pool containing objects that have existed for some time in the survivor space.

When you created a new object, jvm allocate a part of the heap for your object. Visually, it is something as of following.

                   +-----+  
                   |     |  
   <-minor gc->    v     v   <------------- major gc---------------------->  
   +------------+-----+-----+----------------------------------------------+-------------+  
   |            |     |     |                                              |             |
   | Eden       | S0  | S1  |  Tenure Generation                           | Perm gen    |
   |            |     |     |                                              |             |
   +------------+-----+-----+----------------------------------------------+-------------+  
    <---------------------jvm heap (-Xms -Xmx)----------------------------> -XX:PermSize  
    <-- young gen(-Xmn)---->                                                -XX:MaxPermSize  

When eden space is fill with object and minor gc is performed, survive objects will copy to either survivor spaces; s0 or s1. At a time, one of the survivor space is empty. Because the eden space are relatively small in comparison to the tenure generation, hence, the gc that happened in eden space is quick.  Eden and both survivors spaces are also known as young or new generation.

To understand into how young generation heap get free, this article provided detail explanation.

The Sun/Oracle HotSpot JVM further divides the young generation into three sub-areas: one large area named "Eden" and two smaller "survivor spaces" named "From" and "To". As a rule, new objects are allocated in "Eden" (with the exception that if a new object is too large to fit into "Eden" space, it will be directly allocated in the old generation). During a GC, the live objects in "Eden" first move into the survivor spaces and stay there until they have reached a certain age (in terms of numbers of GCs passed since their creation), and only then they are transferred to the old generation. Thus, the role of the survivor spaces is to keep young objects in the young generation for a little longer than just their first GC, in order to be able to still collect them quickly should they die soon afterwards.
Based on the assumption that most of the young objects may be deleted during a GC, a copying strategy ("copy collection") is being used for young generation GC. At the beginning of a GC, the survivor space "To" is empty and objects can only exist in "Eden" or "From". Then, during the GC, all objects in "Eden" that are still being referenced are moved into "To". Regarding "From", the still referenced objects in this space are handled depending on their age. If they have not reached a certain age ("tenuring threshold"), they are also moved into "To". Otherwise they are moved into the old generation. At the end of this copying procedure, "Eden" and "From" can be considered empty (because they only contain dead objects), and all live objects in the young generation are located in "To". Should "to" fill up at some point during the GC, all remaining objects are moved into the old generation instead (and will never return). As a final step, "From" and "To" swap their roles (or, more precisely, their names) so that "To" is empty again for the next GC and "From" contains all remaining young objects.

As you can observed based on the visual diagram above, you can set the amount of heap for the eden and survivor space using -Xmn in the java parameter. There is also -XX:SurvivorRatio=ratio and you can find further information here for java8. Note that in the diagram above, Perm gen has been removed in java8, hence always refer find out what java run your application and refer to the right version of java documentation.

If you want to monitor the statistics of eden , you can use jstats. Previously I have written an article about jstat and you can read here what is jstat and how to use it. You can also enable gc log statistics and so jvm will write the gc statistics into a file, you can further read more here.

Till then we meet again in the next article. Please consider donate, thank you!

Saturday, October 24, 2015

Study MongoDB security by setup and configure server and client on secure line

It's been a while since my last learning on MongoDB. The last learning on MongoDB was on administration. Today, we will learn another topic of mongoDB; MongoDB security. As a general for MongoDB security context, it means

Maintaining a secure MongoDB deployment requires administrators to implement controls to ensure that users and applications have access to only the data that they require. MongoDB provides features that allow administrators to implement these controls and restrictions for any MongoDB deployment.

This article is reference the official documentation which can be found here. As the security context is pretty huge, in this short article, we will focus how to setup mongdb server to use on ssl and how client will access the database resource securely.

First, make sure you have install the server and client package. If you are on deb package linux distribution, it is as easy as sudo apt-get install mongodb-clients mongodb-server. Once both packages are install, you can check in the log file at /var/log/mongodb/mongodb.log similar such as the following. So our mongodb version is 2.6.3 and it has support using openssl library.

 2015-09-27T16:04:48.849+0800 [initandlisten] db version v2.6.3  
 2015-09-27T16:04:48.849+0800 [initandlisten] git version: nogitversion  
 2015-09-27T16:04:48.849+0800 [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014  

Next, let's generate a public and private key and a self sign certifcate.

 user@localhost:~/test1$ openssl req -newkey rsa:2048 -new -x509 -days 365 -nodes -out mongodb-cert.crt -keyout mongodb-cert.key  
 Generating a 2048 bit RSA private key  
 .............................+++  
 ..................................................................................................................................................................................................................+++  
 writing new private key to 'mongodb-cert.key'  
 -----  
 You are about to be asked to enter information that will be incorporated  
 into your certificate request.  
 What you are about to enter is what is called a Distinguished Name or a DN.  
 There are quite a few fields but you can leave some blank  
 For some fields there will be a default value,  
 If you enter '.', the field will be left blank.  
 -----  
 Country Name (2 letter code) [AU]:MY  
 State or Province Name (full name) [Some-State]:KL  
 Locality Name (eg, city) []:Kuala Lumpur  
 Organization Name (eg, company) [Internet Widgits Pty Ltd]:example.com  
 Organizational Unit Name (eg, section) []:Engineering  
 Common Name (e.g. server FQDN or YOUR name) []:Jason Wee  
 Email Address []:jason@example.com  
 user@localhost:~/test1$ ls  
 mongodb-cert.crt mongodb-cert.key  

Now put everything into a file with extension .pem.

 user@localhost:~/test1$ cat mongodb-cert.key mongodb-cert.crt > mongodb.pem  

Now, stop mongodb instance if it is running. As we will now configured the server to use the certificate we generated previously.

 user@localhost:~/test1$ sudo systemctl status mongodb  
 ● mongodb.service - An object/document-oriented database  
   Loaded: loaded (/lib/systemd/system/mongodb.service; enabled; vendor preset: enabled)  
   Active: inactive (dead) since Sun 2015-09-27 16:13:34 MYT; 23min ago  
    Docs: man:mongod(1)  
  Main PID: 15343 (code=exited, status=0/SUCCESS)  
   
 Sep 27 16:04:48 localhost systemd[1]: Started An object/document-oriented database.  
 Sep 27 16:04:48 localhost systemd[1]: Starting An object/document-oriented database...  
 Sep 27 16:13:33 localhost systemd[1]: Stopping An object/document-oriented database...  
 Sep 27 16:13:34 localhost systemd[1]: Stopped An object/document-oriented database.  
 Sep 27 16:36:30 localhost systemd[1]: Stopped An object/document-oriented database.  
 user@localhost:~/test1$ sudo tail -10 /etc/mongodb.conf   
 # Size limit for in-memory storage of op ids.  
 #opIdMem = <bytes>  
   
 # SSL options  
 # Enable SSL on normal ports  
 sslOnNormalPorts = true  
 # SSL Key file and password  
 #sslPEMKeyFile = /etc/ssl/mongodb.pem  
 sslPEMKeyFile = /home/user/test1/mongodb.pem  
 #sslPEMKeyPassword = pass  
 user@localhost:~/test1$   

In the above output, as an example, I have set the file mongodb.pem to the configuration sslPEMKeyFile and also set sslOnNormalPorts to true. Now if you start mongodb instance.

 user@localhost:~/test1$ sudo systemctl start mongodb  
 user@localhost:~/test1$   

In the log file, noticed that ssl is enabled and no ssl related error.

 2015-09-27T16:46:41.648+0800 [initandlisten] options: { config: "/etc/mongodb.conf", net: { bindIp: "127.0.0.1", ssl: { PEMKeyFile: "/home/user/test1/mongodb.pem", mode: "requireSSL" } }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log" } }  
 2015-09-27T16:46:41.788+0800 [initandlisten] journal dir=/var/lib/mongodb/journal  
 2015-09-27T16:46:41.797+0800 [initandlisten] recover : no journal files present, no recovery needed  
 2015-09-27T16:46:42.162+0800 [initandlisten] waiting for connections on port 27017 ssl  

On the server configuration and setup, it is now done. Now, we will focus on the mongdb client. If you connect to mongodb using its client, you will get error.

 user@localhost:~/test1$ mongo foo  
 MongoDB shell version: 2.6.3  
 connecting to: foo  
 2015-09-27T17:22:54.300+0800 DBClientCursor::init call() failed  
 2015-09-27T17:22:54.302+0800 Error: DBClientBase::findN: transport error: 127.0.0.1:27017 ns: admin.$cmd query: { whatsmyuri: 1 } at src/mongo/shell/mongo.js:146  
 exception: connect failed  
 user@localhost:~/test1$ mongo --ssl --sslPEMKeyFile mongodb.pem  
 MongoDB shell version: 2.6.3  
 connecting to: test  
 Server has startup warnings:   
 2015-09-27T16:46:41.647+0800 [initandlisten]   
 2015-09-27T16:46:41.647+0800 [initandlisten] ** NOTE: This is a 32 bit MongoDB binary.  
 2015-09-27T16:46:41.647+0800 [initandlisten] **    32 bit builds are limited to less than 2GB of data (or less with --journal).  
 2015-09-27T16:46:41.647+0800 [initandlisten] **    See http://dochub.mongodb.org/core/32bit  
 2015-09-27T16:46:41.647+0800 [initandlisten]   
 > show dbs  
 admin (empty)  
 local 0.078GB  
 >   

As you can read above, you need to specify parameter ssl and the pem file. That's it for this article, if you want to go the distance, try using tcpdump to listen to this port traffic. Good luck!

Friday, October 23, 2015

cracking wireless wep router encryption

Recently I noticed one of my neighbor router encryption is using WEP and I thought maybe to spend sometime to aircrack it. Googling shown many articles that teach you on how to gain access to wireless router that using WEP as the encryption standard. In this article, I mainly reference this article and running this on my linux workstation.

With that said, let's begin. Install aircrack-ng package that contain essentials applications that sniff the air traffic and crack it.

 root@localhost:~# apt-get install aircrack-ng  
 Reading package lists... Done  
 Building dependency tree      
 Reading state information... Done  
 The following packages were automatically installed and are no longer required:  
  libgee2 libmono-2.0-1 libmono-2.0-dev libmono-accessibility4.0-cil libmono-c5-1.1-cil libmono-cecil-private-cil libmono-codecontracts4.0-cil  
  libmono-compilerservices-symbolwriter4.0-cil libmono-cscompmgd8.0-cil libmono-custommarshalers4.0-cil libmono-db2-1.0-cil libmono-debugger-soft2.0a-cil  
  libmono-debugger-soft4.0a-cil libmono-entityframework-sqlserver6.0-cil libmono-entityframework6.0-cil libmono-http4.0-cil libmono-ldap2.0-cil  
  libmono-ldap4.0-cil libmono-management2.0-cil libmono-management4.0-cil libmono-messaging-rabbitmq2.0-cil libmono-messaging-rabbitmq4.0-cil  
  libmono-messaging4.0-cil libmono-microsoft-build-engine4.0-cil libmono-microsoft-build-framework4.0-cil libmono-microsoft-build-tasks-v4.0-4.0-cil  
  libmono-microsoft-build-utilities-v4.0-4.0-cil libmono-microsoft-build2.0-cil libmono-microsoft-build4.0-cil libmono-microsoft-visualc10.0-cil  
  libmono-microsoft-web-infrastructure1.0-cil libmono-microsoft8.0-cil libmono-npgsql2.0-cil libmono-npgsql4.0-cil libmono-opensystem-c4.0-cil  
  libmono-oracle2.0-cil libmono-oracle4.0-cil libmono-parallel4.0-cil libmono-peapi2.0a-cil libmono-peapi4.0a-cil libmono-profiler libmono-rabbitmq2.0-cil  
  libmono-rabbitmq4.0-cil libmono-relaxng2.0-cil libmono-relaxng4.0-cil libmono-sharpzip2.6-cil libmono-simd2.0-cil libmono-simd4.0-cil  
  libmono-system-componentmodel-composition4.0-cil libmono-system-componentmodel-dataannotations4.0-cil libmono-system-configuration-install4.0-cil  
  libmono-system-data-datasetextensions4.0-cil libmono-system-data-linq4.0-cil libmono-system-data-services-client4.0-cil  
  libmono-system-data-services2.0-cil libmono-system-data-services4.0-cil libmono-system-design4.0-cil libmono-system-drawing-design4.0-cil  
  libmono-system-dynamic4.0-cil libmono-system-identitymodel-selectors4.0-cil libmono-system-identitymodel4.0-cil  
  libmono-system-io-compression-filesystem4.0-cil libmono-system-io-compression4.0-cil libmono-system-json-microsoft4.0-cil libmono-system-json2.0-cil  
  libmono-system-json4.0-cil libmono-system-ldap-protocols4.0-cil libmono-system-ldap2.0-cil libmono-system-ldap4.0-cil libmono-system-management4.0-cil  
  libmono-system-messaging4.0-cil libmono-system-net-http-formatting4.0-cil libmono-system-net-http-webrequest4.0-cil libmono-system-net-http4.0-cil  
  libmono-system-net2.0-cil libmono-system-net4.0-cil libmono-system-numerics4.0-cil libmono-system-reactive-core2.2-cil  
  libmono-system-reactive-debugger2.2-cil libmono-system-reactive-experimental2.2-cil libmono-system-reactive-interfaces2.2-cil  
  libmono-system-reactive-linq2.2-cil libmono-system-reactive-observable-aliases0.0-cil libmono-system-reactive-platformservices2.2-cil  
  libmono-system-reactive-providers2.2-cil libmono-system-reactive-runtime-remoting2.2-cil libmono-system-reactive-windows-forms2.2-cil  
  libmono-system-reactive-windows-threading2.2-cil libmono-system-runtime-caching4.0-cil libmono-system-runtime-durableinstancing4.0-cil  
  libmono-system-servicemodel-activation4.0-cil libmono-system-servicemodel-discovery4.0-cil libmono-system-servicemodel-routing4.0-cil  
  libmono-system-servicemodel-web4.0-cil libmono-system-servicemodel4.0a-cil libmono-system-serviceprocess4.0-cil  
  libmono-system-threading-tasks-dataflow4.0-cil libmono-system-web-abstractions4.0-cil libmono-system-web-dynamicdata4.0-cil  
  libmono-system-web-extensions-design4.0-cil libmono-system-web-extensions4.0-cil libmono-system-web-http-selfhost4.0-cil  
  libmono-system-web-http-webhost4.0-cil libmono-system-web-http4.0-cil libmono-system-web-mvc1.0-cil libmono-system-web-mvc2.0-cil  
  libmono-system-web-mvc3.0-cil libmono-system-web-razor2.0-cil libmono-system-web-routing4.0-cil libmono-system-web-webpages-deployment2.0-cil  
  libmono-system-web-webpages-razor2.0-cil libmono-system-web-webpages2.0-cil libmono-system-windows-forms-datavisualization4.0a-cil  
  libmono-system-windows-forms4.0-cil libmono-system-windows4.0-cil libmono-system-xaml4.0-cil libmono-system-xml-serialization4.0-cil  
  libmono-tasklets2.0-cil libmono-tasklets4.0-cil libmono-webbrowser4.0-cil libmono-webmatrix-data4.0-cil libmono-windowsbase3.0-cil  
  libmono-windowsbase4.0-cil libmono-xbuild-tasks2.0-cil libmono-xbuild-tasks4.0-cil libmonoboehm-2.0-1 libmonoboehm-2.0-dev libmonosgen-2.0-1  
  libnunit-cil-dev libts-0.0-0 libts-0.0-0:i386 libupower-glib1 linux-image-amd64 mono-2.0-service mono-4.0-service mono-csharp-shell mono-jay mono-utils  
  mono-xbuild monodoc-base monodoc-browser monodoc-manual python-gtksourceview2 python3-packagekit tsconf  
 Use 'apt-get autoremove' to remove them.  
 The following extra packages will be installed:  
  ieee-data  
 The following NEW packages will be installed:  
  aircrack-ng ieee-data  
 0 upgraded, 2 newly installed, 0 to remove and 523 not upgraded.  
 Need to get 1,244 kB of archives.  
 After this operation, 5,914 kB of additional disk space will be used.  
 Do you want to continue? [Y/n] Y  
 Get:1 http://cdn.debian.net/debian/ unstable/main aircrack-ng amd64 1:1.2-0~beta3-4 [435 kB]  
 Get:2 http://cdn.debian.net/debian/ unstable/main ieee-data all 20141019.1 [809 kB]  
 Fetched 1,244 kB in 5s (228 kB/s)   
 Selecting previously unselected package aircrack-ng.  
 (Reading database ... 325525 files and directories currently installed.)  
 Preparing to unpack .../aircrack-ng_1%3a1.2-0~beta3-4_amd64.deb ...  
 Unpacking aircrack-ng (1:1.2-0~beta3-4) ...  
 Selecting previously unselected package ieee-data.  
 Preparing to unpack .../ieee-data_20141019.1_all.deb ...  
 Unpacking ieee-data (20141019.1) ...  
 Processing triggers for man-db (2.7.0.2-5) ...  
 Setting up aircrack-ng (1:1.2-0~beta3-4) ...  
 Setting up ieee-data (20141019.1) ...  

okay, the package installed successfully. Next you need to turn the wireless interface to monitor mode and so the interface can intercept any wireless traffic nearby.


As you can read above, I have stop network-manager and so my interface will not automatically connect to my wireless router. In this example, I use one of my wireless interface for this learning adventure. Once the wireless interface is turned into monitor mode, you can start to dump the air traffic using the command airodump-ng .




Identify any wireless router that use WEP encryption protocol, and then start another terminal to write all these traffics into a tcpdump file. You can do that using the command airodump-ng -c 6 -w data-capture wlan0 .


Next, you can inject (send traffic) more packets to the identified router using aireplay . As you can see below, I experiment with a few aireplay parameter and you should too.




If your neighbour WEP wireless router is an active users, airodump-ng should be able to capture sufficient of initialization vectors(IVs) for your next aircrack command. I suggest you leave over few hours to collect maybe 10k of IVs and run this tcpdump capture with a powerful cpu.



That's it. Good luck, have fun and be good.



Sunday, October 11, 2015

How to install D-Link DWA-123 Wireless N 150 adapter in debian

Today I got myself a new wireless usb device. Pricing for D-Link DWA-123 wireless N 150 is very affordable and only at 17MYR (01 september 2015) in local computer store. So I got myself a unit and try out, and it's working fine after more than ten days. I will share with you how do I install this unit in linux debian.


If you are using kernel 4.0 or above, the module should come together with the kernel. You can identify below and you plug the device into the usb port.

 root@localhost:~# modinfo r8188eu  
 filename:    /lib/modules/4.0.0-2-amd64/kernel/drivers/staging/rtl8188eu/r8188eu.ko  
 version:    v4.1.4_6773.20130222  
 author:     Realtek Semiconductor Corp.  
 description:  Realtek Wireless Lan Driver  
 license:    GPL  
 srcversion:   A3DA328AE8853D31D90212F  
 alias:     usb:v0DF6p0076d*dc*dsc*dp*ic*isc*ip*in*  
 alias:     usb:v2001p3311d*dc*dsc*dp*ic*isc*ip*in*  
 alias:     usb:v2001p3310d*dc*dsc*dp*ic*isc*ip*in*  
 alias:     usb:v2001p330Fd*dc*dsc*dp*ic*isc*ip*in*  
 alias:     usb:v07B8p8179d*dc*dsc*dp*ic*isc*ip*in*  
 alias:     usb:v056Ep4008d*dc*dsc*dp*ic*isc*ip*in*  
 alias:     usb:v0BDAp0179d*dc*dsc*dp*ic*isc*ip*in*  
 alias:     usb:v0BDAp8179d*dc*dsc*dp*ic*isc*ip*in*  
 depends:    usbcore  
 staging:    Y  
 intree:     Y  
 vermagic:    4.0.0-2-amd64 SMP mod_unload modversions   
 parm:      rtw_ips_mode:The default IPS mode (int)  
 parm:      ifname:The default name to allocate for first interface (charp)  
 parm:      if2name:The default name to allocate for second interface (charp)  
 parm:      rtw_initmac:charp  
 parm:      rtw_channel_plan:int  
 parm:      rtw_chip_version:int  
 parm:      rtw_rfintfs:int  
 parm:      rtw_lbkmode:int  
 parm:      rtw_network_mode:int  
 parm:      rtw_channel:int  
 parm:      rtw_wmm_enable:int  
 parm:      rtw_vrtl_carrier_sense:int  
 parm:      rtw_vcs_type:int  
 parm:      rtw_busy_thresh:int  
 parm:      rtw_ht_enable:int  
 parm:      rtw_cbw40_enable:int  
 parm:      rtw_ampdu_enable:int  
 parm:      rtw_rx_stbc:int  
 parm:      rtw_ampdu_amsdu:int  
 parm:      rtw_lowrate_two_xmit:int  
 parm:      rtw_rf_config:int  
 parm:      rtw_power_mgnt:int  
 parm:      rtw_smart_ps:int  
 parm:      rtw_low_power:int  
 parm:      rtw_wifi_spec:int  
 parm:      rtw_antdiv_cfg:int  
 parm:      rtw_antdiv_type:int  
 parm:      rtw_enusbss:int  
 parm:      rtw_hwpdn_mode:int  
 parm:      rtw_hwpwrp_detect:int  
 parm:      rtw_hw_wps_pbc:int  
 parm:      rtw_max_roaming_times:The max roaming times to try (uint)  
 parm:      rtw_fw_iol:FW IOL (int)  
 parm:      rtw_mc2u_disable:int  
 parm:      rtw_80211d:Enable 802.11d mechanism (int)  
 parm:      rtw_notch_filter:0:Disable, 1:Enable, 2:Enable only for P2P (uint)  
 parm:      debug:Set debug level (1-9) (default 1) (int)  
 root@localhost:~# dpkg -S /lib/modules/4.0.0-2-amd64/kernel/drivers/staging/rtl8188eu/r8188eu.ko  
 linux-image-4.0.0-2-amd64: /lib/modules/4.0.0-2-amd64/kernel/drivers/staging/rtl8188eu/r8188eu.ko  

If you get the following message in your syslog,

 Sep 1 19:12:33 localhost kernel: [ 385.525522] r8188eu 1-1.1.2.3:1.0: firmware: failed to load rtlwifi/rtl8188eufw.bin (-2)  
 Sep 1 19:12:33 localhost kernel: [ 385.525530] r8188eu 1-1.1.2.3:1.0: Direct firmware load for rtlwifi/rtl8188eufw.bin failed with error -2  
 Sep 1 19:12:33 localhost kernel: [ 385.525534] r8188eu 1-1.1.2.3:1.0: Firmware rtlwifi/rtl8188eufw.bin not available  
 Sep 1 19:12:33 localhost kernel: [ 385.525539] MAC Address = 00:00:00:00:00:00  

What you need to do next is to install the firmware. The firmware is in the repository and you can install as easy as apt-get.

 root@localhost:~# apt-get install firmware-realtek  

Now the userspace application should be able to identify the device correctly. I am using gnome so, it is detected as USB Wi-FI. Note that I have an existing pci wifi as shown in the screenshot below.


Try out to device to retrieve the IP address etc. If you get your wireless to a funky name such as wlxc412f52da87d , then you can create a rule in the udev configuration file.

 # cat /etc/udev/rules.d/70-persistent-net.rules  
 # This file was automatically generated by the /lib/udev/write_net_rules  
 # program, run by the persistent-net-generator.rules rules file.  
 #  
 # You can modify it, as long as you keep each rule on a single  
 # line, and change only the value of the NAME= key.  
   
 # PCI device 0x0000:0x0000 (atl1c)  
 SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:00:00:00:00:00", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"  
   
 # PCI device 0x0000:0x0000 (brcm80211)  
 SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:00:00:00:00:0", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="wlan*", NAME="wlan0"  
   
 # USB device 0x0000:0x0000 (r8188eu)  
 SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:00:00:00:00:00", ATTR{dev_id}=="0x0", ATTR{type}=="1", KERNEL=="wlan*", NAME="wlan1"  

As you can see above, PCI device my existing device and you should add another line similar to the above. You need to replace ATTR{address} value to your device mac address. When the operating system bring up this device, it will be renamed to wlan1 instead of a random interface name next time.

Saturday, October 10, 2015

post mortem of anonymous vs najib on august 2015


When the video is circulating in the social media, it create additional sensational hype on the verge of incoming bersih 4.0 rally at the month end of august of 2015. Some said in the youtube comment this video above is fake, some news circulating even the IGP of malaysia has even commented on this.

Whether it is conspiracies from any parties or if this incident really happened, it has been weeks after the date set forth in the video. In this article, we will take a look at what actually had happened on 29 august 2015.

For a start, one things is ascertain, the sites that were identified by the anonymous groups in the video; on that day were all having ddos protections. Take a look at the screenshots below taken on 29 august 2015.






All the sites above have taken precaution steps to prevent denial of services from happening. From the norse map, as usual, ddos happen every time but nothing particulary on the time declared by the anonymous toward malaysia IPs.




However, sometime around 7.30pm, the royal malaysian police website cannot be access for duraiton of time. It could be just a minor glitch.


So today I take a look at the digital attack map on the day 29 and 30 august 2015. Nothing massive attack happened to malaysia IPs.




It definitely looks like there is no obvious denial of services happened on that two days. But why and what is the motive of the anonymous video was created for? Has anonymous failed in the attack given the additional security prevention has setup in placed? Well, I would say unlikely. It's a strange world indeed.

Friday, October 9, 2015

Mail server setup


“Why is legitimate email sent from my server rejected or flagged as spam?”

In the last eight years working as mail- and spam filter administrator I’ve had to answer this question many times. If you google it on the web, you’re often getting answers like “because you don’t have SPF/DKIM/DMARC”. This might be the reason in some cases, but most of the time the problem lies in the basic server hostname and DNS setup.


Many spam filters first try to find out, if the sending device was meant to send mail or if its just yet another hacked VPS / home PC  out there. So your job is to clearly tell them "yes, the administrator of this device and the ISP agree that this system should send mail".  And you tell them this by setting a distinctive SMTP HELO (which makes clear that the SMTP engine was set up by a mail administrator),
DNS A-record (which defines that IP address is the one meant send mail) and reverse DNS (which confirms that the owner of the IP adddress  agrees). If you get these three things right, chances are already pretty high that your legitimate mail will be accepted even without SPF and DKIM.  

System hostname / SMTP HELO


Most mail servers automatically use the system hostname as SMTP HELO - so this is the first thing that should be set correctly.
  • Configure a fully qualified domain name on your mailserver (something like ‘mail.example.com’, 'exchange.example.com', 'mta.example.com' )
  • DO NOT actually use “example.com” - we will use this domain as an example here, but you should not on your server. Whenever you read ‘mail.example.com’ below, replace it with the fully qualified hostname you’ve chosen in a domain under your control. (yes, I’ve seen people literally configure ‘example.com’ on their servers. It’s a bad idea.)
  • DO NOT use hostname in a domain of your ISP/VPS Provider. Use your OWN domain.
  • DO NOT choose a hostname that looks auto-generated (based your IP address etc). 
  • DO NOT try to be cute and invent your own top level domain ("myserver.home", "exchange.lan") - your hostname should be publicly resolvable

IP

  • Send mail from a static IP address. If you don’t have one, use a smarthost that does
  • If your IP address is listed on the Spamhaus PBL it is not meant to be used to send mail directly. Use a smarthost or ask your ISP for an IP in a different range.
  • if you have multiple static IP addresses available, configure a dedicated IP for your mail server which is not used as gateway by any other devices in your network. This reduces the risk of an infected device causing your mail IP to get blacklisted

Reverse DNS

  • Set the reverse DNS (PTR) entry to the your server’s hostname (x.x.x.x.in-addr.arpa PTR mail.example.com)
  • DO NOT set multiple PTRs. Your sending IP should have exactly one PTR. It does not matter if this server is hosting multiple domains for web / mail. The PTR is used to identify the sending server, not the domains it is hosting
  • DO NOT use generic PTRs from your upstream/IP/vps provider. (like x.x.x.x-static.reverse.softlayer.com). Set this to a domain under your control.

A-record

  • Make sure there is an A record that maps the hostname used in your PTR back to your sending IP address (see FcRDNS)
  • if you used a different FQDN in your smtp HELO for any reason, add an A record for this  as well . Only extremely agressive spam filters will actually check if your helo is resolvable, but RFC 5321 states "Only resolvable, fully-qualified domain names (FQDNs) are permitted  when domain names are used in SMTP.", so better be safe than sorry.