月度归档:2014年07月

用ThreadLocal解决SimpleDateFormat的性能问题

SimpleDateFormat是非线程安全的,在并发下碰到的案例见过好几次了。若场景对性能要求很高,创建大量生命周期很短暂的实例对象也是不小开销(主要是SimpleDateFormat内部比较复杂),可以考虑采用ThreadLocal来优化,以前曾见到过这种优化方式,忘了出处,在tomcat的代码里也有这样的用法,下面代码是tomcat7的DateTool工具类里的:

    public static final ThreadLocal<DateFormat> rfc1123Format = new ThreadLocal<DateFormat>() {
    @Override
    public DateFormat initialValue() {
        DateFormat result = new SimpleDateFormat(RFC1123_PATTERN, Locale.US);
        result.setTimeZone(GMT_ZONE);
        return result;
    }
};

调用的时候,每个线程可以通过rfc1123Format.get().format(...) 来使用。不过tomcat8里已经删除了这个类。

使用strace定位jvm退出的原因范围

今天遇到的一个tomcat启动过程中jvm退出的问题,不是jvm crash的情况,用户日志配置的不正确导致一些信息没有展现出来,只看到pandora执行了shutdownhook的信息。这可能是启动时的逻辑有触发System.exit,或被系统或人为kill掉了。

根据以往的经验,排除了oom killer或ulimit -t设置不当导致被内核给kill掉的情况,OS级别的signal通常不留机会给jvm执行shutdownhook的。如此一来singal的范围应该就是SIGTERM, SIGINT, SIGHUP这3种(参考这里)。

虽然singal范围缩小,但依然不能确定是因为代码里调用了System.exit还是人为(或被其他进程)kill引起的。直接上大招用systemtap需要安装kernal debuginfo,没有权限的话,还要找到对应的人去做;如果现象较容易重现的话,可以先通过strace命令进一步缩小问题的范围,究竟是因为jvm内部执行了System.exit还是外界的kill引起的。

这里通过启动一个scala的repl来模拟java进程,通过strace attach到jvm进程上,然后观察,如果是外界的kill所致,可以看到下面的信息:

$ sudo strace -p 1947
Process 1947 attached - interrupt to quit
futex(0x7fb7635959d0, FUTEX_WAIT, 1948, NULL) = ? ERESTARTSYS (To be restarted)
--- SIGTERM (Terminated) @ 0 (0) ---
futex(0x7fb762762360, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn(0x7fb762762360)            = 202
futex(0x7fb7635959d0, FUTEX_WAIT, 1948, NULLPANIC: attached pid 1947 exited with 143
 <unfinished ... exit status 143>

里面的关键信息是SIGTERMexit status 143(即SIGTERM的code)

如果是kill -2或ctrl-c终止repl,可以看到有关SIGINT的信息

$ sudo strace -p 1813
Process 1813 attached - interrupt to quit
futex(0x7fb24d15a9d0, FUTEX_WAIT, 1814, NULL) = ? ERESTARTSYS (To be restarted)
--- SIGINT (Interrupt) @ 0 (0) ---
futex(0x7fb24c327360, FUTEX_WAKE_PRIVATE, 1) = 1
rt_sigreturn(0x7fb24c327360)            = 202
futex(0x7fb24d15a9d0, FUTEX_WAIT, 1814, NULLPANIC: attached pid 1813 exited with 130
 <unfinished ... exit status 130>

如果是jvm自身执行了System.exit比如:

scala> System.exit(0)

那么在跟踪的信息里,是看不到signal的:

$ sudo strace -p 2131
Process 2131 attached - interrupt to quit
futex(0x7fc14adb49d0, FUTEX_WAIT, 2132, NULLPANIC: attached pid 2131 exited with 0
 <unfinished ... exit status 0>

至此我们可以判断出到底是外部还是内部引起的了,如果是内部就不必麻烦Systemtap了,可以从源码去找。

tomcat关闭应用时的清理工作(3): ThreadLocal

tomcat在关闭时,可能看到与ThreadLocal相关的警告:

Jan 24, 2014 7:18:52 AM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [] created a ThreadLocal with key of type 
[java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4e61bc49]) and a value of type 
[com.alibaba.xxx.Entry] (value [...]) but failed to remove it when the web application was stopped. 
Threads are going to be renewed over time to try and avoid a probable memory leak.

用下面的例子来模拟一下

@WebServlet(name = "MainServlet", urlPatterns = { "/main" }, loadOnStartup = 1)
public class MainServlet extends HttpServlet {

    private static final long serialVersionUID = 1L;

    private static ThreadLocal<MyClass> tl = new ThreadLocal<MyClass>();

    public void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws ServletException, IOException {

        MyClass m = tl.get();
        if (m == null) {
            tl.set(new MyClass());
        }
    }

}

class MyClass {}

请求一次之后,通过脚本会命令停止tomcat,会看到类似的日志:

七月 16, 2014 5:01:35 下午 org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
严重: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] 
(value [java.lang.ThreadLocal@7da150]) and a value of type [org.r113.servlet3.MyClass] 
(value [org.r113.servlet3.MyClass@37e98b70]) but failed to remove it when the web application was stopped. 
Threads are going to be renewed over time to try and avoid a probable memory leak.

这个泄露其实是可能造成classloader的泄露,因为ThreadLocal引用了自定义的类MyClass,绑定到了当前的请求线程上,而请求线程又是线程池里的线程,生存周期可能会比较长。比如上面模拟的情况,要停止应用的时候,请求线程的ThreadLocal仍未释放,那么即使加载MyClass类的classLoader已经不会再被任何地方使用,可以被垃圾回收了,却因为这个MyClass被引用而得不到回收。

tomcat关闭应用时的清理工作(2): 线程的清理

tomcat在关闭时,有时会看到类似下面的警告信息:

2014-7-10 13:44:02 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] appears to have started a thread named 
[com.taobao.xxx.client.Timer] but has failed to stop it. 
This is very likely to create a memory leak.

这是tomcat关闭应用时检测到了应用启动的线程未被终止,tomcat为防止造成内存泄露,给出上面的警告,并根据配置来决定是否强制停止该线程(默认不会强制停止)。

有时也会有另一种相似的警告信息:

2014-7-10 13:44:02 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [] is still processing a request that has yet to finish. 
This is very likely to create a memory leak. 
You can control the time allowed for requests to finish 
by using the unloadDelay attribute of the standard Context implementation.

这是tomcat关闭应用时检测到了仍有请求线程未处理完。

上面的2种警告都是在WebappClassLoaderclearReferencesThreads方法里给出的,该方法也是在stop时调用clearReferences方法时调用的:

protected void clearReferences() {
    ...
    // Stop any threads the web application started
    clearReferencesThreads();
    ...
}

clearReferencesThreads方法里,通过找到最顶层的root thread group获取所有的active线程,然后判断这些线程如果是用户线程的话,给出警告:

if (isRequestThread(thread)) {
    log.error(sm.getString("webappClassLoader.warnRequestThread",
                            contextName, thread.getName()));
} else {
    log.error(sm.getString("webappClassLoader.warnThread",
                            contextName, thread.getName()));
}

我们来模拟一下,先看第一种情况:

@WebServlet(name = "MainServlet", urlPatterns = { "/main" }, loadOnStartup = 1)
public class MainServlet extends HttpServlet {

    private static final long serialVersionUID = 1L;

    public void init() {
        new Thread() {
            public void run() {
              try {
                while (true) {
                    Thread.sleep(1000);
                }
              } catch (Exception e) {
              }
            }
        }.start();
    }
}

在一个servlet初始化时启动了一个线程,没有提供销毁这个线程的机制,当tomcat停止时,会报第一种警告。

再模拟第二种警告情况,在请求时将线程hang住:

public void doGet(HttpServletRequest req, HttpServletResponse resp)
        throws ServletException, IOException {
    try {
        Thread.sleep(1000 * 200);
    } catch (Exception e) {
    }
}

这时关闭tomcat时会报出第二种警告信息。

默认tomcat并不会强制将这些线程终止,除非设置了clearReferencesStopThreads为true,它判断线程属于某个线程池则延迟一段时间将线程终止,否则直接调用了JDK已不鼓励的Thread.stop方法终止线程。

if (usingExecutor) {
    // Executor may take a short time to stop all the
    // threads. Make a note of threads that should be
    // stopped and check them at the end of the method.
    executorThreadsToStop.add(thread);
} else {
    // This method is deprecated and for good reason. This
    // is very risky code but is the only option at this
    // point. A *very* good reason for apps to do this
    // clean-up themselves.
    thread.stop();
}

tomcat关闭应用时的清理工作(1): JDBC Driver

tomcat在关闭应用时,对资源做了一些清理,避免了泄露,这个工作主要是WebappClassLoader里做的,WebappClassLoader也实现自Lifecycle接口,在应用关闭时,会触发其stop方法:

@Override
public void stop() throws LifecycleException {
    // Clearing references should be done before setting started to
    // false, due to possible side effects
    clearReferences();
    ......
}

释放引用的工作主要在clearReferences里:

protected void clearReferences() {

    // De-register any remaining JDBC drivers
    clearReferencesJdbc();

    // Stop any threads the web application started
    clearReferencesThreads();

    // Check for leaks triggered by ThreadLocals loaded by this class loader
    checkThreadLocalsForLeaks();

    // Clear RMI Targets loaded by this class loader
    clearReferencesRmiTargets();

    // Null out any static or final fields from loaded classes,
    // as a workaround for apparent garbage collection bugs
    if (clearReferencesStatic) {
        clearReferencesStaticFinal();
    }

     // Clear the IntrospectionUtils cache.
    IntrospectionUtils.clear();

    // Clear the classloader reference in common-logging
    if (clearReferencesLogFactoryRelease) {
        org.apache.juli.logging.LogFactory.release(this);
    }

    // Clear the resource bundle cache
    // This shouldn't be necessary, the cache uses weak references but
    // it has caused leaks. Oddly, using the leak detection code in
    // standard host allows the class loader to be GC'd. This has been seen
    // on Sun but not IBM JREs. Maybe a bug in Sun's GC impl?
    clearReferencesResourceBundles();

    // Clear the classloader reference in the VM's bean introspector
    java.beans.Introspector.flushCaches();

}

其中对JDBC Driver的清理,是clearReferencesJdbc方法,它检查当前WebappClassLoader加载过的,在关闭时未注销掉的JDBC Driver,给出警告信息,并强行将这些Driver反注册掉。所以我们有时在关闭时会看到类似下面的信息:

SEVERE: The web application [xxx] registered the JDBC driver 
[com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. 
To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

我们通过下面的例子来模拟一下这个现象:

@WebServlet(name = "MainServlet", urlPatterns = { "/main" }, loadOnStartup=1)
public class MainServlet extends HttpServlet {

    private static final long serialVersionUID = 1L;

    public void init() {
        try {
            Class.forName("com.alibaba.druid.mock.MockDriver");
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws ServletException, IOException {
        PrintWriter wr = resp.getWriter();
        wr.write("ok");
        wr.flush();
    }
}

上面的servlet在初始化时注册了一个Driver,但销毁时未将这个Driver给反注册掉;这时不管是显式的通过命令来stop tomcat,还是因为设置了自动reload,而且恰好检查到应用有变,执行了reload的时候(reload也是对app context进行stop,然后再重新start),就会被tomcat判断为泄露,给出警告并强制反注册Driver:

$ kill `pidof java`

// 在日志里会看到
严重: The web application [] registered the JDBC driver [com.alibaba.druid.mock.MockDriver]
but failed to unregister it when the web application was stopped. 
To prevent a memory leak, the JDBC Driver has been forcibly unregistered.

要避免这个信息,应用或框架应该自己来保证在销毁时将JDBC Driver反注册掉。例如在destroy方法里:

@Override
public void destroy() {
    super.destroy();
    try{
        DriverManager.deregisterDriver(DriverManager.getDrivers().nextElement());
    }catch(Exception e){
        e.printStackTrace();
    }
}

因为tomcat自带了DBCP数据库连接池,很多用户在使用DBCP时遇到了这个问题,并建议在 DBCP 的 BasicDataSourceclose方法里执行反注册驱动的行为来解决这个警告。但DBCP的开发者认为这个应该是使用者的责任,不愿意接受这种建议,参考这里

jvm与系统信号(4)

stop 与 cont 信号

这两个信号对于jvm也是可用的,比如让jvm进程停止:

scala> var a=0;

scala>  while(true) { Thread.sleep(2000); println(a); a=a+1 }
0
1
2
3

$ ps -ostat -p `pidof java`
STAT
S+

$ kill -s stop `pidof java`

此时jvm进程被暂停住,进程状态也变为:T (TASK_STOPPED or TASK_TRACED),暂停状态或跟踪状态

$ ps -ostat -p `pidof java`
STAT
T+

发送cont信号恢复进程状态:

$ kill -s cont `pidof java`

repl端会继续输出。

jvm与系统信号(3)

一些相关的参数

1) -Xrs

man java里可以看到这个参数的介绍,大意如下:

这个参数是在java1.3.1 之后增加的,rs是reduce signal的缩写,即忽略系统信号。在java1.3.0添加了 Shutdown Hook,目的是用于在jvm关闭时清除一些代码(比如关闭数据库连接)。对于jvm非正常退出,Sun/Oracle的jvm通过捕获信号来实现shutdown hook。JVM使用 SIGHUP, SIGINT,SIGTERM 来初始化 shutdown hook

JVM使用了相似的机制来实现 pre-1.2 特性,dumping线程栈(用于调试目的)。Sun/Oracle的JVM使用 SIGQUIT 来执行 线程 dump.

应用时常也需要自己捕获SIGINT或SIGTERM,这会导致干扰JVM自己的signal handler,为了避免这种情况, -Xrs 命令行参数在java1.3.1里被增加了进来。当使用 Sun的JVM时, SIGINT, SIGTERM, SIGHUP, SIGQUIT不会被影响JVM,这些信号的handler不会被install。

使用-Xrs要注意这2个后果:

1) SIGQUIT 不再产生 thread dump
2) Shutdown hook将不被执行

尝试一下Xrs参数,使用scala -J-Xrs启动一个repl:

$ scala -J-Xrs

$ kill -3 `pidof java` 

这个时候确实不会产生 thread dump,并且java进程会退出。

有趣的是,我使用 jstack 依然可以看到 scala repl的所有线程,也就是jstack依然会把请求发送到java进程,即使它设置了-Xrs,可能与attach机制有关,这里的细节以后再展开。

2) -XX:-AllowUserSignalHandlers

限于Linux和Solaris,默认不启用。允许为java进程安装信号处理器,信号处理参见类:sun.misc.Signal, sun.misc.SignalHandler

3) -XX:+UseAltSigs

限于Solaris,默认启用。为了防止与其他发送信号的应用程序冲突,允许使用候补信号替代 SIGUSR1和SIGUSR2

jvm与系统信号(2)

core dump 与 thread stack dump

进程处理信号的行为参考这里,很多信号都将导致core dump,比如SIGILL, SIGSEGV等。

比如我们对一个java进程发送SIGILL会让进程退出,并产生core dump:

$ kill -s ILL `pidof java`

在java进程的错误输出流会产生如下信息,并且在home下产生core dump文件:hs_err_pid8385.log

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x00007fff89d1ba1a, pid=8385, tid=1287
#
# JRE version: Java(TM) SE Runtime Environment (7.0_60-b15) (build 1.7.0_60-ea-b15)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.60-b09 mixed mode bsd-amd64 compressed oops)
# Problematic frame:
# C  [libsystem_kernel.dylib+0x11a1a]  mach_msg_trap+0xa
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/hongjiang/hs_err_pid8385.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#
/data/tools/scala/bin/scala: line 21:  8385 Abort trap: 6           "$@"

对于SIGQUIT(kill -3),jvm会捕获该信号并dump线程栈到标准错误流,不会产生core-dump文件;这个信号的处理方式是系统保留用户无法修改:

scala> import sun.misc._

scala>  Signal.handle(
 |          new Signal("QUIT"),
 |          new SignalHandler(){ 
 |              def handle(sig:Signal){ println("down") }
 |          }
 |      )
java.lang.IllegalArgumentException: Signal already used by VM or OS: SIGQUIT
at sun.misc.Signal.handle(Signal.java:166)
... 38 elided

$ kill -s QUIT `pidof java`

会看到repl输出线程堆栈信息以及heap信息,java进程并不退出。

可被用户设定的singals

对于SIGSEGV(kill -11), SIGFPE(kill -8),SIGILL(kill -4), SIGUSR1(kill -10) 等信号,无法被用户设置:

scala> Signal.handle(new Signal("USR1"), 
 |              new SignalHandler(){ def handle(sig:Signal){ println("down") }}
 |      )

java.lang.IllegalArgumentException: Signal already used by VM or OS: SIGUSR1
    at sun.misc.Signal.handle(Signal.java:166)
    ... 32 elided

对于SIGINT(kill -2),SIGTERM(kill -15),SIGUSR2(kill -12),SIGBUS(kill -7),SIGPIPE(kill -13), 可以被用户设置:

scala> Signal.handle(new Signal("INT"), 
 |              new SignalHandler(){ def handle(sig:Signal){ println("down") }}
 |      )

$ kill -s INT `pidof java`

scala> down // 被捕获,进程不会退出

触发shutdown hook的singals

SIGTERM(kill), SIGINT(kill -2), SIGHUP(kill -1) 会触发shutdown hook的执行。

scala>  Runtime.getRuntime().addShutdownHook(
            new Thread() { override def run() { println("ok") } })

$ kill -s INT `pidof java`

scala> ok 

jvm与系统信号(1)

下面是singal部分列表(POSIX.1-1990):

SIGHUP 1 Terminal line hangup
SIGINT 2 Interrupt program
SIGQUIT 3 Quit program
SIGILL 4 Illegal instruction
SIGABRT 6 Abort
SIGFPE 8 Floating-point exception
SIGKILL 9 Kill program
SIGSEGV 11 Invalid memory reference
SIGPIPE 13 Write on a pipe with no one to read it
SIGALRM 14 Alarm clock
SIGTERM 15 Termination signal
SIGUSR1 30,10,16 User-defined signal 1
SIGUSR2 31,12,17 User-defined signal 2
SIGCHLD 20,17,18 Child stopped or terminated
SIGCONT 19,18,25 Continue if stopped
SIGSTOP 17,19,23 Stop process

其中有多个值的singal,它的含义是与硬件体系相关,参考这里:一般alpha和sparc架构用第一个值,x86,arm架构用中间值,mips架构用第三个值, – 表示相应架构的取值未知。

比如SIGUSR1这个信号,有30,10,16 三种值,在x86上会用第二个值,也就是10.

$ /bin/kill -L | xargs -n2 | grep USR1
10 USR1

后来在POSIX.1-2001有部分新增或修改:

SIGTRAP 5 Trace trap
SIGBUS 10,7,10 Bus error (bad memory access)
SIGSYS 12,31,12 Bad system call
SIGURG 16,23,21 Urgent condition on socket (4.2BSD)
SIGXCPU 24,24,30 CPU time limit exceeded (4.2BSD)

现在看一下jvm里对singal的处理,参考这里 (The mention “optional” means that the signal is not necessary when the -Xrs option is specified):

Signal Description
SIGSEGV, SIGBUS, SIGFPE,
SIGPIPE, SIGILL
Used in the implementation for implicit null check, and so forth.
SIGQUIT Thread dump support: To dump Java stack traces at the standard error stream. (Optional.)
SIGTERM, SIGINT, SIGHUP Used to support the shutdown hook mechanism ( java.lang.Runtime.addShutdownHook) when the VM is terminated abnormally. (Optional.)
SIGUSR1 Used in the implementation of the java.lang.Thread.interrupt method. (Configurable.) Not used starting with Solaris 10 OS. Reserved on Linux.
SIGUSR2 Used internally. (Configurable.) Not used starting with Solaris 10 OS.
SIGABRT The HotSpot VM does not handle this signal. Instead it calls the abort function after fatal error handling. If an application uses this signal then it should terminate the process to preserve the expected semantics.

类初始化与并发问题

这个案例是2009年中文站线上曾发生过的事情,实际情况比较复杂,简化后我们可以用下面的例子做demo

// 定义一个锁对象
class Lock {} 

// 在这个类的静态构造块里做一些事情
class Danger {
    static {
        System.out.println("clinit begin...");
        try {
            Thread.sleep(2000);
        } catch (Exception e) {
        }

        synchronized (Lock.class) { 
            System.out.println("clinit done!");
        }
    } 
}

public class Test {
    public static void main(String[] args) {
        // 启动新线程
        new Thread() { 
            public void run() {
                synchronized (Lock.class) { 
                    System.out.println("new thread start!");
                    try {
                        Thread.sleep(1000);
                    } catch (Exception e) {
                    } 
                    new Danger();
                }
                System.out.println("new thread end!"); 

            }
        }.start();

        // 当前线程sleep 500毫秒
        try {
            Thread.sleep(500);
        } catch (Exception e) {
        } 
        // 当前线程创建了Danger对象实例
        System.out.println(new Danger()); 
        // 会走到这里吗?
        System.out.println("DONE!");
    } 
}

运行上面的程序时,java进程挂在那儿不动了,我们通过jstack去观察的话,它先给出死锁检查是没有找到:

Deadlock Detection:
No deadlocks found.

然后看看两个线程的堆栈信息:

Thread 17643: (state = BLOCKED)
    - Test$1.run() @bci=24, line=28 (Interpreted frame)

Thread 17634: (state = BLOCKED)
    - Danger.<clinit>() @bci=24, line=12 (Interpreted frame)
    - Test.main(java.lang.String[]) @bci=23, line=41 (Interpreted frame)

虽然jvm没有把这两个线程看做是死锁,但实际情况是和死锁类似的。在jvm的实现规范里:

当多个线程需要初始化一个类,仅有一个线程会进⾏,其他线程需要等待。当活动的线程完成初始化之后, 它必须通知其他等待线程

所以这个被hang住的问题简单来说就是:线程1持有了锁(Lock.class对象),然后sleep了一会儿;与此同时线程2创建Danger对象,在创建对象的过程中先进行类初始化,在类初始化过程中又因为需要获取锁(Lock.class对象)而等待线程1释放(Lock.class对象);在线程1睡醒了之后,也去创建Danger对象,同样,因为Danger对象当前的状态是处于类初始化进行中的状态,线程1要等待正在初始化Danger类的那个线程要先把类初始化完成才能构造对象。于是造成了相互等待的境况。

在以前的分享的jvm内存管理的ppt里曾提到过这个例子,也可以参考一下这个ppt

同事周忱 曾用pstack看过到底hang在什么地方,参考他的gist,在ubuntu上pstack不太好用,可以直接使用jstack -m来看

----------------- 17643 -----------------
0x00007f897f250d84  __pthread_cond_wait + 0xc4
0x00007f897ea5ec85  _ZN13ObjectMonitor4waitElbP6Thread + 0x785
0x00007f897e865c1b  _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread + 0x31b
0x00007f897e865f19  _ZN13instanceKlass10initializeEP6Thread + 0x49
0x00007f897e8999d4  _ZN18InterpreterRuntime4_newEP10JavaThreadP19constantPoolOopDesci + 0x214
0x00007f897501d98a  * Test$1.run() bci:24 line:28 (Interpreted frame)
0x00007f89750004f7  <StubRoutines>
0x00007f897e8a67bf  _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x54f
0x00007f897e8a5802  _ZN9JavaCalls12call_virtualEP9JavaValue11KlassHandleP6SymbolS4_P17JavaCallArgumentsP6Thread + 0x1c2
0x00007f897e8a585f  _ZN9JavaCalls12call_virtualEP9JavaValue6Handle11KlassHandleP6SymbolS5_P6Thread + 0x4f
0x00007f897e8dcfc6  _ZL12thread_entryP10JavaThreadP6Thread + 0x86
0x00007f897eb8bfc0  _ZN10JavaThread17thread_main_innerEv + 0xc0
0x00007f897ea69ec2  _ZL10java_startP6Thread + 0x132

----------------- 17634 -----------------
0x00007f897f2510fe  __pthread_cond_timedwait + 0x13e
0x00007f897ea5d004  _ZN13ObjectMonitor6EnterIEP6Thread + 0x294
0x00007f897ea5de87  _ZN13ObjectMonitor5enterEP6Thread + 0x297
0x00007f897e89ced9  _ZN18InterpreterRuntime12monitorenterEP10JavaThreadP15BasicObjectLock + 0x99
0x00007f897501e299  * Danger.<clinit>() bci:24 line:12 (Interpreted frame)
0x00007f89750004f7  <StubRoutines>
0x00007f897e8a67bf  _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x54f
0x00007f897e8a6055  _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread + 0x25
0x00007f897e8658ac  _ZN13instanceKlass27call_class_initializer_implE19instanceKlassHandleP6Thread + 0xbc
0x00007f897e8658f5  _ZN13instanceKlass22call_class_initializerEP6Thread + 0x25
0x00007f897e865a9f  _ZN13instanceKlass15initialize_implE19instanceKlassHandleP6Thread + 0x19f
0x00007f897e865f19  _ZN13instanceKlass10initializeEP6Thread + 0x49
0x00007f897e8999d4  _ZN18InterpreterRuntime4_newEP10JavaThreadP19constantPoolOopDesci + 0x214
0x00007f897501d998  * Test.main(java.lang.String[]) bci:23 line:41 (Interpreted frame)
0x00007f89750004f7  <StubRoutines>
0x00007f897e8a67bf  _ZN9JavaCalls11call_helperEP9JavaValueP12methodHandleP17JavaCallArgumentsP6Thread + 0x54f
0x00007f897e8a6055  _ZN9JavaCalls4callEP9JavaValue12methodHandleP17JavaCallArgumentsP6Thread + 0x25
0x00007f897e8b27de  _ZL17jni_invoke_staticP7JNIEnv_P9JavaValueP8_jobject11JNICallTypeP10_jmethodIDP18JNI_ArgumentPusherP6Thread.isra.126.constprop.151 + 0x1ce
0x00007f897e8bae80  jni_CallStaticVoidMethod + 0x160
0x00007f897fa27f7b      ????????

效果不如pstack那么直观,但也可以发现instanceKlass::initialize_impl和ObjectSynchronizer::waitUninterruptibly方法,由此线索可再结合jvm代码进一步深究。