标签归档:diagnose

scala的诊断方法(4) -Ytyper-debug 编译项

在定位之前的那个问题时,发现了-Ytyper-debug选项,可以吧编译器在typer阶段的一些信息打印出来,比-Xprint:typer多出一些类型推导的信息:

$ cat A.scala
object A {
    println(1,2,3)
}

$ scalac -Ytyper-debug  A.scala
|-- <empty> EXPRmode-POLYmode-QUALmode (site: package <root>)
|    \-> <empty>.type
|-- object A BYVALmode-EXPRmode (site: package <empty>)
|    |-- super EXPRmode-POLYmode-QUALmode (silent: <init> in A)
|    |    |-- this EXPRmode (silent: <init> in A)
|    |    |    \-> A.type
|    |    \-> A.type
|    |-- println(1, 2, 3) BYVALmode-EXPRmode (site: value <local A> in A)
|    |    |-- println BYVALmode-EXPRmode-FUNmode-POLYmode (silent: value <local A> in A)
|    |    |    \-> (x: Any)Unit <and> ()Unit
|    |    |-- scala.Tuple3(1, 2, 3) : pt=Any BYVALmode-EXPRmode (silent: value <local A> in A)
|    |    |    |-- scala.Tuple3 BYVALmode-EXPRmode-FUNmode-POLYmode (silent: value <local A> in A)
|    |    |    |    |-- scala.Tuple3.apply BYVALmode-EXPRmode-FUNmode-POLYmode (silent: value <local A> in A)
|    |    |    |    |    [adapt] [T1, T2, T3](_1: T1, _2: T2, _3: T3)(T1, T2, T3) adapted to [T1, T2, T3](_1: T1, _2: T2, _3: T3)(T1, T2, T3)
|    |    |    |    |    \-> (_1: T1, _2: T2, _3: T3)(T1, T2, T3)
|    |    |    |    [adapt] Tuple3.type adapted to [T1, T2, T3](_1: T1, _2: T2, _3: T3)(T1, T2, T3)
|    |    |    |    \-> (_1: T1, _2: T2, _3: T3)(T1, T2, T3)
|    |    |    |-- 1 BYVALmode-EXPRmode-POLYmode (silent: value <local A> in A)
|    |    |    |    \-> Int(1)
|    |    |    |-- 2 BYVALmode-EXPRmode-POLYmode (silent: value <local A> in A)
|    |    |    |    \-> Int(2)
|    |    |    |-- 3 BYVALmode-EXPRmode-POLYmode (silent: value <local A> in A)
|    |    |    |    \-> Int(3)
|    |    |    solving for (T1: ?T1, T2: ?T2, T3: ?T3)
|    |    |    \-> (Int, Int, Int)
|    |    \-> Unit
|    \-> [object A] A.type
|-- (_1: <?>, _2: <?>, _3: <?>)(T1, T2, T3) EXPRmode-FUNmode-POLYmode-TAPPmode (site: object A) implicits disabled
|    |-- new (Int, Int, Int) EXPRmode-POLYmode-QUALmode (site: object A) implicits disabled
|    |    \-> (Int, Int, Int)
|    \-> (_1: Int, _2: Int, _3: Int)(Int, Int, Int)

在解析的时候,对println(1,2,3)里的参数执行了Tuple3(1,2,3)最终参数识别成了Tuple3[Int,Int,Int]类型。暂没发现更大的价值,先记下来以后再补充。

检测最耗cpu的线程的脚本

这个脚本用于定位出当前java进程里最耗cpu的那个线程,给出cpu的占用率和当前堆栈信息。这个脚本仅限于linux上,我没有找到在mac下定位线程使用cpu情况的工具,如果你知道请告诉我一下。

先模拟一个耗cpu的java进程,启动一个scala的repl并在上面跑一段死循环:

scala> while(true) {}

脚本执行效果:

$ ./busythread.sh `pidof java`
tid: 431  cpu: %98.8
"main" prio=10 tid=0x00007f777800a000 nid=0x1af runnable [0x00007f7781c2e000]
    java.lang.Thread.State: RUNNABLE
    at $line3.$read$$iw$$iw$.<init>(<console>:8)
    at $line3.$read$$iw$$iw$.<clinit>(<console>)
    at $line3.$eval$.$print$lzycompute(<console>:7)
    - locked <0x00000000fc201758> (a $line3.$eval$)
    at $line3.$eval$.$print(<console>:6)
    at $line3.$eval.$print(<console>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:739)
    at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:986)
    at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:593)
    at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:592)
    at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
    at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
    at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:592)
    at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:524)
    at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:520)

脚本内容:

#!/bin/bash

if [ $# -eq 0 ];then
    echo "please enter java pid"
    exit -1
fi

pid=$1
jstack_cmd=""

if [[ $JAVA_HOME != "" ]]; then
    jstack_cmd="$JAVA_HOME/bin/jstack"
else
    r=`which jstack 2>/dev/null`
    if [[ $r != "" ]]; then
        jstack_cmd=$r
    else
        echo "can not find jstack"
        exit -2
    fi
fi

#line=`top -H  -o %CPU -b -n 1  -p $pid | sed '1,/^$/d' | grep -v $pid | awk 'NR==2'`

line=`top -H -b -n 1 -p $pid | sed '1,/^$/d' | sed '1d;/^$/d' | grep -v $pid | sort -nrk9 | head -1`
echo "$line" | awk '{print "tid: "$1," cpu: %"$9}'
tid_0x=`printf "%0x" $(echo "$line" | awk '{print $1}')`
$jstack_cmd $pid | grep $tid_0x -A20 | sed -n '1,/^$/p'

脚本已放到服务器上,可以通过下面的方式执行:

$ bash <(curl -s http://hongjiang.info/busythread.sh)  java_pid

update: 感谢容若的反馈,很多环境的procps版本较低,top还不支持-o参数,排序那块用sort解决了,脚本已更新。

lsof查看进程在使用的已删除的文件

有时会遇到这种情况,当一个进程正在向一个文件写数据时,该文件的目录可能被移动,或该文件已被其他进程删除。
lsof +L1 可以查看那些在访问的已被删除的文件

lsof +L1 shows you all open files that have a link count less than 1, often indicative of a cracker trying to hide something

hongjiang@whj ~ % sudo lsof +L1
COMMAND   PID  USER   FD   TYPE DEVICE SIZE/OFF NLINK    NODE NAME
mysqld  22070 mysql    4u   REG  202,1        0     0 1048935 /tmp/ibfwFj0I (deleted)
mysqld  22070 mysql    5u   REG  202,1        0     0 1048937 /tmp/ibpUnKvR (deleted)
mysqld  22070 mysql    6u   REG  202,1        0     0 1048942 /tmp/ibXYbb1Z (deleted)

这个参数不好记,也可以用 lsof -n | grep deleted 来查看。

scala的诊断方法(3) 在repl下统计方法的执行时间

为了方便启动repl时自动加载,把这段函数定义放在.scalarc文件中:

hongjiang@whj-mbp ~ % cat .scalarc

def time[T](code : => T) =  {
    val t0 = System.nanoTime : Double
    val res = code
    val t1 = System.nanoTime : Double
    println("Elapsed time " + (t1 - t0) / 1000000.0 + " msecs")
    res
}

把scala命令alias一下,每次启动时自动load启动脚本:

alias scala='scala -deprecation -feature -i ~/.scalarc'

用time方法测试执行时间

// 测试代码块的执行时间
scala> time{ var sum=0; for(i<-0 to 1000000) sum=sum+i; println(sum) }
    1784293664
    Elapsed time 71.57888 msecs

scala> def fib(n:Int):Int = if (n<2)  n else fib(n-1) + fib(n-2);

// 测试方法的执行时间  
scala> time( fib(40) )
    Elapsed time 407.36 msecs
    res3: Int = 102334155

scala的诊断方法(2) 在repl下用reify查看表达式的翻译结果

之前介绍的-Xprint:typer-Xprint:jvm 方式,用于编译或执行脚本时。
当在repl下也想要查看类似效果时,可以用 universe.reify 方法

scala> import reflect.runtime.universe._

scala> reify( for(i <- 1 to 10) println(i) )
res0: reflect.runtime.universe.Expr[Unit] = Expr[Unit](scala.this.Predef.intWrapper(1).to(10).foreach(((i) => scala.this.Predef.println(i))))

scala> show(res0)
res9: String = Expr[Unit](scala.this.Predef.intWrapper(1).to(10).foreach(((i) => scala.this.Predef.println(i))))

scala> show(res0.tree)
res10: String = scala.this.Predef.intWrapper(1).to(10).foreach(((i) => scala.this.Predef.println(i)))

universe这个对象定义在 scala.reflect 包下的 runtime 包对象(package object)中:
是个 lazy val,类型是 JavaUniverse,继承了scala.reflect.Universe抽象类里的 reify 方法。
reify方法用于对一段表达式参数生成语法树。

这对定位隐式转换有帮助,比如我们不确定一个类型在哪儿被隐式转换了:

scala> 1.min(3) //min在哪儿定义的?
res0: Int = 1

scala> reify(1.min(3)).tree
res7: reflect.runtime.universe.Tree = Predef.intWrapper(1).min(3)

scala的诊断方法(1) 使用-Xprint:typer看语法糖的背后

scala中很多看似花哨的语法表达,其实背后并不是语言级别的支持,都是些语法糖。当我们拿不准一些语句最终被翻译为什么,可以通过一些编译时参数来帮助我们分析。
先看一下scalac编译过程的各个阶段(phases):

$ scalac -Xshow-phases

         phase name  id  description
         ----------  --  -----------
             parser   1  parse source into ASTs, perform simple desugaring
              namer   2  resolve names, attach symbols to named trees
     packageobjects   3  load package objects
              typer   4  the meat and potatoes: type the trees
             patmat   5  translate match expressions
     superaccessors   6  add super accessors in traits and nested classes
         extmethods   7  add extension methods for inline classes
            pickler   8  serialize symbol tables
          refchecks   9  reference/override checking, translate nested objects
       selectiveanf  10
       selectivecps  11
            uncurry  12  uncurry, translate function values to anonymous classes
          tailcalls  13  replace tail calls by jumps
         specialize  14  @specialized-driven class and method specialization
      explicitouter  15  this refs to outer pointers, translate patterns
            erasure  16  erase types, add interfaces for traits
        posterasure  17  clean up erased inline classes
           lazyvals  18  allocate bitmaps, translate lazy vals into lazified defs
         lambdalift  19  move nested functions to top level
       constructors  20  move field definitions into constructors
            flatten  21  eliminate inner classes
              mixin  22  mixin composition
            cleanup  23  platform-specific cleanups, generate reflective calls
              icode  24  generate portable intermediate code
            inliner  25  optimization: do inlining
inlineExceptionHandlers  26  optimization: inline exception handlers
           closelim  27  optimization: eliminate uncalled closures
                dce  28  optimization: eliminate dead code
                jvm  29  generate JVM bytecode
           terminal  30  The last phase in the compiler chain

第一个parser阶段就已经做了部分简单的“脱糖”处理来去除语法糖,不过对我们最有帮助的还是 typer 和 jvm 两个阶段。一般我们可以通过 -Xprint:typer 来看一些语法糖表达式的背后:

$ scalac -Xprint:typer A.scala

或者直接以脚本的方式:

$ scala -Xprint:typer -e 'def incr(i:Int) = i+1'

例子,看一下for表达式的背后:

$ scala -Xprint:typer -e 'val l=List(1,2,3); for(i<-l) {println(i)}'
    ... 
    private[this] val l: List[Int] = immutable.this.List.apply[Int](1, 2, 3);
    <stable> <accessor> private def l: List[Int] = $anon.this.l;
    $anon.this.l.foreach[Unit](((i: Int) => scala.this.Predef.println(i)))
    ...

可见在上面的例子里是被翻译为了foreach操作;对于for表达式,我们后续会专门分析。