标签归档:keep-alive

nginx与tomcat之间的keep-alive配置

今天碰到的一个情况,tomcat与前端nginx之间的存在大量的TIME_WAIT状态的连接,第一反应是这里可能没有配置keep-alive。问ops,回复说启用了;要来nginx的配置看了一下,发现upstream里设置了keepalive参数:

upstream  tomcat {
    server x.x.x.x:8080;
    ...
    keepalive 16;
}

不确定这个参数是不是http的keep-alive,在nginx的网站上找了一下

Syntax: keepalive connections;
Default:    —
Context:    upstream

The connections parameter sets the maximum number of idle keepalive connections to upstream servers that are preserved in the cache of each worker process.

它并不是与后端节点开启http-alive方式的意思,nginx作为反向代理后端并不局限http协议,这里的keepalive设置相当于每个worker连接池的最大空闲keepalive连接数,跟http里的keep-alive不是一回事。

在官方文档里明确要对后端节点使用http keep-alive 可以指定http版本为1.1:

location /http/ {
    proxy_pass http://http_backend;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    ...
}

或者仍使用http 1.0协议,但显式的设置http header里Connection参数为Keep-Alive,这是因为keep-alive是http1.1的默认特性,在1.0里最初并未实现是后来从1.1里backport到1.0的,需要显式设定这个参数才启用:

location /http/ {
    proxy_pass http://http_backend;
    proxy_http_version 1.0;
    proxy_set_header Connection "Keep-Alive";
    ...
}

为了确认有效性,可以对tomcat的logger增加一句

org.apache.coyote.http11.level = FINE

这样可以在tomcat日志里看到每个请求的http header信息:

FINE: Received [GET / HTTP/1.0
Connection: Keep-Alive
Host: localhost:8080
User-Agent: curl/7.37.1
Accept: */*

确实在header里增加了。建议还是配置为http 1.1协议,支持chunked等特性。

ps, keepalive 这个词可能是指连接池或者tcp层面的参数,要看上下文。在http里是keep-alive,注意中间有个连字符。

tomcat-connector的微调(5): keep-alive相关的参数

tomcat默认是开启keep-alive的,有3个相关的参数可以配置:

1) keepAliveTimeout

表示在复用一个连接时,两次请求之间的最大间隔时间;超过这个间隔服务器会主动关闭连接。默认值同connectionTimeout参数,即20秒。不做限制的话可以设置为-1.

2) maxKeepAliveRequests

表示一个连接最多可复用多少次请求,默认是100。不做限制可以设置为-1. 注意如果tomcat是直接对外服务的话,keep-alive特性可能被一些DoS攻击利用,比如以很慢的方式发生数据,可以长时间持有这个连接;从而可能被恶意请求耗掉大量连接拒绝服务。tomcat直接对外的话这个值不宜设置的太大。

3) disableKeepAlivePercentage

注意,这个参数是BIO特有的,默认情况BIO在线程池里的线程使用率超过75%时会取消keep-alive,如果不取消的话可以设置为100.

模拟tomcat bio模式下线程池利用率超过75%关闭keep-alive的情况

模拟一下在BIO模式下,当线程利用率超过75%时,将自动关闭keep-alive的场景。

通过curl命令来观察,默认情况下curl会开启keep-alive选项,不过注意curl复用socket的话是在同一进程内多次访问目标同一地址时才会复用,两次执行curl的话并不会复用,比如:

$ curl http://localhost:7001/main
$ curl http://localhost:7001/main

上面连续执行curl命令并不会复用socket,socket会随着进程的消失而关闭,下次新的进程会重新创建连接。可以通过tcpdump观察,上面两次连接是不同的socket:

$ sudo tcpdump -l -i lo0 port 7001

23:43:19.236948 IP6 localhost.62625 > localhost.afs3-callback
......
23:43:26.071504 IP6 localhost.62626 > localhost.afs3-callback
......

在同一个curl进程里多次访问同一地址的话,会复用socket,通过-v参数就可以观察到:

$ curl -v  http://localhost:7001/main  http://localhost:7001/main
* Hostname was NOT found in DNS cache
*   Trying ::1...
* Connected to localhost (::1) port 7001 (#0)
> GET /main HTTP/1.1
> User-Agent: curl/7.37.1
> Host: localhost:7001
> Accept: */*
>
< HTTP/1.1 200 OK
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< Transfer-Encoding: chunked
< Date: Mon, 18 Aug 2014 15:49:28 GMT
<
* Connection #0 to host localhost left intact
ok
* Found bundle for host localhost: 0x7fa7d8c08c50
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (::1) port 7001 (#0)
> GET /main HTTP/1.1
> User-Agent: curl/7.37.1
> Host: localhost:7001
> Accept: */*
>
< HTTP/1.1 200 OK
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< Transfer-Encoding: chunked
< Date: Mon, 18 Aug 2014 15:49:28 GMT
<
* Connection #0 to host localhost left intact
ok

注意,在第二次请求开头有一句:Re-using existing connection! 表明复用了上次的socket,使用tcpdump也会看到确实是同一个socket端口连的tomcat。

默认情况下,线程池的最大线程数是200个,BIO模式下当线程利用率超过75%的时候,server会对新来的连接不再使用keep-alive。我们先模拟建立151个连接(默认开启keep-alive的):

for i in {1..151}; do 
    ( 
        {echo -ne "GET /main HTTP/1.1\nhost: localhost:7001\n\n"; sleep 20} 
        | telnet localhost 7001
    )&;  
done

上面的zsh脚本模拟了151个连接(脚本里for循环里使用后台子进程方式启动模拟任务,通过jobs命令也可查看到),每次新建立socket并在服务器端响应后保持连接20秒(这也是服务器端默认keep-alive的超时时间)。tomcat对这151个连接保持keep-alive,BIO模式下会有151个线程耗在上面,即使socket上请求已处理完,后续没有新的请求也不会让出线程,而一直阻塞在上面。这时刚好达到了 151/200 ≈ 0.75 的临界值,那么后续建立的socket将不能再享用keep-alive

现在在这个临界值上,再执行curl命令模拟新的连接:

$  curl -v  http://localhost:7001/main http://localhost:7001/main

* Hostname was NOT found in DNS cache
*   Trying ::1...
* Connected to localhost (::1) port 7001 (#0)
> GET /main HTTP/1.1
> User-Agent: curl/7.37.1
> Host: localhost:7001
> Accept: */*
>
< HTTP/1.1 200 OK
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< Transfer-Encoding: chunked
< Date: Mon, 18 Aug 2014 15:22:50 GMT
< Connection: close
<
* Closing connection 0
ok
* Hostname was found in DNS cache
*   Trying ::1...
* Connected to localhost (::1) port 7001 (#1)
> GET /main HTTP/1.1
> User-Agent: curl/7.37.1
> Host: localhost:7001
> Accept: */*
>
< HTTP/1.1 200 OK
* Server Apache-Coyote/1.1 is not blacklisted
< Server: Apache-Coyote/1.1
< Transfer-Encoding: chunked
< Date: Mon, 18 Aug 2014 15:22:50 GMT
< Connection: close
<
* Closing connection 1
ok

注意,这次连接有2次请求,但看不到Re-using existing connection! 关键字,每次请求结束,服务器都显式的关闭了连接,即在header里看到的:Connection: close字段。表明超过75%之后,新建立的连接都不会再使用keep-alive。