Java REST API 调用失败并出现异常 "Async operation timed out" 当负载超过特定大小时

Java REST API call fails with exception "Async operation timed out" when payload exceeds certain size

我正在一堆 Linux 服务器上设置自动 Cron 作业,这需要调用 IBM WebSphere Application Server(8.5.5) 提供的一些 PUT/POST APIs .我在 Java 中编写了自己的 REST 客户端,但奇怪的是有时 API 执行挂起并出现以下异常。

com.ibm.ws.webcontainer.webapp.WebApp logServletError SRVE0293E: [Servlet Error]-[JAX-RS Servlet]: java.net.SocketTimeoutException: Async operation timed out
    at com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:191)
    at com.ibm.ws.tcp.channel.impl.TCPReadRequestContextImpl.read(TCPReadRequestContextImpl.java:111)
    at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.fillABuffer(HttpServiceContextImpl.java:4212)
    at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.readSingleBlock(HttpServiceContextImpl.java:3440)
    at com.ibm.ws.http.channel.impl.HttpServiceContextImpl.readBodyBuffer(HttpServiceContextImpl.java:3546)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundServiceContextImpl.getRequestBodyBuffer(HttpInboundServiceContextImpl.java:1822)
    at com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream.bufferIsGood(WCCByteBufferInputStream.java:371)
    at com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream.read(WCCByteBufferInputStream.java:265)
    at com.ibm.ws.webcontainer.srt.http.HttpInputStream.read(HttpInputStream.java:322)
    at sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:464)
    at sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:506)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:234)
    at java.io.InputStreamReader.read(InputStreamReader.java:188)
    at com.ibm.json.java.JSONArray.parse(JSONArray.java:150)
    at com.ibm.websphere.jaxrs.providers.json4j.JSON4JArrayProvider.readFrom(JSON4JArrayProvider.java:44)
    at com.ibm.websphere.jaxrs.providers.json4j.JSON4JArrayProvider.readFrom(JSON4JArrayProvider.java:25)
    at org.apache.wink.server.internal.registry.ServerInjectableFactory$EntityParam.getValue(ServerInjectableFactory.java:197)
    at org.apache.wink.common.internal.registry.InjectableFactory.instantiate(InjectableFactory.java:68)
    at org.apache.wink.server.internal.handlers.CreateInvocationParametersHandler.handleRequest(CreateInvocationParametersHandler.java:44)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleSubResourceMethod(FindResourceMethodHandler.java:188)
    at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleRequest(FindResourceMethodHandler.java:110)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.internal.handlers.FindRootResourceHandler.handleRequest(FindRootResourceHandler.java:95)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.internal.handlers.HeadMethodHandler.handleRequest(HeadMethodHandler.java:53)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.internal.handlers.OptionsMethodWADLHandler.handleRequest(OptionsMethodWADLHandler.java:51)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.internal.handlers.SearchResultHandler.handleRequest(SearchResultHandler.java:33)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.internal.log.ResourceInvocation.handleRequest(ResourceInvocation.java:92)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.internal.log.Requests.handleRequest(Requests.java:76)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)
    at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)
    at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:75)
    at org.apache.wink.server.handlers.AbstractHandlersChain.run(AbstractHandlersChain.java:60)
    at org.apache.wink.server.internal.RequestProcessor.handleRequestWithoutFaultBarrier(RequestProcessor.java:207)
    at org.apache.wink.server.internal.RequestProcessor.handleRequest(RequestProcessor.java:154)
    at org.apache.wink.server.internal.servlet.RestServlet.service(RestServlet.java:124)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
    at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
    at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
    at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1071)
    at com.ibm.ws.webcontainer.servlet.CacheServletWrapper.handleRequest(CacheServletWrapper.java:87)
    at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:909)
    at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
    at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:459)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:526)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:312)
    at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:283)
    at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
    at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
    at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
    at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
    at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
    at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
    at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
    at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
    at com.ibm.io.async.ResultHandler.run(ResultHandler.java:905)
    at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1862)
Caused by: com.ibm.io.async.AsyncTimeoutException(Async operation timed out, [Timeout, rc=0])
    at com.ibm.io.async.AbstractAsyncFuture.waitForCompletion(AbstractAsyncFuture.java:359)
    at com.ibm.io.async.AsyncFuture.getByteCount(AsyncFuture.java:218)
    at com.ibm.ws.tcp.channel.impl.AioSocketIOChannel.readAIOSync(AioSocketIOChannel.java:215)
    at com.ibm.ws.tcp.channel.impl.AioTCPReadRequestContextImpl.processSyncReadRequest(AioTCPReadRequestContextImpl.java:184)
    ... 78 more

这只发生在一些 Linux 服务器上,当我在我的 Windows 工作站上开发它时它工作得很好。因此,WAS 服务器中的超时设置应该不是问题。我尝试使用 curl 进行调试,如下所示

curl -vX PUT http://<was_server_ip>:9081/RTCWebClient/v0.1/api/testResult/syncTestSuites?testCategory=Kitchen -d @test.json --header "Content-Type: application/json"

发现当有效负载超过 1306 字节时(请参阅下面 curl 输出中的 Content-Length),API 开始挂起并给出上述异常。

> PUT /RTCWebClient/v0.1/api/testResult/syncTestSuites?testCategory=Kitchen
HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 9.51.163.190:9081
> Accept: */*
> Content-Type: application/json
> Content-Length: 1307
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< Content-Length: 0
< Date: Wed, 24 Aug 2016 07:35:23 GMT
< HTTP/1.1 500 Internal Server Error
< Date: Wed, 24 Aug 2016 07:35:23 GMT
< X-Powered-By: Servlet/3.0
< Content-Type: text/html;charset=ISO-8859-1
< $WSEP:
< Content-Language: en-US
< Content-Length: 102
< Connection: Close
<
Error 500: javax.servlet.ServletException: java.net.SocketTimeoutException: Async operation timed out
* Closing connection #0

当有效负载小于 1307 字节时,API 会正常执行。

> PUT /RTCWebClient/v0.1/api/testResult/syncTestSuites?testCategory=Kitchen HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: 9.51.163.190:9081
> Accept: */*
> Content-Type: application/json
> Content-Length: 1306
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< Content-Length: 0
< Date: Wed, 24 Aug 2016 07:37:51 GMT
< HTTP/1.1 200 OK
< Date: Wed, 24 Aug 2016 07:37:51 GMT
< X-Powered-By: Servlet/3.0
< Content-Type: application/json
< Access-Control-Allow-Headers: application/json
< Access-Control-Allow-Methods: GET, POST, PUT, DELETE
< Access-Control-Allow-Origin: *
< Content-Language: en-US
< Transfer-Encoding: chunked
<
* Connection #0 to host 9.51.163.190 left intact
* Closing connection #0
{"Identifier":"syncTestSuites completed","Result":{"Newly Added Test Suites":[]}}

我需要与他们的管理员核实哪些设置可能会在那些有问题的 Linux 机器上导致此问题。但我也将它张贴在这里,看看是否有人可以就这个问题提供一些深思熟虑的见解。

这个问题早就解决了,但是忘记在这里更新了。

将WAS应用服务器的MTU大小从1500位降低到1320位后,问题就解决了。根据网络管理员的说法,这使得服务器的数据段变小了一点,让一些应用程序可以更好地处理数据。

这里有一篇关于最大传输单元的非常好的博客文章https://craigocon.wordpress.com/2012/10/03/whats-mtu-and-how-do-i-change-it-in-windows-2008-r2/

我对网络知识完全陌生,很高兴得到我们网络团队的支持。