Hi,

Service center is unresponsive for 30 - 40 mins in the morning time. I am getting proxy error on time out when i hit the login page.

This is on-premise installation of Outsystems 10 platform

I could see the CheckSystemStautus was running for 50m and 22m and timed out and next try completed in 2m.

Application was working fine without any issue it was only ServiceCenter application which was unresponsive.

What is this timer and what could be the cause of it for running so long?

Checked the memory and CPU utilization on the servers and they were healthy.


Below is what we see in logs


Cleaning the connection pools because: IO Error: Connection timed out (Read failed)

at outsystems.hubedition.databaseprovider.oracle.transactionservice.TransactionService.createConnection()

at outsystems.hubedition.extensibility.data.transactionservice.BaseTransactionService.createConnection()

at outsystems.hubedition.extensibility.data.transactionservice.GenericTransactionManager.getConnection()

at outsystems.hubedition.extensibility.data.transactionservice.GenericTransactionManager.buildTransactionInfo()

at outsystems.hubedition.extensibility.data.transactionservice.GenericTransactionManager.getRequestTransaction()

at outsystems.hubedition.extensibility.data.transactionservice.GenericTransactionManager.commitAllTransactions()

at outsystems.internal.db.DatabaseAccessProvider.commitAllTransactions()

at outsystems.internal.db.DatabaseAccess.$jcc()

at outsystems.internal.db.DatabaseAccess.commitAllTransactions()


... 41 more

Caused by: java.net.SocketException: Broken pipe (Write failed)

at java.net.SocketOutputStream.socketWrite0(Native Method) [rt.jar:1.8.0_221]

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) [rt.jar:1.8.0_221]

at java.net.SocketOutputStream.write(SocketOutputStream.java:155) [rt.jar:1.8.0_221]

at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:706) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:730) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:543) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.coyote.Response.doWrite(Response.java:594) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:424) [jbossweb.jar:7.5.30.Final-redhat-1]

... 52 more

2020-01-08:08:01:57,384 WARNING [org.apache.cxf.phase.PhaseInterceptorChain] (http-0.0.0.0:8080-1) Interceptor for {http://tempuri.org/}ActivityHandler has thrown exception, unwinding now: org.apache.cxf.interceptor.Fault: Could not send Message.

at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64) [cxf-api.jar:2.7.18.SP7-redhat-1]

at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272) [cxf-api.jar:2.7.18.SP7-redhat-1]

at org.apache.cxf.interceptor.AbstractFaultChainInitiatorObserver.onMessage(AbstractFaultChainInitiatorObserver.java:113) [cxf-api.jar:2.7.18.SP7-redhat-1]

at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:336) [cxf-api.jar:2.7.18.SP7-redhat-1]

at org.apache.cxf.interceptor.OutgoingChainInterceptor.handleMessage(OutgoingChainInterceptor.java:83) [cxf-api.jar:2.7.18.SP7-redhat-1]

at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272) [cxf-api.jar:2.7.18.SP7-redhat-1]

at org.apache.cxf.phase.PhaseInterceptorChain.doInterceptStartingAt(PhaseInterceptorChain.java:424) [cxf-api.jar:2.7.18.SP7-redhat-1]

at org.apache.cxf.frontend.WSDLGetInterceptor.handleMessage(WSDLGetInterceptor.java:97) [cxf-rt-frontend-simple.jar:2.7.18.SP7-redhat-1]

at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272) [cxf-api.jar:2.7.18.SP7-redhat-1]

at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-api.jar:2.7.18.SP7-redhat-1]

at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:249) [cxf-rt-transports-http.jar:2.7.18.SP7-redhat-1]

at org.jboss.wsf.stack.cxf.RequestHandlerImpl.handleHttpRequest(RequestHandlerImpl.java:97)

at org.jboss.wsf.stack.cxf.transport.ServletHelper.callRequestHandler(ServletHelper.java:131)

at org.jboss.wsf.stack.cxf.CXFServletExt.invoke(CXFServletExt.java:88)

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:289) [cxf-rt-transports-http.jar:2.7.18.SP7-redhat-1]

at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doGet(AbstractHTTPServlet.java:214) [cxf-rt-transports-http.jar:2.7.18.SP7-redhat-1]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:734) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-2]

at org.jboss.wsf.stack.cxf.CXFServletExt.service(CXFServletExt.java:136)

at org.jboss.wsf.spi.deployment.WSFServlet.service(WSFServlet.java:140) [jbossws-spi.jar:2.3.1.Final-redhat-1]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-2]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb.jar:7.5.30.Final-redhat-1]

at outsystems.hubedition.webwidgets.BaseRequestStartupFilter.doFilter(Unknown Source) [outsystems.hubedition.webwidgets.jar:]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb.jar:7.5.30.Final-redhat-1]

at outsystems.hubedition.webwidgets.filters.HttpContextFilter.doFilter(Unknown Source) [outsystems.hubedition.webwidgets.jar:]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb.jar:7.5.30.Final-redhat-1]

at outsystems.hubedition.webwidgets.filters.CacheControlFilter.doFilter(Unknown Source) [outsystems.hubedition.webwidgets.jar:]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.jboss.as.web.security.SubjectInfoSetupValve.invoke(SubjectInfoSetupValve.java:34) [jboss-as-web.jar:7.5.22.Final-redhat-1]

at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) [jboss-as-web.jar:7.5.22.Final-redhat-1]

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:151) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:560) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:656) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926) [jbossweb.jar:7.5.30.Final-redhat-1]

at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_221]

Caused by: ClientAbortException: java.net.SocketException: Broken pipe (Write failed)

at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:429) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:468) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:441) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:90) [jbossweb.jar:7.5.30.Final-redhat-1]

at java.io.ByteArrayOutputStream.writeTo(ByteArrayOutputStream.java:167) [rt.jar:1.8.0_221]

at outsystems.hubedition.http.iIIiiiiIIiII.$wld(Unknown Source) [outsystems.hubedition.runtimeplatform.jar:]

at outsystems.hubedition.http.iIIiiiiIIiII.close(Unknown Source) [outsystems.hubedition.runtimeplatform.jar:]

at org.apache.cxf.transport.http.AbstractHTTPDestination$WrappedOutputStream.close(AbstractHTTPDestination.java:762) [cxf-rt-transports-http.jar:2.7.18.SP7-redhat-1]

at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56) [cxf-api.jar:2.7.18.SP7-redhat-1]

at org.apache.cxf.transport.http.AbstractHTTPDestination$BackChannelConduit.close(AbstractHTTPDestination.java:692) [cxf-rt-transports-http.jar:2.7.18.SP7-redhat-1]

at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62) [cxf-api.jar:2.7.18.SP7-redhat-1]

... 43 more

Caused by: java.net.SocketException: Broken pipe (Write failed)

at java.net.SocketOutputStream.socketWrite0(Native Method) [rt.jar:1.8.0_221]

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) [rt.jar:1.8.0_221]

at java.net.SocketOutputStream.write(SocketOutputStream.java:155) [rt.jar:1.8.0_221]

at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:706) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:450) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:351) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:730) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:124) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:543) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.coyote.Response.doWrite(Response.java:594) [jbossweb.jar:7.5.30.Final-redhat-1]

at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:424) [jbossweb.jar:7.5.30.Final-redhat-1]

... 55 more

Hi Prabhu Gnanasekar,

Better you connect to Outsystems Support Team.

https://www.outsystems.com/SupportPortal/CaseOpen/

thanks