20
Views
1
Comments
 JBOSS. 100% util all cpu
Question

Dear all.

We have O10.0.1005.2 JAVA EE. JBOSS 6.4.6. Red Hat Linux 7.3.

Please help us find an anomaly in the server behavior.

Periodically, once a week, one cluster server goes to 100% CPU load.

All 16 CPU`s have 100% CPU util.

The server itself is responding and judging by the logs has no anomalies. But the number of active JAVA threads in Linux exceeds all reasonable limits.

The fix for this is to restart the server. 

I would like to understand what is wrong with JBOSS?

We removed the full stack trace from JBOSS threads and saw locks everywhere.

There are more of these errors:

"http-/0.0.0.0:8080-93" #5947 daemon prio=5 os_prio=0 tid=0x00007f7b6c32e800 nid=0x6538 runnable [0x00007f7932610000]

   java.lang.Thread.State: RUNNABLE

at java.util.WeakHashMap.get(WeakHashMap.java:403)

at javax.faces.component.UIComponentBase.populateDescriptorsMapIfNecessary(Unknown Source)

at javax.faces.component.UIComponentBase.<init>(Unknown Source)

at outsystems.hubedition.webwidgets.uicomponent.AbstractWebComponent.<init>(Unknown Source)

at outsystems.hubedition.webwidgets.uicomponent.os_controls.IfComponent.<init>(Unknown Source)

at sun.reflect.GeneratedConstructorAccessor3425.newInstance(Unknown Source)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:422)

at java.lang.Class.newInstance(Class.java:442)

at com.sun.faces.application.ApplicationImpl.newThing(Unknown Source)

at com.sun.faces.application.ApplicationImpl.createComponent(Unknown Source)

at javax.faces.webapp.UIComponentELTag.createComponent(Unknown Source)

at javax.faces.webapp.UIComponentClassicTagBase.createChild(Unknown Source)

at javax.faces.webapp.UIComponentClassicTagBase.findComponent(Unknown Source)

at javax.faces.webapp.UIComponentClassicTagBase.doStartTag(Unknown Source)

at org.apache.jsp.AllTasks_jsp._jspx_meth_os_005fif_005f0(AllTasks_jsp.java:1678)

at org.apache.jsp.AllTasks_jsp._jspx_meth_os_005fcontainer_005f10(AllTasks_jsp.java:1621)

at org.apache.jsp.AllTasks_jsp._jspx_meth_os_005fcontainer_005f9(AllTasks_jsp.java:1566)

at org.apache.jsp.AllTasks_jsp._jspx_meth_os_005fwebBlockInstance_005f3(AllTasks_jsp.java:1503)

at org.apache.jsp.AllTasks_jsp._jspx_meth_os_005fcontainer_005f8(AllTasks_jsp.java:1452)

at org.apache.jsp.AllTasks_jsp._jspx_meth_os_005fcontainer_005f2(AllTasks_jsp.java:1121)

at org.apache.jsp.AllTasks_jsp._jspx_meth_os_005fcontainer_005f1(AllTasks_jsp.java:1072)

at org.apache.jsp.AllTasks_jsp._jspx_meth_os_005fcontainer_005f0(AllTasks_jsp.java:1031)

at org.apache.jsp.AllTasks_jsp._jspx_meth_os_005fwebBlockInstance_005f2(AllTasks_jsp.java:990)

at org.apache.jsp.AllTasks_jsp._jspx_meth_os_005fwebBlockInstance_005f1(AllTasks_jsp.java:939)

at org.apache.jsp.AllTasks_jsp._jspx_meth_os_005fplaceholderArgument_005f0(AllTasks_jsp.java:892)

at org.apache.jsp.AllTasks_jsp._jspx_meth_os_005fwebBlockInstance_005f0(AllTasks_jsp.java:841)

at org.apache.jsp.AllTasks_jsp._jspx_meth_h_005fform_005f0(AllTasks_jsp.java:739)

at org.apache.jsp.AllTasks_jsp._jspx_meth_os_005fpage_005f0(AllTasks_jsp.java:431)

at org.apache.jsp.AllTasks_jsp._jspx_meth_f_005fview_005f0(AllTasks_jsp.java:353)

at org.apache.jsp.AllTasks_jsp._jspService(AllTasks_jsp.java:320)

at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:69)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)

at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:365)

at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:309)

at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:242)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:847)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)

at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:832)

at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:620)

at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:553)

at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:482)

at com.sun.faces.context.ExternalContextImpl.dispatch(Unknown Source)

at com.sun.faces.application.ViewHandlerImpl.executePageToBuildView(Unknown Source)

at com.sun.faces.application.ViewHandlerImpl.renderView(Unknown Source)

at com.sun.faces.lifecycle.RenderResponsePhase.execute(Unknown Source)

at com.sun.faces.lifecycle.Phase.doPhase(Unknown Source)

at com.sun.faces.lifecycle.LifecycleImpl.render(Unknown Source)

at outsystems.hubedition.webwidgets.managedbean.WebPage.doForward(Unknown Source)

at ospledgemain.managedbean.flowapplication.EntryMain.pageLoad(Unknown Source)

at outsystems.hubedition.webwidgets.uicomponent.utils.ComponentUtils.invokeBeanPageLoad(Unknown Source)

at outsystems.hubedition.webwidgets.uicomponent.os_controls.PageComponent.encodeBegin(Unknown Source)

at javax.faces.component.UIComponent.encodeAll(Unknown Source)

at javax.faces.component.UIComponent.encodeAll(Unknown Source)

at com.sun.faces.application.ViewHandlerImpl.doRenderView(Unknown Source)

at com.sun.faces.application.ViewHandlerImpl.renderView(Unknown Source)

at com.sun.faces.lifecycle.RenderResponsePhase.execute(Unknown Source)

at com.sun.faces.lifecycle.Phase.doPhase(Unknown Source)

at com.sun.faces.lifecycle.LifecycleImpl.render(Unknown Source)

at javax.faces.webapp.FacesServlet.service(Unknown Source)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)

at outsystems.hubedition.webwidgets.BaseRequestStartupFilter.doFilter(Unknown Source)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)

at outsystems.hubedition.webwidgets.filters.HttpContextFilter.doFilter(Unknown Source)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)

at outsystems.hubedition.webwidgets.filters.CacheControlFilter.doFilter(Unknown Source)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:231)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149)

at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:150)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97)

at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:559)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)

at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:854)

at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:653)

at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)

at java.lang.Thread.run(Thread.java:745)


   Locked ownable synchronizers:


- <0x00007f81ea9f91e8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)


Rank: #6598

Please help us.