JBoss Error - Unable to create new native thread

JBoss Error - Unable to create new native thread

  
Hello,

I'm using the version 8.0.1.8 on Red Hat with JBoss EAP 6.2.2.GA (AS 7.3.2.Final-redhat-2)

In my solution I published a webservice that will received too many request, around 200 request per minutes.

Inside this web service I need call another external webservice in asynchronous way. For this, I'm using a timer that is waked up in the flow.

After some hours, I'm noting that the JBoss java process are consuming hight memory from SO and even if I stop the request, this memory is not unallocated.

When it happen, I have OutOfMemoryError error:

10:51:15,958 ERROR [stderr] (http-/0.0.0.0:8080-2)      at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:926)
10:51:15,958 ERROR [stderr] (http-/0.0.0.0:8080-2)      at java.lang.Thread.run(Thread.java:662)
10:51:15,975 ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[default-host].[/LifeTimeMonitoring].[TimerHandlerServlet]] (http-/0.0.0.0:8080-2) JBWEB000236: Servlet.service() for servlet TimerHandlerServlet threw exception: java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method) [rt.jar:1.6.0_45]va.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start(Thread.java:640) [rt.jar:1.6.0_45]
        at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681) [rt.jar:1.6.0_45]
        at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727) [rt.jar:1.6.0_45]
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655) [rt.jar:1.6.0_45]
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:78) [rt.jar:1.6.0_45]
        at outsystems.hubedition.util.TimeoutHandler.execute(Unknown Source) [outsystems.hubedition.runtimeplatformutil.jar:]
        at outsystems.hubedition.http.HttpContext.startRequest(Unknown Source) [outsystems.hubedition.runtimeplatform.jar:]
        at outsystems.hubedition.webwidgets.filters.HttpContextFilter.doFilter(Unknown Source) [outsystems.hubedition.webwidgets.jar:]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) [jbossweb.jar:7.3.1.Final-redhat-1]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) [jbossweb.jar:7.3.1.Final-redhat-1]
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) [jbossweb.jar:7.3.1.Final-redhat-1]
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) [jbossweb.jar:7.3.1.Final-redhat-1]

I changed the java start memory params many times to try improve, but the JBoss java process has the same behavior, allocate memory and not free up.

Anybody Know if there is a bug in this version?

Thanks regards.
Hi Tiago,

As far as I know there shouldn't be any known memory leaks on that version. But if you keep having problems you can contact OutSystems Support and they can help troubleshooting it better.

But anyway from what I see on what you explained:
  - You call a web service, is it a web reference also using the platform or is it done in an extension?
  - If you get 200requests per minute how will it run on the timer? it processes all it has in a queue?
  - If the timer is processing all it can does it end and restarts at some point? or it a very long running timer?
  - What are your current memory configurations?

Regards,
João Rosado
Hi João

Thanks for your consideration.

I will contact support to help in this issue.

So, about your questions:

  - You call a web service, is it a web reference also using the platform or is it done in an extension?

The web service "A" that I call is a web reference that I had configured from the studio. It is not in an extension.

Basically I have a web service "B" that I exposed also on the platform (not in an extesion), this web service execute some activity, included call another external web service "A". As I need that the flow don't wait the result ot the external webservice "A", I decided to put this call in a timer where I can wakeup and it will be executed asynchronous. So the request will be completed, who call my web service "B" will receive the response, independent if the external web service "A" was finished.
 
  - If you get 200requests per minute how will it run on the timer? it processes all it has in a queue?

I created a timer that run every 5 minutes. Case something wrong happen, this request will be stored in a queue to be run after. But I also waked up this timer for each request. I need to do it because I can't wait the finish of the external webservice to return the response. This call is not too slow, it take less 1 second to execute.

  - If the timer is processing all it can does it end and restarts at some point? or it a very long running timer?

  Looking the log I noted that the timer is not restarting, I know that it try run any time (defined in the Service Center) if it fall. It is not very long running. I tested for exemple in a small situation, just sending one request each 10 seconds. During this test, the JBoss java process (in the #top command) starting allocate memory, after 1 hour, around 50% of memory is allocated for this java process and all the system begin slow. So if I stop the requests, the memory is not unallocated, it still on 50%, if I restart the jboss-outsystems service it is not solved, I need to kill the process or reboot the SO.

- What are your current memory configurations?

The memory is small, 8 GB, but I think that the system should tolerate for a small volume, the solution is not too complex.

Thanks regards.

I think that this article is very useful

https://plumbr.eu/outofmemoryerror/unable-to-create-new-native-thread

I will try change the OS level.