Hi,
When performing a performance test last week we noticed that our 2 front-end servers were out of memory after a while. It looks like the ultimate PDF component used up about 10GB of RAM for each running browser process on both of the servers.
Is it possible to change something that the memory will be a little bit more agressively cleared when generating for example 1000 pdf's in a very short timeframe?
Kind regards,
Bas
Hey @Bas de Jong and everyone keeping an eye in this forum.Just to let you know that we released a new version of Ultimate PDF with a fix for the memory issue.Just to keep it transparent, this is what was changed:
We added an else to the if clause on the Disposable method of the PooledPage Class to ensure the tabs are closed properly.
This issue was introduced as part of the last release, when we improved the memory allocation for PDFs in timers, but didn't consider other edge use cases.
@Bas de Jong , as you were having the issue before, would be great if you could test it and let us know if you don't have any problems as well with this new version.
Best Regards,
RR :)
That's awesome! Thanks for including the fix. We will try it out in the upcoming days!
Hi Bas,
You can check the memory usage of running processes by using the GetActiveProcesses service action from UltimatePDF_Service module.
I recommend checking, after each generation, if MemoryUsageMegabytes exceeds a defined threshold and then kill this process with the KillProcess service action from UltimatePDF_Service module.
Use GetManagementSettings server action from UltimatePDF module to get the configured TemporaryFolder.
Best regards,
HP
See my post about reentrancy. It could be that the .NET extension is not written in a reentrancy way. This will lead to no issues when the load is low but certainly will generate unpredictical behaviour when the load get's higher. We face the same issues and unfortunately the only thing we can do if the we see memory usage/cpu load rising to max that we recycle the IIS processes.
Hey Hélio,
Thanks for the suggestion. However, is this a "safe" way to do this?
E.G. will no PDF's be impacted by killing this process? This is critical for the high-load environment where we want to generate lots of pdf's.
The management interface warns for this:
and the Service Action you are referring to has "internal use only" in the description:
Maybe @Leonardo Fernandes has a suggestion or can confirm that this Service action is a safe way to do this?
Maybe it could be baked into the PDF management app with a timer or something?
That's an issue for sure, you must guarantee no printing job is running before killing the processes.
A max memory usage setting, it's something @Leonardo Fernandes can easy implemented on the extension side to recycling the processes.
As an workaround, it's better stop printing for 1s than 'killing' the whole machine.
Exactly. Do you know if there is a way to just start a new process alongside a old one and force the new pdf's to use the new process? And let the old one "die"?
Stopping generation for 1 sec is no problem, but failed pdf's is.
Using the UltimatePDF_Service extension you can create different processes working with different TemporaryFolder's, old processes will die for inactivity (10min), during this time they are consuming memory and doing no work.
Without changing the extension, killing the processes it's the way.
Sorry for asking, but how would i do that?
I'm kinda lost in which action to use. The PrintPDF action of the component uses a site property with the Temporary folder. Would changing that property launch a new browser process the next time it's executed?
Otherwise, i see no option to launch a second or third browser process.
My goal would be to launch a new process every x minutes and let the other ones die slowly
You must directly call the PrintPDF service action from the UltimatePDF_Service module
Hi @Hélio Palma. Best wishes for the new Year. We have tried the suggestions to start multiple processes with different folders. However this results in all the current processes being killed and only the newest one survives, resulting in errors for users :( So currently this solution does not work.
@Hélio Palma or @Leonardo Fernandes do you guys have any more suggestions. Currently i'm letting a .NET colleague evaluate the code to see if there are any improvements about memory management to be found there. If he finds something, could it be included in the PDF Generator Forge component?
Hi @Hélio Palma and @Leonardo Fernandes, @Miguel Antunes, @Raphael Ranieri . He proposes to make the following changes:
n the PrintPDFmethod of the UltimatePDFExecutionContext class, we propose adding statements to close the page when rendering completes:
We propose the same for the PNG implementation in the ScreenshotPNG method:
What are your thoughts about this?
Hey @Bas de Jong,
Can you clarify if generating 1000 pdfs at the same time is something you expect in a real use case? Or something that is just happening because you are stressing the application for a test?
If you really need this amount of PDFs to be generated could you queue the requests and do it asynchronously in a timer (for instance) and then send the PDF in an email? Or do you need to return the information immediatally for the user?
I'm afraid that even if you optmize the code, eventually you will face a bottleneck if you need to generate this many PDFs at the same time.
Best Regards, RR :)
Hi Raphael,
That is indeed the case. Our stress test was half the desired load. We use multiple frontend servers and we know we will hit some bottleneck. But what we don't want is a single component filling up gigabytes of ram and with that, take the whole server down. So optimizing will be a huge improvement, we also recognize that we have to scale our setup. But if we don't try to optimize this component that will just be a waste of money
@Bas de Jong ,
Even if you apply this optimization, it still consumes a lot of resources.
After all, you are only closing the page after it is used. However, if you are using 500 pages at the same time, memory will remain allocated, and this will not solve your issue.
Additionally, since you are waiting for the page to close, it may delay the generation of your PDF if something goes wrong, such as Chromium becoming unresponsive.
The component consumes a significant amount of memory because, on the server side, it essentially opens a Chrome tab and exports it to PDF. Even with optimizations, Chrome is not known for being memory-efficient.
This could eventually consume all available resources to the point of impacting other applications or features in your system.
If generating the PDF immediately for the user is not critical, I recommend queuing the requests and generating the PDF asynchronously using a timer, depending on a memory usage threshold.
You can use the GetActiveProcesses action to monitor memory consumption and only generate the PDF if enough memory is available. Otherwise, wait for a few seconds before rechecking.
Would that be an option?
Unfortunately we have to generate most of the PDF's immediately. If we could do it in batch it would not be a problem. So i really would like some memory management improvements.
Hi Raphael, Ultimate PDF team,
Reading this discussion I feel some misunderstanding related to the problem Bas tries to solve. The Ultimate PDF is a generic component used by several applications with high user-interaction providing digital documents (PDF) serving all kind of processes within the health care.
It is not the case that continuously >1000 PDF's are generated at the same time, however it could be possible during peak-moments. The main problem behind this is that regular hours the PDF Generator will not become idle and filling up the memory to critical limits. Generating the documents is critical and filling up the memory is a problem for guaranteeing the continuity. So a solution is needed not to fill-up memory as long as the process is running and don't have an idle time of 10 minutes.
Killing the process is not an option, because you don't know or any other documents are being generated at the moment, as stated by you/Ultimate PDF team as well. The solution provided by Bas makes (a lot of) sense to implement a piece of logic to close the page as soon as the generation is completed. Or the suggested place in the extension is right is hard to tell and maybe better alternatives are applicable, but at least it should be admitted that keeping the page open until there is an idle time of 10 minutes is not a very solid solution.
Addressing this issue takes your already excellent component to the next level. By implementing this enhancement the component transform a strong solution into an even more powerful solution.
Peter
Hi Guys,
This topic is becoming a bit stale. Can you guys please reconsider adding the above code to the PDF forge component. We have tested the code and we see that instead of 10+GB usage we now only use about 2 GB RAM when generating loads of PDF's. It's working stable with closing the tabs, and it would really suck if we had to clone this component just to prevent our servers from going down when generating loads of PDF's.
If you want, we can also directly provide a new version of ultimate PDF to put into the forge, based on the latest update.
You can lower the high CPU usage manually and using an automatic tool.
This reply is not useful at all... can you please remove it, you don't contribute to this discussion.
Sorry, if you are disturbed.
I am suggesting that if you want to remove unwanted space or want to clean the space in your PC.
He has the same problem !! We have a ruge application from a client that has a lot of costumer and we have pick of memory in the server when all the costumers print pdf in the same time. We already scale our server on the cloud, but without success. The outsystems support only tell us that we need to see UltimatePDF, but there isn't another solution to generate PDF like this component in the forge.
@Bas de Jong , @dauster , @Peter Vermeulen ,
We are already duscussing the best plan of action to do it and we will update you asap.
Besides this, there are other fixes we also need to consider and test before releasing, and as this component is not maintained by OutSystems, we do it as we have available time.
I will keep you postedBest Regards,
Hey Raphael,
Thank you so much for considering to add this to the component. It would help us a lot if we could stay up to date with the forge.
I understand this is done in your available time, however i am also able to provide a fixed version of the latest version of this component if that helps speed things up. Let me know.
Some performance information:This is a graph of available memory on our server that generates the PDF's. After deploying the non-forge version with the fix implemented you can now see that the memory is quite stable instead of spiking down to only 0-100 MB free on the machine.