Hello Leonardo,

Thank you for your reply.

1.2.3.5.

Deployment zone is configured. The deployment zone is the IP address of the load balancer. Both #1 and #2 are in the same deployment zone.

Modules containing PDF screens and UltimatePDF are also in this deployment zone.

For debugging, we created the module including PDF screen and UltimatePDF in the deployment zone with localhost as the IP address.

This allows internal communication and avoids communication on Port 443, but now I get "operation timed out" errors.

Is there a possibility that this error will occur if I separate modules that contain PDF screens or UltimatePDF into different deployment zones?

4.

I have unchecked "Use HTTPS for internal communications" in the deployment zone.

6.

There is a reverse proxy.

7.

OutSystems.HubEdition.HTTPtoHTTPSproxyHeader parameter has been inserted.

We appreciate for your confirmation.

Would you be able to publish the following extension in Production, then republish the UltimatePDF_Service espace, and test if it solves your issue?

I have changed the extension to detect your scenario (by detecting the HTTPtoHTTPSproxyHeader setting) and then using HTTP and injecting the header.

If you continue to have problems, please send the error logs from ServiceCenter, including the stack trace. Note that a single PDF might generate multiple error logs (usually 3-4), please send the stack trace of all of them.

Hello, my name is Hiromasa Inoki, a colleague of Hiromi Hayashi.

Hayashi-san is currently busy, so I will reply on her behalf.

Thank you for the suggested solution. We will try it.

Please let me check one thing.

For now, we have the PDF screen and UltimatePDF in the localhost deployment zone, but once we release that extension module, is it correct to put it in the deployment zone where we have specified a load balancer in the IP address? (All modules are placed in the same deployment zone?)

Yes, having them on the same deployment zone should work now.

Thank you very much. Sorry to bother you again, but I have one more question.

What version of UltimatePDF is "UltimatePDF_Service_v72.xif" based on?

Is it based on the latest version 9.0.5? We are using 8.0.2 and would like to know which version to try.

Yes, it was based on 9.0.5. Give me a few minutes, and I'll apply the same changes to 8.0.2 so it has less risk for you.

Ok, please use the extension that is attached.

You should probably publish it in DEV first (don't forget to publish the UltimatePDF_Service extension afterwards) to make sure I used the same version as you had deployed.

Thank you so much for your help.

We are now checking if your file works for us. We will let you know once our confirmation is done.

Hello, thank you for your kind response.

May I ask some additional questions to analyze this issue?

1.

This issue is only occurring in the production environment and not in the QA environment with the same environment configuration.

We investigated the environment configuration differences, but could not find any.

*The above is is about before applying the patch

Can Ultimate PDF operation become unstable if we use Ultimate PDF with separate deployment zones?

2.

If this fix (to use http communication when HTTPtoHTTPSproxyHeader is set) solves the problem, will it be incorporated in future latest versions?

1. Ultimate PDF will always use the deployment zone address to render the screen. Even if you have multiple deployment zones, Ultimate PDF should work.

As to why the problem doesn't occur in QA, maybe your network topology is not the same. Or maybe the load balancer has a valid SSL certificate in QA, but not in Production.

2. Yes, I will release it as soon as you confirm the issue has been fixed.

1.

Thanks for the helpful advice. I will review the network topology and SSL certificate settings.

By the way, I am getting the following error, is there anything else I should investigate?

Sorry for asking several questions, but I would appreciate your advice.

---

①

yyyy-mm-dd hh:mm: ss, 000 ERROR [***.***] (1662fe06-1683-4ee5-90f3-643ac931b4df) The connection has timed out PrintPDF

CommunicationException: The connection has timed out

at c.onTimeout (https://***/***/scripts/OutSystems.js?x3N11wSh4Z4AKwRHy7hWcw:3:7427)

at XMLHttpRequest. (https://***/***/scripts/OutSystems.js?x3N11wSh4Z4AKwRHy7hWcw: 3:2648)

②

yyyy-mm-dd hh:mm: ss, 000 ERROR [***.***] (ec8445bc-3774-44c9-81ac-3051018b53fc) The connection has timed out Print PDF

CommunicationException: The connection has timed out

at c.onTimeout (https://***/***/scripts/OutSystems.js?x3N11wSh4Z4AKwRHy7hWcw:3:7427)

at XMLHttpRequest. (https://***/***/scripts/OutSystems.js?x3N11wSh4Z4AKwRHy7hWcw:3:2648)

③

yyyy-mm-dd hh:mm:ss, 000 ERROR [***.***] (4d486fb7-8fa4-4631-87b8-b86d3f03ee0c) The connection has timed out Search_Address

CommunicationException: The connection has timed out

at c.onTimeout (https://***/***/scripts/OutSystems.js?x3N11wSh4Z4AKwRHy7hWcw:3:7427)

at XMLHttpRequest. (https://***/***/scripts/OutSystems.js?x3N11wSh4Z4AKwRHy7hWcw:3:2648)

---

2.

Thank you so much for your willingness to release it once the problem is fixed.

We are required to be cautious about the environmental differences between production and QA. We will report back as soon as we can confirm.

The three errors you shared are client-side timeout errors. That just means that the PDF generation is taking longer than the client-side timeout. You might want to increase the client-side timeout (Server Request Timeout). You can find this setting in the call to server action in the code (you will need to change the code), or you can raise it for the whole application in Service Center by going to the Operation tab of the application.

You can check how long the PDF generation is taking by checking the service action log. You should see calls to PrintPDF and a duration in milliseconds. Your timeout value needs to be larger than that, and also account for latency in the network.

If you don't see any calls to PrintPDF service action, or any errors in those calls, then there's something else going wrong.

Hello, I would like to add more detailed information about the earlier question from Shimakura.

We have tested in our test environment with UltimatePDF_Service_v72.xif and UltimatePDF ver9.0.5 provided by you.

The results of the three test patterns, along with the deployment zone configuration pattern, are as follows:

[Test 1] 

UltimatePDF ver9.0.5 + UltimatePDF_Service_v72.xif

All modules in the same deployment zone

[Result] Failure ErrorLog_patching(Test1).xlsx in attachment

[Test 2]

UltimatePDF ver9.0.5 only

All modules are in the same deployment zone

[Result] Failure

[Test 3]

UltimatePDF ver9.0.5 only

Separate deployment zone for PDF output module and UltimatePDF, and set the address to localhost

[Result] Success

We had hoped that test 1 would succeed with the extensions you gave us, but it did not. Is there anything you can tell us about this result?

I look forward to hearing back from you.

Also, the following is the error situation we are experiencing in our production environment (UltimatePDF ver. 8.0.2, Separate Deployment Zone).

1. "CommunicationException: The connection has timed out" error occurs on the client side

2. One hour after the above error, "An error occurred while generating the pdf for OutputPDF" timeout error on the server side

We checked the internals of UltimatePDF and found that the extension was called with a timeout value of 1*60*60 before calling Service Action.

It seems that some process stopped inside the extension and it is outputting error 2 after 1 hour.

For example, the PDF output process of puppeteer is called with a timeout value of 0. Is it possible that this kind of process is causing the stop? If so, what could be the cause?

Sorry for the many questions but we appreciate your help. 

Hi. I haven't received the log for service actions. Especially since you claim "I figured it was less than the timeout value and something else was going on" but the error logs have "ssUltimatePDF_Service.ServiceAPIController.ServiceAPIPrintPDF" at the bottom of the stack and seem to indicate that this service action was executing for a long period of time.

The service action times out after 1h, which should be long enough for any use case. If you're receiving errors after 1h, that means some deadlock happened. Evidence of this would be in the service action log, you should see a duration of more than 1 hour (or 3600000ms).

I'm working to improve the logs from Ultimate PDF Management, to help troubleshoot your scenario. That should be ready this week.

Hi.

I've launched the version 10 of the component which reduces the timeout to 5 minutes, instead of 1 hour. Additionally, the default timeout is configurable, and can be overridden per request by using the UseTimeout server action.

That will not fix your timeout, but at least processes will not be using resources for so long.

I still don't know what is causing the timeouts, but I've enabled logging to understand what is happening at the lower level of Google Chrome. This is included in the browser.txt file when you download the troubleshooting zip file from Ultimate PDF Management. This will probably include sensitive information such as internal urls, and request payloads. So there are three options to continue the troubleshooting:

1. Compare the browser.txt file between a successful execution (e.g. from DEV) and from a timeout, to see where they diverge. This might help us narrow down the troubleshooting. However, this may be difficult since the two files will differ in too many details such as GUIDs, and most differences will not be relevant to the timeout. But maybe looking at the differences, something might jump out to you and help you identify what is causing the timeout.

2. Send the browser.txt file to me via email. Send me a private message if you want to go with this route.

3. Censor sensitive information from the browser.txt file, before sharing it. You will probably have to censor all URLs and payload information. I don't have an exhaustive list of everything you would need to censor.

I am confident that, with these logs, we will be able to understand why the timeout is happening and work towards a solution.

Hi. If you were using IP addresses for the deployment zones, then please upgrade to 10.0.1 and it might fix the net::ERR_CERT_AUTHORITY_INVALID error.

If you were not using an IP address for the deployment zones, could you check if the SSL certificate installed on the deployment zone's address is valid?

Hello, 

I apologize for not replying to you despite your polite response so far.

The "net::ERR_CERT_AUTHORITY_INVALID" error has been resolved by reviewing the environment settings and deployment zone.

Thank you for being so cooperative.

However, the "An error occurred while generating the pdf" error has not yet been resolved. We would appreciate your continued cooperation.

I upgraded UltimatePDF from 8.0.2 to 9.0.5 and am getting the same error.

This error does not occur after application deployment, but occurs after a while.

The conditions under which it occurs have not been identified. It may occur 3 weeks after deployment, or it may occur 2 days later.

The following error logs are now being output as a result of the version upgrade.

+++

An error occurred while generating the PDF for /pdf/***: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.

+++

PrintPDF

 CommunicationException: The connection has timed out

+++

PrintPDF

 CommunicationException: Request failed with an error

+++

An error occurred while generating the pdf for /pdf/***: The connection has timed out

+++

I also checked the System Temporary Folder of UltimatePDF and found that some folders remain undeleted.

Two folders remain, with update dates of "2023/07/19 16:35" and "2023/07/24 22:31".

Since these dates coincide with the dates when the PDF generation began to fail, I believe this is somehow causally related to the error.

I also checked the Temporary Folder and found the file "83a4658c-f82b-4bff-9f1e-6761f37e6abd.tmp" is being used by chrome.exe.

We suspect that when Puppteer launches a Chromium process, it may be that an already existing Chromium process has somehow become a zombie process.

Is there any way to deal with this?

We always appreciate your cooperation.

Could you upgrade to version 10.0.4 or higher? I have resolved some race conditions on that version, and had some other users report that it resolved these issues: https://www.outsystems.com/forums/discussion/89221/ultimate-pdf-the-underlying-connection-was-closed-a-connection-that-was-expect/

Thanks again for your quick response.

We will try the UltimatePDF upgrade and report back on how it went after the upgrade.

We would like to ask one additional question below.

After this error, we are considering adding a monitoring mechanism to quickly identify one of our two servers where the PrintPDF Service has stopped working.

The other errors reported so far are shown in the "Monitoring - Errors" tab of the Service Center.

However, when this error occurs, nothing is output to "Monitoring - Service Actions".

(If the service terminates properly, the Service Action Log will be displayed.)

Since Screen Request and Service Action are executed on different servers, we would like to identify the server where the Service Action was executed if possible.

Is there any way to identify the server where the Service Action was executed, such as outputting the Service Action Log even when an error occurs?

 e.g.) Customize to output a log when a PrintPDF Service Action is executed.

Thank you so much for your help.

You can see which server has executed a particular service action from the Service Action logs in Service Center, in the Server column:

I also want to point out that you can monitor Ultimate PDF processes and jobs from the Ultimate PDF Management solution.

Thank you so much for your help. 

The Service Action Log will show all Service Actions that have completed execution.

However, if a Service Action fails, the error will not appear in any log.

What we want to do is to identify the server on which the PrintPDF Service Action failed.

So, for example, we came up with the following idea.

Do you have any other ideas?

+++

Customize PrintPDF Service Action.

 1. Get the server name.

 2. At the beginning of the PrintPDF Service Action process, log the server where the action was executed. Even if any errors occur, the server can be identified.

+++

Thanks for the idea of using UltimatePDF Management. I will consider this idea in parallel.

Sorry for all the requests, but I have an additional question.

1.

Regarding the "Fixed a race condition that could close a browser abruptly and cause errors." in version 10.0.4, could you tell me where the problem was and what was fixed?

2.

I would like to confirm that version 10.0.4 or higher is the solution, but do you know the conditions under which this error occurs when version 9.0.5 is used? This error occurs irregularly and I am wondering how to reproduce it.

Any errors in service actions should be logged as well. But feel free to make that change in the PrintToPDF service action.

1. The race condition is documented here: https://github.com/hardkoded/puppeteer-sharp/issues/2202

There was also some issues with the UltimatePDF_Service extension. I remember there was a lock that was causing contention in some requests, and I believe the logic to kill a browser after it became idle wasn't safe enough and it could happen that the browser was killed while processing a request.

2. The original race condition in the github link I sent is probabilistic. It happens rarely, but given enough requests it will happen eventually.

The other issues in the extension happened more frequently if the PDF requests were frequent enough. Ultimate PDF reuses the same browser up to a certain point, when it decides to recycle the browser and launch a new one, killing the old one. But if a new request came in at the same time, it could happen that Ultimate PDF would kill the browser in the middle of the request.

Dear Leonardo,

Thank you very much for your help on this issue. 

It's been a while after upgrading the UltimatePDF, but no errors have occurred until today and the system is operating stably.

Although we are pretty much confident that the upgraded version will not cause similar errors in the future, we have implemented the UltimatePDF Management to identify the server where the error is occurring and to kill the Chromium process, just in case it happens.

Just one point of feedback.

When calling the PrintToPDF Server Action, the URL parameter omits "http/https" and domain name. (e.g., "/module_name/screen_name")

In this case, the called URL became always https.

In our environment, the internal communication had to be http, so the PDF generation has failed.

We have checked that the "Use HTTPS for internal communications" checkbox in the deployment zone is unchecked and that the OutSystems.HubEdition.HTTPtoHTTPSproxyHeader property is set.

We solved this issue by making the URL parameter when calling PrintToPDF Server Action explicit that it is http, like "http://DeploymentZoneAddress/module name/screen name".

We did this after looking at the UltimatePDF source code and confirmed that there is a process to create the URL if http/https was not specified.

It is now resolved and we are not having trouble. This is just a feedback.

We sincerely appreciate your time and consideration on this case.

Thanks to your help, we could overcome this problem. Have a great day!

Best Regards,

Hiromi