Tip: The underlying connection was closed: An unexpected error occurred on a receive

Symptom

You application is making a call to another system performing an HTTP(S) request. This request can be a:

• Web service call (SOAP or REST)
• A request to a specific web-page
• Sending an email
• ...

The call is failing and an error occurs in the application. In the Error Log you see an entry similar to (below example is for a SOAP web-reference):


Cause
Errors like these happen because the remote server (to which the call was being made) did not deliver a response to your request and cut the connection before an answer could be sent. Meaning:

1. Your application (the caller) sent the request
2. While the application was waiting for the response, the remote server cut the connection.

Resolution
Resolving such a problem cannot be done on the caller side (your application) - you need to work with the party that provides the service for you.

If you are the other party or want to help: common causes on the other side for these errors are:

• The request is taking too long to run and the server "times it out". If another OutSystems Platform is on the receiving end (meaning: the web call is to an OutSystems Platform) you may want to look here.
• The remote server is having issues. E.g. application pool recycling or crashes that interrupt logic that may be runnning.
• A network layer is cutting the network connection half-way. You may see errors in the remote server similar to this one (e.g. "could not flush response);

Regards,
Acácio

Hi,

i'm facing the same error and can't figure out the root cause from my side. I sent to the API providers and they asked me to capture the request and its response and send it to them for debugging. Any idea how i can do that ?
Another question: the close of connection is carried out by outsystems server or the server the API calls ?


Note: i tested the API earlier through a http server and it worked perfectly, when i switched to the live server with https, it failed with this error.

Hello Bil,

Like this post says, there is nothing you can do on the caller side. You need to get in touch with the provider of the service so they can tell you what's happening.

Best regards,
Ricardo Silva

Good Afternoon,

We're getting a very similar message after upgrading to 9.1.300.0 on a webservice, except we're getting "send" at the end not "receive"
 "The underlying connection was closed: An unexpected error occurred on a send."
Does the same explanation apply ?

Regards,

Chris O'Reilly

Hi Chris

I'd like to see the whole error stack, but would say that the explanation would be the same.
The difference is the part of the call where it happens - during sending the request to the end-server rather than when receiving the response.

Acácio

Hello Acácio,

Wow, what a quick reply, thanks.
Here's the error stack (full URL disguised) for you to look at, just to add more detail the remote site is using TLS1.0 

Regards,

Chris.

Error Detail Back to Log Id: 40c44c51-30e2-49ab-9dea-0c854f505b3b Time of Log: 25/02/2016 16:20:46 eSpace: BackupVRMlookup Tenant: Users User:  (2) Session Id: 4tmj1q1w3xted1kgpmwvz4xr Server: OS10689DEV66263 Module: Web Reference Message: The underlying connection was closed: An unexpected error occurred on a send. Environment Information eSpaceVer: 30 (Id=4286, PubId=5381, CompiledWith=9.1.300.0) RequestUrl: XXXXXXXXXXXXXXXXX\/HomePage.aspx?_ts=1456417245861 (Method: POST) AppDomain: /LM/W3SVC/1/ROOT/BackupVRMlookup-91-131008903342359506 FilePath: d:\Outsystems\Platform Server\running\BackupVRMlookup.01915612306\HomePage.aspx Locale: en-US DateFormat: dd/MM/yyyy PID: 372 ('w3wp', Started='2/24/2016 6:41:39 PM', Priv=1113Mb, Virt=9758Mb) TID: 150 Thread Name:  .NET: 4.0.30319.42000 Stack: [1] The underlying connection was closed: An unexpected error occurred on a send.    at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)    at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)    at ssExternalWebservices.WRefCarweBVRRWebServiceMain.WRefCarweBVRRWebService.strB2BGetVehicleByVRM(String inWSstrUserName, Boolean inWSstrUserNameSpecified, String inWSstrPassword, Boolean inWSstrPasswordSpecified, String inWSstrClientRef, Boolean inWSstrClientRefSpecified, String inWSstrClientDescription, Boolean inWSstrClientDescriptionSpecified, String inWSstrKey1, Boolean inWSstrKey1Specified, String inWSstrVRM, Boolean inWSstrVRMSpecified, String inWSstrVersion, Boolean inWSstrVersionSpecified, XmlElement& outWSstrB2BGetVehicleByVRMResult) [2] Authentication failed because the remote party has closed the transport stream.    at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)    at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)    at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)    at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)    at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)    at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)    at System.Net.ConnectStream.WriteHeaders(Boolean async)

Hi Chris

By looking at the stack I'd say it's the exact same explanation. Actually, your error message gives you some more details:

[1] The underlying connection was closed: An unexpected error occurred on a send.
[2] Authentication failed because the remote party has closed the transport stream.

This means:

1. The base error is "connection was closed while sending information"
   == because ==
2. "the remote party has closed the transport stream [the connection]"

By the way, this is a generic mechanism you can use when reading such error messages: if you find things such as [1], [2] in your error stacks, each bigger number is what is called an "inner exception". This means the thing with the lowest number (the "outmost exception") is the error that was caused ultimately by the biggest number ("the inmonst exception"). Think of it as (absurd/generic example):

[1] Could not complete transaction
because
[2] Error writing to SQL Server
because
[3] Syntax error in query

Most of the times you get very valuable information just by reading the first part of each [x] (meaning: the lines that don't start with "at"). Other times, you need to read the lines starting with "at" (which show the lines of code where the problem happens - the "call stack").

Hope this helps
Acácio

By the way, if you're going down the troubleshooting path, this post might be helpful in understanding OutSystems error stacks:

http://www.outsystems.com/forums/discussion/7856/anatomy-of-an-outsystems-error-stack/

APN

Problem solved Acácio!

The endpoints I was originally trying to connect to didn't support TLS 1.2 which gave the original error message, luckily they had another server which did provide TLS1.2 and I connected and got the correct response straightway.
So the moral of the story is 9.1 is much stricter than 9.0 when connecting, and rightly so, it will force service providers to update their servers when vulnerable protocols are still being used.
Thanks for your help and quick responses, and I've learned a lot about error tracing and troubleshooting.

Chris.

Hi Acácio,

I'm having exactly te same error as the one reported by Chris (see below). After discussing and troubleshooting it with the service provider they concluded that we are negotiating on TLS1.0 and they have TLS1.0 disabled on their end (only 1.1 and 1.2 are enabled). Is there a way to enfrce the usage of TLS1.1 or 1.2?

Thanks,
Pedro

Error:
Message: The underlying connection was closed: An unexpected error occurred on a send.

Environment Information
eSpaceVer: 354 (Id=120008, PubId=169933, CompiledWith=9.0.1.50)
RequestUrl: http://127.0.0.1/Ind_Integration/_TimerHandler.asmx (Method: POST)
AppDomain: /LM/W3SVC/1/ROOT/Ind_Integration-700-131051900176010862
FilePath: E:\...\PS\running\Ind_Integration.01560225457\
Locale: en-US
DateFormat: dd-MM-yyyy
PID: 49436 ('w3wp', Started='29-03-2016 12:42:23', Priv=2126Mb, Virt=26611Mb)
TID: 2421
Thread Name:
.NET: 4.0.30319.34209

Stack:
[1] The underlying connection was closed: An unexpected error occurred on a send.
at ssInd_Integration.WRefVcmDirectWebService.WRefVcmDirectWebService.DoeFunctie(String inWSfunctie, Boolean inWSfunctieSpecified, WORCVcmBerichtRecord inWSinvoer, Boolean inWSinvoerSpecified, WORCVcmBerichtRecord& outWSDoeFunctieResult)
at ssInd_Integration.ExtendedActions.WebRefVcmDirectWebService_DoeFunctie(HeContext heContext, Boolean[] optionalValuesBitMask, String inWSfunctie, RCVcmBerichtRecord inWSinvoer, RCVcmBerichtRecord& outWSDoeFunctieResult)
at ssInd_Integration.Actions.ActionANVA_DoeFunctie(HeContext heContext, String inParamFileNumber, String inParamPK_DATA, Int32 inParamElementId, Int32 inParamElementType, Boolean inParamSkipTransactionIdValidation, Object& outParamReturn, String& outParamErrorMsg, Int32& outParamErrorCodeId, Boolean& outParamResumeOnError)

[2] Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.Net.ConnectStream.WriteHeaders(Boolean async)

[3] An existing connection was forcibly closed by the remote host
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)