System.NullReferenceException in Production Enviroment

System.NullReferenceException in Production Enviroment

  
Hello,

I have Platform Server version 4.2.4.78 installed in my Production Enviroment.
And I am having the following error sent to the event viewer at the rate of about 4 or 5 per second:


Event Type:    Error
Event Source:    OutSystems Platform
Event Category:    None
Event ID:    0
Date:        10-10-2011
Time:        11:49:44
User:        N/A
Computer:    BIPWFLOW01
Description:
Error sending log: System.NullReferenceException: Object reference not set to an instance of an object.
   at System.Data.SqlClient.TdsParser.AddSqlErrorToCollection(SqlErrorCollection& temp, SqlErrorCollection& InputCollection)
   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParserStateObject.WriteSni()
   at System.Data.SqlClient.TdsParser.WriteByteArray(Byte[] b, Int32 len, Int32 offsetBuffer, TdsParserStateObject stateObj)
   at System.Data.SqlClient.TdsParser.WriteMarsHeader(TdsParserStateObject stateObj, SqlInternalTransaction transaction)
   at System.Data.SqlClient.TdsParser.TdsExecuteRPC(_SqlRPC[] rpcArray, Int32 timeout, Boolean inSchema, SqlNotificationRequest notificationRequest, TdsParserStateObject stateObj, Boolean isCommandProc)
   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
   at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
   at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
   at OutSystems.HubEdition.RuntimePlatform.DBHelper.ExecuteNonQueryDontCleanParameters(IDbCommand cmd, String description, Boolean isApplication, Boolean skipLog)
   at OutSystems.HubEdition.RuntimePlatform.DBRuntimePlatform.CreateScreenLog(IDbConnection conn, ScreenLog log, Int32 cycleSize)
   at OutSystems.HubEdition.RuntimePlatform.Log.LoggerSynchronous.Log(ScreenLog log)
   at OutSystems.HubEdition.RuntimePlatform.Log.RuntimeLogger.Log(ScreenLog screenLog)

Message generated by eSpace WF_Organization.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.


The database server is not having any access errors logged, the database has plenty of grownth space, and I don't have a clue about where to look to solve this.
The event viewer is being overwriten every 20 minutes due to the huge amount of errors that are being generated.

Can anyone please point me in the right direction?
Thanks,
Ana Santana
Hi Ana,

Do the errors start right after the server is started? or only after a while?
Do you have any database errors other than that on your Error logs in Service Center?

A quick search on google associates those kinds of errors to bad connections in the connection pool.
But the system should be able to recover after some errors.

Regards,
João Rosado

Hello João,

I restarted the server to be able to answer your questions (this is a production server and we tend to restart it only if strictly necessary).
Its been up for over an hour, and the error disappeared!...

There were no errors in the database server, or in service center related to database issues.

I will monitorize it for a while and see if the error shows  up again.

Thank you!
Ana
Hi!

It's been a few days, in which we decided to run a series of windows updates, the machine wasn't updated in a very long time.
Now that everything windows-related is up to date, I've returned to check the event viewer for erros, and they persist!
They do not appear right after the system is restarted, and there are no errors at all at the SQL Server's machine.
There are 3 main errors/warnings that are appearing in our outsystems production machine:

############################################################
Event Type:    Warning
Event Source:    OutSystems Platform
Event Category:    None
Event ID:    0
Date:        21-10-2011
Time:        10:28:40
User:        N/A
Computer:    PRODMACHINE
Description:
Error openning connection to the database: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)
Cleaning the connection pools.

   at OutSystems.HubEdition.RuntimePlatform.DBHelper.releaseAllConnectionPools(String errorMessage)
   at OutSystems.HubEdition.RuntimePlatform.DBHelper.HandleSqlException(SqlException e)
   at OutSystems.HubEdition.RuntimePlatform.DBHelper.ExecuteNonQueryDontCleanParameters(IDbCommand cmd, String description, Boolean isApplication, Boolean skipLog)
   at OutSystems.HubEdition.RuntimePlatform.DBRuntimePlatform.CreateExtensionLog(IDbConnection conn, ExtensionLog log, Int32 cycleSize)
   at OutSystems.HubEdition.RuntimePlatform.Log.LoggerSynchronous.Log(ExtensionLog log)
   at OutSystems.HubEdition.RuntimePlatform.Log.RuntimeLogger.Log(ExtensionLog extensionLog)
   at OutSystems.HubEdition.RuntimePlatform.Log.ExtensionLog.Write(DateTime instant, Int32 duration, String action_name, String sessionId, Int32 espaceId, Int32 tenantId, Int32 userId, Int32 extensionId, String executedBy, String errorId)
   at OutSystems.HubEdition.RuntimePlatform.Log.ExtensionLog.StaticWrite(DateTime instant, Int32 duration, String action_name, String sessionId, Int32 espaceId, Int32 tenantId, Int32 userId, Int32 extensionId, String executedBy, String errorId)
   at ssWF_Common_BANIF.RssExtensionHTTPRequestHandler.MssGetEntryURL(HeContext heContext, String inParamEntryName, String inParameSpaceName, String inParamFirstParameterName, String inParamFirstParameterValue, String inParamSecondParameterName, String inParamSecondParameterValue, String inParamThirdParameterName, String inParamThirdParameterValue, String inParamFourthParameterName, String inParamFourthParameterValue, String inParamFifthParameterName, String inParamFifthParameterValue, String& outParamURL)
   at ssWF_Common_BANIF.Functions.rssextensionhttprequesthandler_UserActionGetEntryURL(HeContext heContext, String inParamEntryName, String inParameSpaceName, String inParamFirstParameterName, String inParamFirstParameterValue, String inParamSecondParameterName, String inParamSecondParameterValue, String inParamThirdParameterName, String inParamThirdParameterValue, String inParamFourthParameterName, String inParamFourthParameterValue, String inParamFifthParameterName, String inParamFifthParameterValue)
   at ssWF_Common_BANIF.Flows.FlowCommon.WBlkMenu.lnk_Linkwidget644023_NavigateUrl()
   at ASP.blocks_wf_common_banif_common_menu_ascx.__DataBindingwt_Linkwidget644023(Object sender, EventArgs e)
   at System.Web.UI.Control.OnDataBinding(EventArgs e)
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at OutSystems.HubEdition.WebWidgets.HyperLink.DataBind()
   at System.Web.UI.Control.DataBindChildren()
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at OutSystems.HubEdition.WebWidgets.Container.DataBind()
   at OutSystems.HubEdition.WebWidgets.If.DataBind()
   at System.Web.UI.Control.DataBindChildren()
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at OutSystems.HubEdition.WebWidgets.PlaceHolder.DataBind()
   at System.Web.UI.Control.DataBindChildren()
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at OutSystems.HubEdition.WebWidgets.Container.DataBind()
   at System.Web.UI.Control.DataBindChildren()
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at OutSystems.HubEdition.WebWidgets.Container.DataBind()
   at OutSystems.HubEdition.WebWidgets.If.DataBind()
   at System.Web.UI.Control.DataBindChildren()
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at OutSystems.HubEdition.WebWidgets.PlaceHolder.DataBind()
   at System.Web.UI.Control.DataBindChildren()
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at OutSystems.HubEdition.WebWidgets.OSUserControl.DataBind()
   at System.Web.UI.Control.DataBindChildren()
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at OutSystems.HubEdition.WebWidgets.Container.DataBind()
   at System.Web.UI.Control.DataBindChildren()
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at OutSystems.HubEdition.WebWidgets.OSUserControl.DataBind()
   at proxy_WFF_Invoce_WF_Common_BANIF.Flows.FlowCommon.WBlkHeader.DataBind()
   at System.Web.UI.Control.DataBindChildren()
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at OutSystems.HubEdition.WebWidgets.Td.DataBind()
   at System.Web.UI.Control.DataBindChildren()
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at System.Web.UI.Control.DataBindChildren()
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at OutSystems.HubEdition.WebWidgets.Table.DataBind()
   at System.Web.UI.Control.DataBindChildren()
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at System.Web.UI.Control.DataBindChildren()
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at System.Web.UI.Control.DataBindChildren()
   at System.Web.UI.Control.DataBind(Boolean raiseOnDataBinding)
   at OutSystems.HubEdition.WebWidgets.OSPage.DataBind()
   at ssWFF_Invoce.Flows.FlowInvoiceFlow.ScrnInvoice_Edit.DataBind()
   at ssWFF_Invoce.Flows.FlowInvoiceFlow.ScrnInvoice_Edit.Page_Load(Object sender, EventArgs e)
   at System.Web.UI.Control.OnLoad(EventArgs e)
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   at System.Web.UI.Page.ProcessRequest()
   at System.Web.UI.Page.ProcessRequest(HttpContext context)
   at ASP.invoice_edit_aspx.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   at System.Web.HttpApplication.ApplicationStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest(HttpContext context, AsyncCallback cb, Object extraData)
   at System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr)
   at System.Web.HttpRuntime.ProcessRequestNoDemand(HttpWorkerRequest wr)
   at System.Web.Hosting.ISAPIRuntime.ProcessRequest(IntPtr ecb, Int32 iWRType)


Message generated by eSpace WF_Organization.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
############################################################
Event Type:    Error
Event Source:    OutSystems Platform
Event Category:    None
Event ID:    0
Date:        21-10-2011
Time:        10:28:40
User:        N/A
Computer:    PRODMACHINE
Description:
Error sending log: System.Data.SqlClient.SqlException: A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)
   at OutSystems.HubEdition.RuntimePlatform.DBHelper.HandleSqlException(SqlException e)
   at OutSystems.HubEdition.RuntimePlatform.DBHelper.ExecuteNonQueryDontCleanParameters(IDbCommand cmd, String description, Boolean isApplication, Boolean skipLog)
   at OutSystems.HubEdition.RuntimePlatform.DBRuntimePlatform.CreateExtensionLog(IDbConnection conn, ExtensionLog log, Int32 cycleSize)
   at OutSystems.HubEdition.RuntimePlatform.Log.LoggerSynchronous.Log(ExtensionLog log)
   at OutSystems.HubEdition.RuntimePlatform.Log.RuntimeLogger.Log(ExtensionLog extensionLog)

Message generated by eSpace WF_Organization.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
############################################################
Event Type:    Error
Event Source:    Windows Search Service
Event Category:    Gatherer
Event ID:    3083
Date:        21-10-2011
Time:        10:28:38
User:        N/A
Computer:    PRODMACHINE
Description:
The protocol handler Search.Mapi2Handler.1 cannot be loaded. Error description: Class not registered
.


For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
############################################################

This error is appearing in any eSpace that tries to write in the log, not just the one showing here.
What's wrong here and how can I solve it?
Thank you
Ana Santana

Since this is a production environment and I'm guessing there is some urgency to it, have you tried OS Support directly?

Best regards,
PC
All the applications seems to be working fine, and there are no complains from the users.
However, this constant errors/warnings is not normal and I'm trying to address it before they turn into a problem to the applications/users.
So, no, I haven't tried outsystems support yet.

The Windows Search error I already found a fix and time will tell if it worked or not.
About the other two Outsystems events, I can't find fixes or solutions anywhere.

Just a wild guess cause I did have some issues that remind me of these errors in the 4.1 version

The DB server is not the same server as the front-end right?

Best regards,
PC
No, it isn't.
It's a machine dedicated to SQL Server, that is used not only to the Outsystems DB but to other production environments DB's as well....
As I though...make sure (with the network team) if there are any constraints in the path between both servers...I had problems simply because someone changed one of the machine cables connecting it to a different switch port which was badly configurated, and performance decayed so much that we started having network problems that would result in a similar behaviour to the one you are experiencing

Best regards,
PC
I talked to our network administrator guy.
Our machines are in a big virtual environment, and those kind of problems are very unlikely to occur, accordingly to him.
What he did do was to check the production machine's network card power saving options, and turn power saving off.
We are now giving the machine some time and see if the erros go away.
I'll come back to tell you the results of this experience.

Best Regards,
Ana Santana
Hello again,

All the possible network problems were verified and nothing is wrong.
It has to be something within the outsystems platform...

I'll get in touch with Outsystems support and see if they can find out what's wrong in our installation.

Thank you for your help anyway.
Ana