Hi Miguel,

Is there any way to measure these parameters, so we can know if we are indeed being affected by this problem and also better choose the appropriate values?
I was thinking maybe performance counters, but which ones would be useful?

Thank you,

Paulo Cunha

Hi Paulo

The Windows operating system doesn't have specific performance counters for ephemeral ports, however, you can use other performance counters to identify the overall number of TCP ports opened on the system.

Using the performance counter TCPv4\Connections Established, you can identify the amount of existing connections on the server. If it's near the default limits, then you might have problems establishing new connections, and usually the symptoms are runtime errors with exceptions like:

An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full 

Hope this information is helpful.

Cheers

Miguel Simões João

Hi Miguel,

                   First of all thank you for writing such a clear and good post. It makes it so clear after reading your post about tuning the TCP/IP Stack and to eleminate the dreaded error "An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full ".

                   But unfortunately for me though it has increased the requests per second on my performance test, which is for huge number 10,000, the error message crops up every time I near the completion of my test; even after tuning the TCP/IP stack described in your post. To say you exactly I get the error whenever I reach the mark of 8500 users in my load test.

                  Before I end just a brief of environment that Iam using, I am suing Visual Studio Team Foundation 2010 for my Test and so I have a Rig created and, My Controller and agents resides on VMs'.

                 I am just puzzled as what more do I need to configure to eleminate this error.

Cheers
Abdul Latif Saquib

Hello Abdul

The error "An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full " is not necessarily only caused by the TCP ephemeral ports exhaustion. There's also memory concerns to take into consideration, for instance, in 32 bit environments when using the /3G boot falg on the operation system, this can be a frequent error.

Do you care to share a little bit more about the architecture of your environment? What operating systems are being used? Are they 32 or 64 bits? Which have OutSystems Deployment Controller running and which are only Front-Ends with the OutSystems Deployment service?

When you're performing the load test, and reach the 8500 users, how many TCP connections do you get with the TCPv4\Connections Established, performance counter?

Cheers

Miguel João

Hi Miguel ,

Thanks for your timely reply. Here are the details of my setup.

Architecture details  :
VSTS 2010 controller -1 (Physical Machine)
VSTS 2010 Agents-10 (Vitual machines)
All machines are on Windows 2008 R2 with latest SP
 
Environment is VMware setup :
Web servers -8
Application 2
SQL Database servers 
 
I see these errors on the VSTS results and also on VSTS controller and agents.
I tried tuning the following
TCPwiatdelay -30
MaxuserPOrt -65537
I have now disabled  TCP Auto-Tuning. Thought the TCPWindow size is what I need to alter.

Yet the test starts to throw the dreaded WSAENOBUFS10055 error just after 8500 user load. The memory is present in abundance and I see no pressure on it.

Thanks and Regards
M.A Latif Saquib



 

Hello Abdul

To have more context about the applications you're testing, are they OutSystems applications, or just ASP.NET applications?

Like I've said, the error is not just due to lack of ephemeral ports. It can happen if your memory resources are scretched to thin. I would recommend to monitor the following performance counters on the servers that are generating such error:

- TCPv4\Connections Established
- Memory\Available MBytes
- Memory\Free System Page Tables Entries
- Memory\Pool Nonpaged Bytes
- Memory\Pool Paged bytes
- Paging File\% Usage

If there's memory strangulation on the test, you'll see the Available MBytes reducing, the Page File usage increasing, and the Pool nonpaged and paged bytes also increasing. If they reach their limits, then you've got a memory strangulation on the process.

Also, make sure the Connections Established are way below the 65535 limit.

Do you care to share some of those metrics for analysis?

Cheers

Miguel

Hi Miguel,

                  First of all let me thank you for replying me and rendering your support though a bit late, for I feel you were off for celeberation.

                 I do not have OutSystems applications but,  just ASP.Net. After reading your good post and many many articles on the web, and monitoring my memorry and other good metrics I could just conclude that, the memory what I had was abundant and their was no way that I got strangulated. One thing which I could conclude was, somehow the load test is opening ports whch where beyond the LIMIT as rightly said by you "65535 " and I have to limit them.

               Alhamdullilah, two days back I was able to curb down the problem. The resolution was just simple at the end, with all the understanding now made me confess that I should use Connection Pool instead of Connection Per User Option which I was using on my Load Test.  In VSTS User Load is not quite proportional to hits or it does not signify exactly as how many hits are being pressurized at the given load, just cannot find the concurrency.
 
           Any ways the solution, I limited the connections by pooling a limited amount of connnections. This avoided the abundance of ports to be opened and also it limited the TCP Stack's backlog Connection Queue from getting flooded.

Well, to be more realistic I may have to use the connection per user option though; Inshallah I feel better off now than before with your support and the progress.

Thanks again Miguel
Best Regards
M.A Latif Saquib
       

I am on windows server 2008 R2.

Got the "parameter incorrect" error on
netsh int ipv4 set dynamicportrange tcp start=32767 num=65535

But the following works
netsh int ipv4 set dynamicportrange tcp start=32767 num=32767

Hello Terry,

Great catch. After all this time I haven't noticed the typo in the command line for windows. I've fixed it in the original post.

Thanks Pradeep for the insight and explanation.

Cheers

Miguel João

hi there
i am having a problem that my NVR is on static ip(intranet) using internet through isp broadband router(DHCP) the issue is my nvr break ping automically unless i change the port of ethernet i want to know how to prevent router to break the ping i fixed all the DHCP range to prevent conflicting but no result kindly help me with this

Hello Jahangir

Unfortunately, that seems to be a network configuration problem with the router, or some firewall. Can't help you there.

Have you tried contacting the router manufactor.

Cheers

HI
THANKS FOR YOUR REPLY I CAN NOT CONTACT WITH THE MANUFACTOR IT IS AN OEM FROM A NATIONAL BROADBAND COMPANY. I DISABLE THE DHCP DIFFRENCE OOCUR A QUITE BIT.I THINK THERE IS SOME PROBLEM I NEED TO FIND THAT HOW TO PREVANT ROUTER TO BREAK THE PING OF SPECIFIED IP IF YOU CAN GUIDE ME.