Troubleshooting HTTP communications and web services - TCP Sniffing

Troubleshooting HTTP communications and web services - TCP Sniffing





Sometimes you may encounter some problems where you find it useful to know what is being passed between the server and the client of a HTTP communications. A frequent situation is that you have a web service that is not producing the desired result, you have already got it all sorted out on the applications (correctly imported, correct data types, ...) but when calling it you either get an error, or no results at all. Other scenarios where this may become useful are when you suspect of connectivity issues between servers, or simply when you are curious to know what is being transfered between your browser and your server when you access a given page.


For this, there is a procedure called TCP sniffing, which is literally to sneak peek at the communications between the two parties involved. Many tools exist for that, and this post explains how to do it with one given tool, for the sake of simplicity, although the explanation is transversal so any other tool can be used. 


The procedure above is generic enough to be used with any of the OutSystems AgilePlatform tools, when troubleshooting 3rd party integrations, or even for non-OutSystems related communications



How it is done - high level


There are two typical ways to sniff TCP communications:

  - Placing the sniffer in the middle of the communication, and altering one or both ends of the communications to explicitly send traffic to it. This procedure demands changing involved URLs in the application logic, but is typically less intrusive since you don't need access to the server and/or sniff all communications;

  - Sniffing the communications between both parties without explicit man-in-the-middle. This option is more faithful to the desired setup, but is more intrusive.


This post focus on the first perspective - explicit sniffer in the middle. The tool chosen for this procedure is TCPTrace, which is a free and simple tool that needs not be installed - simply unzip the program and run it. The sniffer itself runs only in Windows machines, but can be used to sniff communications between any servers. Other programs exist with the same set of functionality, so you might opt for your favorite one.


Just for information, a program that is used for the second perspective is Wireshark, but its use is outside the scope of this post.





TCPTrace can be obtained at . After downloading it, simply run it and configure its options:

  - local port where TCPTrace will be listening;

  - IP of the destination computer

  - port in the destination computer


Local port should be a port that is not being used in the computer where TCPTrace is run. 8080 is usually a good choice.


Make sure to configure the correct ports for destination. For HTTP port, it is typically port 80; SMTP is typically port 25. Other port listings can be used, depending on the context.


To test sniffing, simply open a web browser and access http://computer:8080/ (where computer is the machine where TCPTrace is running). You will see some lines in the window indicating communication with the destination server. If you do, congratulations, you have TCPTrace running.



Some usage scenarios


I. Sniff communication between Service Studio and Service Center

To do this, simply start TCPTrace, and use computer:8080 instead of servername in the dialog in Service Studio.


II. Capture communications (get/post) between browser and server

To do this, rewrite application URLs to use computer:8080 instead of servername in the server part. If using AgilePlatform 4.2, remember to turn off IIS compression and back on when you are done (


III. Capture Web Service Communications

This one is a little trickier, as it involves importing a changed version of the WSDL. Steps are as follows, and assume web reference created in Service Studio:


  1. Get the WSDL from your web service (put the URL in a web browser and save the text file you obtained);

  2. Open it with a text editor, and locate (typically at the bottom) a tag in the form:




          <soap:address location="http://servername/some/path/to/a/service.asmx" />




     Here you need to replace servername with computer:8080. Note that you might have several port/address elements - make sure to change them all. Also note that, in this case, computer must be a machine visible by the OutSystems server - if you can, simply run TCPTrace in the OutSystems server for simplicity.


  3. Refresh the web reference with the altered WSDL (replace the URL you originally inputted with the path to the file, e.g. c:\test.wsdl);

  4. Publish the eSpace

  5. Run the logic that calls the web service. You will be able to see the contents of the communication in TCPTrace.


IV. Capture communications with SMTP server during email send

For this, you will need to configure TCPTrace to listen on port 25 and redirect the traffic to the SMTP server, to the same port. After that, use the name of the computer as the SMTP server in the eSpace logic.





There are several variants you can create to this, for diverse troubleshooting purposes. You can also associate TCPTrace with STunnel to sniff on HTTPS traffic, in case you are having problems troubleshooting communcations that can only be made in a secure channel. More on the subject later on.


Before using any of these procedures, note that you might be able to capture confidential information (usernames / password, ...) and that you must be authorized to do so.


Feel free to comment or correct whatever is said above!

Hi everyone

I would like to post a minor change to the above post. For scenario III. Capture Web Service Communications , instead of changing the WSDL, you can simply change the effective URL of the web service. Simply:
  • Access Service Center;
  • Go to Factory --> eSpaces and select the eSpace containing your web reference;
  • In the Web Services tab, click the name of the web reference;
  • Change the effective URL to be the place where TCPTrace is running (in the example, you would write something like http://computer:8080/some/path/to/a/service.asmx );
  • Republish the eSpace.
This one is a lot easier than messing around with WSDL.
Will there be an extension made available to 'log' the low-level Web-requests?

This would be very nice for debugging purposes where clients directly connect to a webinterface showing the XML data 'log'.
(I'm currently working on a Javascript based "message syntax highlight" E-space to support a possible above extension)

for me, the best free tool for tcpsniffing is, with no doubt, TcpCatcher ( ; SOAP messages are loaded in convenient tree views, messages can be edited at network level, it is crossplateform, works over SSL, ...
I found wireshark to be quite useful

ben tcpcatcher is nice.
Hi Robert, Ben

I use Wireshark frequently, and it is a very good tool for advanced troubleshooting. I chose however not to use it for this post because, in my opinion:
  • It is harder to understand the results. Wireshark will give you more detailed information (at all layers above Ethernet) that for the purpose of the type of troubleshooting usually done is not needed. Typically one only wants to actually see the text in the TCP layer;
  • It must be installed. This is a huge con when troubleshooting production environments. With TCPTrace, you simply deploy the executable and delete it at the end;
  • It will typically require WinPcap (I myself never got good results without it, but I might be doing something wrong), which is yet another con for production environments. Not because it actually interferes, but because sysadmins will not appreciate that.
However, TCPTrace has a huge con - it is very intrusive, in the way you must divert traffic to it instead of just sniffing.

I will look at TCPCatcher (in my first experiments with my Win7 machine I am having multiple issues), and I don't know the level of Java dependency it will have (OutSystems .NET servers will not require Java installed). Maybe Ben could post a quick how-to to using that tool, so others could benefit from that?

Hi all

I just found (kudos to Nuno Teles) Fiddler. It has some advantages on TCPTrace (requires less configuration, just install & run) and can be easily used for all scenarios. The inconvenient is that it must be installed, so be sure that installing it is approved by corporate IT.

I believe that, at least for scenarios I and II, it is a better option than using TCPTrace.

Fiddler can be found at:

Hi All,
Sorry for late answer.
I used TcpCatcher recently again to debug a Web Service client program within Visual Studio 2008 and it worked well for me.
It can't be more simple, just start it and HTTP traffic get automatically captured (from Visual Studio and from IE as well), there is just nothing to do (well you need a java runtime indeed since it is a java app).
How does it work ?
I suppose it modifies your local Windows proxy settings when it starts and becomes your default proxy (just like Fiddler does, doesn't it ?). Once you close it, your usual proxy settings get restored.
What is quite nice with that tool, is that you can turn on a debug mode (called "catch mode") and packets can be modified on the fly. You can also build a basic transformation (add or remove HTTP headers, search and replace HTTP content) with a GUI called "hook builder". The design of the interface is much more modern and clear than Fiddler to me. What do you think ?
I am using Windows Vista, and I didn't try it with Windows 7.  
There is a bunch of detailed tutorials provided here, dealing with different use cases.

Hi All

Just adding more information for those using the Agile Platform for Java.

Although the WireShark, TCPTrace, TCPCatcher and Fiddler are great tools for Windows based systems, for Linux server boxes there are also some great tools to trace the HTTP protocols that can be used to capture the web service soap communications packets.

Wireshark also runs on Linux boxes, but it requires an X server, which usually our Agile Platform for Java servers don't have (ain't needed and consume extra resources).

So the key is to use the TCPDUMP tool that comes along with the default Redhat packages, and allows to capture all kind of TCP traffic into a file. Then, one can actually load the file on the WireShark for processing and analysis.

For tracing HTTP protocol on a Redhat box, use the following command line syntax for the TCPDUMP, assuming the interface eth0 is where the traffic goes through:

tcpdump -i eth0 -w httpdump.cap -s 65535 host webservicehostname and tcp port 80

Then you can load the httpdump.cap file into the WireShark and perform the analysis of the captured data.

A good example of how to use it can be found at

Happy linux traces.


Miguel Simoes Joao

Just an addenda to this topic - starting with Agile Platform 5.1 communications between Service Studio and Platform Server are by default via HTTPS. If you are troubleshooting communications in this scenario, it is easier if you force communications via HTTP. For that, simply append :80 after the server name, when connecting with Service Studio.