#learning bugs

We have a front-end system, which talks different web services and high-level setup is as follows:

--> (load-balancer) webservice1 (2 hosts) | user ---> front-end --- | --> (load-balancer) webservice2 (2 hosts)

After a deployment we started intermittently getting errors in the application. We could just back out the change, but since we didn’t have live users yet, so we had some time to investigate.

Looking at event logs and application logs, the front-end was calling webservice1, but errored out when calling the second service.

In addition, we found the IIS access from the machine hosting webservice2:

#Fields: time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip sc-status sc-substatus sc-win32-status 08:49:11 xx.xx.xx.31 POST /xxx.svc - 80 - xx.xx.xx.252 200 0 64 09:02:25 xx.xx.xx.31 POST /xxx.svc - 80 - xx.xx.xx.252 200 0 64

Hmm, a win32-status code of 64? That shouldn’t be due to code changes, right?

We wanted to check which host was terminating the connection (front-end or one of the webservices), so and a Wireshark capture of the traffic shows what was happening:

Request was sent to webservice2, the server responded, but suddenly, front-end closed the connection! (notice the RST in the end of the line). That’s weird.

A ‘Failed Request Tracing’ would have helped, but sadly we ran IIS 6 at the time.

So there’s something in this data that front-end isn’t liking very much. The fact that we had a win32status set meant that the request wasn’t handed over to the .NET runtime.

Let’s look at the HTTP response we got more closely, before RST was sent:

HTTP/1.1 200 OK X-Served by: xxx Content-Length: 14315 Cache-Control: private Content-Type: text/xml; charset=utf-8 <envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"> // snipped...

Notice the header X-Served by has a space character before “by”. We usually add this header as it helps to know which server we’re dealing with in case of issues.

To make the matters worse, we had only made this error in server, the others were setup without the space. That actually takes care of the “intermittent” part :)

Turns out, you’re not supposed to have a space in the name of the header, as we can see (here)[https://tools.ietf.org/html/rfc822#section-3.1]

The field-name must be composed of printable ASCII characters (i.e., characters that have values between 33. and 126., decimal, except colon). The field-body may be composed of any ASCII characters, except CR or LF. (While CR and/or LF may be present in the actual text, they are removed by the action of unfolding the field.)

It seems so obvious now, but at the time, we had a REALLY hard time trying to find that space.

This was actually added in error during our deployment. Since the receving server tried to parse the first chunk of the response, and errored out it terminated the connection!

So it turnes out code changes could also be the cluprit if we have set an HTTP header in our web.config file.

So, there you go with the summary:

The client machine says error invoking webservice1

Web Service is getting invoked but the client terminated the connection

IIS access logs show 200 0 64