Oh That Innocent Looking Page !

So, you are building a web application that talks to bunch of other applications, shared databases and other web services which aren't in your control. Sounds like a typical project, isn't it ? :-)

Well, I was the (only) tester in such a project sometime back. Maintaining a working test environment was a pain. Broken connectivity between dependent applications, databases was a recurring problem. To make my life as well as environment team's life easy, developers came up with this idea.

A simple html page that shows the connection status of different web services, databases in test environment.

When it was introduced in mid release cycle, I was happy :-) Now I don't have to worry much when the transactions aren't going through. Just check the status page, if any connectivity is broken, shout across to Environment team, get the things fixed. Life is simple and worry free !

We decided to introduce the same status page to Production System as well, with limited access, only to environment team. After all, it made all of our lives easy.

So, moving on to 'Go Live' Day. We run the deploy scripts, I run the sanity tests, things were looking good.

In a matter of an hour or so, environment team got in touch with us, apparently production systems were receiving a huge volume of data. At first shot, it looked like our application is the originator of the traffic.

So, we had to take our application down (luckily it was on a weekend, not much work going on) while we debug the issue.

The way status page was supposed to work was to send a 'Are you up?' question to different systems and was expecting a 'Yes' or 'No' answer. But one of the shared library we used had a bug. Instead of sending a response, it sent back a request, for which our application promptly sent a response. This got into an infinite loop, hence the surge in the traffic. Well, the problem was quickly solved and we went home anyway, albeit little late.

Apparently same thing was happening in Test environment, but me being the only user, there wasn't much havoc. It went unnoticed.

This incident got me in to thinking.

You see, I, as a tester, am worried about the stories, functionality and bunch of other things like performance and security. I wouldn't doubt much about these small tools we use that make our work easy. Big mistake.

As a 'only' tester in my team, it's my responsibility to check every new thing that's being introduced in to the application. I should be on the top of the things, questioning and doubting every little piece of software that gets developed.

So, a lesson learnt: Never ever look at a piece of software and think it's harmless. Not even a simple innocent looking html page.

TL;DR : I goofed up in testing, learnt a lesson.

:-)