Geocortex Blog

One of the numbers I keep track of while working on the Optimizer project is the number of outstanding issues logged against it. We really try hard to ensure that each version of Optimizer is released with an empty bug database. The reason is simple. During a previous coding gig, I calculated that each bug, on average cost about $350 to fix. To come up with that number, I added the salaries of the programmers dedicated to fixing bugs together and divided by the number of bugs they fixed over a year. One of the products had plenty of issues so the getting supporting data was not hard to come by. I didn’t factor in overhead costs or the amount of free juice they consumed. $350 represents the average cost in terms of straight salary. Obviously if we discover bugs before our customers do, the cost is less. If we can automatically find bugs by writing good quality unit tests, or keep bugs from getting into the product in the first place, the cost is much less.

The cost of quality is not negligible and must be taken into consideration when planning projects. Adequate time for doing proper design, writing unit tests, and performing user acceptance testing must be budgeted for. Having said that, I would argue that the cost of quality is often less than we think. An awesome side effect of higher initial quality is that our developers get to spend their time adding new functionality to the product which has a positive impact on the morale of the development team (who doesn’t like doing new development), not to mention happier support staff (fewer issues to deal with) and very happy customers (feature packed releases that help them do their jobs better).

I’ve been battling all week with space issues in Geocortex Optimizer. For those of you not familiar with it, Geocortex Optimizer is our new product that collects information regarding your ArcGIS Server installation and presents this information to you a number of ways in order for you to make good management decisions regarding your GIS investment.

When you install Optimizer for the first time, it examines your system, looking for log files from both IIS and ArcGIS servers. It then reads these log files, massages them and stores them away in a SQL Server database for further analysis. The problem is that the amount of information collected can be huge particularly if you have a busy site that you have been running for a while. On one server we tested, we processed more than 8 million records in a little over a few hours before we ran into a capacity bottleneck.

One of Optimizer’s features is that it will not loose data if the SQL Server database it is using is down. When this happens Optimizer quietly serializes the data to disk that it would have written to the database. When the database becomes available again it writes those saved records to it. The problem I discovered is that .NET datasets serialized to xml are really large. I considered using binary serialization but then a coworker suggested that I serialize to a GZip steam instead of a regular file stream as I had been doing. This sounded promising because database data usually compresses 20 to 1. I had no idea how easy it would be. Check out the following snippet taken from Optimizer

using (Stream stream = new FileStream(filename, FileMode.Create))

{

  using (Stream StreamOut = new GZipStream(stream, CompressionMode.Compress))

  {

    ds.WriteXml(StreamOut, XmlWriteMode.WriteSchema);

  }

}

These few lines of code resulted in a 40x space savings over uncompressed xml. I’ve never used GZIP streams before but I’m beginning to think of all kinds of applications where they could be really be useful.