I'll admit it, I really like graphs. I'm a visual person. So it's not too surprising I'd feel the urge to come up with another stats page, this time showing some server performance metrics
. This page is even more useless for the casual browser than some of my other charts, but from an administration point of view it's quite telling.
What this chart is looking at is the average time it takes to generate pages on the site -- specifically, the home page, all pages on average, and RSS feeds. The chart also includes the number of app recycles occurring on the server -- not directly related to page generation time but a good metric of performance. In general, these "generation times" are related to performance bottlenecks on the server. They're also influenced by the connection speeds of the client ... but from a ratio standpoint it's virtually all server and database performance. Obviously, the lower the number, the better.
I blogged not too long about implementing a sessionless state handler
on the site. The goal at the time was to work around the large number of app recycles I was seeing to provide a faster, cleaner, and web farm friendly state mechanism. It's been fantastic so far. While this solves the issue of state, it doesn't solve a caching issue: when an app pool recycles, the cache is flushed. Looking at two days this month (10/5 and 10/8) there were about 45 application recycles each day! Most days are lighter; some are worse (I nearly fell out of my chair when I saw 90 app recycles on 9/20 -- that's nearly 1 every 15 minutes).
It's pretty hard to run a performant site when the cache is getting flushed this often. And to boot (pun intended), the performance on a shared hosting/shared database leaves a lot to be desired (though, the pricing is pretty slick so I can't complain), so it's really critical to have a good caching solution. In fact, in June I had pushed a copy of my site where I forgot to remove some debug code that prevented the RSS feed from getting cached. It's pretty easy to see the effect this had in the chart.
Over the past few months I have been fine-tuning the caching, and it's helping. (Technical note: I have a delegate fired to log when elements are removed from the cache -- and it's clear to me that these servers are hungry for resources as the reason for the removal are typically low resource warnings or application restarts). Increasing the item's priority in the cache seemed to help (done in the June-July timeframe and visible in the graph with reduced generation times). Ah, the realities of shared hosting.
In my latest reports, though, these chart pages have proven to be the worst performers -- and it's no surprise. The queries that build these charts are very slow, and it so happened that about half the time the pages were hit, they weren't in the cache. Querying the data is 90% of the execution time, rendering the graphs is the remaining 10%. Since the output is a binary image, I decided to alter the caching approach for these pages. Going forward, nothing is in the cache, and these pages simply look for the correct PNGs in the file system, bypassing the charting controls and database altogether.
Based on the file timestamp, the app will decide if it should render a new chart or not. After all, the traffic from November of '04 will never change, so the chart from this month might as well live on indefinitely. Even for charts we want refreshed -- like this months browser or traffic stats -- we don't need them redrawn from scratch every app recycle. Using the file-based "cache" works wonders at speeding these up, allowing for a more controlled refresh cycle. (This kind of caching approach was also done with my image resizer library
, since resizing hundreds of thumbnails each app recycle was not