Daily Temperature This Year:<\/strong><\/p>\n Once you look at the actual daily temperature, it becomes clear what happened to your power bill.\u00c2\u00a0 Last year, the temperature was fairly constant, but this year, there were wild temperature swings.\u00c2\u00a0 You had your AC cranking full blast for the first part of the month, then you kept nudging up the thermostat during the end of the month.\u00c2\u00a0 However, since the temperature extremes offset one another, the average temperature\u00c2\u00a0makes it seem like both months had the same weather.<\/p>\n That’s the key:\u00c2\u00a0 Once you look at the data, it’s often clear that the average doesn’t tell the whole story, or worse, tells the wrong story.\u00c2\u00a0 And the problem is that people typically don’t look at the data.<\/p>\n Let me pull this back to the world of software performance testing and tell a story about a product I’ve worked on and how I first came to realize that typical performance testing was dead wrong.\u00c2\u00a0 The product was a keyword processor at the heart of a larger system.\u00c2\u00a0 Our customer had demanded a 1.5 second response time from our overall\u00c2\u00a0system, and that requirement\u00c2\u00a0got chipped down and used up by other components on its way to the keyword processor I was involved with.\u00c2\u00a0 By the time it got to us, we only had a response time cap of 250 ms in which to return our data, otherwise the larger system would give up on us and continue on its way.\u00c2\u00a0 So, great, I thought.\u00c2\u00a0 I’ll just load up the system with an ever increasing number of concurrent requests and find out how many requests we can process at the same time before our average hits 250 ms.\u00c2\u00a0 I did that and came up with 20 requests at once.<\/p>\n So we set up enough\u00c2\u00a0machines to handle the anticipated load with a maximum of 15 requests per box at one time, so we’d have some room to grow before we needed to add capacity.\u00c2\u00a0 All was well.<\/p>\n Until, that is, the day we launched and our keyword processor kept getting dumped on the floor by the system above us.<\/p>\n Something was obviously wrong.\u00c2\u00a0 We knew what our performance was.\u00c2\u00a0 At 20 concurrent\u00c2\u00a0requests, we had a 250 ms response time, at 15\u00c2\u00a0requests, we has seen an average 200 ms response time.\u00c2\u00a0 We’re fast enough, and we’d proved that we were fast enough.\u00c2\u00a0 Statistics said so!<\/p>\n That right there was the problem.\u00c2\u00a0 We trusted the wrong information.\u00c2\u00a0 Sure, the average response time was 200 ms at the load we were seeing, but that said absolutely nothing about the\u00c2\u00a0something like 30% of the requests that\u00c2\u00a0were hitting the 250 ms timeout.\u00c2\u00a0 We frantically reran the performance tests.\u00c2\u00a0 The results were stunning.\u00c2\u00a0 While the average response time did not hit 250 ms until we reached the 20 concurrent request level, we saw a significant (and SLA violating) number of requests that took more than 250 ms by the time we reached the 10 concurrent request level.<\/p>\n People aren’t very happy when you tell them that a cluster size has to double…<\/p>\n At the time, I thought I might have just made a rookie mistake.\u00c2\u00a0 It was the first major system I’d done the performance testing for, and I’d had no training or mentoring.\u00c2\u00a0 I did what I thought was right and ended up getting burned.\u00c2\u00a0 Surely, I thought, everyone else knows what they’re doing.\u00c2\u00a0 Real performance testers using real performance testing tools will get it right.\u00c2\u00a0 Trouble is, I’ve since discovered that’s not the case.\u00c2\u00a0 Everyone else makes these mistakes and they don’t even realize that they’re making any mistakes.\u00c2\u00a0 And performance testing tools actively encourage testers to make these mistakes by not giving the tester the information that they really need and, in some cases, giving testers information that is just plain invalid. ((I’ve seen several cases of odd numbers coming out of VS Perf Tests, but the one I’m specifically thinking of here is the fact that VS will still report an average response time, even when you’re ramping up the number of users.\u00c2\u00a0 The performance of your system when you have a single user is vastly different than when you have 100 users, so the single “Average Request Time” number that it will report is just plain useless.))<\/p>\n So…\u00c2\u00a0 What do you do about it?\u00c2\u00a0 For starters, don’t use the average in isolation.\u00c2\u00a0 Pull in some other measurement.\u00c2\u00a0 I like using\u00c2\u00a0the 95th percentile for a second opinion. ((You can get the 95th\u00c2\u00a0percentile in\u00c2\u00a0Visual Studio, if you tweak a\u00c2\u00a0setting.\u00c2\u00a0 Set “Timing Details Storage” to “Statistics Only” or “All Individual Details” and it’ll start being recorded.))\u00c2\u00a0 The 95th percentile means that 95% of all requests take less than that amount of time.\u00c2\u00a0 That’s really more what you care about, anyway.\u00c2\u00a0 You’re probably not really concerned with the 5% that lie beyond that point, since they’re usually outliers\u00c2\u00a0or abberations in your performance anyway.\u00c2\u00a0 This will get rid of things like Mr. Lucky’s lottery winnings.\u00c2\u00a0\u00c2\u00a0 Additionally, you’re\u00c2\u00a0probably not really concerned with where the average response time lies.\u00c2\u00a0 People often use the results of performance testing to feed into capacity planning or SLAs.\u00c2\u00a0 When there are dollars on the line tied to the Five Nines, why do you care about a number that you think of as the middle of the road?\u00c2\u00a0 You care about the worst case, not the average case.\u00c2\u00a0 If we’d used the 95th or 99th percentile in our inital performance tests of that keyword processor, we would not have had the problem that we did.<\/p>\n But even so, the 95th percentile has its own set of issues and should also not be used in isolation.\u00c2\u00a0 It, too, does not tell you the complete story, and can easily hide important\u00c2\u00a0trends of the data.<\/p>\n There I go again with “the data”.\u00c2\u00a0 You have to look at the data. \u00c2\u00a0In order to get the full picture of your system performance, you actually\u00c2\u00a0have to look at the full picture of your system performance.\u00c2\u00a0 You can’t go to single aggregate\u00c2\u00a0numbers here or there and call it good.<\/p>\n Of course, that leads to the obvious problem.\u00c2\u00a0 When you run a performance test, you’ll often end up running thousands upon thousands of iterations, collecting thousands upon thousands of data points.\u00c2\u00a0 You cannot\u00c2\u00a0wrap your head around that kind of data in the same way that you can see Mr. Lucky’s income or the daily chart of the temperature.<\/p>\n Well, not without help…<\/p>\n \u00c2\u00a0Whenever I do performance testing now, I rely on a tool that I built that will produce several graphs from the results of a performance test run.\u00c2\u00a0 The simplest graph to produce is a histogram of response times.<\/p>\n A histogram will show you the distribution of the response times for the performance test.\u00c2\u00a0 At a glance, you can see the where the fastest and slowest responses lie, and get a sense for how the service behaves.\u00c2\u00a0\u00c2\u00a0Are the response times consistent, with most of them around 150 ms, or are they spread out between 100 and 200 ms?\u00c2\u00a0 A histogram can also show you when things are acting strangely.\u00c2\u00a0 One piece of software I tested had most of the response times centered\u00c2\u00a0below\u00c2\u00a0400\u00c2\u00a0ms, but there was a secondary bump up between 500-900 ms.\u00c2\u00a0 That secondary bump indicated something was strange, perhaps there was a code path that took three times as long to execute that only some inputs would trigger, or there were random slowdowns due to garbage collection or page swapping or network hiccups.<\/p>\n As you can see, there are a sizeable number of responses in that bump, enough to make you want to investigate the cause.\u00c2\u00a0 This potential problem would have been completely invisible if you were only concerned with the average, and the full extent would not have been known if you’d been looking at the 95th percentile. ((Likely, the 95th percentile would lie in the middle of the bump, leading you to believe that the system performance is slow overall, not that there’s an anomaly.))\u00c2\u00a0 However, it’s plainly visible that something strange is going on when you see the graph like that.<\/p>\n While helpful, simple histograms like this are not enough.\u00c2\u00a0 In particular, they fall down if anything changes during the test run.\u00c2\u00a0 If something like network latency slows down your test for a brief period of time, that will be invisible in this graph.\u00c2\u00a0 If you’re changing the number of users or the number of concurrent requests, then the times from all of them get squeezed together, rendering the graph invalid.\u00c2\u00a0 What you’re missing here is the time dimension.<\/p>\n One way to bring in the time dimension is to animate the histogram.\u00c2\u00a0 You produce multiple histograms, each representing a slice of time in your test.\u00c2\u00a0 That way, you can watch the behavior of your service change over the length of the performance test.\u00c2\u00a0 The problem I have with an animated histogram is that you can’t just glance at the information.\u00c2\u00a0 You have to watch the whole thing, which can be time consuming for a long running performance test.<\/p>\n Instead of animation, I prefer to visualize the time in this way:<\/p>\n \u00c2\u00a0 Going up the side, you have divisions for buckets of response times.\u00c2\u00a0 Going across, you have time slices. ((Time isn’t marked in these examples because the tool I use to generate them\u00c2\u00a0will usually generate other graphs, as well, making it possible to correlate the time on that graph with the\u00c2\u00a0time on this graph.\u00c2\u00a0 For these examples, it’s not really necessary to know how much time\u00c2\u00a0each column is or how many requests are represented by each block.))\u00c2\u00a0\u00c2\u00a0 Essentially, this is what you’d see if you stacked a bunch of histograms together side by side, then looked at them from the top.\u00c2\u00a0 It’s basically like a heat map or a graph of the density of the response times.\u00c2\u00a0 The red zones have the most responses, while the green areas have the least. ((And the white zones are for loading and unloading only.\u00c2\u00a0 There is no parking in the white zones.))\u00c2\u00a0 In the example above, you can see that there are a lot of responses in the 100 and 150 ms buckets and then\u00c2\u00a0it quickly trails off to zero.\u00c2\u00a0 There’s\u00c2\u00a0a lot of\u00c2\u00a0noise up to about 600 ms, and sporadic outliers above that.\u00c2\u00a0 The performance remains fairly stable throughout the test run.\u00c2\u00a0 All in all, this is a fairly standard graph for a well behaved piece of software. ((For comparison,\u00c2\u00a0this graph\u00c2\u00a0is from the same test run that produced the “Average Response Time: 149 ms” histogram that I showed earlier.))<\/p>\n These density graphs aren’t terribly interesting when things are well behaved, though.\u00c2\u00a0 Here’s another graph I saw:<\/p>\n First, there’s bands of slowness between about 210 and 280 and 380-450.\u00c2\u00a0 These bands would appear in a histogram like the secondary hill shown above.\u00c2\u00a0 But what a histogram isn’t going to show you is the apparent pattern in the 380-450 band.\u00c2\u00a0 It appears that there’s groups of slow responses in a slice of time, then none for the next couple of slices, then another group of slow responses, then none, and so on.\u00c2\u00a0 Seeing this kind of behavior can help you find the problem faster.\u00c2\u00a0 In this case, the slow responses may be caused by something else running on the box that’s scheduled to run at a regular interval, like an anti-virus scanner or a file\u00c2\u00a0indexer, or they can be caused by something like\u00c2\u00a0the garbage collector waking up to do a sweep on a somewhat regular basis.<\/p>\n Another benefit of a density graph is that they’re still useful, even if you change the parameters of the test during the test run.\u00c2\u00a0 For instance, a common practice in performance testing is to increase the load on a system during the run, in order to see how performance changes.<\/p>\n![\"ElectricityBill-LastYear\"](\"https:\/\/mathpirate.net\/log\/wp-content\/uploads\/2009\/09\/ElectricityBill-LastYear.png\" "\\"ElectricityBill-LastYear\\"")
<\/a><\/strong><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n