Or how I learned to stop worrying and love the New Relic monitoring service
|The very useful button|
I cannot stress how handy this is. The problem we had did not show up when load testing, where attaching a traditional profiler would be easy. And we have real users using the service, so we cannot afford downtime to stop the application and attach the profiler… so click, click… wait 5 minutes and analyse the results.
|The version with the bug fix was deployed some time around 04:20 (not my time zone).|
- One of the issues with Load Testing is that you have to define your scenarios, and while you can add some elements of randomness to each scenario, you cannot give that true randomness that occurs with real users. In our case I had not put sufficient randomness along the code path that had the bug, so my caching layer was hot on those values and was masking the issue. Real users are not so kind to your code!
- How do you load test the code path where new users sign up via a specific third party OAuth service? You’d need to create a large number of fake accounts on that service and feed them into the load tester… or you cheat and do what I did and just use 2-3 and wipe their records out of the DB every 10-15 seconds… yep… just begging for a hot cache disguised bug!
Stephen Connolly has nearly 20 years experience in software development. He is involved in a number of open source projects, including Jenkins. Stephen was one of the first non-Sun committers to the Jenkins project and developed the weather icons. Stephen lives in Dublin, Ireland - where the weather icons are particularly useful. Follow Stephen on Twitter and on his blog.