Unlike other industries where performance testing is often an after-thought, the retail industry cannot afford to have this approach. After all, Black Friday was last holiday season’s first billion-dollar-plus online shopping day – up a hefty 15% over the year before.
We are often asked by our customers as to how holiday readiness testing is different from routine testing and why should they invest in it.
How Holiday Readiness Testing is different?
Standard user workload model will not work
The arrival pattern of users during the holiday season is different. Not only are customers shopping online, they’re also increasingly shopping via their mobile devices. A user workload model has many attributes: user load levels, user arrival pattern, distribution of user activities etc. All these attributes are different on Black Friday. The load levels are almost thrice as that of peak levels during the rest of the year and the arrival pattern is burst in nature. Large number of users arrive in short bursts of time. It’s imperative to take the marketing strategy into consideration, the expected increase in number of users etc.
More stringent acceptance criteria
Due to the sheer volume of business and transactions on Black Friday, it is important that there is absolutely no leniency in defining stringent acceptance criteria – 0% error rate during peak hours, end user response time of less than two seconds (varies for customers), less than 75% infrastructure resource utilization etc
Point of stress is completely different
Load levels are higher than peak levels observed during the year. Order conversion is critical, hence shopping cart and payments need to be stressed more than what is usually done during normal load testing. It impacts the behavior of apps/DB servers and resource consumption of the physical hardware.
Application deployment architecture could be different
Having a separate database for products on offer is a good strategy. This is done to avoid negative impact on other parts of the website and improve overall experience of the end users. Retailers adopt different load balancing algorithms for better management of traffic across geographies and channels (stores, desktop, mobile / tablet). Configuration settings for App server, DB server and Caching could be different.
Content Delivery Network (CDN) testing
CDN is used to reduce traffic on original server for serving static content. If CDN does not work properly then it may have negative impact on web/app servers, resulting in longer response times. While defining the holiday readiness strategy, working with the CDN provider early-on during planning becomes critical to understand the caching strategy. During testing, the offload ratio from CDN should be validated.
Detection and prevention of BOT traffic
BOT traffic is that part of online traffic and activities which is artificially generated by automated robots and spiders. BOT traffic is evidently hard to evaluate, but according to several sources it can be estimated to range from 10 to 20 % of the total traffic. Since retailers would not want to stop all of the traffic generated BOTs, it is vital to have a customized detection strategy. During testing, it is important to test the mechanism developed to detect BOT traffic and develop test strategies for prevention/throttling.
Many a times, retailers adopt production testing since the testing environment is not configured or sized like production. Production testing provides a mechanism to evaluate the performance during peak production load in all business critical scenarios. Some of the key areas to be monitored while planning for production testing are:
Holiday readiness testing is totally a different ball game and traditional performance testing practices may not provide the necessary business assurance in this digital market. With proactive and exhaustive planning, and tighter collaboration with the stakeholders, this exercise can lead to huge business gains and increase in number of satisfied customers during the holiday season.
To know more about the holiday readiness framework Register here.