Phy-gital Roundtable: Breakfast Roundup from Germany and Netherlands

02 May '15 | Debjyoti Paul

German Shoppers: Meet Them in the Fast Lane to Phy-gital

15 January '15 | Ralf Reich

Shoppers Will Share Personal Information (But They Don’t Want to be “Friends”)

15 January '15 | Anil Venkat

Modernize or Perish: Property and Casualty Insurers and IT Solutions

14 January '15 | Manesh Rajendran

Benelux Reaches the Phy-gital Tipping Point: Omnichannel Readiness is Crucial

13 January '15 | Anil Gandharve

The New Omnichannel Dynamic: Finding Core Principles Across Industries

13 January '15 | Debjyoti Paul

Technology does not disrupt business – CIO day 2014 Roundup

02 December '14 | Anshuman Singh

Apple Pay – The Best Is Yet To Come

02 December '14 | Indy Sawhney

Digital transformation is a business transformation enabled by technology

01 December '14 | Amit Varma

3 Stages of FATCA Testing and Quality Assurance

06 October '14 | Raman Suprajarama

3 Reasons why Apple Pay could dominate the payments space

18 September '14 | Gaurav Johri

Beacon of Hope: Serving Growth and Customer Satisfaction

05 August '14 | Debjyoti Paul

The Dos and Don’ts of Emerging Technologies Like iBeacon

30 July '14 | Debjyoti Paul

What You Sold Us On – eCommerce Award Finalist Selections

17 July '14 | Anshuman Singh

3 Steps to Getting Started with Microsoft Azure Cloud Services

04 June '14 | Koushik Ramani

8 Steps to Building a Successful Self Service Portal

03 June '14 | Giridhar LV

Innovation outsourced – a myth or a mirage or a truth staring at us?

13 January '14 | Ramesh Hosahalli

What does a mobile user want?

03 January '14 | Gopikrishna Aravindan

How Holiday Readiness Testing is different?

Posted on: 20 August '14
Abhishek Raj
Head, Performance Engineering

Unlike other industries where performance testing is often an after-thought, the retail industry cannot afford to have this approach. After all, Black Friday was last holiday season’s first billion-dollar-plus online shopping day – up a hefty 15% over the year before.

We are often asked by our customers as to how holiday readiness testing is different from routine testing and why should they invest in it.

How Holiday Readiness Testing is different?

  • Difference in user behavior – Standard workload model will not work
  • Edge case scenarios need to be tested
  • More stringent production SLAs
  • Point of stress is completely different
  • Change in application deployment architecture
  • Content delivery network testing is recommended
  • Detection and prevention of BOT traffic
  • Production testing might be needed

Standard user workload model will not work
The arrival pattern of users during the holiday season is different. Not only are customers shopping online, they’re also increasingly shopping via their mobile devices. A user workload model has many attributes: user load levels, user arrival pattern, distribution of user activities etc. All these attributes are different on Black Friday. The load levels are almost thrice as that of peak levels during the rest of the year and the arrival pattern is burst in nature. Large number of users arrive in short bursts of time. It’s imperative to take the marketing strategy into consideration, the expected increase in number of users etc.

More stringent acceptance criteria
Due to the sheer volume of business and transactions on Black Friday, it is important that there is absolutely no leniency in defining stringent acceptance criteria – 0% error rate during peak hours, end user response time of less than two seconds (varies for customers), less than 75% infrastructure resource utilization etc

Point of stress is completely different
Load levels are higher than peak levels observed during the year. Order conversion is critical, hence shopping cart and payments need to be stressed more than what is usually done during normal load testing. It impacts the behavior of apps/DB servers and resource consumption of the physical hardware.

Application deployment architecture could be different
Having a separate database for products on offer is a good strategy. This is done to avoid negative impact on other parts of the website and improve overall experience of the end users. Retailers adopt different load balancing algorithms for better management of traffic across geographies and channels (stores, desktop, mobile / tablet). Configuration settings for App server, DB server and Caching could be different.

Content Delivery Network (CDN) testing
CDN is used to reduce traffic on original server for serving static content. If CDN does not work properly then it may have negative impact on web/app servers, resulting in longer response times. While defining the holiday readiness strategy, working with the CDN provider early-on during planning becomes critical to understand the caching strategy. During testing, the offload ratio from CDN should be validated.

Detection and prevention of BOT traffic
BOT traffic is that part of online traffic and activities which is artificially generated by automated robots and spiders. BOT traffic is evidently hard to evaluate, but according to several sources it can be estimated to range from 10 to 20 % of the total traffic. Since retailers would not want to stop all of the traffic generated BOTs, it is vital to have a customized detection strategy. During testing, it is important to test the mechanism developed to detect BOT traffic and develop test strategies for prevention/throttling.

Production testing
Many a times, retailers adopt production testing since the testing environment is not configured or sized like production. Production testing provides a mechanism to evaluate the performance during peak production load in all business critical scenarios. Some of the key areas to be monitored while planning for production testing are:

  • Escalation matrix: Define clear roles and responsibilities to handle alerts, with clearly defined standard operating procedures with Marketing, Analytics, Infrastructure, DBA, Ops and other key stakeholders. This will ensure that early warnings are not missed and critical incidents are avoided in production environment
  • Batch job management: Batch jobs are both time and resource consuming, and they impact the overall performance of the website. Change in timings of batch jobs to non-peak hours helps minimize the impact. The change in batch schedule should be tested to ensure there are no surprises during the actual holiday season.

Holiday readiness testing is totally a different ball game and traditional performance testing practices may not provide the necessary business assurance in this digital market. With proactive and exhaustive planning, and tighter collaboration with the stakeholders, this exercise can lead to huge business gains and increase in number of satisfied customers during the holiday season.

To know more about the holiday readiness framework Register here.

Abhishek Raj

Abhishek Raj has 12+ years of experience and is heading the performance engineering practice at Mindtree. He specializes in setting up Performance Testing and Engineering CoEs. Has proven experience in QA program management; project management and test management; and designing and implementing SDLC suites.