PTLC - Requirement (NFR) Gathering

PerfMatrix Site Link: https://www.perfmatrix.com/non-functional-requirement-gathering/

As per my and other’s experience (which I heard), I think this is the toughest phase of performance testing. Many times a performance tester face challenges to collect the correct performance testing requirement and conclude them as a proper client's expectation.

In most of the cases, applications are new and hence a performance tester does not have any clue on the NFRs (non-functional requirements). I have seen many times, the client randomly pick a number over the call and ask to perform NFT (non-functional testing) against that number and when those NFRs do not meet at the end of the testing they forced to modify them and pass the test. Such performance tests are meant for just to showcase that we have done the performance testing and the application is performing very well. But in reality, 82% of the applications are failed within a year, even in the less time period, due to performance issue when users load increases in production. If you did not face such a situation then you are lucky and if you faced it how did you overcome the situation? I am keen to know (please write in the comment section).

Understand the application architecture: In this phase, you may have to set-up multiple calls and meetings with the client, business associate, application architect, developer, and QA team. It is advisable to collect all the information about the application and its architecture as much as you can and make some handy notes. 

Refer Project Document: The software development documents like HLD (high-level design document), LLD (low-level design document), AO (Architecture Overview document) etc. help a lot to understand the design of the application. 

Set-up Meetings/Calls: In the absence of these documents one-to-one meetings (calls) or combine meetings (calls) with all the stakeholders fulfil the purpose. 

To get the performance requirements from non-technical customers is a valuable question that everyone has. In most cases, customer expectations are either incompletely defined or more conceptually defined. We need to better understand their expectations by asking the right set of questions and translating their conceptual requirements into more quantitative goals. A performance tester needs to play the role of a modem translating between the novice user language and performance testing terminologies. It is the performance tester’s responsibility to get the essence from the customer. One needs to quickly understand the customer and should start talking in their language. Don’t talk using performance testing jargons and make the customer feel that they are ignorant, which is not the case. Don’t expect that your customer will give you the performance goals in a single shot. Rather, start the discussion with an overview of important terminologies like what is the response time, transactions, hits, why is it required before starting the performance tests, etc. so that the customer feels comfortable to explain what he wants. Try to build a good rapport with your customer. During the meeting/call, you can educate the customer and explain with an example to understand what they are expecting from the system and also explain about how you have solved the issues in some other projects in your past. Once you get that, you will have the full liberty to state your expectations from them in order to provide successful results. The most important thing which I always avoid not to scare the client by asking lots of question in one shot or talking too much technical. Your way of asking the requirement should be very polite to impresses the customer. This behaviour helps at the time of test execution when you raise some bugs and developer try to shift the pressure on you by asking the illogical questions which are out of their field. 

The complete understanding of the AUT with all its features is the basic step in requirement gathering activity. You can’t thoroughly test an application unless you have its complete understanding. Instead of explaining the PTLC theoretically, we will understand PTLC with a real-life example:

Let’s take a simple example to understand the complete PTLC. 
An e-commerce website with 3-tier architecture needs to be tested. A performance tester (name Perfingo) set-up some initial meetings with project stakeholders to understand the application architecture and gather the non-functional requirement. After understanding the application architecture, Perfingo got the following requirement:

1. The application should be very fast.

2. The response time of the application should be quick.

3. The web server performance should be as high as possible.

4. The application should support many users.

5. The application should not fail when a sudden load comes during sale and offer periods.

6. The application should run without any failures for a long duration.

From a client point of view, he has given the all the requirements and set his expectation, but from Perfingo perspective, he just got the conceptual requirements. Now, Perfingo explained that the provided information is partial and cannot help him to defined performance testing goal. With a polite manner, he gained the customer's confidence and convinced him to provide the quantitative requirement. He requested project stakeholders to answer some of his basic questions. At last, he managed to gather following requirements in next couple of days:

Q. How many types of users are using this application (GUI)?
A. 4 types of users
1. Admin
2. Seller
3. End-user
4. Call center employee

Q. What are the business scenarios of every user?
A. Business scenarios for each type of users are like: 

Admin: 

1. To approve/reject a new seller

2. To verify a newly added product and approve/reject it

Seller:

1. To add a new product

2. To delete the existing product

Buyer:

1. To buy a product

2. To cancel the order

Call Center Employee: 

1. To register the complain

Q. What are the AUT current and predicted peak user load for all its users’ actions over time?
A: 
Admin

Current: 4 

Predicted: 10

Seller

Current: 25

Predicted: 100

Buyer

Current: 1000

Predicted: 2500

Call Center

Current: 24

Predicted: 24 (No change)

Q. What is the average of active user count (including all types of users) at a time?
A. Total: 304
Admin: 3
Seller: 15
Buyer: 278
Call Center: 8

Q. What could be the active user count during peak hour?
A. Total: 500
Admin: 4
Seller: 50
Buyer: 438
Call Center: 8

Q. Anytime during a day or month when average user count is suddenly increased?
A. 31st of every month there is a 1-minute sale from 09:00 to 09:01 and 10:00 to 10:01. During this period the active buyer count becomes three times i.e. 834 and active seller count becomes 23. None of the other (Admin/Customer Care) users count change.
Admin: 3
Seller: 23
Buyer: 834
Call Center: 8
Total: 868

Q. What would be the response time NFRs for all the scenarios?
A. None of the pages should breach 3 seconds average response time NFR. This NFR is applicable for all type of users, scenarios, and pages.

Q. What would be the server-side NFRs?
A. CPU, memory and disk utilization (of Web, App, and DB) must not be more than 75%; except stress test. 

Q. What is the size of the performance testing environment with respect to production?
A. 100% (Live like environment)

More or less Perfingo got the answers to all of his queries and noted down NFRs in an excel and share with the client for the approval. If you want to have a look at NFRs, click here. After getting the approval on the NFRs, he starts to prepare the performance test strategy.

You can download the non-functional requirement document template from below link:

Non-Functional Requirement Document Template

Next Topic:
Performance Test Strategy (Planning)

Previous Topic:
Performance Test Life Cycle - Introduction