Link: https://www.perfmatrix.com/standard-deviation-in-performance-testing/
Related Topics:
- Result Analysis - Basic Level
- Result Analysis - Intermediate Level
- Result Analysis - Advance Level
- Performance Testing Basics
- Performance Engineering Basics
- PTLC - Result, Report & Recommendation
- Importance of 90th Percentile
Nice 1.
ReplyDeleteKeep posting :)
-Rajat
Excellent explanation for complex PT terms. Please keep posting like that.
ReplyDelete-Hung Nguyen
Hi Owner, please review why step 3 we divide for 4, not 5 which does not match with the formula: https://perfmatrix.blogspot.com/2016/12/standard-deviation.html
ReplyDeleteStep 3: (4 + 9 + 100 + 16 + 1)/4 = 32.5
and in Step 4: √32.5 = 5.7
this data does not match to the deviation in the table of the sample
Corrected the formula... Thanks Truong Vu!
DeleteThis comment has been removed by the author.
DeleteHello Gagan..
ReplyDeleteIt was nice explanation!!
Can you please explain where is wrong calculation? I am bit confused seems to be same formula and calculation. Please provide me clarification---Thanks.
In Standard deviation; arithmetic mean is divided by 1/(N-1) where mistakenly I had divided it by N in step 4
DeleteIn your post regarding 90th percentile, you mentioned about the relationship between the difference between 90th percentile, average response time divided by average response time and deviation.
ReplyDeletethe lesser the ratio -> consistent performance - good
higher the ratio -> high deviation
In the table of response times in this post, pick the search transaction.
It's 90th percentile is 4, average is 5.
the ratio is 0.2.
Here the ratio lesser, so the deviation should be less as per your 90th percentile post which is not the case here.
help me explain this.
Agreed, that method is only applicable if you are not aware of standard deviation. However I have recommended to use standard deviation to draw conclusion on results.
Deletegood information . But If few samples response time is pathetic , Std Dev will be high . so should not we use 90th pct to draw conclusion on results?
DeleteStandard Deviation and 90th Percentile both are used for different purpose. Standard deviation gives you picture about how much data response time is deviated from mean, while 90th percentile excludes the 10% sample of high response time. See the given example. You can conclude all three transactions are performed well, but when you see the deviation then found that "Search" transaction is more deviated than others. The more the transaction deviate, the more probability of failing the transaction.
DeleteNice one sirji.
ReplyDeletethank you for this :)
- Jaydeep
Welcome Jaydeep
DeleteNice explanation,but once after conclusion,what action need to be done by developer?
ReplyDeleteDevelopers needs to investigate the cause which makes fluctuation in response time. They may require to tune the application, DB queries optimisation, network bandwidth etc. There could be multiple reasons which impacts response time. Only the thing is proper analysis and resolution helps to reduce the deviation.
Deletewhat happens if the standard deviation values are not around the mean values?
ReplyDeleteHi Divya,
DeleteAs I mentioned in the post "The smaller the Standard Deviation, the more consistent transaction response time" and vice-versa would be "The more Standard Deviation, the more inconsistency" which means server does not respond all the request in a consistence manner which may leads failure of the transactions.
Also one more thing i wanted to is what is the difference between a crashpoint and degradation in the performance/load testing?
ReplyDeleteThese are not the generic term which are used in the performance testing. May be these are something specific to your project or client who frequently used them. If so please get the exact definition from them.
DeleteI assume,
1. Crash-point term seems like break-point of the application at which system is totally failed and throughput sudden drops to Zero.
2. Degradation point arrives when there is a decline in the metrics (hits/second, TPS, Throughput etc.) due to waiting or queuing of the requests at server end (typical issue). The server respond continuously, but slowly and degrade the above mentioned metrics. The point where decline rely starts called Degradation Point.
Is there any standard deviation threshold value like we have in cpu. Eg could should not be more then 80%, if it reaches then it will be considered as warning. So what is the value in case of standard deviation. How to correlate it with 90% and other metrics.
ReplyDeleteNo, there is no standard value unless it is specifically defined by your project. The key thing is that the more the standard deviation value the poor your application's performance.
DeleteRelation with 90th%ile: Standard Deviation is an independent metric which is having its own definition and purpose. You may get high deviation value in case of 90th percentile is much higher than average response time. But you can not conclude the result only on the basis this relationship (Refer above given example).