Pages

▼

Thursday 23 February 2017

Why Standard Deviation important in Performance Testing?

21 comments:

  1. Anonymous22 December 2016 at 00:55

    Nice 1.
    Keep posting :)

    -Rajat

    ReplyDelete
  2. Hung D Nguyen22 December 2016 at 11:03

    Excellent explanation for complex PT terms. Please keep posting like that.

    -Hung Nguyen

    ReplyDelete
  3. Unknown4 January 2017 at 02:42

    Hi Owner, please review why step 3 we divide for 4, not 5 which does not match with the formula: https://perfmatrix.blogspot.com/2016/12/standard-deviation.html
    Step 3: (4 + 9 + 100 + 16 + 1)/4 = 32.5

    and in Step 4: √32.5 = 5.7

    this data does not match to the deviation in the table of the sample

    ReplyDelete
    Replies
    1. PerfMatrix5 January 2017 at 21:37

      Corrected the formula... Thanks Truong Vu!

      Delete
    2. ravichander18 January 2017 at 04:57

      This comment has been removed by the author.

      Delete
  4. ravichander18 January 2017 at 04:59

    Hello Gagan..

    It was nice explanation!!

    Can you please explain where is wrong calculation? I am bit confused seems to be same formula and calculation. Please provide me clarification---Thanks.

    ReplyDelete
    Replies
    1. PerfMatrix12 February 2017 at 20:24

      In Standard deviation; arithmetic mean is divided by 1/(N-1) where mistakenly I had divided it by N in step 4

      Delete
  5. blogger15 March 2017 at 01:06

    In your post regarding 90th percentile, you mentioned about the relationship between the difference between 90th percentile, average response time divided by average response time and deviation.

    the lesser the ratio -> consistent performance - good
    higher the ratio -> high deviation

    In the table of response times in this post, pick the search transaction.
    It's 90th percentile is 4, average is 5.
    the ratio is 0.2.
    Here the ratio lesser, so the deviation should be less as per your 90th percentile post which is not the case here.

    help me explain this.

    ReplyDelete
    Replies
    1. PerfMatrix15 March 2017 at 03:57

      Agreed, that method is only applicable if you are not aware of standard deviation. However I have recommended to use standard deviation to draw conclusion on results.

      Delete
    2. Anonymous23 August 2017 at 06:45

      good information . But If few samples response time is pathetic , Std Dev will be high . so should not we use 90th pct to draw conclusion on results?

      Delete
    3. PerfMatrix23 August 2017 at 21:12

      Standard Deviation and 90th Percentile both are used for different purpose. Standard deviation gives you picture about how much data response time is deviated from mean, while 90th percentile excludes the 10% sample of high response time. See the given example. You can conclude all three transactions are performed well, but when you see the deviation then found that "Search" transaction is more deviated than others. The more the transaction deviate, the more probability of failing the transaction.

      Delete
  6. Unknown27 April 2017 at 01:11

    Nice one sirji.
    thank you for this :)
    - Jaydeep

    ReplyDelete
  7. Unknown12 September 2017 at 02:57

    Nice explanation,but once after conclusion,what action need to be done by developer?

    ReplyDelete
    Replies
    1. PerfMatrix12 September 2017 at 21:19

      Developers needs to investigate the cause which makes fluctuation in response time. They may require to tune the application, DB queries optimisation, network bandwidth etc. There could be multiple reasons which impacts response time. Only the thing is proper analysis and resolution helps to reduce the deviation.

      Delete
  8. Unknown6 May 2018 at 11:30

    what happens if the standard deviation values are not around the mean values?

    ReplyDelete
    Replies
    1. PerfMatrix6 May 2018 at 22:52

      Hi Divya,

      As I mentioned in the post "The smaller the Standard Deviation, the more consistent transaction response time" and vice-versa would be "The more Standard Deviation, the more inconsistency" which means server does not respond all the request in a consistence manner which may leads failure of the transactions.

      Delete
  9. Unknown6 May 2018 at 11:30

    Also one more thing i wanted to is what is the difference between a crashpoint and degradation in the performance/load testing?

    ReplyDelete
    Replies
    1. PerfMatrix6 May 2018 at 23:14

      These are not the generic term which are used in the performance testing. May be these are something specific to your project or client who frequently used them. If so please get the exact definition from them.

      I assume,
      1. Crash-point term seems like break-point of the application at which system is totally failed and throughput sudden drops to Zero.

      2. Degradation point arrives when there is a decline in the metrics (hits/second, TPS, Throughput etc.) due to waiting or queuing of the requests at server end (typical issue). The server respond continuously, but slowly and degrade the above mentioned metrics. The point where decline rely starts called Degradation Point.

      Delete
  10. Anonymous28 May 2018 at 23:41

    Is there any standard deviation threshold value like we have in cpu. Eg could should not be more then 80%, if it reaches then it will be considered as warning. So what is the value in case of standard deviation. How to correlate it with 90% and other metrics.

    ReplyDelete
    Replies
    1. PerfMatrix29 May 2018 at 03:25

      No, there is no standard value unless it is specifically defined by your project. The key thing is that the more the standard deviation value the poor your application's performance.

      Relation with 90th%ile: Standard Deviation is an independent metric which is having its own definition and purpose. You may get high deviation value in case of 90th percentile is much higher than average response time. But you can not conclude the result only on the basis this relationship (Refer above given example).

      Delete