Performance Test Result Analysis (Level-II - Part B)

Continue where we left off in Part - A .....
Server-side monitoring comprises of Web Server, App Server and DB resource monitoring. Microfocus LoadRunner provides some server resource graph like Linux resource graph, Windows resource graph, etc. which give the resources utilization in % (CPU, disk space, memory, or services) used on remote Linux and Windows servers measured during the performance test. Many times profiling tools are used for deep server monitoring. For this purpose Microfocus SiteScope, Compuware Dynatrace, Microfocus Diagnostic are very helpful tools. Among these profiling tools, some are agent-based while some are agentless tools. Each tool has its own pros and cons.

Majorly, at server side 2 main components which require extra attention than others. If we do a thorough analysis of these two metrics then most of the bottlenecks can be identified:

1. CPU Utilization: Ideally, CPU utilization should be less than 60%, although the utilization percentage totally depends on giving SLA. As a performance tester you could mark less than 60% CPU utilization as GREEN, 60%-80% AMBER and more than 80% as RED but this RAG is generic. If you are testing a high performance system having a tremendous growth of load in the near future then RAG will be changed. A typical CPU Utilization graph is given below:

Tips: 
a. CPU Utilization must be constant (+/- 2% tolerance) during steady state. 
b. There must be gradually increased in utilization % during ramp-up.
c. Continue Peaks in utilization % graph leads the investigation of the bottleneck.

2. Memory Utilization: To calculate the real impact of user load on memory, you must have to note down pre and post-test memory utilization reading. Memory Utilization depends on many factors like users' load, functioning of Garbage Collector, DB queries, etc. There are no such generic figures for memory. Still, I must say to extrapolate your performance test results to get up to what time and how many users can be handled by the server when memory reaches 100%. If memory utilization % does not come down (even test ended long back) then there must be memory leakage bottleneck and you have to investigate the cause. The typical memory utilization graph is given below:

Tips: 
a. Memory Utilization must be constant during steady state. 
b. Increase in memory utilization % during steady state and remain occupied for longer duration are caused by memory leakage. GC may be one of the reasons for this.

The validation of server performance can be directly done by comparing the result with given SLAs. The system SLAs cannot be generic because those numbers vary according to requirements. I have listed down some important counters and server tuning tips below:  

Web Server: Web server counters provide information about resource usage on the web server during the performance test. These counter provide useful information on web server performance and its bottlenecks. There is a long list of metrics which are generated during the performance depending upon the type of web server. Apache HTTP, Microsoft Internet Information Services (Microsoft IIS), Lighttpd, Sun Java System and Jigsaw server are famous web servers. 

Apache Web Server Counters:

• CPULoad: The current percentage of CPU consumed by the Apache server

• ReqPerSec: The number of requests per second (a.k.a. hits per second)

• BytesPerSec: The number of bytes transferred per second

• BytesPerReq: The number of bytes transferred per request

• BusyWorkers: Number of active threads serving requests

• IdleWorkers: Number of inactive/idle threads

Tuning Tips: Continue Reading....... 

ASP (Active Server Pages) Web Server Counters:

• Errors/sec: The average number of errors that occurred per second. 

• Requests/sec: The average number of requests that were executed per second. 

• Requests Executing The number of ASP requests currently executing (for example, the number of active worker threads). 

• Requests Queued The number of queued ASP requests that are waiting to be processed. The maximum number for this counter is determined by the metabase property AspRequestQueueMax. 

• Transactions/sec The average number of transactions that have been started, per second.

Tuning Tips: Continue Reading....... 

Microsoft IIS web server counters:

• Bytes Sent per Second:  It shows the total data flow in bytes from the Transaction Integrator (TI) server to the host per second.

• Bytes Received per Second: It shows the total bytes received from the host per second

• Get Requests per Second: The number of GET requests executed per second

• Post Requests per Second: The number of POST requests executed per second

• Maximum Connections

• Not Found (404) Error per Second

• Private Bytes: Private Bytes is the current size, in bytes, of memory that is allocated for a particular process and cannot be shared with other processes.

• Percent of Processor Time

• System Processor Queue Length

• Percent Disk Time

• Queue Length

• Cache Hit Percent

• Request Wait Time: The number of milliseconds that the most recent request waited in the queue for processing

• Request Queued: The number of requests waiting for service from the queue. When this number starts to increment linearly with increased client load, the Web server computer has reached the limit of concurrent requests that it can process. The default maximum for this counter is 5,000. You can change this setting in the Machine.config file

Application Server: Most of the application computational activities are performed on its application server. The application server is the backbone of the application especially in case of highly complex business applications. Web Sphere, JBoss, Apache Tomcat, Web Logic, Microsoft .Net, and Oracle Application server are some of the famous applications servers being used these days. Every type of application server has its performance counters based on its model. Below are some of the Web Sphere application performance counters which need to be monitored during the performance test, 

Web Sphere Counters 

• Memory Free:

• Total Memory

• Memory Use

• Bean Destroys

• Active Threads

• Total Threads

• Percent Time Maxed

• Connection Wait Time

• Connection Time

• Connection Percent Used

• Active Transactions

• Transactions Suspended

• Rolled Back

• Timeouts

• Servlet Errors

• Sessions Invalidated

Data Base Server: All the application data is stored in the database and the system on which database is installed is called database server. As the database server contains all the critical information of the application, its fast and error-free working is extremely important. Oracle, DB2, Microsoft SQL Server and Informix are famous database servers. Some of the common metrics monitored for all database servers are following, 

• Database Transaction Summary: Database transaction monitor provide the information of volume of data sent and received by the database server. It provides the detail in bytes of requests sent and received during the performance test.
• Database Connection Summary: This metric provides information about the total number of open and close connections on the database server during the test. Excess of database connections can badly affect the database server performance.
• Database Thread Summary: Thread metric provides information about the number of new threads created, connected and used.
• Database Query Summary: Read, write and delete data are main database functions and their detail is very important to check the database server performance during the performance test. Database Query metric provides information about the total number of queries read, wrote and deleted by the database server.

Microsoft SQL Server Counters: 

• Transactions per Second

• Log Cache Hit Ratio

• Log Cache Reads per Second

• Log Bytes Flushed per Second

• Log Flush Wait Time

• Log Flush Waits per Second

• Log Flushes per Second

• Percent Log Used

Next Topic:
Performance Test Result Analysis (Level-II - Part C)

Previous Topic:
Performance Test Result Analysis (Level-II - Part A)

Related Topics:

• Result Analysis - Basic Level
• Result Analysis - Advance Level
• Performance Testing Basics
• Performance Engineering Basics
• PTLC - Result, Report & Recommendation