Applying PDQ in R to Load Testing

May 19, 2011
By

(This article was first published on The Pith of Performance, and kindly contributed to R-bloggers)

PDQ is a library of functions that helps you to express and solve performance questions about computer systems using the abstraction of queues. The queueing paradigm is a natural choice because, whether big (a web site) or small (a laptop), all computer systems can be represented as a network or circuit of buffers and a buffer is a type of queue.

As a performance analyst, there are several things I really like about using PDQ in R; as opposed to the other programming languages: C, Perl, Python, etc. It enables you to:
  1. easily import (large) data with a variety formats
  2. perform sophisticated statistical analysis
  3. extract input parameters for a PDQ model
  4. construct and execute the PDQ model within R
  5. plot the PDQ output and compare it with the original data
  6. test your ideas in the R console and save the best into a script
In applying this approach, you could find yourself using a number of R library packages. To improve clarity in your modeling script, you might like to identify clearly which routines belong to PDQ; especially if you're new to PDQ and not familiar with all the functions.

R syntax for naming function dependency is the same as Perl. The :: operator is used for explicitly exported names. It also avoids conflict between packages the export different functions with the same name. The ::: operator is used for access to functions that are not exported in the package namespace.

Let's look at the above steps in the context of an example based on load testing data. A key point to observe here is how the performance data and the performance model play together to provide validation of the measurements.

Performance data

We begin by importing the load test data from measurements of an application intended for a three-tier architecture.

library(ineq)
library(pdq)

# Read in the performance measurements
gdat <- read.csv("/Users/njg/.../gcap.dat",header=TRUE)
Even though the ineq package is part of base R functionality, I've loaded it explicitly so as to name its functions explicitly. This will also provide a contrast with explicitly named functions from the PDQ package.

> gdat
Vusr Xgps Rms Uweb Uapp Udbm
1 1 24 26.0 0.21 0.08 0.04
2 2 48 26.0 0.41 0.13 0.05
3 4 85 29.3 0.74 0.20 0.05
4 7 100 44.7 0.95 0.23 0.05
5 10 115 66.0 0.96 0.22 0.06
6 20 112 140.0 0.97 0.22 0.06
The columns are respectively the client load, measured throughput, response time (in milliseconds), and system utilization on each of the three tiers.

Statistical analysis

We can now perform various kinds of statistical analysis on these data.

> summary(gdat)
Vusr Xgps Rms Uweb Uapp
Min. : 1.000 Min. : 24.00 Min. : 26.00 Min. :0.2100 Min. :0.0800
1st Qu.: 2.500 1st Qu.: 57.25 1st Qu.: 26.82 1st Qu.:0.4925 1st Qu.:0.1475
Median : 5.500 Median : 92.50 Median : 37.00 Median :0.8450 Median :0.2100
Mean : 7.333 Mean : 80.67 Mean : 55.33 Mean :0.7067 Mean :0.1800
3rd Qu.: 9.250 3rd Qu.:109.00 3rd Qu.: 60.67 3rd Qu.:0.9575 3rd Qu.:0.2200
Max. :20.000 Max. :115.00 Max. :140.00 Max. :0.9700 Max. :0.2300
Udbm
Min. :0.04000
1st Qu.:0.05000
Median :0.05000
Mean :0.05167
3rd Qu.:0.05750
Max. :0.06000
More significantly, we can use R statistical functions to derive appropriate parameters for a PDQ model.

# Apply Little's law to get mean service times + CoVs
Sweb <- mean(gdat$Uweb/gdat$Xgps)
Sapp <- mean(gdat$Uapp/gdat$Xgps)
Sdbm <- mean(gdat$Udbm/gdat$Xgps)

Csw <- ineq::var.coeff(gdat$Uweb/gdat$Xgps)
Csa <- ineq::var.coeff(gdat$Uapp/gdat$Xgps)
Csd <- ineq::var.coeff(gdat$Udbm/gdat$Xgps)

s1 <- sprintf("System: %6s %6s %6s\n", "Web","App","DBMS")
s2 <- sprintf("Mean S: %6.4f %6.4f %6.4f\n", Sweb, Sapp, Sdbm)
s3 <- sprintf("CoV S: %6.4f %6.4f %6.4f\n", Csw, Csa, Csd)
cat("\n",s1,s2,s3)
In particular, we calculate the average service times on each tier (second row) by applying Little's law.

System: Web App DBMS
Mean S: 0.0088 0.0024 0.0008
CoV S: 0.0411 0.1989 0.5271

PDQ model

As shown in Figure 1, the service times for each of the three tiers in the load-test platform can be represented as queueing resources in PDQ.

There is a finite number of requests allowed in the system corresponding to the load clients or virtual users that range between N = 1 and N = 20 Vusers, represented by the octagonal box in Figure 1. Using the diagram, we set up the following PDQ model. Note the use of explicitly named functions from the PDQ library

# Plotting variables
xc <- 0 # Vuser loads
yc <- 0 # PDQ throughputs
rc <- 0 # PDQ response times

# Define and solve the PDQ model
for(n in 1:max(gdat$Vusr)) {
pdq::Init("Three-Tier Model")

pdq::CreateClosed("httpGETs", TERM, as.numeric(n), 0.028)

pdq::CreateNode("WebServer", CEN, FCFS)
pdq::CreateNode("AppServer", CEN, FCFS)
pdq::CreateNode("DBMServer", CEN, FCFS)

pdq::SetDemand("WebServer", "httpGETs", Sweb)
pdq::SetDemand("AppServer", "httpGETs", Sapp)
pdq::SetDemand("DBMServer", "httpGETs", Sdbm)

pdq::Solve(EXACT)

xc[n] <- n
yc[n] <- pdq::GetThruput(TERM, "httpGETs")
rc[n] <- pdq::GetResponse(TERM, "httpGETs") * 10^3
}
In the above PDQ model, we've selected the predicted throughput and the predicted response times to compare with the original load-test data.

Plot PDQ results


# Plot throughput and response time models
par(mfrow=c(2,1))
plot(xc, yc, type="l", lwd=1, col="blue", ylim=c(0,120), main="PDQ Throughput Model", xlab="Vusers (N)", ylab="Gets/s X(N)")
points(gdat$Vusr, gdat$Xgps)
plot(xc, rc, type="l", lwd=1, col="blue", ylim=c(0,220), main="PDQ Response Time Model", xlab="Vusers (N)", ylab="ms R(N)")
points(gdat$Vusr, gdat$Rms)
The above R code produces the following plot array:

We see that the data and PDQ model are in good agreement with the throughput saturating above N = 5 vusers with the corresponding response time rising up the proverbial "hockey stick" handle.

PDQ report

Optionally, we can produce a formal PDQ report to examine the performance of each of the three tiers, even if we don't have any corresponding performance measurements from the load-test platform. This is one way by which bottlenecks can be predicted and checked before deploying into production.

> pdq::Report()
***************************************
****** Pretty Damn Quick REPORT *******
***************************************
*** of : Sun May 15 18:26:21 2011 ***
*** for: Three-Tier Model ***
*** Ver: PDQ Analyzer v5.0 030211 ***
***************************************
***************************************

=======================================
****** PDQ Model INPUTS *******
=======================================

Node Sched Resource Workload Class Demand
---- ----- -------- -------- ----- ------
CEN FCFS WebServer httpGETs TERML 0.0088
CEN FCFS AppServer httpGETs TERML 0.0024
CEN FCFS DBMServer httpGETs TERML 0.0008

Queueing Circuit Totals:
Streams: 1
Nodes: 3

WORKLOAD Parameters:
httpGETs 20.00 0.0120 0.03


=======================================
****** PDQ Model OUTPUTS *******
=======================================

Solution Method: EXACT

****** SYSTEM Performance *******

Metric Value Unit
------ ----- ----
Workload: "httpGETs"
Mean concurrency 16.8004 Users
Mean throughput 114.2725 Users/Sec
Response time 0.1470 Sec
Round trip time 0.1750 Sec
Stretch factor 12.2633

Bounds Analysis:
Max throughput 114.2725 Users/Sec
Min response 0.0120 Sec
Max Demand 0.0088 Sec
Tot demand 0.0120 Sec
Think time 0.0280 Sec
Optimal clients 4.5696 Clients


****** RESOURCE Performance *******

Metric Resource Work Value Unit
------ -------- ---- ----- ----
Throughput WebServer httpGETs 114.2725 Users/Sec
Utilization WebServer httpGETs 100.0000 Percent
Queue length WebServer httpGETs 16.3144 Users
Waiting line WebServer httpGETs 15.3144 Users
Waiting time WebServer httpGETs 0.1340 Sec
Residence time WebServer httpGETs 0.1428 Sec

Throughput AppServer httpGETs 114.2725 Users/Sec
Utilization AppServer httpGETs 27.7529 Percent
Queue length AppServer httpGETs 0.3841 Users
Waiting line AppServer httpGETs 0.1066 Users
Waiting time AppServer httpGETs 0.0009 Sec
Residence time AppServer httpGETs 0.0034 Sec

Throughput DBMServer httpGETs 114.2725 Users/Sec
Utilization DBMServer httpGETs 9.2447 Percent
Queue length DBMServer httpGETs 0.1019 Users
Waiting line DBMServer httpGETs 0.0094 Users
Waiting time DBMServer httpGETs 0.0001 Sec
Residence time DBMServer httpGETs 0.0009 Sec

To leave a comment for the author, please follow the link and comment on his blog: The Pith of Performance.

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...



If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Comments are closed.