The following configuration executes a multi-threaded Postmark benchmark, and is included in the distribution as postmark.ap:
1 #!/usr/bin/perl /usr/local/bin/auto-pilot 2 # 3 # Package: auto-pilot 4 # Erez Zadok <ezk AT cs.sunysb.edu> 5 # Charles P. Wright <cwright AT cs.sunysb.edu> 6 # Copyright (c) 2001-2005 Stony Brook University 7 8 # What is the name of this test? Auto-pilot doesn't internally 9 # use it for Postmark, but we may want it for setting paths, etc. 10 VAR BENCH=postmark 11 12 # How many times do we run it? 13 # We could for example, run it ten times 14 # VAR TERMINATE=10 15 # However, it is more interesting to run it five times, then run 16 # getstats to check if the half-width of our confidence intervals 17 # are less than 5% of the mean for elapsed, user, and system time. 18 # We also don't want it to run away and run more than 30 times on us. 19 VAR TERMINATE=10 1 getstats --predicate \ 20 '("$name" ne "User" && "$name" ne "System" && "$name" ne "Elapsed") || \ 21 ("$delta" < 0.05 * $mean) || ($count > 30)' -- 22 23 # How many threads should we go up to? 24 VAR NTHREADS=32 25 26 # What file systems do we want to test? 27 VAR TESTFS=ext2 ext3 reiserfs 28 29 # Include common.inc. You can override variables and include your own 30 # commands by creating local.inc, which is automatically included by 31 # common.inc. 32 INCLUDE common.inc 33 34 # Do the actual tests 35 FOREACH FS %TESTFS% 36 FOR THREADCOUNT=1 TO %NTHREADS% FACTOR 2 37 THREADS=%THREADCOUNT% 38 39 TEST %FS%-%THREADS% %TERMINATE% 40 SETUP fs-setup.sh %FS% 41 EXEC postmark.sh 42 CLEANUP fs-cleanup.sh %FS% 43 DONE 44 DONE 45 DONE 46 47 # All done 48 INCLUDE ok.inc
Line 1 informs the OS to use Perl as the interpreter for this Auto-pilot configuration. Only one level of interpreter is allowed, so you must execute Perl with Auto-pilot's path as the argument, because Auto-pilot itself uses an interpreter. Auto-pilot uses the pound sign (#) as a comment symbol, and any text after a pound sign is ignored. Lines 3–6 are comments containing copyright notice. Blank lines are ignored.
Line 10 assigns the value postmark
to the
variable bench. This variable is not directly used by this
configuration file, but later on we will use it to chose where to store
the Auto-pilot results.
Lines 12–19
set a variable named TERMINATE, which is used later in the configuration
to determine at what point a TEST
should end. The simplest method is to
specify a fixed number of tests (e.g., VAR TERMINATE=10
).
A more powerful method is to run a predicate command. In this example,
we execute the test at least ten times. For each iteration (denoted by
the "1" in the variable), we execute a predicate. To execute the
predicate after every other iteration, we would replace the "1" with a
"2". It can be useful to run the predicate only after several runs, so
that computing the predicate does not use more time than the benchmark
itself.
In this case we execute the command getstats --predicate '("$name"
ne "User" && "$name" ne "System" && "$name" ne "Elapsed") || ("$delta" <
0.05 * $mean) || ($count > 30)' --
. Getstats is our generic results
processing engine; for a complete description See Getstats.
Getstats can run arbitrary predicates using various summary statistics
(e.g., the mean, median, or standard deviation). The predicate is
checked for each quantity that Auto-pilot measures. The first part of
the predicate, ("$name" ne "User" && "$name" ne "System" &&
"$name" ne "Elapsed")
, skips all but User, System, and Elapsed time,
because these are the measured quantities we are generally interested
in. Wait time and CPU utilization are excluded because they are derived
from these quantities.
The next portion of the predicate, ("$delta" < 0.05 * $mean)
,
returns true if the half-width of a confidence interval (the confidence
level is 95% by default) is less than 5% of the mean.
The final portion, ($count > 30)
prevents a poorly behaving test
from executing for more than 30 iterations.
The TERMINATE variable is rather long, and spans multiple lines using a backslash. To continue a line, the backslash must be the very last character before the newline (no spaces are allowed).
We then set another two variables NTHREADS (line 24), which is later used to determine the maximum number of threads, and TESTFS (line 27) which is used to determine the file systems to execute the benchmark on.
The next directive (on line 32), is INCLUDE
common.inc
(this file is included with the Auto-pilot distribution). The
common.inc file performs
several tasks:
FASTFAIL
directive is used so that benchmarks stop after one
fails.
To override any previous settings, and define new ones without editing the installed configuration files you should create a local.inc in the directory where you run the benchmarks. For example, if you wanted to use another predicate, then you would override TERMINATE here. You also need to set the TESTDEV, TESTROOT, RESULTS, LOGS, OKADDR, and FAILADDR.
An example local.inc could have the following:
RESULTS=$HOME$/results/%BENCH% LOGS=$HOME$/logs/%BENCH% VAR TESTFS=ext2 VAR NOSERVICES=1 ENV FORMAT=1 ENV TESTDEV=/dev/sda1 ENV TESTROOT=/mnt/test ENV OKADDR=you@example.com ENV FAILADDR=you@example.com
In this example, the results files are stored under the current user's home directory in results/%BENCH%, where %BENCH% is the name of the benchmark as defined in the configuration file. The stdout/stderr logs are similarly stored in logs/%BENCH% under the current user's home directory.
The third directive, VAR TESTFS=ext2
, overrides the default value
of TESTFS, so that only ext2 is tested. The fourth causes
common.inc to shutdown excess services.
The remaining lines set environment variables that are used by the default Auto-pilot shell scripts. Setting FORMAT to 1 causes the scripts to recreate the file system. TESTDEV determines what device should be used, and TESTROOT determines what directory is used.
The two remaining lines set addresses that should be used to send email. OKADDR, FAILADDR are the address that mail is sent to on successful completion and failure, respectively. I have found it useful to add .procmailrc recipes like the following to forward these messages to a mobile phone:
# Send a copy of all messages with a subject starting with Benchmarks to # another email address for Auto-pilot paging. :0 c * ^Subject: Benchmarks ! yourphone@telco.com
The remaining configuration files are similar to postmark.ap, and using these as templates you should be able to create your own.