The default function library can be broken down into several categories: manipulating global variables, time related functions, warning primitives, built-in warnings, the default transformations that each relation undergoes, predicate evaluation, and relation renaming.
There are several functions that manipulate global replacement variables (which are stored in %globals).
set(
var,
val)
setexpr(
var,
expr)
seteval(
var,
expr)
setexpr
performs global replacement first, and seteval
first runs
Perl's eval function on expr.
unset(
var)
push(
var, val)
pusheval(
var,
expr)
pusheval
, "expr" is first passed through
Perl's eval function.
pop(
var)
There are also several transformation functions for creating more useful output from times.
aggthreads
procdiff
savestats
$globals{'stathashes'}
, this is used for computing overheads.
unifycommand
configure
and
make
).
There are three types of warning primitives. Warnings that should be evaluated for each row ("warnrow"), column ("warncol"), or each cell ("warnval").
warnrow(expression, output)
Assuming that the exist status is stored in a column named status, the following transformation warns when tests fail:
warnrow("$status != 0", "Failure for epoch $epoch, thread $thread, exit status = $status.")
When run on Relation 7.15, the output is:
Failure for epoch 3, thread 1, exit status = 127.
warncol(expression, output)
An example column warning would be if the half-width of a confidence interval is greater than 10% of the mean:
warncol("$delta > $mean * 0.05", "$name has a half-width of $delta.")
warnval(expression, output)
The following transformations use the previously described warning primitives to raise warnings if $warn is set (which it is by default). You can turn these warnings off by setting $warn to 0.
exitfail
negio
otherexec
You can control the percentage by setting the global variable
otherexec-thresh.
warnregress
You can control the slope by setting the global variable regress-thresh.
zscore
You can control the zscore by setting the global variable zscore-thresh.
The following functions define the four default passes that getstats performs over the relations. Any transformations specified on the command line are executed after readpass, but before warnpass.
readpass
warnpass
ohpass
summary
predicate(expr)
Predicate takes a single argument, which is the predicate to evaluate over each column of data. If the predicate is not true for any of the columns, Getstats dies. Auto-pilot picks up on this failure and continues to run the tests.
twosamplet
It also performs a two-sample t-test with a null hypotheses of "current mean - baseline mean <= twosampledelta", "current mean - baseline mean >= twosampledelta", and "current mean - baseline mean = twosampledelta". The p-value (the probability of observing this data, assuming the null hypothesis is true). If the p-value is small, then you can reject the null hypothesis. "REJECT" or "ACCEPT" is also printed, based on the confidence level you have specified (95% by default).
The following example compares a one threaded Postmark run (Sample 1) against a two threaded postmark run (the baseline is Sample 2).
Comparing samples/ext2:2.res (Sample 1) to samples/ext2:1.res (Sample 2). Elapsed: 95%CI for samples/ext2:2.res - samples/ext2:1.res = (7.945, 8.029) H_0: u1 - u2 <= 0.000 H_a: u1 - u2 > 0.000 P = 1.000 ACCEPT H_0 H_0: u1 - u2 >= 0.000 H_a: u1 - u2 < 0.000 P = 0.000 REJECT H_0 H_0: u1 - u2 == 0.000 H_a: u1 - u2 != 0.000 P = 0.000 REJECT H_0
As we can see, we must accept the assumption that two threads takes less time than one thread. We can reject the assumption that two threads takes longer than one thread, and the assumption that they take the same amount of time. Remember that Getstats runs all of these tests, but you need to choose which assumption makes sense for your case. For example, if you have code that should improve performance, you can't make the assumption that it does improve performance. Instead, you must make the assumption that it does not improve performance, and either accept or reject that conclusion.
Several global variables control the ttest. ttestcolumns is a comma separate list of columns to compare. twosampledelta controls the delta for the t-test, by default the delta is zero. This is used to show that two samples are different by more than some value. rejectonly causes only rejected hypothesis to be displayed, this can greatly reduce the amount of output when comparing mostly similar samples. verbosettest prints the relation names instead of u1 and u2. confidencelevel controls when to reject or accept tests (and the width of the confidence interval). precision controls how many decimal places are printed.
pairwiset
rename_relation(expr)
basename