benchtest: script to compare two benchmarks

This script is a sample implementation that uses import_bench to
construct two benchmark objects and compare them.  If detailed timing
information is available (when one does `make DETAILED=1 bench`), it
writes out graphs for all functions it benchmarks and prints
significant differences in timings of the two benchmark runs.  If
detailed timing information is not available, it points out
significant differences in aggregate times.

Call this script as follows:

  compare_bench.py schema_file.json bench1.out bench2.out

Alternatively, if one wants to set a different threshold for warnings
(default is a 10% difference):

  compare_bench.py schema_file.json bench1.out bench2.out 25

The threshold in the example above is 25%.  schema_file.json is the
JSON schema (which is $srcdir/benchtests/scripts/benchout.schema.json
for the benchmark output file) and bench1.out and bench2.out are the
two benchmark output files to compare.

The key functionality here is the compress_timings function which
groups together points that are close together into a single point
that is the mean of all its representative points.  Any point in such
a group is at most 1.5x the smallest point in that group.  The
detailed derivation is a comment in the function.

	* benchtests/scripts/compare_bench.py: New file.
	* benchtests/scripts/import_bench.py (mean): New function.
	(split_list): Likewise.
	(do_for_all_timings): Likewise.
	(compress_timings): Likewise.

1 files changed, 6 insertions, 0 deletions

diff --git a/ChangeLog b/ChangeLog
index 624e6f0f4e..6d295e720b 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,5 +1,11 @@
 2015-06-01  Siddhesh Poyarekar  <siddhesh@redhat.com>
 
+	* benchtests/scripts/compare_bench.py: New file.
+	* benchtests/scripts/import_bench.py (mean): New function.
+	(split_list): Likewise.
+	(do_for_all_timings): Likewise.
+	(compress_timings): Likewise.
+
 	* benchtests/scripts/import_bench.py: New file.
 	* benchtests/scripts/validate_benchout.py: Import import_bench
 	instead of jsonschema.