bench_gcc - benchmark gcc optimization flags on a given code base
bench_gcc [options]
GCC compiler options are a twisty, dark maze of evil smelling passages. Finding the right options for a given system can be quite tricky even without all the myths that surround compiler options.
This script aims to help people test a set of options for a given
program. This script will not tell you the Perfect CFLAGS(tm) nor
help you find God or anything else. What it will do is let you
determine quantitatively how different options change you code.
FLAG OPTIONS: --base <flags> Sets the baseline options for comparison --peak <flags> Sets the new options that you want to benchmark
COMPILER OPTIONS:
To alter the paths for base and peak compilers, use:
--compiler_base <path>
--compiler_peak <path>
SOURCE OPTIONS: --compile <command> The command needed to compile the code. (ex: "./configure; make") --clean <command> The command to reset the code in the current directory. (ex: "make clean"); --run <command> The command to use for benchmarking. This is very application dependent and must be set by the user. [required option]
CODE SIZE OPTIONS: --sizeof <file> The file to compare size changes of. Specifying this flag enables code size reporting.
STATISTICAL OPTIONS: --iterate <number> How many times the run command is executed. --bestof Select the best time of all the runs. --average Select the average time (mean) of all the runs.
OUTPUT OPTIONS: --dotevery <number> Sets the number of lines of output per dot printed.
HELP OPTIONS: --man Show a man page for this program. --help Print a short option summary.
- To eliminate disk cache from the benchmarks, place the testing directory on a ramdisk.
- The final report is appended in CSV format to ``bench_gcc.csv'' in the main directory. A simple perl script can be written to parse this or alternately it can be loaded into a spreadsheet program with Gnumeric.
- You can use --compiler_base and --compile_peak to set different paths for gcc and g++ in order to test differnt version of gcc with the same base and peak options.
- To test more than two combinations of flags you can run the script multiple times with all the options. Ignore the reports and use the CSV file that will have all the runs results in it.
- More complex stats can be had by hacking this perl script to suit your specific needs.
- Very involved tests can be done with the run command by writing a small testing script (ex: create foo.sh and use --run foo.sh). For instance, if you are interested in benchmarking options on KMail, you could write a test script that starts KMail, uses dcop to do some KMail things and shut it down.
- Try to make your run commands reflect the behaviours that you want to see in the application.
To benchmark the differnces in the options ``-01'' and ``-03'' on MP3 encoding you might perform the following steps.
You need a suitable WAV file to run this example. If you do not have one you can make one with ``mpg123 -w foo.wav foo.mp3'' from foo.mp3. For this example I assume that you have a suitable WAV file already called ``/tmp/foo.wav''.
# cd /tmp # tar zxvf /usr/portage/distfiles/lame-3.96.1.tar.gz # cd lame-3.96.1 # bench_gcc \ --compile "./configure --disable-gtk; make" \ --run "frontend/lame --preset standard /tmp/foo.wav foo.mp3 >&/dev/null" \ --clean "make clean" \ --average --iterate 3 --base "-O1" --peak "-O3" \ --sizeof frontend/lame
This might give you output similar to the screen scrape below that shows a 9.87% speedup in runtime.
------------------------------------------------------------------------------------- Starting base benchmark... - compiling... (log available in base/compile.log) ................................................................ -> compile took 40.37 seconds. - starting runtime test... (log available in base/run.log.*) -> iteration 1... run took 33.34 seconds. -> iteration 2... run took 34.81 seconds. -> iteration 3... run took 33.58 seconds. => average time: 33.91 - cleaning up... (log available in base/clean.log) Starting peak benchmark... - compiling... (log available in peak/compile.log) ................................................................ -> compile took 48.57 seconds. - starting runtime test... (log available in peak/run.log.*) -> iteration 1... run took 30.92 seconds. -> iteration 2... run took 30.96 seconds. -> iteration 3... run took 30.72 seconds. => average time: 30.86 - cleaning up... (log available in peak/clean.log)
Stats Method: average Number of iterations: 3
Base Path: default Base Options: -O1 Base Compile Time: 40.37 seconds Base Run Time: 33.91 seconds Base Code Size: 319191 bytes
Peak Path: default Peak Options: -O3 Peak Compile Time: 48.57 seconds Peak Run Time: 30.86 seconds Peak Code Size: 328284 bytes
Diff Compile 8.20 seconds (16.88% increase) Diff Run: 3.05 seconds (-9.87% decrease) Diff Code Size: -9093 bytes (2.77% increase) -------------------------------------------------------------------------------------
Or perhaps you want to compare gcc 3.3 to gcc 3.4. If you replace the last line in the above command with:
--bestof --iterate 1 --base "-O2" --peak "-O2" --compiler_base /usr/i686-pc-linux-gnu/gcc-bin/3.3 --compiler_peak /usr/i686-pc-linux-gnu/gcc-bin/3.4
You might get something like this showing the expected increase in compile time and a very nice 15% decrease in run time with the same flags.
------------------------------------------------------------------------------------ Starting base benchmark... - compiling... (log available in base/compile.log) ................................................................ -> compile took 49.27 seconds. - starting runtime test... (log available in base/run.log.*) -> iteration 1... run took 36.53 seconds. => best time: 36.53 - cleaning up... (log available in base/clean.log) Starting peak benchmark... - compiling... (log available in peak/compile.log) ................................................................ -> compile took 36.10 seconds. - starting runtime test... (log available in peak/run.log.*) -> iteration 1... run took 31.68 seconds. => best time: 31.68 - cleaning up... (log available in peak/clean.log)
Stats Method: bestof Number of iterations: 1
Base Path: /usr/i686-pc-linux-gnu/gcc-bin/3.3/ Base Options: -O2 Base Compile Time: 49.27 seconds Base Run Time: 36.53 seconds Base Code Size: not_run
Peak Path: /usr/i686-pc-linux-gnu/gcc-bin/3.4/ Peak Options: -O2 Peak Compile Time: 36.10 seconds Peak Run Time: 31.68 seconds Peak Code Size: not_run
Diff Compile -13.17 seconds (-36.47% decrease) Diff Run: 4.85 seconds (-15.31% decrease) Diff Code Size: not_run ------------------------------------------------------------------------------------
After running both of the above examples, you would have a gcc_bench.csv that looked like:
------------------------------------------------------------------------------------- path,opts,compile_time,run_time,size,method,iterations default,--bypass -O1,37.92,69.41,319191,average,3 default,--bypass -O3,47.81,64.46,328284,average,3 /usr/i686-pc-linux-gnu/gcc-bin/3.3/,-O2,46.11,74.91,not_run,bestof,1 /usr/i686-pc-linux-gnu/gcc-bin/3.4/,-O2,34.45,67.71,not_run,bestof,1 -------------------------------------------------------------------------------------
Good statistics are hard to come by. Really. This script measures wall time of the run command. Compile times are not expected to be terribly accurate but you can get reasonable accuracy from the run times but using more iterations.
Written by Patrick Audley <paudley@blackcat.ca> http://blackcat.ca
Please send me any comments, bug reports, suggestions or flame mail :)
Copyright 2003,2004 by Patrick Audley <paudley@blackcat.ca>
This program is licensed under the terms of the GPL. If you did not recieve a copy of the license, you can get it from http://www.gnu.org/licenses/gpl.html