This page links to material that is supplemental to publications, such as raw data and source code. The paper title may not be reported for publications that are currently under review, but papers should be identifiable to reviewers using the manuscript id or other information.

Manuscript JSS-D-13-00860 submitted to Journal of Systems and Software (Special Issue on SBSE)

To view the directory of supplemental files, click here.

The directory contains:

  • The instrumented source of SUTs used as case studies in the empirical demonstration. The instrumentation is achieved via precompiler macros OPT_INSTRUMENT_DECISION and OPT_INSTRUMENT_CASE. I am unable to share the additional (uninstrumented) sources that form the full test harness of each SUT, such as the rest of the epuck simulator, since I do not hold the copyright to this software.
  • The raw data from the algorithm trials in the compressed archive raw_trial_data.tar.gz (note: this archive is 123 MB). The files are named: _E2[1-4][1-5][y|z]_nn_[fitnesses|executions].txt. The digit [1-4] indicates the SUT: 1=tcas, 2=circbuff, 3=epuck, 4=replace. The digit [1-5] indicates the algorithm: 1=full, 2=rand, 3=\cond, 4=\part, 5=\both. The digits nn are the index of the algorithm trial between 1 and 64. The files with the suffix _fitnesses list the fitness (the estimated minimum coverage probability) after each iteration of the algorithm trial. The files with the suffix _executions list the cumulative number of executions made of the instrumented SUT after each iteration of the algorithm trial.
The Use of Automated Search in Deriving Software Testing Strategies
PhD Thesis (2013)

To view the directory of supplemental files, click here.

The directory contains the source code for the following SUTs used as case studies in the thesis: simpleFunc, bestMove, nsichneu, cArcsin, tcas, circBuff, and epuck. For each SUT, the code provided is the file that was instrumented; for epuck, for instance, this file is the simple controller function. In order to show which structural elements were covered, the code includes the instrumentation (implemented using preprocessor macros beginning _AST_INSTRUMENT or OPT_INSTRUMENT), plus additional code that enables the SUT to be linked directly with the search algorithm.

A Principled Evaluation of the Effect of Directed Mutation on Search-Based Statistical Testing
4th International Workshop on Search-Based Software Testing - SBST 2011

To view the directory of supplemental files, click here.

The subdirectory data contains the responses from individual experimental trails (algorithm runs), with filenames of the form ssss_aaaa_pppp.csv, where:

  • ssss is the SUT (bestMove, nischneu, cArcsin, fct3);
  • aaaa is the algorithm (DM- or DM+); and
  • pppp is the type of experiment (untuned - parameters before tuning, tuned - parameters at predicted optimal parameters, or RSM iteration/phase in the form ixpy where x is the iteration and y the phase).

In these files, the first column is the response (logarithm of the adjusted number of SUT executions as described in the paper). For RSM phase 2 experiments only, the second column is the distance in steps along the steepest descent path. The remaining 9 (for DM-) or 10 (for DM+) columns are the coded parameters at which the experiment was to run (converted back from the actual parameter values to account accurately for rounding).

The subdirectory suts contains source of the SUTS, apart from that for FCT3 which we unable to share. The macros beginning _AST, and the additional function arguments elementCountArray, outputs and elementCount, implement the instrumentation.

Efficient Software Verification: Statistical Testing Using Automated Search
Transactions on Software Engineering, 36(6) 763–777, 2010

To view the directory of supplemental files, click here.

The directory contains:

  • Source code for each of the three SUTs.
  • The function itself is in the header file and includes macros that enable instrumentation to detect branch coverage. Building the corresponding *.c file (e.g. simpleFunc.c) produces an executable without instrumentation, while building the *_inst.c file (e.g. simpleFunc_inst.c) produces an executable that outputs instrumentation data to stderr. For both executables, function arguments are passed as command line parameters.

    • simpleFunc: simpleFunc.h simpleFunc.c simpleFunc_inst.c
    • bestMove: bestMove5.h bestMove5.c bestMove5_inst.c
    • nsichneu: nsichneu.h nsichneu.c nsichneu_inst.c

  • Data for each of the four experiments.
    • Experiment A
    • expt_A_*_response.txt contains the response (probability lower bound) for each trial. Each row corresponds to a separate trial. The columns are explained in expt_A_response_desc.txt; in particular, note that '1' in the second column indicates a 'succesful' trial, and '0' an 'unsuccessful' one.

      expt_A_*_timing.txt contains the timings (as CPU user time) for each trial.

    • Experiment B
    • expt_B_*_response.txt contains the response (mutation scores) for each trial. The columns are explained in expt_B_response_desc.txt. Each probability distribution is uniquely identified by a distribution id. Statistical testing distributions have non-zero ids; the uniform distribution used for random testing has an id of zero. (The file expt_B_bestMove_response_150.txt is an additional run of the experiment for only the uniform random distribution for test sizes up to 150. The data are used to determine the test size at which a mutation score of 0.55 is achievable using random testing, in order to perform a more realistic cost-benefit analysis than would be possible with the original experimental data that consider test sizes up to 50.)

      expt_B_*_coverage.txt determines the number of samples taken from each distribution until the coverage criterion (all branches) is achieved. The columns are explained in expt_B_coverage_desc.txt. Each row corresponds to the equivalent row in expt_B_*_response.txt in that the same samples, in the same order are taken from the distribution. Note that the data for uniform distributions (zero id) are not used in the analysis described in the paper (and is incomplete for some SUTs).

    • Experiment C
    • expt_C_*_response.txt contains the response (mutation scores) for each trial. The columns are explained in expt_C_response_desc.txt. Each probability distribution is uniquely identified by a trajectory iteration id.

    • Experiment D
    • expt_D_*_response.txt contains the response (mutation scores) for each trial. The columns are explained in expt_D_response_desc.txt. Each probability distribution is uniquely identified by a combination of diversity constraint and distribution id (i.e. distribution id is unique only within the same setting of diversity constraint). A value of '1' in the second column indicates a diversity constraint was used during the search, and a value of '0' that it was not.