Enforcer project, v0.9
======================

Enforcer - a tool to enforce exception coverage for difficult-to-test cases.

Enforcer will run all JUnit-based unit tests, analyze exception
coverage, and re-run an instrumented version of your program which
"manually" forces code exception to go through previously uncovered
exceptions.

This allows code to be tested that is normally not directly testable,
e.g. error handling for a network connection.

* Architect, designer, project leader, manager, and currently sole author:
  Cyrille Artho (c.artho@aist.go.jp)

* REQUIRES Java 1.5 JVM! Java 1.4 does not support this new form
  of dynamic code instrumentation.

* Should work with any kind of JUnit 3.8.1 test setup, but has only
  been tested for tests called by junit.textui.TestRunner.run(TestSuite
  suite). Other test runners are supported as long as they use the
  same signature (most importantly, an argument of type TestSuite).

* You should only call unit tests once (which is normally the case),
  and that code should be reachable from the main method:
  static void main(String[] args).

Installation:
-------------

* From source: compile the source files using
  ./configure
  make

* Install enforcer.jar, serp.jar, junit3.8.1.jar in your CLASSPATH.

* Make sure you use the correct path names for enforcer.jar and
  your own test application.

How to run:
-----------

java -javaagent:<path to jar file> <normal program and options>

So essentially you just have to add, as the first argument, option 
"-javaagent:<path>/enforcer.jar" to the command line.

Note: Java 1.4 or lower does not support this option.

Example:
--------

See TestExample.java. To run the unit test suite normally, use
  java TestExample
One test will be executed, exceptions will not be tested.
To test the exceptions, use Enforcer:
  java -javaagent:enforcer.jar TestExample
After the first normal test run, Enforcer detects four untested
exceptions; the given unit test will be re-run four more times,
each time triggering a different exception.

Caveats:
--------

Errors in instrumentation may not result in any error message, but
broken class files. Your program may not run at all, or your JVM
may just hang. This makes it difficult to debug this tool. Such
problems can even occur if the classpath is not correct, e.g. if
the serp.jar instrumentation library is not included.

How it works:
-------------

The code instrumentation package analyzes each exception and
instruments coverage measuring code. This is done on the fly, while
classes are loaded, i.e., even after the first loaded classes are
already executing. Whenever a call to
junit.<something>.TestRunner.run(TestSuite suite) is discovered,
it is instrumented such that coverage measurement is enabled.

After execution of the normal test run, a summary information is
printed, giving the number of instrumented try blocks, the number
of executed try and catch blocks, and uncovered blocks.

In a second test execution, control flow is artificially altered
to run through uncovered catch blocks after each try block has
been executed until its last instruction. Only one try block is
artificially altered during each execution, in order to make
debugging possible. This may result in a single unit test to be
run several times, forcing execution through different paths each
time.

Such instrumented test re-runs may occur several times, in order
to cover nested try/catch blocks. Note that the nesting depth
supported by the tool is finite. This was a design decision made
for efficiency.

Note: unreachable try blocks (obviously) cannot be tested with this
tool; you have to improve your test suite such that such dead code
becomes reachable.

Supported configuration:
------------------------

Configuration is performed by environment variables.

ENFORCER_LOGLEVEL:	Default "2" = "INFO".
			Lower it to suppress output, increase it
			for debugging information.

ENFORCER_INSTR_DUMP:	Default: empty.
			Directory where the currently instrumented
			class file is written to (for debugging
			purposes only).

ENFORCER_INSTR_TESTCODE:	Default: no.
			If set to "1", "yes", or "true",
			instrumentation will also instrument code
			in test case classes, which should normally
			not be instrumented.

ENFORCER_DYN_EXC_CHECK:	Default: no
			If set to "1", "yes", or "true",
			exception types will again be checked
			dynamically. Currently slow but may reduce
			false positives if subclasses do not throw
			exceptions declared in the super class.

ENFORCER_SUPPR_ARTIF_EXC:	Default; yes
			If set to "1", "yes", or "true",
			the stack trace of artificially generated
			exceptions will not be shown. This affects
			both calls within test cases to
			Exception.printStackTrace(), as well as
			the evaluation of such tests within JUnit
			as "failures".
			Set env. var. to "0" to disable this.