Jacks (Jacks is an Automated Compiler Killing Suite)

Jacks Logo Betting on Java? You had better be holding a good hand.

What is Jacks?

Jacks is a free test suite designed to detect bugs in a Java compiler. We guarantee that it will find at least one bug in any supported Java compiler, in fact it may find more than one. Jacks is not designed to test a Java runtime (JVM) or Java class libraries. The Mauve project already aims to do that.

Jacks includes a collection of Java compiler tests contributed by people on the net. Jacks is free software licensed under the terms of the GPL. A test case must be licensed under the terms of the GPL to be suitable for inclusion into Jacks, but authors retain the original copyright. The license used by the Jacks test suite does not effect your ability to run the suite with any free or non-free Java compiler.

The layout of directories in the Jacks test suite mimics chapters from the JLS.
The Java Language Specification Version 2 can be found at:

  • http://java.sun.com/docs/books/jls/second_edition/html/jTOC.doc.html
  • http://java.sun.com/docs/books/jls/html (Old Version)

    • Install Requirements

    Tcl version 8.3 is required as it provides the regression testing framework Jacks depends on. Older versions of Tcl will not work as they do not include the tcltest extension and they do not support Unicode character sets.

    Windows users can download the Tcl/Tk for Windows installer .

    Red Hat 6.X Linux (x86) users can download an RPM . Red Hat 7.X ships with Tcl 8.3.

    Everyone else will need to build it from source: tcl8.3.4.tar.gz .

    Once Tcl is installed, you will need the Jacks source code from the CVS. The CVS module for jacks can be fetched like so: 

    setenv CVSROOT :pserver:anoncvs@oss.software.ibm.com:/usr/cvs/jikes

cvs login
paswsd anoncvs

cvs checkout jacks

    Configure Jacks for your system

    Before running the tests, you will need to tell Jacks where you have installed each of the compilers you intend to use. A setup file exists for each supported compiler. For example, if you want to run tests with the jikes compiler, you would need to edit the jikes_setup configuration file so that includes the path name where your jikes compiler is installed. You may also want to set the CLASSPATH that will be used for the tests and the optional path name of the Java runtime to use for runtime tests. You can add any number of configuration files. To remove a configuration you do not want, just delete the setup file for that configuration.

    Running Jacks regression tests

    To run the tests, invoke the jacks shell script and pass it the name of the compiler you want to test. You will want to put the toplevel directory on your PATH so that you can run the jacks shell script. Test the javac compiler like this:

    % jacks javac

    The above command will recurse into subdirectories and run all the tests it finds using the javac compiler. If any of the tests fail, an error will be displayed.

    You can also run tests with multiple compilers using a single command. This example runs the tests using javac and then jikes.

    % jacks "javac jikes"

    It is also possible to run tests using all of the supported compilers. The following command shows how this is done:

    % jacks all

    You can run the tests in a specific subdirectory like this: 

    % cd tests/jls/packages/package-declarations/unnamed-packages
% jacks jikes

    In addition, you can pass patterns as the third argument. In this way, you can run only those tests that match a given pattern. 

    % jacks jikes "*int*"
    The above command will run all the test with names that match the pattern *int*. You can also pass multiple patterns. 
    % jacks jikes "*int* test-1 *foo-1"

    The last and perhaps most interesting feature of Jacks is the ability to generate regression reports. These reports show how the current test results have changed with respect to the previous results. This is very useful since you can run the tests and check the results to see if any changes you made caused regressions. This really makes a big difference when there are a large number of tests, since otherwise you would not know if a failing test was failing before or is now failing because of your changes. To generate a changes file for the jikes target, run: 

    % jacks loggen jikes
    That will generate a logging/jikes.changes. The first set of entries are the most recent changes. A transition of {PASSED FAILED} indicates a regression while a transition of {FAILED PASSED} indicates a bug fix. Note that generating a changes file requires a cvs client and network access.

    The Jacks suite also includes test that require a Java runtime in addition to a Java compiler. These tests are located in the tests/runtime directory.

    Tests that apply to only a specific compiler are included in a top level directory named after the compiler. For example, tests in the tests/gcj directory will only be run when testing with the gcj compiler.

    You can clean up generated files after a test run with the following command: 

    % jacks clean

    Adding a new test case

    Adding a new test case to Jacks is very easy. One of the main goals for Jacks was to create a set of tests that could be extended with minimal effort. A detailed example of Adding A Test Case is provided.

    Review existing tests

    A nicely cross referenced index of every test in the Jacks suite is provided. You can look up a test case by test number, find the JLS section that a given test directory refers to, review each of the test that apply to a given compiler, or simply review all the test that apply to a given JLS section.

    Jacks mailing list

    You are welcome to join the Jacks mailing list and discuss anything related to the Jacks test-suite.