This is Mauve, a free test suite for the Java Class Libraries.

Mauve is intended to test several different varieties of the
libraries.  For instance, it will contain tests that are specific to a
particular JDK version.  Tags in the test files help the test
framework decide which tests should or should not be run against a
given runtime.


To build, first run configure.  You can control the configuration with
some environment variables:

     JAVA   Name of Java interpreter to use
     JAVAC  Name of Java (to class) compiler to use
     GCJ    Name of Java (to object) compiler to use

GCJ is only used when the `--with-gcj' option is given to configure.

Note that you will need GNU make to use this testsuite.  If your
installation provides GNU make under a different name, such as gmake,
replace `make' with `gmake' in the following.

Use `make check' to run the tests.  You can set the make variable
`KEYS' to select a subset of the tests.  KEYS is a list of keys that
must be matched by the files to be tested.  Some values:

     * Any key starting with `java.' is taken to be the name of a
       class hierarchy.
       E.g., the key `java.lang' matches only test classes in java.lang.*.

     * Any key starting with `!java.' is used to exclude a class
       hierarchy.
       E.g., `!java.lang.reflect' can be used to omit all reflection
       tests, while still including the rest of java.lang.

     JDK1.0  Run JDK1.0 tests only
     JDK1.1  Run JDK1.1 tests only
     JDK1.2  Run JDK1.2 tests only

If an otherwise unrecognized tag QUUX is seen, and the file
`mauve-QUUX' exists in the mauve source directory, then the contents
of this file are treated as a list of tags.  For instance, here is the
current version of `mauve-libjava', which is used by the `libjava'
library implementation:

	# Config file that tells mauve about the `libjava' tag.
	JDK1.0
	JDK1.1
	!java.beans

Note that anything after a `#' on a line is treated as a comment and
is stripped.

File inclusion and exclusion commands are processed in order.  So, for
instance, specifying `!java.lang java.lang.System' will omit all
java.lang tests except for System.  Every file starts off in the
`included' state; an explicit exclusion is needed to reject it.  (So,
e.g., specifying just `java.lang' is insufficient to select only
java.lang.  You must specify `!java. java.lang'.  Note the `.'!)

If no tags are given, "JDK1.0 JDK1.1" are assumed.

You can use `make recheck' to re-run `make check' with the key list
you last used.


If you want to run a single test by hand, you can feed its name
directly to the test harness.  Make sure to remember the `gnu.testlet'
prefix for the test cases.  E.g.:

  echo gnu.testlet.java.lang.Character.classify | \
    java gnu.testlet.SimpleTestHarness -verbose -debug PATH-TO-SRCDIR

The optional `-verbose' command-line argument makes the test suite a
little noisier about what it is doing.  In particular it will print
information about caught exceptions in this case.

Some tests may provide even more information about failures to aid
with debugging a particular run-time system.  These messages are enabled
by specifying the optional command-line argument `-debug'

If you are only interested in the results of the tests (not only which
ones FAIL but also which ones PASS). Then you can use the '-resultsonly'
command-line argument. This will also suppress the printing of a summary
at the end.

You may use the environment variable TESTFLAGS to provide these
flags to the invocation of the SimpleTestHarness in the `check' target.
For instance,

  make check "TESTFLAGS=-verbose -debug"

will run the testsuite in verbose and debug mode.

================================================================

Tags in a test are specified a little differently from tags on the
command line.

Each tag on the command line is first mapped to a list of actual tags.
E.g., "JDK1.2" implies all the tags "JDK1.0", "JDK1.1", and "JDK1.2".

If any tag from the expanded list is matched by the test case, then
the test case is chosen.

However, if one of the tags specified on the command line appears in
the test with a `!' prefix, then the test is rejected.

Tags must all appear on a single line beginning "// Tags: ".

Many files test functionality that has existed since JDK1.0.  The
corresponding line in the source:

    // Tags: JDK1.0

Here is how you would tag something that first appeared in JDK1.2:

    // Tags: JDK1.2

Here is how you would tag something that was eliminated in JDK1.2:

    // Tags: JDK1.0 !JDK1.2

The idea behind this scheme is that it is undesirable to update all
the files whenever we add a tag.  So instead most tags are defined in
terms of primitive tags, and then we note the exceptions.

When adding a new tag, change the `choose' program to map the
specified tag onto the implied tags.  There is some code near the top
that handles this transformation.

Files should only hold tags describing their prerequisites.  In
particular, limitations of a given library implementation should not
be mentioned in file tags (because when the library changes, this
would necessitate global edits).  Instead, put such limitations in a
`mauve-QUUX' tag expansion file.

================================================================

Some test cases may require extra utility classes to run.  When the
choose script selects a test case for running, the framework
identifies the supporting classes through another magic comment in the
test source.

Support classes must all appear on a single line beginning with the
string "// Uses: ".  The framework assumes that all utility classes
used are found in that same package as the test case.

================================================================

The test harness can also ignore known failures.  Simply create a
file 'xfails' in the directory where the tests are being run which
contains 'FAIL:' entries from previous test runs.  The order of the
lines in the file is immaterial.  Also, the -verbose flag must be
used.  Totals for both XFAILs and XPASSes will be output at the
end of the run.

In this way, implementations can track known failures and subsequent
test runs can thus highlight regressions.

================================================================

There are still a few things to do in the test framework.


It would be nice if we could have tests that can specify their
expected output.  The expected output could be encoded directly in the
test, e.g.:

      /*{
      expected output here
      }*/

The test harness would be reponsible for extracting this from the test
source and then setting things up so that the checking is done
correctly.  SimpleTestHarness could do this by setting System.out to
point to some string buffer for the duration of the test.


Change things so that the .o files can be built directly from the
.java files without any intermediate .class files.  (Use the same
configuration options that libjava uses.)


Some tests probably should be run in their own environment.  This
could be implemented using a new "group" magic comment, along with
changes to the `choose' program to generate the list of classes in
chunks.  Each such chunk would be fed into SimpleTestHarness in a
separate invocation.


It would be interesting to be able to compare test results for
unspecified things against Sun's implementation.  This could be done
by adding a new method to the test harness.  The `expected' argument
would come from Sun's implementation.  Unlike `check', a failure here
would simply be informative.
