test/README.md

# Apache Thrift - integration test suite

This is the cross everything integration test suite for Apache Thrift.

## Run

### A. Using Make

The test can be executed by:

    make cross

This starts the [test.py](test.py) script which does the real cross test with
different transports, protocols and languages.

Note that this skips any language that is not built locally. It also skips
tests that are known to be failing. If you need more control over which tests
to run, read following section.

### B. Using test script directly

Alternatively, you can invoke [test.py](test.py) directly. You need to run`make
precross` once before executing it for the first time.

For example, if you changed something in `nodejs` library and need to verify
the patch, you can skip everything except `nodejs` itself and some reference
implementation (currently `cpp` and `java` are recommended) like this:

    ./configure --without-c_glib --without-erlang --without-lua ...
    make precross -j8
    test/test.py --server cpp,java --client nodejs
    test/test.py --server nodejs --client cpp,java

Another useful flag is --regex. For example, to run all tests that involve
Java TBinaryProtocol:

    test/test.py --regex "java.*binary"

## Test case definition file

The cross test cases are defined in [tests.json](tests.json).
The root element is collection of test target definitions.
Each test target definition looks like this:

    {
      "name": "somelib",

      "client": {
        "command": ["somelib_client_executable"],
        "workdir": "somelib/bin",
        "protocols": ["binary"],
        "transports": ["buffered"],
        "sockets": ["ip"],
      },
      "server": {
        "command": ["somelib_server_executable"],
        "workdir": "somelib/bin",
        "protocols": ["binary"],
        "transports": ["buffered"],
        "sockets": ["ip", "ip-ssl"],
      }
    }

Either client or server definition or both should be present.

Parameters that are common to both `client` and `server` can be put to target
definition root:

    {
      "name": "somelib",

      "workdir": "somelib/bin",
      "protocols": ["binary"],
      "transports": ["buffered"],
      "sockets": ["ip"],

      "client": { "command": ["somelib_client_executable"] },
      "server": {
        "command": ["somelib_server_executable"],
        "sockets": ["ip-ssl"]
      }
    }

For the complete list of supported keys and their effect, see source code
comment at the opt of [crossrunner/collect.py](crossrunner/collect.py).


## List of known failures

Since many cross tests currently fail (mainly due to partial incompatibility
around exception handling), the test script specifically report for "not known
before" failures.

For this purpose, test cases known to (occasionally) fail are listed in
`known_failures_<platform>.json` where `<platform>` matches with python
`platform.system()` string.

Currently, only Linux version is included.

FYI, the file is initially generated by

    test/test.py --update-expected-failures=overwrite

after a full test run, then repeatedly

    test/test.py --skip-known-failures
    test/test.py --update-expected-failures=merge

to update the known failures, run

    make fail

## Test executable specification

### Command line parameters

Unit tests for languages are usually located under lib/<lang>/test/
cross language tests according to [ThriftTest.thrift](ThriftTest.thrift) shall be
provided for every language including executables with the following command
line interface:

**Server command line interface:**

    $ ./TestServer -h
    Allowed options:
      -h | --help                  produce help message
      --port=arg (9090)            Port number to listen
      --domain-socket=arg          Unix Domain Socket (e.g. /tmp/ThriftTest.thrift)
      --pipe=arg                   Windows Named Pipe (e.g. MyThriftPipe)
      --server-type=arg (simple)   type of server, "simple", "thread-pool",
                                   "threaded", or "nonblocking"
      --transport=arg (buffered)   transport: buffered, framed, http, anonpipe, zlib
      --protocol=arg (binary)      protocol: binary, compact, header, json
      --multiplex                  Add TMultiplexedProtocol service name "ThriftTest"
      --abstract-namespace         Create the domain socket in the Abstract Namespace 
                                   (no connection with filesystem pathnames)
      --ssl                        Encrypted Transport using SSL
      --zlib                       Wrapped Transport using Zlib
      --processor-events           processor-events
      -n=arg | --workers=arg (=4)  Number of thread pools workers. Only valid for
                                   thread-pool server type

**Client command line interface:**

    $ ./TestClient -h
    Allowed options:
      -h | --help                  produce help message
      --host=arg (localhost)       Host to connect
      --port=arg (9090)            Port number to connect
      --domain-socket=arg          Domain Socket (e.g. /tmp/ThriftTest.thrift),
                                   instead of host and port
      --pipe=arg                   Windows Named Pipe (e.g. MyThriftPipe)
      --anon-pipes hRead hWrite    Windows Anonymous Pipes pair (handles)
      --abstract-namespace         Create the domain socket in the Abstract Namespace
                                   (no connection with filesystem pathnames)
      --transport=arg (buffered)   Transport: buffered, framed, http, evhttp, zlib
      --protocol=arg (binary)      Protocol: binary, compact, header, json
      --multiplex                  Add TMultiplexedProtocol service name "ThriftTest"
      --ssl                        Encrypted Transport using SSL
      --zlib                       Wrap Transport with Zlib
      -n=arg | --testloops=arg (1) Number of Tests
      -t=arg | --threads=arg (1)   Number of Test threads

If you have executed the **make check** or **make cross** then you will be able to browse
[gen-html/ThriftTest.html](gen-html/ThriftTest.html) with the test documentation.

### Return code

The return code (exit code) shall be 0 on success, or an integer in the range 1 - 255 on errors.
In order to signal failed tests, the return code shall be composed from these bits to indicate
failing tests:

      #define TEST_BASETYPES     1  // 0000 0001
      #define TEST_STRUCTS       2  // 0000 0010
      #define TEST_CONTAINERS    4  // 0000 0100
      #define TEST_EXCEPTIONS    8  // 0000 1000
      #define TEST_UNKNOWN      64  // 0100 0000 (Failed to prepare environment etc.)
      #define TEST_TIMEOUT     128  // 1000 0000
      #define TEST_NOTUSED      48  // 0011 0000 (reserved bits)

Tests that have not been executed at all count as errors.

**Example:**

During tests, the test client notices that some of the Struct tests fail.
Furthermore, due to some other problem none of the Exception tests is executed.
Therefore, the test client returns the code `10 = 2 | 8`, indicating the failure
of both test 2 (TEST_STRUCTS) and test 8 (TEST_EXCEPTIONS).


## SSL
Test Keys and Certificates are provided in multiple formats under the following
directory [test/keys](keys)