THRIFT-2578 Moving 'make cross' from test.sh to test.py
diff --git a/test/README.md b/test/README.md
index bc1828c..ef874ac 100755
--- a/test/README.md
+++ b/test/README.md
@@ -1,17 +1,116 @@
# Apache Thrift - integration test suite
This is the cross everything integration test suite for Apache Thrift.
-executed by
+
+## Run
+
+### A. Using Make
+
+The test can be executed by:
make cross
-at the moment, this starts the test.sh script which does the real cross test
-with different transports, protocols and languages.
+This starts the [test.py](test.py) script which does the real cross test with
+different transports, protocols and languages.
+
+Note that this requires basically everything to be built, otherwise test cases
+for missing languages will fail. If you skip building some languages, or simply
+need more control, read following section.
+
+### B. Using test script directly
+
+Alternatively, you can invoke [test.py](test.py) directly. You need to run`make
+precross` once before executing it for the first time.
+
+For example, if you changed something in `nodejs` library and need to verify
+the patch, you can skip everything except `nodejs` itself and some reference
+implementation (currently `cpp` and `java` are recommended) like this:
+
+ ./configure --without-c_glib -without-csharp --without-erlang --without-lua ...
+ make precross -j8
+ test/test.py --server cpp,java --client nodejs
+ test/test.py --server nodejs --client cpp,java
+
+## Test case definition file
+
+The cross test cases are defined in [tests.json](tests.json).
+The root element is collection of test target definitions.
+Each test target definition looks like this:
+
+ {
+ "name": "somelib",
+
+ "client": {
+ "command": ["somelib_client_executable"],
+ "workdir": "somelib/bin",
+ "protocols": ["binary"],
+ "transports": ["buffered"],
+ "sockets": ["ip"],
+ },
+ "server": {
+ "command": ["somelib_server_executable"],
+ "workdir": "somelib/bin",
+ "protocols": ["binary"],
+ "transports": ["buffered"],
+ "sockets": ["ip", "ip-ssl"],
+ }
+ }
+
+Either client or server definition or both should be present.
+
+Parameters that are common to both `client` and `server` can be put to target
+definition root:
+
+ {
+ "name": "somelib",
+
+ "workdir": "somelib/bin",
+ "protocols": ["binary"],
+ "transports": ["buffered"],
+ "sockets": ["ip"],
+
+ "client": { "command": ["somelib_client_executable"] },
+ "server": {
+ "command": ["somelib_server_executable"],
+ "sockets": ["ip-ssl"]
+ }
+ }
+
+For the complete list of supported keys and their effect, see source code
+comment at the opt of [crossrunner/collect.py](crossrunner/collect.py).
+
+
+## List of known failures
+
+Since many cross tests currently fail (mainly due to partial incompatibility
+around exception handling), the test script specifically report for "not known
+before" failures.
+
+For this purpose, test cases known to (occasionally) fail are listed in
+`known_failures_<platform>.json` where `<platform>` matches with python
+`platform.system()` string.
+
+Currently, only Linux version is included.
+
+FYI, the file is initially generated by
+
+ test/test.py --update-expected-failures=overwrite
+
+after a full test run, then repeatedly
+
+ test/test.py --skip-known-failures
+ test/test.py --update-expected-failures=merge
+
+## Test executable specification
+
+### Command line parameters
Unit tests for languages are usually located under lib/<lang>/test/
cross language tests according to [ThriftTest.thrift](ThriftTest.thrift) shall be
provided for every language including executables with the following command
-line interface for servers:
+line interface:
+
+**Server command line interface:**
$ ./cpp/TestServer -h
Allowed options:
@@ -19,23 +118,23 @@
--port arg (=9090) Port number to listen
--domain-socket arg Unix Domain Socket (e.g. /tmp/ThriftTest.thrift)
--named-pipe arg Windows Named Pipe (e.g. MyThriftPipe)
- --server-type arg (=simple) type of server, "simple", "thread-pool",
+ --server-type arg (=simple) type of server, "simple", "thread-pool",
"threaded", or "nonblocking"
--transport arg (=buffered) transport: buffered, framed, http, anonpipe
--protocol arg (=binary) protocol: binary, compact, json
--ssl Encrypted Transport using SSL
--processor-events processor-events
- -n [ --workers ] arg (=4) Number of thread pools workers. Only valid for
+ -n [ --workers ] arg (=4) Number of thread pools workers. Only valid for
thread-pool server type
-and this for clients:
+**Client command line interface:**
$ ./cpp/TestClient -h
Allowed options:
-h [ --help ] produce help message
--host arg (=localhost) Host to connect
--port arg (=9090) Port number to connect
- --domain-socket arg Domain Socket (e.g. /tmp/ThriftTest.thrift),
+ --domain-socket arg Domain Socket (e.g. /tmp/ThriftTest.thrift),
instead of host and port
--named-pipe arg Windows Named Pipe (e.g. MyThriftPipe)
--anon-pipes hRead hWrite Windows Anonymous Pipes pair (handles)
@@ -43,13 +142,15 @@
--protocol arg (=binary) Protocol: binary, compact, json
--ssl Encrypted Transport using SSL
-n [ --testloops ] arg (=1) Number of Tests
- -t [ --threads ] arg (=1) Number of Test threads
+ -t [ --threads ] arg (=1) Number of Test threads
If you have executed the **make check** or **make cross** then you will be able to browse
[gen-html/ThriftTest.html](gen-html/ThriftTest.html) with the test documentation.
-The return code (exit code) shall be 0 on success, or an integer in the range 1 - 255 on errors.
-In order to signal failed tests, the return code shall be composed from these bits to indicate
+### Return code
+
+The return code (exit code) shall be 0 on success, or an integer in the range 1 - 255 on errors.
+In order to signal failed tests, the return code shall be composed from these bits to indicate
failing tests:
#define TEST_BASETYPES 1 // 0000 0001
@@ -59,16 +160,15 @@
#define TEST_NOTUSED 240 // 1111 0000 (reserved bits)
Tests that have not been executed at all count as errors.
-
-### Example:
-During tests, the test client notices that some of the Struct tests fail.
-Furthermore, due to some other problem none of the Exception tests is executed.
-Therefore, the test client returns the code `10 = 2 | 8`, indicating the failure
-of both test 2 (TEST_STRUCTS) and test 8 (TEST_EXCEPTIONS).
+**Example:**
+
+During tests, the test client notices that some of the Struct tests fail.
+Furthermore, due to some other problem none of the Exception tests is executed.
+Therefore, the test client returns the code `10 = 2 | 8`, indicating the failure
+of both test 2 (TEST_STRUCTS) and test 8 (TEST_EXCEPTIONS).
## SSL
Test Keys and Certificates are provided in multiple formats under the following
directory [test/keys](keys)
-