nwsystest

introduction

nwsystest is a simple tool to help test full software systems - potentially consisting of multiple concurrent applications and threads - in an automated manner. It has only been developed (on and off - mostly "off") since August 2010, but already offers useful functionality. It was rapidly created to help develop a complex client-server filesharing system, and already works well for this purpose. There are many imperfections and missing things, but I'm releasing this project regardless in order to solicite ideas for improvements.

nwsystest drives the execution of your software, and collects and evaluates results in accordance with textual test description files. The idea is to notify nwsystest of things ("events") that are happening in your software, thereby enabling it to trace and verify whether the software is doing the right (expected) thing. You can think of it as a more formalized type of debugging messages which serve as checkpoints for verification.

It is your task to:

The concept is best explained with a few examples. Let's consider a trivial testcase, as defined by the following test description file:

# Test 1
participant prog = ./program
expectexit prog
{ prog { event hello_world } }

This test description file, when executed by nwsystest, will start your program with the identity "prog". It then expects your program to produce exactly one event named "hello_world", not more and not less. No other events are allowed. The test can be executed by saving it to a *.test file - e.g. hello.test - and either running

cat hello.test | nwsystest

or just running

nwsystest

in the same directory (this will execute every .test file in the current working directory). Here is a corresponding, valid program which will "pass" that test outlined above:

/*
 * Program to be tested by test 1
 */
 #include "nwsystest.h"
 
 int

 main(int argc, char **argv) {
         int sock;

 
         nwst_eval_args(&argc, argv); /* Initialization */
         if (!nwst_is_testing()) {

                 /*
                 * This is NOT a test run. The program is
		 * executed normally, presumably by the
                 * end user
                 */
                 ;
         } else {
                 /*
                  * The program was started by nwsystest -
                  * we need to connect to it to take part
                  * in the intended test
                  */
                 /* Connect to nwsystest */ 
                 sock = nwst_connect(NULL); 
                 if (sock != -1) {

                         /*
                          * Indicate readiness (nwsystest
                          * may now send commands)
                          */
                         nwst_ready(sock); 
                          /* Send event */
                         nwst_emit_event(sock, "hello_world");

                 }
         }
 }

In this test, nwsystest does not control your application; It does not tell it what to do but merely awaits the unconditional hello_world event. Let's consider a more elaborate example in which commands are sent to your sofware, and which also has more than one participant: A dummy client-server system in which we wish to test a file transfer function.

# Test 2

# Start participants client and server, redirecting their output (stdout/
# stderr) to the respective log files client.log and server.log
participant client = ./client >client.log
participant server = ./server >server.log

# Tell nwsystest that both participants may exit by themselves after
# processing all commands and emitting all expected events (without
# this allowance they'd have to wait for nwsystest to send a termination
# request. This flags crashes). This is done to simplify the test
# programs
expectexit client
expectexit server

# Wait for server connection to be established
{ client { event server_online } }


# Send download request to client
cmd client get foo.mp3 {
        # List of events that we expect the server to generate in
        # response to the "get foo.mp3" command
        server {
                event sending_file = "foo.mp3"
                event transfer_complete
        }
        # List of events that we expect the client to generate in
        # response to the "get foo.mp3" command
        client {
                event receiving_file
                event transfer_complete = download.dat = reference_foo.mp3
        }
}

You can find the full client and server program source code along with Makefile and test description file in the examples/tests.full sub directory of the nwsystest package (they are much simplified and do not pay full attention to good programming practice).

The server will listen for a client connection. After the client has successfully connected, it will read commands from the client - in normal operation, that's the user typing at the keyboard, and while testing it is input received from nwsystest (the commands you listed in your test description file). The client then expects to receive a file from the server socket - since "get" is the only implemented program feature - and will save it as "download.dat". After processing the first and only command, both programs exit.

The client is expected to produce a file download.dat, which is expected to have the exact same content as the file reference_foo.mp3, which has to exist prior to running the test. Both of these assumptions may be violated by faulty client or server code, in which case the test fails.

In order to set up this test, we have to:

nwsystest © Nils Weller 2011 SourceForge Logo