nwsystest

Contents



Basics

Running just

nwsystest

in a directory will execute every file with a .test ending in that directory. Alternatively, a single test case can be executed by passing it through stdin;

cat name.test | nwsystest

A test is considered passed if all software participants involved produced the events and data that were expected by the test description file. Otherwise it failed. It is possible to check your software with valgrind automatically as well - more about that later.

nils@trashcan ~/nwlanshare/tests [0]> nwsystest
Test `/home/nils/nwlanshare/tests/startup0.test' ... OK
Test `/home/nils/nwlanshare/tests/list2.test' ... OK
Test `/home/nils/nwlanshare/tests/list1.test' ... OK
Test `/home/nils/nwlanshare/tests/list0.test' ... OK
Test `/home/nils/nwlanshare/tests/get0.test' ... ERROR
Test `/home/nils/nwlanshare/tests/list3.test' ... OK

NOTE: nwsystest does NOT handle hanging software gracefully yet! If your system stops producing events and data, then nwsystest will wait indefinitely for it to resume. Additionally, nwsystest does not properly keep failed test results apart yet. That is to say that if 10 of 100 tests in your test suite fail, then you may only be able to reconstruct the sequence of events for the last failed test. Note also that nwsystest does not limit memory or disk usage of your software; A runaway system is not stopped by nwsystest.

Therefore, it is recommended (for now) to debug tests individually and in order. Run "nwsystest -loop" to stop the test suite as soon as the first test fails, such that you can look into what happened (this option will also rerun the test suite indefinitely - more about that later) You can run

nwsystest -verbose

to get more info about what is happening (whether participants connect to the test server properly, whether they emit expected events, which commands are being executed, and so on). A copy of the verbose output for the last test run is saved to the file nwsystest.log. So you can run "nwsystest -loop" without -verbose option and look at nwsystest.log after a test failed to see what happened.



Test description file

    General

A test description file represents the entire life cycle of an instance of your software; Your software is started, receives commands, emits events and data, and is then terminated (first by asking it to, later by shutting it down forcefully).

The test description file has three fundamental and related concepts:

Test description files may use "include" directives to move commonly used settings into external central data files. Various things can be tweaked - whether you wish to ignore certain events, whether you expect events to arrive in no particular order, and so on.

Essentially a test description file looks like this:

participant1
participant2
....
cmd1 { eventlist }
cmd2 { eventlist }
....

... potentially mixed with a few control directives.

Commands and event lists in the test description file are executed in the order in which they appear in the file, but most other control directives apply to the entire file - if the last line in your file is a participant definition, then that participant will still be started at the very beginning (currently only the "ordering" directive honors order as well).

    include directives

The "include" directive allows you to move things to external files. For example, if all (or at least a few) tests in your test suite involve the exact same participants with the exact same command line arguments, then you can simply write those participant definitions in an external file - e.g. "participants.part" - and put an

include participants.part

... at the top of every test description file. This also allows you to reuse code - common code passages need only be changed in one place.

Note that the include directive currently does not allow nesting yet; An included file may not include other files.

    Initialization/Deinitialization

You will probably wish to provide your system with a predictable environment. For example, if your system accesses a corrupted database, it may perform a repair routine, delete or otherwise mark it as invalid. Your system may save state to configuration and data files.

Persistent results of your system can cause tests to behave differently every time they are executed. A test may fail in the first iteration but work in the second. Test 1 may affect the outcome of test 2 because it left behind certain data files that make a difference.

In order to avoid this, you should reset your system to a clean state any time a test is started. nwsystest offers two ways to accomplish this:

These things are intended to offer a way in which you can delete data and configuration files, create test data files, and so on.



Programming basics

Your program needs to call various nwsystest library routines to connect to the test server and receive commands and report results. It has to include the library header file "nwshlib.h", and link the static library libnwsystest.a (e.g. by passing the full path "/usr/local/lib/libnwsystest.a" on your compiler link line). The easiest way is to keep these library calls compiled into your program at all times, but to avoid calling some of them if your program is not in testing mode. To determine whether your program is being run through nwsystest, you can check:

nwst_is_testing()

This allows you to decide things like whether to take input from the normal user interface or from nwsystest - and whether you connect to nwsystest at all.

Your program should always call

nwst_eval_args(&argc, argv);

at the beginning of the main() function regardless of nwst_is_testing(). This will pass data from nwsystest to your program if environment variables aren't possible (because your program is executed through valgrind).

Here is a simple template program which demonstrates the basic outline of nwsystest communication:

        #include "nwsystest.h"

        static void

        process_command(const char *cmd) {
                /* ... evaluate command ... */
        }

        int
        main(int argc, char **argv) {

                int     testsock = -1;

                nwst_eval_args(&argc, argv); /* Should always be called */

                if (nwst_is_testing()) {
                        /* Program was invoked by nwsystest */
                        char    *p;

                        /* Connect to test server */
                        testsock = nwst_connect(NULL);
                        /* Indicate readiness to receive commands */

                        nwst_ready(testsock);

                        /* Receive commands */
                        while ((p = nwst_get_command(testsock)) != NULL) {

                                process_command(p);
                                free(p);
                        }
                } else {
                        char    buf[128];

                        /*
                         * Program was invoked normally - this isn't a
                         * test so we get input from the user rather
                         * than nwsystest
                         */
                        while (fgets(buf, sizeof buf, stdin) != NULL) {

                                process_command(buf);
                        }
                }
                return 0;
        }


Participants

A participant is a piece of software that takes part in the test. Commonly it is a program, but it might also be a thread, and a single thread or program could in theory also play multiple participants.

The current primary mode of operation is to make a participant a full program, and to have nwsystest start and terminate that program. Participants should be defined at the beginning of the test description file. The format of the "participant" directive is as follows:

participant foo = ../prog -arg1 -arg2 >foo.log

This will start the executable file "../prog" with the identity foo, pass it the command line arguments -arg1 and -arg2, and redirect its console output (stdout/stderr) to the file foo.log (the latter is really advisable, and future versions of nwsystest may just automatically perform this redirection).

In order to become a registered and ready participant, your program has to call nwst_connect() to join the test server:

int testsock = -1;
if (nwst_is_testing()) {
        testsock = nwst_connect(NULL);

It then has to call

nwst_ready(testsock);

to indicate to nwsystest that it has been properly initialized and is now ready to receive commands. All participants have to indicate readiness before the first command is executed.

If you omit the executable file in the participant declaration, you can have participants join without having nwsystest start them. You can first start the test, and then start the software which takes part in the test. It is probably also possible to let a single participant program create multiple thread participants:

participant foo = ./prog >foo.log
participant foo_bla_thread
participant foo_bla2_thread

In this case, one program is started and receives the identity foo. Two threads created by that program then too connect to nwsystest to take part in the test with their own identities.

(A participant that was not directly started by nwsystest has to declare its identity explicitly:

nwst_connect("my_identity")

... whereas a participant started by nwsystest can simply call

nwst_connect(NULL)

because it receives its identity through an environment variable that is used by the library. NOTE: The latter mode of operation is mostly untested.)

A participant is generally expected to be terminated by nwsystest; Calling

nwst_get_command()

automatically exits your application if nwsystest requested this. You can avoid exiting automatically (e.g. because you may wish to save data files and clean up other things) by calling

nwst_get_command_without_exiting()

The program should perform its cleanup and exit if the string returned by nwst_get_command_without_exiting() is identified as an exit command;

        char    *p = nwst_get_command_without_exiting(testsock);

        if (p != NULL) {
                if (nwst_is_exit_command(p)) {

                        cleanup();
                        exit(EXIT_SUCCESS);
                }
        }

By default your program is not permitted to terminate without first having received a termination request from nwsystest. You can use the "expectexit" test description directive if you wish to do so;

participant prog = ./prog
expectexit prog

Note that the participant still has to emit all expected events for all commands in the test case correctly; It is simply allowed to terminate by itself after all work is done.



Commands



    General

A command is a message that is sent to a participant, and which is commonly expected to trigger one or more events and associated data emissions. For example,

cmd client get foo.mp3 {}

... will send the message "get foo.mp3" to participant "client". The client can read this message by calling

nwst_get_command()

... on the nwsystest connection socket. If your program is a command line application of some sort anyway, it is advisble to simply pass it commands in the same syntax as a user would type them. That way all command parsing code can also be reused and tested.

In the example above, the command is associated with an empty event list ("{}"), meaning the command is not expected to generate any events. If it does generate events, the test fails.

    nwsystest I/O

Receiving commands from nwsystest using nwst_get_command() may be problematic if you wish to do other things while waiting for nwsystest to send new commands - you may instead wish to serve other socket connections for one reason or another.

nwsystest offers the function

nwst_try_get_command()

which reads a command from nwsystest if available, but returns immediately with a null pointer return value if there's nothing to read yet.

Otherwise you can also use select()/poll() to wait for events on multiple file descriptors - as described in any introductory Unix programming text - or even run a thread dedicated to nwsystest communication.

Note that your program should generally be responsive to the nwsystest interface because that's also the channel through which your program is terminated at the end of a test.



Events

    Plain

The test description file can contain event lists which describe certain things happening in your software. For example, an unconditional event list:

{
        server { event hello_world }
        client { event hello_world }
}

... will expect the participants "server" and "client" to generate one hello_world event each. nwsystest does not continue processing commands in the test description file before both of these events have arrived (if a different set of events arrives, the test failed).

Your software can emit an event by calling the nwst_emit_event() function:

nwst_emit_event(testsock, "hello_world");

You do not need to guard this event emission with nwst_is_testing() because it will do this internally - no event emission is attempted if we're not in testing mode.

A "conditional" event list is a set of events that is expected in response to a certain command. For example

cmd client put foo.mp3 {
        server {
                event file_received
        }
        client {
                event file_sent
        }
}

This will first send the command "put foo.mp3" to participant "client" and then wait for the associated event lists to be completed.

    With strings

The last example in the preceding section leaves a lot to be desired. It allows server and client to claim that a file was transferred, but nothing at all is done to verify this claim for correctness.

In order to back up claims with data, nwsystest allows you to transfer events with data - simple strings or entire files.

For example, the client could announce just which file it believes was transferred:

event file_sent = "foo.mp3"

This event can be emitted with

nwst_emit_event_with_string(testsock, "file_sent", "foo.mp3");

(... which is a variadic function and allows you to use a format string with arguments in place of foo.mp3)

You can also compare the received string against an existing file:

event file_sent == dat0_transfer_result.txt

The advantage of this approach is that you can run

nwsystest -reference

... to populate the text file with the initial results. Note that the == operator is only for strings!

Strings may contain the variable "$PWD" in order to insert the current working directory path into the text. This is intended to allow dynamic absolute paths. Your software may wish to handle test data files with fully resolved absolute paths. For example, it may wish to work with - and report - the path

/home/nils/project/tests/dummy.txt

But writing

event file_uploaded = "/home/nils/project/tests/dummy.txt"

would tie that test case to a particular directory. It would fail on OS X, for example, where the home directory is /Users/nils, not /home/nils, such that the file name comparison breaks.

The solution is to write it like this:

event file_uploaded = "$PWD/dummy.txt"

    With files

The server in our preceding example received a file and presumably also produced a copy of that file in the file system, so it would be beneficial to check that file for correctness (i.e. against the original file, or another copy of the original).

A file can be sent either inline or simply by passings its path to the test server.

event file_received = foo.mp3 = reference_foo.mp3

... will expect your program to create a file foo.mp3 which has the same contents as an existing file named reference_foo.mp3. You can create such an event by calling

        /*
         * Last argument means "not inline" - just the relative path
         * "foo.mp3" is passed (nwsystest has to be running in the
         * same current working directory for this to work)
         */
        nwst_emit_event_with_file(testsock, "file_received", "foo.mp3", 0);

If you wish to send the file inline - e.g. because the participant may be running on a different machine or work in directories that are inaccessible to nwsystest - you can do it like this:

        /*
         * Last argument means "inline" - nwsystest will receive the
         * contents of the file "./foo.mp3" and save it to a temp
         * file
         */
        nwst_emit_event_with_file(testsock, "file_received", "foo.mp3", 1);

The latter case can be checked in the test description file as follows:

# Compare unnamed data file received to reference file
event file_received = reference_foo.mp3

(NOTE: These ways to compare things are a bit inconsistent and confusing and may be improved in the future.)

    Ordering

By default, an event list is "strongly" ordered. That means the events are expected to arrive in the order in which they are listed in the event list;

client {
        event event_one # must arrive first
        event event_two # must arrive second
}

The "ordering" directive allows you to specify that you wish a "weak" ordering, i.e. that events may arrive in any order:

ordering weak
client {
        event event_one # may arrive at any time
        event event_two # may arrive at any time
}

You can switch between strong and weak ordering at the top level of the test description file, and between individual participant event lists in an event list block:

participant gnu
participant foo

ordering weak

cmd client bla {
        gnu { ...weakly ordered event list ... }

        ordering strong # switch to strong ordering

        foo { ...strongly ordered event list ... }
}

{ gnu { ...still strongly ordered event list... } }

    Ignoring events

Sometimes it may be desirable to ignore events emitted by your software. Your software may always emit a couple of status events that are only interesting for some test cases.

In order to ignore events, you can use the ignore directive;

ignore event_name

You can put all events that you usually intend to ignore into an include file, e.g. "common.ignores", which you include at the top of every test file that does not need any of those events.

Igoring events is an "all or nothing" thing, and it applies globally - so if the last line in your test description is an ignore directive, that will also affect all preceding commands and events. In the future, there may also be an "unignore" directive to enable you to ignore events for a particular portion of the test file.



Stress testing

It is possible for a test case to work most of the time, but to fail every once in a while. This is particularly true in the presence of multiple participants which end up competing in race conditions.

Therefore, it can be useful to rerun the test suite over and over again until a test case breaks. Running

nwsystest -loop

... will do just that. It will stop as soon as a test breaks, such that you can look at the various log files produced to infer the problem.



valgrind

valgrind is a fantastic debugging tool which is capable of detecting various fatal memory errors in your software. It is full of truth - valgrind reports should generally be taken very seriously (there are a few expections, such as programs compiled with profiling support on some systems, which do not report correct results).

By running

nwsystest -valgrind

you can have nwsystest run all of your participants through valgrind and check for errors. If a fatal memory error (buffer overflow, use of uninitialized variable, etc.) occurred, the test is reported as failed even though it may have fulfilled all of the other event criteria.



High-level programming considerations

A lot of things are completely under your control and nwsystest does not tell you how to realize them. This section is intended to give you an overview to some of the high level design considerations involved in setting up the communication between your software and nwsystest.

    Test coverage

Tests will be as fine-grained and useful as you make them. A meaningless test might just emit a single "all_done" event when a tested feature has finished execution. A better test would at least include a few data results, and distinguish between event names (e.g. featureA and featureB shouldn't both emit an "all_done" done event because that would make the result ambiguous; It would be possible that one feature incorrectly ended up executing a code path from the other one).

In general you should always supply some data to make things more verifiable. For example, if nwsystest sends a command stating "please change your current working directory to /home/backup", then your software shouldn't just respond with

"directory changed"

but rather

"directory changed to /home/backup"

i.e. an event message with a string (your software should also strive to tell the truth; It should only indicate that the directory was changed if the chdir() call succeeded). This will help verify that the command was correctly transmitted and parsed by the receiver, and that the directory change was most likely triggered by that command rather than some other software component.

There are no guidelines on how to structure the relationship between commands and events - it is up to you.

    Test system communication

You can choose who talks to nwsystest and how. When developing a client-server application, for example, it might be sensible to only ever send commands from nwsystest to the client - the client will then propagate all requests to the server, as it normally would when operated by a user. However, even though the execution of a feature may be initiated by the client, it is still perfectly reasonable to have the server report results directly to nwsystest.



Future ideas

nwsystest is a young and as of yet immature project. Some shortcuts were taken to get it up and working quickly; The test description language is inconsistent in some ways (event data comparison operator, scope of "ignore" directives), the test driver is not hardened against runaway systems which can bring a test to a halt, and test results and program output logs are not saved permanently but always overwritten with the results from the next test.

These imperfections will be addressed. In addition, there are various new features under consideration:



Appendix A: Command line arguments

-verbose

Will output specific information about what is happening in the test case (participants being started, commands sent, events received, etc). A copy of this data for the last test run - even if -verbose wasn't actually used - is saved in the nwsystest.log file in the test directory.

-reference

Causes data file results produced by the test run to be assumed as "reference" files, i.e. for every comparison

event file_received = new_file = reference_file

... reference_file is not expected to exist and have the same contents as the newly created file new_file, but it becomes a copy of new_file. Note that you should check all new reference files carefully for correctness to ensure that you don't end up testing for the wrong results.

-loop

Keeps rerunning the test suite until a test case fails, in which case the test run is stopped immediately.

-valgrind

Runs all participants of the test through valgrind. Memory errors cause the test to fail. valgrind output is logged to participant.vlog



Appendix B: Keywords

participant - Defines a participant
include - Include a text file in the test description
cmd - Send a command to a participant
run - Execute a system command
event - Expect an event
ignore - Ignore an event type
ordering - Specify an event receipt ordering
weak - Weakly ordered
strong - Strongly ordered
expectexit - Allow a participant to exit by itself


Appendix C: Library reference

int nwst_connect(const char *identity);

Connect to test server. Pass NULL to pick our identity from the environment variable passed by nwsystest (if started by nwsystest), or a custom name

Returns a server socket handle for all subsequent operations on success, -1 on failure

void nwst_ready(int sock);

Announce readiness to test server. The server will not begin sending commands before all participants are ready. This is intended to allow for some initialization time

char *nwst_get_command(int sock);

Read a command (as defined in the test description file by using the "cmd" directive) from the test server, waits if no command is available.

Returns the command on success, NULL on failure (server exited). Note that a termination request from the server will not be returned, but automatically closes your program. Use nwst_get_command_without_exiting() to exert control about what to do when the server requests termination

Command must be freed with free()

char *nwst_try_get_command(int sock);

Attempts to read a command from the test server, but doesn't wait if no data is available.

Returns the command on success, NULL on failure (no data, or server exited)

Command must be freed with free()

char *nwst_get_command_without_exiting(int sock);

Reads a command from the server. This behaves the same as nwst_get_command(), except that a server termination request does not automatically cause it to exit the program. You can use

nwst_is_exit_command()

on the string returned to determine whether nwsystest requested termination.

int nwst_is_exit_command(const char *);

States whether a server command constitutes an exit request

char *nwst_get_identity(int sock);

Obtains identity name associated with socket, intended to distinguish between multiple identities in the same process (multiple nwst_connect() calls)

void nwst_emit_event(int sock, const char *name);

Sends a named event to the server, for receipt with

event name
void nwst_emit_event_error(int sock, const char *name);

Sends a named error even to the server, for receipt with

event errorevent = "name"
void nwst_emit_event_with_string(int sock, const char *name, const char *data, ...);

Sends a named event with a string data payload to the server, for receipt with

event name = "string" # "string" is expected data

or

event name = file.txt # file.txt is a file containing expected data
void nwst_emit_event_with_data(int sock, const char *name, const char *data, size_t data_size);

Sends a named event with a data payload to the server XXX Untested/unsupported/undocumented

void nwst_emit_event_with_file(int sock, const char *name, const char *path, int send_inline);

Sends a named event with a file data payload If send_inline is 1, the file designated by "path" is opened, read, transferred and saved in an unnamed temp file. In that case, the file can be checked with

event name = reference_file

If send_inline is 0, the file path "path" is transferred and can be checked with

# Expect file "path" to be produced, and to have the same
# contents as file "reference_file"
event name = path = reference_file
int nwst_is_testing(void);

Returns 1 if a test is running (the program was started by nwsystest) and 0 if not.

void nwst_set_testing(int val);

Set result reported by subsequent nwst_is_testing() calls. Useful to enable testing mode in program instaces that weren't started by nwsystest, but join the test regardless (e.g. instructed by command line arguments)

void nwst_eval_args(int *, char **);

Evaluates directions from nwsystest passed through command line arguments. Should be called at the begining of your main() function

nwsystest © Nils Weller 2011 SourceForge Logo