
$Id: tests.txt 5744 2012-07-17 22:54:13Z fields_t $

This file explains the basics of executing multi-host tests using the
framework supplied here.

Tests are intended to be configured entirely using configuration files
and executed without need to establish an external environment.  This is
done by building and packaging the test executables and scripts so that
they can be moved to each host participating in the test and executed
directly from the packaged files.

The steps involved in setting up and executing tests using this framework
include:

1) Define and configure your scenario;  this involves selection of
   existing scenario configuration files or creation of new ones.  See
   the available scenario files located in the 'tests' directory.  The
   'doc/scenarios' file describes the available test scenarios.

2) Select or create a service configuration file.  This will mostly
   involve definition of transport configurations and any service wide
   configuration settings.  Use of the 'etc/transport.ini' file will
   provide 10 default transport instances for each of 5 transport types
   and select a round-robin scheduler for use when TRANSPORT_PRIORITY
   values other than the default are used.

3) Package the test framework.  Make sure that if you have created new
   scenario or service configuration files that they are included in the
   packaged directories.

   This is facilitated by the 'tools/mkpkg' script.  Use it as follows:

   > tools/mkpkg <name>

   Where <name> names a directory where the files to be packaged are
   copied to.  Any existing directory of this name will be removed.  The
   default <name> is 'pkg'.

   All libraries, executables, scripts and configuration files are copied
   into the package directory and library version links created.

   It is the responsibility of the test developer to copy in any custom
   scenario or service configuration files into the package directories.

   Once all files for the test have been copied into the package
   directory, the directory can be transfered to each host participating
   in the test.  This can be done by recursively copying the directories
   or archiving the directories and transferring the archive.  When the
   directories are visible from each host participating in the test via
   network file access, such as nfs, then a separate directory should be
   created for each host to allow for identically named output files to
   be generated.  If name collisions are guaranteed to not occur it is
   possible to execute from the same directories.

3) Determine which hosts will be executing which test processes.
   Transfer and unpack any packaged archives or locate the network
   accessible directories on each of these hosts.

   Identify which test processes will run on each host during test
   execution.

4) Ensure that a fully qualified domain name is available on each host
   and resolves correctly.  An entry in the /etc/hosts file may be
   required where network name resolution is not adequately setup.

5) Acquire privileged access on hosts which have non-default values for
   TRANSPORT_PRIORITY specified.  This is required in order to setup the
   appropriate scheduler and establish individual thread priorities.

6) Execute an OpenDDS repository for use by the test processes.  A host
   and port to locate the repository should be specified.  Do this using
   the run_test script as:

   > bin/run_test -v -S -ORBListenEndpoints iiop://repohost.domain:<port>

7) Execute test processes on each host.  Only a single invocation of the
   run_test script is needed on each host as the script will start a
   process for each specified scenario configuration file.

   An example of starting the execution of 2 processes on each of 3
   hosts for scenario2 is:

   Host1> bin/run_test -v -P \
            -DCPSInfoRepo corbaloc:iiop:repohost.domain:<port>/DCPSInfoRepo \
            -s tests/scenarios/s2-p1.ini,tests/scenarios/s2-p5.ini

   Host2> bin/run_test -v -P \
            -DCPSInfoRepo corbaloc:iiop:repohost.domain:<port>/DCPSInfoRepo \
            -s tests/scenarios/s2-p2.ini,tests/scenarios/s2-p4.ini

   Host2> bin/run_test -v -P \
            -DCPSInfoRepo corbaloc:iiop:repohost.domain:<port>/DCPSInfoRepo \
            -s tests/scenarios/s2-p3.ini,tests/scenarios/s2-p6.ini

Some examples of executing tests on multiple hosts is included here.  Use
this as a guide as your local network configuratio and directory
structures are likely to be unique to your environment.

In all of the examples below, it is best to start the testprocess
execution (invocations of bin/run_tests ... -P ...) in the order shown
and within a few seconds of each other.

If a duration is not specified for the test (-t <seconds>) then use '>
pkill -INT testprocess' to stop the testprocess executions gracefully so
that they will report collected data at test termination.  This should
be done on the hosts in the same order that the test scripts were started.

A simple example to execute scenario1 is:

   host1> make
   host1> tools/mkpkg testing
   host1> tar jcf scenario1.tar.bz2 testing
   host1> scp scenario1.tar.bz2 host2:
   host1> cd testing
   host1> bin/run_test -v -S -h iiop://host1.localdomain:2809

   ...then in another window on host1...

   host1> cd testing
   host1> sudo bin/run_test -v -P -t 60 -h host1.localdomain \
          -i etc/transport.ini -s tests/scenarios/s1-p1.ini

   ...and on host2...

   host2> tar xf scenario1.tar.bz2
   host2> cd testing
   host2> sudo bin/run_test -v -P -h host1.localdomain \
          -i etc/transport.ini -s tests/scenarios/s1-p2.ini

A more complex example executing scenario3 on 3 hosts is:

   host1> make
   host1> tools/mkpkg testing
   host1> tar jcf scenario3.tar.bz2 testing
   host1> scp scenario3.tar.bz2 host2:
   host1> scp scenario3.tar.bz2 host3:
   host1> cd testing
   host1> bin/run_test -v -S -h iiop://host1.localdomain:2809

   ...then in another window on host1...

   host1> cd testing
   host1> sudo bin/run_test -v -P -t 60 -h host1.localdomain \
          -i etc/transport.ini -s scenarios/s3-p1.ini

   ...and on host2...

   host2> tar xf scenario3.tar.bz2
   host2> cd testing
   host2> sudo bin/run_test -v -P -h host1.localdomain \
          -i etc/transport.ini -s scenarios/s3-p2.ini

   ...and on host3...

   host3> tar xf scenario3.tar.bz2
   host3> cd testing
   host3> sudo bin/run_test -v -P -h host1.localdomain \
          -i etc/transport.ini -s scenarios/s3-p3.ini

An example of executing a custom defined scenario using files created in
tests/myscenario-p*.ini and etc/myconfig.ini is:

   host1> make
   host1> tools/mkpkg testing
   host1> cp tests/mysecenario-p*.ini testing/tests
   host1> cp etc/myconfig.ini testing/etc
   host1> tar jcf custom.tar.bz2 testing
   host1> scp custom.tar.bz2 host2:
   host1> scp custom.tar.bz2 host3:
   host1> cd testing
   host1> bin/run_test -v -S -h iiop://host1.localdomain:2525

   ...then in another window on host1...

   host1> cd testing
   host1> sudo bin/run_test -v -P -t 60 -h host1.localdomain:2525 \
          -i etc/myconfig.ini -s myscenario-p1.ini

   host2> tar xf custom.tar.bz2
   host2> cd testing
   host2> sudo bin/run_test -v -P -h host1.localdomain:2525 \
          -i etc/myconfig.ini -s myscenario-p2.ini

   host3> tar xf custom.tar.bz2
   host3> cd testing
   host3> sudo bin/run_test -v -P -h host1.localdomain:2525 \
          -i etc/myconfig.ini -s myscenario-p3.ini

Currently it is advisable to make sure that each process is indeed
terminated and kill the repository process and restart it prior to each
execution run.  This ensures that no problems are caused due to problems
that might persist between executions.  This can be done as follows:

  > pkill -9 testprocess
  > pkill -9 DCPSInfoRepo

And check by:

  > pgrep testprocess
  > pgrep DCPSInfoRepo

In all cases the tests are configured initially to gather data from each
subscription in the test.  The collected data will only be accurate for
the measurements made on the same host.  That means only the full path
traversal latency measured in the same process as the data was sent will
be correct.  All other data, including per hop data, is dependent on the
clock synchronization between hosts.  Using the ntp clock synchronization
mechanism, the jitter of the synchronization is on the order of the
measurements.  This means that these measurements are likely not useful.

