A Cool CS Co : Team : Phil Hollenback : white paper

HMS Testing with ETET

Phil Hollenback
A Cool Computer Science Company

April 30, 2001

Abstract:

A discussion of the use of the ETET test harness to remotely test functionality of the HMS hardware platform.

Contents

• Contents
• Overview
• Terminology
• Methodology
• Test Coverage
  • Audio
  • Data CD Duplication
  • DHCP Client
  • Display
  • Memory
  • PPP
  • Samba
  • Network Bandwidth
  • USB
  • IrDA
  • Network Security
  • Reiserfs Crash Recovery
  • Rio 500
• Test Configuration
• Test Operation
• Theory Of Operation
• ETET Configuration Variables
• Test Scenarios
• Configuring ssh RSA authentication

Overview

A set of tests are desired for the HMS system under development by HP. These tests must exercise a defined set of system capabilities. The tests must be integrated into a test harness which provides for consistent test operation and journalling. The tests must be controlled via a remote computer system.

Terminology

The controlling system is referred to in this document as the host. The system being controlled is referred to as the target. Following this logic, the HMS is the target.

Monospaced sections are commands to be executed. or configuration file entries. Commands starting with # are to be executed as root, and command starting with $ can be executed as any user. The system to execute the command on should be clear in context.

Methodology

The HMS is a relatively full-featured PC running Linux. It has network access via an ethernet connection. We can take advantage of the power and network connectivity of the HMS by running the actual test software on it and gathering the results on the host machine. This provides other advantages, for example it is possible to power-cycle the HMS and see if it restarts correctly from the safety of the host system. This would be very difficult to implement if the tests were run solely on the HMS.

The test harness selected is ETET, the Extended Test Environment Toolkit. ETET provides the test execution and journalling mechanisms. Extensions to ETET written by ACCSC provide the remote execution capability over Secure Shell.

The general test strategy is thus to run the test controller (ETET) on the host and control the target (HMS) via an ssh connection.

Test Coverage

Tests have been developed to cover the following areas:

• audio playback and recording
• data CD duplication and verification
• DHCP client operation
• X display with automatic image comparison
• memory read/write with system load
• data transfer over ppp over a modem connection
• samba server functionality
• network bandwidth usage
• USB operation
• network security (port scan)
• reiserfs system crash recovery
• rio500 mp3 player control

All tests use a set of common ETET configuration variables as listed in 8. Variables specific to each test are noted in each test section. See the section on 5 for details on how to configure the tests.

Audio

Various tests of audio subsystem. Tests are manual (user intervention is required). Tests include the following:

• Sample audio files of differing sample rates and formats are played and user is asked if audio was heard. Yes answer is test pass.
• 30 seconds of audio is played from audio CD in drive, user is asked if audio was heard.
• User records input from microphone, then recorded sample is played back and user is asked if audio was heard.

Assumptions:

• An audio CD is present in the drive on the target
• A microphone is connected.
• Audio is configured under Linux.

ETET configuration variables:

• SBLIVE_AUDIO_HACK - if set to True, mixer is switched as necessary to output CD audio. This is a workaround for peculiarities in the Sound Blaster Live! sound driver. Set this variable to 'False' for all other sound cards.
• AUDIO_MONOAU8 - name of a mono 8 bit file in .au format. All audio test files must be in the directory /usr/test/data on the target.
• AUDIO_MONOWAV8 - name of mono 8 bit file in .wav format.
• AUDIO_MONOWAV16 - name of mono 16 bit file in .wav format.
• AUDIO_STEREOWAV16 - name of stereo 16 bit file in .wav format.

The audio ETET test scenario is named audio , so an example execution would be (as root):

# tcc -e hms audio

Data CD Duplication

Data CD is duplicated and MD5 checksums are used to verify the new CD matches the original. This is a manual test (user must swap CDs when prompted).

Assumptions:

• CD-ROM writer is configured.

ETET configuration variables:

• none at this time

This ETET test scenario is named cd.

DHCP Client

DHCP client functionality is tested. If DHCP client (pump) is not running, it is started. These are automated tests. Tests include the following:

• pump -status command is tested for success (indicating system is operating as a DHCP client).
• pump -renew command is tested for success (indicating system can query DHCP server for updated DHCP configuration settings).

Assumptions:

• DHCP server is configured on local network to respond to target system.

ETET configuration variables:

• none at this time

This ETET test scenario is named dhcp.

Display

Ability of HMS to display images is tested. Image is displayed and compared to reference image. These are automated tests. Tests include:

• Positive display test. A new X server is started on the HMS and an image is displayed. Image is then dumped to a file and compared to reference image. Test passes if dumped image matches reference image.
• Negative display test. Same mechanism as positive display test, except dumped display image is compared to a different reference image which should not match. Test passes if dumped image does not match reference image.

Assumptions:

• X is configured on HMS

ETET configuration variables:

• none at this time

This ETET test scenario is named display.

Memory

HMS memory is tested. A block of memory is read and written continuously for a specified time while a specified system load is generated. This is an automated test.

Assumptions:

• none at this time

ETET configuration variables:

• MEMTEST_LOAD - system load to generate while memory test runs. A load of 2 would create two memtest process, for example. A load of 0 would create no additional memtest processes.
• MEMTEST_SIZE - memory size to test, in kilobytes. This amount of memory will be allocated and tested repeatedly during the test run.
• MEMTEST_TIME - time to run memory test, in seconds.

This ETET test scenario is named memory.

Example: to test 4MB of memory for 10 minutes with a system load of 1, set the following tetexec.cfg variables:

MEMTEST_LOAD=1

MEMTEST_SIZE=4096

MEMTEST_TIME=6000

PPP

Data transfer over a dialup ppp connection is tested. This is an automated test. Connects to PPP server and transfers a file via tftpbandwidth. PPP server can be the host, but this is not required. PPP server also acts as tftp server to test file transfer.

This test is difficult to set up because ppp client/server configuration is difficult in general. Some hints:

• The usual way to configure ppp on the target and server is to use PAP authentication. To do this, edit the file /etc/ppp/pap-secrets on both systems. On the target, it should look like this:

  #client  server        secret   IP address

  hms      <servername>  <secret> 10.1.1.1

  On the server, it should be similar to:

  #client  server        secret   IP address

  hms      <servername>  <secret> *

• You may need to use the local-address ppp option to force the target to supply the correct IP address for authentication.
• The server can use pppd as the user shell. If it isn't configured this way, the test will attempt to start pppd on the server after gaining shell login.
• If you use a USB modem, the modem device will be /dev/ttyACMx. This device must already exist. It can be created via

  mknod /dev/ttyACM0 c 166 0

Assumptions:

• Modem is configured on HMS. This may include ensuring that appropriate driver (such as USB ACM driver) is already loaded.
• Dialup PPP server is configured
• PPP server is also TFTP server and configured to server vmlinux.srec test file.

ETET configuration variables:

• PPP_DEVICE - modem device such as /dev/modem or /dev/ttyS2.
• PPP_PHONE_NUMBER - number to dial
• PPP_USERNAME - username to connect to ppp server with
• PPP_PASSWORD - password to use on ppp server

This ETET test scenario is named ppp.

Samba

Target system is tested as a samba server. This is an automated test. smbclient is run on target connecting to localhost. Zero return status from smbclient program indicates samba server on hms is operational.

Assumptions:

• samba server is running on hms

ETET configuration variables:

• none at this time

This ETET test scenario is named 'samba'.

Network Bandwidth

Ability to transfer data over ethernet connection is tested. This is an automated test. Test program is tftpbandwidth. Target system connects to tftp server and downloads a file.

Assumptions:

• tftp server is configured to allow connections from target system (typically this server is the host system, although this is not required).
• tftp server is reachable via ethernet.

ETET configuration variables:

• TFTP_SERVER - IP address or DNS name of tftp server to connect to. This system must be configured to allow hms to download a file named vmlinux.srec.
• TFTP_LOAD - system load to generate while tftp transfer is running. 0 means no additional load, 1 means 1 more running process, etc.

This ETET test scenario is named bandwidth.

USB

Tests hms system ability to deal with USB devices. This is a manual test. Tests include:

• User is asked to plug in a USB device. Test succeeds if a new device is reported by OS.
• User is asked to unplug a USB device. Test succeeds if OS reports a device has been removed.

Any USB device can be used.

Assumptions:

• USB is configured and operational on hms.

ETET configuration variables:

• none at this time

This ETET test scenario is named usb.

IrDA

Tests transferring files over IrDA. This is an automated test. Files are transferred over an IrOBEX connection and checksummed. Tests include:

• Transfer file from target to host system and checksum. Test passes if file checksums match.
• Transfer file from host system to target. Test passes if file checksums match.

Assumptions:

• IrDA is configured and operational on target and host systems.
• IR transceivers on host and target are pointed at each other and in range (within a few feet).
• File copy over IrOBEX software is installed on both systems. This includes libobex and ircp.

ETET configuration variables:

• IRDA_FILESIZE - size of file in kilobytes to generate and send over IR link. File is generated on the fly for each test by reading random data from /dev/urandom.

This ETET test scenario in named irda.

Network Security

Tests the network security of the hms by doing a port scan with nmap. This is an automated test. Note that this is not a remote test, i.e. nothing gets run on the hms.

Assumptions:

• nmap is installed on the host.

ETET configuration variables:

• NMAP_ALLOWED_PORTS - list of acceptable open ports. This list must be separated with the bar character '|', for example ``21|22|512''.

This ETET test scenario is named security.

Reiserfs Crash Recovery

Tests the ability of a reiserfs filesystem to handle a system power loss. This is an automated test. Files are opened for reading and writing on a reiserfs filesystem, reads and writes are started, and the system is power-cycled. ETET on host then waits a predetermined amount of time before attempting to contact target again. If a new ssh connection can then be made to the target, it is assumed the target reset correctly without data loss.

WARNING: this test is potentially destructive, as it power-cycles the target system. This could cause data corruption and damage to the OS. In practice this test usually does not damage the system.

Assumptions:

• Target system supports reiserfs (kernel is properly configured)
• A filesystem on the target is configured as type reiserfs and mounted.

ETET configuration variables:

• REISER_FS - name of reiserfs filesystem on target to use.

This ETET test scenario is named reiserfs.

Note that this test is not included in the 'all' scenario due to it's potential for system damage. It is included in the separate 'dangerous' scenario.

Rio 500

Tests connectivity to a rio500 handheld mp3 player. These are not automated tests, as the rio automatically powers down after a period of inactivity. Thus the rio must be manually checked for power on before running the tests. Tests include:

• Collect statistics from the device
• Format the memory of the device
• Upload a song
• Delete a song

Assumptions:

• USB is configured on target.
• rio 500 is connected and powered on.
• rio 500 kernel driver is installed (test script attempts to modprobe it if not currently loaded).
• rio 500 user utilities are installed.

Note also that these tests are partially ordered. For example, running the delete test immediately after the format test will fail because there is nothing on the device to delete. Run the add test first.

ETET configuration variables:

• RIO_FILE - name of an mp3 file to upload to rio 500. This file must exist in /usr/test/data on the target.

This ETET test scenario is named rio.

Test Configuration

1. Install ETET on host system. Using the CoolCS-supplied RPM package:

   # rpm -ivh tet-3.3g-1.i386.rpm

2. Install tests on target system. Using the CoolCS-supplied RPM package:

   # rpm -ivh hms_tests_target-1.0-1.i386.rpm

3. Install tests on host. Using the CoolCS-supplied RPM package:

   # rpm -ivh hms_tests_host-1.0-1.i386.rpm

4. Configure target system. Examples:
   1. Start USB.
   2. Set up a Reiserfs filesystem.
   3. Configure the sound card.
5. Configure external servers as needed for ppp, IrDA, etc. In general these servers can be the host or other systems, as preferred. If you are running tests on multiple host/target combos in parallel, it may be more convenient to configure some central server for things like PPP.
6. Set the TET_ROOT environment variable on the host system to point to top of the ETET installation. If you are using the CoolCS-supplied ETET rpm, this is:

   # export TET_ROOT=/usr/local

7. Set the TET_SUITE_ROOT environment variable on the host system to point to the top of the actual test suite. For example:

   # export TET_SUITE_ROOT=/usr/coolcs/etet

The convention is that remote tests start with 'r', for example rppptest is the test which runs on the remote system to control the ppp test program (ppptest) on the hms.

All tests which run on the target live in the directory /usr/test/bin on the target. Test data files live in /usr/test/data on the target.

Test Operation

All tests must be run as root, and connection without password prompts (i.e. ssh RSA authentication, see 10) must be configured before tests are run.

1. Examine $TET_SUITE_ROOT/hms/tet_scen to determine test scenario you wish to run. See 8 and the individual scenarios under 4for details.
2. Invoke the ETET test controller tcc to run the tests you desire, for example:

   # tcc -e hms display

   will execute all display tests.

3. Examine the ETET log file in $TET_SUITE_ROOT/hms/results/XXXXe/journal where XXXX is a number incremented on every test run, starting with 0001. Passed test scenarios are denoted with PASS, failed test scenarios are denoted with FAIL. NORESULT or UNRESOLVED indicates something went wrong and the test did not run to completion.

Alternately, you can run individual tests on the target directly. See 7 for detains on how this works. All of the tests in /usr/test/bin on the target can be executed independently if you supply the proper command-line arguments. For example, you run the positive display test on the target directly by invoking:

# /usr/test/bin/xtest-pos

One advantage of running the tests right on the target is you can see all the generated output and error messages. Thus this is suitable for debugging test execution. Be advised that you loose test logging doing this as the host-based ETET component isn't invoked.

Finally, you can run individual tests via ETET on the host system without invoking tcc. For example, you can run the irda tests by invoking rirdatest directly on the host system. In this case, all irda tests would be run. You can specify tests to run with a numerical argument:

# $TET_SUITE_ROOT/hms/ts/rirdatest 2

which run the second irda test. Note that this also bypasses the standard ETET logging facilities. After running this example, you will find the test results in ETET log format in the file tet_xres in the current directory.

Theory Of Operation

A few notes on how the tests actually work, using the memory tests as an example. Assume you have just invoked the tests on the host with tcc -e hms memory:

1. tcc checks the scenario file for the scenario you have indicated. Since $TET_SUITE_ROOT is set to /usr/coolcs/etet and you select the hms suite, this is the file $TET_SUITE_ROOT/hms/tet_scen. This tells ETET which test to actually invoke. In this case, the test program is rmemtest in $TET_SUITE_ROOT/hms/ts/.
2. tcc opens the configuration variable file and reads in the contents. This would be $TET_SUITE_ROOT/hms/tetexec.cfg.
3. tcc starts logging in $TET_SUITE_ROOT/hms/results/00001e/journal.
4. tcc invokes the test program rmemtest. This is all on the host.
5. rmemtest does it's initialization and runs each test case in turn (there is only one test case currently in rmemtest).
6. Here's where things get interesting: rmemtest calls the actual test case. This is a function inside rmemtest called testmem(). For the purposes of understanding the tests, it's probably best to think of testmem() as a separate program called by rmemtest.
7. testmem() uses a here-document to feed a script to expect. expect is a tcl-based tool that can automate any command-line program. In this case, expect starts an ssh session to the target system.
8. expect works through the test. First it feeds the target system the name of a program to run: memtest, along with arguments (the contents of MEMTEST_LOAD, MEMTEST_SIZE, and MEMTEST_TIME from tetexec.cfg).
9. expect normally then watches the connection and waits for the test program on the target to complete. memtest is a special case, as the test program on the target will run indefinitely. So, expect sleeps for MEMTEST_TIME seconds and then sends a C to kill the memtest on the target.
10. expect checks for a shell prompt on the target. This indicates that the test exited cleanly. For tests other than memtest, expect would check for a keyword of 'success' or 'failure'.
11. If everything went ok, expect exits with an exit code of 0. If not, expect exits with 1.
12. testmem() catches that and passes the code back to rmemtest.
13. rmemtest checks that testmem() exited successfully (exit code 0). If not, a flag of FAIL is set.
14. rmemtest checks that testmem() didn't print anything to stderr. If it did, rmemtest again sets the FAIL flag and logs stderr to the journal.
15. rmemtest terminates. tcc catches the result of FAIL or PASS and logs it to the journal.
16. test execution terminates.

ETET Configuration Variables

ETET configuration variables are stored in the file $TET_SUITE_ROOT/hms/tetexec.cfg.

The following configuration variables should be left as-is for proper test execution:

TET_OUTPUT_CAPTURE=False 

TET_EXTENDED=True 

TET_API_COMPLIANT=True 

EXPECT_TIMEOUT=60 

RSH_CMD is the command used to connect to the hms. ssh is the only tested mechanism, however rsh would probably work with a little tweaking too.

RSH_CMD=ssh -q 

Address of the hms under test. Either IP address or DNS name.

HMS_ADDRESS=quark 

Other tetexec.cfg variables are defined in individual test descriptions.

Test Scenarios

The following preconfigured etet test scenarios are available. A scenario is standard unit of action for ETET - you run a scenario, and that in turn invokes tests which can contain multiple test cases. Scenarios are controlled by the file $TET_SUITE_ROOT/hms/tet_scen.

1. all - all tests (except dangerous tests)
2. auto - all automated tests. These are:
   • rtftpbandwidth
   • rdisplay
   • rirdatest
   • rppptest
   • rdhcptest
   • securitytest
3. dangerous - tests which may be hazardous to the system:
   • rriesertest is currently the only dangerous test.
4. manual - all tests which require user intervention:
   • raudiotest
   • rusbtest
   • rcddup
   • rriotest

Finally each test has it's own individual scenario, as follows:

1. bandwidth - network bandwidth tests (rtftpbandwidth, 1 test)
2. memory - memory tests (rmemtest, 1 test)
3. display - display tests (rdisplay, 2 test cases)
4. ppp - ppp tests (rppptest, 1 test case)
5. irda - IrDA tests (rirdatest, 2 test cases)
6. cd - CD-ROM tests (rcddup, 1 test case)
7. audio - audio tests (raudiotest, 3 test cases)
8. dhcp - DHCP client tests (rdhcptest, 2 test cases)
9. samba - Samba server tests (rsambatest, 1 test case)
10. rio - rio500 tests (rriotest, 4 test cases)
11. security - security tests (securitytest, 1 test case)
12. reiserfs - reiserfs tests (rreisertest, 1 test case)

Configuring ssh RSA authentication

The most secure and reliable way to run the tests is over ssh using RSA authentication. This can be configured as follows:

• Run ssh-keygen on your host system to create a RSA key pair. Use the default location of /.ssh/identity.pub. Don't supply a passphrase (just press <enter> when prompted).
• Copy your public key to the server you wish to use. This is the file identity.pub. The identity file is your private key and should never be copied anywhere.
• Append your public key to the file /.ssh/authorized_keys on the machine you wish to connect to (target). You may have to create the .ssh directory on the server (and give it the right permissions):

  # mkdir /.ssh

  # chmod 700 /.ssh 

  # cat identity.pub >> /.ssh/authorized_keys 

• Test the connection. From the host, attempt to log in to the target with ssh -v <target>. The -v turns on verbose mode, which will tell you what authentication mechanism ssh is trying to use. If everything works fine, you will be logged right in. If not, examine the debug messages.

Other considerations:

• The tests must be run as root.
• Check the sshd config file /etc/ssh/sshd_config on the target to make sure PermitRootLogon is set to 'Yes'

About this document ...

HMS Testing with ETET

This document was generated using the LaTeX2HTML translator Version 99.2beta5 (1.38)

Copyright © 1993, 1994, 1995, 1996, Nikos Drakos, Computer Based Learning Unit, University of Leeds.
Copyright © 1997, 1998, 1999, Ross Moore, Mathematics Department, Macquarie University, Sydney.

The command line arguments were:
latex2html -nonavigation -split 0 test-guide.tex

The translation was initiated by Philip J. Hollenback on 2001-07-24