ABRT integration test suite documentation

Author: Richard Marko <rmarko@redhat.com>
Description:Since 2011, ABRT is using BeakerLib library based test suite for integration testing. This document describes the usage and the architecture of the test suite.

Table of Contents

About

At the time of writing this document, we have 32 tests which tests ABRT, libreport and btparser. These tests are run nightly for each of the currently supported releases of Fedora and Red Hat Enterprise Linux. You can find the results on our public mirror.

Running the test suite

Caution!

Don't run the test suite on your machine directly, use virtual machine or machine dedicated for the testing.

In your virtual machine or dedicated machine:
  • clone the ABRT repository: git clone git://git.fedorahosted.org/abrt.git
  • go to abrt/tests/runtest/
  • run ./run

This will run all the tests in order specified in ./aux/test_order. When the testing is done, the output can be found in /tmp/abrt-testsuite/.

To override the destination directory or the other options, you can edit ./aux/config.sh.

Running single test

To run only one of the tests, you have to run it via ./aux/runner.sh. This script prepares your system by starting abrtd and setting correct /proc/sys/kernel/core_pattern.

For example, to run dbus-api test, you have to pass the whole path to runtest.sh to runner.sh:

./aux/runner.sh dbus-api/runtest.sh

If you don't need to start/stop abrtd or reset core_pattern for each run (if it's already running in your system) you can also run the runtest.sh directly.

Running only a subset of the tests

To run only part of the tests, edit ./aux/test_order file and comment out the tests you don't want to run.

For example, when using the test suite to test already installed packages, you don't need to build and install git versions. In this case, you can comment out abrt-nightly-build test and all the *-make-check tests.

Writing new test

To create a new test you have to:
  • create a directory reflecting its name
  • create two files in that directory:
    • executable runtest.sh file
    • PURPOSE file
  • add the test to ./aux/test_order

The tests are written in bash using BeakerLib library. BeakerLib Quick Reference [PDF] is what you need.

It's always good to copy one of the existing tests as a base for your test. Good candidates are:
  • run-abrtd which basically serves as an example,
  • ccpp-plugin if you need to test crash handling (if you require crash directory),
  • dbus-api if you need to test dbus functionality.

Complex tests

Please provide an explanation for complex tests. Either explain certain sections of the test in comments in runtest.sh or create README file inside the directory of the test.

Output directory structure

To avoid painful digging in several distinct files logging is now divided into several logical levels:

$OUTPUT_ROOT variable is defined in aux/config.sh which represents the root for all logs.

For each stage of the test suite run there is corresponding directory in root directory. It contains stage.log file which contains output of script governing the stage.

/tmp/abrt-testsuite
├── format/
├── post/
├── pre/
├── report/
├── results
└── test/

Stages:

  • PRE

    install required packages, collect log files, collect system information

  • TEST

    run all the tests

  • FORMAT

    format the results for REPORT stage

  • REPORT

    report the results

  • POST

    collect logs, cleanup

For TEST stage there is an additional subdirectory for each test case:

/tmp/abrt-testsuite/test/
├── abrt-make-check
├── abrt-nightly-build
├── abrt-should-return-rating-0-on-fail
├── blacklisted-package
...

Each directory contains several files:

/tmp/abrt-testsuite/test/systemd-init/
├── dmesg
├── avc
├── fail.log
├── full.log
├── messages
└── protocol.log

Only full.log is mandatory. It contains stdout and stderr of the test run. protocol.log only contains the protocol generated by BeakerLib. If the test fails with FATAL error, protocol.log is not generated. In case of other failures, these are extracted to fail.log along with line numbers pointing to lines in full.log.

dmesg, messages and avc each contains log file messages written during the test run.

Helper files

Several helper files and scripts are stored in abrt/tests/helpers directory. These includes kickstarts for Fedora.

Using virt-install to install virtual machine

First you need to prepare your kickstart file. Use one of the available kickstarts and run:

ksflatten fedora_16.kickstart.cfg > custom.ks

ksflatten utility is provided by pykickstart package.

Afterwards yo can use following virt-install command to install your virtual machine

virt-install --name "abrt-testing" --ram "1222" \
  --connect qemu:///system \
  --location "http://download.fedoraproject.org/pub/fedora/linux/releases/16/Fedora/x86_64/os/" \
  --disk path=/var/lib/libvirt/images/abrt-testing.img,size=4,sparse=true \
  --accelerate \
  --initrd-inject=./custom.ks \
  --extra-args "ks=file:/custom.ks" \
  --graphics type=vnc \
  --noautoconsole

For this to work, you need qemu:///system access and virt-install utility, provided by python-virtinst package. To allow non-root users access to qemu:///system you need to create policykit file with following contents:

# cat /etc/polkit-1/localauthority/50-local.d/org.libvirt.unix.manage.pkla
[Allow user to manage virtual machines]
Identity=unix-user:your_user_name
Action=org.libvirt.unix.manage
ResultAny=yes
ResultInactive=yes
ResultActive=yes

and make sure you use replace your_user_name with the username you use. Alternatively you can allow group access using unix-group:your_group_name.

After you run the virt-install command you can watch the progress of the installation by running:

vncviewer localhost

After the installation is done, proceed with Running the test suite.