Strategy and Workflow

Due to the amount of data that LKFT produces, test results need to be clear and actionable. This means there is a very low tolerance for any tests which sometimes pass and sometimes fail. Further, any test that is expected to fail is marked as a known issue in qa-reports so that results are always in a ‘all green’ state. This makes triaging new regressions straightforward.

Whenever a test becomes a known issue, a bug is opened in bugzilla. All bugzilla bugs are regularly reviewed, investigated, and reported internally or upstream.

All regressions are triaged and reported internally and/or upstream. Typically the test will also be added as a known issue until it is fixed.

Test Suites

Linux Test Project (LTP)

The following test sets are run in LTP:

  • cap_bounds
  • containers
  • fcntl-locktests
  • filecaps
  • fs_bind
  • fs_perms_simple
  • fsx
  • fs
  • hugetlb
  • io
  • ipc
  • math
  • nptl
  • pty
  • sched
  • securebits
  • syscalls
  • timers

The version of LTP used is the most recent release, plus backports that fix regressions, if any.

LTP Issues

LTP issues are tracked in several places.

Bugs are tracked in bugzilla and can be seen at bugs.linaro.org (query for product ‘Kernel Functional Testing’ and component ‘Linux Test Project (LTP)’).

A skipfile is maintained which contains a list of tests that are not suitable to run for a variety of reasons. LKFT’s LTP skipfile can be found in test-definitions repository.

Finally, known issues are used to annotate failures in qa-reports, so that known failures show up as ‘xfail’ (expected fail), instead of a failure. The known issues that LKFT uses in LTP can be found at qa-reports-known-issues.

Linux Kernel Selftests

Linux Kernel Selftests, or kselftests, are tests that are included in the linux kernel tree under tools/testing/selftests/.

The version used depends on the kernel version under test:

  • linux-next uses selftest included in linux-next
  • mainline uses selftest included in mainline
  • Latest stable uses selftest included in latest stable
  • All other LTS branches also use the version from latest stable

Running a mismatched kselftest version does create false positives from time to time, but it is the best balance between a supported and up to date set of tests and an older kernel.

Kselftest Issues

Kselftest issues are tracked in several places.

Bugs are tracked in bugzilla and can be seen at bugs.linaro.org (query for product ‘Kernel Functional Testing’ and component ‘kselftest’).

A skipfile is maintained which contains a list of tests that are not suitable to run for a variety of reasons. LKFT’s Kselftest skipfile can be found in test-definitions repository.

Finally, known issues are used to annotate failures in qa-reports, so that known failures show up as ‘xfail’ (expected fail), instead of a failure. The known issues that LKFT uses in kselftest can be found at qa-reports-known-issues.

Libhugetlbfs

libhugetlbfs is run using the latest released version.