Tests in LKFT

Strategy and Workflow

Due to the amount of data that LKFT produces, test results need to be clear and actionable. This means there is a very low tolerance for any tests which sometimes pass and sometimes fail. Further, any test that is expected to fail is marked as a known issue in qa-reports so that results are always in a ‘all green’ state. This makes triaging new regressions straightforward.

Whenever a test becomes a known issue, a bug is opened in bugzilla. All bugzilla bugs are regularly reviewed, investigated, and reported internally or upstream.

All regressions are triaged and reported internally and/or upstream. Typically the test will also be added as a known issue until it is fixed.

Running Test Suites

Linux Test Project (LTP)

The following test sets are run in LTP:

  • cap_bounds
  • commands
  • containers
  • controllers
  • crypto
  • cve
  • dio
  • fcntl-locktests
  • filecaps
  • fs_bind
  • fs_perms_simple
  • fs
  • hugetlb
  • io
  • ipc
  • math
  • mm
  • nptl
  • pty
  • sched
  • securebits
  • syscalls
  • tracing

The version of LTP used is the most recent release, plus backports that fix regressions, if any.

LTP Issues

LTP issues are tracked in several places.

Bugs are tracked in bugzilla and can be seen at bugs.linaro.org (query for product ‘Kernel Functional Testing’ and component ‘Linux Test Project (LTP)’).

A skipfile is maintained which contains a list of tests that are not suitable to run for a variety of reasons. LKFT’s LTP skipfile can be found in test-definitions repository.

Finally, known issues are used to annotate failures in qa-reports, so that known failures show up as ‘xfail’ (expected fail), instead of a failure. The known issues that LKFT uses in LTP can be found at qa-reports-known-issues.

Linux Kernel Selftests

Linux Kernel Selftests, or kselftests, are tests that are included in the linux kernel tree under tools/testing/selftests/.

The version used depends on the kernel version under test:

  • linux-next uses selftest included in linux-next
  • mainline uses selftest included in mainline
  • Latest stable uses selftest included in latest stable
  • All other LTS branches also use the version from latest stable

Running a mismatched kselftest version does create false positives from time to time, but it is the best balance between a supported and up to date set of tests and an older kernel.

Subsystems tested

The kernel includes a thorough set of tests for the various kernel subsystems. This is a list of kernel selftests subsystems that get executed:

  • kselftest-android
  • kselftest-arm64
  • kselftest-breakpoints
  • kselftest-capabilities
  • kselftest-cgroup
  • kselftest-clone3
  • kselftest-core
  • kselftest-cpu-hotplug
  • kselftest-cpufreq
  • kselftest-drivers-dma-buf
  • kselftest-efivarfs
  • kselftest-filesystems
  • kselftest-filesystems-binderfs
  • kselftest-firmware
  • kselftest-fpu
  • kselftest-futex
  • kselftest-gpio
  • kselftest-intel_pstate
  • kselftest-ipc
  • kselftest-ir
  • kselftest-kcmp
  • kselftest-kexec
  • kselftest-kvm
  • kselftest-lib
  • kselftest-livepatch
  • kselftest-membarrier
  • kselftest-memfd
  • kselftest-memory-hotplug
  • kselftest-mincore
  • kselftest-mount
  • kselftest-mqueue
  • kselftest-net
  • kselftest-net-forwarding
  • kselftest-net-mptcp
  • kselftest-netfilter
  • kselftest-nsfs
  • kselftest-openat2
  • kselftest-pid_namespace
  • kselftest-pidfd
  • kselftest-proc
  • kselftest-pstore
  • kselftest-ptrace
  • kselftest-rseq
  • kselftest-rtc
  • kselftest-seccomp
  • kselftest-sigaltstack
  • kselftest-size
  • kselftest-splice
  • kselftest-static_keys
  • kselftest-sync
  • kselftest-sysctl
  • kselftest-tc-testing
  • kselftest-timens
  • kselftest-timers
  • kselftest-tmpfs
  • kselftest-tpm2
  • kselftest-user
  • kselftest-vm
  • kselftest-x86
  • kselftest-zram

Kselftest Issues

Kselftest issues are tracked in several places.

Bugs are tracked in bugzilla and can be seen at bugs.linaro.org (query for product ‘Kernel Functional Testing’ and component ‘kselftest’).

A skipfile is maintained which contains a list of tests that are not suitable to run for a variety of reasons. LKFT’s Kselftest skipfile can be found in test-definitions repository.

Finally, known issues are used to annotate failures in qa-reports, so that known failures show up as ‘xfail’ (expected fail), instead of a failure. The known issues that LKFT uses in kselftest can be found at qa-reports-known-issues.

Perf

Linux perf is built and tested using a basic set of tests implemented at test-definitions.

LKFT Integration with Perf testing on Arm64

Build Kernel with extra debug configurations

Except the default config, we need to enable extra Linux kernel configurations for building kernel image. Below is some missed configurations that we can enable with the commands:

  export ARCH=arm64
  export CROSS_COMPILE=aarch64-linux-gnu-
  cd $LINUX_SRC
  make defconfig
  ./scripts/config -e CONFIG_BPF_SYSCALL
  ./scripts/config -e CONFIG_BPF_JIT_ALWAYS_ON
  ./scripts/config -e CONFIG_TRACEPOINTS
  ./scripts/config -e CONFIG_KPROBES
  ./scripts/config -e CONFIG_UPROBES
  ./scripts/config -e CONFIG_KRETPROBES
  ./scripts/config -e CONFIG_PROC_KCORE
  ./scripts/config -e CONFIG_NOP_TRACER
  ./scripts/config -e CONFIG_TRACER_MAX_TRACE
  ./scripts/config -e CONFIG_RING_BUFFER
  ./scripts/config -e CONFIG_EVENT_TRACING
  ./scripts/config -e CONFIG_CONTEXT_SWITCH_TRACER
  ./scripts/config -e CONFIG_TRACING
  ./scripts/config -e CONFIG_GENERIC_TRACER
  ./scripts/config -e CONFIG_FTRACE
  ./scripts/config -e CONFIG_FUNCTION_TRACER
  ./scripts/config -e CONFIG_FUNCTION_GRAPH_TRACER
  ./scripts/config -e CONFIG_SCHED_TRACER
  ./scripts/config -e CONFIG_FTRACE_SYSCALLS
  ./scripts/config -e CONFIG_TRACER_SNAPSHOT
  ./scripts/config -e CONFIG_KPROBE_EVENTS
  ./scripts/config -e CONFIG_UPROBE_EVENTS
  ./scripts/config -e CONFIG_BPF_EVENTS
  ./scripts/config -e CONFIG_DYNAMIC_EVENTS
  ./scripts/config -e CONFIG_PROBE_EVENTS
  ./scripts/config -e CONFIG_DYNAMIC_FTRACE
  yes "" | make oldconfig

Installation dependency libs on target board

# apt-get install flex bison libelf-dev libaudit-dev libdw-dev libunwind* python-dev binutils-dev \
  libnuma-dev libbfd-dev libelf1 libperl-dev libslang2 libslang2-dev libunwind8 libunwind8-dev \
  binutils-multiarch-dev elfutils libiberty-dev libncurses5-dev
# apt-get install clang-7 llvm-7

The kernel source code path

The kernel source code is fatal for ‘perf probe’ related testings; usually, on Arm platform the perf probe related testings fail due to perf tool cannot find the kernel source code path.

On x86 platform, it’s quite common to use the same PC to build kernel and perf tool, and then can directly run the perf test on it. Thus it’s smooth for x86 to allow debugging tools to find the source code path based on vmlinux’s dwarf info.

On the other hand, Arm platforms, usually, we cross compile kernel image on our PC or laptop and then run the testing on Arm boards. So the issue is perf tool uses the vmlinux’s dwarf info to find the source code path but the path is PC’s path but not the kernel source code path on the target board.

To fix this issue, the simple method is to create the same path on Arm platforms with the PC; for example, on the PC side, build the kernel in the folder “/home/leoy/Work/opensource/linux-cs-dev”; then on Arm board, should create the path “/home/leoy/Work/opensource/linux-cs-dev” and mount NFS on this folder. Thus the Arm board can use the same code base and this allows perf tool to easily to find kernel source code.

Exporting perf python lib

Perf provides python’s .so, which needs to tell python the right path to export ‘perf’ class. This can be finished with below command:

export PYTHONPATH=/home/leoy/Work/opensource/linux-cs-dev/tools/perf/python/

Perf config file

We need to configure clang related options thus allows perf tool to have knowledge for clang and llvm:

# cd $LINUX_SRC
# ./tools/perf/perf config llvm.clang-path=clang-7
# ./tools/perf/perf config llvm.kbuild-dir=/home/leoy/Work/opensource/linux-cs-dev/
# ./tools/perf/perf config llvm.clang-opt="-g"

Testing result

# cd $LINUX_SRC
# ./tools/perf/perf test
Test ID Test Case DB410c Juno Comment
1 vmlinux symtab matches kallsyms Ok Ok vmlinux must be placed in the current folder.
2 Detect openat syscall event Ok Ok  
3 Detect openat syscall event on all cpus Ok Ok  
4 Read samples using the mmap interface Ok Ok  
5 Test data source output Ok Ok  
6 Parse event definition strings Ok Ok  
7 Simple expression parser Ok Ok  
8 PERFRECORD* events & perf_sample fields Ok Ok  
9 Parse perf pmu format Ok Ok  
10 DSO data read Ok Ok  
11 DSO data cache Ok Ok  
12 DSO data reopen Ok Ok  
13 Roundtrip evsel->name Ok Ok  
14 Parse sched tracepoints fields Ok Ok  
15 syscalls:sys_enter_openat event fields Ok Ok  
16 Setup struct perf_event_attr Ok Ok  
17 Match and link multiple hists Ok Ok  
18 ‘import perf’ in python Ok Ok  
19 Breakpoint overflow signal handler Disabled Disabled The signal handler doesn’t work well with breakpoint on Arm/Arm64.
20 Breakpoint overflow sampling Disabled Disabled The signal handler doesn’t work well with breakpoint on Arm/Arm64.
21 Breakpoint accounting Ok Ok  
22.1 Read Only Watchpoint Ok Ok  
22.2 Write Only Watchpoint Ok Ok  
22.3 Read / Write Watchpoint Ok Ok  
22.4 Modify Watchpoint Ok Ok  
23 Number of exit events of a simple workload Ok FAILED Any big.LITTLE system cannot migrate events between different CPU variants, thus this case will fail on big.LITTLE system. This case can pass on SMP platform (e.g. DB410c).
24 Software clock events period values Ok Ok  
25 Object code reading Ok Ok  
26 Sample parsing Ok Ok  
27 Use a dummy software event to keep tracking Ok Ok  
28 Parse with no sample_id_all bit set Ok Ok  
29 Filter hist entries Ok Ok  
30 Lookup mmap thread Ok Ok  
31 Share thread mg Ok Ok  
32 Sort output of hist entries Ok Ok  
33 Cumulate child hist entries Ok Ok  
34 Track with sched_switch Ok Ok  
35 Filter fds with revents mask in a fdarray Ok Ok  
36 Add fd to a fdarray, making it autogrow Ok Ok  
37 kmod_path__parse Ok Ok  
38 Thread map Ok Ok  
39.1 Basic BPF llvm compile Ok Ok  
39.2 kbuild searching Ok Ok  
39.3 Compile source for BPF prologue generation Ok Ok  
39.4 Compile source for BPF relocation Ok Ok  
40 Session topology Ok Ok  
41.1 Basic BPF filtering Ok Ok  
41.2 BPF pinning Ok Ok  
41.3 BPF prologue generation Ok Ok  
41.4 BPF relocation checker Ok Ok  
42 Synthesize thread map Ok Ok  
43 Remove thread map Ok Ok  
44 Synthesize cpu map Ok Ok  
45 Synthesize stat config Ok Ok  
46 Synthesize stat Ok Ok  
47 Synthesize stat round Ok Ok  
48 Synthesize attr update Ok Ok  
49 Event times Ok Ok  
50 Read backward ring buffer Ok Ok  
51 Print cpu map Ok Ok  
52 Probe SDT events Ok Ok  
53 is_printable_array Ok Ok  
54 Print bitmap Ok Ok  
55 perf hooks Ok Ok  
56 builtin clang support Skip Skip  
57 unit_number__scnprintf Ok Ok  
58 mem2node Ok Ok  
59 time utils Ok Ok  
60 map_groups__merge_in Ok Ok  
61 DWARF unwind Ok Ok  
62 probe libc’s inet_pton & backtrace it with ping Skip Skip  
63 Check open filename arg using perf trace + vfs_getname Ok Ok If fail, the failure is caused by cannot find kernel source code.
64 Zstd perf.data compression/decompression Skip Skip  
65 Add vfs_getname probe to get syscall args filenames Ok Ok If fail, the failure is caused by cannot find kernel source code.
66 Use vfs_getname probe to get syscall args filenames Ok Ok If fail, the failure is caused by cannot find kernel source code.

Video4Linux (v4l2)

The v4l2-compliance tests are run using a virtual driver (VIVI)

KVM Unit Tests

KVM-unit-tests are run to verify basic functionality of KVM.

S Suite

S I/O Benchmark Suite are enabled for a subset of systems to verify I/O performance.

For the moment, this test makes it possible to execute the following two benchmarks with the suite:

  • responsiveness, by measuring start-up times of real applications under real background workloads;
  • throughput with processes doing filesystem or raw I/O in parallel (figure of merit measured by many other suites too).

KUnit

Separate test runs of KUnit are enabled for Qemu on x86 (64/32 bits) and arm (64/32 bits).

Future Test Suites

We are continuously evaluating test suites for use in LKFT. We are guided by feedback from kernel engineers and the community at large. If you would like to see a particular test or test suite added to LKFT, please contact us.