diff --git a/BANANAPI.md b/BANANAPI.md index 3a76d28..4390425 100644 --- a/BANANAPI.md +++ b/BANANAPI.md @@ -6,16 +6,60 @@ time distributions of the measurement runs. ## Base Setup -First step is to get a linux image running on the banana PI. Armbian worked very well for us, -follow these setups to get it up and running: - -- Download the mainline base kernel from armbian: https://www.armbian.com/bananapi-m3/#kernels-archive-all -- Unbpack the download -- Prepare a micro SD card for flashing -- Use etcher (https://www.balena.io/etcher/) or similar to burn the image on the sd card -- Insert the micro SD card into the banana PI and power it on -- Follow the instructions to setup an user account (best done using HDMI out and keyboard attatched) -- Test network connection/ssh login (armbian from our experience just works, that why we chose it) +First step is to get a linux image running on the banana PI. We choose to use the +[armbian](https://www.armbian.com/) project as it is the only one with support for the +bananaPI m3. For this we generally +[follow the instructions given](https://docs.armbian.com/Developer-Guide_Build-Preparation/), +below are notes on what to do to get the rt kernel patch into it and to build. + +You can also use [our pre-build image](https://drive.google.com/open?id=1RiHymBO_XjOk5tMAL31iOSJGfncrWFQh) +and skip the build process below. Just use etcher (https://www.balena.io/etcher/) or similar, +flash an sd card and the PI should boot up. Default login is root/1234, follow the instructions, +then continue with the isolating system setup steps for more accurate measurements. + +General Setup: +- Setup an ubuntu bionic 18.04 virtual box VM +- `# apt-get -y -qq install git` +- `$ git clone --depth 1 https://github.com/armbian/build` +- `$ cd build` +- To verify the environment first go for a 'clean build' without patch: `# ./compile.sh` +- Select the bananaPI m3 board and a minimal console build + +Apply RT Pach: +- Find the current kernel version armbian is using, e.g. from the previous build logs +- Download and unpack the matching rt path from https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/ +- You should have a single .patch file, place it in build/userpatches/kernel/sunix-current/patch-5.4.28-rt19.patch +- Re-run the `# ./compile.sh` script +- Select BananaPI M3, Command Line Minimal and SHOW KERNEL CONFIG +- The build should pick up the patch (and show it in the logs) +- You will be ask to fill in some settings. Choose (4) fully preemptive at the first option +- Fill out the other asked settings to your liking. To avoid issues just leave them at default. +- You will then be in the kernel config window +- Here disable the file systems AUFS and NFS in the settings (they cause build issues and we do not need them) +- Store the settings and build the image +- If successfull, the flashed image should show the preempt patch with `uname -a` and should have good latencies in cyclictest + +## Run project + +First setup some base dependencies for running the benchmark and tests: + +- `# apt-get install rt-tests` +- `# apt-get install build-essential` +- `# apt-get install cmake` + +Next EMBB is required as a comparison in the benchmark suite. Install it using the following or similar +(as described on their github page, https://github.com/siemens/embb): +- `$ wget https://github.com/siemens/embb/archive/v1.0.0.zip` +- `$ unzip v1.0.0.zip` +- `$ cd embb-1.0.0` +- `$ mkdir cmake-build-release` +- `$ cd cmake-build-release` +- `$ cmake ../` +- `$ cmake --build .` +- `# cmake --build . --target install` + +This are all dependencies needed for executing the benchmark project and pls itself. +Follow the project specific instructions for how to use them. ## Tweaking Scheduler, CPU and Interrupts @@ -48,13 +92,16 @@ Additionally, disabling any dynamic frequency scaling makes tests more reproduca Create a file called 'setup_cpu.sh' and modify it with 'chmod +x setup_cpu.sh': ```shell script +#!/bin/bash + echo "Writing frequency utils settings file..." echo "ENABLE=true -MIN_SPEED=1008000 -MAX_SPEED=1008000 -GOVERNOR=performance" > vim /etc/default/cpufrequtils +MIN_SPEED=1412000 +MAX_SPEED=1412000 +GOVERNOR=performance" > /etc/default/cpufrequtils echo "Restarting frequency utils service..." +systemctl restart cpufrequtils echo "Done!" echo "Try ./watch_cpu.sh to see if everything worked." @@ -63,9 +110,18 @@ echo "Test your cooling by stressing the cpu and watching the temperature output Create a file called 'watch_cpu.sh' and modify it with 'chmod +x watch_cpu.sh': ````shell script -echo "Max Frequencies" +#!/bin/bash + +echo "Min/Max Frequencies" +cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_min_freq +echo "-----" cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_max_freq +echo "Scaling Min/Max Frequencies" +cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq +echo "-----" +cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq + echo "Actual Frequencies" cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq @@ -108,7 +164,9 @@ try to get a very predictable behavior in our RT scheduler. Create a file called 'setup_rt.sh' and modify it with 'chmod +x setup_rt.sh': ```shell script -sysctl -w kernel.sched_rt_runtime_us=-1 +#!/bin/bash + +sysctl -w kernel.sched_rt_runtime_us=1000000 sysctl -w kernel.sched_rt_period_us=1000000 ```` @@ -123,7 +181,7 @@ sysctl -w kernel.sched_rt_period_us=1000000 To run the tests use the following (or a similar command with different rt policy): -`taskset FFFE chrt -rr 80 sudo -u $SUDO_USER ` +`taskset FFFE chrt --fifo 80 sudo -u $SUDO_USER ` This maps the process to all cores but core 0 and runs them using the round robin real time schedule. Rplace -rr with --fifo to use the first in first out scheduler. diff --git a/NOTES.md b/NOTES.md index 3bf294c..dcb81a0 100644 --- a/NOTES.md +++ b/NOTES.md @@ -16,11 +16,23 @@ The main concern is software. Options are the vendor image (kernel 3.4 with RT p an [arch image from the forum](http://forum.banana-pi.org/t/bananapi-bpi-m3-new-image-archlinux-4-18-1-1-arch-2018-08-19/6544) (kernel 4.19 with RT patch) or [armbian](https://www.armbian.com/bananapi-m3/) (kernel 5.4, NO RT patch). We tried all three. The vendor image is simply too old to be comparable to modern changes in linux. -The arch kernel is relatively new and has an RT patch, but it requires quite a bit manual setup. -We therefore settle for the armbian project, as it offers frequent updates and it is questionable -if the rt patch has even big advantages for our test cases. - -Full setup documentation (tune for isolated performance): [BANANAPI.md](./BANANAPI.md) +We then compared the arch rt kernel and the most recent armbian kernel using +[cyclictest](https://manpages.debian.org/jessie/rt-tests/cyclictest.8). We ran a normal smp test +using `cycltest --smp -p95 -m` on both systems while stressing the processor with lower priority +applications to 100% utilization. The rt arch kernel had maximum latencies between +100us and 180us on different cores, the armbian kernel had latencies between 550us and 730us on +different cores. As our benchmarks are in the magnitude of a few milliseconds (e.g. the matrix +with 3ms) we think it is important to choose an rt kernel, especially if we later look at +applications running periodically. However, the arch kernel is 2 years old, making updating its +software components with arch rolling releases very hard. Even with tinkering we can not get it +to update the system without the kernel and without boot issues. + +Following the above discussion we see that armbian is easy to setup, but has no prebuild rt image. +Because of this we decide to use the armbian build script and add the rt patch to it, giving us the best +of both worlds. + +Full setup documentation for the os and how to further tune isolated performance can be found in a +separate document: [BANANAPI.md](./BANANAPI.md) # 24.03.2020 - mmap stack