Add initial notes on banana pi benchmark setup.

Our benchmarks require isolation from other system effects. Additionally,
we would like to have a system with 'very symmetric cores' (no hyperthreading,
no different memory connections, ....) that falls more into  the embedded
class of devices than desktop processors do. The notes describe how to setup
an banana pi m3 with a linux distri with rt preemt patch to achieve this.

BANANAPI.md

@@ -6,16 +6,60 @@ time distributions of the measurement runs.
 
 
 ## Base Setup
 ## Base Setup
 
 
-First step is to get a linux image running on the banana PI. Armbian worked very well for us,
-follow these setups to get it up and running:
-
-- Download the mainline base kernel from armbian: https://www.armbian.com/bananapi-m3/#kernels-archive-all
-- Unbpack the download
-- Prepare a micro SD card for flashing
-- Use etcher (https://www.balena.io/etcher/) or similar to burn the image on the sd card
-- Insert the micro SD card into the banana PI and power it on
-- Follow the instructions to setup an user account (best done using HDMI out and keyboard attatched)
-- Test network connection/ssh login (armbian from our experience just works, that why we chose it)
+First step is to get a linux image running on the banana PI. We choose to use the
+[armbian](https://www.armbian.com/) project as it is the only one with support for the
+bananaPI m3. For this we generally 
+[follow the instructions given](https://docs.armbian.com/Developer-Guide_Build-Preparation/),
+below are notes on what to do to get the rt kernel patch into it and to build.
+
+You can also use [our pre-build image](https://drive.google.com/open?id=1RiHymBO_XjOk5tMAL31iOSJGfncrWFQh)
+and skip the build process below. Just use etcher (https://www.balena.io/etcher/) or similar,
+flash an sd card and the PI should boot up. Default login is root/1234, follow the instructions,
+then continue with the isolating system setup steps for more accurate measurements.
+
+General Setup:
+- Setup an ubuntu bionic 18.04 virtual box VM
+- `# apt-get -y -qq install git`
+- `$ git clone --depth 1 https://github.com/armbian/build`
+- `$ cd build`
+- To verify the environment first go for a 'clean build' without patch: `# ./compile.sh` 
+- Select the bananaPI m3 board and a minimal console build
+
+Apply RT Pach:
+- Find the current kernel version armbian is using, e.g. from the previous build logs
+- Download and unpack the matching rt path from https://mirrors.edge.kernel.org/pub/linux/kernel/projects/rt/
+- You should have a single .patch file, place it in build/userpatches/kernel/sunix-current/patch-5.4.28-rt19.patch
+- Re-run the `# ./compile.sh` script
+- Select BananaPI M3, Command Line Minimal and SHOW KERNEL CONFIG
+- The build should pick up the patch (and show it in the logs)
+- You will be ask to fill in some settings. Choose (4) fully preemptive at the first option
+- Fill out the other asked settings to your liking. To avoid issues just leave them at default.
+- You will then be in the kernel config window
+- Here disable the file systems AUFS and NFS in the settings (they cause build issues and we do not need them)
+- Store the settings and build the image
+- If successfull, the flashed image should show the preempt patch with `uname -a` and should have good latencies in cyclictest
+
+## Run project
+
+First setup some base dependencies for running the benchmark and tests:
+
+- `# apt-get install rt-tests`
+- `# apt-get install build-essential`
+- `# apt-get install cmake`
+
+Next EMBB is required as a comparison in the benchmark suite. Install it using the following or similar
+(as described on their github page, https://github.com/siemens/embb):
+- `$ wget https://github.com/siemens/embb/archive/v1.0.0.zip`
+- `$ unzip v1.0.0.zip`
+- `$ cd embb-1.0.0`
+- `$ mkdir cmake-build-release`
+- `$ cd cmake-build-release`
+- `$ cmake ../`
+- `$ cmake --build .`
+- `# cmake --build . --target install`
+
+This are all dependencies needed for executing the benchmark project and pls itself.
+Follow the project specific instructions for how to use them.
 
 
 ## Tweaking Scheduler, CPU and Interrupts
 ## Tweaking Scheduler, CPU and Interrupts
 
 
@@ -48,13 +92,16 @@ Additionally, disabling any dynamic frequency scaling makes tests more reproduca
 
 
 Create a file called 'setup_cpu.sh' and modify it with 'chmod +x setup_cpu.sh':
 Create a file called 'setup_cpu.sh' and modify it with 'chmod +x setup_cpu.sh':
 ```shell script
 ```shell script
+#!/bin/bash
+
 echo "Writing frequency utils settings file..."
 echo "Writing frequency utils settings file..."
 echo "ENABLE=true
 echo "ENABLE=true
-MIN_SPEED=1008000
-MAX_SPEED=1008000
-GOVERNOR=performance" > vim /etc/default/cpufrequtils
+MIN_SPEED=1412000
+MAX_SPEED=1412000
+GOVERNOR=performance" > /etc/default/cpufrequtils
 
 
 echo "Restarting frequency utils service..."
 echo "Restarting frequency utils service..."
+systemctl restart cpufrequtils
 
 
 echo "Done!"
 echo "Done!"
 echo "Try ./watch_cpu.sh to see  if everything worked."
 echo "Try ./watch_cpu.sh to see  if everything worked."
@@ -63,9 +110,18 @@ echo "Test your cooling by stressing the cpu and watching the temperature output
 
 
 Create a file called 'watch_cpu.sh' and modify it with 'chmod +x watch_cpu.sh':
 Create a file called 'watch_cpu.sh' and modify it with 'chmod +x watch_cpu.sh':
 ````shell script
 ````shell script
-echo "Max Frequencies"
+#!/bin/bash
+
+echo "Min/Max Frequencies"
+cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_min_freq
+echo "-----"
 cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_max_freq
 cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_max_freq
 
 
+echo "Scaling Min/Max Frequencies"
+cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq
+echo "-----"
+cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
+
 echo "Actual Frequencies"
 echo "Actual Frequencies"
 cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq
 cat /sys/devices/system/cpu/cpu*/cpufreq/cpuinfo_cur_freq
 
 
@@ -108,7 +164,9 @@ try to get a very predictable behavior in our RT scheduler.
 
 
 Create a file called 'setup_rt.sh' and modify it with 'chmod +x setup_rt.sh':
 Create a file called 'setup_rt.sh' and modify it with 'chmod +x setup_rt.sh':
 ```shell script
 ```shell script
-sysctl -w kernel.sched_rt_runtime_us=-1
+#!/bin/bash
+
+sysctl -w kernel.sched_rt_runtime_us=1000000
 sysctl -w kernel.sched_rt_period_us=1000000
 sysctl -w kernel.sched_rt_period_us=1000000
 ````
 ````
 
 
@@ -123,7 +181,7 @@ sysctl -w kernel.sched_rt_period_us=1000000
 
 
 To run the tests use the following (or a similar command with different rt policy):
 To run the tests use the following (or a similar command with different rt policy):
 
 
-`taskset FFFE chrt -rr 80 sudo -u $SUDO_USER <benchmark>`
+`taskset FFFE chrt --fifo 80 sudo -u $SUDO_USER <benchmark>`
 
 
 This maps the process to all cores but core 0 and runs them using the round robin real time schedule.
 This maps the process to all cores but core 0 and runs them using the round robin real time schedule.
 Rplace -rr with --fifo to use the first in first out scheduler.
 Rplace -rr with --fifo to use the first in first out scheduler.

NOTES.md

@@ -16,11 +16,23 @@ The main concern is software. Options are the vendor image (kernel 3.4 with RT p
 an [arch image from the forum](http://forum.banana-pi.org/t/bananapi-bpi-m3-new-image-archlinux-4-18-1-1-arch-2018-08-19/6544)
 an [arch image from the forum](http://forum.banana-pi.org/t/bananapi-bpi-m3-new-image-archlinux-4-18-1-1-arch-2018-08-19/6544)
 (kernel 4.19 with RT patch) or [armbian](https://www.armbian.com/bananapi-m3/) (kernel 5.4, NO RT patch).
 (kernel 4.19 with RT patch) or [armbian](https://www.armbian.com/bananapi-m3/) (kernel 5.4, NO RT patch).
 We tried all three. The vendor image is simply too old to be comparable to modern changes in linux.
 We tried all three. The vendor image is simply too old to be comparable to modern changes in linux.
-The arch kernel is relatively new and has an RT patch, but it requires quite a bit manual setup.
-We therefore settle for the armbian project, as it offers frequent updates and it is questionable 
-if the rt patch has even big advantages for our test cases. 
-
-Full setup documentation (tune for isolated performance): [BANANAPI.md](./BANANAPI.md)
+We then compared the arch rt kernel and the most recent armbian kernel using 
+[cyclictest](https://manpages.debian.org/jessie/rt-tests/cyclictest.8). We ran a normal smp test
+using `cycltest --smp -p95 -m` on both systems while stressing the processor with lower priority
+applications to 100% utilization. The rt arch kernel had maximum latencies between
+100us and 180us on different cores, the armbian kernel had latencies between 550us and 730us on
+different cores. As our benchmarks are in the magnitude of a few milliseconds (e.g. the matrix
+with 3ms) we think it is important to choose an rt kernel, especially if we later look at
+applications running periodically. However, the arch kernel is 2 years old, making updating its
+software components with arch rolling releases very hard. Even with tinkering we can not get it
+to update the system without the kernel and without boot issues.
+
+Following the above discussion we see that armbian is easy to setup, but has no prebuild rt image.
+Because of this we decide to use the armbian build script and add the rt patch to it, giving us the best
+of both worlds.
+
+Full setup documentation for the os and how to further tune isolated performance can be found in a
+separate document: [BANANAPI.md](./BANANAPI.md)
 
 
 # 24.03.2020 - mmap stack
 # 24.03.2020 - mmap stack