From 028887218f56eb9aa6d438a44882ca17607b676c Mon Sep 17 00:00:00 2001 From: FritzFlorian Date: Fri, 17 Apr 2020 15:16:34 +0200 Subject: [PATCH] Finalize banana pi setup notes. --- BANANAPI.md | 11 +++++------ NOTES.md | 2 +- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/BANANAPI.md b/BANANAPI.md index 02c8f8d..69de631 100644 --- a/BANANAPI.md +++ b/BANANAPI.md @@ -90,7 +90,7 @@ Here are some good articles discussing jitter on linux systems: We use the following settings: ```shell script -mce=ignore_ce nosoftlockup nmi_watchdog=0 transparent_hugepage=never processor.max_cstate=1 idle=poll nohz=on nohz_full=1-7 isolcpus=1-7 +mce=ignore_ce nosoftlockup nmi_watchdog=0 transparent_hugepage=never processor.max_cstate=1 idle=poll nohz=on nohz_full=1-7 ``` - ***mce=ignore_ce*** do not scan for hw errors. Reduce the jitter introduced by periodic runs @@ -99,15 +99,13 @@ mce=ignore_ce nosoftlockup nmi_watchdog=0 transparent_hugepage=never processor.m - ***transparent_hugepage=never*** do not scan for small pages to combine to hugepages. We have no issues with memory usage, spare us of this periodic jitter. - ***processor.max_cstate=1 idle=poll*** do not switch to CPU power saving modes (c-states). Just run all cores at full speed all the time (we do not care about energy during our tests). - ***nohz=on nohz_full=1-7*** disable houskeeping os ticks on our isolated benchmark cores. core 0 will handle these when needed. -- ***isolcpus=1-7*** ignores cores 1 to 7 for scheduling processes (expect system processes that are bound to cores). ### Pin all other processes to core 0 (crgoups) We want to isolate our measurements to cores 1 to 7 and use core 0 for all non benchmark related processes. -isolcpus does some of this during startup (TODO: see if there are issues with task migration if we use isolcpus), -however, we need to configure the system to schedule our threads specifically onto the isolated cores. - -cgroups is a tool well suited for this. See the tutorial for further information: https://github.com/lpechacek/cpuset/blob/master/doc/tutorial.txt +isolcpus is often used for this, however, we found that it disables the scheduler from balancing tasks +between the isolated cores. A better approach is to use cgroups. +See the tutorial for further information: https://github.com/lpechacek/cpuset/blob/master/doc/tutorial.txt Essentially, we can partition our cores into two isolated groups, then map all tasks that can be moved away from our benchmark cores, to ensure low influence of background tasks. Cgroups also nicely interact with the real time scheduler, as described here https://www.linuxjournal.com/article/10165, because @@ -196,6 +194,7 @@ echo "Some might fail because they can not be mapped (e.g. core specific timers echo "" echo "" +echo 1 > /proc/irq/default_smp_affinity for dir in /proc/irq/*/ do echo "Mapping $dir ..." diff --git a/NOTES.md b/NOTES.md index dcb81a0..a7ccac7 100644 --- a/NOTES.md +++ b/NOTES.md @@ -28,7 +28,7 @@ software components with arch rolling releases very hard. Even with tinkering we to update the system without the kernel and without boot issues. Following the above discussion we see that armbian is easy to setup, but has no prebuild rt image. -Because of this we decide to use the armbian build script and add the rt patch to it, giving us the best +Because of this, we decide to use the armbian build script and add the rt patch to it, giving us the best of both worlds. Full setup documentation for the os and how to further tune isolated performance can be found in a -- libgit2 0.26.0