Add notes on our current measurement setup.

NOTES.md

@@ -4,6 +4,59 @@ A collection of stuff that we noticed during development.
 Useful later on two write a project report and to go back
 in time to find out why certain decisions where made.
 
+## 24.06.2019 - Further Benchmark Settings and First Results
+
+As a further option to reduce potential inference of our benchmark
+process we add the `CPUAffinity` option to systemd, setting the
+CPU affinity of all processes started by the startup procedure
+to core 0, thus isolating cores 1 to 7 for our benchmarks.
+For docs on this see [the systemd manual](https://www.freedesktop.org/software/systemd/man/systemd.exec.html).
+(Please note that this is not prefect, it seems that a kernel parameter
+would perform even better, but our current measurements show this
+working well enough to not go that deep a long as we do not see the
+need in further testing).
+
+We could potentially better our results further by using a preemtive
+patch on a newer kernel, e.g. Arch linux with preemt patch.
+
+Our first results look promising, and show that our measurement
+methodology can indeed work (measured on the banana PI with the
+above settings). Backgroud tasks are started from core 1 to 7
+with 500us work, 5000us pause. Benchmarks are started pinned
+to core 1 to 7:
+
+FFT without background tasks:
+
+<img src="./media/2ad04ce5_fft_boxplot_isolated.png" width="400"/>
+
+
+FFT with background tasks:
+
+<img src="./media/2ad04ce5_fft_boxplot_multiprogrammed.png" width="400"/>
+
+Matrix without background tasks:
+
+<img src="./media/2ad04ce5_matrix_boxplot_isolated.png" width="400"/>
+
+Matrix with background tasks:
+
+<img src="./media/2ad04ce5_matrix_boxplot_multiprogrammed.png" width="400"/>
+
+
+The plot's include all outliers, showing that the workst case times can
+be measured and compared under these circumstances.
+One option we might change in the future is how/when the background
+preempting workers are executed, to better 'produce' bad situations
+for the scheduler, e.g. have a higher active time for the workers
+(close to maybe 1/4 of the benchmark times), to actually fully
+remove one worker from a few tasks (we would like to do that with
+our tests on different queue types, to e.g. see if the locknig queue
+actualy is worse than lock free implementations by blocking up other
+tasks).
+
+
+
+
 ## 14.06.2019 - Benchmark Settings/Real Time Execution on Linux
 
 Our set goal of this project is predictable execution times.