Add some notes on current project state.

NOTES.md

@@ -4,6 +4,58 @@ The new version of pls uses a more complicated/less user friendly
 API in favor of performance and memory guarantees.
 API in favor of performance and memory guarantees.
 For the old version refer to the second half of this document.
 For the old version refer to the second half of this document.
 
 
+# 13.03.2020 - Coroutines and 'Nice API'
+
+Before Christmas we finished the 'ugly' inline/object based API.
+Because this was very hard to use (even for us), the only options
+where to go to a class based solution like TBB (everything MUST be a
+class, as soon as you call into user code promises are gone) or to
+go with a fully fledged continuation based API.
+
+We decide to go for the continuation API, as one of our project questions
+was if we can incorporate a nice high level API with a library only solution.
+
+Following this we need coroutines and we also need the stealing to work for more
+than two child tasks.
+
+## Coroutines basic implementation
+
+We go for an implementation similar to [boost.context](https://github.com/boostorg/context),
+where the context is stored and changed using custom assembly.
+We do so, as jumpbuf is not portable at all (i.e. behaviour for the jumps
+we intent to do are EXPLICITLY not allowed in the standard, see 
+[docs](https://en.cppreference.com/w/c/program/setjmp) on jumping other
+than what expections are allowed to do).
+
+Inspirations for what we use in our end-product are the [deboost.context](https://github.com/septag/deboost.context),
+ [tinycoroutine](https://github.com/soundsrc/tinycoroutine).
+ 
+Our final implementatino uses custom assembly on supported platforms,
+being faster than boost.context as we choose to ommit unneccesary setup steps,
+as the focus is on quick switching into a new routine. We plan to also use
+this for 'tail calls' in our future code (and think that this is a good fit if
+we plan for composable APIs in future work).
+ 
+## TSan support...
+ 
+We want to integrate tsan into our fiber for sure, as it has helped us in the
+past to catch various bugs. Unfortionetly, we found that our programm crashes 
+when running it with it. The first solution was to add the new (mid 2019) API
+for explicit fiber switching. This seemed to work, however, the programm crased
+after some time. After long research, we found that it was a memory leak in
+thet tsan implementation. [We fixed it](https://reviews.llvm.org/D76073) 
+and integrated support for it into a seperate coroutine library.
+
+## Multiple child steal
+
+We then needed to adopt our stealing procedure/queue to support multiple
+child tasks. To do so in a lock-free manner required some tinkering.
+-> further details are in our paper notebook for now. In summary,
+we use a flag to atomicially trade in resources when stealing a tasks
+and use a lock free stack to store the resources pending on a task.
+
+
+
 # 05.11.2019 - Memory Allocation, 'Fat Objects'
 # 05.11.2019 - Memory Allocation, 'Fat Objects'
 
 
 We change our memory allocation for all memory the scheduler requires
 We change our memory allocation for all memory the scheduler requires