Documentation/epicardium/api.rst

+.. _epicardium_api:
+
 Epicardium API
 ==============
 
@@ -5,8 +7,17 @@ Epicardium API
 
    .. warning::
 
-      This version of the documentation was built without
-      `Hawkmoth <https://github.com/jnikula/hawkmoth>`_ available.  This means,
-      no documentation for the Epicardium API was generated.
+      This version of the documentation was built without `Hawkmoth`_ working
+      (most likely python3-clang was not installed).  This means, no
+      documentation for the Epicardium API was generated.
+
+   .. _Hawkmoth: https://github.com/jnikula/hawkmoth
+
+This page details all available *Epicardium API* functions.  All these
+functions are defined in
+
+.. code-block:: c++
+
+   #include "epicardium.h"
 
 .. c:autodoc:: epicardium/epicardium.h

Documentation/epicardium/internal.rst

+.. _epicardium_internal_apis:
+
+Epicardium Internal APIs
+========================
+
+Core OS APIs
+------------
+.. c:autodoc:: epicardium/os/core.h

Documentation/epicardium/mutex.rst

+Mutex
+=====
+Ontop of FreeRTOS, we have our own mutex implementation.  **Never use the
+FreeRTOS mutexes directly!  Always use this abstraction layer instead**.  This
+mutex implementation tries to make reasoning about program flow and locking
+behavior easier.  And most importantly tries to help with debugging possible
+dead-locks.
+
+Design
+------
+There are a few guiding design principles:
+
+- Mutexes can only be used from tasks, **never** from interrupts!
+- Timers can use mutexes, but only with :c:func:`mutex_trylock`, **never** with
+  :c:func:`mutex_lock` (Because they are not allowed to block).
+- Locking can *never* fail (if it does, we consider this a fatal error ⇒ panic).
+- No recursive locking.
+- An unlock can only occur from the task which previously acquired the mutex.
+- An unlock is only allowed if the mutex was previously acquired.
+
+For a more elaborate explanation of the rationale behind these rules take a
+look at the :ref:`mutex-design-reasons`.
+
+Definitions
+-----------
+.. c:autodoc:: epicardium/os/mutex.h
+
+.. _mutex-design-reasons:
+
+Reasons for this Design
+-----------------------
+
+Locking can *never* fail
+^^^^^^^^^^^^^^^^^^^^^^^^
+This might seem like a bold claim at first but in the end, it is just a matter
+of definition and shifting responsibilities.  Instead of requiring all code to
+be robust against a locking attempt failing, we require all code to properly
+lock and unlock their mutexes and thus never producing a situation where
+locking would fail.
+
+Because all code using any of the mutexes is contained in the Epicardium
+code-base, we can - *hopefully* - audit it properly behaving ahead of time and
+thus don't need to add code to ensure correctness at runtime.  This makes
+downstream code easier to read and easier to reason about.
+
+History of this project has shown that most code does not properly deal with
+locking failures anyway: There was code simply skipping the mutexed action on
+failure, code blocking a module entirely until reboot, and worst of all: Code
+exposing the locking failure to 'user-space' (Pycardium) instead of retrying.
+This has lead to spurious errors where technically there would not need to be
+any.
+
+Only from tasks
+^^^^^^^^^^^^^^^
+Locking a mutex from an ISR, a FreeRTOS software timer or any other context
+which does not allow blocking is complicated to do right.  The biggest
+difficulty is that a task might be holding the mutex during execution of such a
+context and there is no way to wait for it to release the mutex.  This requires
+careful design of the program flow to choose an alternative option in such a
+case.  A common approach is to 'outsource' the relevant parts of the code into
+an 'IRQ worker' which is essentially just a task waiting for the IRQ to wake it
+up and then attempts to lock the mutex.
+
+If you absolutely do need it (and for legacy reasons), software timers *can*
+lock a mutex using :c:func:`mutex_trylock` (which never blocks).  I strongly
+recommend **not** doing that, though.  As shown above, you will have to deal
+with the case of the mutex being held by another task and it is very well
+possible that your timer will get starved of the mutex because the scheduler
+has no knowledge of its intentions.  In most cases, it is a better idea to use
+a task and attempt locking using :c:func:`mutex_lock`.
+
+.. todo::
+
+   We might introduce a generic IRQ worker queue system at some point.
+
+No recursive locking
+^^^^^^^^^^^^^^^^^^^^
+Recursive locking refers to the ability to 'reacquire' a mutex already held by
+the current task, deeper down in the call-chain.  Only the outermost unlock
+will actually release the mutex.  This feature is sometimes implemented to
+allow more elegant abstractions where downstream code does not need to know
+about the mutexes upstream code uses and can still also create a larger region
+where the same mutex is held.
+
+But exactly by hiding the locking done by a function, these abstractions make
+it hard to trace locking chains and in some cases even make it impossible to
+create provably correct behavior.  As an alternative, I would suggest using
+different mutexes for the different levels of abstraction.  This also helps
+keeping each mutex separated and 'local' to its purpose.
+
+Only unlock from the acquiring task
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Because of the above mentioned mutex locking semantics, there should never be a
+need to force-unlock a forgein mutex.  Even in cases of failures, all code
+should still properly release all mutexes it holds.  One notable exceptions is
+``panic()``\s which will abort all ongoing operations anyway.
+
+Only unlock once after acquisition
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Justified with an argument of robustness, sometimes the :c:func:`mutex_unlock`
+call is written in a way that allows unlocking an already unlocked mutex.  But
+robustness of downstream code will not really be improved by the upstream API
+dealing with arguably invalid usage.  For example, this could encourage
+practices like unlocking everything again at the end of a function "just to be
+sure".
+
+Instead, code should be written in a way where the lock/unlock pair is
+immediately recognizable as belonging together and is thus easily auditable to
+have correct locking behavior.  A common pattern to help with readability in
+this regard is the *Single Function Exit* which looks like this:
+
+.. code-block:: cpp
+
+   int function()
+   {
+           int ret;
+           mutex_lock(&some_mutex);
+
+           ret = foo();
+           if (ret) {
+                   /* Return with an error code */
+                   ret = -ENODEV;
+                   goto out_unlock;
+           }
+
+           ret = bar();
+           if (ret) {
+                   /* Return the return value from foo */
+                   goto out_unlock;
+           }
+
+           ret = 0;
+   out_unlock:
+           mutex_unlock(&some_mutex);
+           return ret;
+   }

Documentation/epicardium/overview.rst

+.. _epicardium_api_overview:
+
+Overview
+========
+Epicardium, the "main" firmware running on core 0 (more about our dual
+core design can be found :ref:`here <firmware_overview>`), exposes a lot of
+functionality to user-code via the so-called *Epicardium API*.  This API
+consists of a number of calls that can be issued by core 1 using an
+auto-generated library.
+
+API Design
+----------
+.. note::
+
+   This is the current design.  We might adjust it in the future to allow for
+   more performance in cases where this approach is too slow.  This will most
+   likely be the addition of some form of asynchroneous calls.
+
+The API is strictly synchroneous.  This means, an API call looks exactly the
+same as calling any other function.  Internally, the call will wake up
+Epicardium and there the call will be dispatched.  It will then block until
+Epicardium finished executing the call and return whatever the call has as a
+return value.  In code:
+
+.. code-block:: c++
+
+   #include "epicardium.h"
+
+   int main(void)
+   {
+   	/* ... */
+
+   	/* Call the API to write to UART. */
+   	epic_uart_write_str("Hello from core 1!\r\n", 20);
+
+   	/*
+   	 * Call the API to receive a byte from UART.
+   	 * This will block until at least one character is received.
+   	 */
+   	char chr = epic_uart_read_chr();
+
+   	/* ... */
+   }
+
+Internals
+---------
+In most cases, you won't need to care about the actual implementation of the
+API as it is well hidden inside the auto-generated library.  If you want to
+know anyway, here is a rough overview:
+
+The current design is based around a shared memory region, the semaphore
+peripherals provided by MAX32666, and the SEV/WFE mechanism.  When issuing a
+call, this is what happens:
+
+.. image:: ../static/synchroneous-api-call.svg
+
+There is one important thing to note here:  Right now, the API is only ever
+polled for new calls from Epicardium inside the idle task.  This means, that if
+it is busy with other things, the API will have the least priority.  For now,
+this has not been an issue as Epicardium is sleeping most of the time anyway.
+If it becomes one in the future we will have to introduce another
+synchronization point.

Documentation/epicardium/sensor-streams.rst

+Sensor Streams
+==============
+Sensor drivers can make their data available to core 1 in a stream-like format.
+This allows batch-reading many samples and shoud reduce pressure on the
+Epicardium API this way.  Sensor streams are read on core 1 using
+:c:func:`epic_stream_read`.
+
+This page intends to document how to add this stream interface to a sensor driver.
+It also serves as a reference of existing streams.  For that, take a look at the
+definitions in the :c:type:`stream_descriptor` enum.
+
+Adding a new Stream
+-------------------
+The list of possible sensor streams must be known at compile time.  Each stream
+gets a unique ID in the :c:type:`stream_descriptor` enum.  Please do not assign
+IDs manually but instead let the enum assign sequencial IDs.  :c:macro:`SD_MAX`
+must always be the highest stream ID.  Additionally, please document what this
+stream is for using a doc-comment so it shows up on this page.
+
+When a sensor driver enables data collection, it should also register its
+respective stream.  This is done using a :c:type:`stream_info` object.  Pass
+this object to :c:func:`stream_register` to make your stream available.  Your
+driver must guarantee the :c:member:`stream_info.queue` handle to be valid until
+deregistration using :c:func:`stream_deregister`.
+
+Definitions
+-----------
+
+.. c:autodoc:: epicardium/modules/stream.h

Documentation/hawkmoth/README.rst

+Hawkmoth - Sphinx Autodoc for C
+===============================
+
+Hawkmoth is a minimalistic Sphinx_ `C Domain`_ autodoc directive extension to
+incorporate formatted C source code comments written in reStructuredText_ into
+Sphinx based documentation. It uses Clang Python Bindings for parsing, and
+generates C Domain directives for C API documentation, and more. In short,
+Hawkmoth is Sphinx Autodoc for C.
+
+Hawkmoth aims to be a compelling alternative for documenting C projects using
+Sphinx, mainly through its simplicity of design, implementation and use.
+
+.. _Sphinx: http://www.sphinx-doc.org
+
+.. _C Domain: http://www.sphinx-doc.org/en/stable/domains.html
+
+.. _reStructuredText: http://docutils.sourceforge.net/rst.html
+
+Example
+-------
+
+Given C source code with rather familiar looking documentation comments::
+
+  /**
+   * Get foo out of bar.
+   */
+  void foobar();
+
+and a directive in the Sphinx project::
+
+  .. c:autodoc:: filename.c
+
+you can incorporate code documentation into Sphinx. It's as simple as that.
+
+You can document functions, their parameters and return values, structs, unions,
+their members, macros, function-like macros, enums, enumeration constants,
+typedefs, variables, as well as have generic documentation comments not attached
+to any symbols.
+
+Documentation
+-------------
+
+Documentation on how to install and configure Hawkmoth, and write documentation
+comments, with examples, is available in the ``doc`` directory in the source
+tree, obviously in Sphinx format and using the directive extension. Pre-built
+documentation `showcasing what Hawkmoth can do`_ is available at `Read the
+Docs`_.
+
+.. _showcasing what Hawkmoth can do: https://hawkmoth.readthedocs.io/en/latest/examples.html
+
+.. _Read the Docs: https://hawkmoth.readthedocs.io/
+
+Installation
+------------
+
+You can install Hawkmoth from PyPI_ with::
+
+  pip install hawkmoth
+
+You'll additionally need to install Clang and Python 3 bindings for it through
+your distro's package manager; they are not available via PyPI. For further
+details, see the documentation.
+
+Alternatively, installation packages are available for:
+
+* `Arch Linux`_
+
+In Sphinx ``conf.py``, add ``hawkmoth`` to ``extensions``, and point
+``cautodoc_root`` at the source tree. See the extension documentation for
+details.
+
+.. _PyPI: https://pypi.org/project/hawkmoth/
+
+.. _Arch Linux: https://aur.archlinux.org/packages/?K=hawkmoth
+
+Development and Contributing
+----------------------------
+
+Hawkmoth source code is available on GitHub_. The development version can be
+checked out via ``git`` using this command::
+
+  git clone https://github.com/jnikula/hawkmoth.git
+
+Please file bugs and feature requests as GitHub issues. Contributions are
+welcome both as emailed patches to the mailing list and as pull requests.
+
+.. _GitHub: https://github.com/jnikula/hawkmoth
+
+Dependencies
+------------
+
+- Python 3.4
+- Sphinx 3
+- Clang 6.0
+- Python 3 Bindings for Clang 6.0
+- sphinx-testing 1.0.0 (for development)
+
+These are the versions Hawkmoth is currently being developed and tested
+against. Other versions might work, but no guarantees.
+
+License
+-------
+
+Hawkmoth is free software, released under the `2-Clause BSD License`_.
+
+.. _2-Clause BSD License: https://opensource.org/licenses/BSD-2-Clause
+
+Contact
+-------
+
+IRC channel ``#hawkmoth`` on freenode_.
+
+Mailing list hawkmoth@freelists.org. Subscription information at the `list home
+page`_.
+
+.. _freenode: https://freenode.net/
+
+.. _list home page: https://www.freelists.org/list/hawkmoth

Documentation/hawkmoth/README.txt

-Hawkmoth - Sphinx Autodoc for C
-===============================
-
-Hawkmoth is a minimalistic Sphinx_ `C Domain`_ autodoc directive extension to
-incorporate formatted C source code comments written in reStructuredText_ into
-Sphinx based documentation. It uses Clang Python Bindings for parsing, and
-generates C Domain directives for C API documentation, and more. In short,
-Hawkmoth is Sphinx Autodoc for C.
-
-Hawkmoth aims to be a compelling alternative for documenting C projects using
-Sphinx, mainly through its simplicity of design, implementation and use.
-
-.. _Sphinx: http://www.sphinx-doc.org
-
-.. _C Domain: http://www.sphinx-doc.org/en/stable/domains.html
-
-.. _reStructuredText: http://docutils.sourceforge.net/rst.html
-
-Example
--------
-
-Given C source code with rather familiar looking documentation comments::
-
-  /**
-   * Get foo out of bar.
-   */
-  void foobar();
-
-and a directive in the Sphinx project::
-
-  .. c:autodoc:: filename.c
-
-you can incorporate code documentation into Sphinx. It's as simple as that.
-
-You can document functions, their parameters and return values, structs, unions,
-their members, macros, function-like macros, enums, enumeration constants,
-typedefs, variables, as well as have generic documentation comments not attached
-to any symbols.
-
-Documentation
--------------
-
-Documentation on how to configure Hawkmoth and write documentation comments,
-with examples, is available in the ``doc`` directory in the source tree,
-obviously in Sphinx format and using the directive extension. Pre-built
-documentation `showcasing what Hawkmoth can do`_ is available at `Read the
-Docs`_.
-
-.. _showcasing what Hawkmoth can do: https://hawkmoth.readthedocs.io/en/latest/examples.html
-
-.. _Read the Docs: https://hawkmoth.readthedocs.io/
-
-Installation
-------------
-
-You can install Hawkmoth from PyPI_ with::
-
-  pip install hawkmoth
-
-You'll additionally need to install Clang and Python 3 bindings for it through
-your distro's package manager; they are not available via PyPI. You may also
-need to set ``LD_LIBRARY_PATH`` so that the Clang library can be found. For
-example::
-
-  export LD_LIBRARY_PATH=$(llvm-config --libdir)
-
-Alternatively, installation packages are available for:
-
-* `Arch Linux`_
-
-In Sphinx ``conf.py``, add ``hawkmoth`` to ``extensions``, and point
-``cautodoc_root`` at the source tree. See the extension documentation for
-details.
-
-.. _PyPI: https://pypi.org/project/hawkmoth/
-
-.. _Arch Linux: https://aur.archlinux.org/packages/?K=hawkmoth
-
-Development and Contributing
-----------------------------
-
-Hawkmoth source code is available on GitHub_. The development version can be
-checked out via ``git`` using this command::
-
-  git clone https://github.com/jnikula/hawkmoth.git
-
-Please file bugs and feature requests as GitHub issues. Contributions are
-welcome both as emailed patches to the mailing list and as pull requests.
-
-.. _GitHub: https://github.com/jnikula/hawkmoth
-
-Dependencies
-------------
-
-- Python 3.4
-- Sphinx 1.8
-- Clang 6.0
-- Python 3 Bindings for Clang 6.0
-- sphinx-testing 1.0.0 (for development)
-
-These are the versions Hawkmoth is currently being developed and tested
-against. Other versions might work, but no guarantees.
-
-License
--------
-
-Hawkmoth is free software, released under the `2-Clause BSD License`_.
-
-.. _2-Clause BSD License: https://opensource.org/licenses/BSD-2-Clause
-
-Contact
--------
-
-IRC channel ``#hawkmoth`` on freenode_.
-
-Mailing list hawkmoth@freelists.org. Subscription information at the `list home
-page`_.
-
-.. _freenode: https://freenode.net/
-
-.. _list home page: https://www.freelists.org/list/hawkmoth

Documentation/hawkmoth/VERSION

-0.4
+0.7.0

Documentation/hawkmoth/__init__.py

@@ -52,7 +52,10 @@ class CAutoDocDirective(Directive):
         env = self.state.document.settings.env
 
         for (severity, filename, lineno, msg) in errors:
+            if filename:
                 toprint = '{}:{}: {}'.format(filename, lineno, msg)
+            else:
+                toprint = '{}'.format(msg)
 
             if severity.value <= env.app.verbosity:
                 self.logger.log(self._log_lvl[severity], toprint,
@@ -109,7 +112,7 @@ class CAutoDocDirective(Directive):
         return node.children
 
 def setup(app):
-    app.require_sphinx('1.8')
+    app.require_sphinx('3.0')
     app.add_config_value('cautodoc_root', app.confdir, 'env')
     app.add_config_value('cautodoc_compat', None, 'env')
     app.add_config_value('cautodoc_clang', None, 'env')

Documentation/hawkmoth/__main__.py

@@ -39,7 +39,10 @@ def main():
         print(doc)
 
     for (severity, filename, lineno, msg) in errors:
+        if filename:
             print('{}: {}:{}: {}'.format(severity.name,
                                          filename, lineno, msg), file=sys.stderr)
+        else:
+            print('{}: {}'.format(severity.name, msg), file=sys.stderr)
 
 main()

Documentation/hawkmoth/parser.py

 # Copyright (c) 2016-2017 Jani Nikula <jani@nikula.org>
-# Copyright (c) 2018-2019 Bruno Santos <brunomanuelsantos@tecnico.ulisboa.pt>
+# Copyright (c) 2018-2020 Bruno Santos <brunomanuelsantos@tecnico.ulisboa.pt>
 # Licensed under the terms of BSD 2-Clause, see LICENSE for details.
 """
 Documentation comment extractor
@@ -34,7 +34,7 @@ Otherwise, documentation comments are passed through verbatim.
 """
 
 import enum
-import itertools
+import re
 import sys
 
 from clang.cindex import CursorKind, TypeKind
@@ -75,18 +75,22 @@ def comment_extract(tu):
             current_comment = token
             continue
 
+        # Store off the token's cursor for a slight performance improvement
+        # instead of accessing the `cursor` property multiple times.
+        token_cursor = token.cursor
+
         # cursors that are 1) never documented themselves, and 2) allowed
         # between comment and the actual cursor being documented
-        if (token.cursor.kind == CursorKind.INVALID_FILE or
-            token.cursor.kind == CursorKind.TYPE_REF or
-            token.cursor.kind == CursorKind.PREPROCESSING_DIRECTIVE or
-            token.cursor.kind == CursorKind.MACRO_INSTANTIATION):
+        if (token_cursor.kind == CursorKind.INVALID_FILE or
+            token_cursor.kind == CursorKind.TYPE_REF or
+            token_cursor.kind == CursorKind.PREPROCESSING_DIRECTIVE or
+            token_cursor.kind == CursorKind.MACRO_INSTANTIATION):
             continue
 
-        if cursor is not None and token.cursor == cursor:
+        if cursor is not None and token_cursor == cursor:
             continue
 
-        cursor = token.cursor
+        cursor = token_cursor
 
         # Note: current_comment may be None
         if current_comment != None and docstr.is_doc(current_comment.spelling):
@@ -125,16 +129,18 @@ def _get_macro_args(cursor):
     if cursor.kind != CursorKind.MACRO_DEFINITION:
         return None
 
+    tokens = cursor.get_tokens()
+
     # Use the first two tokens to make sure this starts with 'IDENTIFIER('
-    x = [token for token in itertools.islice(cursor.get_tokens(), 2)]
-    if (len(x) != 2 or x[0].spelling != cursor.spelling or
-        x[1].spelling != '(' or x[0].extent.end != x[1].extent.start):
+    one = next(tokens)
+    two = next(tokens, None)
+    if two is None or one.extent.end != two.extent.start or two.spelling != '(':
         return None
 
     # Naïve parsing of macro arguments
     # FIXME: This doesn't handle GCC named vararg extension FOO(vararg...)
     args = []
-    for token in itertools.islice(cursor.get_tokens(), 2, None):
+    for token in tokens:
         if token.spelling == ')':
             return args
         elif token.spelling == ',':
@@ -161,8 +167,26 @@ def _recursive_parse(comments, cursor, nest, compat):
         return _result(comment, cursor=cursor, fmt=fmt,
                        nest=nest, name=name, args=args, compat=compat)
 
-    elif cursor.kind == CursorKind.VAR_DECL:
+    elif cursor.kind in [CursorKind.VAR_DECL, CursorKind.FIELD_DECL]:
+        if cursor.kind == CursorKind.VAR_DECL:
             fmt = docstr.Type.VAR
+        else:
+            fmt = docstr.Type.MEMBER
+
+        # If this is an array, the dimensions should be applied to the name, not
+        # the type.
+        dims = ttype.rsplit(' ', 1)[-1]
+        if dims.startswith('[') and dims.endswith(']'):
+            ttype = ttype.rsplit(' ', 1)[0]
+            name = name + dims
+
+        # If this is a function pointer, or an array of function pointers, the
+        # name should be within the parenthesis as in (*name) or (*name[N]).
+        fptr_type = re.sub(r'\((\*+)(\[[^]]*\])?\)', r'(\1{}\2)'.format(name),
+                           ttype, count=1)
+        if fptr_type != ttype:
+            name = fptr_type
+            ttype = ''
 
         return _result(comment, cursor=cursor, fmt=fmt,
                        nest=nest, name=name, ttype=ttype, compat=compat)
@@ -188,9 +212,15 @@ def _recursive_parse(comments, cursor, nest, compat):
 
         # FIXME: Handle anonymous enumerators.
 
-        fmt = docstr.Type.TYPE
+        fmts = {CursorKind.STRUCT_DECL: docstr.Type.STRUCT,
+                CursorKind.UNION_DECL: docstr.Type.UNION,
+                CursorKind.ENUM_DECL: docstr.Type.ENUM}
+
+        fmt = fmts[cursor.kind]
+
+        # name may be empty for typedefs
         result = _result(comment, cursor=cursor, fmt=fmt,
-                         nest=nest, name=ttype, compat=compat)
+                         nest=nest, name=name if name else ttype, compat=compat)
 
         nest += 1
         for c in cursor.get_children():
@@ -205,12 +235,6 @@ def _recursive_parse(comments, cursor, nest, compat):
         return _result(comment, cursor=cursor, fmt=fmt,
                        nest=nest, name=name, compat=compat)
 
-    elif cursor.kind == CursorKind.FIELD_DECL:
-        fmt = docstr.Type.MEMBER
-
-        return _result(comment, cursor=cursor, fmt=fmt,
-                       nest=nest, name=name, ttype=ttype, compat=compat)
-
     elif cursor.kind == CursorKind.FUNCTION_DECL:
         # FIXME: check args against comment
         # FIXME: children may contain extra stuff if the return type is a
@@ -261,7 +285,8 @@ def clang_diagnostics(errors, diagnostics):
            4: ErrorLevel.ERROR}
 
     for diag in diagnostics:
-        errors.extend([(sev[diag.severity], diag.location.file.name,
+        filename = diag.location.file.name if diag.location.file else None
+        errors.extend([(sev[diag.severity], filename,
                         diag.location.line, diag.spelling)])
 
 # return a list of (comment, metadata) tuples

Documentation/hawkmoth/util/docstr.py

-# Copyright (c) 2016-2017 Jani Nikula <jani@nikula.org>
-# Copyright (c) 2018-2019 Bruno Santos <brunomanuelsantos@tecnico.ulisboa.pt>
+# Copyright (c) 2016-2020 Jani Nikula <jani@nikula.org>
+# Copyright (c) 2018-2020 Bruno Santos <brunomanuelsantos@tecnico.ulisboa.pt>
 # Licensed under the terms of BSD 2-Clause, see LICENSE for details.
 """
 Documentation strings manipulation library
@@ -17,6 +17,9 @@ class Type(Enum):
     TEXT = auto()
     VAR = auto()
     TYPE = auto()
+    STRUCT = auto()
+    UNION = auto()
+    ENUM = auto()
     ENUM_VAL = auto()
     MEMBER = auto()
     MACRO = auto()
@@ -31,10 +34,13 @@ _doc_fmt = {
     Type.TEXT:       (0, '\n{text}\n'),
     Type.VAR:        (1, '\n.. c:var:: {ttype} {name}\n\n{text}\n'),
     Type.TYPE:       (1, '\n.. c:type:: {name}\n\n{text}\n'),
-    Type.ENUM_VAL:   (1, '\n.. c:macro:: {name}\n\n{text}\n'),
+    Type.STRUCT:     (1, '\n.. c:struct:: {name}\n\n{text}\n'),
+    Type.UNION:      (1, '\n.. c:union:: {name}\n\n{text}\n'),
+    Type.ENUM:       (1, '\n.. c:enum:: {name}\n\n{text}\n'),
+    Type.ENUM_VAL:   (1, '\n.. c:enumerator:: {name}\n\n{text}\n'),
     Type.MEMBER:     (1, '\n.. c:member:: {ttype} {name}\n\n{text}\n'),
     Type.MACRO:      (1, '\n.. c:macro:: {name}\n\n{text}\n'),
-    Type.MACRO_FUNC: (1, '\n.. c:function:: {name}({args})\n\n{text}\n'),
+    Type.MACRO_FUNC: (1, '\n.. c:macro:: {name}({args})\n\n{text}\n'),
     Type.FUNC:       (1, '\n.. c:function:: {ttype} {name}({args})\n\n{text}\n')
 }
 

Documentation/hawkmoth/util/readthedocs.py

+# Copyright (c) 2021, Jani Nikula <jani@nikula.org>
+# Licensed under the terms of BSD 2-Clause, see LICENSE for details.
+"""
+Read the Docs Helpers
+=====================
+
+Helpers for setting up and using Hawkmoth on Read the Docs.
+"""
+
+import os
+import subprocess
+
+def clang_setup(force=False):
+    """Try to find and configure libclang location on RTD."""
+    if 'READTHEDOCS' in os.environ or force:
+        try:
+            result = subprocess.run(['llvm-config', '--libdir'],
+                                    check=True, capture_output=True, encoding='utf-8')
+            libdir = result.stdout.strip()
+
+            # For some reason there is no plain libclang.so symlink on RTD.
+            from clang.cindex import Config
+            Config.set_library_file(os.path.join(libdir, 'libclang.so.1'))
+        except Exception as e:
+            print(e)

Documentation/how-to-build.rst

+.. _how_to_build:
+
+How To Build
+============
+If you just want to write MicroPython code for card10, you probably **won't**
+need to build the firmware yourself.  This page is for people **who want to work
+on the underlying firmware itself**.
+
+Dependencies
+------------
+* **gcc**, **binutils** & **newlib** for ``arm-none-eabi``:  The packages have
+  slightly different names on different distros.
+
+  - Ubuntu / Debian:
+
+    .. code-block:: shell-session
+
+       apt install gcc-arm-none-eabi binutils-arm-none-eabi libnewlib-arm-none-eabi
+
+  - Arch:
+
+    .. code-block:: shell-session
+
+       pacman -S arm-none-eabi-gcc arm-none-eabi-binutils arm-none-eabi-newlib
+
+  - Fedora
+
+    .. code-block:: shell-session
+
+        dnf install arm-none-eabi-gcc arm-none-eabi-binutils arm-none-eabi-newlib
+
+  - macOS (Note: The card10 firmware team used Linux so far. macOS recommendations here are experimental.)
+
+    You can use `Homebrew`_ to install the required tools.  The version of the
+    ARM crosscompiler tool chain is quite important; with the wrong version,
+    e.g. strip and/or ld might throw strange errors.
+
+    .. code-block:: shell-session
+
+        brew tap px4/px4
+        brew install px4/px4/gcc-arm-none-eabi-63
+        brew install coreutils
+
+    .. _Homebrew: https://brew.sh/
+
+  - Alternative: Download `ARM's GNU toolchain`_.  **TODO**
+
+
+* **python3**:  For meson and various scripts needed for building.
+* **meson** (>0.43.0) & **ninja**:  Unfortunately most distros only have very old versions
+  of meson in their repositories.  Instead, you'll probably save yourself a lot
+  of headaches by installing meson from *pip*.
+
+  - Ubuntu / Debian:
+
+    .. code-block:: shell-session
+
+       apt install ninja-build
+       pip3 install --user meson
+
+  - Arch (has latest *meson* in the repos):
+
+    .. code-block:: shell-session
+
+       pacman -S meson
+
+  - macOS
+
+    .. code-block:: shell-session
+
+        brew install ninja
+        pip3 install --user meson  # see https://mesonbuild.com/Getting-meson.html - you will have to add ~/.local/bin to your PATH.
+
+* One of two CRC packages is required. Pick one:
+
+  - Ubuntu / Debian / macOS
+
+    .. code-block:: shell-session
+
+        pip3 install --user crc16
+
+or
+
+    .. code-block:: shell-session
+
+        apt install python3-crcmod
+
+or
+
+    .. code-block:: shell-session
+
+        pip3 install --user crcmod
+
+  - Arch
+
+    .. code-block:: shell-session
+
+       pacman -S python-crc16
+
+* **python3-pillow**: Python Image Library
+       .. code-block:: shell-session
+
+            pip3 install --user pillow
+
+  - Arch
+
+    .. code-block:: shell-session
+
+       pacman -S python-pillow
+
+.. _ARM's GNU toolchain: https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm/downloads
+
+Cloning
+-------
+Clone the ``master`` branch of the firmware repository:
+
+.. code-block:: shell-session
+
+   $ git clone https://git.card10.badge.events.ccc.de/card10/firmware.git
+
+Build Configuration
+-------------------
+Initialize the build-system using
+
+.. code-block:: shell-session
+
+   $ ./bootstrap.sh
+
+Additional arguments to ``bootstrap.sh`` will be passed to *meson*.  You can
+use this to for example, to enable one or more of the following optional
+firmware features:
+
+- ``-Ddebug_prints=true``: Print more verbose debugging log messages
+- ``-Dble_trace=true``: Enable BLE tracing.  This will output lots of status
+  info related to BLE.
+- ``-Ddebug_core1=true``: Enable the core 1 SWD lines which are exposed on the
+  SAO connector.  Only use this if you have a debugger which is modified for core 1.
+
+.. warning::
+
+   Our build-system contains a few workarounds around short-comings in meson.
+   These workarounds might break on some setups which we did not yet test.  If
+   this is the case for you, please open an issue in our `issue tracker`_!
+
+.. _issue tracker: https://git.card10.badge.events.ccc.de/card10/firmware/issues
+
+Building
+--------
+Build using *ninja*:
+
+.. code-block:: shell-session
+
+   $ ninja -C build/
+
+If ninja succeeds, the resulting binaries are in ``build/``.  They are
+available in two formats:  As an ``.elf`` which can be flashed using a debugger
+and as a ``.bin`` which can be loaded using the provided bootloader.  Here is a
+list of the binaries:
+
+- ``build/bootloader/bootloader.elf``: Our bootloader.  It should already be on
+  your card10.  The bootloader can only be flashed using a debugger.
+- ``build/pycardium/pycardium_epicardium.bin``: The entire firmware in one ``.bin``.
+- ``build/epicardium/epicardium.elf``: The core 0 part of the firmware, called Epicardium.
+- ``build/pycardium/pycardium.elf``: Our MicroPython port, the core 1 part of the firmware.
+
+In order to do a rebuild you can issue a clean command to ninja via
+
+.. code-block:: shell-session
+
+  $ ninja -C build/ -t clean
+
+Otherwise, rerunning ``./bootstrap.sh`` will also clean the build-directory.
+
+.. note::
+
+   **macOS**: If ``strip`` fails to work on the freshly compiled ``mpy-cross``:
+   "strip: object: (...)/lib/micropython/micropython/mpy-cross/mpy-cross
+   malformed object (unknown load command 9)", you a likely not using the
+   `strip` that matches to your ``clang``. Do ``which strip && which clang``,
+   and if the paths don't match, clean up your PATHs, or as a quick hack,
+   create a symlink for strip.
+
+.. note::
+
+   If you try to flash pycardium_epicardium.bin (renamed to card10.bin)
+   and the bootloader does not finish updating, the file might be too large.
+   ~700kB is the normal size, but problems were reported where the file size
+   was >1MB. This was caused by the ``tr`` tool in the build process
+   (it's supposed to create a large file with 0xff in it) - this requires the
+   LC_ALL environment variable to be not set, or set to "C"
+   (but not UTF8 or similar).
+
+Docker
+======
+Alternatively, clone the ``master`` branch of the firmware repository and enter it:
+
+.. code-block:: shell-session
+
+   $ git clone https://git.card10.badge.events.ccc.de/card10/firmware.git
+   $ cd firmware
+
+Afterwards, build a docker-container which will build the firmware via:
+
+.. code-block:: shell-session
+
+   $ docker build -f docker/Dockerfile_fwbuild -t card10-firmware-builder .
+
+Now, you can start the container with the firmware directory mounted, which will build the
+firmware into the firmware/build directory:
+
+.. code-block:: shell-session
+
+   $ docker run -v $(pwd):/firmware card10-firmware-builder

Documentation/how-to-flash.rst

+How To Flash
+============
+Depending on whether you have a debugger or not, you have to use a different
+method of flashing:
+
+Flash Without Debugger
+----------------------
+
+If you do not have a debugger, you have to update the firmware using our
+bootloader by going into :ref:`usb_file_transfer`.
+
+After you get your badge into :ref:`usb_file_transfer`, you can drop the firmware's
+``.bin`` (from ``build/pycardium/pycardium_epicardium.bin`` in most cases) into
+this flash-storage.  You **must** call the file ``card10.bin`` for the
+bootloader to use it.
+
+The bootloader will then display ``Writing.`` in red while it is actually
+writing the file to external flash.  Please wait until it displays ``Ready.``
+again before resetting card10 by pressing the power button again.
+
+Flash Using Debugger
+--------------------
+
+First, setup everything as explained on the :ref:`debugger` page.  Following
+that and after connecting to card10, you can flash your binary using the
+``load`` command.  After loading, you need to use ``reset`` to reboot card10
+using your new firmware.
+
+.. warning::
+
+   With the current version of the bootloader, before attempting to flash using
+   the debugger, make sure there is no ``card10.bin`` stored on the device.
+   If there is, the bootloader will overwrite whatever you just flashed after
+   reboot every time.
+
+.. code-block:: text
+
+   (gdb) load
+   Loading section .text, size 0x12514 lma 0x10010000
+   Loading section .ARM.exidx, size 0x8 lma 0x10022514
+   Loading section .data, size 0x8d8 lma 0x1002251c
+   Start address 0x10012160, load size 77300
+   Transfer rate: 19 KB/sec, 11042 bytes/write.
+   (gdb)
+
+.. note::
+
+   If OpenOCD was able to connect, but GDB gives you an
+
+   .. code-block:: text
+
+      Error erasing flash with vFlashErase packet
+
+   error, issue a ``reset`` command, quickly followed by a ``load`` command.
+
+   Reason: The Epicardium puts parts of the CPU to sleep and the debugging
+   interface is part of that. After a reset the bootloader starts up
+   and lets OpenOCD/GDB take control again.

Documentation/index.rst

 card10 firmware docs
 ====================
 
-This is the API documentation for card10's default firmware.
+**Dear traveller,**
 
-Overview
---------
+these transcripts describe how you can write code for your card10.  This
+includes the Python modules that are available but also documentation of the
+lower level firmware components.
 
-The design roughly looks like this:
+If you want to write Python code for card10, you will want to take a look at
+the :ref:`Pycardium <pycardium_overview>` docs.  If you are interested in writing applications
+in other languages, you'll probably want to interface with
+:ref:`Epicardium API <epicardium_api_overview>` directly.
 
-.. image:: static/overview.svg
+Last but not least, if you want to start hacking the lower-level firmware, the
+:ref:`Firmware <firmware_overview>` section of these docs is a good starting place.
+
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Pycardium
+
+   pycardium/overview
+   pycardium/stdlib
+   pycardium/bhi160
+   pycardium/ble_hid
+   pycardium/bme680
+   pycardium/max30001
+   pycardium/max86150
+   pycardium/buttons
+   pycardium/color
+   pycardium/config
+   pycardium/display
+   pycardium/gpio
+   pycardium/leds
+   pycardium/light-sensor
+   pycardium/os
+   pycardium/personal_state
+   pycardium/png
+   pycardium/power
+   pycardium/pride
+   pycardium/simple_menu
+   pycardium/utime
+   pycardium/vibra
+   pycardium/ws2812
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Firmware
+
+   overview
+   card10-cfg
+   usb-file-transfer
+   how-to-build
+   how-to-flash
+   debugger
+   pycardium-guide
+   memorymap
+   epicardium/mutex
+   epicardium/sensor-streams
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Epicardium API
+
+   epicardium/overview
+   epicardium/api
+   epicardium-guide
+   epicardium/internal
 
 .. toctree::
-   :maxdepth: 2
-   :caption: Contents
+   :maxdepth: 1
+   :caption: Bluetooth
+
+   bluetooth/overview
+   bluetooth/ess
+   bluetooth/file-transfer
+   bluetooth/card10
+   bluetooth/ecg
+   bluetooth/nimble
+
+Indices and tables
+==================
 
-   epicardium
-   pycardium
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

Documentation/memorymap.rst

+Memory Map
+==========
+
+Flash::
+
+  | Address    | Usage              |
+  |------------|--------------------|
+  |0x100F FFFF |                    |
+  |            | Core 1: 512 kB     |
+  |0x1008 0000 |                    |
+  |------------|--------------------|
+  |0x1007 FFFF |                    |
+  |            | Core 0: 448 kB     |
+  |0x1001 0000 |                    |
+  |------------|--------------------|
+  |0x1000 FFFF |                    |
+  |            | Bootloader: 64 kB  |
+  |0x1000 0000 |                    |
+  |------------|--------------------|
+
+
+Ram::
+
+  | Address    | Usage                 |
+  |------------|-----------------------|
+  |0x2008 BFFF |                       |
+  |            | API Parameters: 48 kB |
+  |0x2008 0000 |                       |
+  |------------|-----------------------|
+  |0x2007 FFFF |                       |
+  |            | Core 1: 256 kB        |
+  |0x2004 0000 |                       |
+  |------------|-----------------------|
+  |0x2003 FFFF |                       |
+  |            | Core 0: 256 kB        |
+  |0x2000 0000 |                       |
+  |------------|-----------------------|
+
+External Flash::
+
+  | Address    | Usage                 |
+  |------------|-----------------------|
+  |0x0080 0000 |                       |
+  |            | FAT file system: 8 MB |
+  |0x0000 0000 |                       |
+  |------------|-----------------------|
+

Documentation/overview.rst

+.. _firmware_overview:
+
+Overview
+========
+To make the most of card10's dual-core processor, its firmware will have been
+divided into two parts: The "main" firmware running on core 0 which will have
+been called *Epicardium* and the "user-code" running on core 1.  In most cases
+this will have been *Pycardium*, our MicroPython port.
+
+.. image:: ./static/overview.svg
+
+Epicardium
+----------
+Epicardium is based on `FreeRTOS <https://www.freertos.org/>`_.  There are a
+number of tasks that will have been keeping card10 running.  These are:
+
++--------------------+-------------------------------+----------+-------------------------------------------+
+| Name               | ID Global                     | Priority | Description                               |
++====================+===============================+==========+===========================================+
+| `vPmicTask`_       | ``pmic_task_id`` (static)     | +4       | Power Management (and Reset Button)       |
++--------------------+-------------------------------+----------+-------------------------------------------+
+| `vLifecycleTask`_  | ``lifecycle_task`` (static)   | +3       | Control of the payload running on core 1. |
++--------------------+-------------------------------+----------+-------------------------------------------+
+| `vBleTask`_        | ``ble_task_id`` (static)      | +3       | Bluetooth Low Energy Stack                |
++--------------------+-------------------------------+----------+-------------------------------------------+
+| `vSerialTask`_     | ``serial_task_id``            | +3       | Serial Output via UART/CDC-ACM/BLE        |
++--------------------+-------------------------------+----------+-------------------------------------------+
+| `vApiDispatcher`_  | ``dispatcher_task_id``        | +2       | Epicardium API dispatcher                 |
++--------------------+-------------------------------+----------+-------------------------------------------+
+| `vInterruptsTask`_ | ``interrupts_task`` (static)  | +2       | Interrupt dispatcher worker               |
++--------------------+-------------------------------+----------+-------------------------------------------+
+| `vMAX30001Task`_   | ``max30001_task_id`` (static) | +1       | `MAX30001`_ ECG driver                    |
++--------------------+-------------------------------+----------+-------------------------------------------+
+| `vBhi160Task`_     | ``bhi160_task_id`` (static)   | +1       | `BHI160`_ sensor fusion driver            |
++--------------------+-------------------------------+----------+-------------------------------------------+
+
+.. _vPmicTask: https://git.card10.badge.events.ccc.de/card10/firmware/blob/master/epicardium/modules/pmic.c#L281
+.. _vLifecycleTask: https://git.card10.badge.events.ccc.de/card10/firmware/blob/master/epicardium/modules/lifecycle.c#L361
+.. _vBleTask: https://git.card10.badge.events.ccc.de/card10/firmware/blob/master/epicardium/ble/ble.c#L237
+.. _vSerialTask: https://git.card10.badge.events.ccc.de/card10/firmware/blob/master/epicardium/modules/serial.c#L289
+.. _vApiDispatcher: https://git.card10.badge.events.ccc.de/card10/firmware/blob/master/epicardium/modules/dispatcher.c#L25
+.. _vInterruptsTask: https://git.card10.badge.events.ccc.de/card10/firmware/blob/master/epicardium/modules/interrupts.c#L119
+.. _vLedTask: https://git.card10.badge.events.ccc.de/card10/firmware/blob/master/epicardium/modules/personal_state.c#L58
+.. _vMAX30001Task: https://git.card10.badge.events.ccc.de/card10/firmware/blob/master/epicardium/modules/max30001.c#L378
+.. _vBhi160Task: https://git.card10.badge.events.ccc.de/card10/firmware/blob/master/epicardium/modules/bhi.c#L419
+
+.. _MAX30001: https://www.maximintegrated.com/en/products/analog/data-converters/analog-front-end-ics/MAX30001.html
+.. _BHI160: https://www.bosch-sensortec.com/bst/products/all_products/bhi160
+
+Epicardium API
+--------------
+Epicardium exposes lots of functionality via the *Epicardium API*.  The
+technical details of this API can be found in this :ref:`overview
+<epicardium_api_overview>`.  If you are interested in adding new API calls,
+you should probably read the :ref:`epicardium_api_guide` guide.
+
+Pycardium
+---------
+
+Pycardium is our MicroPython fork.  Its purpose is to make it as easy as
+possible to interact with card10.  If you are interested in working on
+Pycardium, take a look at the :ref:`pycardium_guide` guide.
+
+.. _l0dables:
+
+l0dables
+--------
+
+Next to Pycardium, other bare-metal code can also run on core 1.  For example,
+a `Rustcardium`_ or C-cardium.  These l0dables must be compiled using our special
+linker script and should link against the api-caller library so they can
+interface with the :ref:`epicardium_api`.
+
+Note: this feature is disabled by default and has to be enabled in :ref:`card10_cfg`. A :ref:`card10_cfg` file dropped into the :ref:`usb_file_transfer` of the badge containing ``execute_elf = true`` is enough.
+
+l0dables are currently built within the source tree of the main repository. See |l0dables_blinky|_ for an example of a hello-world-like program. Within those programs, you can access the :ref:`epicardium_api` to control the hardware and behaviour of the badge.
+
+Once you have a built ELF file, you can drop it into the FAT filesystem of the flash (eg. via :ref:`usb_file_transfer`) and it will be available from the menu program of the badge.
+
+.. _Rustcardium: https://git.card10.badge.events.ccc.de/astro/rust-card10
+.. |l0dables_blinky| replace:: ``l0dables/blinky``
+.. _l0dables_blinky: https://git.card10.badge.events.ccc.de/card10/firmware/-/tree/master/l0dables
+
+Program Flow Diagram
+--------------------
+The following diagram is a rough overview of the program flow in this fimware:
+
+.. image:: ./static/firmware-flow.svg

Documentation/pycardium-guide.rst

+.. _pycardium_guide:
+
+Pycardium Module Development
+============================
+This is a step-by-step guide on augmenting Pycardium with new modules.  These
+module are either written in Python or in C:
+
+* **Python Modules**: Python modules live in ``pycardium/modules/py`` and are
+  pre-compiled and packed into Pycardium at build-time.  Python modules are
+  meant to be used for creating abstractions and helpers based on lower level
+  functionality.  The rationale is that it is easier to write pythonic modules
+  in python instead of having to wrap all that in C code.  Though care has to
+  be taken as these modules can eat quite a lot of space.
+
+* **C Modules**: C modules allow MicroPython code to interface with the rest of
+  the system or allow a much faster implementation of performance-critical
+  code. For interfacing with card10, these modules are mostly meant to make use
+  of the :ref:`Epicardium API <epicardium_api_overview>`.
+
+Python Modules
+--------------
+Adding a new Python module
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+Add a file in ``pycardium/modules/py``.  The name of the file dictates the name
+of the module.  Next, add you file to the list in
+``pycardium/modules/py/meson.build`` to make the build-system aware of it.
+Rebuild, and you should see your module pop up in the list shown by
+
+.. code-block:: python
+
+   help("modules")
+
+Size Consideration
+~~~~~~~~~~~~~~~~~~
+Python modules can get quite big.  If you want to know exact sizes, take a look at
+the output of
+
+.. code-block:: shell-session
+
+   ls -lh build/pycardium/*@@frozen.c@*/
+
+If your module contains some kind of big lookup-table or data-block, consider
+pushing that into external flash.  You can then read the data using standard
+Python ``open()`` and if you need to decode it, use the ``ustruct`` or ``ujson``
+module.
+
+Creating a new C module
+-----------------------
+Module Source
+~~~~~~~~~~~~~
+Create a new file in ``pycardium/modules``.  The basic structure of a C module
+looks like this:
+
+.. code-block:: c++
+
+   #include "py/obj.h"
+
+   /* Define a function in this module */
+   static mp_obj_t mp_example_hello(mp_obj_t number)
+   {
+           int num = mp_obj_get_int(number);
+           printf("Hello from example: %d\r\n", num);
+           return mp_const_none;
+   }
+   /* Create an object for this function */
+   static MP_DEFINE_CONST_FUN_OBJ_1(example_hello_obj, mp_example_hello);
+
+   /* The globals table for this module */
+   static const mp_rom_map_elem_t example_module_globals_table[] = {
+           {MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_example)},
+           {MP_ROM_QSTR(MP_QSTR_hello), MP_ROM_PTR(&example_hello_obj)},
+   };
+   static MP_DEFINE_CONST_DICT(example_module_globals, example_module_globals_table);
+
+   const mp_obj_module_t example_module = {
+           .base    = {&mp_type_module},
+           .globals = (mp_obj_dict_t*)&example_module_globals,
+   };
+
+   /* This is a special macro that will make MicroPython aware of this module */
+   /* clang-format off */
+   MP_REGISTER_MODULE(MP_QSTR_example, example_module, MODULE_EXAMPLE_ENABLED);
+
+.. note::
+
+   Please keep the ``/* clang-format off */``.  It ensures that *clang-format*
+   won't touch the following line.  This is necessary because MicroPython's
+   codegen scripts can'd deal with newlines in the ``MP_REGISTER_MODULE`` macro.
+
+Build Integration
+~~~~~~~~~~~~~~~~~
+Next, you need to add your module to ``pycardium/meson.build``. There is a list
+of module sources at the very top called ``modsrc`` where you need to add your
+file (e.g. ``modules/example.c``).
+
+QSTR Definitions
+~~~~~~~~~~~~~~~~
+If you now run ``ninja -C build/``, you will hit a few errors regarding missing
+QSTR definitions.  With the example module above, they will look similar to
+this:
+
+.. code-block:: text
+
+   ../pycardium/modules/example.c:15:46: error: 'MP_QSTR_example' undeclared here (not in a function)
+      15 |  {MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_example)},
+
+To fix these errors, you need to add all QSTRs your module needs to
+``pycardium/modules/qstrdefs.h``.  Add a section for your module where you
+define all QSTRs you need:
+
+.. code-block:: cpp
+
+   /* example */
+   Q(example)
+   Q(hello)
+
+Each ``Q(...)`` will define into a corresponding ``MP_QSTR_...``.  So
+``Q(example)`` corresponds to ``MP_QSTR_example``.
+
+Enable Module
+~~~~~~~~~~~~~
+The last step is to actually enable inclusion of your module into the firmware.
+Do this by adding a define in ``pycardium/mpconfigport.h``:
+
+.. code-block:: cpp
+
+   #define MODULE_EXAMPLE_ENABLED              (1)
+
+The name of the define is the one from the last line in the module source above.
+
+Wrapping Epicardium API
+-----------------------
+Most modules will probably make use of the :ref:`Epicardium API
+<epicardium_api_overview>`. Doing so does not require any extra work, you can
+just call the API from your module code.  You should check the input that your
+module got from MicroPython before sending data off to Epicardium.  For
+example, raise a ``ValueError`` if an integer is too big to fit into the type
+specified by the API.  You should also gracefully handle errors returned by API
+calls.  As most API calls use *errno* codes, you can just wrap them in an
+``OSError``:
+
+.. code-block:: cpp
+
+   int ret = epic_bma_get_accel(&values);
+
+   if (ret < 0) {
+          mp_raise_OSError(-ret);
+   }
+
+QSTRs
+-----
+QSTRs are so called “interned strings”. This means they are not allocated like
+normal python objects but instead live in flash and are indexed. This allow
+MicroPython to very efficiently use them as identifiers. According to them,
+comparing two QSTR is as fast as comparing integers.
+
+Unfortunately, the way these QSTRs are collected from the source files is quite
+weird.  MicroPython comes with a few python scripts (namely `makeqstrdefs.py`_
+and `makeqstrdata.py`_) that parse the C source files and search for uses of
+``MP_QSTR_*``.  These are then sorted and indexed into a header file called
+``qstrdefs.collected.h``. This is closely tied in with their Makefiles.
+
+As we use our own build system, we had to somehow wrap this generation to work
+for us as well. This is done using a few scripts in `lib/micropython`_.
+Currently, our build system does **not** parse the module sources to search for
+QSTRs.  Instead all QSTRs needed by modules need to be defined in the header
+``pycardium/modules/qstrdefs.h``.
+
+.. _makeqstrdefs.py: https://github.com/micropython/micropython/blob/master/py/makeqstrdefs.py
+.. _makeqstrdata.py: https://github.com/micropython/micropython/blob/master/py/makeqstrdata.py
+.. _lib/micropython: https://git.card10.badge.events.ccc.de/card10/firmware/tree/master/lib/micropython
+
+Functions for MicroPython
+-------------------------
+As shown in the example above, you can create functions that can be called from
+MicroPython code.  These functions always have one of the following signatures.
+To create a MicroPython object for a function, you need the macro call shown
+after each signature.  Please place these calls directly after the function
+body as shown above.
+
+.. code-block:: cpp
+
+   /* Function with 0 arguments */
+   mp_obj_t mp_example_fun0(void);
+   static MP_DEFINE_CONST_FUN_OBJ_0(example_fun0_obj, mp_example_fun0);
+
+   /* Function with 1 argument */
+   mp_obj_t mp_example_fun1(mp_obj_t arg0_in);
+   static MP_DEFINE_CONST_FUN_OBJ_1(example_fun1_obj, mp_example_fun0);
+
+   /* Function with 2 arguments */
+   mp_obj_t mp_example_fun2(mp_obj_t arg0_in, mp_obj_t arg1_in);
+   static MP_DEFINE_CONST_FUN_OBJ_2(example_fun2_obj, mp_example_fun0);
+
+   /* Function with 3 arguments */
+   mp_obj_t mp_example_fun3(mp_obj_t arg0_in, mp_obj_t arg1_in, mp_obj_t arg2_in);
+   static MP_DEFINE_CONST_FUN_OBJ_3(example_fun3_obj, mp_example_fun0);
+
+   /* Function with 4 or more arguments */
+   mp_obj_t mp_example_fun4(size_t n_args, mp_obj_t *args);
+   static MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN(example_fun4_obj, 4, 4, mp_example_fun4);
+
+For functions with 4 or more arguments, you need to use the
+``MP_DEFINE_CONST_FUN_OBJ_VAR_BETWEEN`` macro and instead of directly accessing
+the arguments, you get an array.  The macro gets two numbers (they are the same
+in the example above):  The minimum and maximum number of arguments.
+
+MicroPython Objects
+-------------------
+**TL;DR**: Look at |obj.h|_.  It contains most functions needed to create,
+access, and modify MicroPython objects.
+
+.. |obj.h| replace:: ``lib/micropython/micropython/py/obj.h``
+.. _obj.h: https://github.com/micropython/micropython/blob/master/py/obj.h
+
+For C modules to interface with MicroPython, you need to be able to interface
+with MicroPython objects.  The generic type of an object is ``mp_obj_t``.  As
+you can see in the example above, this is also what a function gets as
+arguments and returns back to MicroPython.  You can cast ``mp_obj_t`` to
+concrete object types if you made sure it is actually the correct type.
+
+Booleans
+~~~~~~~~
+``True`` and ``False`` are *const* objects, called ``mp_const_true`` and
+``mp_const_false`` respectively.  A usual equality check can be used:
+
+.. code-block:: cpp
+
+   #include "py/obj.h"
+   #include "py/runtime.h"
+
+   if (bool_obj == mp_const_true) {
+       /* is a boolean true */
+   } else if (bool_obj == mp_const_false) {
+       /* is a boolean false */
+   } else {
+       mp_raise_TypeError("arg 0 is not a boolean");
+   }
+
+Integers
+~~~~~~~~
+As long as your integers stay within **31**-bit limits, integers are stored very
+efficiently and can be accessed and created like this:
+
+.. code-block:: cpp
+
+   #include "py/obj.h"
+
+   /* Create a new integer which is < 2^31 */
+   mp_obj_t int_obj = MP_OBJ_NEW_SMALL_INT(0xc0ffee);
+
+   /* Check if an integer is small and if so, extract it */
+   if (mp_obj_is_small_int(int_obj)) {
+           int int_value = MP_OBJ_SMALL_INT_VALUE(int_obj);
+   }
+
+For bigger integers or if you are uncertain about the limits, use the following
+functions:
+
+.. code-block:: cpp
+
+   #include "py/obj.h"
+
+   /* Create new integer objects in various sizes and signedness */
+   mp_obj_t int0_obj = mp_obj_new_int((mp_int_t)value);
+   mp_obj_t int1_obj = mp_obj_new_int_from_uint((mp_uint_t)value);
+   mp_obj_t int2_obj = mp_obj_new_int_from_ll((long long)value);
+   mp_obj_t int3_obj = mp_obj_new_int_from_ull((unsigned long long)value);
+
+   /* Check if a value is an integer */
+   if (mp_obj_is_integer(int_obj)) {
+
+           /* Get an integer */
+           int int0 = mp_obj_get_int(int_obj);
+           int int0 = mp_obj_get_int_truncated(int_obj);
+   }
+
+   int value;
+   if (!mp_obj_get_int_maybe(int_obj, &value)) {
+      /* Not an integer! */
+   }
+
+To get really big integers you have to use
+
+.. code-block:: cpp
+
+   #include "py/obj.h"
+   #include "py/objint.h"
+
+   long long value;
+   mp_obj_int_to_bytes_impl(int_obj, 8, (byte*)&value);
+
+Strings
+~~~~~~~
+As in CPython, MicroPython also has multiple string representations:  There is
+``str``, ``bytes``, and ``bytearray`` but also the above mentions ``QSTR``\ s.
+Ideally, code should work with as many of these as possible and to ensure this,
+please use these generic functions:
+
+.. code-block:: cpp
+
+   #include "py/obj.h"
+
+   /* Create a new string object */
+   char buf[] = "Hello MicroPython!";
+   mp_obj_t str_obj = mp_obj_new_str(buf, sizeof(buf));
+
+   /* Check if an object is a string */
+   if (mp_obj_is_str(str_obj)) {
+           /* Either str or QSTR */
+   }
+   if (mp_obj_is_str_or_bytes) {
+           /* Either str, QSTR, or bytes */
+   }
+
+   /*
+    * Get a char array from a string object.
+    * CAUTION! This string is not necessarily null terminated!
+    */
+   char *str_data;
+   size_t str_len;
+   str_data = mp_obj_str_get_data(str_obj, &str_len);
+
+Lists & Tuples
+~~~~~~~~~~~~~~
+While lists and tuples can be accessed using type specific functions, you
+should refrain from doing so:  Use generic indexing functions instead, as they
+also allow types derived from lists or tuples or types with custom indexing
+implementations (i.e. duck-typing).
+
+.. code-block:: cpp
+
+   #include "py/obj.h"
+   #include "py/runtime.h"
+
+   /* Get the length of a list-like object */
+   mp_int_t len = mp_obj_get_int(mp_obj_len(list_obj));
+
+   /* Get element at a specific index */
+   mp_obj_t elem5 = mp_obj_subscr(
+           list_obj,
+           MP_OBJ_NEW_SMALL_INT(5),
+           MP_OBJ_SENTINEL
+   );
+
+To create a list or tuple:
+
+.. code-block:: cpp
+
+   mp_obj_t values[] = {
+           MP_OBJ_NEW_SMALL_INT(0xde),
+           MP_OBJ_NEW_SMALL_INT(0xad),
+           MP_OBJ_NEW_SMALL_INT(0xbe),
+           MP_OBJ_NEW_SMALL_INT(0xef),
+   };
+
+   /* Create a tuple */
+   mp_obj_t tuple = mp_obj_new_tuple(4, values);
+
+   /* Create a list */
+   mp_obj_t list = mp_obj_new_list(4, values);

Documentation/pycardium.rst

-Pycardium
-=========
-
-Pycardium is a core1 payload based on MicroPython.  It can interface with
-card10 using the Epicardium API, which is wrapped in a bunch of python modules.
-These modules are documented in this section.
-
-.. toctree::
-   :maxdepth: 1
-   :caption: Modules:
-
-   pycardium/color
-   pycardium/leds