*: use plain ESP-IDF cmake machinery

This refactors the build system to be more in line with the average
ESP-IDF project.

This means that:
  1. `idf.py` from the main directory now works
  2. Adding components, running menuconfig, etc. should now be trivial.
  3. We don't have to edit files in micropython/ports/esp32/....

Hopefully this is the first step to yeeting the micropython checkout
from source and just depending on a submodule instead.

CMakeLists.txt

+cmake_minimum_required(VERSION 3.12)
+
+set(IDF_TARGET esp32s3)
+
+include($ENV{IDF_PATH}/tools/cmake/project.cmake)
+
+# TODO(q3k): migrate badge23/components/* to components/*.
+list(APPEND EXTRA_COMPONENT_DIRS "${CMAKE_CURRENT_SOURCE_DIR}/badge23/components/")
+
+project(badge23)

README.md

-## current functionality
-micropython repl hooked up, hw functionality partly broken
+## Current functionality
+
+Micropython repl hooked up, hw functionality partly broken>
+
+Some fun commands to try:
 
-some fun commands to try:
 ```
 import hardware
 #turn on sound
@@ -25,18 +27,19 @@ hardware.count_sources();
 #...don't know how to hook up gc to __del__, maybe wrong approach
 ```
 
-files can be transferred with mpremote, such as:
+Files can be transferred with mpremote, such as:
+
 ```
 mpremote fs cp python_payload/boot.py :boot.py
 mpremote fs cp python_payload/cap_touch_demo.py :cap_touch_demo.py
 ```
 
-## how to install dependencies
+## How to install dependencies
 
 ### Generic
 
 1. install esp-idf v4.4:
-(copied from https://www.wemos.cc/en/latest/tutorials/others/build_micropython_esp32.html)
+(copied from https://www.wemos.cc/en/latest/tutorials/others/build\_micropython\_esp32.html)
 ```
 $ cd ~
 $ git clone https://github.com/espressif/esp-idf.git
@@ -57,48 +60,62 @@ you need to run it in every new terminal and adding it to autostart did bother u
 $ nix-shell nix/shell.nix
 ```
 
-## how to build
+## How to build and flash
+
+Standard ESP-IDF project machinery present and working. You can run `idf.py` from the git checkout and things should just work.
+
+### Building
 
-1. prepare build
-```
-$ cd micropython/
-$ make -C mpy-cross
-$ cd ports/esp32
-$ make submodules
-```
-2. build/flash
-make sure esp-idf is sourced as in step 1 and that you are in micropython/ports/esp32
-build:
 ```
-$ make
+$ idf.py build
 ```
-flash + build: (put badge into bootloader mode*)
+
+### Flashing
+
+Put badge into bootloader mode by holding left should button down during boot.
+
 ```
-$ make deploy PORT=/dev/ttyACM0
+$ idf.py -p /dev/ttyACM0 flash
 ```
-_*press right shoulder button down during boot (on modified "last gen" prototypes)_
 
-empty build cache (useful when moving files around):
+You can skip `-p /dev/ttyACM0` if you set the environment variable `ESPPORT=/dev/ttyACM0`. This environment variable is also set by default when using Nix.
+
+After flashing, remember to powercycle your badge to get it into the user application.
+
+### Accessing MicroPython REPL:
+
 ```
-$ make clean
+$ picocom -b 115200 /dev/ttyACM0
+$ # or
+$ screen /dev/ttyACM0
+$ # or (will eat newlines in REPL, though)
+$ idf.py -p /dev/ttyACM0 monitor
 ```
 
-3. access micropython repl:
+### Use CMake
+
+`idf.py` calls cmake under the hood for most operations. If you dislike using wrappers you can do the same work yourself:
 
 ```
-$ picocom -b 115200 /dev/ttyACM0
-$ # OR
-# screen /dev/ttyACM0
+mkdir build
+cd build
+cmake .. -G Ninja
+ninja
 ```
 
-## how to modify
+There's `flash/monitor` targets, too (but no openocd/gdb...). To pick what port to flash to/monitor, set the ESPPORT environment variable.
+
+## How to modify
+
+### General info
+
+Global + micropython entry point: app\_main() in micropython/ports/esp32/main.c (includes badge23/espan.h).
+
+C entry point, called by^: os\_app\_main() in badge23/espan.c
 
-### general info
+Register new c files for compilation: add to set(BADGE23\_SOURCE) in main/CMakelists.txt
 
-global + micropython entry point: app_main() in micropython/ports/esp32/main.c (includes badge23/espan.h)
-c entry point, called by^: os_app_main() in badge23/espan.c
-register new c files for compilation: add to set(BADGE23_LIB) in micropython/ports/esp32/main/CMakelists.txt
-change output volume in the set_global_vol_dB(int8_t) call; -90 for mute
+Change output volume in the set\_global\_vol\_dB(int8\_t) call; -90 for mute
 
 ### Debugging
 
@@ -120,7 +137,7 @@ static const char *TAG = "misery";
 ESP_LOGI(TAG, "i love C");
 ```
 
-However, this will **only work** if you modify `micropython/ports/esp32/boards/sdkconfig.base` to set `CONFIG_LOG_DEFAULT_LEVEL_INFO=y` (which will likely break programs interacting with micropython REPL as many components of the ESP-IDF will suddenly become very chatty). But that's fine for troubleshooting some C-land bugs.
+However, this will **only work** if you first set `CONFIG_LOG_DEFAULT_LEVEL_INFO=y` (which will likely break programs interacting with micropython REPL as many components of the ESP-IDF will suddenly become very chatty). But that's fine for troubleshooting some C-land bugs.
 
 If you want to only log errors or just add temporary logs, use `ESP_LOGE` instead, which will always print to the USB console.
 
@@ -130,26 +147,28 @@ If you want to only log errors or just add temporary logs, use `ESP_LOGE` instea
 
 First, make sure your badge is running in application mode (not bootloader mode! that will stay in bootloader mode).
 
+Then, start OpenOCD:
+
 ```
-$ make -C micropython/ports/esp32 openocd
+$ OPENOCD_COMMANDS="-f board/esp32s3-builtin.cfg" idf.py openocd
 ```
 
+(you can skip setting `OPENOCD_COMMANDS` if you're using Nix)
+
 Then, in another terminal:
 
 ```
-$ make -C micropython/ports/esp32 gdb
+$ idf.py gdb
 ```
 
-Good luck.
+Good luck. The idf.py gdb/openocd scripts seem somewhat buggy.
 
 ### ESP-IDF functionality
 
-Micropython splits up sdkconfig into a bunch of small files. Unfortunately, that doesn't work well with ESP-IDFs menuconfig.
-
-If you want to permanently toggle options, you'll have to do that by modifying `micropython/ports/esp32/boards/sdkconfig.badge23`, or another file referenced by `micropython/ports/esp32/GENERIC_S3_BADGE/mpconfigboard.cmake`. Keep in mind the usual Kconfig shenanigants: disabling an options requires removing its line (not setting it to 'n'!), and that some options might be enabled automatically by other options. After modifying files, you must `make clean` in micropython/ports/esp32. Bummer.
-
-To verify whether your configuration is what you expect it to be, you can still run menuconfig on the effective/calculated sdkconfig that micropython assembles. You can even modify the settings which will affect the build, but will be lost on `make clean` (and on git push, of course). To do so, run:
+Currently we have one large sdkconfig file. To modify it, run:
 
 ```
-$ make -C micropython/ports/esp32 menuconfig
-```
\ No newline at end of file
+$ idf.py menuconfig
+```
+
+TODO(q3k): split into defaults

main/CMakeLists.txt

+# This copies micropython/ports/esp32/CMakeLists.txt but modifies it to remove
+# the concept of 'boards' as it's not conducive to a sensible
+# directory/project structure for our badge project:
+#
+# 1. We don't want to have stuff deeply inside micropython/ports/esp/boards.
+#    Ideally, the micropython directory will be a submodule that's tracking
+#    upstream without any modifications.
+# 2. We don't want to have to call cmake with -DMICROPY_BOARD. Calling plain
+#    `cmake` or `idf.py` should just work and do the thing we want.
+# 3. We don't target multiple boards anyways, so all that board functionality
+#    is just dead weight.
+
+set(MICROPY_DIR "${PROJECT_DIR}/micropython")
+set(MICROPY_PORT_DIR "${MICROPY_DIR}/ports/esp32")
+set(MICROPY_QSTRDEFS_PORT "${MICROPY_PORT_DIR}/qstrdefsport.h")
+
+include("${MICROPY_DIR}/py/py.cmake")
+
+if(NOT CMAKE_BUILD_EARLY_EXPANSION)
+    # Enable extmod components that will be configured by extmod.cmake.
+    # A board may also have enabled additional components.
+    set(MICROPY_PY_BTREE ON)
+
+    include(${MICROPY_DIR}/py/usermod.cmake)
+    include(${MICROPY_DIR}/extmod/extmod.cmake)
+endif()
+
+set(MICROPY_SOURCE_SHARED
+    ${MICROPY_DIR}/shared/readline/readline.c
+    ${MICROPY_DIR}/shared/netutils/netutils.c
+    ${MICROPY_DIR}/shared/timeutils/timeutils.c
+    ${MICROPY_DIR}/shared/runtime/interrupt_char.c
+    ${MICROPY_DIR}/shared/runtime/stdout_helpers.c
+    ${MICROPY_DIR}/shared/runtime/sys_stdio_mphal.c
+    ${MICROPY_DIR}/shared/runtime/pyexec.c
+)
+
+set(MICROPY_SOURCE_LIB
+    ${MICROPY_DIR}/lib/littlefs/lfs1.c
+    ${MICROPY_DIR}/lib/littlefs/lfs1_util.c
+    ${MICROPY_DIR}/lib/littlefs/lfs2.c
+    ${MICROPY_DIR}/lib/littlefs/lfs2_util.c
+    ${MICROPY_DIR}/lib/mbedtls_errors/mp_mbedtls_errors.c
+    ${MICROPY_DIR}/lib/oofatfs/ff.c
+    ${MICROPY_DIR}/lib/oofatfs/ffunicode.c
+)
+
+set(MICROPY_SOURCE_DRIVERS
+    ${MICROPY_DIR}/drivers/bus/softspi.c
+    ${MICROPY_DIR}/drivers/dht/dht.c
+)
+
+set(MICROPY_SOURCE_PORT
+    ${MICROPY_PORT_DIR}/badge23_mp_hardware.c
+    ${MICROPY_PORT_DIR}/badge23_mp_synth.c
+    ${MICROPY_PORT_DIR}/main.c
+    ${MICROPY_PORT_DIR}/uart.c
+    ${MICROPY_PORT_DIR}/usb.c
+    ${MICROPY_PORT_DIR}/usb_serial_jtag.c
+    ${MICROPY_PORT_DIR}/gccollect.c
+    ${MICROPY_PORT_DIR}/mphalport.c
+    ${MICROPY_PORT_DIR}/fatfs_port.c
+    ${MICROPY_PORT_DIR}/help.c
+    ${MICROPY_PORT_DIR}/modutime.c
+    ${MICROPY_PORT_DIR}/machine_bitstream.c
+    ${MICROPY_PORT_DIR}/machine_timer.c
+    ${MICROPY_PORT_DIR}/machine_pin.c
+    ${MICROPY_PORT_DIR}/machine_touchpad.c
+    ${MICROPY_PORT_DIR}/machine_adc.c
+    ${MICROPY_PORT_DIR}/machine_adcblock.c
+    ${MICROPY_PORT_DIR}/machine_dac.c
+    ${MICROPY_PORT_DIR}/machine_i2c.c
+    ${MICROPY_PORT_DIR}/machine_i2s.c
+    ${MICROPY_PORT_DIR}/machine_uart.c
+    ${MICROPY_PORT_DIR}/modmachine.c
+    ${MICROPY_PORT_DIR}/network_common.c
+    ${MICROPY_PORT_DIR}/network_lan.c
+    ${MICROPY_PORT_DIR}/network_ppp.c
+    ${MICROPY_PORT_DIR}/network_wlan.c
+    ${MICROPY_PORT_DIR}/mpnimbleport.c
+    ${MICROPY_PORT_DIR}/modsocket.c
+    ${MICROPY_PORT_DIR}/modesp.c
+    ${MICROPY_PORT_DIR}/esp32_nvs.c
+    ${MICROPY_PORT_DIR}/esp32_partition.c
+    ${MICROPY_PORT_DIR}/esp32_rmt.c
+    ${MICROPY_PORT_DIR}/esp32_ulp.c
+    ${MICROPY_PORT_DIR}/modesp32.c
+    ${MICROPY_PORT_DIR}/machine_hw_spi.c
+    ${MICROPY_PORT_DIR}/machine_wdt.c
+    ${MICROPY_PORT_DIR}/mpthreadport.c
+    ${MICROPY_PORT_DIR}/machine_rtc.c
+    ${MICROPY_PORT_DIR}/machine_sdcard.c
+)
+
+set(MICROPY_SOURCE_QSTR
+    "${MICROPY_SOURCE_PY}"
+    "${MICROPY_SOURCE_EXTMOD}"
+    "${MICROPY_SOURCE_USERMOD}"
+    "${MICROPY_SOURCE_SHARED}"
+    "${MICROPY_SOURCE_LIB}"
+    "${MICROPY_SOURCE_PORT}"
+)
+
+set(IDF_COMPONENTS
+    app_update
+    bootloader_support
+    bt
+    driver
+    esp_adc_cal
+    esp_common
+    esp_eth
+    esp_event
+    esp_hw_support
+    esp_ringbuf
+    esp_rom
+    esp_pm
+    esp_wifi
+    esp_system
+    esp_timer
+    esp_netif
+    esp32s3
+    freertos
+    hal
+    heap
+    log
+    lwip
+    mbedtls
+    mdns
+    newlib
+    nvs_flash
+    sdmmc
+    soc
+    spi_flash
+    tcpip_adapter
+    ulp
+    vfs
+    xtensa
+)
+
+set(MICROPY_FROZEN_MANIFEST ${MICROPY_PORT_DIR}/boards/manifest.py)
+set(MICROPY_CROSS_FLAGS -march=xtensawin)
+
+# Include badge23/* via this component for now.
+# TODO(q3k): make this into a real component.
+set(BADGE23_SOURCE
+    ${PROJECT_DIR}/badge23/apa102LEDStrip.c
+    ${PROJECT_DIR}/badge23/audio.c
+    ${PROJECT_DIR}/badge23/captouch.c
+    ${PROJECT_DIR}/badge23/display.c
+    ${PROJECT_DIR}/badge23/espan.c
+    ${PROJECT_DIR}/badge23/leds.c
+    ${PROJECT_DIR}/badge23/scope.c
+    ${PROJECT_DIR}/badge23/synth.c
+    ${PROJECT_DIR}/badge23/components/gc9a01/gc9a01.c
+)
+
+idf_component_register(
+    SRCS
+        ${MICROPY_SOURCE_PY}
+        ${MICROPY_SOURCE_EXTMOD}
+        ${MICROPY_SOURCE_USERMOD}
+        ${MICROPY_SOURCE_SHARED}
+        ${MICROPY_SOURCE_LIB}
+        ${MICROPY_SOURCE_DRIVERS}
+        ${MICROPY_SOURCE_PORT}
+        ${BADGE23_SOURCE}
+    INCLUDE_DIRS
+        # Actual micropython include paths.
+        ${MICROPY_INC_CORE}
+        ${MICROPY_INC_USERMOD}
+        ${MICROPY_PORT_DIR}
+        # Needed for genhdr/* which for some reason is placed directly into
+        # the output dir. Gross.
+        ${CMAKE_BINARY_DIR}
+        # Needed for include/mpconfigboard.h.
+        "include"
+    REQUIRES
+        ${IDF_COMPONENTS}
+)
+
+# micropython machinery uses this to inspect include directories and compile
+# definitions. We're effectively looping back MICROPY_{SOURCE/INC} through
+# COMPONENT_TARGET (as generated by idf_component_register) back into
+# micropython.
+set(MICROPY_TARGET ${COMPONENT_TARGET})
+
+# Feed information from IDF component targets into micropy qstr machinery.
+foreach(comp ${IDF_COMPONENTS})
+    micropy_gather_target_properties(__idf_${comp})
+endforeach()
+
+# micropython/qstr wants to intern strings from nimble private headers. Tell
+# them how to find them.
+target_include_directories(${COMPONENT_TARGET} PUBLIC
+    "${IDF_PATH}/components/bt/host/nimble/nimble"
+)
+
+target_compile_definitions(${COMPONENT_TARGET} PUBLIC
+    # micropython includes FatFS which is configurable via its own
+    # configuration file, which is expected to be dfined as FFCONF_H.
+    # micropython also ships such a configuration file. I don't know why
+    # micropython doesn't just hardcode this...
+    FFCONF_H=\"${MICROPY_OOFATFS_DIR}/ffconf.h\"
+)
+
+include("${MICROPY_DIR}/py/mkrules.cmake")

main/include/mpconfigboard.h

+#define MICROPY_HW_BOARD_NAME "badge23"
+#define MICROPY_HW_MCU_NAME "ESP32S3"
+
+#define MICROPY_ENABLE_FINALISER (1)
+#define MICROPY_PY_MACHINE_DAC (0)
+#define MICROPY_HW_I2C0_SCL (9)
+#define MICROPY_HW_I2C0_SDA (8)
+
+#define MICROPY_ESP_IDF_4 1
+#define MICROPY_VFS_FAT 1
+#define MICROPY_VFS_LFS2 1
+
+// These kinda freak me out, but that's what micropython does...
+#define LFS1_NO_MALLOC
+#define LFS1_NO_DEBUG
+#define LFS1_NO_WARN
+#define LFS1_NO_ERROR
+#define LFS1_NO_ASSERT
+#define LFS2_NO_MALLOC
+#define LFS2_NO_DEBUG
+#define LFS2_NO_WARN
+#define LFS2_NO_ERROR
+#define LFS2_NO_ASSERT

nix/shell.nix

@@ -26,5 +26,9 @@ in with nixpkgs; pkgs.mkShell {
     export OPENOCD_SCRIPTS="${pkgs.openocd-esp32-bin}/share/openocd/scripts"
     # For GDB to be able to find libpython2.7 (????).
     export LD_LIBRARY_PATH="${pkgs.python2}/lib:$LD_LIBRARY_PATH"
+
+    # Some nice-to-have defaults.
+    export ESPPORT=/dev/ttyACM0
+    export OPENOCD_COMMANDS="-f board/esp32s3-builtin.cfg"
   '';
 }