Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found
Select Git revision
  • 9Rmain
  • anon/gpndemo
  • anon/update-sim
  • anon/webflasher
  • audio_input
  • audio_io
  • bl00mbox
  • bl00mbox_old
  • captouch-threshold
  • ch3/bl00mbox_docs
  • ci-1690580595
  • compressor
  • dev_p4
  • dev_p4-iggy
  • dev_p4-iggy-rebased
  • dos
  • dos-main-patch-50543
  • events
  • fm_fix
  • fm_fix2
  • fpletz/flake
  • history-rewrite
  • icon-flower
  • iggy/stemming
  • iggy/stemming_merge
  • json-error
  • main
  • main+schneider
  • media-buf
  • micropython_api
  • moon2_applications
  • moon2_demo_temp
  • moon2_gay_drums
  • passthrough
  • phhw
  • pippin/display-python-errors-on-display
  • pippin/make_empty_drawlists_skip_render_and_blit
  • pippin/media_framework
  • pippin/uhm_flash_access_bust
  • pressable_bugfix
  • q3k/doom-poc
  • rahix/big-flow3r
  • rahix/flow3rseeds
  • raw_captouch_new
  • raw_captouch_old
  • release/1.0.0
  • release/1.1.0
  • release/1.1.1
  • rev4_micropython
  • schneider/application-remove-name
  • schneider/bhi581
  • schneider/factory_test
  • schneider/recovery
  • scope
  • scope_hack
  • sdkconfig-spiram-tinyusb
  • sec/auto-nick
  • sec/blinky
  • simtest
  • slewtest
  • t
  • test
  • test2
  • uctx-wip
  • view-think
  • vm-pending
  • vsync
  • wave
  • wip-docs
  • wip-tinyusb
  • v1.0.0
  • v1.0.0+rc1
  • v1.0.0+rc2
  • v1.0.0+rc3
  • v1.0.0+rc4
  • v1.0.0+rc5
  • v1.0.0+rc6
  • v1.1.0
  • v1.1.0+rc1
  • v1.1.1
  • v1.2.0
  • v1.2.0+rc1
  • v1.3.0
83 results

Target

Select target project
  • flow3r/flow3r-firmware
  • Vespasian/flow3r-firmware
  • alxndr42/flow3r-firmware
  • pl/flow3r-firmware
  • Kari/flow3r-firmware
  • raimue/flow3r-firmware
  • grandchild/flow3r-firmware
  • mu5tach3/flow3r-firmware
  • Nervengift/flow3r-firmware
  • arachnist/flow3r-firmware
  • TheNewCivilian/flow3r-firmware
  • alibi/flow3r-firmware
  • manuel_v/flow3r-firmware
  • xeniter/flow3r-firmware
  • maxbachmann/flow3r-firmware
  • yGifoom/flow3r-firmware
  • istobic/flow3r-firmware
  • EiNSTeiN_/flow3r-firmware
  • gnudalf/flow3r-firmware
  • 999eagle/flow3r-firmware
  • toerb/flow3r-firmware
  • pandark/flow3r-firmware
  • teal/flow3r-firmware
  • x42/flow3r-firmware
  • alufers/flow3r-firmware
  • dos/flow3r-firmware
  • yrlf/flow3r-firmware
  • LuKaRo/flow3r-firmware
  • ThomasElRubio/flow3r-firmware
  • ai/flow3r-firmware
  • T_X/flow3r-firmware
  • highTower/flow3r-firmware
  • beanieboi/flow3r-firmware
  • Woazboat/flow3r-firmware
  • gooniesbro/flow3r-firmware
  • marvino/flow3r-firmware
  • kressnerd/flow3r-firmware
  • quazgar/flow3r-firmware
  • aoid/flow3r-firmware
  • jkj/flow3r-firmware
  • naomi/flow3r-firmware
41 results
Select Git revision
  • 89-apps-should-be-able-to-specify-if-they-want-wifi-to-be-disabled-when-entering-them
  • 9Rmain
  • allow-reloading-sunmenu
  • always-have-a-wifi-instance
  • anon/gpndemo
  • anon/update-sim
  • anon/webflasher
  • app_text_viewer
  • audio_input
  • audio_io
  • blm_dev_chan
  • ch3/bl00mbox_docs
  • ci-1690580595
  • dev_p4
  • dev_p4-iggy
  • dev_p4-iggy-rebased
  • dx/dldldld
  • dx/fb-save-restore
  • dx/hint-hint
  • dx/jacksense-headset-mic-only
  • events
  • fil3s-limit-filesize
  • fil3s-media
  • fpletz/flake
  • gr33nhouse-improvements
  • history-rewrite
  • icon-flower
  • iggy/stemming
  • iggy/stemming_merge
  • led_fix_fix
  • main
  • main+schneider
  • media_has_video_has_audio
  • micropython_api
  • mixer2
  • moon2_demo_temp
  • moon2_migrate_apps
  • more-accurate-battery
  • pippin/ctx_sprite_sheet_support
  • pippin/display-python-errors-on-display
  • pippin/make_empty_drawlists_skip_render_and_blit
  • pippin/more-accurate-battery
  • pippin/tcp_redirect_hack
  • pippin/tune_ctx_config_update_from_upstream
  • pippin/uhm_flash_access_bust
  • pressable_bugfix
  • py_only_update_fps_overlay_when_changing
  • q3k/doom-poc
  • q3k/render-to-texture
  • rahix/flow3rseeds
  • raw_captouch_new
  • raw_captouch_old
  • release/1.0.0
  • release/1.1.0
  • release/1.1.1
  • release/1.2.0
  • release/1.3.0
  • release/1.4.0
  • restore_blit
  • return_of_melodic_demo
  • rev4_micropython
  • schneider/application-remove-name
  • schneider/bhi581
  • schneider/factory_test
  • schneider/recovery
  • scope_hack
  • sdkconfig-spiram-tinyusb
  • sec/auto-nick
  • sec/blinky
  • sector_size_512
  • shoegaze-fps
  • smaller_gradient_lut
  • store_delta_ms_and_ins_as_class_members
  • task_cleanup
  • uctx-wip
  • w1f1-in-sim
  • widgets_draw
  • wifi-json-error-handling
  • wip-docs
  • wip-tinyusb
  • v1.0.0
  • v1.0.0+rc1
  • v1.0.0+rc2
  • v1.0.0+rc3
  • v1.0.0+rc4
  • v1.0.0+rc5
  • v1.0.0+rc6
  • v1.1.0
  • v1.1.0+rc1
  • v1.1.1
  • v1.2.0
  • v1.2.0+rc1
  • v1.3.0
  • v1.4.0
94 results
Show changes
Showing
with 2218 additions and 0 deletions
micropython/docs/develop/img/collision.png

3.38 KiB

micropython/docs/develop/img/linprob.png

3.57 KiB

MicroPython Internals
=====================
This chapter covers a tour of MicroPython from the perspective of a developer, contributing
to MicroPython. It acts as a comprehensive resource on the implementation details of MicroPython
for both novice and expert contributors.
Development around MicroPython usually involves modifying the core runtime, porting or
maintaining a new library. This guide describes at great depth, the implementation
details of MicroPython including a getting started guide, compiler internals, porting
MicroPython to a new platform and implementing a core MicroPython library.
.. toctree::
:maxdepth: 3
gettingstarted.rst
writingtests.rst
compiler.rst
memorymgt.rst
library.rst
optimizations.rst
qstr.rst
maps.rst
publiccapi.rst
extendingmicropython.rst
porting.rst
.. _internals_library:
Implementing a Module
=====================
This chapter details how to implement a core module in MicroPython.
MicroPython modules can be one of the following:
- Built-in module: A general module that is be part of the MicroPython repository.
- User module: A module that is useful for your specific project that you maintain
in your own repository or private codebase.
- Dynamic module: A module that can be deployed and imported at runtime to your device.
A module in MicroPython can be implemented in one of the following locations:
- py/: A core library that mirrors core CPython functionality.
- extmod/: A CPython or MicroPython-specific module that is shared across multiple ports.
- ports/<port>/: A port-specific module.
.. note::
This chapter describes modules implemented in ``py/`` or core modules.
See :ref:`extendingmicropython` for details on implementing an external module.
For details on port-specific modules, see :ref:`porting_to_a_board`.
Implementing a core module
--------------------------
Like CPython, MicroPython has core builtin modules that can be accessed through import statements.
An example is the ``gc`` module discussed in :ref:`memorymanagement`.
.. code-block:: bash
>>> import gc
>>> gc.enable()
>>>
MicroPython has several other builtin standard/core modules like ``io``, ``array`` etc.
Adding a new core module involves several modifications.
First, create the ``C`` file in the ``py/`` directory. In this example we are adding a
hypothetical new module ``subsystem`` in the file ``modsubsystem.c``:
.. code-block:: c
#include "py/builtin.h"
#include "py/runtime.h"
#if MICROPY_PY_SUBSYSTEM
// info()
STATIC mp_obj_t py_subsystem_info(void) {
return MP_OBJ_NEW_SMALL_INT(42);
}
MP_DEFINE_CONST_FUN_OBJ_0(subsystem_info_obj, py_subsystem_info);
STATIC const mp_rom_map_elem_t mp_module_subsystem_globals_table[] = {
{ MP_ROM_QSTR(MP_QSTR___name__), MP_ROM_QSTR(MP_QSTR_subsystem) },
{ MP_ROM_QSTR(MP_QSTR_info), MP_ROM_PTR(&subsystem_info_obj) },
};
STATIC MP_DEFINE_CONST_DICT(mp_module_subsystem_globals, mp_module_subsystem_globals_table);
const mp_obj_module_t mp_module_subsystem = {
.base = { &mp_type_module },
.globals = (mp_obj_dict_t *)&mp_module_subsystem_globals,
};
MP_REGISTER_MODULE(MP_QSTR_subsystem, mp_module_subsystem);
#endif
The implementation includes a definition of all functions related to the module and adds the
functions to the module's global table in ``mp_module_subsystem_globals_table``. It also
creates the module object with ``mp_module_subsystem``. The module is then registered with
the wider system via the ``MP_REGISTER_MODULE`` macro.
After building and running the modified MicroPython, the module should now be importable:
.. code-block:: bash
>>> import subsystem
>>> subsystem.info()
42
>>>
Our ``info()`` function currently returns just a single number but can be extended
to do anything. Similarly, more functions can be added to this new module.
.. _maps:
Maps and Dictionaries
=====================
MicroPython dictionaries and maps use techniques called open addressing and linear probing.
This chapter details both of these methods.
Open addressing
---------------
`Open addressing <https://en.wikipedia.org/wiki/Open_addressing>`_ is used to resolve collisions.
Collisions are very common occurrences and happen when two items happen to hash to the same
slot or location. For example, given a hash setup as this:
.. image:: img/collision.png
If there is a request to fill slot ``0`` with ``70``, since the slot ``0`` is not empty, open addressing
finds the next available slot in the dictionary to service this request. This sequential search for an alternate
location is called *probing*. There are several sequence probing algorithms but MicroPython uses
linear probing that is described in the next section.
Linear probing
--------------
Linear probing is one of the methods for finding an available address or slot in a dictionary. In MicroPython,
it is used with open addressing. To service the request described above, unlike other probing algorithms,
linear probing assumes a fixed interval of ``1`` between probes. The request will therefore be serviced by
placing the item in the next free slot which is slot ``4`` in our example:
.. image:: img/linprob.png
The same methods i.e open addressing and linear probing are used to search for an item in a dictionary.
Assume we want to search for the data item ``33``. The computed hash value will be 2. Looking at slot 2
reveals ``33``, at this point, we return ``True``. Searching for ``70`` is quite different as there was a
collision at the time of insertion. Therefore computing the hash value is ``0`` which is currently
holding ``44``. Instead of simply returning ``False``, we perform a sequential search starting at point
``1`` until the item ``70`` is found or we encounter a free slot. This is the general way of performing
look-ups in hashes:
.. code-block:: c
// not yet found, keep searching in this table
pos = (pos + 1) % set->alloc;
if (pos == start_pos) {
// search got back to starting position, so index is not in table
if (lookup_kind & MP_MAP_LOOKUP_ADD_IF_NOT_FOUND) {
if (avail_slot != NULL) {
// there was an available slot, so use that
set->used++;
*avail_slot = index;
return index;
} else {
// not enough room in table, rehash it
mp_set_rehash(set);
// restart the search for the new element
start_pos = pos = hash % set->alloc;
}
}
} else {
return MP_OBJ_NULL;
}
.. _memorymanagement:
Memory Management
=================
Unlike programming languages such as C/C++, MicroPython hides memory management
details from the developer by supporting automatic memory management.
Automatic memory management is a technique used by operating systems or applications to automatically manage
the allocation and deallocation of memory. This eliminates challenges such as forgetting to
free the memory allocated to an object. Automatic memory management also avoids the critical issue of using memory
that is already released. Automatic memory management takes many forms, one of them being
garbage collection (GC).
The garbage collector usually has two responsibilities;
#. Allocate new objects in available memory.
#. Free unused memory.
There are many GC algorithms but MicroPython uses the
`Mark and Sweep <https://en.wikipedia.org/wiki/Tracing_garbage_collection#Basic_algorithm>`_
policy for managing memory. This algorithm has a mark phase that traverses the heap marking all
live objects while the sweep phase goes through the heap reclaiming all unmarked objects.
Garbage collection functionality in MicroPython is available through the ``gc`` built-in
module:
.. code-block:: bash
>>> x = 5
>>> x
5
>>> import gc
>>> gc.enable()
>>> gc.mem_alloc()
1312
>>> gc.mem_free()
2071392
>>> gc.collect()
19
>>> gc.disable()
>>>
Even when ``gc.disable()`` is invoked, collection can be triggered with ``gc.collect()``.
The object model
----------------
All MicroPython objects are referred to by the ``mp_obj_t`` data type.
This is usually word-sized (i.e. the same size as a pointer on the target architecture),
and can be typically 32-bit (STM32, nRF, ESP32, Unix x86) or 64-bit (Unix x64).
It can also be greater than a word-size for certain object representations, for
example ``OBJ_REPR_D`` has a 64-bit sized ``mp_obj_t`` on a 32-bit architecture.
An ``mp_obj_t`` represents a MicroPython object, for example an integer, float, type, dict or
class instance. Some objects, like booleans and small integers, have their value stored directly
in the ``mp_obj_t`` value and do not require additional memory. Other objects have their value
store elsewhere in memory (for example on the garbage-collected heap) and their ``mp_obj_t`` contains
a pointer to that memory. A portion of ``mp_obj_t`` is the tag which tells what type of object it is.
See ``py/mpconfig.h`` for the specific details of the available representations.
**Pointer tagging**
Because pointers are word-aligned, when they are stored in an ``mp_obj_t`` the
lower bits of this object handle will be zero. For example on a 32-bit architecture
the lower 2 bits will be zero:
``********|********|********|******00``
These bits are reserved for purposes of storing a tag. The tag stores extra information as
opposed to introducing a new field to store that information in the object, which may be
inefficient. In MicroPython the tag tells if we are dealing with a small integer, interned
(small) string or a concrete object, and different semantics apply to each of these.
For small integers the mapping is this:
``********|********|********|*******1``
Where the asterisks hold the actual integer value. For an interned string or an immediate
object (e.g. ``True``) the layout of the ``mp_obj_t`` value is, respectively:
``********|********|********|*****010``
``********|********|********|*****110``
While a concrete object that is none of the above takes the form:
``********|********|********|******00``
The stars here correspond to the address of the concrete object in memory.
Allocation of objects
----------------------
The value of a small integer is stored directly in the ``mp_obj_t`` and will be
allocated in-place, not on the heap or elsewhere. As such, creation of small
integers does not affect the heap. Similarly for interned strings that already have
their textual data stored elsewhere, and immediate values like ``None``, ``False``
and ``True``.
Everything else which is a concrete object is allocated on the heap and its object structure is such that
a field is reserved in the object header to store the type of the object.
.. code-block:: bash
+++++++++++
+ +
+ type + object header
+ +
+++++++++++
+ + object items
+ +
+ +
+++++++++++
The heap's smallest unit of allocation is a block, which is four machine words in
size (16 bytes on a 32-bit machine, 32 bytes on a 64-bit machine).
Another structure also allocated on the heap tracks the allocation of
objects in each block. This structure is called a *bitmap*.
.. image:: img/bitmap.png
The bitmap tracks whether a block is "free" or "in use" and use two bits to track this state
for each block.
The mark-sweep garbage collector manages the objects allocated on the heap, and also
utilises the bitmap to mark objects that are still in use.
See `py/gc.c <https://github.com/micropython/micropython/blob/master/py/gc.c>`_
for the full implementation of these details.
**Allocation: heap layout**
The heap is arranged such that it consists of blocks in pools. A block
can have different properties:
- *ATB(allocation table byte):* If set, then the block is a normal block
- *FREE:* Free block
- *HEAD:* Head of a chain of blocks
- *TAIL:* In the tail of a chain of blocks
- *MARK :* Marked head block
- *FTB(finaliser table byte):* If set, then the block has a finaliser
.. _natmod:
Native machine code in .mpy files
=================================
This section describes how to build and work with .mpy files that contain native
machine code from a language other than Python. This allows you to
write code in a language like C, compile and link it into a .mpy file, and then
import this file like a normal Python module. This can be used for implementing
functionality which is performance critical, or for including an existing
library written in another language.
One of the main advantages of using native .mpy files is that native machine code
can be imported by a script dynamically, without the need to rebuild the main
MicroPython firmware. This is in contrast to :ref:`cmodules` which also allows
defining custom modules in C but they must be compiled into the main firmware image.
The focus here is on using C to build native modules, but in principle any
language which can be compiled to stand-alone machine code can be put into a
.mpy file.
A native .mpy module is built using the ``mpy_ld.py`` tool, which is found in the
``tools/`` directory of the project. This tool takes a set of object files
(.o files) and links them together to create a native .mpy files. It requires
CPython 3 and the library pyelftools v0.25 or greater.
Supported features and limitations
----------------------------------
A .mpy file can contain MicroPython bytecode and/or native machine code. If it
contains native machine code then the .mpy file has a specific architecture
associated with it. Current supported architectures are (these are the valid
options for the ``ARCH`` variable, see below):
* ``x86`` (32 bit)
* ``x64`` (64 bit x86)
* ``armv6m`` (ARM Thumb, eg Cortex-M0)
* ``armv7m`` (ARM Thumb 2, eg Cortex-M3)
* ``armv7emsp`` (ARM Thumb 2, single precision float, eg Cortex-M4F, Cortex-M7)
* ``armv7emdp`` (ARM Thumb 2, double precision float, eg Cortex-M7)
* ``xtensa`` (non-windowed, eg ESP8266)
* ``xtensawin`` (windowed with window size 8, eg ESP32)
When compiling and linking the native .mpy file the architecture must be chosen
and the corresponding file can only be imported on that architecture. For more
details about .mpy files see :ref:`mpy_files`.
Native code must be compiled as position independent code (PIC) and use a global
offset table (GOT), although the details of this varies from architecture to
architecture. When importing .mpy files with native code the import machinery
is able to do some basic relocation of the native code. This includes
relocating text, rodata and BSS sections.
Supported features of the linker and dynamic loader are:
* executable code (text)
* read-only data (rodata), including strings and constant data (arrays, structs, etc)
* zeroed data (BSS)
* pointers in text to text, rodata and BSS
* pointers in rodata to text, rodata and BSS
The known limitations are:
* data sections are not supported; workaround: use BSS data and initialise the
data values explicitly
* static BSS variables are not supported; workaround: use global BSS variables
So, if your C code has writable data, make sure the data is defined globally,
without an initialiser, and only written to within functions.
Linker limitation: the native module is not linked against the symbol table of the
full MicroPython firmware. Rather, it is linked against an explicit table of exported
symbols found in ``mp_fun_table`` (in ``py/nativeglue.h``), that is fixed at firmware
build time. It is thus not possible to simply call some arbitrary HAL/OS/RTOS/system
function, for example.
New symbols can be added to the end of the table and the firmware rebuilt.
The symbols also need to be added to ``tools/mpy_ld.py``'s ``fun_table`` dict in the
same location. This allows ``mpy_ld.py`` to be able to pick the new symbols up and
provide relocations for them when the mpy is imported. Finally, if the symbol is a
function, a macro or stub should be added to ``py/dynruntime.h`` to make it easy to
call the function.
Defining a native module
------------------------
A native .mpy module is defined by a set of files that are used to build the .mpy.
The filesystem layout consists of two main parts, the source files and the Makefile:
* In the simplest case only a single C source file is required, which contains all
the code that will be compiled into the .mpy module. This C source code must
include the ``py/dynruntime.h`` file to access the MicroPython dynamic API, and
must at least define a function called ``mpy_init``. This function will be the
entry point of the module, called when the module is imported.
The module can be split into multiple C source files if desired. Parts of the
module can also be implemented in Python. All source files should be listed in
the Makefile, by adding them to the ``SRC`` variable (see below). This includes
both C source files as well as any Python files which will be included in the
resulting .mpy file.
* The ``Makefile`` contains the build configuration for the module and list the
source files used to build the .mpy module. It should define ``MPY_DIR`` as the
location of the MicroPython repository (to find header files, the relevant Makefile
fragment, and the ``mpy_ld.py`` tool), ``MOD`` as the name of the module, ``SRC``
as the list of source files, optionally specify the machine architecture via ``ARCH``,
and then include ``py/dynruntime.mk``.
Minimal example
---------------
This section provides a fully working example of a simple module named ``factorial``.
This module provides a single function ``factorial.factorial(x)`` which computes the
factorial of the input and returns the result.
Directory layout::
factorial/
├── factorial.c
└── Makefile
The file ``factorial.c`` contains:
.. code-block:: c
// Include the header file to get access to the MicroPython API
#include "py/dynruntime.h"
// Helper function to compute factorial
STATIC mp_int_t factorial_helper(mp_int_t x) {
if (x == 0) {
return 1;
}
return x * factorial_helper(x - 1);
}
// This is the function which will be called from Python, as factorial(x)
STATIC mp_obj_t factorial(mp_obj_t x_obj) {
// Extract the integer from the MicroPython input object
mp_int_t x = mp_obj_get_int(x_obj);
// Calculate the factorial
mp_int_t result = factorial_helper(x);
// Convert the result to a MicroPython integer object and return it
return mp_obj_new_int(result);
}
// Define a Python reference to the function above
STATIC MP_DEFINE_CONST_FUN_OBJ_1(factorial_obj, factorial);
// This is the entry point and is called when the module is imported
mp_obj_t mpy_init(mp_obj_fun_bc_t *self, size_t n_args, size_t n_kw, mp_obj_t *args) {
// This must be first, it sets up the globals dict and other things
MP_DYNRUNTIME_INIT_ENTRY
// Make the function available in the module's namespace
mp_store_global(MP_QSTR_factorial, MP_OBJ_FROM_PTR(&factorial_obj));
// This must be last, it restores the globals dict
MP_DYNRUNTIME_INIT_EXIT
}
The file ``Makefile`` contains:
.. code-block:: make
# Location of top-level MicroPython directory
MPY_DIR = ../../..
# Name of module
MOD = factorial
# Source files (.c or .py)
SRC = factorial.c
# Architecture to build for (x86, x64, armv6m, armv7m, xtensa, xtensawin)
ARCH = x64
# Include to get the rules for compiling and linking the module
include $(MPY_DIR)/py/dynruntime.mk
Compiling the module
--------------------
The prerequisite tools needed to build a native .mpy file are:
* The MicroPython repository (at least the ``py/`` and ``tools/`` directories).
* CPython 3, and the library pyelftools (eg ``pip install 'pyelftools>=0.25'``).
* GNU make.
* A C compiler for the target architecture (if C source is used).
* Optionally ``mpy-cross``, built from the MicroPython repository (if .py source is used).
Be sure to select the correct ``ARCH`` for the target you are going to run on.
Then build with::
$ make
Without modifying the Makefile you can specify the target architecture via::
$ make ARCH=armv7m
Module usage in MicroPython
---------------------------
Once the module is built there should be a file called ``factorial.mpy``. Copy
this so it is accessible on the filesystem of your MicroPython system and can be
found in the import path. The module can now be accessed in Python just like any
other module, for example::
import factorial
print(factorial.factorial(10))
# should display 3628800
Further examples
----------------
See ``examples/natmod/`` for further examples which show many of the available
features of native .mpy modules. Such features include:
* using multiple C source files
* including Python code alongside C code
* rodata and BSS data
* memory allocation
* use of floating point
* exception handling
* including external C libraries
.. _optimizations:
Optimizations
=============
MicroPython uses several optimizations to save RAM but also ensure the efficient
execution of programs. This chapter discusses some of these optimizations.
.. note::
:ref:`qstr` and :ref:`maps` details other optimizations on strings and
dictionaries.
Frozen bytecode
---------------
When MicroPython loads Python code from the filesystem, it first has to parse the file into
a temporary in-memory representation, and then generate bytecode for execution, both of which
are stored in the heap (in RAM). This can lead to significant amounts of memory being used.
The MicroPython cross compiler can be used to generate
a ``.mpy`` file, containing the pre-compiled bytecode for a Python module. This will still
be loaded into RAM, but it avoids the additional overhead of the parsing stage.
As a further optimisation, the pre-compiled bytecode from a ``.mpy`` file can be "frozen"
into the firmware image as part of the main firmware compilation process, which means that
the bytecode will be executed from ROM. This can lead to a significant memory saving, and
reduce heap fragmentation.
See :ref:`manifest` for more information.
Variables
---------
MicroPython processes local and global variables differently. Global variables
are stored and looked up from a global dictionary that is allocated on the heap
(note that each module has its own separate dict, so separate namespace).
Local variables on the other hand are are stored on the Python value stack, which may
live on the C stack or on the heap. They are accessed directly by their offset
within the Python stack, which is more efficient than a global lookup in a dict.
The length of global variable names also affects how much RAM is used as identifiers
are stored in RAM. The shorter the identifier, the less memory is used.
The other aspect is that ``const`` variables that start with an underscore are treated as
proper constants and are not allocated or added in a dictionary, hence saving some memory.
These variables use ``const()`` from the MicroPython library. Therefore:
.. code-block:: python
from micropython import const
X = const(1)
_Y = const(2)
foo(X, _Y)
Compiles to:
.. code-block:: python
X = 1
foo(1, 2)
Allocation of memory
--------------------
Most of the common MicroPython constructs are not allocated on the heap.
However the following are:
- Dynamic data structures like lists, mappings, etc;
- Functions, classes and object instances;
- imports; and
- First-time assignment of global variables (to create the slot in the global dict).
For a detailed discussion on a more user-centric perspective on optimization,
see `Maximising MicroPython speed <https://docs.micropython.org/en/latest/reference/speed_python.html>`_
.. _porting_to_a_board:
Porting MicroPython
===================
The MicroPython project contains several ports to different microcontroller families and
architectures. The project repository has a `ports <https://github.com/micropython/micropython/tree/master/ports>`_
directory containing a subdirectory for each supported port.
A port will typically contain definitions for multiple "boards", each of which is a specific piece of
hardware that that port can run on, e.g. a development kit or device.
The `minimal port <https://github.com/micropython/micropython/tree/master/ports/minimal>`_ is
available as a simplified reference implementation of a MicroPython port. It can run on both the
host system and an STM32F4xx MCU.
In general, starting a port requires:
- Setting up the toolchain (configuring Makefiles, etc).
- Implementing boot configuration and CPU initialization.
- Initialising basic drivers required for development and debugging (e.g. GPIO, UART).
- Performing the board-specific configurations.
- Implementing the port-specific modules.
Minimal MicroPython firmware
----------------------------
The best way to start porting MicroPython to a new board is by integrating a minimal
MicroPython interpreter. For this walkthrough, create a subdirectory for the new
port in the ``ports`` directory:
.. code-block:: bash
$ cd ports
$ mkdir example_port
The basic MicroPython firmware is implemented in the main port file, e.g ``main.c``:
.. code-block:: c
#include "py/compile.h"
#include "py/gc.h"
#include "py/mperrno.h"
#include "py/stackctrl.h"
#include "shared/runtime/gchelper.h"
#include "shared/runtime/pyexec.h"
// Allocate memory for the MicroPython GC heap.
static char heap[4096];
int main(int argc, char **argv) {
// Initialise the MicroPython runtime.
mp_stack_ctrl_init();
gc_init(heap, heap + sizeof(heap));
mp_init();
// Start a normal REPL; will exit when ctrl-D is entered on a blank line.
pyexec_friendly_repl();
// Deinitialise the runtime.
gc_sweep_all();
mp_deinit();
return 0;
}
// Handle uncaught exceptions (should never be reached in a correct C implementation).
void nlr_jump_fail(void *val) {
for (;;) {
}
}
// Do a garbage collection cycle.
void gc_collect(void) {
gc_collect_start();
gc_helper_collect_regs_and_stack();
gc_collect_end();
}
// There is no filesystem so stat'ing returns nothing.
mp_import_stat_t mp_import_stat(const char *path) {
return MP_IMPORT_STAT_NO_EXIST;
}
// There is no filesystem so opening a file raises an exception.
mp_lexer_t *mp_lexer_new_from_file(const char *filename) {
mp_raise_OSError(MP_ENOENT);
}
We also need a Makefile at this point for the port:
.. code-block:: Makefile
# Include the core environment definitions; this will set $(TOP).
include ../../py/mkenv.mk
# Include py core make definitions.
include $(TOP)/py/py.mk
include $(TOP)/extmod/extmod.mk
# Set CFLAGS and libraries.
CFLAGS += -I. -I$(BUILD) -I$(TOP)
LIBS += -lm
# Define the required source files.
SRC_C = \
main.c \
mphalport.c \
shared/readline/readline.c \
shared/runtime/gchelper_generic.c \
shared/runtime/pyexec.c \
shared/runtime/stdout_helpers.c \
# Define the required object files.
OBJ = $(PY_CORE_O) $(addprefix $(BUILD)/, $(SRC_C:.c=.o))
# Define the top-level target, the main firmware.
all: $(BUILD)/firmware.elf
# Define how to build the firmware.
$(BUILD)/firmware.elf: $(OBJ)
$(ECHO) "LINK $@"
$(Q)$(CC) $(LDFLAGS) -o $@ $^ $(LIBS)
$(Q)$(SIZE) $@
# Include remaining core make rules.
include $(TOP)/py/mkrules.mk
Remember to use proper tabs to indent the Makefile.
MicroPython Configurations
--------------------------
After integrating the minimal code above, the next step is to create the MicroPython
configuration files for the port. The compile-time configurations are specified in
``mpconfigport.h`` and additional hardware-abstraction functions, such as time keeping,
in ``mphalport.h``.
The following is an example of an ``mpconfigport.h`` file:
.. code-block:: c
#include <stdint.h>
// Python internal features.
#define MICROPY_ENABLE_GC (1)
#define MICROPY_HELPER_REPL (1)
#define MICROPY_ERROR_REPORTING (MICROPY_ERROR_REPORTING_TERSE)
#define MICROPY_FLOAT_IMPL (MICROPY_FLOAT_IMPL_FLOAT)
// Enable u-modules to be imported with their standard name, like sys.
#define MICROPY_MODULE_WEAK_LINKS (1)
// Fine control over Python builtins, classes, modules, etc.
#define MICROPY_PY_ASYNC_AWAIT (0)
#define MICROPY_PY_BUILTINS_SET (0)
#define MICROPY_PY_ATTRTUPLE (0)
#define MICROPY_PY_COLLECTIONS (0)
#define MICROPY_PY_MATH (0)
#define MICROPY_PY_IO (0)
#define MICROPY_PY_STRUCT (0)
// Type definitions for the specific machine.
typedef intptr_t mp_int_t; // must be pointer size
typedef uintptr_t mp_uint_t; // must be pointer size
typedef long mp_off_t;
// We need to provide a declaration/definition of alloca().
#include <alloca.h>
// Define the port's name and hardware.
#define MICROPY_HW_BOARD_NAME "example-board"
#define MICROPY_HW_MCU_NAME "unknown-cpu"
#define MP_STATE_PORT MP_STATE_VM
This configuration file contains machine-specific configurations including aspects like if different
MicroPython features should be enabled e.g. ``#define MICROPY_ENABLE_GC (1)``. Making this Setting
``(0)`` disables the feature.
Other configurations include type definitions, root pointers, board name, microcontroller name
etc.
Similarly, an minimal example ``mphalport.h`` file looks like this:
.. code-block:: c
static inline void mp_hal_set_interrupt_char(char c) {}
Support for standard input/output
---------------------------------
MicroPython requires at least a way to output characters, and to have a REPL it also
requires a way to input characters. Functions for this can be implemented in the file
``mphalport.c``, for example:
.. code-block:: c
#include <unistd.h>
#include "py/mpconfig.h"
// Receive single character, blocking until one is available.
int mp_hal_stdin_rx_chr(void) {
unsigned char c = 0;
int r = read(STDIN_FILENO, &c, 1);
(void)r;
return c;
}
// Send the string of given length.
void mp_hal_stdout_tx_strn(const char *str, mp_uint_t len) {
int r = write(STDOUT_FILENO, str, len);
(void)r;
}
These input and output functions have to be modified depending on the
specific board API. This example uses the standard input/output stream.
Building and running
--------------------
At this stage the directory of the new port should contain::
ports/example_port/
├── main.c
├── Makefile
├── mpconfigport.h
├── mphalport.c
└── mphalport.h
The port can now be built by running ``make`` (or otherwise, depending on your system).
If you are using the default compiler settings in the Makefile given above then this
will create an executable called ``build/firmware.elf`` which can be executed directly.
To get a functional REPL you may need to first configure the terminal to raw mode:
.. code-block:: bash
$ stty raw opost -echo
$ ./build/firmware.elf
That should give a MicroPython REPL. You can then run commands like:
.. code-block:: bash
MicroPython v1.13 on 2021-01-01; example-board with unknown-cpu
>>> import sys
>>> sys.implementation
('micropython', (1, 13, 0))
>>>
Use Ctrl-D to exit, and then run ``reset`` to reset the terminal.
Adding a module to the port
---------------------------
To add a custom module like ``myport``, first add the module definition in a file
``modmyport.c``:
.. code-block:: c
#include "py/runtime.h"
STATIC mp_obj_t myport_info(void) {
mp_printf(&mp_plat_print, "info about my port\n");
return mp_const_none;
}
STATIC MP_DEFINE_CONST_FUN_OBJ_0(myport_info_obj, myport_info);
STATIC const mp_rom_map_elem_t myport_module_globals_table[] = {
{ MP_OBJ_NEW_QSTR(MP_QSTR___name__), MP_OBJ_NEW_QSTR(MP_QSTR_myport) },
{ MP_ROM_QSTR(MP_QSTR_info), MP_ROM_PTR(&myport_info_obj) },
};
STATIC MP_DEFINE_CONST_DICT(myport_module_globals, myport_module_globals_table);
const mp_obj_module_t myport_module = {
.base = { &mp_type_module },
.globals = (mp_obj_dict_t *)&myport_module_globals,
};
MP_REGISTER_MODULE(MP_QSTR_myport, myport_module);
You will also need to edit the Makefile to add ``modmyport.c`` to the ``SRC_C`` list, and
a new line adding the same file to ``SRC_QSTR`` (so qstrs are searched for in this new file),
like this:
.. code-block:: Makefile
SRC_C = \
main.c \
modmyport.c \
mphalport.c \
...
SRC_QSTR += modmyport.c
If all went correctly then, after rebuilding, you should be able to import the new module:
.. code-block:: bash
>>> import myport
>>> myport.info()
info about my port
>>>
.. _publiccapi:
The public C API
================
The public C-API comprises functions defined in all C header files in the ``py/``
directory. Most of the important core runtime C APIs are exposed in ``runtime.h`` and
``obj.h``.
The following is an example of public API functions from ``obj.h``:
.. code-block:: c
mp_obj_t mp_obj_new_list(size_t n, mp_obj_t *items);
mp_obj_t mp_obj_list_append(mp_obj_t self_in, mp_obj_t arg);
mp_obj_t mp_obj_list_remove(mp_obj_t self_in, mp_obj_t value);
void mp_obj_list_get(mp_obj_t self_in, size_t *len, mp_obj_t **items);
At its core, any functions and macros in header files make up the public
API and can be used to access very low-level details of MicroPython. Static
inline functions in header files are fine too, such functions will be
inlined in the code when used.
Header files in the ``ports`` directory are only exposed to the functionality
specific to a given port.
.. _qstr:
MicroPython string interning
============================
MicroPython uses `string interning`_ to save both RAM and ROM. This avoids
having to store duplicate copies of the same string. Primarily, this applies to
identifiers in your code, as something like a function or variable name is very
likely to appear in multiple places in the code. In MicroPython an interned
string is called a QSTR (uniQue STRing).
A QSTR value (with type ``qstr``) is a index into a linked list of QSTR pools.
QSTRs store their length and a hash of their contents for fast comparison during
the de-duplication process. All bytecode operations that work with strings use
a QSTR argument.
Compile-time QSTR generation
----------------------------
In the MicroPython C code, any strings that should be interned in the final
firmware are written as ``MP_QSTR_Foo``. At compile time this will evaluate to
a ``qstr`` value that points to the index of ``"Foo"`` in the QSTR pool.
A multi-step process in the ``Makefile`` makes this work. In summary this
process has three parts:
1. Find all ``MP_QSTR_Foo`` tokens in the code.
2. Generate a static QSTR pool containing all the string data (including lengths
and hashes).
3. Replace all ``MP_QSTR_Foo`` (via the preprocessor) with their corresponding
index.
``MP_QSTR_Foo`` tokens are searched for in two sources:
1. All files referenced in ``$(SRC_QSTR)``. This is all C code (i.e. ``py``,
``extmod``, ``ports/stm32``) but not including third-party code such as
``lib``.
2. Additional ``$(QSTR_GLOBAL_DEPENDENCIES)`` (which includes ``mpconfig*.h``).
*Note:* ``frozen_mpy.c`` (generated by mpy-tool.py) has its own QSTR generation
and pool.
Some additional strings that can't be expressed using the ``MP_QSTR_Foo`` syntax
(e.g. they contain non-alphanumeric characters) are explicitly provided in
``qstrdefs.h`` and ``qstrdefsport.h`` via the ``$(QSTR_DEFS)`` variable.
Processing happens in the following stages:
1. ``qstr.i.last`` is the concatenation of putting every single input file
through the C pre-processor. This means that any conditionally disabled code
will be removed, and macros expanded. This means we don't add strings to the
pool that won't be used in the final firmware. Because at this stage (thanks
to the ``NO_QSTR`` macro added by ``QSTR_GEN_CFLAGS``) there is no
definition for ``MP_QSTR_Foo`` it passes through this stage unaffected. This
file also includes comments from the preprocessor that include line number
information. Note that this step only uses files that have changed, which
means that ``qstr.i.last`` will only contain data from files that have
changed since the last compile.
2. ``qstr.split`` is an empty file created after running ``makeqstrdefs.py split``
on qstr.i.last. It's just used as a dependency to indicate that the step ran.
This script outputs one file per input C file, ``genhdr/qstr/...file.c.qstr``,
which contains only the matched QSTRs. Each QSTR is printed as ``Q(Foo)``.
This step is necessary to combine the existing files with the new data
generated from the incremental update in ``qstr.i.last``.
3. ``qstrdefs.collected.h`` is the output of concatenating ``genhdr/qstr/*``
using ``makeqstrdefs.py cat``. This is now the full set of ``MP_QSTR_Foo``'s
found in the code, now formatted as ``Q(Foo)``, one-per-line, with duplicates.
This file is only updated if the set of qstrs has changed. A hash of the QSTR
data is written to another file (``qstrdefs.collected.h.hash``) which allows
it to track changes across builds.
4. Generate an enumeration, each entry of which maps a ``MP_QSTR_Foo`` to it's corresponding index.
It concatenates ``qstrdefs.collected.h`` with ``qstrdefs*.h``, then it transforms
each line from ``Q(Foo)`` to ``"Q(Foo)"`` so they pass through the preprocessor
unchanged. Then the preprocessor is used to deal with any conditional
compilation in ``qstrdefs*.h``. Then the transformation is undone back to
``Q(Foo)``, and saved as ``qstrdefs.preprocessed.h``.
5. ``qstrdefs.generated.h`` is the output of ``makeqstrdata.py``. For each
``Q(Foo)`` in qstrdefs.preprocessed.h (plus some extra hard-coded ones), it outputs
``QDEF(MP_QSTR_Foo, (const byte*)"hash" "Foo")``.
Then in the main compile, two things happen with ``qstrdefs.generated.h``:
1. In qstr.h, each QDEF becomes an entry in an enum, which makes ``MP_QSTR_Foo``
available to code and equal to the index of that string in the QSTR table.
2. In qstr.c, the actual QSTR data table is generated as elements of the
``mp_qstr_const_pool->qstrs``.
.. _`string interning`: https://en.wikipedia.org/wiki/String_interning
Run-time QSTR generation
------------------------
Additional QSTR pools can be created at runtime so that strings can be added to
them. For example, the code::
foo[x] = 3
Will need to create a QSTR for the value of ``x`` so it can be used by the
"load attr" bytecode.
Also, when compiling Python code, identifiers and literals need to have QSTRs
created. Note: only literals shorter than 10 characters become QSTRs. This is
because a regular string on the heap always takes up a minimum of 16 bytes (one
GC block), whereas QSTRs allow them to be packed more efficiently into the pool.
QSTR pools (and the underlying "chunks" that store the string data) are allocated
on-demand on the heap with a minimum size.
.. _writingtests:
Writing tests
=============
Tests in MicroPython are located at the path ``tests/``. The following is a listing of
key directories and the run-tests.py runner script:
.. code-block:: bash
.
├── basics
├── extmod
├── float
├── micropython
├── run-tests.py
...
There are subfolders maintained to categorize the tests. Add a test by creating a new file in one of the
existing folders or in a new folder. It's also possible to make custom tests outside this tests folder,
which would be recommended for a custom port.
For example, add the following code in a file ``print.py`` in the ``tests/unix/`` subdirectory:
.. code-block:: python
def print_one():
print(1)
print_one()
If you run your tests, this test should appear in the test output:
.. code-block:: bash
$ cd ports/unix
$ make tests
skip unix/extra_coverage.py
pass unix/ffi_callback.py
pass unix/ffi_float.py
pass unix/ffi_float2.py
pass unix/print.py
pass unix/time.py
pass unix/time2.py
Tests are run by comparing the output from the test target against the output from CPython.
So any test should use print statements to indicate test results.
For tests that can't be compared to CPython (i.e. micropython-specific functionality),
you can provide a ``.py.exp`` file which will be used as the truth for comparison.
The other way to run tests, which is useful when running on targets other than the Unix port, is:
.. code-block:: bash
$ cd tests
$ ./run-tests.py
Then to run on a board:
.. code-block:: bash
$ ./run-tests.py --target minimal --device /dev/ttyACM0
And to run only a certain set of tests (eg a directory):
.. code-block:: bash
$ ./run-tests.py -d basics
$ ./run-tests.py float/builtin*.py
.. _cpython_diffs:
MicroPython differences from CPython
====================================
MicroPython implements Python 3.4 and some select features of Python 3.5 and
above. The sections below describe the current status of these features.
.. toctree::
../differences/python_35.rst
../differences/python_36.rst
../differences/python_37.rst
../differences/python_38.rst
../differences/python_39.rst
../differences/python_310.rst
For the features of Python that are implemented by MicroPython, there are
sometimes differences in their behaviour compared to standard Python. The
operations listed in the sections below produce conflicting results in
MicroPython when compared to standard Python.
.. toctree::
:maxdepth: 2
.. _python_310:
Python 3.10
===========
Python 3.10.0 (final) was released on the 4 October 2021. The Features for 3.10 are
defined in `PEP 619 <https://www.python.org/dev/peps/pep-0619/#features-for-3-10>`_
and a detailed description of the changes can be found in
`What's New in Python 3.10 <https://docs.python.org/3/whatsnew/3.10.html>`_.
.. table::
:widths: 20 60 20
+--------------------------------------------------------+----------------------------------------------------+--------------+
| **New syntax features** | **Status** |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| `PEP 634 <https://www.python.org/dev/peps/pep-0634/>`_ | Structural Pattern Matching: Specification | [#spm]_ |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| `PEP 635 <https://www.python.org/dev/peps/pep-0635/>`_ | Structural Pattern Matching: Motivation and | [#spm]_ |
| | Rationale | |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| `PEP 636 <https://www.python.org/dev/peps/pep-0636/>`_ | Structural Pattern Matching: Tutorial | [#spm]_ |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| `bpo-12782 | Parenthesized context managers are now officially | |
| <https://github.com/python/cpython/issues/56991>`_ | allowed | |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| **New features in the standard library** |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| `PEP 618 <https://www.python.org/dev/peps/pep-0618/>`_ | Add Optional Length-Checking To zip | |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| **Interpreter improvements** |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| `PEP 626 <https://www.python.org/dev/peps/pep-0626/>`_ | Precise line numbers for debugging and other tools | |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| **New typing features** |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| `PEP 604 <https://www.python.org/dev/peps/pep-0604/>`_ | Allow writing union types as X | Y | |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| `PEP 613 <https://www.python.org/dev/peps/pep-0613/>`_ | Explicit Type Aliases | |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| `PEP 612 <https://www.python.org/dev/peps/pep-0612/>`_ | Parameter Specification Variables | |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| **Important deprecations, removals or restrictions** |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| `PEP 644 <https://www.python.org/dev/peps/pep-0644/>`_ | Require OpenSSL 1.1.1 or newer | |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| `PEP 632 <https://www.python.org/dev/peps/pep-0632/>`_ | Deprecate distutils module. | Not relevant |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| `PEP 623 <https://www.python.org/dev/peps/pep-0623/>`_ | Deprecate and prepare for the removal of the wstr | Not relevant |
| | member in PyUnicodeObject. | |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| `PEP 624 <https://www.python.org/dev/peps/pep-0624/>`_ | Remove Py_UNICODE encoder APIs | Not relevant |
+--------------------------------------------------------+----------------------------------------------------+--------------+
| `PEP 597 <https://www.python.org/dev/peps/pep-0597/>`_ | Add optional EncodingWarning | |
+--------------------------------------------------------+----------------------------------------------------+--------------+
Other Language Changes:
.. table::
:widths: 90 10
+-------------------------------------------------------------------------------------------------------------+---------------+
| The :class:`int` type has a new method :meth:`int.bit_count`, returning the | |
| number of ones in the binary expansion of a given integer, also known | |
| as the population count. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| The views returned by :meth:`dict.keys`, :meth:`dict.values` and | |
| :meth:`dict.items` now all have a ``mapping`` attribute that gives a | |
| :class:`types.MappingProxyType` object wrapping the original | |
| dictionary. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| :pep:`618`: The :func:`zip` function now has an optional ``strict`` flag, used | |
| to require that all the iterables have an equal length. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Builtin and extension functions that take integer arguments no longer accept | |
| :class:`~decimal.Decimal`\ s, :class:`~fractions.Fraction`\ s and other | |
| objects that can be converted to integers only with a loss (e.g. that have | |
| the :meth:`~object.__int__` method but do not have the | |
| :meth:`~object.__index__` method). | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| If :func:`object.__ipow__` returns :const:`NotImplemented`, the operator will | |
| correctly fall back to :func:`object.__pow__` and :func:`object.__rpow__` as expected. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Assignment expressions can now be used unparenthesized within set literals | |
| and set comprehensions, as well as in sequence indexes (but not slices). | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Functions have a new ``__builtins__`` attribute which is used to look for | |
| builtin symbols when a function is executed, instead of looking into | |
| ``__globals__['__builtins__']``. The attribute is initialized from | |
| ``__globals__["__builtins__"]`` if it exists, else from the current builtins. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Two new builtin functions -- :func:`aiter` and :func:`anext` have been added | |
| to provide asynchronous counterparts to :func:`iter` and :func:`next`, | |
| respectively. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Static methods (:func:`@staticmethod <staticmethod>`) and class methods | |
| (:func:`@classmethod <classmethod>`) now inherit the method attributes | |
| (``__module__``, ``__name__``, ``__qualname__``, ``__doc__``, | |
| ``__annotations__``) and have a new ``__wrapped__`` attribute. | |
| Moreover, static methods are now callable as regular functions. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Annotations for complex targets (everything beside ``simple name`` targets | |
| defined by :pep:`526`) no longer cause any runtime effects with ``from __future__ import annotations``. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Class and module objects now lazy-create empty annotations dicts on demand. | |
| The annotations dicts are stored in the object’s ``__dict__`` for | |
| backwards compatibility. This improves the best practices for working | |
| with ``__annotations__``. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Annotations consist of ``yield``, ``yield from``, ``await`` or named expressions | |
| are now forbidden under ``from __future__ import annotations`` due to their side | |
| effects. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Usage of unbound variables, ``super()`` and other expressions that might | |
| alter the processing of symbol table as annotations are now rendered | |
| effectless under ``from __future__ import annotations``. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Hashes of NaN values of both :class:`float` type and | |
| :class:`decimal.Decimal` type now depend on object identity. Formerly, they | |
| always hashed to ``0`` even though NaN values are not equal to one another. | |
| This caused potentially quadratic runtime behavior due to excessive hash | |
| collisions when creating dictionaries and sets containing multiple NaNs. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| A :exc:`SyntaxError` (instead of a :exc:`NameError`) will be raised when deleting | |
| the :const:`__debug__` constant. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| :exc:`SyntaxError` exceptions now have ``end_lineno`` and | |
| ``end_offset`` attributes. They will be ``None`` if not determined. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
Changes to built-in modules:
.. table::
:widths: 90 10
+---------------------------------------------------------------------------------------------------------------+---------------+
| `asyncio <https://docs.python.org/3/whatsnew/3.10.html#asyncio>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Add missing :meth:`~asyncio.events.AbstractEventLoop.connect_accepted_socket` | |
| method. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| `array <https://docs.python.org/3/whatsnew/3.10.html#array>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| The :meth:`~array.array.index` method of :class:`array.array` now has | |
| optional *start* and *stop* parameters. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| `gc <https://docs.python.org/3/whatsnew/3.10.html#gc>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Add audit hooks for :func:`gc.get_objects`, :func:`gc.get_referrers` and | |
| :func:`gc.get_referents`. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| `hashlib <https://docs.python.org/3/whatsnew/3.10.html#hashlib>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| The hashlib module requires OpenSSL 1.1.1 or newer. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| The hashlib module has preliminary support for OpenSSL 3.0.0. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| The pure-Python fallback of :func:`~hashlib.pbkdf2_hmac` is deprecated. In | |
| the future PBKDF2-HMAC will only be available when Python has been built with | |
| OpenSSL support. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| `os <https://docs.python.org/3/whatsnew/3.10.html#os>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Add :func:`os.cpu_count()` support for VxWorks RTOS. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Add a new function :func:`os.eventfd` and related helpers to wrap the | |
| ``eventfd2`` syscall on Linux. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Add :func:`os.splice()` that allows to move data between two file | |
| descriptors without copying between kernel address space and user | |
| address space, where one of the file descriptors must refer to a | |
| pipe. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Add :data:`~os.O_EVTONLY`, :data:`~os.O_FSYNC`, :data:`~os.O_SYMLINK` | |
| and :data:`~os.O_NOFOLLOW_ANY` for macOS. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| `platform <https://docs.python.org/3/whatsnew/3.10.html#platform>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Add :func:`platform.freedesktop_os_release()` to retrieve operation system | |
| identification from `freedesktop.org os-release | |
| <https://www.freedesktop.org/software/systemd/man/os-release.html>`_ standard file. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| `socket <https://docs.python.org/3/whatsnew/3.10.html#socket>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| The exception :exc:`socket.timeout` is now an alias of :exc:`TimeoutError`. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Add option to create MPTCP sockets with ``IPPROTO_MPTCP``. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Add ``IP_RECVTOS`` option to receive the type of service (ToS) or DSCP/ECN fields. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| `ssl <https://docs.python.org/3/whatsnew/3.10.html#ssl>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| The ssl module requires OpenSSL 1.1.1 or newer. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| The ssl module has preliminary support for OpenSSL 3.0.0 and new option | |
| :data:`~ssl.OP_IGNORE_UNEXPECTED_EOF`. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Deprecated function and use of deprecated constants now result in | |
| a :exc:`DeprecationWarning`. :attr:`ssl.SSLContext.options` has | |
| :data:`~ssl.OP_NO_SSLv2` and :data:`~ssl.OP_NO_SSLv3` set by default and | |
| therefore cannot warn about setting the flag again. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| The ssl module now has more secure default settings. Ciphers without forward | |
| secrecy or SHA-1 MAC are disabled by default. Security level 2 prohibits | |
| weak RSA, DH, and ECC keys with less than 112 bits of security. | |
| :class:`~ssl.SSLContext` defaults to minimum protocol version TLS 1.2. | |
| Settings are based on Hynek Schlawack's research. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| The deprecated protocols SSL 3.0, TLS 1.0, and TLS 1.1 are no longer | |
| officially supported. Python does not block them actively. However | |
| OpenSSL build options, distro configurations, vendor patches, and cipher | |
| suites may prevent a successful handshake. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Add a *timeout* parameter to the :func:`ssl.get_server_certificate` function. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| The ssl module uses heap-types and multi-phase initialization. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| A new verify flag :data:`~ssl.VERIFY_X509_PARTIAL_CHAIN` has been added. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| `sys <https://docs.python.org/3/whatsnew/3.10.html#sys>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Add :data:`sys.orig_argv` attribute: the list of the original command line | |
| arguments passed to the Python executable. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Add :data:`sys.stdlib_module_names`, containing the list of the standard library | |
| module names. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| `_thread <https://docs.python.org/3/whatsnew/3.10.html#_thread>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| :func:`_thread.interrupt_main` now takes an optional signal number to | |
| simulate (the default is still :data:`signal.SIGINT`). | |
+---------------------------------------------------------------------------------------------------------------+---------------+
.. rubric:: Notes
.. [#spm] The structural pattern matching feature is discussed in `issue #7847 <https://github.com/micropython/micropython/issues/7847>`_.
.. _python_35:
Python 3.5
==========
Below is a list of finalised/accepted PEPs for Python 3.5 grouped into their impact to MicroPython.
.. table::
:widths: 30 50 20
+--------------------------------------------------------------------------------------------------------------+--------------------+
| **Extensions to the syntax** | **Status** |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| `PEP 448 <https://www.python.org/dev/peps/pep-0448/>`_ | Additional unpacking generalizations | Partial |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| `PEP 465 <https://www.python.org/dev/peps/pep-0465/>`_ | A new matrix multiplication operator | Complete |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| `PEP 492 <https://www.python.org/dev/peps/pep-0492/>`_ | Coroutines with ``async`` and ``await`` syntax | Complete |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| **Extensions and changes to runtime** |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| `PEP 461 <https://www.python.org/dev/peps/pep-0461/>`_ | % formatting for binary strings | Complete |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| `PEP 475 <https://www.python.org/dev/peps/pep-0475/>`_ | Retrying system calls that fail with ``EINTR`` | Complete |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| `PEP 479 <https://www.python.org/dev/peps/pep-0479/>`_ | Change ``StopIteration`` handling inside generators | Complete |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| **Standard library changes** |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| `PEP 471 <https://www.python.org/dev/peps/pep-0471/>`_ | ``os.scandir()`` | |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| `PEP 485 <https://www.python.org/dev/peps/pep-0485/>`_ | ``math.isclose()``, a function for testing | Complete |
| | approximate equality | |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| **Miscellaneous changes** |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| `PEP 441 <https://www.python.org/dev/peps/pep-0441/>`_ | Improved Python zip application support | |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| `PEP 486 <https://www.python.org/dev/peps/pep-0486/>`_ | Make the Python Launcher aware of virtual | Not relevant |
| | environments | |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| `PEP 484 <https://www.python.org/dev/peps/pep-0484/>`_ | Type hints (advisory only) | Complete [#fth]_ |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| `PEP 488 <https://www.python.org/dev/peps/pep-0488/>`_ | Elimination of PYO files | Not relevant |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
| `PEP 489 <https://www.python.org/dev/peps/pep-0489/>`_ | Redesigning extension module loading | |
+--------------------------------------------------------+-----------------------------------------------------+--------------------+
Other Language Changes:
.. table::
:widths: 90 10
+-----------------------------------------------------------------------------------------------------------+---------------+
| Added the *namereplace* error handlers. The *backslashreplace* error handlers now work with decoding and | |
| translating. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| Property docstrings are now writable. This is especially useful for collections.namedtuple() docstrings | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| Circular imports involving relative imports are now supported. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
New Modules:
* `typing <https://docs.python.org/3/whatsnew/3.5.html#typing>`_
* `zipzap <https://docs.python.org/3/whatsnew/3.5.html#zipapp>`_
Changes to built-in modules:
.. table::
:widths: 90 10
+-----------------------------------------------------------------------------------------------------------+---------------+
| `collections <https://docs.python.org/3/whatsnew/3.5.html#collections>`_ |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The *OrderedDict* class is now implemented in C, which makes it 4 to 100 times faster. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| *OrderedDict.items()* , *OrderedDict.keys()* , *OrderedDict.values()* views now support reversed() | |
| iteration. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The deque class now defines *index()*, *insert()*, and *copy()*, and supports the + and * operators. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| Docstrings produced by namedtuple() can now be updated. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The UserString class now implements the *__getnewargs__()*, *__rmod__()*, *casefold()*, *format_map()*, | |
| *isprintable()*, and *maketrans()* methods to match the corresponding methods of str. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| `heapq <https://docs.python.org/3/whatsnew/3.5.html#heapq>`_ |
+-----------------------------------------------------------------------------------------------------------+---------------+
| Element comparison in *merge()* can now be customized by passing a key function in a new optional key | |
| keyword argument, and a new optional *reverse* keyword argument can be used to reverse element comparison | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| `io <https://docs.python.org/3/whatsnew/3.5.html#io>`_ |
+-----------------------------------------------------------------------------------------------------------+---------------+
| A new *BufferedIOBase.readinto1()* method, that uses at most one call to the underlying raw stream's | |
| *RawIOBase.read()* or *RawIOBase.readinto()* methods | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| `json <https://docs.python.org/3/whatsnew/3.5.html#json>`_ | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| JSON decoder now raises JSONDecodeError instead of ValueError to provide better context information about | |
| the error. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| `math <https://docs.python.org/3/whatsnew/3.5.html#math>`_ |
+-----------------------------------------------------------------------------------------------------------+---------------+
| Two new constants have been added to the math module: *inf* and *nan*. | Complete |
+-----------------------------------------------------------------------------------------------------------+---------------+
| A new function *isclose()* provides a way to test for approximate equality. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| A new *gcd()* function has been added. The *fractions.gcd()* function is now deprecated. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| `os <https://docs.python.org/3/whatsnew/3.5.html#os>`_ |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The new *scandir()* function returning an iterator of DirEntry objects has been added. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The *urandom()* function now uses the *getrandom()* syscall on Linux 3.17 or newer, and *getentropy()* on | |
| OpenBSD 5.6 and newer, removing the need to use /dev/urandom and avoiding failures due to potential file | |
| descriptor exhaustion. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| New *get_blocking()* and *set_blocking()* functions allow getting and setting a file descriptor's blocking| |
| mode (O_NONBLOCK.) | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| There is a new *os.path.commonpath()* function returning the longest common sub-path of each passed | |
| pathname | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| `re <https://docs.python.org/3/whatsnew/3.5.html#re>`_ | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| References and conditional references to groups with fixed length are now allowed in lookbehind assertions| |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The number of capturing groups in regular expressions is no longer limited to 100. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The *sub()* and *subn()* functions now replace unmatched groups with empty strings instead of raising an | |
| exception. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The *re.error* exceptions have new attributes, msg, pattern, pos, lineno, and colno, that provide better | |
| context information about the error | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| `socket <https://docs.python.org/3/whatsnew/3.5.html#socket>`_ |
+-----------------------------------------------------------------------------------------------------------+---------------+
| Functions with timeouts now use a monotonic clock, instead of a system clock. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| A new *socket.sendfile()* method allows sending a file over a socket by using the high-performance | |
| *os.sendfile()* function on UNIX, resulting in uploads being from 2 to 3 times faster than when using | |
| plain *socket.send()* | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The *socket.sendall()* method no longer resets the socket timeout every time bytes are received or sent. | |
| The socket timeout is now the maximum total duration to send all data. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The backlog argument of the *socket.listen()* method is now optional. By default it is set to SOMAXCONN or| Complete |
| to 128, whichever is less. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| `ssl <https://docs.python.org/3/whatsnew/3.5.html#ssl>`_ |
+-----------------------------------------------------------------------------------------------------------+---------------+
| Memory BIO Support | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| Application-Layer Protocol Negotiation Support | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| There is a new *SSLSocket.version()* method to query the actual protocol version in use. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The SSLSocket class now implements a *SSLSocket.sendfile()* method. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The *SSLSocket.send()* method now raises either the *ssl.SSLWantReadError* or *ssl.SSLWantWriteError* | |
| exception on a non-blocking socket if the operation would block. Previously, it would return 0. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The *cert_time_to_seconds()* function now interprets the input time as UTC and not as local time, per RFC | |
| 5280. Additionally, the return value is always an int. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| New *SSLObject.shared_ciphers()* and *SSLSocket.shared_ciphers()* methods return the list of ciphers sent | |
| by the client during the handshake. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The *SSLSocket.do_handshake()*, *SSLSocket.read()*, *SSLSocket.shutdown()*, and *SSLSocket.write()* | |
| methods of the SSLSocket class no longer reset the socket timeout every time bytes are received or sent. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The *match_hostname()* function now supports matching of IP addresses. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| `sys <https://docs.python.org/3/whatsnew/3.5.html#sys>`_ |
+-----------------------------------------------------------------------------------------------------------+---------------+
| A new *set_coroutine_wrapper()* function allows setting a global hook that will be called whenever a | |
| coroutine object is created by an async def function. A corresponding *get_coroutine_wrapper()* can be | |
| used to obtain a currently set wrapper. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| A new *is_finalizing()* function can be used to check if the Python interpreter is shutting down. | |
+-----------------------------------------------------------------------------------------------------------+---------------+
| `time <https://docs.python.org/3/whatsnew/3.5.html#time>`_ |
+-----------------------------------------------------------------------------------------------------------+---------------+
| The *monotonic()* function is now always available | |
+-----------------------------------------------------------------------------------------------------------+---------------+
.. rubric:: Notes
.. [#fth] The MicroPython parser correct ignores all type hints. However, the ``typing`` module is not built-in.
.. _python_36:
Python 3.6
==========
Python 3.6 beta 1 was released on 12 Sep 2016, and a summary of the new features can be found here:
.. table::
:widths: 30 50 20
+-----------------------------------------------------------------------------------------------------------+-----------------+
| **New Syntax Features** | **Status** |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 498 <https://www.python.org/dev/peps/pep-0498/>`_ | Literal String Formatting | Complete |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 515 <https://www.python.org/dev/peps/pep-0515/>`_ | Underscores in Numeric Literals | Complete |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 525 <https://www.python.org/dev/peps/pep-0525/>`_ | Asynchronous Generators | |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 526 <https://www.python.org/dev/peps/pep-0526/>`_ | Syntax for Variable Annotations (provisional) | Complete |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 530 <https://www.python.org/dev/peps/pep-0530/>`_ | Asynchronous Comprehensions | |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| **New Built-in Features** |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 468 <https://www.python.org/dev/peps/pep-0468/>`_ | Preserving the order of *kwargs* in a function | |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 487 <https://www.python.org/dev/peps/pep-0487/>`_ | Simpler customization of class creation | |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 520 <https://www.python.org/dev/peps/pep-0520/>`_ | Preserving Class Attribute Definition Order | |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| **Standard Library Changes** |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 495 <https://www.python.org/dev/peps/pep-0495/>`_ | Local Time Disambiguation | |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 506 <https://www.python.org/dev/peps/pep-0506/>`_ | Adding A Secrets Module To The Standard Library | |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 519 <https://www.python.org/dev/peps/pep-0519/>`_ | Adding a file system path protocol | |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| **CPython Internals** |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 509 <https://www.python.org/dev/peps/pep-0509/>`_ | Add a private version to dict | Won't do |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 523 <https://www.python.org/dev/peps/pep-0523/>`_ | Adding a frame evaluation API to CPython | |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| **Linux/Window Changes** |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 524 <https://www.python.org/dev/peps/pep-0524/>`_ | Make ``os.urandom()`` blocking on Linux | |
| | (during system startup) | |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 528 <https://www.python.org/dev/peps/pep-0528/>`_ | Change Windows console encoding to UTF-8 | |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
| `PEP 529 <https://www.python.org/dev/peps/pep-0529/>`_ | Change Windows filesystem encoding to UTF-8 | |
+--------------------------------------------------------+--------------------------------------------------+-----------------+
Other Language Changes:
.. table::
:widths: 90 10
+-------------------------------------------------------------------------------------------------------------+---------------+
| A *global* or *nonlocal* statement must now textually appear before the first use of the affected name in | |
| the same scope. Previously this was a SyntaxWarning. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| It is now possible to set a special method to None to indicate that the corresponding operation is not | |
| available. For example, if a class sets *__iter__()* to *None* , the class is not iterable. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Long sequences of repeated traceback lines are now abbreviated as *[Previous line repeated {count} more | |
| times]* | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Import now raises the new exception *ModuleNotFoundError* when it cannot find a module. Code that currently | |
| checks for ImportError (in try-except) will still work. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Class methods relying on zero-argument *super()* will now work correctly when called from metaclass methods | |
| during class creation. | |
+-------------------------------------------------------------------------------------------------------------+---------------+
Changes to built-in modules:
.. table::
:widths: 90 10
+--------------------------------------------------------------------------------------------------------------+----------------+
| `array <https://docs.python.org/3.6/whatsnew/3.6.html#array>`_ | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| Exhausted iterators of *array.array* will now stay exhausted even if the iterated array is extended. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| `binascii <https://docs.python.org/3.6/whatsnew/3.6.html#binascii>`_ | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The b2a_base64() function now accepts an optional newline keyword argument to control whether the newline | Complete |
| character is appended to the return value | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| `cmath <https://docs.python.org/3.6/whatsnew/3.6.html#cmath>`_ | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The new cmath.tau (τ) constant has been added | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| New constants: *cmath.inf* and *cmath.nan* to match *math.inf* and *math.nan* , and also *cmath.infj* and | |
| *cmath.nanj* to match the format used by complex repr | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| `collections <https://docs.python.org/3.6/whatsnew/3.6.html#collections>`_ |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The new Collection abstract base class has been added to represent sized iterable container classes | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The new *Reversible* abstract base class represents iterable classes that also provide the *__reversed__()* | |
| method. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The new *AsyncGenerator* abstract base class represents asynchronous generators. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The *namedtuple()* function now accepts an optional keyword argument module, which, when specified, is used | |
| for the *__module__* attribute of the returned named tuple class. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The verbose and rename arguments for *namedtuple()* are now keyword-only. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| Recursive *collections.deque* instances can now be pickled. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| `hashlib <https://docs.python.org/3.6/whatsnew/3.6.html#hashlib>`_ |
+--------------------------------------------------------------------------------------------------------------+----------------+
| BLAKE2 hash functions were added to the module. *blake2b()* and *blake2s()* are always available and support | |
| the full feature set of BLAKE2. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The SHA-3 hash functions *sha3_224()*, *sha3_256()*, *sha3_384()*, *sha3_512()*, and *SHAKE* hash functions | |
| *shake_128()* and *shake_256()* were added. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The password-based key derivation function *scrypt()* is now available with OpenSSL 1.1.0 and newer. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| `json <https://docs.python.org/3.6/whatsnew/3.6.html#json>`_ |
+--------------------------------------------------------------------------------------------------------------+----------------+
| *json.load()* and *json.loads()* now support binary input. Encoded JSON should be represented using either | |
| UTF-8, UTF-16, or UTF-32. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| `math <https://docs.python.org/3.6/whatsnew/3.6.html#math>`_ |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The new math.tau (τ) constant has been added | Complete |
+--------------------------------------------------------------------------------------------------------------+----------------+
| `os <https://docs.python.org/3.6/whatsnew/3.6.html#os>`_ |
+--------------------------------------------------------------------------------------------------------------+----------------+
| A new *close()* method allows explicitly closing a *scandir()* iterator. The *scandir()* iterator now | |
| supports the context manager protocol. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| On Linux, *os.urandom()* now blocks until the system urandom entropy pool is initialized to increase the | |
| security. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The Linux *getrandom()* syscall (get random bytes) is now exposed as the new *os.getrandom()* function. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| `re <https://docs.python.org/3.6/whatsnew/3.6.html#re>`_ |
+--------------------------------------------------------------------------------------------------------------+----------------+
| Added support of modifier spans in regular expressions. Examples: *'(?i:p)ython'* matches 'python' and | |
| 'Python', but not 'PYTHON'; *'(?i)g(?-i:v)r'* matches *'GvR'* and *'gvr'*, but not *'GVR'*. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| Match object groups can be accessed by *__getitem__*, which is equivalent to *group()*. So *mo['name']* is | |
| now equivalent to *mo.group('name')*. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| Match objects now support index-like objects as group indices. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| `socket <https://docs.python.org/3.6/whatsnew/3.6.html#socket>`_ |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The *ioctl()* function now supports the *SIO_LOOPBACK_FAST_PATH* control code. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The *getsockopt()* constants *SO_DOMAIN* , *SO_PROTOCOL*, *SO_PEERSEC* , and *SO_PASSSEC* are now supported. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The *setsockopt()* now supports the *setsockopt(level, optname, None, optlen: int)* form. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The socket module now supports the address family *AF_ALG* to interface with Linux Kernel crypto API. | |
| *ALG_*, *SOL_ALG* and *sendmsg_afalg()* were added. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| New Linux constants *TCP_USER_TIMEOUT* and *TCP_CONGESTION* were added. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| `ssl <https://docs.python.org/3.6/whatsnew/3.6.html#ssl>`_ |
+--------------------------------------------------------------------------------------------------------------+----------------+
| ssl supports OpenSSL 1.1.0. The minimum recommend version is 1.0.2. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| 3DES has been removed from the default cipher suites and ChaCha20 Poly1305 cipher suites have been added. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| *SSLContext* has better default configuration for options and ciphers. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| SSL session can be copied from one client-side connection to another with the new *SSLSession* class. TLS | |
| session resumption can speed up the initial handshake, reduce latency and improve performance. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The new *get_ciphers()* method can be used to get a list of enabled ciphers in order of cipher priority. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| All constants and flags have been converted to *IntEnum* and *IntFlags*. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| Server and client-side specific TLS protocols for *SSLContext* were added. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| Added *SSLContext.post_handshake_auth* to enable and *ssl.SSLSocket.verify_client_post_handshake()* to | |
| initiate TLS 1.3 post-handshake authentication. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| `struct <https://docs.python.org/3.6/whatsnew/3.6.html#struct>`_ | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| now supports IEEE 754 half-precision floats via the 'e' format specifier. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| `sys <https://docs.python.org/3.6/whatsnew/3.6.html#sys>`_ | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The new *getfilesystemencodeerrors()* function returns the name of the error mode used to convert between | |
| Unicode filenames and bytes filenames. | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| `zlib <https://docs.python.org/3.6/whatsnew/3.6.html#zlib>`_ | |
+--------------------------------------------------------------------------------------------------------------+----------------+
| The *compress()* and *decompress()* functions now accept keyword arguments | |
+--------------------------------------------------------------------------------------------------------------+----------------+
.. _python_37:
Python 3.7
==========
New Features:
.. table::
:widths: 20 60 20
+--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
| **Feature** | **Status** |
+--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
| `PEP 538 <https://www.python.org/dev/peps/pep-0538/>`_ | Coercing the legacy C locale to a UTF-8 based | |
| | locale | |
+--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
| `PEP 539 <https://www.python.org/dev/peps/pep-0539/>`_ | A New C-API for Thread-Local Storage in CPython | |
+--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
| `PEP 540 <https://www.python.org/dev/peps/pep-0540/>`_ | UTF-8 mode | |
+--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
| `PEP 552 <https://www.python.org/dev/peps/pep-0552/>`_ | Deterministic pyc | |
+--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
| `PEP 553 <https://www.python.org/dev/peps/pep-0553/>`_ | Built-in ``breakpoint()`` | |
+--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
| `PEP 557 <https://www.python.org/dev/peps/pep-0557/>`_ | Data Classes | |
+--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
| `PEP 560 <https://www.python.org/dev/peps/pep-0560/>`_ | Core support for typing module and generic types | |
+--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
| `PEP 562 <https://www.python.org/dev/peps/pep-0562/>`_ | Module ``__getattr__`` and ``__dir__`` | Partial |
+--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
| `PEP 563 <https://www.python.org/dev/peps/pep-0563/>`_ | Postponed Evaluation of Annotations | |
+--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
| `PEP 564 <https://www.python.org/dev/peps/pep-0564/>`_ | Time functions with nanosecond resolution | Partial [#ftimenanosec]_ |
+--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
| `PEP 565 <https://www.python.org/dev/peps/pep-0565/>`_ | Show DeprecationWarning in ``__main__`` | |
+--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
| `PEP 567 <https://www.python.org/dev/peps/pep-0567/>`_ | Context Variables | |
+--------------------------------------------------------+--------------------------------------------------+--------------------------------------+
Other Language Changes:
.. table::
:widths: 90 10
+-----------------------------------------------------------------------------------------------------------------+----------------+
| ``async`` and ``await`` are now reserved keywords | Complete |
+-----------------------------------------------------------------------------------------------------------------+----------------+
| ``dict`` objects must preserve insertion-order | |
+-----------------------------------------------------------------------------------------------------------------+----------------+
| More than 255 arguments can now be passed to a function; a function can now have more than 255 parameters | |
+-----------------------------------------------------------------------------------------------------------------+----------------+
| ``bytes.fromhex()`` and ``bytearray.fromhex()`` now ignore all ASCII whitespace, not only spaces | |
+-----------------------------------------------------------------------------------------------------------------+----------------+
| ``str``, ``bytes``, and ``bytearray`` gained support for the new ``isascii()`` method, which can be used to | |
| test if a string or bytes contain only the ASCII characters | |
+-----------------------------------------------------------------------------------------------------------------+----------------+
| ``ImportError`` now displays module name and module ``__file__`` path when ``from ... import ...`` fails | |
+-----------------------------------------------------------------------------------------------------------------+----------------+
| Circular imports involving absolute imports with binding a submodule to a name are now supported | |
+-----------------------------------------------------------------------------------------------------------------+----------------+
| ``object.__format__(x, '')`` is now equivalent to ``str(x)`` rather than ``format(str(self), '')`` | |
+-----------------------------------------------------------------------------------------------------------------+----------------+
| In order to better support dynamic creation of stack traces, ``types.TracebackType`` can now be | |
| instantiated from Python code, and the ``tb_next`` attribute on tracebacks is now writable | |
+-----------------------------------------------------------------------------------------------------------------+----------------+
| When using the ``-m`` switch, ``sys.path[0]`` is now eagerly expanded to the full starting directory path, | |
| rather than being left as the empty directory (which allows imports from the current working directory | |
| at the time when an import occurs) | |
+-----------------------------------------------------------------------------------------------------------------+----------------+
| The new ``-X importtime`` option or the ``PYTHONPROFILEIMPORTTIME`` environment variable can be used to | |
| show the timing of each module import | |
+-----------------------------------------------------------------------------------------------------------------+----------------+
Changes to built-in modules:
.. table::
:widths: 90 10
+------------------------------------------------------------------------------------------------------------+----------------+
| `asyncio <https://docs.python.org/3/whatsnew/3.7.html#asyncio>`_ | |
+------------------------------------------------------------------------------------------------------------+----------------+
| Too many to list | |
+------------------------------------------------------------------------------------------------------------+----------------+
| `gc <https://docs.python.org/3/whatsnew/3.7.html#gc>`_ | |
+------------------------------------------------------------------------------------------------------------+----------------+
| New features include *gc.freeze()*, *gc.unfreeze()*, *gc-get_freeze_count* | |
+------------------------------------------------------------------------------------------------------------+----------------+
| `math <https://docs.python.org/3/whatsnew/3.7.html#math>`_ | |
+------------------------------------------------------------------------------------------------------------+----------------+
| math.remainder() added to implement IEEE 754-style remainder | |
+------------------------------------------------------------------------------------------------------------+----------------+
| `re <https://docs.python.org/3/whatsnew/3.7.html#re>`_ | |
+------------------------------------------------------------------------------------------------------------+----------------+
| A number of tidy up features including better support for splitting on empty strings and copy support for | |
| compiled expressions and match objects | |
+------------------------------------------------------------------------------------------------------------+----------------+
| `sys <https://docs.python.org/3/whatsnew/3.7.html#sys>`_ | |
+------------------------------------------------------------------------------------------------------------+----------------+
| sys.breakpointhook() added. sys.get(/set)_coroutine_origin_tracking_depth() added | |
+------------------------------------------------------------------------------------------------------------+----------------+
| `time <https://docs.python.org/3/whatsnew/3.7.html#time>`_ | |
+------------------------------------------------------------------------------------------------------------+----------------+
| Mostly updates to support nanosecond resolution in PEP564, see above | |
+------------------------------------------------------------------------------------------------------------+----------------+
.. rubric:: Notes
.. [#ftimenanosec] Only :func:`time.time_ns` is implemented.
.. _python_38:
Python 3.8
==========
Python 3.8.0 (final) was released on the 14 October 2019. The Features for 3.8
are defined in `PEP 569 <https://www.python.org/dev/peps/pep-0569/#id9>`_ and
a detailed description of the changes can be found in `What's New in Python
3.8. <https://docs.python.org/3/whatsnew/3.8.html>`_
.. table::
:widths: 20 60 20
+--------------------------------------------------------+---------------------------------------------------+---------------+
| **Features** | **Status** |
+--------------------------------------------------------+---------------------------------------------------+---------------+
| `PEP 570 <https://www.python.org/dev/peps/pep-0570/>`_ | Positional-only arguments | |
+--------------------------------------------------------+---------------------------------------------------+---------------+
| `PEP 572 <https://www.python.org/dev/peps/pep-0572/>`_ | Assignment Expressions | Complete |
+--------------------------------------------------------+---------------------------------------------------+---------------+
| `PEP 574 <https://www.python.org/dev/peps/pep-0574/>`_ | Pickle protocol 5 with out-of-band data | |
+--------------------------------------------------------+---------------------------------------------------+---------------+
| `PEP 578 <https://www.python.org/dev/peps/pep-0578/>`_ | Runtime audit hooks | |
+--------------------------------------------------------+---------------------------------------------------+---------------+
| `PEP 587 <https://www.python.org/dev/peps/pep-0587/>`_ | Python Initialization Configuration | |
+--------------------------------------------------------+---------------------------------------------------+---------------+
| `PEP 590 <https://www.python.org/dev/peps/pep-0590/>`_ | Vectorcall: a fast calling protocol for CPython | |
+--------------------------------------------------------+---------------------------------------------------+---------------+
| **Miscellaneous** |
+------------------------------------------------------------------------------------------------------------+---------------+
| f-strings support = for self-documenting expressions and debugging | Complete |
+------------------------------------------------------------------------------------------------------------+---------------+
Other Language Changes:
.. table::
:widths: 90 10
+------------------------------------------------------------------------------------------------------------+-------------+
| A *continue* statement was illegal in the *finally* clause due to a problem with the implementation. In | Complete |
| Python 3.8 this restriction was lifted | |
+------------------------------------------------------------------------------------------------------------+-------------+
| The *bool*, *int* , and *fractions.Fraction* types now have an *as_integer_ratio()* method like that found | |
| in *float* and *decimal.Decimal* | |
+------------------------------------------------------------------------------------------------------------+-------------+
| Constructors of *int*, *float* and *complex* will now use the *__index__()* special method, if available | |
| and the corresponding method *__int__()*, *__float__()* or *__complex__()* is not available | |
+------------------------------------------------------------------------------------------------------------+-------------+
| Added support of *\N{name}* escapes in regular expressions | |
+------------------------------------------------------------------------------------------------------------+-------------+
| Dict and dictviews are now iterable in reversed insertion order using *reversed()* | |
+------------------------------------------------------------------------------------------------------------+-------------+
| The syntax allowed for keyword names in function calls was further restricted. In particular, | |
| f((keyword)=arg) is no longer allowed | |
+------------------------------------------------------------------------------------------------------------+-------------+
| Generalized iterable unpacking in yield and return statements no longer requires enclosing parentheses | |
+------------------------------------------------------------------------------------------------------------+-------------+
| When a comma is missed in code such as [(10, 20) (30, 40)], the compiler displays a SyntaxWarning with a | |
| helpful suggestion | |
+------------------------------------------------------------------------------------------------------------+-------------+
| Arithmetic operations between subclasses of *datetime.date* or *datetime.datetime* and *datetime.timedelta*| |
| objects now return an instance of the subclass, rather than the base class | |
+------------------------------------------------------------------------------------------------------------+-------------+
| When the Python interpreter is interrupted by *Ctrl-C (SIGINT)* and the resulting *KeyboardInterrupt* | |
| exception is not caught, the Python process now exits via a SIGINT signal or with the correct exit code | |
| such that the calling process can detect that it died due to a *Ctrl-C* | |
+------------------------------------------------------------------------------------------------------------+-------------+
| Some advanced styles of programming require updating the *types.CodeType* object for an existing function | |
+------------------------------------------------------------------------------------------------------------+-------------+
| For integers, the three-argument form of the pow() function now permits the exponent to be negative in the | |
| case where the base is relatively prime to the modulus | |
+------------------------------------------------------------------------------------------------------------+-------------+
| Dict comprehensions have been synced-up with dict literals so that the key is computed first and the value | |
| second | |
+------------------------------------------------------------------------------------------------------------+-------------+
| The *object.__reduce__()* method can now return a tuple from two to six elements long | |
+------------------------------------------------------------------------------------------------------------+-------------+
Changes to built-in modules:
.. table::
:widths: 90 10
+------------------------------------------------------------------------------------------------------------+-------------+
| `asyncio <https://docs.python.org/3/whatsnew/3.8.html#asyncio>`_ |
+------------------------------------------------------------------------------------------------------------+-------------+
| *asyncio.run()* has graduated from the provisional to stable API | Complete |
+------------------------------------------------------------------------------------------------------------+-------------+
| Running *python -m asyncio* launches a natively async REPL | |
+------------------------------------------------------------------------------------------------------------+-------------+
| The exception *asyncio.CancelledError* now inherits from *BaseException* rather than *Exception* and no | Complete |
| longer inherits from *concurrent.futures.CancelledError* | |
+------------------------------------------------------------------------------------------------------------+-------------+
| Added *asyncio.Task.get_coro()* for getting the wrapped coroutine within an *asyncio.Task* | |
+------------------------------------------------------------------------------------------------------------+-------------+
| Asyncio tasks can now be named, either by passing the name keyword argument to *asyncio.create_task()* or | |
| the *create_task()* event loop method, or by calling the *set_name()* method on the task object | |
+------------------------------------------------------------------------------------------------------------+-------------+
| Added support for Happy Eyeballs to *asyncio.loop.create_connection()*. To specify the behavior, two new | |
| parameters have been added: *happy_eyeballs_delay* and interleave. | |
+------------------------------------------------------------------------------------------------------------+-------------+
| `gc <https://docs.python.org/3/whatsnew/3.8.html#gc>`_ |
+------------------------------------------------------------------------------------------------------------+-------------+
| *get_objects()* can now receive an optional generation parameter indicating a generation to get objects | |
| from. (Note, though, that while *gc* is a built-in, *get_objects()* is not implemented for MicroPython) | |
+------------------------------------------------------------------------------------------------------------+-------------+
| `math <https://docs.python.org/3/whatsnew/3.8.html#math>`_ |
+------------------------------------------------------------------------------------------------------------+-------------+
| Added new function *math.dist()* for computing Euclidean distance between two points | |
+------------------------------------------------------------------------------------------------------------+-------------+
| Expanded the *math.hypot()* function to handle multiple dimensions | |
+------------------------------------------------------------------------------------------------------------+-------------+
| Added new function, *math.prod()*, as analogous function to *sum()* that returns the product of a "start" | |
| value (default: 1) times an iterable of numbers | |
+------------------------------------------------------------------------------------------------------------+-------------+
| Added two new combinatoric functions *math.perm()* and *math.comb()* | |
+------------------------------------------------------------------------------------------------------------+-------------+
| Added a new function *math.isqrt()* for computing accurate integer square roots without conversion to | |
| floating point | |
+------------------------------------------------------------------------------------------------------------+-------------+
| The function *math.factorial()* no longer accepts arguments that are not int-like | Complete |
+------------------------------------------------------------------------------------------------------------+-------------+
| `sys <https://docs.python.org/3/whatsnew/3.8.html#sys>`_ |
+------------------------------------------------------------------------------------------------------------+-------------+
| Add new *sys.unraisablehook()* function which can be overridden to control how "unraisable exceptions" | |
| are handled | |
+------------------------------------------------------------------------------------------------------------+-------------+
.. _python_39:
Python 3.9
==========
Python 3.9.0 (final) was released on the 5th October 2020. The Features for 3.9 are
defined in `PEP 596 <https://www.python.org/dev/peps/pep-0596/#features-for-3-9>`_
and a detailed description of the changes can be found in
`What's New in Python 3.9 <https://docs.python.org/3/whatsnew/3.9.html>`_
.. table::
:widths: 20 60 20
+--------------------------------------------------------+----------------------------------------------------+----------------------+
| **Features** | | **Status** |
+--------------------------------------------------------+----------------------------------------------------+----------------------+
| `PEP 573 <https://www.python.org/dev/peps/pep-0573/>`_ | Fast access to module state from methods of C | Not relevant |
| | extension types | |
+--------------------------------------------------------+----------------------------------------------------+----------------------+
| `PEP 584 <https://www.python.org/dev/peps/pep-0584/>`_ | Union operators added to dict | Complete [#pep584]_ |
+--------------------------------------------------------+----------------------------------------------------+----------------------+
| `PEP 585 <https://www.python.org/dev/peps/pep-0584/>`_ | Type hinting generics in standard collections | |
+--------------------------------------------------------+----------------------------------------------------+----------------------+
| `PEP 593 <https://www.python.org/dev/peps/pep-0593/>`_ | Flexible function and variable annotations | |
+--------------------------------------------------------+----------------------------------------------------+----------------------+
| `PEP 602 <https://www.python.org/dev/peps/pep-0602/>`_ | CPython adopts an annual release cycle. Instead of | Not relevant |
| | annual, aiming for two month release cycle | |
+--------------------------------------------------------+----------------------------------------------------+----------------------+
| `PEP 614 <https://www.python.org/dev/peps/pep-0614/>`_ | Relaxed grammar restrictions on decorators | |
+--------------------------------------------------------+----------------------------------------------------+----------------------+
| `PEP 615 <https://www.python.org/dev/peps/pep-0615/>`_ | The IANA Time Zone Database is now present in the | |
| | standard library in the zoneinfo module | |
+--------------------------------------------------------+----------------------------------------------------+----------------------+
| `PEP 616 <https://www.python.org/dev/peps/pep-0616/>`_ | String methods to remove prefixes and suffixes | |
+--------------------------------------------------------+----------------------------------------------------+----------------------+
| `PEP 617 <https://www.python.org/dev/peps/pep-0617/>`_ | CPython now uses a new parser based on PEG | Not relevant |
+--------------------------------------------------------+----------------------------------------------------+----------------------+
Other Language Changes:
.. table::
:widths: 90 10
+-------------------------------------------------------------------------------------------------------------+---------------+
| *__import__()* now raises *ImportError* instead of *ValueError* | Complete |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Python now gets the absolute path of the script filename specified on the command line (ex: *python3* | |
| *script.py*): the *__file__* attribute of the *__main__* module became an absolute path, rather than a | |
| relative path | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| By default, for best performance, the errors argument is only checked at the first encoding/decoding error | |
| and the encoding argument is sometimes ignored for empty strings | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| *"".replace("", s, n)* now returns *s* instead of an empty string for all non-zero n. It is now consistent | |
| with *"".replace("", s)* | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Any valid expression can now be used as a decorator. Previously, the grammar was much more restrictive | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Parallel running of *aclose()* / *asend()* / *athrow()* is now prohibited, and *ag_running* now reflects | |
| the actual running status of the async generator | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Unexpected errors in calling the *__iter__* method are no longer masked by TypeError in the in operator and | |
| functions contains(), indexOf() and countOf() of the operator module | |
+-------------------------------------------------------------------------------------------------------------+---------------+
| Unparenthesized lambda expressions can no longer be the expression part in an if clause in comprehensions | |
| and generator expressions | |
+-------------------------------------------------------------------------------------------------------------+---------------+
Changes to built-in modules:
.. table::
:widths: 90 10
+---------------------------------------------------------------------------------------------------------------+---------------+
| `asyncio <https://docs.python.org/3/whatsnew/3.9.html#asyncio>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Due to significant security concerns, the reuse_address parameter of *asyncio.loop.create_datagram_endpoint()*| |
| is no longer supported | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Added a new coroutine *shutdown_default_executor()* that schedules a shutdown for the default executor that | |
| waits on the *ThreadPoolExecutor* to finish closing. Also, *asyncio.run()* has been updated to use the new | |
| coroutine. | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Added *asyncio.PidfdChildWatcher*, a Linux-specific child watcher implementation that polls process file | |
| descriptors | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| added a new *coroutine asyncio.to_thread()* | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| When cancelling the task due to a timeout, *asyncio.wait_for()* will now wait until the cancellation is | |
| complete also in the case when timeout is <= 0, like it does with positive timeouts | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| *asyncio* now raises *TyperError* when calling incompatible methods with an *ssl.SSLSocket* socket | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| `gc <https://docs.python.org/3/whatsnew/3.9.html#gc>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Garbage collection does not block on resurrected objects | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Added a new function *gc.is_finalized()* to check if an object has been finalized by the garbage collector | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| `math <https://docs.python.org/3/whatsnew/3.9.html#math>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Expanded the *math.gcd()* function to handle multiple arguments. Formerly, it only supported two arguments | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Added *math.lcm()*: return the least common multiple of specified arguments | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Added *math.nextafter()*: return the next floating-point value after x towards y | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Added *math.ulp()*: return the value of the least significant bit of a float | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| `os <https://docs.python.org/3/whatsnew/3.9.html#os>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Exposed the Linux-specific *os.pidfd_open()* and *os.P_PIDFD* | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| The *os.unsetenv()* function is now also available on Windows | Complete |
+---------------------------------------------------------------------------------------------------------------+---------------+
| The *os.putenv()* and *os.unsetenv()* functions are now always available | Complete |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Added *os.waitstatus_to_exitcode()* function: convert a wait status to an exit code | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| `random <https://docs.python.org/3/whatsnew/3.9.html#random>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Added a new *random.Random.randbytes* method: generate random bytes | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| `sys <https://docs.python.org/3/whatsnew/3.9.html#sys>`_ |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Added a new *sys.platlibdir* attribute: name of the platform-specific library directory | |
+---------------------------------------------------------------------------------------------------------------+---------------+
| Previously, *sys.stderr* was block-buffered when non-interactive. Now stderr defaults to always being | |
| line-buffered | |
+---------------------------------------------------------------------------------------------------------------+---------------+
.. rubric:: Notes
.. [#pep584] PEP 584 ``dict`` union operator is only available on MicroPython builds with ``MICROPY_CPYTHON_COMPAT`` enabled.
.. _esp32_general:
General information about the ESP32 port
========================================
The ESP32 is a popular WiFi and Bluetooth enabled System-on-Chip (SoC) by
Espressif Systems.
Multitude of boards
-------------------
There is a multitude of modules and boards from different sources which carry
the ESP32 chip. MicroPython tries to provide a generic port which would run on
as many boards/modules as possible, but there may be limitations. Espressif
development boards are taken as reference for the port (for example, testing is
performed on them). For any board you are using please make sure you have a
datasheet, schematics and other reference materials so you can look up any
board-specific functions.
To make a generic ESP32 port and support as many boards as possible the
following design and implementation decision were made:
* GPIO pin numbering is based on ESP32 chip numbering. Please have the manual/pin
diagram of your board at hand to find correspondence between your board pins and
actual ESP32 pins.
* All pins are supported by MicroPython but not all are usable on any given board.
For example pins that are connected to external SPI flash should not be used,
and a board may only expose a certain selection of pins.
Technical specifications and SoC datasheets
-------------------------------------------
The datasheets and other reference material for ESP32 chip are available
from the vendor site: https://www.espressif.com/en/support/download/documents?keys=esp32 .
They are the primary reference for the chip technical specifications, capabilities,
operating modes, internal functioning, etc.
For your convenience, some of technical specifications are provided below:
* Architecture: Xtensa Dual-Core 32-bit LX6
* CPU frequency: up to 240MHz
* Total RAM available: 528KB (part of it reserved for system)
* BootROM: 448KB
* Internal FlashROM: none
* External FlashROM: code and data, via SPI Flash; usual size 4MB
* GPIO: 34 (GPIOs are multiplexed with other functions, including
external FlashROM, UART, etc.)
* UART: 3 RX/TX UART (no hardware handshaking), one TX-only UART
* SPI: 4 SPI interfaces (one used for FlashROM)
* I2C: 2 I2C (bitbang implementation available on any pins)
* I2S: 2
* ADC: 12-bit SAR ADC up to 18 channels
* DAC: 2 8-bit DACs
* RMT: 8 channels allowing accurate pulse transmit/receive
* Programming: using BootROM bootloader from UART - due to external FlashROM
and always-available BootROM bootloader, the ESP32 is not brickable
For more information see the ESP32 datasheet: https://www.espressif.com/sites/default/files/documentation/esp32_datasheet_en.pdf
MicroPython is implemented on top of the ESP-IDF, Espressif's development
framework for the ESP32. This is a FreeRTOS based system. See the
`ESP-IDF Programming Guide <https://docs.espressif.com/projects/esp-idf/en/latest/index.html>`_
for details.