Import common modules into Pycardium

added 1-Necessary 3-Pycardium 8-Easy labels

I already enabled a few of the micropython specific u* modules in the following commits:

1572a35c: ubinascii
3760b6b2: uheapq
9b2cf5ca: ure
435883e6: ujson
0e5e760f: urandom

Some more analysis of micropython-lib:

category: "useless as of now"

binhex (only a stub)
csv (only a stub)
ipaddress (only a stub)
math (only a stub)
numbers (only a stub)

category: "not possible directly"

base64 (needs re (pcre) + struct + binascii)
datetime (needs time + math) but only math.modf which could be stubbed out ("Return the fractional and integer parts of x. Both results carry the sign of x and are float")

struct can be replaced by ustruct which is already enabled IIRC. Same thing for binascii and ubinascii. We also have ure which we should probably reexport as re so it is easier to find.

assigned to @wink

>>> from binascii import hexlify
>>> from binascii import unhexlify
>>> a = 'FF'                            
>>> x = unhexlify(a)
>>> y = hexlify(x)
>>> y
b'ff'

works as expected, didn't try base64 yet - unclear why they have binascii.py

>>> from itertools import tee
>>> a = [0,1,2]
>>> x = tee(a, 2)
>>> idx = 0
>>> for n in x[0]:
...   print(n)
... 
0
1
2
>>> for n in x[1]:
...   print(n)
... 
>>> for n in x[0]:
...   print(n)
... 
>>> a
[0, 1, 2]
>>> x
[<iterator>, <iterator>]
>>> for n in x[0]:
...   print(n)

# compare to cpython
>>> from itertools import tee
>>> a = [0,1,2]
>>> x = tee(a, 2)
>>> x
(<itertools.tee object at 0x7fd4f93ff8c0>, <itertools.tee object at 0x7fd4f93ff950>)
>>> for n in x[0]:
...   print(n)
... 
0
1
2
>>> for n in x[1]:
...   print(n)
... 
0
1
2

Needs more testing, but it wouldn't iterate the 2nd iterator, didn't check the other functions

>>> from functools import partial
>>> b2 = partial(int, base=2)
>>> x = b2('10010')
Traceback (most recent call last):
  File "<stdin>", in <module>
  File "functools.py", in _partial
TypeError: function doesn't take keyword arguments

this seems downright broken

# just using 'hhl' from the manual example
>>> from struct import Struct
>>> s = Struct('hhl')
>>> v = (1,2,3)
>>> s.pack(*v)
b'\x01\x00\x02\x00\x03\x00\x00\x00'
>>> s.pack(1,2,3)
b'\x01\x00\x02\x00\x03\x00\x00\x00'
>>> a = s.pack(1,2,3)
>>> s.unpack( a)
(1, 2, 3)

# explicit > <
>>> s = Struct('>hhl')
>>> a = s.pack(1,2,3)
>>> a
b'\x00\x01\x00\x02\x00\x00\x00\x03'
>>> s = Struct('<hhl')
>>> a = s.pack(1,2,3)
>>> a
b'\x01\x00\x02\x00\x03\x00\x00\x00'


# compare to cpython:
>>> a = pack('hhl', *(1,2,3))
>>> a
'\x01\x00\x02\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00'
>>> a = pack('>hhl', *(1,2,3))
>>> a
'\x00\x01\x00\x02\x00\x00\x00\x03'
>>> a = pack('<hhl', *(1,2,3))
>>> a
'\x01\x00\x02\x00\x03\x00\x00\x00'





>>> s2 = Struct('ci')
Traceback (most recent call last):
  File "<stdin>", in <module>
  File "struct.py", in __init__
ValueError: bad typecode

>>> s2 = Struct('ci', '*')
Traceback (most recent call last):
  File "<stdin>", in <module>
TypeError: function takes 2 positional arguments but 3 were given

# again, cpython:
>>> a = pack('ci', '*', 0x12131415)
>>> a
'*\x00\x00\x00\x15\x14\x13\x12'

Hi!

itertools.tee

Yes, tee does not behave correctly. The implementation in micropython-lib is not tested in their test-suite and from taking a look at it, I can't make out how it is supposed to ever produce correct results ... Still, the other iterator tools would be quite nice to have so I'd argue to include the module anyway and just add a warning in our docs. Maybe at some point upstream a fix ...

functools.patial

From what I can see, the example you showed is not an issue directly related to functools. It is the int() function in MicroPython not having a base= parameter. Would you mind checking it again with another function?

struct.*

FIY, MicroPython's struct module also has .pack() & .unpack(). They are re-exported from ustruct.

The inconsistencies you discovered are weird ... I'll take a closer look. The missing c specifier is probably just an optimization because it is only used rarely.

As I wrote in IRC today, I would have liked to include pickle, but the provided implementation is just a dummy. So we'll have to search for an alternative ...

added 1-Required label and removed 1-Necessary label

@wink, what's your status on this?

Sorry, didn't have much time since last week.

I want to continue but maybe the ticket should be split a little.

I guess I should commit "just binascii" (including reimport from ubinascii and double-checking compatibility) and then we merge that.

And then split off investigation of the other modules. or just do them one after the other. I tried to do all at the same time, but it's not feasible with the state of micropython-lib.

But then it's at least clearer how to continue and anyone else can join if they feel like it.

Can we start with itertools, functools, contextlib & string for now? I have no problem with tackling this issue in multiple merge-requests, do it as you see fit.

@wink, ping?

Yeah sorry, didn't manage to do that, feel free to unassign and maybe someone else has time. I don't think any code I wrote has been helpful here, maybe the findings I got were.

unassigned @wink

assigned to @rahix

mentioned in merge request !86 (merged)

marked the checklist item contextlib (ucontextlib) as completed

marked the checklist item collections (+ deque & defaultdict) as completed

marked the checklist item functools as completed

marked the checklist item itertools as completed

marked the checklist item string as completed

marked the checklist item struct as completed

binascii raises an assertion for some reason (which I couldn't reproduce manually) and for uuid we need to either strip down the C-Python version or write our own. If someone is willing to do continue work here, I would be very grateful!

unassigned @rahix

assigned to @rahix

I am trying to get binascii working atm. I'll comment again once I have something working and tested.

@hellerbarde, in lib/micropython/micropython-lib/binascii/binascii.py:41, there is an assertion. Last I checked, this was triggered when attempting to load the module.

Curiously, when pasting the offending lines into the REPL, the assertion is not triggered. I did not spend any more time trying to figure out what is going wrong ...

If you want to continue working on this, add a symlink to the above file in pycardium/modules/py/binascii.py and add it to pycardium/modules/py/meson.build. Then, the binascii module should be available in MicroPython (after flashing the new firmware).

Regarding uuid: I had stripped down C-Python's uuid and pushed that in b0348430. For some reason it seems I have never tested it properly as it isn't even included in the build right now. When trying to add this, mpy-cross raises an error related to big integer constants in the file (L25, L34).

Digging deeper, we have to rewrite this module from scratch: C-Python uses a 128-bit integer to represent the UUIDs but in MicroPython with have at most 63 bits available. This means the best approach is probably to store the UUID as a bytes() object.

unassigned @rahix

All right. I can import binascii fine. I'll try to investigate and make sure it works. And then I'll take a look at uuid

I have an UUID implementation almost ready. Binascii looks fine. PR hopefully incoming tonight.

marked the checklist item uuid ( micropython-lib does not yet implement this module, so we need a version from somewhere else. Maybe even contribute an implementation upstream?) as completed

marked the checklist item binascii as completed

closed

Import common modules into Pycardium

Designs

Child items ...

Activity