pycc-next 0.3.0: a real PyPI package, unchanged, compiled and passing its own tests
Two days after 0.2.0, 0.3.0 is out. This one is about the code people actually write:
several files, imports, dataclasses, Counter, itertools. And one package straight off
PyPI.
🏁 The scorecard
| goal | target | result |
|---|---|---|
| multi-module | import m, packages, relative imports, one compiled unit |
native, --non-strict and --ext; cross-module calls are direct calls |
| stdlib | dataclasses, enum, functools.cache, deque/Counter/defaultdict, itertools | all native in strict mode, oracle-tested in three modes |
| corpus, strict | >= 150/200 real programs unchanged | 161/200, 0 wrong (was 124) |
| corpus, non-strict | 200/200 with >= 98% native | 200/200, 98.1% native, 0 wrong |
| real package | a PyPI package passes its own tests as --ext |
diff-match-patch: 28/28, CI-gated |
| platforms | Linux aarch64 archive | three archives: Linux x86_64, Linux aarch64, macOS arm64 |
📦 A real package, not a demo
I picked diff-match-patch: Google’s diff/match/patch algorithms, Apache-2.0, pure Python,
and it ships its own test suite. Every module builds unchanged with
pycc build --ext --non-strict. All 40 functions and methods compile, none fall back to
CPython, and its 28 tests pass against the compiled .so files. CI downloads the pinned sdist,
checks its hash, builds, swaps the extensions in, and runs the suite on every push.
The honest part: its speed test runs 1.7x faster, not 30x. The package has no type
annotations, so its text parameters stay CPython str objects and most of the work still goes
through CPython’s string machinery. Untyped code gets correctness first and modest speed. Typed
code gets the 20-680x numbers below.
🧩 Imports that are really imports
import helper, from pkg.sub import f, from ..util import g, __init__.py: each
module keeps its own namespace, its body runs once in CPython’s import order, and
helper.add(1, 2) compiles to a direct native call. Two modules can both define main. An
uncaught exception prints as helper.E: msg, like CPython. Circular imports are refused with a
clear error for now.
🧰 The stdlib you actually use
@dataclasswith defaults,field(default_factory=...),order=True,frozen=TrueEnum,IntEnum,functools.cache/lru_cachedeque,Counter,defaultdictitertools.product/permutations/combinations/accumulate/chain/islice/groupby/..., as lazy native iterators
A BFS over a grid with deque runs 6.8x faster than CPython. A word count with Counter
is about 4x faster.
⚡ Numbers
python3 bench/run.py, best of 7, on a shared 4-core box:
| program | vs CPython | vs Rust |
|---|---|---|
| fib | 19.9x faster | 2.0x |
| mandelbrot | 43.9x faster | 0.96x |
| nbody | 29.4x faster | 2.6x |
| sieve | 25.0x faster | 1.9x |
| spectral_norm | 46.1x faster | 1.7x |
| collections_bfs | 6.8x faster | 2.9x |
| stencil2d (numpy) | 677x faster | 2.3x |
| prefix_sum_np (numpy) | 78x faster | 1.7x |
🛡️ What review caught
Every push went through security review, and it earned its keep:
- a
defaultdictfactory that drops the last reference to its own dict (use-after-free); - a module name that could reach generated C unescaped;
- a
__str__with the wrong signature that the checker skipped but codegen still called.
My own pre-release pass also found a leak: every fresh str passed to a CPython call lost
about 40 bytes. json.dumps(str(i)) in a loop grew by megabytes. It’s fixed, and 10 new
constant-memory probes guard that boundary.
🔭 Next
Untyped code is correct but only modestly faster, so the next big lever is inferring native
str and list types through untyped functions. And there are still 23 non-strict statements
that run in CPython. The roadmap is in docs/ROADMAP.md.