Skip to content

Lodum Roadmap

This document outlines the planned future enhancements for lodum. Our goal is to maintain the library's high performance while expanding its utility in the Python ecosystem.

0. Completed Milestones

v0.2.0 — First Public Release

  • AST Bytecode Compiler: Replaced string-based exec with a robust and maintainable AST generation engine using a specialized DSL.
  • Modular Architecture: Refactored the core engine into decoupled compiler/ and handlers/ subpackages.
  • Thread Safety: Implemented thread-local Context management with a lock-free fast path for cache performance.
  • Centralized Schema API: Promoted lodum.schema() as the primary entry point for JSON Schema generation.
  • Extended Type Support: Added native support for numpy, pandas, polars, and numerous standard library types.
  • Community & Governance: Established Code of Conduct, Security policy, and standard issue/PR templates.

v0.3.0 — WASM & Concurrency Support

  • Robust Concurrency Detection: Implemented active probing for native threading capability in Pyodide/WASM environments.
  • WASM Shims: Introduced SequentialThread, DummyLock, and DummyLocal to ensure graceful degradation in restricted environments.
  • Unified Threading API: Refactored core internals to use a cross-platform concurrency abstraction, enabling the same code to run on high-concurrency servers and in single-threaded browsers.

1. Current Milestone (v0.4.0+)

Streaming Support (Plan)

  • Goal: Enable processing of datasets larger than memory.
  • Implementation: Add dump_stream(obj, stream) and load_stream(cls, stream) to the IDumper/ILoader protocols.
  • Target Formats: JSON (via ijson or similar), MsgPack, CBOR.

Partial Serialization (Plan)

  • Goal: Allow users to serialize only a subset of fields or only fields that have changed.
  • API Idea: dumps(obj, include={'id', 'name'}) or field(serialize_if_changed=True).

2. Short-Term Goals (v0.5.0+)

Data Science & ML Optimization (Plan)

  • Goal: Enable efficient handling of multi-gigabyte quantized datasets in memory-constrained environments (Pyodide/WASM).
  • Key Components:
    • Lazy Loading: Two-tier on-demand materialization (Tensor proxies and Complex Object proxies).
    • Quantization Handlers: Native AST-generated loaders for 4-bit, 8-bit, and bit-packed data.
    • Zero-Copy Architecture: Integrated mmap and memoryview views to prevent memory expansion.

Framework Integrations

  • FastAPI (Plan): Create a LodumResponse class and a Depends provider for automatic parsing.
  • OpenAPI Support (Plan): Utilities to convert lodum schemas directly into OpenAPI/Swagger specifications.

3. Medium-Term Goals (v0.7.0+)

Advanced Schema Features

  • Schema-to-Code (Plan): A CLI tool to generate @lodum classes from an existing JSON Schema.

Compiler Enhancements

  • Native Handlers (Plan): Explore compiling the generated bytecode handlers into C via Cython or MyPyC for even greater speed.

4. Long-Term Vision (v1.0.0)

Ecosystem & Tooling

  • Protocol Buffers (Plan): Native Protobuf support via the AST compiler.
  • IDE Plugin (Plan): A VS Code / PyCharm plugin to auto-generate field() definitions and validate schema compliance in real-time.

Contributing

We welcome contributions on any of these items! Please open an issue to discuss your implementation plan before submitting a PR.