AsciiDoctrine Documentation

Michael R. Bernstein zopemaven@gmail.com

v0.1.0

Welcome to the official documentation for AsciiDoctrine, a native Python implementation of the AsciiDoc Language Specification.

AsciiDoctrine is designed to provide a structured, type-safe representation of AsciiDoc documents, perfect for building custom renderers, IDE plugins, and static site generator backends.

Key Capabilities

  • TCK Validation: Ensures standard-compliant rendering by validating its pipeline against the official AsciiDoc Technology Compatibility Kit (TCK) test suites.

  • Precise Source Coordinate Mapping: Enables robust IDE integrations, precise diagnostic errors, and syntactical formatting tools by attaching optional line and column coordinates to every structural block and inline element.

  • Attribute and Macro Resolution: Implements standard AsciiDoc dynamic attribute substitutions, conditional preprocessor passes, multi-file includes, and block/inline macros.

  • Docutils & Sphinx Power: Integrates cleanly with the Python documentation ecosystem by compiling AsciiDoc files natively into the Docutils component tree.

  • Wasm and Pyodide Portability: Runs seamlessly inside browsers and serverless sandboxes via pure-Python, zero-native-dependency Pyodide builds.

This documentation is itself authored in AsciiDoc and compiled using sphinx-asciidoctrine, a custom Sphinx extension built on top of AsciiDoctrine. You can view its source and integrate it into your own project via the sphinx-asciidoctrine repository.

Installation & Setup

Install the stable, released package from PyPI using pip:

pip install asciidoctrine

Quick Start

AsciiDoctrine implements a two-pass resolution pipeline. First, parse the raw source to a syntax-level AST. Then, resolve the AST to a semantic, queryable Abstract Semantic Graph (ASG):

from asciidoctrine import parse_to_ast
from asciidoctrine.resolver import ASGResolver

source = """
== Section Title
This is a *bold* word in a paragraph.
"""

# 1. Parse raw source to syntax-level AST
ast = parse_to_ast(source)

# 2. Resolve to semantic ASG (resolves attributes, includes, and filters comments)
resolver = ASGResolver(ast)
asg = resolver.resolve(ast)

# Iterate through sections in the ASG blocks list
for block in asg.get("blocks", []):
    if block.get("name") == "section":
        # The title is a list of inline nodes in the ASG schema
        title_nodes = block.get("title", [])
        title_text = "".join(node.get("value", "") for node in title_nodes if node.get("name") == "text")
        print(f"Found section: {title_text}")

ASG Representation

Calling asg.to_dict() yields a structured, spec-compliant representation matching the official AsciiDoc Language ASG schema:

{
  "name": "document",
  "type": "block",
  "blocks": [
    {
      "name": "section",
      "type": "block",
      "level": 1,
      "title": {
        "name": "title",
        "type": "inline",
        "inlines": [
          { "name": "text", "type": "string", "value": "Section Title" }
        ]
      },
      "blocks": [
        {
          "name": "paragraph",
          "type": "block",
          "inlines": [
            { "name": "text", "type": "string", "value": "This is a " },
            {
              "name": "span",
              "type": "inline",
              "variant": "strong",
              "form": "constrained",
              "inlines": [
                { "name": "text", "type": "string", "value": "bold" }
              ]
            },
            { "name": "text", "type": "string", "value": " word in a paragraph." }
          ]
        }
      ]
    }
  ],
  "attributes": {}
}

Project Guides

Explore detailed design and usage documentation for library users and developers:

  • Architecture & Design Guide: The parser pipeline, Earley algorithm details, and location tracking mechanics.

  • Custom ASG Renderer Tutorial: Walkthrough and fully executable Python implementation of a custom Markdown converter.

  • Feature Matrix: Complete comparison of supported language features, specification documents, and test coverage.

  • Contributor’s Guide: Virtualenv setup, Pytest protocols, Mypy type checks, and pre-release packaging verification checklists.

Motivation

The Python ecosystem has long lacked a modern, maintainable, and specification-compliant AsciiDoc parser. Historically, AsciiDoc was defined entirely by its implementations (originally the Python-based asciidoc.py, and later the Ruby-based Asciidoctor). This lack of a formal specification made developing third-party tooling—such as IDE plugins, formatters, and custom translators—extremely fragile and prone to compatibility drift.

With the working group at the Eclipse Foundation actively standardizing the AsciiDoc Language and providing an authoritative Technology Compatibility Kit (TCK), building a native, robust Python implementation became both possible and necessary.

Rather than relying on brittle, regex-heavy line-by-line parsing models that struggle with context-sensitive nested blocks, AsciiDoctrine implements a formal context-free grammar parsed via Lark’s Earley parsing engine. This formal parsing approach guarantees:

  1. Strict Spec Alignment: Flawless validation against the official AsciiDoc specification using the TCK test suites.

  2. Rich AST Construction: A complete, type-safe Abstract Syntax Tree (AST) representing every structural block and inline element, which makes writing custom renderers or compilers straightforward.

  3. Inclusive Location Coordinates: Precise tracking of source-to-AST bounding box coordinates (inclusive start/end line and column numbers), providing the foundation for professional IDE support and linting diagnostics.

Under the Hood

Parser Pipeline Architecture

The parser operates in a multi-pass pipeline to handle the inherent complexity of AsciiDoc, while supporting complete, lossless round-trip serialization back to source:

           +---------------------------------+
           |          Raw Source             |<---------------+
           +---------------------------------+                |
                            |                                 |
                            |  [Preprocessed LF/includes]     |
                            v                                 |
           +---------------------------------+                |
           |        Lark Parser Engine       |                |  [Serializer]
           +---------------------------------+                |
                            |                                 |
                            |  [Concrete Syntax Tree]         |
                            v                                 |
           +---------------------------------+                |
           |        CST Transformer          |                |
           +---------------------------------+                |
                            |                                 |
                            |  [Abstract Syntax Tree (AST)]---+
                            v
           +---------------------------------+
           |       ASG Semantic Resolver     |
           +---------------------------------+
                            |
                            |  [Abstract Semantic Graph (ASG)]
                            v
           +---------------------------------+
           |       Publishing Backends       |
           +---------------------------------+

  • AST (Abstract Syntax Tree): Represented in nodes.py, this is a structural tree of the document elements.

  • ASG (Abstract Semantic Graph): The final resolved state where attributes, cross-references, and includes are fully processed.

Roadmap to Syntax Parity

The path to 1:1 syntactic specification compliance is tracked through the following phases:

Phase

Focus

Status

0

Foundations: PEG/Earley grammar, structured AST, and TCK test harness.

✅

1

Advanced Blocks: Admonitions ✅, Sidebars ✅, Source blocks ✅, Example blocks ✅, and Open blocks ✅.

✅

2

Document Infra: Headers ✅, dynamic attribute resolution ✅, and multi-file includes ✅.

✅

3

Tables & Description Lists: Nested/mixed description lists, checklists ✅, and advanced table formatting/spans ✅.

✅

4

Testing & Developer Experience: Callout stripping, strict AST syntax auditing ✅, and precise source location coordinate tracking ✅.

✅

5

Structural Resolution & SSG: TOC outline extraction ✅, cross-references (xref) ✅, and bare URL/email autolinks ✅.

✅

6

Sphinx Extension Support: 100% Docutils node visitor coverage ✅, metadata alignment ✅, and permissive error recovery ✅.

✅

7

Spec & TCK Conformance: Authoring language specification modules, ASG schema convergence, and upstream TCK contributions.

🔄

Project Links & Resources