= Architecture & Design Michael R. Bernstein v0.1.0 :idprefix: :idseparator: - :sectanchors: :sectlinks: This document provides a deep dive into the design and architectural choices behind *AsciiDoctrine*. It is intended for both advanced users and core package contributors. --- == The Multi-Pass Compilation Pipeline AsciiDoc is a context-sensitive markup language. Unlike simpler markup formats like Markdown, correctly resolving AsciiDoc requires knowing document-wide context. For example, attributes defined anywhere in a document can substitute values in earlier or later paragraphs, and include files must be fetched and spliced before block-level formatting is resolved. To handle this complexity reliably, AsciiDoctrine processes input source code through a sequence of distinct, decoupled compiler passes, while preserving syntactic AST states for lossless round-trip serialization: [source,text] ---- +---------------------------+ | Raw AsciiDoc Source Code |<---------------+ +---------------------------+ | | | | [1. Preprocessor Pass] | v | +---------------------------+ | | Lark Earley Parser | | +---------------------------+ | | | [Serializer] | [2. Concrete Syntax Tree] | v | +---------------------------+ | | CST-to-AST Transformer | | +---------------------------+ | | | | [3. Syntactic AST Nodes]-----+ v +---------------------------+ | ASG Semantic Resolver | +---------------------------+ | | [4. Resolved ASG Structure] v +---------------------------+ | Target-Specific Back | | (Docutils, Sphinx) | +---------------------------+ ---- 1. *Preprocessor Pass*: Scans for include directives and circular references, loading external files safely. 2. *Earley Parsing Pass*: Feeds standardized lines into the Lark parsing engine to produce a raw Concrete Syntax Tree (CST). 3. *AST Transformation Pass*: Converts the complex, nested Lark CST into structured, typed Python AST nodes defined in `nodes.py`. 4. *Semantic ASG Pass*: Traverses the AST to perform attribute substitution, filter out comments/entries, and resolve structural constraints, yielding the final Abstract Semantic Graph (ASG). --- == AST vs. ASG Separation A key design choice in AsciiDoctrine is the strict architectural boundary between the *Abstract Syntax Tree (AST)* and the *Abstract Semantic Graph (ASG)*. === 1. Abstract Syntax Tree (AST) * *Goal*: Capture exact syntax-level structures. * *Nodes*: Created by `parse_to_ast()`. * *State*: Retains elements that do not affect final document semantics but are crucial for formatting or developer tools. For example, standalone `:name: value` (`attribute_entry`) nodes and raw comment blocks are preserved exactly as written. * *Line Coordinates*: Highly precise, matching the exact physical positions of characters in the original text file. === 2. Abstract Semantic Graph (ASG) * *Goal*: Represent the resolved, semantic meaning of the document, aligned with the official AsciiDoc ASG JSON schema. * *Nodes*: Produced by `ASGResolver.resolve(ast)`. * *State*: Standalone `attribute_entry` and `comment` nodes are consumed and filtered out of block lists. Their values are resolved and compiled into a unified `attributes` dictionary at the root of the Document. * *Resolution*: All inline text containing `{name}` attribute references is dynamically substituted with their resolved semantic string values. --- == Lark and the Earley Parsing Engine AsciiDoc contains numerous syntactic ambiguities (for example, whether `*` represents a bullet list item or the start of a bold text span). Standard LALR(1) or LL(k) parsers cannot parse AsciiDoc without massive lexer hacks or context-sensitive workarounds. AsciiDoctrine uses *Lark* configured with the *Earley parsing algorithm*. * *Why Earley?* Earley is a chart-parsing algorithm that parses all context-free grammars. It handles ambiguities gracefully by evaluating all matching parse trees in parallel. * *Tie-Breaking via Priorities*: When multiple rules match the same source lines (e.g., `toc::[]` matches both a block macro and a description list item), Lark uses rule-level and terminal-level priorities to break ties. * *Performance Optimizations*: * All input files are pre-normalized to Unix line endings (`\n`) to streamline regex lookaheads. * Structural block rules are assigned high priorities (e.g., `table.10`) while general formatting inline rules are given moderate priorities to prevent greedy asterisks from "stealing" list markers. --- == Location Tracking For compiler-style diagnostic messages and static analysis tools, knowing the precise coordinates of parsed elements is vital. Because source locations are optional in the official AsciiDoc specification, this capability provides significant value for custom tooling and developer environments. AsciiDoctrine implements *inclusive source coordinate tracking* on all transformed nodes: * *Leaf Tokens*: Leaf terminals (like strings or identifiers) get coordinate ranges directly from the Lark lexer's `meta` attributes. * *Polymorphic Propagation*: As nodes are transformed, parent blocks calculate their overall start and end line/column coordinates dynamically using a specialized helper method `_set_location_from_children()`. * *Accuracy*: The coordinates reflect the exact lines in the source, where `start_column` is 1-indexed and `end_column` is inclusive (`end_column - 1`). === Recursive Include Coordinate Mapping When a document recursively resolves `include::` directives, tracking AST nodes back to their original physical files can be challenging. AsciiDoctrine solves this by building an in-memory recursive coordinate map during the `Preprocessor` pass: * *Position Mapping*: Every consolidated, preprocessed line index is tracked and mapped back to its true `(file_path, original_line_number)` coordinate tuple inside the preprocessor's `line_map`. * *Lark Integration*: If the Lark parser encounters an unexpected token or syntax error, the exception is caught, unpacked, and translated through the `line_map` to report the exact physical coordinates and filename of the syntax error (e.g., `Syntax error in included_fragment.adoc at line 14`). * *AST Syntax Auditing*: An `ASTSyntaxAuditor` executes immediately after tree transformation. It performs post-parse structural validation (such as verifying unclosed delimited blocks, malformed inline elements, or mismatched macro brackets), tracing violations straight back to their exact sub-file origin coordinates via the line map. === Permissive Parsing & Error Recovery To support robust developer workflows, live-preview editors, and resilient compilation environments, the `parse_to_ast()` entry point supports a `strict` configuration flag: * *Strict Mode (`strict=True`, default)*: Any syntactic violation, unclosed block, or parser error throws a structured `AsciiDocSyntaxError` immediately to fail fast. * *Permissive Mode (`strict=False`)*: Syntactic or structural failures cleanly degrade into standard paragraph blocks containing the literal representation of the offending text instead of crashing execution. This ensures the parser can always compile incomplete or work-in-progress drafts. === Configurable URI Schemes & Extension Rationale AsciiDoc documents frequently reference web URLs (`https://`), file paths (`file://`), email addresses (`mailto:`), and telephone links (`tel:`). However, open-ended URI scheme matching presents serious syntactic ambiguities in formal EBNF grammars: 1. *Earley State Explosion & Greedy Matching*: If the parser used a generic URI regex matching any string before a colon (such as `[a-zA-Z]+:`), common prose terms (`practices include:`, `NOTE:`, `WARNING:`, `date:`, `time:`) would be mistakenly tokenized as URI schemes. 2. *Blast Radius & Parser Stability*: Hardcoding hundreds of proprietary or vendor-specific schemes (like `chrome:`, `slack:`, `zoommtg:`) directly into the core grammar increases the parser's maintenance surface area and risks shadowing valid prose. To balance spec compliance, high performance, and safe extensibility, AsciiDoctrine adopts a *configurable URI scheme architecture*: * *Core Default Schemes*: The OOTB Lark grammar contains only standardized, universal URI schemes (`http/https`, `ftp/ftps`, `file`, `irc/ircs`, `ws/wss`, `git`, `ssh`, `mailto`, `data`, `tel`, `sms`). * *Configuration-Driven Injection*: Developers can register additional authority schemes (`://`) or opaque schemes (`:`) at runtime via `parse_to_ast()`: [source,python] ---- doc = parse_to_ast( source, extra_authority_schemes=["slack", "zoommtg"], extra_opaque_schemes=["bitcoin"], ) ---- * *Strict Scheme Validation & Blacklisting*: To prevent regex injection or syntax corruption, all custom schemes pass through `validate_custom_scheme()`: - *Length*: Must be between 2 and 10 characters. - *Character Set*: Must contain only lowercase ASCII letters, digits, and hyphens (`[a-z0-9\-]+`). - *Reserved Blacklist*: Blacklists all inline macro prefixes (`link`, `image`, `xref`), block/admonition labels (`note`, `tip`, `warning`), metadata headers (`author`, `version`, `title`), and common colon-followed terms (`to`, `cc`, `bcc`, `date`, `time`, `ssn`, `id`).