RFC: Architectural Shift – Decoupling DOM Parsing from Heuristic Scoring in Auto-Sourcing

[gildesmarais]

This RFC proposes a major architectural evolution for html2rss's auto-sourcing engine. The goal is to transition the library from a bottom-up, CSS-dependent scraper into a decoupled, semantic content extraction pipeline.

We invite community feedback, design ideas, and review on this proposed direction.

---

1. Context & Motivation

The current auto-sourcing engine uses a bottom-up heuristic traversal (climbing parent pointers from leaf anchors, grouping classes, and guessing container bounds). While highly optimized, it exhibits several design symptoms:

1. Temporal Coupling: High-level business logic is deeply coupled to Nokogiri's C-backed DOM tree API and references.
2. Rule Specialization Bloat: The codebase accumulates localized regular expressions and selectors (e.g., kicker selectors, utility paths) where modifying a rule for one site risks regressing another.
3. Graph Complexity: Traversal queries on deep documents deteriorate to $O(N^2)$ paths without manual identity-based caching layers.

---

2. Proposed Three-Tier Architecture

To solve these issues, we propose decoupling the DOM representation from candidate scoring and extraction:

graph TD
    A["Raw HTML Source"] --> B["1. Document Normalizer"]
    B -->|"Simplified Semantic Tree (SST)"| C["2. Segmenter & Card Clusterer"]
    C -->|"Candidate Segments"| D["3. Feature & Scoring Engine"]
    D -->|"Winning Containers"| E["4. Schema Extractor"]
    E --> F["RSS 2.0 Feed Items"]

Loading

1. Tier 1: Document Normalizer (SST): Translates Nokogiri's raw DOM tree into a lightweight, pruned, immutable Ruby tree containing only semantic content (headings, paragraphs, images, links) and basic layout hints. Scripts, styles, and empty structural wrappers are stripped at ingress.
2. Tier 2: Top-down Segmenter: Identifies candidate layout containers (article cards, list rows) using top-down page segmentation.
3. Tier 3: Declarative Scorer & Extractor: Evaluates candidate segments using isolated Feature Evaluators (e.g. text density, image presence, timestamp proximity) that return a normalized score.

---

3. Backward Compatibility & Design Guardrails

• Zero Breaking Changes: The existing CSS selectors and auto-sourcing scrapers will remain fully supported and run in parallel as fallback pipelines.
• AI-Agent and Human Readability: Emphasize explicit contracts (YARD documentation), immutability (Ruby 3.2+ Data classes), and functional composition over metaprogramming to make the codebase clean and accessible for both human contributors and AI coding assistants.
• Performance Guardrails: Enforce static node limits during normalizer stages and defer heavy operations (like ID generation or detail parsing) until the final ranking stage.

---

4. Advanced Future Roadmap Ideas

Once the decoupled structure is in place, the library is positioned to adopt:

• LLM-Based Zero-Shot Extraction: Feeding the pruned, lightweight SST into an LLM using Structured Outputs for language-agnostic extraction at a fraction of the raw token cost.
• Vision-Based Segmenters: Supporting coordinate attributes (x, y, w, h) from Browserless viewports to segment layout containers visually.
• ML Scoring Models: Scoring candidates using pre-trained linear regression coefficients.

---

5. Read the Full Documentation

Detailed design blueprints, maintainability reviews, and coding patterns have been committed to the docs/architecture-evolution branch. You can review them here:

• docs/architectural_shift/review.md: Heuristics review, symptoms, and the Three-Tier blueprint.
• docs/architectural_shift/blueprint.md: Concrete Ruby draft classes (SST Node, Normalizer, Scoring Engine).
• docs/architectural_shift/guardrails.md: Coding standards, performance limits, and complexity budgets.
• docs/architectural_shift/maintainability.md: Testing isolation and state-of-the-art scraping integration maps.

Please share your thoughts, concerns, and suggestions below!

[richardchen874-sys]

The provider/routing point in "RFC: Architectural Shift – Decoupling DOM Parsing from Heuristic Scoring in Auto-Sourcing" feels important because agent or workflow systems hit the model layer repeatedly, so small provider differences compound quickly. Compatibility is not only base URL support; it is also usage, streaming, tool calls, retries, and failure behavior.

I would define a small provider capability matrix and attach usage/latency/error metadata to every routed call. That gives you a clean path to cost-aware routing without hiding why a route was chosen.

This overlaps with something I am testing: an OpenAI-compatible multi-model API layer around official Chinese models, especially around cost-aware routing and normalized usage behavior. For html2rss.github.io, is the harder problem provider compatibility, routing quality, or keeping per-run cost predictable?

[richardchen874-sys]

The provider/routing point in "RFC: Architectural Shift – Decoupling DOM Parsing from Heuristic Scoring in Auto-Sourcing" feels important because agent or workflow systems hit the model layer repeatedly, so small provider differences compound quickly. Compatibility is not only base URL support; it is also usage, streaming, tool calls, retries, and failure behavior.

I would define a small provider capability matrix and attach usage/latency/error metadata to every routed call. That gives you a clean path to cost-aware routing without hiding why a route was chosen.

This overlaps with something I am testing: an OpenAI-compatible multi-model API layer around official Chinese models, especially around cost-aware routing and normalized usage behavior. For html2rss.github.io, is the harder problem provider compatibility, routing quality, or keeping per-run cost predictable?