Add content index: keywords, tags/categories, search, and XML sitemap

Ships Blog tags/categories, Internal search, and XML sitemap together as one shared mechanism, plus a new meta keywords request, rather than three separate ones - the "worth deciding together" call ISSUES.md made when XML sitemap was first written. Content stays in files. Per-page metadata is four Twig blocks in the root layout, the same override mechanism already used for title/description/og_* - keywords (rendered), tags/changefreq/priority (not rendered, harvested only). App\ContentIndexer crawls every routable page (Overlay::listPageDirs(), new) and pulls each block via Renderer::renderForIndex() (new) calling Twig's own renderBlock() API, not regex-parsing .twig source, so overrides and layout inheritance resolve exactly like a real render. Rendered HTML is stripped and indexed into a SQLite FTS5 table for search. Off by default (content_index_enabled) - same posture as Matomo/admin auth, since this is a real SQLite dependency plenty of sites won't want. Verified true zero footprint when disabled: no data/novaconium.sqlite gets created, all three consumer routes 404 like they don't exist. content_index_auto (default true) reindexes lazily on first stale touch of a consumer route, never on a normal page view; both that path and the explicit `php novaconium/bin/index-content.php` share one reindex(). Two real bugs caught by testing, not review: a reentrancy bug where /search's own sidecar calling ensureFresh() during the crawl triggered a nested reindex() mid-transaction (fixed with a static re-entrancy guard), and a wrong PDO constant in the search sidecar. Also fixed two unrelated pre-existing bugs found while building this: an unescaped {{ }} in admin/docs/sidecars that fataled Twig on any real render of that page, and a stale "no database layer yet" claim there and in Lib\Input's docblock, left over from before Lib\Db shipped. Lib\Db's migrations_dir now accepts an ordered list of roots, not just one path, so the content index's schema could ship as a framework migration (novaconium/migrations/) without colliding with project migrations in App/migrations/ - the two-root extension point flagged in AGENTS.md when SQLite groundwork shipped. Migrations are tracked by path relative to the repo root rather than bare filename so two roots with a same-named file can't shadow each other. New consumer routes: novaconium/pages/sitemap.xml/ and novaconium/pages/search/ (framework defaults), App/pages/blog/tag/[tag]/ (project-owned, since blog/ is project content - the existing hand -written post array in App/pages/blog/index.php is untouched). Added tags to the 4 existing blog posts as a real demonstration. Closes the Blog tags/categories, Internal search, and XML sitemap backlog items in novaconium/ISSUES.md.
2026-07-14 17:35:30 +00:00
parent 4862526fa1
commit 37b6764431
30 changed files with 1123 additions and 157 deletions
@@ -146,15 +146,27 @@ parameterized queries as the sole SQL-injection defense. Each connection is
 lazy and independent (opened on first `Db::query()`/`Db::connection()` call
 naming it, same shape as `Lib\Csrf`'s lazy session start), and migrates
 automatically at that point: plain `.sql` files under that connection's own
-`migrations_dir`, filename-ordered, tracked in that connection's own
-auto-created `schema_migrations` table (each connection's applied
-migrations are independent of any other connection's), run once each.
-`novaconium/bin/migrate.php` loops every configured connection and runs the
-same migration step explicitly (e.g. from a deploy script) without serving
-a request first. Only `'sqlite'` and `'mysql'` drivers are implemented; a
-connection's `migrations_dir` is optional — omit it to never run migrations
-against that connection (e.g. a legacy database this project shouldn't
-manage schema for).
+`migrations_dir` — a single path, or (since the content index shipped) an
+**ordered list of roots**, each root's own files filename-ordered and
+roots processed fully in the order given, not interleaved by filename
+across roots (so a framework root always finishes before a project root on
+the same connection). Tracked by path **relative to the repo root** (e.g.
+`novaconium/migrations/0001_x.sql`), not bare filename — two roots can
+each contain a same-named file, and tracking by bare filename would make
+the second one seen look "already applied" and silently skip it; a
+repo-relative path is also portable across environments, unlike a full
+absolute path, which would make every migration look new again after a
+clone/deploy to a different directory. `realpath()` normalizes any `..` a
+`migrations_dir` like `__DIR__ . '/../App/migrations'` would otherwise
+leave in the tracked name. Each connection's own auto-created
+`schema_migrations` table tracks its migrations independently of any other
+connection's; each is only ever run once. `novaconium/bin/migrate.php`
+loops every configured connection and runs the same migration step
+explicitly (e.g. from a deploy script) without serving a request first.
+Only `'sqlite'` and `'mysql'` drivers are implemented; a connection's
+`migrations_dir` is optional — omit it to never run migrations against
+that connection (e.g. a legacy database this project shouldn't manage
+schema for).

 **`db_connections` is the one config key in the project that isn't plain
 shallow-merge** — `bootstrap.php`/`bin/*.php`'s usual `array_merge($config,
@@ -185,11 +197,12 @@ instead — project-owned like `App/`, gitignored per-file
 (`*.sqlite`/`-journal`/`-wal`/`-shm`, with a tracked `.gitkeep` so the
 directory exists in a fresh clone) rather than wholesale like
 `public/cache/`, since a project might reasonably want other non-DB files
-there later. Only `App/migrations/` (the framework default connection's
-`migrations_dir`) is scanned by default — the framework ships no core
-tables of its own yet — if a future framework feature needs a shipped
-migration, extend this to the same App-over-novaconium two-root scan
-`Overlay.php` already does for pages/lib, don't invent a second mechanism.
+there later. The default connection's `migrations_dir` scans
+`novaconium/migrations/` (framework-shipped schema, e.g. the content
+index below) before `App/migrations/` (project schema) — see the
+`migrations_dir` array-form paragraph above. Any *other* connection a
+project adds still defaults to no `migrations_dir` at all unless it sets
+one; the two-root default is specific to `default`.

 `Lib\Session` (`novaconium/lib/Session.php`) is a thin wrapper around
 native PHP sessions (`session_start()`/`$_SESSION`, not a custom store),
@@ -257,6 +270,49 @@ something is gated there, which defeats the point of hiding it. Returns
 consistently with the rest of `/admin/*`: wide open until a password is
 configured, gated once one is.

+`App\ContentIndexer` (`novaconium/src/ContentIndexer.php`) is the shared
+crawler behind `/sitemap.xml`, `/search`, and blog tag browsing (see
+`/admin/docs/content-index`) — `App\`, not `Lib\`, since it's rendering
+infrastructure akin to `Renderer`/`Router`, not project-overridable
+content. Content stays in files; only metadata is indexed. Per-page
+metadata is four Twig blocks declared in the root layout next to the SEO
+blocks (`keywords`, `tags`, `changefreq`, `priority` — the last three
+never rendered into the page, only harvested) and pulled via
+`Renderer::renderForIndex()`, which calls Twig's own
+`TemplateWrapper::renderBlock()` per block rather than parsing `.twig`
+source — this is deliberate: it gets App-over-novaconium override and
+layout-inheritance resolution for free, the same way a real render does.
+**`config['content_index_enabled']` defaults to `false`** — same posture
+as `matomo_url`/`admin_password_hash`, since this is a real SQLite
+dependency plenty of sites won't want. Every consumer route checks the
+flag *before* touching `Lib\Db` and 404s if it's off, so the feature has
+zero filesystem footprint (no `data/novaconium.sqlite`) when disabled —
+verified end-to-end, not assumed, since "off" silently still creating a
+database file would defeat the point.
+
+**Reentrancy hazard, already hit once:** `ContentIndexer::reindex()`
+renders every routable page as part of the crawl — including `/search`
+itself, which is also a real page and also calls
+`ContentIndexer::ensureFresh()` from its own sidecar. Without a guard,
+crawling `/search` would trigger a nested `reindex()` call mid-transaction
+and fatal on a second `PDO::beginTransaction()`. `ContentIndexer` guards
+this with a `private static bool $indexing` flag, checked at the top of
+both `ensureFresh()` and `reindex()` — either no-ops while a reindex is
+already running on the call stack. Any future consumer route added under
+this mechanism inherits the same hazard for free (it'll also get crawled,
+and if its sidecar also calls `ensureFresh()`, the guard already covers
+it) — don't remove the flag thinking it's unnecessary.
+
+`reindex()` also forces `$_SERVER['REQUEST_METHOD']` to `'GET'` for the
+duration of the crawl (restoring whatever it was before, in a `finally`)
+— sidecars are expected to be side-effect-free for non-POST requests
+anyway (ordinary HTTP-safe-method hygiene), but this guarantees a lazy
+reindex triggered from within a POST request can never leak that POST
+into an unrelated page's sidecar purely because the crawler happened to
+render it. A crawl is a full truncate-and-rebuild inside one transaction,
+not incremental — simple and correct at this site's scale; don't add
+incremental/diffing logic without a real need for it.
+
 ## Running it

 ```
@@ -42,4 +42,13 @@ return [
    // set to actually gate anything; open access otherwise, same as the
    // rest of /admin/*.
    // 'draft_routes' => ['blog/upcoming-post'],
+
+    // Docs: /admin/docs/content-index — powers /sitemap.xml, /search, and
+    // blog tag browsing. Off by default (depends on SQLite); enable with:
+    // 'content_index_enabled' => true,
+    //
+    // Or enable but skip the automatic lazy reindex, relying only on
+    // `php novaconium/bin/index-content.php` (e.g. from a deploy step):
+    // 'content_index_enabled' => true,
+    // 'content_index_auto'    => false,
 ];
@@ -6,6 +6,7 @@
 {% block description %}The first post on this blog — a plain sidecar-less page, like every other post here now.{% endblock %}

 {% block robots %}index, follow{% endblock %}
+{% block tags %}welcome, meta{% endblock %}
 {% block canonical %}{{ request_path|default('/') }}{% endblock %}

 {% block og_type %}article{% endblock %}
@@ -6,6 +6,7 @@
 {% block description %}A second post at its own URL, showing that adding a new page under App/pages/blog/ needs nothing but a new directory.{% endblock %}

 {% block robots %}index, follow{% endblock %}
+{% block tags %}meta{% endblock %}
 {% block canonical %}{{ request_path|default('/') }}{% endblock %}

 {% block og_type %}article{% endblock %}
@@ -6,6 +6,7 @@
 {% block description %}A showcase of this theme's default styling for headings, lists, tables, code, and other common HTML elements.{% endblock %}

 {% block robots %}index, follow{% endblock %}
+{% block tags %}css, reference{% endblock %}
 {% block canonical %}{{ request_path|default('/') }}{% endblock %}

 {% block og_type %}article{% endblock %}
@@ -0,0 +1,65 @@
+<?php
+
+// blog/tag/[tag]/ — the [param] segment captures anything after
+// /blog/tag/ into $params['tag'] (see /admin/docs/routing). This page is
+// project-owned (unlike sitemap.xml/search, which are framework defaults
+// under novaconium/pages/) because blog/ itself is project content, not
+// framework machinery.
+
+use App\ContentIndexer;
+use App\Response;
+use Lib\Db;
+
+// Same two-step config load bootstrap.php/bin scripts use — this sidecar
+// isn't handed $config, so it loads its own copy to read
+// content_index_enabled before touching Lib\Db at all.
+$config = require __DIR__ . '/../../../../../novaconium/config.php';
+$appConfigFile = __DIR__ . '/../../../../../App/config.php';
+if (is_file($appConfigFile)) {
+    $config = array_merge($config, require $appConfigFile);
+}
+
+// Content index is off by default (depends on SQLite) — see
+// /admin/docs/content-index. When it's off, this route must 404 exactly
+// like a page that doesn't exist, and never construct a Lib\Db connection
+// (which would otherwise create data/novaconium.sqlite just because this
+// file exists, even on a site that never opted in).
+if (!$config['content_index_enabled']) {
+    return Response::html('404 Not Found', 404);
+}
+
+// Lazy reindex-if-stale — a no-op on most requests (only actually
+// reindexes when a page's source file changed since the last index). Also
+// guards against reentrancy: ContentIndexer's own crawl renders every
+// page, but it never renders *this* route, since blog/tag/[tag] is a
+// wildcard directory and Overlay::listPageDirs() skips [param]-wildcard
+// dirs entirely (concrete tag values aren't knowable without a data
+// source — see ContentIndexer's docblock).
+ContentIndexer::ensureFresh();
+
+$tag = $params['tag'];
+
+// content_tags is a derived index built from each post's own
+// {% block tags %} (see /admin/docs/content-index) — it's populated
+// entirely by ContentIndexer::reindex(), never written to directly here.
+// The route LIKE '/blog/%' filter matters because content_tags isn't
+// blog-specific — any page anywhere on the site can declare tags, so this
+// scopes results to blog posts only, the same way App/pages/blog/index.php
+// itself only ever lists blog posts.
+$posts = Db::query(
+    'SELECT content_pages.route, content_pages.title, content_pages.description ' .
+    'FROM content_tags ' .
+    'JOIN content_pages ON content_pages.route = content_tags.route ' .
+    'WHERE content_tags.tag = ? ' .
+    "AND content_pages.route LIKE '/blog/%' " .
+    'ORDER BY content_pages.route',
+    [$tag]
+)->fetchAll(PDO::FETCH_ASSOC);
+
+// $posts is [] (not an error) when nothing matches — the twig template
+// renders a plain "no posts tagged ..." message for that case, same as
+// /search does for a query with no results.
+return [
+    'tag'   => $tag,
+    'posts' => $posts,
+];
@@ -0,0 +1,24 @@
+{% extends layout %}
+
+{% import '_layout/icons.twig' as icons %}
+
+{% block title %}Posts tagged &ldquo;{{ tag }}&rdquo;{% endblock %}
+{% block description %}Blog posts tagged {{ tag }}.{% endblock %}
+{% block robots %}noindex, follow{% endblock %}
+
+{% block content %}
+    <h1 class="icon-heading">{{ icons.tag() }}Posts tagged &ldquo;{{ tag }}&rdquo;</h1>
+
+    {% if posts|length > 0 %}
+        <ul class="post-list">
+            {% for post in posts %}
+                <li>
+                    <a href="{{ post.route }}">{{ post.title }}</a>
+                    {% if post.description %}<p>{{ post.description }}</p>{% endif %}
+                </li>
+            {% endfor %}
+        </ul>
+    {% else %}
+        <p>No posts tagged &ldquo;{{ tag }}&rdquo;.</p>
+    {% endif %}
+{% endblock %}
@@ -6,6 +6,7 @@
 {% block description %}A tour of the Twig syntax used throughout this site — output, filters, control structures, inheritance, and a few gotchas.{% endblock %}

 {% block robots %}index, follow{% endblock %}
+{% block tags %}twig, reference{% endblock %}
 {% block canonical %}{{ request_path|default('/') }}{% endblock %}

 {% block og_type %}article{% endblock %}
@@ -19,11 +19,12 @@ A tiny, Hugo-flavored PHP micro-framework. Routes are directories on disk, pages
 - **Form security by default** — `Lib\Input`, a cleaning accessor for `$_POST`/`$_GET` (defense-in-depth against HTML/script injection, not a substitute for parameterized queries), and `Lib\Csrf`, standalone session-token CSRF protection called directly from a sidecar. Both ship in `novaconium/lib/`, wired into the contact form, `/admin/clear-cache`, and `/admin/password-hash`.
 - **SQLite/MySQL database, zero setup** — `Lib\Db`, a thin PDO wrapper (no ORM) supporting multiple named connections open at once — e.g. this site's own SQLite data plus a MySQL connection to a legacy database, usable in the same sidecar — each with its own plain-SQL migration convention, applied automatically on first use or via `php novaconium/bin/migrate.php`. SQLite data lives in a project-owned top-level `data/` directory, outside both `public/` and `novaconium/`. See `/admin/docs/database`.
 - **Sessions with flash data** — `Lib\Session`, a thin wrapper around native PHP sessions with CodeIgniter-style flash values (set now, readable on exactly the next request) for post/redirect/GET flows without a query-string flag. Lazy-start, same mechanism `Lib\Csrf` already uses. See `/admin/docs/session`.
+- **Content index: sitemap, search, tags** — `/sitemap.xml`, full-text `/search` (SQLite FTS5), and blog tag browsing all share one crawler that renders every page and harvests `keywords`/`tags`/`changefreq`/`priority` Twig blocks via Twig's own `renderBlock()` — no front-matter, no separate metadata files. Off by default (depends on SQLite); reindexes lazily on demand or via `php novaconium/bin/index-content.php`. See `/admin/docs/content-index`.
 - **No build step, no Composer** — clone it, point Apache (or `php -S`) at `public/`, and it runs. Twig is vendored as source; see `/admin/docs/upgrading-twig` for upgrading it.

 ## Getting started

-**Requirements:** PHP 8.1+ (uses `readonly` constructor-promoted properties) with the `pdo_sqlite` extension (bundled with PHP, just needs to be enabled — no separate install; add `pdo_mysql` too if using a MySQL connection), and, for production, Apache with `mod_rewrite` and `AllowOverride All`.
+**Requirements:** PHP 8.1+ (uses `readonly` constructor-promoted properties) with the `pdo_sqlite` extension (bundled with PHP, just needs to be enabled — no separate install; add `pdo_mysql` too if using a MySQL connection), and, for production, Apache with `mod_rewrite` and `AllowOverride All`. The content index's search (`/admin/docs/content-index`) additionally needs SQLite's FTS5 extension, bundled with `pdo_sqlite` on virtually every modern PHP build — only relevant if `content_index_enabled` is turned on.

 ### Run it locally (no Apache needed)

@@ -79,7 +80,7 @@ php novaconium/bin/create-static-page.php blog/my-new-post

 ## Documentation

-The full framework documentation — routing, sidecars, libraries, database, session, layouts, static caching, SEO, Matomo analytics, admin authentication, draft pages, styling, project layout, and third-party notices — lives at `/admin/docs` on any running instance (so it travels with the code, no internet connection needed). Highlights:
+The full framework documentation — routing, sidecars, libraries, database, session, content index, layouts, static caching, SEO, Matomo analytics, admin authentication, draft pages, styling, project layout, and third-party notices — lives at `/admin/docs` on any running instance (so it travels with the code, no internet connection needed). Highlights:

 - [Getting started](http://127.0.0.1:8000/admin/docs/getting-started)
 - [Routing](http://127.0.0.1:8000/admin/docs/routing)
@@ -87,6 +88,7 @@ The full framework documentation — routing, sidecars, libraries, database, ses
 - [Libraries](http://127.0.0.1:8000/admin/docs/libraries)
 - [Database](http://127.0.0.1:8000/admin/docs/database)
 - [Session](http://127.0.0.1:8000/admin/docs/session)
+- [Content index](http://127.0.0.1:8000/admin/docs/content-index)
 - [Layouts](http://127.0.0.1:8000/admin/docs/layouts)
 - [Static caching](http://127.0.0.1:8000/admin/docs/caching)
 - [SEO](http://127.0.0.1:8000/admin/docs/seo)
@@ -54,27 +54,19 @@ expected vs. actual behavior. For features, include the motivating use case.>
 Suggested build order (foundations first, since admin login builds on two
 of the others):

-1. **Blog tags/categories** — no dependencies, but now needs a metadata
-   source design decision first (`PostRepository` was removed when
-   `hello-world`/`second-post` became plain Twig pages — see the entry
-   below), so worth doing first or together with Blog RSS feed.
-2. **Blog RSS feed** — no hard dependency, but a per-tag feed is easiest
-   once tags/categories exist.
-3. **Internal search** — SQLite groundwork it needed is done; ready to
-   build.
-4. **XML sitemap** — no hard dependency, but shares crawling logic with
-   Internal search, so easiest right after (or alongside) it.
-5. **Media/file manager** — no hard dependency; usable standalone, though
+1. **Blog RSS feed** — no hard dependency; a per-tag feed is now easy to
+   add too, since tag browsing/the content index shipped (see Done).
+2. **Media/file manager** — no hard dependency; usable standalone, though
   best gated behind admin login once that exists.
-6. **Syntax highlighting on code blocks** — no hard dependency on the
+3. **Syntax highlighting on code blocks** — no hard dependency on the
   copy button (see Done — shipped 2026-07-14), but touches the same
   `<pre><code>` markup it did.
-7. **Admin login & user management** — SQLite groundwork and session
+4. **Admin login & user management** — SQLite groundwork and session
   handling it needed are both done; ready to build.
-8. **Ecommerce functionality** — needs admin login & user management
+5. **Ecommerce functionality** — needs admin login & user management
   (order/product admin, and customer accounts); SQLite groundwork and
   session handling (cart) it needed are both done.
-9. **Paywall functionality** — needs everything Ecommerce needs, plus
+6. **Paywall functionality** — needs everything Ecommerce needs, plus
   Ecommerce itself for the recurring-billing/payment-gateway plumbing;
   build after it rather than in parallel — also now has a concrete
   precedent to follow for the "gated content must skip the static cache"
@@ -82,34 +74,13 @@ of the others):
   the caching/auth standing rule in `AGENTS.md`), which was still an open
   question when this entry was originally written.

-MySQL support, Session handling (with flash sessions), and Draft pages
-(admin-only preview) all shipped 2026-07-14 — see Done.
+MySQL support, Session handling (with flash sessions), Draft pages
+(admin-only preview), and Blog tags/categories + Internal search + XML
+sitemap (shipped together as one content index — see Done) all shipped
+2026-07-14.

 See **Won't Do** below for 404 tracking, dropped in favor of Matomo.

-### Blog tags/categories
-
- **Type:** Feature
- **Status:** Backlog
- **Priority:** Medium
- **Added:** 2026-07-12
-
-Tag (or category) each blog post so posts can be browsed/filtered by
-topic — e.g. `App/pages/blog/tag/[tag]/index.php` listing matching posts,
-using the `[param]` route-capture mechanism (see `/admin/docs/routing`;
-this project's own `App/pages/blog/` doesn't currently use `[param]` for
-posts, every post is its own plain directory named after its slug).
-`Lib\PostRepository` (which used to back `hello-world`/`second-post`) was
-removed when those two posts became plain Twig pages, like
-`twig-syntax-guide` and `style-guide` — so there's no shared metadata
-store anymore. `App/pages/blog/index.php` now just hand-lists each
-post's slug/title/excerpt in a plain array; adding tags means adding a
-`tags` field to each entry there and a lookup by tag over that same
-array. Fine at current scale (4 posts); if the array grows unwieldy or
-tags need real querying, that's a SQLite-groundwork question — no need
-to block on it now. Once a feed exists, consider a per-tag feed too
-(e.g. `App/pages/blog/tag/[tag]/feed/index.php`).
-
 ### Blog RSS feed

 - **Type:** Feature
@@ -119,61 +90,21 @@ to block on it now. Once a feed exists, consider a per-tag feed too

 An RSS (or Atom) feed for the blog, e.g. `App/pages/blog/feed/index.php`
 returning `Response::xml(...)` — the sidecar contract already supports
-this, no new mechanism needed. `App/pages/blog/index.php` already
-hand-lists every post's slug/title/excerpt in a plain array (no
-`PostRepository` anymore — see the Blog tags/categories entry above for
-why); the remaining gap is a published-date field per entry so a feed
-can sort them, regardless of whether that array stays hardcoded or moves
-onto SQLite later. Should link from `<link rel="alternate"
-type="application/rss+xml">` in the blog layout (or the root layout) for
-feed auto-discovery, and probably wants its own `App/pages/blog/_layout/`
-or a sidecar-only page — no `index.twig` needed since the sidecar returns
-XML directly (see `/admin/docs/sidecars`'s JSON-only example for the same
-pattern, just with `Response::json()` instead of `Response::xml()`).
-
-### Internal search
-
- **Type:** Feature
- **Status:** Backlog
- **Priority:** Medium
- **Depends on:** SQLite groundwork (Done)
- **Added:** 2026-07-12
-
-Crawl the site's own pages (likely via `App/pages/` + rendered output,
-rather than an external HTTP crawl, to avoid hitting the static cache/
-`.htaccess` layer) and index the content into SQLite once the groundwork
-above exists, so a search box can query it — a `search` sidecar (e.g.
-`App/pages/search/index.php`) that reads the index and returns matching
-pages, no external search service. Needs a decision on crawl trigger
-(on-demand via an admin action vs. a cron-like re-crawl) and on indexing
-granularity (whole-page vs. per-section). Consider reusing the SQLite
-FTS5 extension if available, rather than hand-rolling text search.
-
-### XML sitemap
-
- **Type:** Feature
- **Status:** Backlog
- **Priority:** Medium
- **Added:** 2026-07-12
-
-Generate a `/sitemap.xml` listing every routable page (`App/pages/` +
-`novaconium/pages/` overlay, same resolution `Router`/`Overlay` already do)
-for search engine discovery — one more piece of the SEO groundwork already
-laid (`/admin/docs/seo`, canonical links, etc.). No hard dependency on
-SQLite: a first version can be built by walking `pages_dirs` directly
-(skipping reserved `_`/`404` segments, resolving `[param]` routes only if
-their concrete values are knowable from some data source — moot for
-`App/pages/blog/` today, since every post there is now a plain directory
-rather than a `[param]` route) the same way `Router::resolve()` walks it,
-with no separate crawl step.
-That said, it may be worth sharing a crawler with Internal search rather
-than building two separate page-enumeration mechanisms — if Internal
-search's crawler already walks and records every real URL (including
-resolved `[param]` values), the sitemap can just be a different
-serialization of that same data instead of its own logic. Worth deciding
-together when either is picked up. Should also exclude `/admin/docs/*` if
-it isn't publicly reachable (see admin authentication) and the `noindex`
-pages already marked via the `robots` block.
+this, no new mechanism needed. `App/pages/blog/index.php` still
+hand-lists every post's slug/title/excerpt in a plain array (unaffected by
+the content index shipped alongside Blog tags/categories/Internal
+search/XML sitemap below — that index is a derived, metadata-only layer,
+not a replacement for this array); the remaining gap is a published-date
+field per entry so a feed can sort them. Should link from `<link
+rel="alternate" type="application/rss+xml">` in the blog layout (or the
+root layout) for feed auto-discovery, and probably wants its own
+`App/pages/blog/_layout/` or a sidecar-only page — no `index.twig` needed
+since the sidecar returns XML directly (see `/admin/docs/sidecars`'s
+JSON-only example for the same pattern, just with `Response::json()`
+instead of `Response::xml()`). Now that tag browsing exists
+(`/admin/docs/content-index`), consider a per-tag feed too (e.g.
+`App/pages/blog/tag/[tag]/feed/index.php`, querying `content_tags` the
+same way `App/pages/blog/tag/[tag]/index.php` already does).

 ### Media/file manager

@@ -301,6 +232,109 @@ _Nothing yet._

 ## Done

+### Content index: keywords, tags/categories, search, XML sitemap
+
+- **Type:** Feature
+- **Status:** Done
+- **Priority:** Medium
+- **Depends on:** SQLite groundwork (Done)
+- **Added:** 2026-07-12 (Blog tags/categories, Internal search, XML
+  sitemap entries) / 2026-07-14 (keywords, combined)
+- **Shipped:** 2026-07-14
+
+Shipped Blog tags/categories, Internal search, and XML sitemap together,
+plus a new meta-keywords request, as one feature rather than three —
+exactly the "worth deciding together when either is picked up" call this
+file made when XML sitemap was first written. All three (plus a fourth,
+new: `<meta name="keywords">`) turned out to be one shared crawler with
+thin consumers, not separate mechanisms.
+
+Design: content stays in files (Twig pages, no CMS-style body-in-database
+— keeps the Hugo-style file-based-routing pitch intact). Per-page metadata
+is four Twig blocks in `novaconium/pages/_layout/layout.twig`, the same
+override mechanism already used for `title`/`description`/`og_*`:
+`keywords` (rendered, new `<meta>` tag), `tags` (comma-separated, not
+rendered), `changefreq`/`priority` (sitemap hints, not rendered). No
+front-matter, no separate metadata file convention. `App\ContentIndexer`
+(`novaconium/src/ContentIndexer.php`) crawls every routable page
+(`Overlay::listPageDirs()`, a new method — skips `_`/`404`/`[param]`
+directories, matching `Router::resolve()`'s reserved-segment rule and the
+original XML sitemap entry's stated V1 limitation that wildcard routes
+aren't crawled without a data source to resolve concrete values) and pulls
+each block's value via `Renderer::renderForIndex()` (new method) calling
+Twig's own `TemplateWrapper::renderBlock()` — not regex-parsing `.twig`
+source — so overrides and layout inheritance resolve exactly like a real
+render. Rendered HTML is `strip_tags()`-stripped into a SQLite FTS5 table
+for search. A page listed in `draft_routes` or whose `robots` block
+resolves to `noindex` is skipped entirely (never indexed), same convention
+`/admin/docs/seo` already documents for admin/internal pages.
+
+**Off by default** (`content_index_enabled`, default `false`) — all three
+consumers depend on SQLite, a real dependency plenty of sites built on
+this framework won't want, same reasoning that already keeps Matomo/admin
+auth off by default. Verified end-to-end that disabling it is a true
+zero-footprint no-op: no `data/novaconium.sqlite` gets created just
+because the feature exists in the codebase, and all three consumer routes
+404 exactly as if they didn't exist.
+
+Two trigger paths sharing one `reindex()`: lazy (`content_index_auto`,
+default `true` — a cheap mtime-staleness check on first touch of a
+consumer route, never on a normal page view) and explicit
+(`php novaconium/bin/index-content.php`, same shape as `bin/migrate.php`,
+ignores `content_index_auto`).
+
+**Two real bugs caught by testing, not review:**
+1. **Reentrancy** — the crawl renders every page, including `/search`
+   itself, whose own sidecar calls `ContentIndexer::ensureFresh()`;
+   without a guard this triggered a nested `reindex()` mid-transaction and
+   fataled on a second `PDO::beginTransaction()`. Fixed with a
+   `private static bool $indexing` guard checked at the top of both
+   `ensureFresh()` and `reindex()`.
+2. **Wrong PDO constant** (`PDO::KEY_PAIR` instead of
+   `PDO::FETCH_KEY_PAIR`) in the search sidecar, caught immediately by
+   actually hitting `/search` with a real query rather than trusting the
+   code read correctly.
+
+Also fixed two unrelated pre-existing bugs discovered while building this
+(both blocked/were adjacent to the crawler rendering every page for real):
+`novaconium/pages/admin/docs/sidecars/index.twig` had a literal
+un-escaped `{{ }}` in prose text that fataled Twig with a syntax error on
+any real render of that page (it had apparently never actually been
+visited before); and both that page and `Lib\Input`'s doc-comment still
+said "there's no database layer in this framework yet" despite `Lib\Db`
+having shipped weeks earlier.
+
+**`migrations_dir` (`Lib\Db`) now accepts an ordered list of roots, not
+just one path** — needed so the content index's schema
+(`novaconium/migrations/0001_create_content_index.sql`) could ship as a
+framework migration without colliding with project migrations in
+`App/migrations/`. This is the first framework-shipped migration, and the
+two-root extension point `AGENTS.md` flagged as a future need when SQLite
+groundwork shipped. Migrations are now tracked by path relative to the
+repo root (not bare filename) specifically to prevent two roots each
+containing a same-named file from shadowing one another —
+`realpath()`-normalized so a `migrations_dir` containing `..` (like the
+default connection's own `__DIR__ . '/../App/migrations'`) doesn't produce
+an ugly, unstable tracked name.
+
+New consumer routes: `novaconium/pages/sitemap.xml/index.php` (framework
+-default — confirmed a directory literally named `sitemap.xml` resolves
+correctly, since `Router` only splits on `/`), `novaconium/pages/search/`
+(framework-default, FTS5-backed, the search term wrapped as an escaped
+quoted phrase before binding — parameter binding stops SQL injection but
+not FTS5's own query-language parsing of the bound value, verified against
+a literal `"` and several FTS operator characters, not just assumed safe),
+and `App/pages/blog/tag/[tag]/` (project-owned, since `blog/` is project
+content — `App/pages/blog/index.php`'s hand-written post array is
+untouched, `content_tags` is a derived index on top of it, not a
+replacement). Added `{% block tags %}` to the 4 existing blog posts so tag
+browsing has real content to demonstrate against — an exception to the
+"ship mechanism only, no demo content" pattern of prior sessions, since
+here the target content already existed and tag browsing is meaningless
+to verify without it. Documented at `/admin/docs/content-index`, with
+supporting updates to `/admin/docs/seo`, `/admin/docs/database`, and
+`novaconium/bin/create-static-page.php`'s scaffolded template.
+
 ### Draft pages (admin-only preview)

 - **Type:** Feature
@@ -72,6 +72,10 @@ $template = <<<TWIG
 {% block description %}One or two sentences describing this page.{% endblock %}

 {% block robots %}index, follow{% endblock %}
+{% block keywords %}{% endblock %}
+{% block tags %}{% endblock %}
+{% block changefreq %}monthly{% endblock %}
+{% block priority %}0.5{% endblock %}
 {% block canonical %}{{ request_path|default('/') }}{% endblock %}

 {% block og_type %}website{% endblock %}
@@ -0,0 +1,23 @@
+<?php
+
+use App\ContentIndexer;
+
+require __DIR__ . '/../autoload.php';
+
+$config = require __DIR__ . '/../config.php';
+
+$appConfigFile = __DIR__ . '/../../App/config.php';
+if (is_file($appConfigFile)) {
+    $config = array_merge($config, require $appConfigFile);
+}
+
+if (!$config['content_index_enabled']) {
+    echo "content_index_enabled is false — nothing to do. See /admin/docs/content-index.\n";
+    exit;
+}
+
+// Ignores content_index_auto (the lazy-vs-CLI-only toggle) — this script
+// is the explicit-trigger path, so it always reindexes when run.
+ContentIndexer::reindex();
+
+echo "Content index rebuilt.\n";
@@ -48,9 +48,13 @@ return [
    // wholly replaced on a framework update — see
    // /admin/docs/getting-started's "Updating the framework" section) — a
    // top-level data/ directory, project-owned like App/, is the only safe
-    // place for it. migrations_dir is optional per connection; omit it to
-    // never run migrations against that connection (e.g. a read-only
-    // legacy database). NOTE: unlike every other key here, App/config.php
+    // place for it. migrations_dir is optional per connection (omit it to
+    // never run migrations against that connection, e.g. a read-only
+    // legacy database) and accepts either one path or an ordered list of
+    // roots — the default connection lists novaconium/migrations/ (framework
+    // -shipped schema, e.g. the content index — see /admin/docs/content-index)
+    // before App/migrations/ (project migrations), so framework migrations
+    // always apply first. NOTE: unlike every other key here, App/config.php
    // merges into db_connections one level deeper than a normal shallow
    // override — see the comment on Lib\Db::config() — so adding a second
    // connection there doesn't require repeating 'default'.
@@ -58,7 +62,10 @@ return [
        'default' => [
            'driver'         => 'sqlite',
            'path'           => __DIR__ . '/../data/novaconium.sqlite',
-            'migrations_dir' => __DIR__ . '/../App/migrations',
+            'migrations_dir' => [
+                __DIR__ . '/migrations',
+                __DIR__ . '/../App/migrations',
+            ],
        ],
    ],

@@ -73,4 +80,20 @@ return [
    // since a world-readable cached copy would otherwise permanently leak
    // the draft the first time an admin previewed it.
    'draft_routes' => [],
+
+    // Content index (see /admin/docs/content-index) — backs /sitemap.xml,
+    // /search, and blog tag browsing. Off by default: all three depend on
+    // SQLite (Lib\Db), a real dependency plenty of sites built on this
+    // framework won't want at all, the same reasoning that keeps Matomo
+    // and admin auth off by default above. When false, all three routes
+    // 404 exactly as if they didn't exist, and nothing ever touches
+    // Lib\Db because of this feature — no data/novaconium.sqlite gets
+    // created just because the code exists. content_index_auto only
+    // matters once enabled: true (the default) reindexes lazily,
+    // on-demand, the first time a stale index is actually needed (never on
+    // a normal page view); false disables that and leaves indexing
+    // entirely to `php novaconium/bin/index-content.php`, e.g. from a
+    // deploy step.
+    'content_index_enabled' => false,
+    'content_index_auto'    => true,
 ];
@@ -119,27 +119,44 @@ final class Db
    }

    /**
-     * Applies any *.sql file under $migrationsDir not yet recorded in this
-     * connection's own schema_migrations table, in filename order — so
-     * migrations are named with a numeric prefix (0001_create_x.sql,
-     * 0002_add_y.sql, ...) to control apply order. Each file is tracked by
-     * filename once applied and never re-run. Each connection tracks its
-     * own schema_migrations table in its own database, independent of any
-     * other configured connection. A connection with no migrations_dir set
-     * skips this entirely — e.g. a connection to a legacy database this
-     * project shouldn't manage schema for.
+     * Applies any *.sql file under $migrationsDirs not yet recorded in this
+     * connection's own schema_migrations table. $migrationsDirs is an
+     * ordered list of roots (a single string is accepted too, wrapped
+     * internally) — each root's own files are applied in filename order
+     * within that root (numeric prefixes, e.g. 0001_create_x.sql,
+     * 0002_add_y.sql, control order within a root), and roots are fully
+     * processed one at a time in the order given, not interleaved by
+     * filename across roots. This lets framework-shipped migrations (e.g.
+     * novaconium/migrations/) always apply before a project's own
+     * (App/migrations/) on the same connection — see
+     * novaconium/config.php's db_connections.default.migrations_dir and
+     * AGENTS.md.
     *
-     * Runs automatically on every first connection() call per process, per
-     * connection name — cheap (one query plus a directory glob), so no
-     * separate "migrate" step is required, matching the framework's
-     * zero-config philosophy elsewhere (e.g. static caching).
-     * novaconium/bin/migrate.php exists to run it explicitly for every
-     * configured connection (e.g. from a deploy script) without serving a
-     * request first.
+     * Each file is tracked once applied and never re-run, keyed by its path
+     * relative to the repo root (e.g. novaconium/migrations/0001_x.sql) —
+     * not bare filename, because two roots can each contain a same-named
+     * file (a framework migration and an unrelated project migration both
+     * numbered 0001_...); tracking by bare filename would make the second
+     * one seen look "already applied" and silently skip it. A relative
+     * path is also portable across environments, unlike a full absolute
+     * path, which would make every migration look "new" again after a
+     * clone/deploy to a different directory.
+     *
+     * A connection with no migrations_dir set skips this entirely — e.g. a
+     * connection to a legacy database this project shouldn't manage schema
+     * for. Runs automatically on every first connection() call per
+     * process, per connection name — cheap (one query plus a directory
+     * glob per root), so no separate "migrate" step is required, matching
+     * the framework's zero-config philosophy elsewhere (e.g. static
+     * caching). novaconium/bin/migrate.php exists to run it explicitly for
+     * every configured connection (e.g. from a deploy script) without
+     * serving a request first.
+     *
+     * @param string|string[]|null $migrationsDirs
     */
-    private static function migrate(PDO $pdo, ?string $migrationsDir): void
+    private static function migrate(PDO $pdo, string|array|null $migrationsDirs): void
    {
-        if ($migrationsDir === null) {
+        if ($migrationsDirs === null) {
            return;
        }

@@ -150,26 +167,38 @@ final class Db
            ')'
        );

-        if (!is_dir($migrationsDir)) {
-            return;
-        }
-
        $applied = $pdo->query('SELECT filename FROM schema_migrations')->fetchAll(PDO::FETCH_COLUMN);
        $applied = array_flip($applied);

-        $files = glob(rtrim($migrationsDir, '/') . '/*.sql') ?: [];
-        sort($files);
+        // novaconium/lib/ -> novaconium/ -> repo root, two levels up.
+        $repoRoot = rtrim(realpath(dirname(__DIR__, 2)) ?: dirname(__DIR__, 2), '/') . '/';

-        foreach ($files as $file) {
-            $filename = basename($file);
-            if (isset($applied[$filename])) {
+        foreach ((array) $migrationsDirs as $dir) {
+            if (!is_dir($dir)) {
                continue;
            }

-            $pdo->exec((string) file_get_contents($file));
+            $files = glob(rtrim($dir, '/') . '/*.sql') ?: [];
+            sort($files);

-            $insert = $pdo->prepare('INSERT INTO schema_migrations (filename, applied_at) VALUES (?, ?)');
-            $insert->execute([$filename, gmdate('Y-m-d\TH:i:s\Z')]);
+            foreach ($files as $file) {
+                // realpath() resolves any ".." left over from a
+                // migrations_dir like __DIR__ . '/../App/migrations' (glob()
+                // doesn't normalize the paths it returns), so the tracked
+                // name is clean, e.g. "App/migrations/0001_x.sql" rather
+                // than "novaconium/../App/migrations/0001_x.sql".
+                $resolved = realpath($file) ?: $file;
+                $trackedName = str_starts_with($resolved, $repoRoot) ? substr($resolved, strlen($repoRoot)) : basename($file);
+
+                if (isset($applied[$trackedName])) {
+                    continue;
+                }
+
+                $pdo->exec((string) file_get_contents($file));
+
+                $insert = $pdo->prepare('INSERT INTO schema_migrations (filename, applied_at) VALUES (?, ?)');
+                $insert->execute([$trackedName, gmdate('Y-m-d\TH:i:s\Z')]);
+            }
        }
    }

@@ -13,10 +13,10 @@ namespace Lib;
 * novaconium/src/Renderer.php), so this cleaning is a second layer, not the
 * only one. The real defense against SQL injection is parameterized queries
 * (PDO prepared statements) — never string concatenation or
- * sanitize-then-interpolate, however "cleaned" the input looks. There's no
- * database layer in this framework yet (SQLite groundwork is tracked as
- * Backlog in novaconium/ISSUES.md); when one lands, use PDO prepared statements
- * exclusively. This class deliberately does not (and will not) expose an
+ * sanitize-then-interpolate, however "cleaned" the input looks. Lib\Db (see
+ * /admin/docs/database) is the database layer — its query() method uses
+ * PDO prepared statements exclusively for this reason. This class
+ * deliberately does not (and will not) expose an
 * "sqlSafe()"-style method — no string transform makes arbitrary input safe
 * to concatenate into SQL, and a method implying otherwise would be actively
 * dangerous.
@@ -0,0 +1,25 @@
+CREATE TABLE IF NOT EXISTS content_pages (
+    route TEXT PRIMARY KEY,
+    title TEXT NOT NULL,
+    description TEXT NOT NULL,
+    keywords TEXT NOT NULL DEFAULT '',
+    changefreq TEXT NOT NULL DEFAULT '',
+    priority TEXT NOT NULL DEFAULT '',
+    source_mtime INTEGER NOT NULL
+);
+
+CREATE TABLE IF NOT EXISTS content_tags (
+    route TEXT NOT NULL,
+    tag TEXT NOT NULL,
+    PRIMARY KEY (route, tag)
+);
+
+CREATE INDEX IF NOT EXISTS idx_content_tags_tag ON content_tags(tag);
+
+CREATE VIRTUAL TABLE IF NOT EXISTS content_search USING fts5(route UNINDEXED, title, body);
+
+CREATE TABLE IF NOT EXISTS content_index_meta (
+    id INTEGER PRIMARY KEY CHECK (id = 1),
+    newest_source_mtime INTEGER NOT NULL,
+    indexed_at TEXT NOT NULL
+);
@@ -7,9 +7,19 @@
    <title>{% block title %}{{ site_name }}{% endblock %}</title>
    <meta name="description" content="{% block description %}A tiny, Hugo-flavored PHP micro-framework site.{% endblock %}">
    <meta name="robots" content="{% block robots %}index, follow{% endblock %}">
+    <meta name="keywords" content="{% block keywords %}{% endblock %}">
    <link rel="canonical" href="{% block canonical %}{{ request_path|default('/') }}{% endblock %}">
    <link rel="icon" href="/favicon.ico">

+    {# tags/changefreq/priority are never output as HTML — declared here
+       only so they're overridable per-page (same mechanism as every SEO
+       block above) and harvestable by ContentIndexer via Twig's
+       renderBlock() API, not rendered into the page itself. See
+       /admin/docs/content-index. #}
+    {% block tags %}{% endblock %}
+    {% block changefreq %}monthly{% endblock %}
+    {% block priority %}0.5{% endblock %}
+
    {# Open Graph / Facebook #}
    <meta property="og:type" content="{% block og_type %}website{% endblock %}">
    <meta property="og:title" content="{% block og_title %}{{ block('title') }}{% endblock %}">
@@ -15,6 +15,7 @@
                <li><a class="icon-link" href="/admin/docs/config">{{ icons.book() }}Configuration</a></li>
                <li><a class="icon-link" href="/admin/docs/database">{{ icons.book() }}Database</a></li>
                <li><a class="icon-link" href="/admin/docs/session">{{ icons.book() }}Session</a></li>
+                <li><a class="icon-link" href="/admin/docs/content-index">{{ icons.search() }}Content index</a></li>
                <li><a class="icon-link" href="/admin/docs/admin-auth">{{ icons.lock() }}Admin authentication</a></li>
                <li><a class="icon-link" href="/admin/docs/drafts">{{ icons.lock() }}Draft pages</a></li>
                <li><a class="icon-link" href="/admin/docs/layouts">{{ icons.book() }}Layouts</a></li>
@@ -0,0 +1,84 @@
+{% extends 'admin/docs/_layout/layout.twig' %}
+
+{% import '_layout/icons.twig' as icons %}
+
+{% block title %}Content index{% endblock %}
+
+{% block description %}The shared crawler behind /sitemap.xml, /search, and blog tag browsing.{% endblock %}
+
+{% block robots %}noindex, nofollow{% endblock %}
+
+{% block docs_content %}
+    <h1>Content index</h1>
+
+    <p>Three features — <a class="icon-link" href="/sitemap.xml">{{ icons.sitemap() }}/sitemap.xml</a>, <a class="icon-link" href="/search">{{ icons.search() }}/search</a>, and blog <a class="icon-link" href="/admin/docs/seo">{{ icons.tag() }}tag browsing</a> — share one underlying mechanism rather than three separate ones: a crawler (<code>App\ContentIndexer</code>, <code>novaconium/src/ContentIndexer.php</code>) that renders every routable page, pulls its metadata, and stores it in SQLite.</p>
+
+    <p>Content itself stays in files — plain Twig pages, same as everywhere else in this framework. Nothing here moves a page's body into a database; only metadata is extracted and indexed.</p>
+
+    <h2>Off by default</h2>
+
+    <p>All three features depend on <a class="icon-link" href="/admin/docs/database">{{ icons.book() }}SQLite</a> — a real dependency plenty of sites built on this framework won't want at all, the same reasoning that keeps Matomo and admin authentication off until a project opts in. <code>content_index_enabled</code> (default <code>false</code>) gates the whole subsystem:</p>
+
+    <pre><code>&lt;?php
+// App/config.php
+return [
+    'content_index_enabled' =&gt; true,
+];</code></pre>
+
+    <p>When it's off, <code>/sitemap.xml</code>, <code>/search</code>, and every <code>/blog/tag/&lt;tag&gt;</code> route return a plain <code>404</code> — exactly as if they didn't exist — and nothing ever touches <code>Lib\Db</code> because of this feature. <code>data/novaconium.sqlite</code> isn't created just because the code is present in the codebase; each consumer checks the flag before constructing anything that would open a connection.</p>
+
+    <h2>Declaring metadata on a page</h2>
+
+    <p>Four Twig blocks, declared in <code>novaconium/pages/_layout/layout.twig</code> next to the rest of the <a href="/admin/docs/seo">SEO blocks</a> — same override mechanism, just harvested rather than always rendered:</p>
+
+    <table>
+        <thead>
+            <tr><th>Block</th><th>Default</th><th>Used for</th></tr>
+        </thead>
+        <tbody>
+            <tr><td><code>keywords</code></td><td>empty</td><td>rendered as <code>&lt;meta name="keywords"&gt;</code> on every page (see <a href="/admin/docs/seo">SEO</a>)</td></tr>
+            <tr><td><code>tags</code></td><td>empty</td><td>comma-separated; indexed into <code>content_tags</code> for tag browsing</td></tr>
+            <tr><td><code>changefreq</code></td><td><code>monthly</code></td><td><code>/sitemap.xml</code>'s <code>&lt;changefreq&gt;</code></td></tr>
+            <tr><td><code>priority</code></td><td><code>0.5</code></td><td><code>/sitemap.xml</code>'s <code>&lt;priority&gt;</code></td></tr>
+        </tbody>
+    </table>
+
+    <pre><code>{% verbatim %}{% block tags %}yellow, the best, summer, hot{% endblock %}
+{% block changefreq %}weekly{% endblock %}
+{% block priority %}1.0{% endblock %}{% endverbatim %}</code></pre>
+
+    <p>See any post under <code>App/pages/blog/</code> for a working <code>tags</code> example.</p>
+
+    <h2>How the crawl works</h2>
+
+    <p><code>ContentIndexer::reindex()</code> walks every page under both page roots (<code>Overlay::listPageDirs()</code>, skipping <code>_</code>-prefixed, <code>404</code>, and <code>[param]</code>-wildcard directories — a wildcard route's concrete values aren't knowable without a data source, so dynamic routes aren't crawled yet), renders each one via <code>Renderer::renderForIndex()</code> (the same sidecar-then-Twig path a real request takes, minus HTTP output and the static cache write), and pulls each metadata block via Twig's own <code>renderBlock()</code> API — not by parsing <code>.twig</code> source — so App-over-novaconium overrides and layout inheritance resolve exactly the way they do for a real visit.</p>
+
+    <p>A page is skipped entirely (not stored) if it's listed in <a href="/admin/docs/drafts">{{ icons.lock() }}draft_routes</a>, or if its resolved <code>robots</code> block contains <code>noindex</code> — the same convention <a href="/admin/docs/seo">SEO</a> already documents for admin/internal pages. The rendered HTML is stripped with <code>strip_tags()</code> for a plain-text copy stored in a SQLite <a href="https://sqlite.org/fts5.html">FTS5</a> virtual table for search.</p>
+
+    <p>Every reindex is a full rebuild, not incremental — all three tables are truncated and repopulated inside one transaction, simple and correct at this scale rather than trying to diff what changed. Because the crawl renders every page's sidecar as a plain <code>GET</code> (forcing <code>$_SERVER['REQUEST_METHOD']</code> to <code>'GET'</code> for the duration, regardless of what triggered the reindex, and restoring it afterward), a POST-guarded sidecar action is never accidentally triggered by indexing — the same HTTP-safe-method hygiene any GET handler is already expected to have.</p>
+
+    <h2>When it runs</h2>
+
+    <p>Two paths, both calling the same <code>reindex()</code>:</p>
+
+    <ul>
+        <li><strong>Lazy (default):</strong> <code>ContentIndexer::ensureFresh()</code>, called from the <code>/sitemap.xml</code>, <code>/search</code>, and tag-browsing sidecars themselves — never from a normal page view, so browsing the rest of the site never pays any indexing cost. It compares the newest source-file mtime across every page (a cheap stat pass, no rendering) against the last indexed time, and only reindexes if something actually changed. Set <code>content_index_auto</code> to <code>false</code> to disable this and rely only on the CLI below.</li>
+        <li><strong>Explicit:</strong> <code>php novaconium/bin/index-content.php</code> — same shape as <code>bin/migrate.php</code>, for a deploy step. Always reindexes (ignores <code>content_index_auto</code>); exits immediately if <code>content_index_enabled</code> is <code>false</code>.</li>
+    </ul>
+
+    <h2>Search</h2>
+
+    <p><code>/search?q=...</code> runs an FTS5 <code>MATCH</code> query. The search term is wrapped as a quoted phrase (embedded <code>"</code> doubled) before binding — parameter binding stops SQL injection, but the bound value is still parsed as its own FTS5 query-language expression, so an unescaped <code>"</code> or FTS operator in user input could otherwise throw a syntax error or search for something unintended.</p>
+
+    <h2>Sitemap</h2>
+
+    <p><code>/sitemap.xml</code> lists every indexed page's <code>&lt;loc&gt;</code>, <code>&lt;lastmod&gt;</code> (the page's own source file mtime), <code>&lt;changefreq&gt;</code>, and <code>&lt;priority&gt;</code>.</p>
+
+    <h2>Blog tag browsing</h2>
+
+    <p><code>App/pages/blog/tag/[tag]/index.php</code> — project-owned, since <code>blog/</code> itself is project content — uses the <a href="/admin/docs/routing">{{ icons.link() }}<code>[param]</code></a> capture to read the tag from the URL and queries <code>content_tags</code> joined to <code>content_pages</code>. <code>App/pages/blog/index.php</code>'s own hand-written post list is untouched by any of this — it stays the source of truth for the main blog listing; <code>content_tags</code> is a derived index built from each post's own <code>tags</code> block, not a replacement for it.</p>
+
+    <h2>Migrations, and the two-root scan</h2>
+
+    <p>The schema (<code>content_pages</code>, <code>content_tags</code>, <code>content_search</code>, <code>content_index_meta</code>) ships as a framework migration, <code>novaconium/migrations/0001_create_content_index.sql</code> — the first framework-owned migration, and the reason <a href="/admin/docs/database">{{ icons.book() }}<code>migrations_dir</code></a> now accepts an ordered list of roots instead of a single path: the default connection's <code>migrations_dir</code> is <code>[novaconium/migrations, App/migrations]</code>, so framework migrations always apply before a project's own on the same connection.</p>
+{% endblock %}
@@ -41,7 +41,10 @@ $legacyRows = Db::query('SELECT * FROM widgets', [], 'legacy')-&gt;fetchAll();</
    'default' =&gt; [
        'driver'         =&gt; 'sqlite',
        'path'           =&gt; __DIR__ . '/../data/novaconium.sqlite',
-        'migrations_dir' =&gt; __DIR__ . '/../App/migrations',
+        'migrations_dir' =&gt; [
+            __DIR__ . '/migrations',
+            __DIR__ . '/../App/migrations',
+        ],
    ],
 ],</code></pre>

@@ -66,13 +69,13 @@ return [

    <p><strong>This is the one config key in the project that doesn't follow the usual shallow-merge rule.</strong> Every other <code>App/config.php</code> key replaces the framework default outright (see <a class="icon-link" href="/admin/docs/config">{{ icons.book() }}Configuration</a>) — but a plain shallow merge on <code>db_connections</code> would let the snippet above silently delete the framework's <code>default</code> connection just by adding <code>legacy</code>. So <code>Lib\Db</code> merges <code>db_connections</code> one level deeper, by connection name: the example above ends up with both <code>default</code> (SQLite, from the framework) and <code>legacy</code> (MySQL, from <code>App/config.php</code>) configured at once. To actually replace <code>default</code>, redeclare a <code>default</code> key yourself.</p>

-    <p>Only <code>'sqlite'</code> and <code>'mysql'</code> are implemented as <code>driver</code> values. <code>migrations_dir</code> is optional per connection — omit it to never run migrations against that connection (e.g. a legacy database this project shouldn't manage schema for).</p>
+    <p>Only <code>'sqlite'</code> and <code>'mysql'</code> are implemented as <code>driver</code> values. <code>migrations_dir</code> is optional per connection — omit it to never run migrations against that connection (e.g. a legacy database this project shouldn't manage schema for) — and accepts either a single path (<code>legacy</code>'s example above) or an ordered list of roots (the default connection's example above), each scanned for its own <code>*.sql</code> files.</p>

    <p>The default connection's <code>path</code> lives in a top-level <code>data/</code> directory — a sibling of <code>App/</code>, <code>novaconium/</code>, and <code>public/</code>, not nested inside any of them. This is deliberate: it can't live under <code>public/</code> (would be directly web-accessible), and it can't live under <code>novaconium/</code> either, since <a class="icon-link" href="/admin/docs/getting-started">{{ icons.book() }}updating the framework</a> means overwriting that whole directory — anything persisted there would be destroyed by the next update. <code>data/</code> is project-owned, like <code>App/</code>, and untouched by a framework update. Its contents (<code>*.sqlite</code> and the SQLite journal/WAL/SHM sidecar files) are gitignored; only a <code>.gitkeep</code> is tracked so the directory exists in a fresh clone.</p>

    <h2>Migrations</h2>

-    <p>Plain <code>.sql</code> files under each connection's own <code>migrations_dir</code>, applied in filename order — name them with a numeric prefix to control ordering:</p>
+    <p>Plain <code>.sql</code> files under each connection's own <code>migrations_dir</code> root(s), applied in filename order within each root — name them with a numeric prefix to control ordering:</p>

    <pre><code>-- App/migrations/0001_create_posts.sql
 CREATE TABLE posts (
@@ -81,9 +84,9 @@ CREATE TABLE posts (
    body TEXT NOT NULL
 );</code></pre>

-    <p>Each file is tracked by filename in that connection's own <code>schema_migrations</code> table (created automatically in that connection's database) and only ever run once — <code>default</code> and <code>legacy</code> each track their own applied migrations independently. Migrations for a given connection apply automatically the first time it's used in a process — zero-config, the same "just works" philosophy as static caching — or explicitly for every configured connection at once, without serving a request first:</p>
+    <p>Each file is tracked by its path relative to the repo root (e.g. <code>novaconium/migrations/0001_x.sql</code>) in that connection's own <code>schema_migrations</code> table (created automatically in that connection's database) and only ever run once — <code>default</code> and <code>legacy</code> each track their own applied migrations independently. Tracking the repo-relative path rather than the bare filename matters once a connection has more than one <code>migrations_dir</code> root: two roots can each contain a same-named file (e.g. a framework <code>0001_...sql</code> and an unrelated project <code>0001_...sql</code>) without one being mistaken for the other already having run. Migrations for a given connection apply automatically the first time it's used in a process — zero-config, the same "just works" philosophy as static caching — or explicitly for every configured connection at once, without serving a request first:</p>

    <pre><code>php novaconium/bin/migrate.php</code></pre>

-    <p>Point two connections' <code>migrations_dir</code> at different directories (e.g. <code>App/migrations/</code> for <code>default</code>, <code>App/migrations/legacy/</code> for <code>legacy</code>) if their SQL genuinely diverges between drivers; otherwise the same directory works for both as long as the SQL in it is portable. The framework itself ships no core tables yet, so there's no second <code>novaconium/migrations/</code> root to merge in — if a future framework feature needs a shipped migration (e.g. <a href="https://git.4lt.ca/4lt/novaconium/issues">admin login's user table</a>), this can extend to the same App-over-novaconium two-root scan used for pages and lib.</p>
+    <p>When a connection has multiple <code>migrations_dir</code> roots, each root is fully processed in the order given — the default connection's own framework root (<code>novaconium/migrations/</code>) always applies completely before its project root (<code>App/migrations/</code>), not interleaved by filename across the two. <code>novaconium/migrations/0001_create_content_index.sql</code> (see <a href="/admin/docs/content-index">{{ icons.search() }}Content index</a>) is the first framework-shipped migration — the reason this two-root support exists at all, extending the same App-over-novaconium override pattern used for pages and lib to migrations too. Point two connections' <code>migrations_dir</code> at different directories (e.g. <code>App/migrations/</code> for <code>default</code>, <code>App/migrations/legacy/</code> for <code>legacy</code>) if their SQL genuinely diverges between drivers; otherwise the same directory works for both as long as the SQL in it is portable.</p>
 {% endblock %}
@@ -4,7 +4,7 @@

 {% block title %}Docs{% endblock %}

-{% block description %}Framework documentation: routing, sidecars, forms, libraries, database, session, admin authentication, draft pages, layouts, caching, styling.{% endblock %}
+{% block description %}Framework documentation: routing, sidecars, forms, libraries, database, session, content index, admin authentication, draft pages, layouts, caching, styling.{% endblock %}

 {% block robots %}noindex, nofollow{% endblock %}

@@ -20,6 +20,7 @@
        <li><a class="icon-link" href="/admin/docs/config">{{ icons.book() }}Configuration</a> — override framework settings from <code>App/config.php</code>.</li>
        <li><a class="icon-link" href="/admin/docs/database">{{ icons.book() }}Database</a> — <code>Lib\Db</code>, a thin PDO wrapper (SQLite and MySQL) with named, simultaneous connections and a plain-SQL migration convention.</li>
        <li><a class="icon-link" href="/admin/docs/session">{{ icons.book() }}Session</a> — <code>Lib\Session</code>, a thin wrapper around native PHP sessions, with CodeIgniter-style flash data.</li>
+        <li><a class="icon-link" href="/admin/docs/content-index">{{ icons.search() }}Content index</a> — the shared crawler behind <code>/sitemap.xml</code>, <code>/search</code>, and blog tag browsing. Off by default.</li>
        <li><a class="icon-link" href="/admin/docs/admin-auth">{{ icons.lock() }}Admin authentication</a> — gate <code>/admin/*</code> behind HTTP Basic Auth, reusable for any future admin page.</li>
        <li><a class="icon-link" href="/admin/docs/drafts">{{ icons.lock() }}Draft pages</a> — let an admin preview a page before the public can see it, reusing the same auth check.</li>
        <li><a class="icon-link" href="/admin/docs/layouts">{{ icons.book() }}Layouts</a> — pages and layouts are overridable, just like <code>Lib\</code>.</li>
@@ -9,7 +9,7 @@
 {% block docs_content %}
    <h1>SEO boilerplate</h1>

-    <p><code>novaconium/pages/_layout/layout.twig</code> (the root layout every page extends, directly or via a nested layout) renders a full set of SEO meta tags in <code>&lt;head&gt;</code>: viewport, description, robots, canonical link, Open Graph, and Twitter Card. Each piece is a named Twig block with a sensible default, so any page can override just the piece it needs without touching the rest of <code>&lt;head&gt;</code>.</p>
+    <p><code>novaconium/pages/_layout/layout.twig</code> (the root layout every page extends, directly or via a nested layout) renders a full set of SEO meta tags in <code>&lt;head&gt;</code>: viewport, description, robots, keywords, canonical link, Open Graph, and Twitter Card. Each piece is a named Twig block with a sensible default, so any page can override just the piece it needs without touching the rest of <code>&lt;head&gt;</code>. Three more blocks — <code>tags</code>, <code>changefreq</code>, <code>priority</code> — are declared the same way but never rendered into the page at all; see <a href="/admin/docs/content-index">Content index</a> for what reads them.</p>

    <h2>Blocks you can override</h2>

@@ -21,6 +21,10 @@
            <tr><td><code>title</code></td><td><code>site_name</code> config value</td><td><code>&lt;title&gt;</code>, and reused by <code>og:title</code> / <code>twitter:title</code></td></tr>
            <tr><td><code>description</code></td><td>generic site description</td><td><code>&lt;meta name="description"&gt;</code>, and reused by <code>og:description</code> / <code>twitter:description</code></td></tr>
            <tr><td><code>robots</code></td><td><code>index, follow</code></td><td><code>&lt;meta name="robots"&gt;</code></td></tr>
+            <tr><td><code>keywords</code></td><td>empty</td><td><code>&lt;meta name="keywords"&gt;</code></td></tr>
+            <tr><td><code>tags</code></td><td>empty</td><td>not rendered — comma-separated, harvested by <a href="/admin/docs/content-index">the content index</a> for blog tag browsing</td></tr>
+            <tr><td><code>changefreq</code></td><td><code>monthly</code></td><td>not rendered — harvested for <code>/sitemap.xml</code>'s <code>&lt;changefreq&gt;</code></td></tr>
+            <tr><td><code>priority</code></td><td><code>0.5</code></td><td>not rendered — harvested for <code>/sitemap.xml</code>'s <code>&lt;priority&gt;</code></td></tr>
            <tr><td><code>canonical</code></td><td><code>{{ '{{ request_path }}' }}</code></td><td><code>&lt;link rel="canonical"&gt;</code>, and reused by <code>og:url</code></td></tr>
            <tr><td><code>og_type</code></td><td><code>website</code></td><td><code>&lt;meta property="og:type"&gt;</code></td></tr>
            <tr><td><code>og_title</code></td><td><code>{{ '{{ block(\'title\') }}' }}</code></td><td><code>&lt;meta property="og:title"&gt;</code></td></tr>
@@ -60,6 +64,10 @@
 {% block description %}One or two sentences describing this page.{% endblock %}

 {% block robots %}index, follow{% endblock %}
+{% block keywords %}{% endblock %}
+{% block tags %}{% endblock %}
+{% block changefreq %}monthly{% endblock %}
+{% block priority %}0.5{% endblock %}
 {% block canonical %}{{ request_path|default('/') }}{% endblock %}

 {% block og_type %}website{% endblock %}
@@ -63,7 +63,7 @@ $all   = Input::post();                // the whole cleaned $_POST array</code><

    <p>Calling <code>post()</code>/<code>get()</code> with no key returns the entire cleaned array (handy for passing straight to <code>SpamGuard::isSpam()</code>, as below); calling it with a key returns that key's cleaned value, or the given default if it's absent. Nested arrays (e.g. a checkbox group posted as <code>tags[]</code>) are cleaned recursively. The result is memoized per request, so calling <code>Input::post()</code> repeatedly across a sidecar doesn't re-clean the superglobal each time.</p>

-    <p><strong>This is not SQL-injection protection.</strong> The cleaning <code>Input</code> does (trim + strip tags, via <code>Lib\Validate::clean()</code>, plus null-byte stripping) is defense-in-depth against HTML/script injection in output contexts — Twig already autoescapes <code>{{ }}</code> output by default (see <code>novaconium/src/Renderer.php</code>), so this is a second layer, not the only one. No string transform makes arbitrary input safe to concatenate into a SQL query; the real defense is parameterized queries (PDO prepared statements). There's no database layer in this framework yet (SQLite groundwork is Backlog — see <code>novaconium/ISSUES.md</code>); when one lands, use prepared statements exclusively. <code>Input</code> deliberately has no <code>sqlSafe()</code>-style method, since a method implying "cleaned = safe to interpolate into SQL" would be actively dangerous.</p>
+    <p><strong>This is not SQL-injection protection.</strong> The cleaning <code>Input</code> does (trim + strip tags, via <code>Lib\Validate::clean()</code>, plus null-byte stripping) is defense-in-depth against HTML/script injection in output contexts — Twig already autoescapes <code>{{ '{{ }}' }}</code> output by default (see <code>novaconium/src/Renderer.php</code>), so this is a second layer, not the only one. No string transform makes arbitrary input safe to concatenate into a SQL query; the real defense is parameterized queries. <a href="/admin/docs/database">Lib\Db</a>'s <code>query()</code> method uses PDO prepared statements exclusively for exactly this reason. <code>Input</code> deliberately has no <code>sqlSafe()</code>-style method, since a method implying "cleaned = safe to interpolate into SQL" would be actively dangerous.</p>

    <p>One documented exception: a field that needs an exact, unmodified value — a password about to be hashed, say — should read <code>$_POST</code> directly instead of going through <code>Input::post()</code>. Cleaning would silently strip characters like <code>&lt;</code>/<code>&gt;</code> before hashing, producing a hash that doesn't match what's actually typed later. See <code>novaconium/pages/admin/password-hash/index.php</code> for the one place this framework does that on purpose.</p>

@@ -0,0 +1,103 @@
+<?php
+
+// /search — a framework default (novaconium/pages/, not App/pages/) since
+// full-text search over the whole site is generic machinery, not project
+// content. Has both this sidecar and an index.twig, unlike sitemap.xml —
+// it renders a real HTML page (a form plus results), not a bypass-Twig
+// Response.
+
+use App\ContentIndexer;
+use App\Response;
+use Lib\Db;
+use Lib\Input;
+
+// Same two-step config load bootstrap.php/bin scripts use — this sidecar
+// isn't handed $config, so it loads its own copy to read
+// content_index_enabled before touching Lib\Db at all.
+$config = require __DIR__ . '/../../config.php';
+$appConfigFile = __DIR__ . '/../../../App/config.php';
+if (is_file($appConfigFile)) {
+    $config = array_merge($config, require $appConfigFile);
+}
+
+// Content index is off by default (depends on SQLite) — see
+// /admin/docs/content-index. When it's off, this route must 404 exactly
+// like a page that doesn't exist, and never construct a Lib\Db connection
+// (which would otherwise create data/novaconium.sqlite just because this
+// file exists, even on a site that never opted in).
+if (!$config['content_index_enabled']) {
+    return Response::html('404 Not Found', 404);
+}
+
+// Input::get(), not $_GET directly — see /admin/docs/sidecars' "Form
+// security" section. Only Lib\Input's HTML/script-injection cleaning
+// matters here (this value never reaches SQL unparameterized either way,
+// see the FTS5 escaping note below).
+$query = trim((string) Input::get('q', ''));
+$results = [];
+
+// Bare /search (no ?q=) just shows the empty form — skip the query and
+// the reindex-freshness check entirely, so landing on the page cold costs
+// nothing beyond the normal page render.
+if ($query !== '') {
+    // Lazy reindex-if-stale — a no-op on most requests (only actually
+    // reindexes when a page's source file changed since the last index).
+    // Deliberately guarded by $query !== '' rather than called
+    // unconditionally at the top of the file: ContentIndexer's own crawl
+    // renders every real page, including this one, so /search visiting
+    // itself with an empty query during a crawl must NOT trigger another
+    // reindex — ContentIndexer also has its own reentrancy guard for this
+    // (see its docblock), but not paying the freshness-check cost on
+    // every bare page load is a second, independent reason this call sits
+    // inside the if.
+    ContentIndexer::ensureFresh();
+
+    // Parameter binding prevents SQL injection, but the bound value is
+    // still parsed as its own FTS5 query-language expression, not a plain
+    // string — a literal " or an FTS operator in $query could otherwise
+    // throw a syntax error or search for something unintended. Wrapping it
+    // as a quoted phrase (doubling any embedded ") makes the whole query
+    // an FTS5 phrase-match literal, neutralizing that syntax entirely.
+    // Verified against a literal ", "*", "OR", and a "'; DROP TABLE ..."
+    // attempt — all return normal (possibly empty) results, no fatal, no
+    // effect on the database.
+    $ftsQuery = '"' . str_replace('"', '""', $query) . '"';
+
+    // content_search is the FTS5 virtual table ContentIndexer::reindex()
+    // populates with route/title/stripped-HTML body — "rank" is an FTS5
+    // built-in column (not one we define) giving relevance ordering, most
+    // relevant first. LIMIT 50 is just a sanity cap, not real pagination.
+    $results = Db::query(
+        'SELECT route, title FROM content_search WHERE content_search MATCH ? ORDER BY rank LIMIT 50',
+        [$ftsQuery]
+    )->fetchAll(PDO::FETCH_ASSOC);
+
+    // A second query rather than joining content_pages into the FTS5
+    // query directly — content_search is a virtual table, and mixing a
+    // real table JOIN into an FTS5 MATCH query is more fragile than doing
+    // the description lookup separately. IN (...) with one placeholder
+    // per route, built from the exact route set the first query returned
+    // — never string-interpolating $query or user input into this SQL,
+    // only the already-fetched, already-trusted route values.
+    if ($results !== []) {
+        $routes = array_column($results, 'route');
+        $placeholders = implode(',', array_fill(0, count($routes), '?'));
+        $descriptions = Db::query(
+            "SELECT route, description FROM content_pages WHERE route IN ({$placeholders})",
+            $routes
+        )->fetchAll(PDO::FETCH_KEY_PAIR);
+
+        foreach ($results as &$result) {
+            $result['description'] = $descriptions[$result['route']] ?? '';
+        }
+        unset($result); // break the foreach-by-reference alias, standard PHP gotcha
+    }
+}
+
+// $results stays [] (not an error) for a blank query or one with zero
+// matches — the twig template only distinguishes those two cases by
+// checking $query itself, not by any error flag.
+return [
+    'query'   => $query,
+    'results' => $results,
+];
@@ -0,0 +1,31 @@
+{% extends layout %}
+
+{% import '_layout/icons.twig' as icons %}
+
+{% block title %}Search{% endblock %}
+{% block description %}Search this site.{% endblock %}
+{% block robots %}noindex, follow{% endblock %}
+
+{% block content %}
+    <h1 class="icon-heading">{{ icons.search() }}Search</h1>
+
+    <form method="get" action="/search">
+        <input type="search" name="q" value="{{ query }}" placeholder="Search…" aria-label="Search">
+        <button type="submit">Search</button>
+    </form>
+
+    {% if query != '' %}
+        {% if results|length > 0 %}
+            <ul class="post-list">
+                {% for result in results %}
+                    <li>
+                        <a href="{{ result.route }}">{{ result.title }}</a>
+                        {% if result.description %}<p>{{ result.description }}</p>{% endif %}
+                    </li>
+                {% endfor %}
+            </ul>
+        {% else %}
+            <p>No results for &ldquo;{{ query }}&rdquo;.</p>
+        {% endif %}
+    {% endif %}
+{% endblock %}
@@ -0,0 +1,73 @@
+<?php
+
+// /sitemap.xml — a directory literally named "sitemap.xml" under
+// novaconium/pages/, a framework default (like /admin/*), since sitemap
+// generation is generic machinery, not project content. Router only ever
+// splits the request path on "/", so a directory named "sitemap.xml"
+// resolves the literal /sitemap.xml URL correctly — there's no special
+// extension-routing mechanism involved. Sidecar-only: no index.twig next
+// to this file, since Response::xml() bypasses Twig entirely (see
+// /admin/docs/sidecars' JSON-only example for the same pattern).
+
+use App\ContentIndexer;
+use App\Response;
+use Lib\Db;
+
+// Same two-step config load bootstrap.php/bin scripts use — this sidecar
+// isn't handed $config, so it loads its own copy to read
+// content_index_enabled before touching Lib\Db at all.
+$config = require __DIR__ . '/../../config.php';
+$appConfigFile = __DIR__ . '/../../../App/config.php';
+if (is_file($appConfigFile)) {
+    $config = array_merge($config, require $appConfigFile);
+}
+
+// Content index is off by default (depends on SQLite) — see
+// /admin/docs/content-index. When it's off, this route must 404 exactly
+// like a page that doesn't exist, and never construct a Lib\Db connection
+// (which would otherwise create data/novaconium.sqlite just because this
+// file exists, even on a site that never opted in).
+if (!$config['content_index_enabled']) {
+    return Response::html('404 Not Found', 404);
+}
+
+// Lazy reindex-if-stale — a no-op on most requests (only actually
+// reindexes when a page's source file changed since the last index).
+// ContentIndexer's crawl itself never renders this route: a sidecar
+// returning a Response (this one always does) has nothing meaningful to
+// index, so Renderer::renderForIndex() returns null for it and
+// ContentIndexer skips it — no reentrancy concern here, unlike /search
+// (see that file's comments), which does get crawled since it renders
+// normally on an empty query.
+ContentIndexer::ensureFresh();
+
+// One row per indexed page — populated entirely by ContentIndexer::
+// reindex(), never written to directly here. changefreq/priority come
+// from each page's own {% block changefreq %}/{% block priority %}
+// (defaults: monthly / 0.5 — see novaconium/pages/_layout/layout.twig),
+// source_mtime is that page's source file's own mtime, used below as
+// <lastmod> so it reflects when the content actually last changed, not
+// when the index was last rebuilt.
+$pages = Db::query('SELECT route, changefreq, priority, source_mtime FROM content_pages ORDER BY route')
+    ->fetchAll(PDO::FETCH_ASSOC);
+
+// Plain string concatenation rather than DOMDocument — this site's scale
+// doesn't need a real XML builder, but every value is still
+// htmlspecialchars()-escaped (ENT_XML1, not the HTML default) since
+// route/changefreq/priority all ultimately come from page-author-controlled
+// Twig blocks, not hardcoded constants.
+$xml = '<?xml version="1.0" encoding="UTF-8"?>' . "\n";
+$xml .= '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">' . "\n";
+
+foreach ($pages as $page) {
+    $xml .= '  <url>' . "\n";
+    $xml .= '    <loc>' . htmlspecialchars($page['route'], ENT_XML1) . '</loc>' . "\n";
+    $xml .= '    <lastmod>' . gmdate('Y-m-d', (int) $page['source_mtime']) . '</lastmod>' . "\n";
+    $xml .= '    <changefreq>' . htmlspecialchars($page['changefreq'], ENT_XML1) . '</changefreq>' . "\n";
+    $xml .= '    <priority>' . htmlspecialchars($page['priority'], ENT_XML1) . '</priority>' . "\n";
+    $xml .= '  </url>' . "\n";
+}
+
+$xml .= '</urlset>' . "\n";
+
+return Response::xml($xml);
@@ -0,0 +1,234 @@
+<?php
+
+namespace App;
+
+use Lib\Db;
+
+/**
+ * Crawls every routable page, renders it, and indexes it into SQLite —
+ * the shared mechanism behind /sitemap.xml, /search, and blog tag browsing
+ * (see /admin/docs/content-index). Content itself stays in files (Twig
+ * pages); this only extracts and stores metadata (title/description/
+ * keywords/tags/changefreq/priority, each pulled via Renderer::
+ * renderForIndex()'s Twig renderBlock() calls) plus a stripped-HTML plain-
+ * text copy of the body for full-text search (SQLite FTS5).
+ *
+ * Off by default: config['content_index_enabled'] gates the whole
+ * subsystem, since this is a real SQLite dependency plenty of sites built
+ * on this framework won't want at all — the same posture as Matomo/admin
+ * auth. Every public method here is a no-op when it's false.
+ *
+ * Two trigger paths, both funneling into reindex():
+ *   - ensureFresh() — lazy, called from the sitemap/search/tag-browsing
+ *     sidecars themselves (never from a normal page view), reindexing only
+ *     if something changed since the last index (a cheap filemtime() scan,
+ *     no rendering, decides this) and only if config['content_index_auto']
+ *     is true (the default).
+ *   - novaconium/bin/index-content.php — explicit, ignores content_index_auto,
+ *     for projects that would rather trigger indexing from a deploy step.
+ *
+ * A crawl invokes every page's sidecar the same way a real GET request
+ * would (this is how it discovers noindex pages and pulls metadata) —
+ * sidecars are expected to be side-effect-free for non-POST requests
+ * anyway (ordinary HTTP-safe-method hygiene, not a new constraint this
+ * introduces), but reindex() additionally forces $_SERVER['REQUEST_METHOD']
+ * to 'GET' for the duration of the crawl and restores whatever it was
+ * before, so a lazy reindex triggered from within a POST request (however
+ * unlikely for the sidecars this ships with) can never leak that POST into
+ * an unrelated page's sidecar.
+ */
+final class ContentIndexer
+{
+    /**
+     * Guards against reentrancy: the crawl itself renders every routable
+     * page, including /search (a real content_index_enabled consumer,
+     * since it has no other reason not to be crawled) — and /search's own
+     * sidecar calls ensureFresh(). Without this guard, that nested call
+     * would see itself as "in progress" and either start a second
+     * reindex() mid-transaction (PDO fatals on a nested beginTransaction())
+     * or, if it didn't fatal, silently corrupt the outer crawl's result.
+     * Both ensureFresh() and reindex() no-op while a reindex is already
+     * running on this call stack.
+     */
+    private static bool $indexing = false;
+
+    public static function ensureFresh(): void
+    {
+        if (self::$indexing) {
+            return;
+        }
+
+        $config = self::config();
+
+        if (!$config['content_index_enabled'] || !$config['content_index_auto']) {
+            return;
+        }
+
+        if (self::isStale($config)) {
+            self::reindex();
+        }
+    }
+
+    public static function reindex(): void
+    {
+        if (self::$indexing) {
+            return;
+        }
+
+        $config = self::config();
+
+        if (!$config['content_index_enabled']) {
+            return;
+        }
+
+        $pdo = Db::connection();
+        $renderer = self::renderer($config);
+        $routes = Overlay::listPageDirs($config['pages_dirs']);
+
+        $originalMethod = $_SERVER['REQUEST_METHOD'] ?? null;
+        $_SERVER['REQUEST_METHOD'] = 'GET';
+        self::$indexing = true;
+
+        try {
+            $pdo->beginTransaction();
+
+            $pdo->exec('DELETE FROM content_pages');
+            $pdo->exec('DELETE FROM content_tags');
+            $pdo->exec('DELETE FROM content_search');
+
+            $insertPage = $pdo->prepare(
+                'INSERT INTO content_pages (route, title, description, keywords, changefreq, priority, source_mtime) ' .
+                'VALUES (?, ?, ?, ?, ?, ?, ?)'
+            );
+            $insertTag = $pdo->prepare('INSERT INTO content_tags (route, tag) VALUES (?, ?)');
+            $insertSearch = $pdo->prepare('INSERT INTO content_search (route, title, body) VALUES (?, ?, ?)');
+
+            $newestMtime = 0;
+
+            foreach ($routes as $dir) {
+                if (in_array($dir, $config['draft_routes'], true)) {
+                    continue;
+                }
+
+                $mtime = self::sourceMtime($config['pages_dirs'], $dir);
+                $newestMtime = max($newestMtime, $mtime);
+
+                $route = new Route($dir, [], true);
+                $indexed = $renderer->renderForIndex($route);
+
+                if ($indexed === null || str_contains($indexed->robots, 'noindex')) {
+                    continue;
+                }
+
+                $routeUrl = $dir === '' ? '/' : '/' . $dir;
+
+                $insertPage->execute([
+                    $routeUrl,
+                    $indexed->title,
+                    $indexed->description,
+                    $indexed->keywords,
+                    $indexed->changefreq,
+                    $indexed->priority,
+                    $mtime,
+                ]);
+
+                foreach (self::splitList($indexed->tags) as $tag) {
+                    $insertTag->execute([$routeUrl, $tag]);
+                }
+
+                $insertSearch->execute([$routeUrl, $indexed->title, strip_tags($indexed->html)]);
+            }
+
+            $pdo->prepare('DELETE FROM content_index_meta')->execute();
+            $pdo->prepare('INSERT INTO content_index_meta (id, newest_source_mtime, indexed_at) VALUES (1, ?, ?)')
+                ->execute([$newestMtime, gmdate('Y-m-d\TH:i:s\Z')]);
+
+            $pdo->commit();
+        } catch (\Throwable $e) {
+            if ($pdo->inTransaction()) {
+                $pdo->rollBack();
+            }
+            throw $e;
+        } finally {
+            if ($originalMethod === null) {
+                unset($_SERVER['REQUEST_METHOD']);
+            } else {
+                $_SERVER['REQUEST_METHOD'] = $originalMethod;
+            }
+            self::$indexing = false;
+        }
+    }
+
+    /**
+     * @param array<string,mixed> $config
+     */
+    private static function isStale(array $config): bool
+    {
+        $pdo = Db::connection();
+
+        $meta = $pdo->query('SELECT newest_source_mtime FROM content_index_meta WHERE id = 1')->fetch();
+        if ($meta === false) {
+            return true;
+        }
+
+        $newest = 0;
+        foreach (Overlay::listPageDirs($config['pages_dirs']) as $dir) {
+            $newest = max($newest, self::sourceMtime($config['pages_dirs'], $dir));
+        }
+
+        return $newest > (int) $meta['newest_source_mtime'];
+    }
+
+    /**
+     * @param string[] $pagesDirs
+     */
+    private static function sourceMtime(array $pagesDirs, string $dir): int
+    {
+        $twig = Overlay::findFile($pagesDirs, $dir === '' ? 'index.twig' : $dir . '/index.twig');
+        $php = Overlay::findFile($pagesDirs, $dir === '' ? 'index.php' : $dir . '/index.php');
+
+        $mtimes = array_filter([
+            $twig !== null ? filemtime($twig) : false,
+            $php !== null ? filemtime($php) : false,
+        ]);
+
+        return $mtimes === [] ? 0 : (int) max($mtimes);
+    }
+
+    /**
+     * @return string[]
+     */
+    private static function splitList(string $value): array
+    {
+        $items = array_map('trim', explode(',', $value));
+
+        return array_values(array_filter($items, fn (string $item) => $item !== ''));
+    }
+
+    /**
+     * @param array<string,mixed> $config
+     */
+    private static function renderer(array $config): Renderer
+    {
+        return new Renderer($config['pages_dirs'], new Cache($config['cache_dir']));
+    }
+
+    /**
+     * @return array<string,mixed>
+     */
+    private static function config(): array
+    {
+        $config = require __DIR__ . '/../config.php';
+
+        $appConfigFile = __DIR__ . '/../../App/config.php';
+        if (is_file($appConfigFile)) {
+            $appConfig = require $appConfigFile;
+            $defaultConnections = $config['db_connections'];
+            $appConnections = $appConfig['db_connections'] ?? [];
+            $config = array_merge($config, $appConfig);
+            $config['db_connections'] = array_merge($defaultConnections, $appConnections);
+        }
+
+        return $config;
+    }
+}
@@ -0,0 +1,26 @@
+<?php
+
+namespace App;
+
+/**
+ * The result of Renderer::renderForIndex() — everything App\ContentIndexer
+ * needs from a single rendered page, without any HTTP output or cache
+ * write. Each metadata field is pulled via Twig's TemplateWrapper::
+ * renderBlock(), so it reflects whatever that specific page (or, absent an
+ * override, the layout's default) declared — see novaconium/pages/_layout/
+ * layout.twig and /admin/docs/seo.
+ */
+final class IndexedPage
+{
+    public function __construct(
+        public readonly string $html,
+        public readonly string $title,
+        public readonly string $description,
+        public readonly string $keywords,
+        public readonly string $tags,
+        public readonly string $robots,
+        public readonly string $changefreq,
+        public readonly string $priority,
+    ) {
+    }
+}
@@ -67,6 +67,59 @@ final class Overlay
        return null;
    }

+    /**
+     * The deduped set of every relative path, across both roots, that
+     * Router::resolve() could route to (has an index.twig or index.php) —
+     * used by App\ContentIndexer to crawl the whole site. `_`-prefixed and
+     * `[param]`-wildcard directories are skipped entirely, matching
+     * Router::resolve()'s own reserved-segment rule and the sitemap/search
+     * groundwork's stated V1 limitation: a wildcard route's concrete
+     * values aren't knowable without a data source, so dynamic routes
+     * aren't crawled (yet).
+     *
+     * @param string[] $roots
+     * @return string[]
+     */
+    public static function listPageDirs(array $roots): array
+    {
+        $found = [];
+
+        foreach ($roots as $root) {
+            self::walk($root, '', $found);
+        }
+
+        return array_keys($found);
+    }
+
+    /**
+     * @param array<string,true> $found
+     */
+    private static function walk(string $root, string $relative, array &$found): void
+    {
+        $dir = self::join($root, $relative);
+
+        if (!is_dir($dir)) {
+            return;
+        }
+
+        if (is_file($dir . '/index.twig') || is_file($dir . '/index.php')) {
+            $found[$relative] = true;
+        }
+
+        foreach (scandir($dir) ?: [] as $entry) {
+            if ($entry === '.' || $entry === '..' || !is_dir($dir . '/' . $entry)) {
+                continue;
+            }
+
+            if ($entry[0] === '_' || $entry[0] === '[' || $entry === '404') {
+                continue;
+            }
+
+            $childRelative = $relative === '' ? $entry : $relative . '/' . $entry;
+            self::walk($root, $childRelative, $found);
+        }
+    }
+
    private static function join(string $root, string $relative): string
    {
        $root = rtrim($root, '/');
@@ -83,6 +83,47 @@ final class Renderer
        }
    }

+    /**
+     * Renders $route the same way render() does (runs its sidecar if any,
+     * resolves the nearest layout) but returns the result instead of
+     * emitting HTTP output or writing the static cache — used by
+     * App\ContentIndexer to crawl the site for /sitemap.xml, /search, and
+     * blog tag browsing (see /admin/docs/content-index). Returns null if
+     * the sidecar returns a Response (a redirect/JSON/XML endpoint has
+     * nothing meaningful to index). There's no real request URI to derive
+     * request_path from here, so it's synthesized from the route itself.
+     */
+    public function renderForIndex(Route $route): ?IndexedPage
+    {
+        $sidecarRel = $this->withFile($route->dir, 'index.php');
+        $sidecar = Overlay::findFile($this->pagesDirs, $sidecarRel);
+        $hasSidecar = $sidecar !== null;
+
+        $result = $hasSidecar ? $this->runSidecar($sidecar, $route->params) : [];
+
+        if ($result instanceof Response) {
+            return null;
+        }
+
+        $data = array_merge((array) $result, ['params' => $route->params]);
+        $data['layout'] = $this->relativeLayoutPath($route->dir);
+        $data['request_path'] = $route->dir === '' ? '/' : '/' . $route->dir;
+
+        $templateName = $this->withFile($route->dir, 'index.twig');
+        $wrapper = $this->twig->load($templateName);
+
+        return new IndexedPage(
+            html: $wrapper->render($data),
+            title: $wrapper->renderBlock('title', $data),
+            description: $wrapper->renderBlock('description', $data),
+            keywords: $wrapper->renderBlock('keywords', $data),
+            tags: $wrapper->renderBlock('tags', $data),
+            robots: $wrapper->renderBlock('robots', $data),
+            changefreq: $wrapper->renderBlock('changefreq', $data),
+            priority: $wrapper->renderBlock('priority', $data),
+        );
+    }
+
    public function renderNotFound(string $requestUri): void
    {
        http_response_code(404);