SEO & indexing

This page documents what Typeroll emits at build time, what the runtime layer does on every request, and how it all interacts with custom domains. If you’re trying to understand why your site looks the way it does in Search Console — start here.

The two-URL problem

Every Typeroll site is reachable from at least two URLs:

The fallback subdomain — <slug>.sites.typeroll.com (or your self-hosted equivalent). Auto-provisioned when the site is created. Never goes away.
The custom domain — www.example.com. Optional. Activated through the domain lifecycle.

Both serve the same Cloudflare Pages build. Without deliberate effort, Google would treat them as duplicate content, split page rank between them, and possibly surface the wrong URL in SERPs.

Typeroll’s defense is layered: a strong canonical URL signal at the HTML layer, and a stricter X-Robots-Tag header at the Edge for crawlers that don’t honour canonicals.

Canonical URL

Every page in a Typeroll build carries a <link rel="canonical"> tag in <head>. The href points at the URL the site should be indexed under:

Site state	Canonical points at
No custom domain configured	`https://<slug>.sites.typeroll.com/<path>`
Custom domain status `pending` / `verified` / `failed`	`https://<slug>.sites.typeroll.com/<path>`
Custom domain status `live`	`https://<custom-domain>/<path>`
Site with legacy `domain` field (no lifecycle metadata)	`https://<custom-domain>/<path>` (legacy compat — treats as live)

The canonical is baked at build time from Astro.site, which the deploy runner sets based on the site doc. The URL also feeds into:

sitemap.xml — every page’s <loc> is the canonical URL
sitemap-images.xml — image URLs and their parent page URLs
<meta property="og:url"> — OpenGraph URL for social shares
JSON-LD mainEntityOfPage.@id — for Article schema and related types

Fallback-subdomain de-indexing

The canonical URL alone is enough for Google and the major Western search engines in clean duplicate-content scenarios — they consolidate ranking on the canonical and drop the duplicate from SERPs. But:

Smaller / regional crawlers don’t always respect canonicals.
Bing and Yandex have historically been less consistent.
AI training crawlers (GPTBot, ClaudeBot, etc.) often skip canonical consolidation entirely.

To close that gap, when a site has a live custom domain, the deploy runner emits a path-routed Cloudflare Pages Function at functions/robots.txt.js:

const LIVE_HOST = "www.example.com";
const NORMAL_ROBOTS = "User-agent: *\nAllow: /\n…";
const DISALLOW_ALL = "User-agent: *\nDisallow: /\n";

export function onRequestGet({ request }) {
  const host = request.headers.get("host");
  const body = host === LIVE_HOST ? NORMAL_ROBOTS : DISALLOW_ALL;
  return new Response(body, {
    status: 200,
    headers: {
      "content-type": "text/plain; charset=utf-8",
      "cache-control": "public, max-age=300",
    },
  });
}

The function fires only on /robots.txt requests — not on every page view. Crawlers fetch robots.txt first before crawling the site; if it tells them to stay out, they don’t come back. The function returns Disallow: / for any host that isn’t the activated custom domain, and the normal robots.txt body (with any per-site overrides) for the live host.

What it deliberately doesn’t do:

No 301 redirect. The fallback URL stays usable for the customer — for previewing during DNS changes, debugging cert provisioning, comparing two builds side-by-side.
No per-page X-Robots-Tag injection. That would require middleware fired on every request, which costs invocations per pageview. Robots.txt Disallow is the standard signal crawlers already check; the extra header is overkill when the canonical URL is already in place for the few crawlers that don’t read robots.txt.

Invocation overhead

Path-routed Functions only fire on the matching path. /robots.txt is hit a few times per crawler pass — typically dozens of requests per day per site, total. Even a high-traffic site never approaches the Cloudflare Pages free-tier limit (100,000 invocations / day) from robots.txt alone.

The function is only emitted when the site has a live custom domain. Pre-activation, the fallback IS the live URL and stays fully indexable — no function generated.

Zero-invocation alternative — Cloudflare Transform Rule

For operators who want zero per-request cost (no Pages Function at all), the same noindex policy can be enforced at the Edge via a Cloudflare Transform Rule applied at the fallback zone level (not per Pages project). Configured once in the Cloudflare Dashboard, applies to every customer’s fallback URL.

Hosted typeroll.com: a single Transform Rule on the typeroll.com zone (sites.typeroll.com is a subdomain within that zone, not its own zone). Dashboard path:

Cloudflare → typeroll.com → Rules → Transform Rules → "Modify Response Header"
  Rule name:    Noindex Typeroll fallback subdomains
  When:         (http.host wildcard "*.sites.typeroll.com")
  Then:         Set static header  X-Robots-Tag = noindex, nofollow
  Deploy:       enabled

Or via API (one shot):

CF_TOKEN=$(gcloud secrets versions access latest --secret=cf-api-token --project=typeroll-production)
ZONE_ID=9ec781fe…   # typeroll.com — full id in Secret Manager / CF Dashboard
curl -X PUT "https://api.cloudflare.com/client/v4/zones/${ZONE_ID}/rulesets/phases/http_response_headers_transform/entrypoint" \
  -H "Authorization: Bearer ${CF_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "rules": [
      {
        "action": "rewrite",
        "expression": "(http.host wildcard \"*.sites.typeroll.com\")",
        "description": "Noindex Typeroll fallback subdomains",
        "enabled": true,
        "action_parameters": {
          "headers": {
            "X-Robots-Tag": { "operation": "set", "value": "noindex, nofollow" }
          }
        }
      }
    ]
  }'

(GET the entrypoint first if other Transform Rules already exist — the PUT replaces the entire ruleset, so you’d need to merge.)

Self-hosted: configure the same on whatever zone hosts your fallback subdomain. The Pages Function in your build still serves as a fallback if the Transform Rule isn’t set, so you get both belt and braces if you want.

The Transform Rule beats the Function on three axes:

Zero invocations.
Zero deploy-time generation logic.
Applies globally, not per-project — one rule covers all current and future customer fallbacks.

But the Transform Rule requires ops setup outside the Typeroll deploy pipeline. The Function works without it. Pick based on which fits your operating model.

robots.txt

Generated by the renderer at /robots.txt. Default policy:

User-agent: *
Allow: /

Sitemap: https://<canonical-host>/sitemap.xml

Per-site overrides:

Site setting robots_txt — when non-empty, replaces the default body entirely. Use this for site-specific allow/disallow rules.
Version flag robots_blocked — when true, the renderer emits a hard Disallow: / regardless of the override. Branches default to robots_blocked: true so previews don’t get crawled; main is always indexable.

The User-agent: * block keeps it permissive by default — Typeroll doesn’t enumerate specific bots. To block a specific crawler (e.g. AI training), add it explicitly via the robots_txt override:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Sitemap

Two XML sitemaps emitted per build:

/sitemap.xml — every published page + every collection item that has an item_template_html and route_template. URLs use the canonical host.
/sitemap-images.xml — images referenced from published content, grouped by parent page.

Both filter out items with noindex: true. Both honour the canonical URL rules above.

The sitemap is referenced from robots.txt and is picked up automatically by Google Search Console / Bing Webmaster Tools when you verify ownership. You don’t need to submit it manually after every deploy — search engines re-crawl it on their own schedule.

Structured data (JSON-LD)

Pages can carry arbitrary JSON-LD via the Page.json_ld field. Common patterns:

Article — for blog posts, set kind: "article" on the page. The renderer emits Article schema with headline, datePublished, dateModified, author, image. Combines with the canonical URL via mainEntityOfPage.@id.
Organization — site-level. Configured under Settings → Organization (name, logo, sameAs profile URLs). Emitted on every page so search engines pick up the entity consistently.
PostalAddress — site-level. Configured under Settings → Contact → Address (either a single string or a structured streetAddress / addressLocality / postalCode / addressCountry object). The structured form emits a full Schema.org PostalAddress in the Organization block.
Custom JSON-LD — paste raw JSON into the page’s json_ld field. The renderer escapes </script and <!-- defensively before injecting.

Per-page JSON-LD is added to the page’s <head>; site-level JSON-LD is added by SEOHead.astro on every page automatically.

Meta tags emitted on every page

<title>Page title — Site Name</title>
<meta name="description" content="…" />
<meta name="robots" content="index,follow" />
<!-- or noindex,nofollow if blocked -->
<link rel="canonical" href="https://www.example.com/path" />
<meta name="generator" content="Typeroll" />

<!-- OpenGraph (Facebook, LinkedIn, Slack previews) -->
<meta property="og:title" content="Page title" />
<meta property="og:description" content="…" />
<meta property="og:type" content="website" />
<!-- or "article" for article pages -->
<meta property="og:url" content="https://www.example.com/path" />
<meta property="og:image" content="https://www.example.com/og.png" />
<meta property="og:locale" content="en_US" />
<!-- derived from site language -->

<!-- Twitter card -->
<meta name="twitter:card" content="summary_large_image" />
<meta name="twitter:site" content="@yoursitehandle" />
<meta name="twitter:title" content="Page title" />
<meta name="twitter:description" content="…" />
<meta name="twitter:image" content="https://www.example.com/og.png" />

<!-- Optional, when site-level fields are set -->
<meta name="author" content="…" />
<link rel="alternate" hreflang="…" href="…" />
<!-- if multiple language versions exist -->

The full set, plus per-page overrides, lives in packages/site-template/src/components/SEOHead.astro.

Custom domains and SEO timing

The order of operations when bringing a new domain live:

Add the domain in Settings → Custom domain. Status: pending. Canonical: still the fallback URL. Site behaviour unchanged.
Configure DNS at the registrar — CNAME to the target the dialog shows.
Cloudflare verifies + issues the SSL cert. Click “Check status” to refresh. Status: verified.
Activate the domain. Status: live. With auto-deploy on, a new build is enqueued that bakes the custom domain into canonical / sitemap / OG. The fallback URL starts emitting X-Robots-Tag: noindex.
Wait for Google to recrawl. Search Console will surface the new canonical within 1–2 weeks for established sites. To accelerate, request indexing of the homepage manually in Search Console and submit the new sitemap if needed.

Trying to switch domains without re-deploying leaves the canonical pointing at the previous URL. That’s not catastrophic — Google still consolidates via the new domain’s response headers and content — but it’s slow and confusing, so the activate flow defaults to auto-deploying. See Custom Domain guide for the user-facing walkthrough.