# Data dictionary (v2, de-identified release)

This release holds class labels only: no header values, no cookies, no CSP text, no nonces, no timestamps, no raw header map.

## Glossary (standard across all artifacts)

1. Sampled rows (n=7,040): directory rows drawn at random, covering 7,022 unique initial
   registrable domains.
2. Registrable domain: the domain name with the last two labels, or three when the
   second-level part is a common public suffix such as co.uk. A heuristic, not a full
   Public Suffix List lookup.
3. Final registrable domain: the registrable domain of the final URL after redirects.
   "Final host" means the full hostname, used only for the HSTS host-scope table.
4. Usable responses (n=5,642 rows, 5,614 unique final registrable domains): rows with a
   usable HTTPS response. Tables label this base "all-usable".
5. HTTP-200 responses (n=4,701 rows): usable rows whose final status was 200.
6. Principal base, also called "dedup-200" (n=4,688): unique final registrable domains
   among the HTTP-200 responses. One row kept per final registrable domain - the LAST row
   in scan-2 check-completion order (the released file's row order; sorting by row_id,
   i.e. scan order, changes the dedup). Every headline figure uses this base. Code and
   column names such as n_dedup200 refer to this base.

## data/scan2-2026-09-24-deidentified.jsonl - one object per sampled directory row

Coverage: ALL 7,040 sampled rows are present (directory rows covering 7,022 unique initial
registrable domains) - 5,642 with a usable HTTPS response (ok=true) and 1,398 without
(ok=false; the error class is recorded and no per-check class fields are set). No sampled
row was dropped or excluded after sampling. The analysis bases are filters over this file:
all-usable = usable responses = ok rows (5,642 rows, 5,614 unique final registrable
domains), HTTP-200 = ok rows with status 200 (4,701), principal base (dedup-200) = LAST row
per dedup_group within the HTTP-200 rows (4,688).

Row order is scan-2 check-completion order. The principal base keeps the LAST row per
dedup_group in file order.

- row_id: sample row number in scan order, r0001-r7040 (see README caveat on re-identification).
- sector: first Curlie path segment after /Business_and_Economy ("General" at the category root).
- state: US state/territory segment of the Curlie category path (underscores for spaces).
- ok: whether a usable HTTPS response was received.
- error: failure class when ok=false (e.g. "Domain does not resolve").
- status / hops: final HTTP status and redirect count when ok=true.
- dedup_group: keyed pseudonym of the final registrable domain (HMAC-SHA256 truncated to 12
  hex chars; key not released). Rows sharing a dedup_group reached the same registrable domain.
- final_host_kind: apex | www-subdomain | other-subdomain - shape of the final host relative
  to its registrable domain (for the HSTS host-scope table; carries no identity).

### Per-check classes (final rubric: rubric3.py + csp3.py v3)

- hsts: missing | short | base | base+preload | strict | strict+preload.
  base = max-age >= 31,536,000; strict = base + includeSubDomains; +preload = preload token.
  Joined duplicate fields: first max-age wins (browser behavior). max-age=0 -> short.
- csp.site_class: no-csp | no-script-directive | allowlist | nonce-hash-strict |
  strict-dynamic+anchor | invalid.
- csp booleans: script_restricting, unsafe_inline_effective (element channel),
  unsafe_inline_attr_effective (attribute channel, script-src-attr chain),
  unsafe_eval_effective, wildcard_effective (bare * or bare scheme in the effective script
  directive), wildcard_subdomain_effective, unsafe_inline_ignored, effective (passes the
  header-only script-CSP rule = script-restricting with no effective element-inline,
  attribute-inline, eval or full-wildcard weakness; the composite-index CSP point).
- csp_report_only: a Content-Security-Policy-Report-Only header was present.
- clickjacking: effective | ineffective | missing. An enforced frame-ancestors directive
  decides (effective = no bare * token); X-Frame-Options only when no enforced
  frame-ancestors exists. Joined identical XFO duplicates normalize to one token;
  conflicting repeats and ALLOW-FROM earn no credit.
- xcto: pass (exact nosniff, case-insensitive) | other | missing.
- rp: secure | weak | invalid | missing. Full vocabulary - secure: no-referrer, same-origin,
  origin, strict-origin, origin-when-cross-origin, strict-origin-when-cross-origin; weak:
  unsafe-url, no-referrer-when-downgrade. List processing: the last recognized token wins;
  all-unknown = invalid (browser default applies; no credit). Missing is not a failure.
- pp: restrictive (>=1 parsed directive whose allowlist is not a bare *) | wildcard-only |
  unparsable | missing. Unknown directive names with a non-wildcard allowlist count.
- coop: effective-strong (same-origin) | effective-popups (same-origin-allow-popups) |
  effective-noopener-popups | restrict-properties | noop (unsafe-none) | unknown-token |
  malformed | missing. Parameters after ';' are stripped; joined identical duplicates
  normalize; conflicting repeats and the space variant 'same-origin allow-popups' are
  malformed (browsers ignore them). Index point = the four effective-*/restrict classes.
- version_leak: a digit appears in any of server, x-powered-by, x-aspnet-version,
  x-aspnetmvc-version. version_leak_headers: which of those headers tripped (names only).
  version_headers_present: which of the four were present at all (names only).
- platform: hosting label or null. Rules, first match wins (mutually exclusive by
  construction; a site matching two signals takes the earlier one):
    1. GoDaddy builder - CSP contains 'godaddy.com' OR Server starts with 'DPS'
    2. Flywheel (managed WP) - Server starts with 'Flywheel'
    3. Pagely (managed WP) - Server starts with 'Pagely'
    4. Shopify - CSP starts with exactly "block-all-mixed-content; frame-ancestors 'none'; upgrade-insecure-requests"
    5. Wild Apricot - CSP contains 'wildapricot'
    6. Cloudflare-edge - Server is exactly 'cloudflare'
    7. Microsoft IIS - Server starts with 'Microsoft-IIS'
    8. Apache - Server starts with 'Apache'
    9. nginx - Server starts with 'nginx'
    10. OpenResty - Server starts with 'openresty'
    11. Amazon S3 - Server starts with 'AmazonS3'
  Builder/platform signatures rank above Server tokens because they name the configuration
  owner. Matches are case-sensitive as emitted.

### Derived in reanalysis4.py (not stored)

- idx8: count of passed checks - hsts strict-or-better, csp.effective, xcto pass,
  clickjacking effective, rp secure, pp restrictive, coop index class, no version leak.
- idx7: idx8 minus the version-hygiene item (seven protective controls).

## data/state-counts-2026-09-24.tsv

state, n_http200, none7_http200, mean8_http200, n_dedup200, none7_dedup200, mean8_dedup200.
The n_dedup200 / none7_dedup200 / mean8_dedup200 columns are computed on the principal base
(dedup-200). none7 = count of rows with idx7 == 0. The omnibus chi-square in final-tables.md section 6
uses the n>=150 rows of the HTTP-200 columns.