Refresh & CTR Opportunity Scoring — A Content Action Playbook

Abstract

This study asks which content pages on FlyRank's client portfolio are worth reviewing first for a refresh, using March 2026 daily search performance data. A transparent CTR-position rule baseline and a Random Forest classifier were both evaluated on a client-grouped, out-of-sample split to avoid overstating performance. The Random Forest showed directional improvement over the rule baseline (precision@50 of 0.56 versus 0.16), though a naive random split inflated this figure to 0.84, revealing meaningful client-specific memorization. The result is a ranked, reason-coded action queue with explicit human-review requirements and no-automation rules, intended as decision-support — not a guarantee of outcome.

01Introduction

Which pages deserve a reviewer's time first?

Content teams managing large portfolios of pages need a repeatable way to decide which pages deserve attention first — refreshing outdated content, fixing poor click-through rates, or simply monitoring stable performers. Manual review of every page doesn't scale.

This project builds and honestly validates a ranked action system for FlyRank's Refresh / Content Opportunity Scoring lane: given a page's observed search performance in a given month, which pages are worth a human reviewer's time first, and why?

02Data

One month, one release, clearly scoped.

Data comes from the FlyRank internship warehouse release, table fact_content_daily_performance (daily grain, aggregated to monthly per content item), for month=2026-03 — a mid-panel month, held apart from the sealed final month (2026-06) used for no part of this analysis.

Features used: trailing GSC impressions and clicks, average position, trailing GA4 sessions/pageviews/engaged sessions, days observed, days with GSC data available, and three engineered ratios — click-through rate, engagement rate, and GSC coverage.

Deliberately excluded

impression_change — the exact quantity the label is derived from (confirmed via a deliberate leak test, see Results)
last_optimized_date, optimization_eligible_date — prior decision flags, not world signal
AI-referral columns — present in 69% of rows but non-zero in only 0.03%, too sparse to carry signal
Client and content identifiers — used only for joining and grouping, never as model inputs

No client names, domains, URLs, or raw exports appear anywhere in this report or its supporting notebooks.

03Methodology

A transparent rule first, testing before assuming.

Before writing any rule, three candidate signals were tested against observed March performance:

Signal	Test	Verdict
Staleness (days since content update)	Stale vs. fresh content, avg. clicks	OPPOSITE
CTR vs. search position	Top-10 vs. below-10 position, avg. CTR	CONFIRMED
Search volume vs. captured clicks	High vs. low keyword volume, avg. clicks	OPPOSITE

Stale content outperformed fresh content — the reverse of the initial assumption, likely because unupdated pages are older, established performers that simply don't need touching. High search-volume keywords also underperformed low-volume ones, likely due to competition absorbing more of the larger opportunity. Only the CTR-position relationship held in the expected direction, so the baseline rule is built on it: flag a page if it holds a top-10 average position but its CTR falls below the average CTR of other top-10 pages — a likely title or snippet problem, not a ranking problem.

Model: a Random Forest classifier (200 trees, max depth 6) predicting is_declining — whether a content item's impressions fell in the second half of March versus the first half, a within-month, past-observable proxy for decline. Eleven features, all trailing and past-only.

Split: client-grouped (70/30), not random. A random split lets the same client's content appear in both train and test, letting the model partly memorize client-specific patterns rather than learn a generalizable signal.

04Results

The honest split costs you something — and that's the point.

Model versus baseline, same client-grouped split, same metric:

Method	precision@50	AUC
Base rate (random ranking)	0.18	0.500
Rule-based baseline	0.16	—
Logistic Regression	0.66	0.775
Random Forest	0.58	0.803

The baseline falls slightly below the random base rate — expected, since it was built to catch CTR-fix opportunities, not to predict decline; it was never designed for this exact question. Both models substantially outrank it on this task.

The split itself matters more than it might seem:

0.84precision@50, naive random split

0.56precision@50, honest grouped split

The gap shows the naive split overstated performance — part of the "before" score reflected the model recognizing clients it had already trained on, not genuinely predicting decline on unseen ones. 0.56 is the number this report trusts.

Leakage check Adding the label-derived impression_change column pushed AUC to a perfect 1.0 — confirming both that the test harness catches leakage, and why that column stays permanently excluded. A single-feature scan found no individual feature above 0.95 AUC on its own (highest: GSC data-coverage days, at 0.814 — plausible, not suspicious).

Errors: of 49,823 grouped-split test predictions, 8,987 (18%) were wrong. Errors clustered around content with thin GSC coverage (under half the month's days with usable data), where predictions are inherently less reliable — though at least one wrong case had full data coverage, showing aggregate monthly stats alone don't catch every decline pattern.

05Limitations & honest framing

What this can and cannot tell you.

This is observed, decision-support evidence from one month of data (March 2026) on one 104-client cohort — not a causal claim that acting on a flagged page improves its performance, and not validated on other months or client sets.
The two dominant model features (GSC data availability and coverage) may partly reflect a data-availability artifact rather than genuine performance decline, since the label itself is built from the same underlying metric.
58% of scored content in this queue has thin GSC coverage (under 50% of the month) — predictions on these rows should be treated as lower confidence, not ignored.
The panel is unbalanced: clients differ by as much as 17 months in how much history they have, so cross-client comparisons aren't always apples-to-apples.

06Ranked recommendations

The action playbook.

Content is flagged using percentile-based thresholds (90th percentile of predicted decline probability, not a fixed cutoff) into four reason codes:

high_risk_ranked_page — top-10% decline risk and good search position — highest priority
high_risk_low_visibility — top-10% decline risk, lower search position
thin_data_coverage — under 50% of the month has usable data; verify before acting
lower_priority — monitor only

All 4,988 rows flagged as high-risk carry reliable data coverage, so a reviewer can trust the top of the queue directly. A reviewer should open the live page and current search results before acting on any flag — a numeric score never replaces reading the content.

Never automate Auto-publishing or auto-editing content from an action label; auto-deprioritizing or deleting lower_priority content; client-facing traffic-recovery guarantees; or applying this exact model to a new month or client cohort without re-validation.

07Reproducibility

Re-run it yourself.

All notebooks live in work/notebooks/ of the linked repository. Random seed fixed at 42 throughout. To reproduce: open a notebook in Colab, add a Hugging Face read token as a Colab secret named HF_TOKEN (with dataset access granted), then Runtime → Run all.