Broken Link Checker

Eventually this could be a service to SatoshiHost customers. It could periodically check their sites (if selected by them) and help them to fix the bad links. Along with the uptime service and other stuff, this is what will most distinguish us from other hosts. Working on a WP plugin for immediate use but will be a stand-alone site.

Summary: Custom External Links Checker Project (as of Feb 2026)

Goal: Create a lightweight, reliable way to find potentially broken or problematic external links on a website—without the false positives, bloat, paywalls, or aggressive blocking common in existing WordPress plugins (Broken Link Checker, etc.).

Core problems we’re trying to solve

  • Most plugins mark working links as broken due to Cloudflare challenges, rate limiting, anti-bot measures, temporary errors, or strict timeouts.
  • Free versions are limited (link count caps, forced sign-ups).
  • Paid/bloated ones add too much overhead.
  • We want something we control and can refine over time, with no third-party restrictions.

Current working prototype (WordPress plugin)

  • Name: Simple External Links Lister (Suspicious Only) v0.2
  • Location: Tools → Suspicious Links in WP admin
  • What it does:
    • Scans only published posts & pages
    • Extracts external http/https links using DOMDocument (reliable parsing)
    • Performs a quick HEAD request + limited GET fallback for Cloudflare hints
    • Filters out clean successes (HTTP 200, 301, 302, 303, 307, 308) → only shows suspicious / non-OK links
    • Processes posts in batches of 100 to avoid timeouts/memory issues on large sites
    • Displays paginated table (50 rows/page): Post Title (with edit link) | Anchor Text | URL | Status/Error
    • Highlights 4xx/5xx in red, connection errors in orange
    • No permanent storage — fresh scan every page load (refresh to re-check)
  • Status messages include basic Cloudflare detection when possible (e.g., “403 – likely Cloudflare protection”)

Key design choices so far

  • Prioritize accuracy over automation → manual verification in real browser is encouraged
  • Hide obviously healthy links to keep the list manageable
  • No “mark as fixed” that permanently hides items (because we want to re-notice if it actually breaks later)
  • Polite checking: timeouts, user-agent mimicking browser, small delays

Future evolution ideas (to be added when needed)

  1. Standalone PHP version (no WordPress dependency)
    • Single file or small app, upload anywhere
    • Input: any starting URL
    • Crawl limited depth/pages → find external links across non-WP sites too
    • Same suspicious-only filtering
    • Restrict access (e.g., basic auth, IP list, or simple login) so we can be more aggressive without worrying about public abuse
  2. Smarter prioritization instead of binary pass/fail
    • Allow “mark as working for now” → but re-check it next time
    • If the same failure signature appears again (same status code + similar body/headers) → move to bottom of list / mark low priority
    • Track history per link (e.g., last 3 results) to detect patterns:
      • Consistent Cloudflare challenge → treat as “protected, not broken”
      • Intermittent 503/timeout → deprioritize temporarily
      • Sudden 404 after previously OK → bump to top/high priority
  3. Other possible refinements
    • CSV export button
    • Ignore list for domains we know are protected (e.g., certain CDNs)
    • Configurable batch size, timeout, depth (for standalone)
    • Optional robots.txt respect + crawl delay
    • Anchor text and surrounding context in table for easier manual checking

This is intentionally kept minimal right now so it’s easy to test and iterate. The WP plugin is the immediate deliverable—once that’s validated on the main site, we can decide whether to port/refine it into the standalone multi-site tool.

(February 2026—conversation with Grok)
Latest updates (March 2026): Adding daily Telegram reporting to the existing Python checker and running it via cron on VPS. Main script development chat: Grok conversation about broken link checker + Telegram integration