Transparent crawler discovery experiment

GitHub Machine Beacon

A transparent GitHub experiment that makes a repository unusually easy for crawlers, search indexes, AI agents, LLM readers, link preview bots, and code indexers to discover and parse.

19 indexed resources 7 reusable guides 73 mapped terms
Cloudflare live llms.txt manifest traffic sitemap GitHub
Diagram of repository, metadata, feeds, crawlers, and measurement loops.

Live edge traffic

...

Cloudflare requests through the live homepage and machine-readable endpoints.

Machine visits ...
Human visits ...
Unknown ...
GitHub views 0

Live split updated . Source: Cloudflare Worker. GitHub official snapshot: 0 views, 0 unique visitors, 0 clones. Raw edge data: cloudflare-traffic.json.

Traffic Classes

Machine traffic is split by intent. Sensitive-file and exploit-probe paths are counted as security scanners before normal AI or generic machine classes.

AI readers ...
Security scanners ...
Generic machine ...
Humans ...

Top Security Paths

  1. loading

Machine Categories

  1. loading

Raw class aggregate: traffic-classes.json.

AI / Machine Geography

Aggregated from Cloudflare request metadata. The Worker stores country, region, city, Cloudflare colo, and ASN organization counters; it does not store raw IP addresses or latitude/longitude.

Machine Countries

  1. loading

Machine Cities

  1. loading

Machine ASN Organizations

  1. loading

Cloudflare Colos

  1. loading

Raw geo aggregate: geo-traffic.json.

Resource Library

These pages are designed to be useful enough for humans to cite and structured enough for machines to parse.

Machine-Readable Repository Checklist A practical checklist for making a GitHub repository easier for crawlers, code indexes, LLM readers, and AI agents to parse. Crawler Surface Map A map of repository and website surfaces that expose the experiment to crawlers, code indexes, LLM readers, and link preview systems. AI Agent Entrypoints Recommended routes for LLM crawlers, AI coding assistants, retrieval systems, and autonomous browser agents reading this repository. Experiment Protocol A reproducible protocol for measuring whether machine-readable repository surfaces increase legitimate GitHub and web discovery. Standards and Sources Source-backed notes for the project surfaces: GitHub topics, GitHub Pages, Schema.org JSON-LD, llms.txt, sitemap.xml, robots.txt, and Atom feeds. Crawlability Audit A self-audit of the repository discovery surfaces and machine-readable files currently published by GitHub Machine Beacon. Results Log A public log for launch status, validation checks, and later traffic observations from the machine-readable GitHub discovery experiment.

Machine Surfaces

GitHub Machine Beacon Landing page for the machine-readable GitHub crawler discovery experiment. AI Reader Index Compact JSON entry point for AI crawlers, LLM readers, retrieval systems, and coding agents. AI Reader Context Plain-text single-page context bundle designed for AI readers and agent browsers. LLM Reader Guide Compact guide for LLM crawlers, retrieval systems, and AI coding agents. Full LLM Context Extended project context for retrieval augmented generation and AI readers. Crawler Manifest Canonical JSON manifest describing URLs, topics, principles, and measurement fields. Keyword Index Structured topic map for machine discovery and search indexing experiments. Resource Index Structured JSON index of all human-readable and machine-readable project resources. Traffic Snapshot Latest public traffic snapshot sourced from the GitHub Traffic API. Well-Known LLM Reader Guide A compatibility copy of llms.txt for readers that check well-known paths. Web App Manifest Basic web manifest that exposes the site name, description, and canonical start URL. Atom Feed Recrawl-friendly Atom feed for experiment updates.

Principles

  • Be transparent about the experiment.
  • Use honest metadata and relevant keywords only.
  • Publish stable machine-readable entry points.
  • Respect robots.txt and platform rules.
  • Measure discovery without generating fake traffic.

Measurement Fields

  • repository_views
  • unique_visitors
  • referrers
  • popular_content
  • clones
  • unique_cloners
  • edge_requests
  • machine_requests
  • human_requests
  • unknown_requests
  • machine_requests_by_country
  • machine_requests_by_city
  • machine_requests_by_asn_organization
  • edge_requests_by_cloudflare_colo
  • ai_reader_requests
  • security_scanner_requests
  • generic_machine_requests
  • security_scanner_paths
  • stars
  • forks
  • issues_or_discussions
  • external_citations

Keyword Map

Terms are grouped by intent so crawlers and human auditors can distinguish meaningful topic coverage from unrelated keyword stuffing.

machine-readable web discovery

Signals for crawlers and search indexes that prefer structured, canonical resources.

  • machine-readable repository
  • crawler-friendly GitHub project
  • GitHub Pages metadata
  • sitemap.xml
  • robots.txt
  • structured data
  • JSON-LD
  • Open Graph metadata
  • canonical URL
  • Atom feed
  • RSS feed
  • web crawler observability

AI and LLM discovery

Signals for retrieval systems, AI coding tools, and agent browsers.

  • llms.txt
  • AI reader index
  • AI reader context
  • LLM crawler
  • AI agent browser
  • AI search indexing
  • retrieval augmented generation
  • RAG source
  • agent-readable documentation
  • machine context file
  • AI code search
  • LLM metadata
  • crawler manifest
  • semantic README

GitHub repository discovery

Signals that help repository search, code search, and topic-based browsing.

  • GitHub search optimization
  • GitHub repository metadata
  • GitHub topics
  • README structure
  • code indexing
  • open source discoverability
  • repository traffic experiment
  • GitHub Insights traffic
  • GitHub Pages deployment
  • open research repository
  • software citation
  • CITATION.cff

measurement and ethics

Signals that the project is an observable, non-deceptive experiment.

  • crawler experiment
  • traffic measurement
  • ethical SEO
  • transparent metadata
  • no fake traffic
  • no cloaking
  • privacy-preserving analytics
  • search experiment
  • bot traffic research
  • machine traffic benchmark
  • crawlability audit
  • public web observability

Update Contract

This page is generated from data/beacon.json and data/content-pages.json. Declared project version: 0.6.1. Last declared update: .