Article
Profiled: From Radio to Porn, British Spies Track Web Users' Online Identities
Documents published with this article 29
-
Supporting Internet Operations - Special Source Access [pdf]
In this document GCHQ outlines its plans for expanding internet operations, focusing on increasing access to 10G bearers and enhancing its intercept and egress capabilities by 2011. The agency highlighted that “25 percent of all internet traffic” transits the U.K. through approximately 1,600 different 10-gigabit cables, and it noted that it could “survey the majority of the 1,600” to “select the most valuable to switch into our processing systems.”
-
Event (SIGINT) [pdf]
5 p.This document explains that within GCHQ and NSA SIGINT operations, an “Event” refers to metadata about observable communications or user actions rather than message content, categorized into types like Communications, Presence, Social, and Geo-reference Events. These events are collected, stored in various databases, and analyzed through convergence and correlation processes, with standardized formats defined by Interface Control Documents such as INTERSTELLAR DUST (IDUST) and Actor Action (AA).
Programs: INTERSTELLAR DUST · HAUSTORIUM · SOCIAL ANTHROPOID · ACTOR-ACTION · SAMREF · GEOFUSION · HARD ASSOC · SALAMANCA · ASSOCIATION · BANYAN · MAINWAY · FASCIA · CULTWEAVE II · VOICESAIL · TEEDALE · PILBEAM · MUTANTBROTH · AUTOASSOC · MARBLEDGECKO · KARMAPOLICE · HRMAP · MEMORYHOLE · INFINITEMONKEYS · SOCIALANIMAL · AUTOTDI · SAMUELPEPYS · Mobile Applications Project · NGE · PPF · SLR -
Access: Vision 2013 [pdf]
This July 2010 GCHQ strategic document outlines the agency's vision for 2013 to create "the world's biggest SIGINT engine" with access to 800 10-gigabit connections (up from existing capacity), expanding overseas operations to 60% while reducing UK-based to 40%, and developing new capabilities including computer network exploitation (CNE), exfiltration, beaconry, geolocation, and collection of "Internet Pocket Litter" like cookies and browsing data. The strategy emphasizes "scale and pace," "balance," and "agility" while claiming to operate within legal frameworks of "necessity, proportionality and authorisation," and highlights partnerships with second and third parties to access "intractable and hard targets."
Marking: SECRET -
BLACK HOLE ANALYTICS, ADD/SD briefing, September 2009 [pdf]
This September 2009 GCHQ briefing describes the BLACK HOLE analytics infrastructure and Next Generation Events (NGE) project, which planned to scale from ~8 10-gigabit connections to ~200 10Gs by March 2011, storing intercepted data in massive flat-file repositories at Benhall and Bude before loading it into "Question Focused Dataset" (QFD) databases that power surveillance tools like MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS, and others. The document outlines how "probes" extract Target Detection Identifiers (TDIs) from intercepted data feeds, store them in BLACK HOLE repositories, then enable analysts to profile targets' online activities, create social networks, investigate websites and forums, track web searches and Google Earth usage, and perform large-scale data mining across internet and telephony data.
Programs: BLACKHOLE · MUTANTBROTH · AUTOASSOC · INFINITEMONKEYS · KARMAPOLICE · SOCIALANIMAL · HRMAP · MEMORYHOLE · MARBLEDGECKO · HARDASSOC · SAMUELPEPYS · BLAZINGSADDLES · TELLURIAN · BUFFER · DISTILLERY · LOOKINGGLASS · NGEPeople: bin Laden, Osama -
Report on Architectural Risk 2012 - Summary [pdf]
This 2012 GCHQ report identifies that a historical under-investment in IT architecture has created significant risk, especially with new challenges like massive data growth, cyber defense, and increased partnering. The document recommends immediate, prioritized investment in big data strategies, integrated analytics frameworks, and security services to address the strain on the system.
Programs: BLACKHOLE -
Blazing Saddles [pdf]
This document describes BLAZING SADDLES, a GCHQ program designed to massively scale the storage and analysis of bulk internet surveillance data through multiple specialized databases that track website visits, online identities, communications patterns, and geolocation queries across the global internet population.
Programs: BLAZINGSADDLES · BLACKHOLE · SOCIALANTHROPOID · AUTOASSOC · HRMAP · INFINITEMONKEYS · KARMAPOLICE · MARBLEDGECKO · MUTANTBROTH · MEMORYHOLE · SAMUELPEPYS · SOCIALANIMAL · HAUSTORIUM · GOLDENAXE · NGE -
Broadcast/Internet Radio Exploitation and Analysis, 6th November 2009, GCHQ [pdf]
This GCHQ report analyzes 6.68 million internet radio listening events collected over three months in 2009, detailing global broadcasting patterns, user behaviors, and specifically examining Islamic radio stations and Pakistan-UK connections for potential radicalization intelligence, while recommending integration with bulk surveillance databases like KARMA POLICE for enhanced user profiling.
-
ICTR Cloud Efforts [pdf]
A GCHQ presentation details how the agency developed cloud-based big data analytics to achieve "population scale" surveillance—processing trillions of internet events to build geographic and behavioral profiles on every online identifier worldwide, enabling discovery of unknown targets through pattern analysis rather than traditional keyword searches.
Programs: EVERYASSOC · PROBABILITYCLOUD · SILVERLINING · CULTWEAVE · KARMAPOLICE · EVERYPOLICE · AWKWARDTURTLE · GORDIANKNOT · SANDIA · BLOODHOUND · FIVEALIVE · NGE · MAPREDUCE -
Content-Metadata Matrix [pdf]
GCHQ's internal taxonomy sorts intercepted data into "content" or "metadata" to determine what can be searched without individual warrants—email subject lines, full URLs, and buddy lists fall on opposite sides of the line, enabling analysts to mine massive troves of IP addresses, phone numbers, location data, and browsing history with just a few keystrokes and a brief justification.
-
Cyber Defence Operations Legal Policy [pdf]
GCHQ's internal wiki laying out the legal plumbing for its Cyber Defence Operations (formerly NDIST)—detailing warrant types (RIPA, ISA, CNE), data retention schedules, targeting rules for UK/Five Eyes infrastructure, XKEYSCORE query protocols, and the bureaucratic hoops analysts must jump through to search attacker IPs, fingerprint UK networks, and share malware intelligence without tripping over location/nationality safeguards or commercial sensitivities.
Programs: JTRIG · SPAY · GORDIANKNOT · CHORDAL · DISCOVER · EVENTS · CASPORT · XKEYSCORE · CONTENT CLOUD · SQUEAL · MIRAGE · MUTANTBROTH · ANXIOUS · HALTERHITCH · BROADOAK · DONKEYKONG · ZOOL · PALANTIR · OVERLINK · RT · GATEKEEPER · IRONHAND · HARUSPEX · B3M · LOOKINGGLASS · GOLDENEYE · BLAZINGSADDLES · MOONRAKER · TRITON -
Data Stored in BLACK HOLE [pdf]
This document describes BLACK HOLE, a TOP SECRET data repository that stores approximately 200GB per day of internet activity "events" such as webmail, chat, and browsing. It details the system's various datamarts, including MUTANT BROTH and KARMA POLICE, which collectively hold terabytes of data for intelligence analysis.
Programs: BLACKHOLE · B13B · MUTANTBROTH · KARMAPOLICE · MEMORYHOLE · AUTOASSOC · SOCIALANIMAL · HRMAP · INFINITEMONKEYS -
Demystifying NGE Rock Ridge [pdf]
This presentation outlines the ROCK RIDGE project, which delivers a suite of "Question Focused Databases" (QFDs) like MUTANT BROTH and KARMA POLICE to analyze massive internet event data. These specialized databases are designed to answer specific intelligence questions and are developed using the BLACK HOLE data repository, a key component of the ROUGH DIAMOND program.
Programs: ROCKRIDGE · ROUGHDIAMONG · BLACKHOLE · HAUSTORIUM · HRMAP · MUTANTBROTH · KARMAPOLICE · GOOBZS · INFINITEMONKEYS · AUTOASSOC · SOCIALANIMAL · MARBLEDGECKO · MEMORYHOLE · SAMUELPEPYS · SALAMANCA · MONOPOLY · RADIUS · HARBOURPILOT · LOOKINGGLASS · FIRESTORM · SUPERDRAKE · BLAZINGSADDLES · BLUESHIFT · SUPPORTINGINO · TERRAINS · GEOFUSION · NGEPeople: Lobban, Iain -
GCHQ Analytic Cloud Challenges [pdf]
The 2012 GCHQ Analytic Cloud document outlines a sophisticated data processing infrastructure handling approximately 50 billion daily events across multiple sites, utilizing QFD databases and HADOOP clusters for various analytic capabilities targeting intelligence gathering and analysis. It describes architectural components, challenges in scaling and enabling analysts to cope with data complexity, and proposes future technology options including consolidation approaches and new cloud architectures to enhance capabilities.
Programs: EVOLVEDMUTANTBROTH · MUTANTBROTH · HARDASSOC · HRMAP · KARMAPOLICE · SOCIALANTHROPOID · AUTOASSOC · GEOFUSION · MVR · TERRAIN · SALAMANCA · BLACKHOLE · SAMUELPEPYS · BROKER · VAIL · GUIDINGLIGHT · FIRSTCONTACT · CLOUDYCOBRA · RUMOURMILL · EVERYASSOC · EVERYPOLICE · EVERYCREATURE · EVERYEAD · EVERYCIPHER · GREYFOX · SILVERFOX · GOLDENAXE · TRIBALCARNEM · PUBLICANEMONE · EPICFAIL · STERLINGMOTH · FOGHORN · LOOKINGGLASS · FIREENGINE · NGGEO · ALPHACENTAURI · CLOUDBASE · ACCUMULO · ASEM · SEM · HAKIM · STREAMING · HBASE · LAUREL · SKINNYCLOUD · HARDY · GCHQCORE -
HRA auditing [pdf]
This GCHQ internal wiki page provides detailed procedural guidance for conducting Human Rights Act (HRA) compliance audits of analyst activities using surveillance databases like BROAD OAK and Events, establishing quarterly and six-monthly random sampling schedules, defining compliance metrics (Red <75%, Amber 75-94%, Green ≥95%), and specifying the responsibilities of analysts, auditors, and business unit heads to demonstrate that targeting, tasking, and examination of intercept content or metadata meet legal requirements for necessity and proportionality.
Marking: UK CONFIDENTIAL -
HTTP Activity in XKEYSCORE [pdf]
This training presentation from March 2009 provides NSA and Five Eyes analysts with detailed instructions on exploiting HTTP web browsing activity collected by XKEYSCORE, explaining how to differentiate client-to-server requests (GET/POST) from server-to-client responses, extract metadata like search terms, URLs, cookies, language, and browser types, and use this data to discover new surveillance targets, track individuals' online behavior, and conduct "pattern of life" analysis.
-
Laws on interception - Oversight - Implications for JCE (JCE UK Legalities Context) [pdf]
This GCHQ training presentation on UK surveillance laws, oversight mechanisms, and implications for Joint Collaborative Environment operations explicitly acknowledges that the UK has "a light oversight regime compared with US," detailing how RIPA, ISA, commissioners, and the Investigatory Powers Tribunal function to provide legal authorization while noting the Tribunal has "always found in our favour."
Programs: JTRIG -
Legalities - GCHQ Databases eg Pilbeam, Salamanca, UDAQ etc [pdf]
This October 2007 GCHQ document outlines the legal authorization requirements for querying surveillance databases, revealing that GCHQ analysts can search their own databases (like PILBEAM, SALAMANCA, UDAQ) for metadata on anyone regardless of nationality without additional authorization, while NSA databases (like MARINA, MAINWAY, DISHFIRE) prohibit searches for Americans and other Five Eyes citizens but permit searches for UK persons.
-
Events Analysis [pdf]
This GCHQ document explains authorization policies for querying metadata databases (SALAMANCA, HAUSTORIUM, THUGGEE, IMMINGLE), stating that metadata is "less intrusive than communications content" and that authorization is "not needed for individuals in the UK," though searches are logged and audited with necessity and proportionality considerations still applying.
-
Next Generation Events [pdf]
A 2009 TOP SECRET briefing outlines the “Next Generation Events” program to scale internet metadata and content analysis across Five Eyes, addressing explosive growth in data volume and the limits of legacy ingest systems. It presents a phased roadmap combining cloud infrastructure, large‑scale event processing, and enhanced profiling to enable bulk analytics and contact chaining.
Programs: NGE · HAUSTORIUM · BLAZINGSADDLES · HARBOURPILOT · MUTANTBROTH · AUTOASSOC · KARMAPOLICE · SOCIALANIMAL · INFINITEMONKEYS · HRMAP · MEMORYHOLE · MARBLEDGECKO · SIMMER · ROCKRIDGE · SAMUELPEPYS · CAFFEINEHIT · LAUGHINGHYENA · SALAMANCA · CLOUD · MONTEVISTA · LOOKINGGLASS · XKEYSCORE · FUMECUPBOARD · IIB · MoaG -
NGE BLACK HOLE ConOp [pdf]
This document outlines the "BLACK HOLE" Concept of Operations (ConOp) for the "Next Generation Events" (NGE) project within "BLAZING SADDLES," detailing a flat file data store designed to house various intelligence feeds. Its primary purpose is to serve as a source for Query Focused Datasets (QFDs) and support research, bulk analysis, and target discovery, with two operational instances at Benhall and Bude handling massively increased data volumes.
Programs: NGE · BLAZINGSADDLES · BLACKHOLE · MUTANTBROTH · KARMAPOLICE · SOCIALANIMAL · TELLURIAN · BUFFER · MAILORDER · MARBLEDGECKO · SALAMANCA · SMOKINGSADDLES · CIRCUIT · BLACKFIND · BLACKCAT · DATAFLOWCAB · CHORDAL · DEBIT CARD · B13B -
OP HIGHLAND FLING - Event Log [pdf]
A TOP SECRET operational event log from GCHQ's GCWiki documenting a January 2011 operation targeting Gemalto employees in France, Poland and elsewhere, using MUTANT BROTH cookie analysis, Facebook credentials, LinkedIn profiles and Computer Network Exploitation to identify and implant targets as part of the operation to steal SIM card encryption keys.
-
Operational Engineering November 2010 [pdf]
This internal GCHQ briefing from late 2010 documents the Events Product Centre's suite of bulk internet and communications surveillance tools — including IMMINGLE, SALAMANCA, SOCIAL ANTHROPOID, MUTANT BROTH, KARMA POLICE, SAMUEL PEPYS and GUIDING LIGHT — covering scaling targets, new data sources, and operational capabilities across telephony, webmail, instant messaging, social media, GPRS and VoIP collection.
Programs: EVENTS · IMMINGLE · SALAMANCA · GUIDINGLIGHT · B3M · HARDASSOC · MAINWAY · THUGGEE · FASCIA · HAUSTORIUM · FIRESTORM · TERRAIN · SHAREOWN · ESCHAR · BRIO · INFINITEMONKEY · KARMAPOLICE · MARBLEDGECKO · MEMORYHOLE · MUTANTBROTH · SAMUELPEPYS · SOCIALANIMAL · SOCIALANTHROPOID · HEARTBEAT · SANTHROPOID · LEAKYGATEWAYS · GTPTUNNEL · LUSTRE · GLASSBACK · COLLATERAL · HRMAP · SWORDPLAY · BROADOAK · REFORMER · ARTEMIS · MERLOT · CONTRAOCTAVE · CRYINGFOWL · CULTWEAVE · SAMDYCE · GCQUERY · BLAZZINGSADDLES · EREPO · JTRIG · SOUNDER -
Pull Steering Group Minutes [pdf]
A top secret STRAP1 GCHQ PullThrough Steering Group meeting record from February 2008 documenting the approval process for five surveillance and intelligence prototypes including KARMA POLICE — the mass internet browsing profiling system — alongside ORB FINDER for CNE operational relay box identification, SALTY OTTER for cross-media timing analysis, MOOSE MILK for telephone kiosk surveillance, and LIGHTWOOD for email address extraction used by JTRIG.
Programs: MYOFIBRIL · SALTYOTTER · PRIMETIME · KARMAPOLICE · ORBFINDER · MOOSEMILK · LIGHTWOOD · JTRIG · MERAPEAK · MONTEVISTA · KED -
QFD BLACKHOLE Technology Behind INOC [pdf]
A UK SECRET STRAP2 COMINT ORCON GCHQ Applied Research presentation from March 2009 documenting the BLACK HOLE bulk data repository's scale and structure, and the suite of analytical tools — MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS and MARBLED GECKO — built on top of it to answer specific intelligence questions about online presence, browsing activity, forum usage and geolocation searches.
Programs: BLACKHOLE · MESME · NETEZZA · CLOUDS · MUTANTBROTH · KARMAPOLICE · INFINITEMONKEYS · MARBLEDGECKO · GOOB · AUTOASSOC · PSOUP · SOCIALANIMAL · SAMUELPEPYS · MEMORYHOLE · HRMAP · PSOUPALERT · LAUGHINGHYENA -
Sensitive Targeting Authorisation [pdf]
A GCHQ internal form template for Sensitive Targeting Authorisation (STA), used to obtain approval for surveillance targeting of individuals or organisations with sensitive nationalities or in sensitive locations, covering both real-time tasking and historical database datamining, with separate authorisation tracks for overseas targets and UK-based targets.
-
SOCIAL ANTHROPOID [pdf]
A SECRET STRAP1 GCHQ internal training and rollout presentation for SOCIAL ANTHROPOID — a new converged communications database replacing SALAMANCA, SOCIAL ANIMAL and HAUSTORIUM — demonstrating its query interface, data types, filtering capabilities and operational examples including telephony, webmail, instant messaging, BlackBerry, GPRS tunnels and leaky gateway events, with rollout planned across GCHQ analyst divisions in tranches during January-February 2011.
Programs: SOCIALANTHROPOID · SOCIALANIMAL · REFORMER · HAUSTORIUM · SALAMANCA · BROADOAK · DISCOVER · BLACKHOLE · HARDASSOCIATION · CHORDAL -
Crypt Discovery Joint Collaboration Activity 20 January 2011 [pdf]
A TOP SECRET NSA/GCHQ joint collaboration activity plan from January 2011 describing a research programme to understand and mitigate the threat that rising encryption adoption poses to target discovery and development — measuring encryption prevalence across SIGINT collection, assessing the impact on metadata-based targeting tradecraft, and developing new techniques for identifying targets behind encrypted traffic using a joint cloud environment fed by MUTANT BROTH, BEARDED PIGGY, KARMA POLICE, MEMORY HOLE, MARBLED GECKO, SOCIAL ANIMAL and ASDF metadata.
Marking: TOP SECRETPrograms: TICKETWINDOW · MUTANTBROTH · BEARDEDPIGGY · KARMAPOLICE · MEMORYHOLE · MARBLEDGECKO · SOCIALANIMAL · XKEYSCORE · ASDF“Prying Eyes: Inside the NSA's War on Internet Security” - Der Spiegel ↗ (2014-12-28) · “Profiled: From Radio to Porn, British Spies Track Web Users' Online Identities” - The Intercept ↗ (2015-09-25) · Appelbaum, Jacob · Gibson, Aaron · Grothoff, Christian · Müller-Maguhn, Andy · Poitras, Laura · Sontheimer, Michael · Stöcker, Christian · Gallagher, Ryan -
Target Detection Identifiers [pdf]
A GCHQ presentation from March 2009 introducing the concept of Target Detection Identifiers — the "fundamental atom of the Internet age" — describing how cookies and other unique persistent identifiers extracted from bulk internet traffic at 2,500 per second are used to build the surveillance databases, with a live AutoTDI discovery table showing cookies extracted from Facebook, Yahoo, Google, Reuters, YouPorn, Bebo and other sites.
Programs: MUTANTBROTH · AUTOASSOC · KARMAPOLICE · INFINITEMONKEYS · SOCIALANIMAL · MEMORYHOLE · MARBLEDGECKO · HRMAP · AUTOTDI -
TINT External July 2009 [pdf]
A GCHQ TINT external document from July 2009 describing the BLACKHOLE bulk metadata repository — designed to export unselected metadata in SLR format to enable higher-order analytics — noting anticipated very high volumes expected to exceed MAILORDER transport capacity and proposing a direct connection instead.