Legalities - GCHQ Databases eg Pilbeam, Salamanca, UDAQ etc
Open PDFDocument date: 2007-10-01
Pages: 1
Genre: other
Classification: -
Dissemination: -
This October 2007 GCHQ document outlines the legal authorization requirements for querying surveillance databases, revealing that GCHQ analysts can search their own databases (like PILBEAM, SALAMANCA, UDAQ) for metadata on anyone regardless of nationality without additional authorization, while NSA databases (like MARINA, MAINWAY, DISHFIRE) prohibit searches for Americans and other Five Eyes citizens but permit searches for UK persons.
PILBEAM, SALAMANCA, UDAQ, MARINA, MAINWAY, DISHFIRE
People mentioned:
-
Countries:
-
Tags:
surveillance databases, metadata repositories, content database, warrant, surveillance authorization, intelligence sharing, access control, intelligence loophole, mutual surveillance, surveillance oversight, nationality, location exemptions, targeting procedures
Identifier:
legalities
PDF SHA-256:
52a64b44d949f7e18bc4f79a4d250cd23235f673fc48b7a405ef43606d5c4583
My notes
As of Oct 2007
Legalities - GCHQ Databases eg Pilbeam, Salamanca, UDAQ etc
In the event that both the nationality and the location of the target is confirmed:
| metadata / content | UK persons | 2nd Party persons | Others |
|---|---|---|---|
| UK location | OK / warrant required | OK / warrant required | OK / warrant required |
| 2nd Party location | OK / STA | OK / STA | OK / STA |
| Other country | OK / STA | OK / STA | OK / OK |
Legalities - NSA Databases eg Marina, Mainway, Dishfire etc
In the event that both the nationality and the location of the target is confirmed:
| metadata / content | UK persons | 2nd Party persons | Others |
|---|---|---|---|
| UK location | OK / NO | NO / NO | OK / NO |
| 2nd Party location | NO / NO | NO / NO | NO / NO |
| Other country | OK / 1-off STA | NO / NO | OK / OK |
NB-Dishfire is dealt with as a CONTENT database
Marina is dealt with as a METADATA database
Published in the Same Report
(10 of 28 documents)
-
Supporting Internet Operations - Special Source Access [pdf]
In this document GCHQ outlines its plans for expanding internet operations, focusing on increasing access to 10G bearers and enhancing its intercept and egress capabilities by 2011. The agency highlighted that “25 percent of all internet traffic” transits the U.K. through approximately 1,600 different 10-gigabit cables, and it noted that it could “survey the majority of the 1,600” to “select the most valuable to switch into our processing systems.”
-
Event (SIGINT) [pdf]
5 p.This document explains that within GCHQ and NSA SIGINT operations, an “Event” refers to metadata about observable communications or user actions rather than message content, categorized into types like Communications, Presence, Social, and Geo-reference Events. These events are collected, stored in various databases, and analyzed through convergence and correlation processes, with standardized formats defined by Interface Control Documents such as INTERSTELLAR DUST (IDUST) and Actor Action (AA).
Programs: INTERSTELLAR DUST · HAUSTORIUM · SOCIAL ANTHROPOID · ACTOR-ACTION · SAMREF · GEOFUSION · HARD ASSOC · SALAMANCA · ASSOCIATION · BANYAN · MAINWAY · FASCIA · CULTWEAVE II · VOICESAIL · TEEDALE · PILBEAM · MUTANTBROTH · AUTOASSOC · MARBLEDGECKO · KARMAPOLICE · HRMAP · MEMORYHOLE · INFINITEMONKEYS · SOCIALANIMAL · AUTOTDI · SAMUELPEPYS · Mobile Applications Project · NGE · PPF · SLR -
Access: Vision 2013 [pdf]
This July 2010 GCHQ strategic document outlines the agency's vision for 2013 to create "the world's biggest SIGINT engine" with access to 800 10-gigabit connections (up from existing capacity), expanding overseas operations to 60% while reducing UK-based to 40%, and developing new capabilities including computer network exploitation (CNE), exfiltration, beaconry, geolocation, and collection of "Internet Pocket Litter" like cookies and browsing data. The strategy emphasizes "scale and pace," "balance," and "agility" while claiming to operate within legal frameworks of "necessity, proportionality and authorisation," and highlights partnerships with second and third parties to access "intractable and hard targets."
Marking: SECRET -
BLACK HOLE ANALYTICS, ADD/SD briefing, September 2009 [pdf]
This September 2009 GCHQ briefing describes the BLACK HOLE analytics infrastructure and Next Generation Events (NGE) project, which planned to scale from ~8 10-gigabit connections to ~200 10Gs by March 2011, storing intercepted data in massive flat-file repositories at Benhall and Bude before loading it into "Question Focused Dataset" (QFD) databases that power surveillance tools like MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS, and others. The document outlines how "probes" extract Target Detection Identifiers (TDIs) from intercepted data feeds, store them in BLACK HOLE repositories, then enable analysts to profile targets' online activities, create social networks, investigate websites and forums, track web searches and Google Earth usage, and perform large-scale data mining across internet and telephony data.
Programs: BLACKHOLE · MUTANTBROTH · AUTOASSOC · INFINITEMONKEYS · KARMAPOLICE · SOCIALANIMAL · HRMAP · MEMORYHOLE · MARBLEDGECKO · HARDASSOC · SAMUELPEPYS · BLAZINGSADDLES · TELLURIAN · BUFFER · DISTILLERY · LOOKINGGLASS · NGEPeople: bin Laden, Osama -
Report on Architectural Risk 2012 - Summary [pdf]
This 2012 GCHQ report identifies that a historical under-investment in IT architecture has created significant risk, especially with new challenges like massive data growth, cyber defense, and increased partnering. The document recommends immediate, prioritized investment in big data strategies, integrated analytics frameworks, and security services to address the strain on the system.
Programs: BLACKHOLE -
Blazing Saddles [pdf]
This document describes BLAZING SADDLES, a GCHQ program designed to massively scale the storage and analysis of bulk internet surveillance data through multiple specialized databases that track website visits, online identities, communications patterns, and geolocation queries across the global internet population.
Programs: BLAZINGSADDLES · BLACKHOLE · SOCIALANTHROPOID · AUTOASSOC · HRMAP · INFINITEMONKEYS · KARMAPOLICE · MARBLEDGECKO · MUTANTBROTH · MEMORYHOLE · SAMUELPEPYS · SOCIALANIMAL · HAUSTORIUM · GOLDENAXE · NGE -
Broadcast/Internet Radio Exploitation and Analysis, 6th November 2009, GCHQ [pdf]
This GCHQ report analyzes 6.68 million internet radio listening events collected over three months in 2009, detailing global broadcasting patterns, user behaviors, and specifically examining Islamic radio stations and Pakistan-UK connections for potential radicalization intelligence, while recommending integration with bulk surveillance databases like KARMA POLICE for enhanced user profiling.
-
ICTR Cloud Efforts [pdf]
A GCHQ presentation details how the agency developed cloud-based big data analytics to achieve "population scale" surveillance—processing trillions of internet events to build geographic and behavioral profiles on every online identifier worldwide, enabling discovery of unknown targets through pattern analysis rather than traditional keyword searches.
Programs: EVERYASSOC · PROBABILITYCLOUD · SILVERLINING · CULTWEAVE · KARMAPOLICE · EVERYPOLICE · AWKWARDTURTLE · GORDIANKNOT · SANDIA · BLOODHOUND · FIVEALIVE · NGE · MAPREDUCE -
Content-Metadata Matrix [pdf]
GCHQ's internal taxonomy sorts intercepted data into "content" or "metadata" to determine what can be searched without individual warrants—email subject lines, full URLs, and buddy lists fall on opposite sides of the line, enabling analysts to mine massive troves of IP addresses, phone numbers, location data, and browsing history with just a few keystrokes and a brief justification.
-
Cyber Defence Operations Legal Policy [pdf]
GCHQ's internal wiki laying out the legal plumbing for its Cyber Defence Operations (formerly NDIST)—detailing warrant types (RIPA, ISA, CNE), data retention schedules, targeting rules for UK/Five Eyes infrastructure, XKEYSCORE query protocols, and the bureaucratic hoops analysts must jump through to search attacker IPs, fingerprint UK networks, and share malware intelligence without tripping over location/nationality safeguards or commercial sensitivities.
Programs: JTRIG · SPAY · GORDIANKNOT · CHORDAL · DISCOVER · EVENTS · CASPORT · XKEYSCORE · CONTENT CLOUD · SQUEAL · MIRAGE · MUTANTBROTH · ANXIOUS · HALTERHITCH · BROADOAK · DONKEYKONG · ZOOL · PALANTIR · OVERLINK · RT · GATEKEEPER · IRONHAND · HARUSPEX · B3M · LOOKINGGLASS · GOLDENEYE · BLAZINGSADDLES · MOONRAKER · TRITON
Related Documents
-
Operational Legalities – GCHQ Powerpoint Presentation [pdf]
GCHQ’s “Operational Legalities” deck is an internal compliance playbook that translates the UK’s surveillance laws into day‑to‑day SIGINT procedure—drilling staff on authorisation, necessity, and proportionality, and walking through warrants, targeting justifications, database queries, dissemination, and oversight. Half slides and half speaker notes, it shows how analysts are trained to document, audit, and defend operations under ISA, RIPA, HRA, and WTA while navigating edge cases like location, nationality, second‑party constraints, and “indirect targeting.”
-
Menwith databases as of Aug 2008 [pdf]
This document outlines database access and SIGINT missions at Menwith Hill Station (MHS) as of August 2008, detailing the specific programs, language branches, and target development teams supported, including repositories like DISHFIRE, MAINWAY, and PINWALE. It also notes that U.S. personnel use the SIGINT Contact Center (SCC) process for access, with certain NOFORN restrictions, and lists additional analytic tools available to MHS analysts.
Marking: CONFIDENTIALPrograms: ASSOCIATION · DISHFIRE · HOMEBASE GEO · MAINWAY · NUCLEON · PINWALE · SPOTBEAM · XKEYSCORE · BANYAN · BROOMSTICK · CULTWEAVE II · MARINA · TRAFFICTHIEF · FASCIA · OCTSKYWARD · ROADBED · TUNNINGFORK · AGILITY · CADENCE · CONTRAOCTAVE · GAMUT · UTT · NKB · OCTAVE · PUZZLECUBE · TKB -
Event (SIGINT) [pdf]
5 p.This document explains that within GCHQ and NSA SIGINT operations, an “Event” refers to metadata about observable communications or user actions rather than message content, categorized into types like Communications, Presence, Social, and Geo-reference Events. These events are collected, stored in various databases, and analyzed through convergence and correlation processes, with standardized formats defined by Interface Control Documents such as INTERSTELLAR DUST (IDUST) and Actor Action (AA).
Programs: INTERSTELLAR DUST · HAUSTORIUM · SOCIAL ANTHROPOID · ACTOR-ACTION · SAMREF · GEOFUSION · HARD ASSOC · SALAMANCA · ASSOCIATION · BANYAN · MAINWAY · FASCIA · CULTWEAVE II · VOICESAIL · TEEDALE · PILBEAM · MUTANTBROTH · AUTOASSOC · MARBLEDGECKO · KARMAPOLICE · HRMAP · MEMORYHOLE · INFINITEMONKEYS · SOCIALANIMAL · AUTOTDI · SAMUELPEPYS · Mobile Applications Project · NGE · PPF · SLR -
SSO dictionary relevant entries [pdf]
This document provides a detailed list of relevant entries related to NSA's Special Source Operations (SSO), highlighting various NSA surveillance and data collection systems used to monitor global communications. It includes descriptions of programs such as BIGBIRD and DISHFIRE, which focus on cable and SMS interception, as well as tools like LOPERS and COURIERSKILL for processing telephony metadata and refining target selection. The entries emphasize the extensive and automated nature of the NSA's intelligence-gathering operations, particularly in relation to undersea cables and telecommunication networks, and underscore the secretive and expansive scope of SSO's global surveillance activities.
Programs: ARTIFICE · BIGBIRD · FAIRVIEW · SSO · POORWILL · PINECONE · BLACKBELT · CADENCE · BLACKNIGHT · SHIPMASTER · COURIERSKILL · THEORYMASTER · WEALTHYCLUSTER · CLIFFSIDE · DISHFIRE · MARINA · MAINWAY · NUCLEON · PINWALE · FASCIA · BANYAN · LAMPSHADE · ASSOCIATION · HOMEBASE · SEDB · FRIAR · KEYCARD · TURMOIL · TURBULENCE · UTT · IRISHBEAUTY · OCTAVE · KOZYKOVE · LITHIUM · BLARNEY · LOPERS · MAILORDER · LUMBERYARD · MERLIN · NODDY · SAGUARO · NUTHATCH · CONTRAOCTAVE · XKEYSCORE · PLANK · RODEOSTAR · SAGURA · SERENADE · STARGATE · SILVERCOLLAM · SEALION · SEAGULL · SLIVER · SORA-2 · TITANPOINTE · TUBE · VPCS · STONEGATE“NSA Spying Relies on AT&T’s ‘Extreme Willingness to Help’” - ProPublica ↗ (2015-08-15) · “AT&T Helped U.S. Spy on Internet on a Vast Scale” - The New York Times ↗ (2015-08-15) · “The Wiretap Rooms: The NSA’s Hidden Spy Hubs in Eight U.S. Cities” - The Intercept ↗ (2018-06-25) · Angwin, Julia · Larson, Jeff · Savage, Charlie · Moltke, Henrik · Poitras, Laura · Risen, James · Gallagher, Ryan -
FAIRVIEW dataflow charts April 2012 [pdf]
This document outlines data flow diagrams that show how various types of sensitive communication, including voice, fax, and modem traffic, are handled through systems operated by the NSA and its partners. These partners include corporations, telecommunications companies, ISPs, and other entities involved in processing and managing the data through secure and classified channels. The document covers processing flows for various data types, metadata, and communications systems under NSA's control, detailing how data is routed, stored, and analyzed across different platforms and sites.
Programs: FAIRVIEW · SAGUARO · TURNSTILE · MESA · TITANPOINT · RIMROCK · MAILORDER · PINECONE · BLARNEY · STONEGATE · TINSEL · CONVEYANCE · FISHWAY · NUCLEON · PINWALE · SCISSORS · COURIERSKILL · DRIFTWOOD · ASDF · FALLOUT · MARINA · LOPERS · WEALTHYCLUSTER · PUPR · CLEARSIGHT · MIDFIELD · CADENCE · BUGCATCHER · WAYLAND · DESOTO · GATEKEEP · HIGHDECIBEL · XKEYSCORE · YANKEE · TOPROCK · FASCIA · DISHFIRE · SEAGULL · BIGDIPPER · CORPORATE · LILDIPPER · HAVASU · WEALTHYCLUSTER2 · RAGTIME · TUMULT · TATTOO · DARKTHUNDER · ICYTWINS · TML-STATUS · WATERFRONT · VULCANDEATHGRIP · STEELFLAUTA · UNITY · KEYCARD · UTT · BTUBE · ISLANDTRANSPORT · SSO · MAINWAY · SSDM · TUBE · TITANPOINTE · TURMOIL · VPCS“NSA Spying Relies on AT&T’s ‘Extreme Willingness to Help’” - ProPublica ↗ (2015-08-15) · “AT&T Helped U.S. Spy on Internet on a Vast Scale” - The New York Times ↗ (2015-08-15) · “Titanpointe: The NSA’s Spy Hub in New York, Hidden in Plain Sight” - The Intercept ↗ (2016-11-16) · Angwin, Julia · Larson, Jeff · Savage, Charlie · Moltke, Henrik · Poitras, Laura · Risen, James · Gallagher, Ryan