Tag: metadata collection 47 documents
Search documents matching “metadata collection” →Free-text subjects the curators added. Unlike the other axes these are not necessarily from the documents' own vocabulary, so they are the loosest thing to browse by (and the most forgiving).
Graphs
By Banner
By Classification
Refine (combine with another filter, via search)
Sort: Title
Documents
-
Access Breakdown [pdf]
This GCHQ document provides a technical breakdown of surveillance access points showing 11 different interception sites (with codenames like DONNINGTON, LATUS, SCEPTRE) accessing various undersea cables including Flag, Apollo North/South, BT, Cable & Wireless (C&W), SMW-3, Solas, and Verizon, totaling 541 10-gigabit connections representing 5,410 gigabits of surveillance capacity across the network.
Programs: DONNINGTON · LATUS · SCEPTRE · PRESCOTT · GIBUS · KIRKISTOWN · MONDELLO · LINNELL · CATEGORY · KNAPWEED · ELAPSE · BROKER“Vodafone – der lange Arm des britischen Geheimdienstes?” - WDR ↗ (2014-11-20) · “Spy cable revealed: how telecoms firm worked with GCHQ” - Channel 4 ↗ (2014-11-20) · “Der lange Arm des Geheimdienstes?” - NDR ↗ (2014-11-21) · “Snowden-Leaks: How Vodafone-Subsidiary Cable & Wireless Aided GCHQ’s Spying Efforts” - Süddeutsche Zeitung ↗ (2014-11-25) · Restle, Georg · White, Geoff · Strozyk, Jan · Moltke, Henrik · Poitras, Laura · Obermaier, Frederik -
Access: Vision 2013 [pdf]
This July 2010 GCHQ strategic document outlines the agency's vision for 2013 to create "the world's biggest SIGINT engine" with access to 800 10-gigabit connections (up from existing capacity), expanding overseas operations to 60% while reducing UK-based to 40%, and developing new capabilities including computer network exploitation (CNE), exfiltration, beaconry, geolocation, and collection of "Internet Pocket Litter" like cookies and browsing data. The strategy emphasizes "scale and pace," "balance," and "agility" while claiming to operate within legal frameworks of "necessity, proportionality and authorisation," and highlights partnerships with second and third parties to access "intractable and hard targets."
Marking: SECRET -
Advanced HTTP Activity Analysis [pdf]
This NSA training document from 2009 teaches intelligence analysts how to analyze HTTP internet traffic and exploit it using the XKEYSCORE surveillance system to monitor and identify internet users globally.
“XKEYSCORE: NSA’s Google for the World’s Private Communications” - The Intercept ↗ (2015-07-01) · “No Place to Hide” - Metropolitan Books ↗ (2014-05-13) · Marquis-Boire, Morgan · Greenwald, Glenn · Lee, Micah -
Analyzing Mobile/Cellular DNI in XKEYSCORE, May 2009 [pdf]
The XKEYSCORE document explains how the agency analyzes mobile/cellular device internet usage by collecting and processing both converged DNI/DNR data within cellular networks and traditional internet traffic, using various tools and indicators to identify mobile users even when their device type isn't immediately evident. The document demonstrates how the system can track mobile internet activity across applications like Yahoo Mail and identifies devices through various technical "hints" including MSISDN codes, browser fingerprints, and user agent strings.
-
Blazing Saddles [pdf]
This document describes BLAZING SADDLES, a GCHQ program designed to massively scale the storage and analysis of bulk internet surveillance data through multiple specialized databases that track website visits, online identities, communications patterns, and geolocation queries across the global internet population.
Programs: BLAZINGSADDLES · BLACKHOLE · SOCIALANTHROPOID · AUTOASSOC · HRMAP · INFINITEMONKEYS · KARMAPOLICE · MARBLEDGECKO · MUTANTBROTH · MEMORYHOLE · SAMUELPEPYS · SOCIALANIMAL · HAUSTORIUM · GOLDENAXE · NGE -
BOUNDLESSINFORMANT - Describing Mission Capabilities from Metadata Records [pdf]
This presentation on the Boundless Informant program (BOUNDLESSINFORMANT), which uses big data tools to analyze and visualize global SIGINT metadata collection in near real time. It explains how the system replaced older manual methods, enabling the NSA to track collection volumes, coverage, and trends for specific countries or sites through cloud-based infrastructure.
People: Patel, Vimesh“NSA-files repeatedly show collection of data «against countries» - not «from»” - Dagbladet ↗ (2013-11-22) · “Boundless Informant: the NSA's secret tool to track global surveillance data” - The Guardian ↗ (2013-06-08) · “Claves y giros del espionaje masivo en España” - El Mundo ↗ (2013-11-04) · Greenwald, Glenn · MacAskill, Ewen · Aranda, Germán -
Cable Master List [pdf]
This GCHQ document is a comprehensive master list of undersea fiber optic cables tracked for surveillance purposes, including both cables with UK landing points and international cables without UK connections, detailing technical specifications, ownership, capacity data, and code names for telecommunications partner companies that provided the intelligence agency with surveillance access.
Programs: PINNAGE · PFENNINGALPHA · NIGELLA · REMEDY · HIASCO · GERONTIC · SCEPTRE · PRESCOTT · KIRKISTOWN · LINNELL · CATEGORY · DACRON · KNAPWEED · ELAPSE · VITREOUS · LITTLE · STREETCAR · GIBBUS · VISAGE · LATUS · NUMDAH“Vodafone – der lange Arm des britischen Geheimdienstes?” - WDR ↗ (2014-11-20) · “Spy cable revealed: how telecoms firm worked with GCHQ” - Channel 4 ↗ (2014-11-20) · “Der lange Arm des Geheimdienstes?” - NDR ↗ (2014-11-21) · “Snowden-Leaks: How Vodafone-Subsidiary Cable & Wireless Aided GCHQ’s Spying Efforts” - Süddeutsche Zeitung ↗ (2014-11-25) · Restle, Georg · White, Geoff · Strozyk, Jan · Moltke, Henrik · Poitras, Laura · Obermaier, Frederik -
Cables we do not currently have good access [pdf]
This GCHQ document, titled "Cables we do not currently have good access," lists numerous undersea fiber-optic cables along with their active data capacity ("Lit Capacity"), indicating the substantial portion of global telecommunications infrastructure where the agency lacks adequate surveillance access, in contrast to other internal reports detailing cables it successfully monitors.
“Vodafone – der lange Arm des britischen Geheimdienstes?” - WDR ↗ (2014-11-20) · “Spy cable revealed: how telecoms firm worked with GCHQ” - Channel 4 ↗ (2014-11-20) · “Der lange Arm des Geheimdienstes?” - NDR ↗ (2014-11-21) · “Snowden-Leaks: How Vodafone-Subsidiary Cable & Wireless Aided GCHQ’s Spying Efforts” - Süddeutsche Zeitung ↗ (2014-11-25) · Restle, Georg · White, Geoff · Strozyk, Jan · Moltke, Henrik · Poitras, Laura · Obermaier, Frederik -
CSEC Surveillance of Travellers Through Airport Wi-Fi Tracking [pdf]
The leaked 2012 top secret CSEC presentation details a trial program that used airport Wi-Fi metadata to identify and track travellers’ devices across Canada and internationally, developing advanced “IP profiling” techniques to map movements through hotels, airports, and public hotspots. The program, conducted with NSA support, aimed to build powerful analytic tools for surveillance and real-time alerts within the Five Eyes intelligence network.
Marking: TOP SECRET -
DSD’s Risky Metadata Sharing: Australian Citizens’ Privacy Jeopardized in 5-Eyes Surveillance [pdf]
The document reveals that in 2008, Australia’s Defence Signals Directorate (DSD) was willing to share bulk, unselected, unminimised metadata about ordinary citizens with its 5-Eyes partners, including sensitive medical, legal, and religious information.
Marking: SECRET -
Excerpt of May 2008 NSA Afghanistan Dataflow Analysis [pdf]
The document outlines how the NSA’s Real Time Regional Gateway (RT-RG) in Afghanistan progressively expanded from late 2007 through 2008 to ingest massive amounts of GSM, VHF, HF, and telephony metadata from dozens of locations across Afghanistan, supplemented by third-party “dirtbox” collection from allied nations such as the Netherlands, Denmark, Norway, Germany, and Spain. It details the specific collection sites, data types, and timelines (SPIN 9, 10, 11, etc.), illustrating how partner contributions and multiple signals sources fed into the RT-RG system to enable real-time intelligence.
Programs: FOURCORNERS · FORTRESS · WARRIOR · ECHIDNA · CASSIEOPIA · KARUS · SOUNDER · SNICK · TEMPOWOLFE · CALYPSO · TATTERAN · AIRBORNE · LEAVEBEHIND · UDEAUS · NELEUS · CROSSHAIR · SSO · DRT · RIVET JOINT · VOICESAIL“Mission Creep: How the NSA’s Game-Changing Targeting System Built for Iraq and Afghanistan Ended Up on the Mexico Border” - The Intercept ↗ (2019-05-29) · “Norges hemmelige krig i Afghanistan” - NRK ↗ (2019-05-29) · Moltke, Henrik · Skille, Øyvind Bye · Døvik, Olav · Hansen, Ståle Ellingsgard · Fjørtoft, Martin -
FORNSAT technologies [pdf]
This document outlines a Foreign Satellite Communication Exploitation Model targeting satellite systems such as Intelsat, Iridium, Hughes, ND SatCom, and Inmarsat, using both fixed and mobile satellite services. It describes specific collection and geolocation systems—SEADIVER, ZODIACARRAY, and CANYONDUST—that capture and process voice, fax, data, and terminal location information across global coverage areas.
Programs: SHAREDVISION · DARKQUEST · EKB · GLOBALVIEW · TRYST · ZODIACARRAY · CANYONDUST · IRIDIUM · SEADIVER -
GCHQ Hopscotch [pdf]
An internal note argues for consolidating “edges”—summarized contact pairs—into a single cloud to unify relationship graphs, streamline analytics, and simplify management of HOPSCOTCH. It’s a mundane-sounding data-migration pitch with sharp operational teeth: centralize the social graph so surveillance queries run faster and wider.
Programs: HOPSCOTCH“Operation Socialist: The Inside Story of How British Spies Hacked Belgium’s Largest Telco” - The Intercept ↗ (2014-12-13) · “Lees hier hoe de Britse geheime dienst GCHQ Belgacom aanviel” - NRC ↗ (2014-12-13) · “Britse geheime dienst bespioneerde jarenlang Belgacom-klanten” - De Standaard ↗ (2014-12-13) · Gallagher, Ryan · Modderkolk, Huib · Vanhecke, Nikolas · Eeckhaut, Mark -
HIMR Data Mining Research Problem Book [pdf]
This document is a research roadmap created by GCHQ for the Heilbronn Institute for Mathematical Research (HIMR). Its purpose is to guide long-term mathematical and data-mining research that could improve signals intelligence (SIGINT) capabilities. It outlines key research problems, datasets, and collaboration plans between intelligence agencies and academic researchers. The main goal is to develop new algorithms and mathematical tools to analyze extremely large communications datasets produced by modern surveillance systems.
Programs: CLOUD · DISTILLERY · DISCOVER · PRESTON · LECKWITH · TERRAIN · BROADOAK · NGE · FIVEALIVE · CHARTBREAKER · FIRSTCONTACT · GEOFUSION · RADONSHARPEN-B · PRIMETIME · HIDDENOTTER · BAKERSDOZEN · CLASP · SALTYOTTER · BIRCH · SOLIDINK · FLUIDINK · MAMBA · GRINNINGROACH · PIRATECAREBEAR · AUTOASSOC · CROUCHINGSQUIRREL · HRMAP · KACHINA · SQUEAL · QUESTA · METEORSHOWER · SUNSTORM · GOLDMINE · HAGERAWEL · WOODY · BUZZ · SILVERLIBRARY · SILVERLINING · VALHALLA · MOUNTMCKINLEY · SEPANG · BLACKHOLE · SALAMANCA · HAGELAWEL · LUCKYSTRIKE · SPIKYROCK · MYOFIBRIL · INJUNCTION · PSYCHICSALMON · RAGINGBULLFROG · ROBOTICFISH · TIMIDTOAD · AURA · CARBONCOPY · CASK · CHORDAL · CRAN · INSTINCT · MADFORGE · SWAMP · VOLSUNGA · WHITERAVEN · MVR · PPF · JTRIG · MAPREDUCE · MAILORDERPeople: Tukey, John -
HTTP Activity vs User Activity [pdf]
This NSA training document from June 2009 explains the distinction between HTTP Activity (general web browsing data) and User Activity (communications metadata from specific protocols like webmail and chat) within the XKEYSCORE surveillance system, demonstrating through Iranian examples how analysts combine both search types to develop comprehensive intelligence on targets.
Programs: XKEYSCORE · MARINA · FOXTRAIL · DECODEORDAIN · YACHTSHOP · BRUTUS · WINDCHASER · NKB · UIS · STARPROC -
Kaspersky User-Agent Strings [pdf]
This September 2008 NSA technical report reveals that the agency discovered Kaspersky antivirus software's user-agent strings contain encoded serial numbers and can function as unique machine identifiers, allowing tracking of individual computers running the Russian security software across intercepted network traffic collected through the YACHTSHOP surveillance program.
-
Leaky phone apps [pdf]
These GCHQ slides describe the "WARRIOR PRIDE" mobile intrusion capability being ported to iPhone and Android, with plugins for power management, remote microphone, high‑precision geolocation, kernel stealth, and self‑protection. It claims the ability to retrieve almost any content from infected phones (SMS/MMS, email, call records, photos, calendar, etc.). Part of GCHQ's project "Mobile Applications Project".
Programs: Mobile Applications Project · WARRIORPRIDE · DREAMYSMURF · NOSEYSMURF · TRACKERSMURF · PORUS · PARANOIDSMURF -
Mobile Theme Briefing (May 28, 2010) [pdf]
This 2010 GCHQ briefing document outlines the “Mobile Applications Project,” which aims to expand intelligence capabilities to exploit the growing shift from fixed to mobile internet usage. It details objectives for targeting mobile apps and roaming network traffic (GRX), enabling converged analysis of voice, text, command-and-control, and geolocation data across multiple mobile platforms such as iPhone, Symbian, Android, and Windows Mobile. See also: EFF_-_20140128-guard-leaky_phone_apps_0 (includes other slides from the same slideset)
Programs: Mobile Applications Project -
Next Generation Events [pdf]
A 2009 TOP SECRET briefing outlines the “Next Generation Events” program to scale internet metadata and content analysis across Five Eyes, addressing explosive growth in data volume and the limits of legacy ingest systems. It presents a phased roadmap combining cloud infrastructure, large‑scale event processing, and enhanced profiling to enable bulk analytics and contact chaining.
Programs: NGE · HAUSTORIUM · BLAZINGSADDLES · HARBOURPILOT · MUTANTBROTH · AUTOASSOC · KARMAPOLICE · SOCIALANIMAL · INFINITEMONKEYS · HRMAP · MEMORYHOLE · MARBLEDGECKO · SIMMER · ROCKRIDGE · SAMUELPEPYS · CAFFEINEHIT · LAUGHINGHYENA · SALAMANCA · CLOUD · MONTEVISTA · LOOKINGGLASS · XKEYSCORE · FUMECUPBOARD · IIB · MoaG -
NSA BOUNDLESSINFORMANT Chart on Afghanistan [pdf]
The Afghanistan BOUNDLESSINFORMANT chart covering Dec 10, 2012 to Jan 8, 2013. About 35.8 million cellphone metadata records was collected and shared with the NSA, primarily through DRTBOX systems. These records were gathered in Afghanistan as part of the Afghanistan SIGINT Coalition.