Tag: bulk collection 47 documents
Search documents matching “bulk collection” →Free-text subjects the curators added. Unlike the other axes these are not necessarily from the documents' own vocabulary, so they are the loosest thing to browse by (and the most forgiving).
Graphs
By Banner
By Classification
Refine (combine with another filter, via search)
Sort: Title
Documents
-
Bulk collection of webmail address books and emerging protocols [pdf]
TOP SECRET NSA document details the large-scale collection of email address books and contact lists from major webmail and social media providers, emphasizing their metadata value despite low attribution rates to known targets. It also highlights ongoing efforts to identify and analyze emerging communication protocols, such as those used by Hotmail Messenger.
-
Access Breakdown [pdf]
This GCHQ document provides a technical breakdown of surveillance access points showing 11 different interception sites (with codenames like DONNINGTON, LATUS, SCEPTRE) accessing various undersea cables including Flag, Apollo North/South, BT, Cable & Wireless (C&W), SMW-3, Solas, and Verizon, totaling 541 10-gigabit connections representing 5,410 gigabits of surveillance capacity across the network.
Programs: DONNINGTON · LATUS · SCEPTRE · PRESCOTT · GIBUS · KIRKISTOWN · MONDELLO · LINNELL · CATEGORY · KNAPWEED · ELAPSE · BROKER“Vodafone – der lange Arm des britischen Geheimdienstes?” - WDR ↗ (2014-11-20) · “Spy cable revealed: how telecoms firm worked with GCHQ” - Channel 4 ↗ (2014-11-20) · “Der lange Arm des Geheimdienstes?” - NDR ↗ (2014-11-21) · “Snowden-Leaks: How Vodafone-Subsidiary Cable & Wireless Aided GCHQ’s Spying Efforts” - Süddeutsche Zeitung ↗ (2014-11-25) · Restle, Georg · White, Geoff · Strozyk, Jan · Moltke, Henrik · Poitras, Laura · Obermaier, Frederik -
Advanced HTTP Activity Analysis [pdf]
This NSA training document from 2009 teaches intelligence analysts how to analyze HTTP internet traffic and exploit it using the XKEYSCORE surveillance system to monitor and identify internet users globally.
“XKEYSCORE: NSA’s Google for the World’s Private Communications” - The Intercept ↗ (2015-07-01) · “No Place to Hide” - Metropolitan Books ↗ (2014-05-13) · Marquis-Boire, Morgan · Greenwald, Glenn · Lee, Micah -
Blazing Saddles [pdf]
This document describes BLAZING SADDLES, a GCHQ program designed to massively scale the storage and analysis of bulk internet surveillance data through multiple specialized databases that track website visits, online identities, communications patterns, and geolocation queries across the global internet population.
Programs: BLAZINGSADDLES · BLACKHOLE · SOCIALANTHROPOID · AUTOASSOC · HRMAP · INFINITEMONKEYS · KARMAPOLICE · MARBLEDGECKO · MUTANTBROTH · MEMORYHOLE · SAMUELPEPYS · SOCIALANIMAL · HAUSTORIUM · GOLDENAXE · NGE -
Broadcast/Internet Radio Exploitation and Analysis, 6th November 2009, GCHQ [pdf]
This GCHQ report analyzes 6.68 million internet radio listening events collected over three months in 2009, detailing global broadcasting patterns, user behaviors, and specifically examining Islamic radio stations and Pakistan-UK connections for potential radicalization intelligence, while recommending integration with bulk surveillance databases like KARMA POLICE for enhanced user profiling.
-
Cable Master List [pdf]
This GCHQ document is a comprehensive master list of undersea fiber optic cables tracked for surveillance purposes, including both cables with UK landing points and international cables without UK connections, detailing technical specifications, ownership, capacity data, and code names for telecommunications partner companies that provided the intelligence agency with surveillance access.
Programs: PINNAGE · PFENNINGALPHA · NIGELLA · REMEDY · HIASCO · GERONTIC · SCEPTRE · PRESCOTT · KIRKISTOWN · LINNELL · CATEGORY · DACRON · KNAPWEED · ELAPSE · VITREOUS · LITTLE · STREETCAR · GIBBUS · VISAGE · LATUS · NUMDAH“Vodafone – der lange Arm des britischen Geheimdienstes?” - WDR ↗ (2014-11-20) · “Spy cable revealed: how telecoms firm worked with GCHQ” - Channel 4 ↗ (2014-11-20) · “Der lange Arm des Geheimdienstes?” - NDR ↗ (2014-11-21) · “Snowden-Leaks: How Vodafone-Subsidiary Cable & Wireless Aided GCHQ’s Spying Efforts” - Süddeutsche Zeitung ↗ (2014-11-25) · Restle, Georg · White, Geoff · Strozyk, Jan · Moltke, Henrik · Poitras, Laura · Obermaier, Frederik -
Content Acquisition Optimization PowerPoint Slides [pdf]
An NSA presentation on content acquisition optimisation — describing how incidental bulk collection from Yahoo Webmessenger, email address books, buddy lists and webmail inboxes was generating massive unintended collection volumes.
-
Content-Metadata Matrix [pdf]
GCHQ's internal taxonomy sorts intercepted data into "content" or "metadata" to determine what can be searched without individual warrants—email subject lines, full URLs, and buddy lists fall on opposite sides of the line, enabling analysts to mine massive troves of IP addresses, phone numbers, location data, and browsing history with just a few keystrokes and a brief justification.
-
Cyber Defence Operations Legal Policy [pdf]
GCHQ's internal wiki laying out the legal plumbing for its Cyber Defence Operations (formerly NDIST)—detailing warrant types (RIPA, ISA, CNE), data retention schedules, targeting rules for UK/Five Eyes infrastructure, XKEYSCORE query protocols, and the bureaucratic hoops analysts must jump through to search attacker IPs, fingerprint UK networks, and share malware intelligence without tripping over location/nationality safeguards or commercial sensitivities.
Programs: JTRIG · SPAY · GORDIANKNOT · CHORDAL · DISCOVER · EVENTS · CASPORT · XKEYSCORE · CONTENT CLOUD · SQUEAL · MIRAGE · MUTANTBROTH · ANXIOUS · HALTERHITCH · BROADOAK · DONKEYKONG · ZOOL · PALANTIR · OVERLINK · RT · GATEKEEPER · IRONHAND · HARUSPEX · B3M · LOOKINGGLASS · GOLDENEYE · BLAZINGSADDLES · MOONRAKER · TRITON -
Dealing With a 'Tsunami' of Intercept SIDtoday [pdf]
As part of NSAs efforts to implement Human Language Technology, prototype processors located at collection sites will soon be able to query content at "line speed" based on the language, it is written in.
People: Kulman, Anita H.The Intercept ↗ (2019-05-29) · “Inside NSA, Officials Privately Criticize "Collect It All" Surveillance” - The Intercept ↗ (2015-05-28) · Maass, Peter -
Event (SIGINT) [pdf]
5 p.This document explains that within GCHQ and NSA SIGINT operations, an “Event” refers to metadata about observable communications or user actions rather than message content, categorized into types like Communications, Presence, Social, and Geo-reference Events. These events are collected, stored in various databases, and analyzed through convergence and correlation processes, with standardized formats defined by Interface Control Documents such as INTERSTELLAR DUST (IDUST) and Actor Action (AA).
Programs: INTERSTELLAR DUST · HAUSTORIUM · SOCIAL ANTHROPOID · ACTOR-ACTION · SAMREF · GEOFUSION · HARD ASSOC · SALAMANCA · ASSOCIATION · BANYAN · MAINWAY · FASCIA · CULTWEAVE II · VOICESAIL · TEEDALE · PILBEAM · MUTANTBROTH · AUTOASSOC · MARBLEDGECKO · KARMAPOLICE · HRMAP · MEMORYHOLE · INFINITEMONKEYS · SOCIALANIMAL · AUTOTDI · SAMUELPEPYS · Mobile Applications Project · NGE · PPF · SLR -
FASCIA: The NSA's huge trove of location records [pdf]
FASCIA is the National Security Agency's enormous database containing trillions of metadata records, many with device locations, that are collected from a variety of sources. This document shows the volume and types of device-location data collected by the agency.
-
Forwarding Selected MAILORDER Files Read Me [pdf]
An UNCLASSIFIED//FOR OFFICIAL USE ONLY XKEYSCORE administrator guide explaining how to configure an XKS cluster to forward selected intercepted sessions via MAILORDER, including category throttling limits, output directory configuration and the process setup required to switch all Digital Network Intelligence filtering and selection to the XKEYSCORE collection system.
-
GCHQ - OPTIC NERVE [pdf]
GCHQ’s OPTIC NERVE project built a web interface to display images from Yahoo webcam sessions captured via bulk, unselected interception, then used that stream for “proportionate” target discovery. The system sampled one frame every five minutes per feed, flagged the prevalence of nudity, and bolted on imperfect face-detection “censoring” alongside warnings about handling offensive material.
Programs: OPTICNERVE -
GCHQ access to international internet communications [pdf]
This document describes the scale of the agency's access to international internet communications, stating that GCHQ receives upwards of 50 billion events per day and that this volume is growing.
Programs: GUIDINGLIGHT -
GCHQ Analytic Cloud Challenges [pdf]
The 2012 GCHQ Analytic Cloud document outlines a sophisticated data processing infrastructure handling approximately 50 billion daily events across multiple sites, utilizing QFD databases and HADOOP clusters for various analytic capabilities targeting intelligence gathering and analysis. It describes architectural components, challenges in scaling and enabling analysts to cope with data complexity, and proposes future technology options including consolidation approaches and new cloud architectures to enhance capabilities.
Programs: EVOLVEDMUTANTBROTH · MUTANTBROTH · HARDASSOC · HRMAP · KARMAPOLICE · SOCIALANTHROPOID · AUTOASSOC · GEOFUSION · MVR · TERRAIN · SALAMANCA · BLACKHOLE · SAMUELPEPYS · BROKER · VAIL · GUIDINGLIGHT · FIRSTCONTACT · CLOUDYCOBRA · RUMOURMILL · EVERYASSOC · EVERYPOLICE · EVERYCREATURE · EVERYEAD · EVERYCIPHER · GREYFOX · SILVERFOX · GOLDENAXE · TRIBALCARNEM · PUBLICANEMONE · EPICFAIL · STERLINGMOTH · FOGHORN · LOOKINGGLASS · FIREENGINE · NGGEO · ALPHACENTAURI · CLOUDBASE · ACCUMULO · ASEM · SEM · HAKIM · STREAMING · HBASE · LAUREL · SKINNYCLOUD · HARDY · GCHQCORE -
GCSB Update - 22 April 2010 [pdf]
This GCSB briefing from April 2010 provides an internal update on the operational status and development pathway for the Waihopai satellite surveillance station (codenamed IRONSAND). It reports near-perfect system availability statistics for numerous NSA/GCSB systems like LEGALREPTILE and JUGGERNAUT, details collaboration on virtualization and collection initiatives, and notes consequences arising from external court decisions.
Programs: IRONSAND · CALIX · COMMONGROUND · DUALTIRE · FLUTE · FRESNELEFFECT · JACKHAMMER · JUGGERNAUT · KILOMISER · KRAUSS · LATENTHEAT · LEGALREPTILE · LOPERS · MOONSCAPE · SEMITONE · SSA · STONEGATE · SURFBOARD · VENUSAFFECT · VINEYARD · VINTAGE · WEALTHYCLUSTER · VQUEST · TORUS · SHAREDQUEST -
ICTR Cloud Efforts [pdf]
A GCHQ presentation details how the agency developed cloud-based big data analytics to achieve "population scale" surveillance—processing trillions of internet events to build geographic and behavioral profiles on every online identifier worldwide, enabling discovery of unknown targets through pattern analysis rather than traditional keyword searches.
Programs: EVERYASSOC · PROBABILITYCLOUD · SILVERLINING · CULTWEAVE · KARMAPOLICE · EVERYPOLICE · AWKWARDTURTLE · GORDIANKNOT · SANDIA · BLOODHOUND · FIVEALIVE · NGE · MAPREDUCE -
Is It the End of the SIGINT World as We Have Come to Know It? [pdf]
Two excerpts from a 2012 NSA presentation titled "Is It the End of the SIGINT World as We Have Come to Know It?" prepared by a member of the Large-Access Exploitation Group. The table excerpt lists MAINWAY among the systems at highest risk of being overwhelmed by the volume of data the agency was ingesting. The diagram excerpt shows MAINWAY feeding into the Graph-in-Memory (alongside WESTERNSTAR), illustrating how MAINWAY's continuous contact-chaining pushed its precomputed results into the in-memory graph database that analysts actually queried.
Programs: FALLOUT · METADNI · MARINA · MAINWAY · FASCIA · GMHALO · DPS · EKS · WESTERNSTAR · GRAPH-IN-MEMORY -
Is There a Sustainable Ops Tempo in S2? How Can Analysts Deal with the Flood of Collection? SIDtoday [pdf]
The NSA's Deputy Director for Analysis and Production discusses the unsustainable operational tempo caused by overwhelming data collection volumes, advocating for "Analytic Modernization" that would shift from treating SIGINT as "radioactive" NSA-only material to sharing raw intelligence with other agencies for collaborative exploitation.
Programs: A-SPACE