Tag: bulk collection 47 documents
Search documents matching “bulk collection” →Free-text subjects the curators added. Unlike the other axes these are not necessarily from the documents' own vocabulary, so they are the loosest thing to browse by (and the most forgiving).
Graphs
By Banner
By Classification
Refine (combine with another filter, via search)
Sort: Title
Documents
-
MILKWHITE Enrichment Service (MES) Programme [pdf]
This March 2011 GCHQ document details the MILKWHITE program, which is enhancing domestic interception and data analysis capabilities. It outlines expanded funding for advanced analytics, bulk data storage, and service scaling to meet growing demand from multiple law enforcement and intelligence agencies, including SOCA, HMRC, MI5, and SIS, thereby broadening access to large volumes of internet-derived metadata for operational use.
-
Minimization Procedures Used By The National Security Agency In Connection With Acquisitions Of Foreign Intelligence, Information Pursuant To Section 702 Of The Foreign Intelligence Court Surveillance Act Of 1978 [pdf]
The NSA’s Section 702 minimization rules expose how the agency can collect, keep, and use Americans’ communications without warrants, despite claiming to target only foreigners. While the document sets out procedures for handling U.S. data, it includes broad loopholes—like keeping “inadvertently” collected domestic messages for years, sharing unfiltered data with the FBI, CIA, and even foreign governments, and giving NSA analysts wide discretion over who gets surveilled—all under secret FISA Court approval.
People: Holder, Eric H. Jr. -
N.S.A. Timeline of Surveillance Law Developments [pdf]
This chart, apparently from an internal N.S.A. presentation, lists milestones in surveillance rules over nearly four decades. The July 2002 “Raw Take” sharing order, whose existence has not previously been disclosed, is just one of three rulings by the Foreign Intelligence Surveillance Court to make the list. The other two were 2004 and 2006 rulings allowing bulk collection of data about emails and phone calls.
-
NGE BLACK HOLE ConOp [pdf]
This document outlines the "BLACK HOLE" Concept of Operations (ConOp) for the "Next Generation Events" (NGE) project within "BLAZING SADDLES," detailing a flat file data store designed to house various intelligence feeds. Its primary purpose is to serve as a source for Query Focused Datasets (QFDs) and support research, bulk analysis, and target discovery, with two operational instances at Benhall and Bude handling massively increased data volumes.
Programs: NGE · BLAZINGSADDLES · BLACKHOLE · MUTANTBROTH · KARMAPOLICE · SOCIALANIMAL · TELLURIAN · BUFFER · MAILORDER · MARBLEDGECKO · SALAMANCA · SMOKINGSADDLES · CIRCUIT · BLACKFIND · BLACKCAT · DATAFLOWCAB · CHORDAL · DEBIT CARD · B13B -
NSA inspector general report on email and internet data collection under Stellar Wind, 24 March 2009 [Working Draft] [pdf]
This 2009 NSA Inspector General draft report details the origins and legal framework of the President’s Surveillance Program (PSP), launched after 9/11 to bypass FISA limits and authorize bulk interception of internet and telephony content and metadata, including communications with one end inside the U.S. It reveals how NSA, with White House backing, expanded surveillance authority to cover “collection gaps” that existing law didn’t allow, effectively legalizing large-scale warrantless spying under a secret presidential directive.
People: Bush, George W. · Hayden, Michael · Tenet, George · Gonzales, Alberto R. · Addington, David S. · Graham, Robert · Deitz, Robert L. · Shelby, Richard · Pelosi, Nancy · Cheney, Dick · Libby, I. Lewis · Lamberth, Royce · Inouye, Daniel · Levin, Daniel · Stevens, Theodore · Kollar-Kotelly, Colleen · Rockefeller, Jay · Alexander, Keith B. · Potenza, Vito -
NSA June 2012 SSO Weekly Brief Excerpt [pdf]
This NSA Special Source Operations (SSO) briefing from June 2012 describes SPINNERET, a surveillance program that became the second-largest source of phone calling records and "tips" feeding into SHELLTRUMPET, a system that obscures the origin of collected metadata before forwarding it to NSA's Real-Time Regional Gateway (RT-RG) in Texas. The document indicates SHELLTRUMPET "obfuscates the source designators" and filters metadata to only include call events within NSA-Texas's area of responsibility.
“Mission Creep: How the NSA’s Game-Changing Targeting System Built for Iraq and Afghanistan Ended Up on the Mexico Border” - The Intercept ↗ (2019-05-29) · “Norges hemmelige krig i Afghanistan” - NRK ↗ (2019-05-29) · Moltke, Henrik · Skille, Øyvind Bye · Døvik, Olav · Hansen, Ståle Ellingsgard · Fjørtoft, Martin -
Operational Engineering November 2010 [pdf]
This internal GCHQ briefing from late 2010 documents the Events Product Centre's suite of bulk internet and communications surveillance tools — including IMMINGLE, SALAMANCA, SOCIAL ANTHROPOID, MUTANT BROTH, KARMA POLICE, SAMUEL PEPYS and GUIDING LIGHT — covering scaling targets, new data sources, and operational capabilities across telephony, webmail, instant messaging, social media, GPRS and VoIP collection.
Programs: EVENTS · IMMINGLE · SALAMANCA · GUIDINGLIGHT · B3M · HARDASSOC · MAINWAY · THUGGEE · FASCIA · HAUSTORIUM · FIRESTORM · TERRAIN · SHAREOWN · ESCHAR · BRIO · INFINITEMONKEY · KARMAPOLICE · MARBLEDGECKO · MEMORYHOLE · MUTANTBROTH · SAMUELPEPYS · SOCIALANIMAL · SOCIALANTHROPOID · HEARTBEAT · SANTHROPOID · LEAKYGATEWAYS · GTPTUNNEL · LUSTRE · GLASSBACK · COLLATERAL · HRMAP · SWORDPLAY · BROADOAK · REFORMER · ARTEMIS · MERLOT · CONTRAOCTAVE · CRYINGFOWL · CULTWEAVE · SAMDYCE · GCQUERY · BLAZZINGSADDLES · EREPO · JTRIG · SOUNDER -
Partner Cables [pdf]
A classified table mapping which GCHQ cable interception programs had access to specific undersea fiber optic cables worldwide, revealing the codenames and access types — either Direct Cable Order or Indefeasible Right of Use/Landing Capacity — of seven surveillance programs across dozens of international cable systems, with Cable & Wireless (GERONTIC) providing access to the largest number.
“Vodafone – der lange Arm des britischen Geheimdienstes?” - WDR ↗ (2014-11-20) · “Spy cable revealed: how telecoms firm worked with GCHQ” - Channel 4 ↗ (2014-11-20) · “Der lange Arm des Geheimdienstes?” - NDR ↗ (2014-11-21) · “Snowden-Leaks: How Vodafone-Subsidiary Cable & Wireless Aided GCHQ’s Spying Efforts” - Süddeutsche Zeitung ↗ (2014-11-25) · Restle, Georg · White, Geoff · Strozyk, Jan · Moltke, Henrik · Poitras, Laura · Obermaier, Frederik -
PRESTON Study [pdf]
This GCHQ document details the PRESTON surveillance program, revealing that over a six-month period, 97% of intercepted communications—including calls, texts, and internet data—were not viewed by analysts despite being fully warranted, highlighting systemic challenges in processing and exploiting the vast volume of collected intelligence. It underscores the reliance on traditional PRESTON sources for events like the London G20, while noting gaps in data feeds from other repositories, reflecting both operational scale and limitations in effective oversight.
-
PRISM/US-984XN Overview [pdf]
A 2013 NSA training presentation revealing that major U.S. tech companies — including Google, Facebook, Apple, Microsoft, and Skype — secretly gave the NSA direct access to their servers, allowing the agency to collect users' emails, chats, photos, and files in real time, with the FBI acting as intermediary.
Programs: PRISM · FAIRVIEW · STORMBREW · BLARNEY · OAKSTAR · UTT · PRINTAURA · PINWALE · NUCLEON · TRAFFICTHIEF · SCISSORS · MARINA · MAINWAY · FALLOUT · CONVEYANCE · REPRISMFISA · SSO“U.S., British intelligence mining data from nine U.S. Internet companies in broad secret program” - The Washington Post ↗ (2013-06-06) · “NSA Prism program taps in to user data of Apple, Google and others” - The Guardian ↗ (2013-06-07) · “L'ampleur de l'espionnage mondial par la NSA” - Le Monde ↗ (2013-10-21) · “NSA files decoded: Edward Snowden's surveillance revelations explained” - The Guardian ↗ (2013-11-01) · “No Place to Hide” - Metropolitan Books ↗ (2014-05-13) · “Dark Mirror: Edward Snowden and the American Surveillance State” - Penguin Press ↗ (2020-05-19) · Gellman, Barton · Poitras, Laura · Greenwald, Glenn · MacAskill, Ewen · Untersinger, Martin · Dance, Gabriel -
Procedures Used By The National Security Agency For Targeting Non-United States Persons Reasonably Believed To Be Located Outside The United States To Acquire Foreign Intelligence Court Information Pursuant To Section 702 Of The Foreign Intelligence Surveillance Act Of 1978 [pdf]
The NSA’s 2009 Section 702 targeting procedures lay out how analysts decide who can be surveilled without a warrant, supposedly limiting spying to foreigners abroad. In reality, the rules give the agency wide latitude to presume people are foreign, rely on vague “totality of the circumstances” tests, and allow broad technical methods like IP filters and metadata checks. Oversight is mostly internal and after-the-fact, creating significant potential for U.S. communications to be swept up with little accountability.
People: Holder, Eric H. Jr. -
PROFILING SSL AND ATTRIBUTING PRIVATE NETWORKS [pdf]
A TOP SECRET GCHQ presentation introducing two prototype cloud analytics — FLYING PIG, a knowledge base for investigating TLS/SSL traffic at scale across approximately 20 billion events per week, and HUSH PUPPY, a tool for attributing private network traffic to identifiable organisations — with live interface screenshots demonstrating queries against mail.ru infrastructure and operational successes including identification of the Afghan Ministry of Defence network and a foreign Ministry of Foreign Affairs VSAT network.
Programs: FLYINGPIG · HUSHPUPPY · QUICKANT · GEOFUSION · LEGIONJADE · LEGIONRUBY · MAKERSMARK · XKEYSCORE · NOSEYPARKER“Prying Eyes: Inside the NSA's War on Internet Security” - Der Spiegel ↗ (2014-12-28) · “'Brasil é um grande alvo', diz jornalista sobre vigilância dos EUA” - O Globo ↗ (2013-07-07) · Appelbaum, Jacob · Gibson, Aaron · Grothoff, Christian · Müller-Maguhn, Andy · Poitras, Laura · Sontheimer, Michael · Stöcker, Christian · Greenwald, Glenn · Kaz, Roberto · Casado, José -
QFD BLACKHOLE Technology Behind INOC [pdf]
A UK SECRET STRAP2 COMINT ORCON GCHQ Applied Research presentation from March 2009 documenting the BLACK HOLE bulk data repository's scale and structure, and the suite of analytical tools — MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS and MARBLED GECKO — built on top of it to answer specific intelligence questions about online presence, browsing activity, forum usage and geolocation searches.
Programs: BLACKHOLE · MESME · NETEZZA · CLOUDS · MUTANTBROTH · KARMAPOLICE · INFINITEMONKEYS · MARBLEDGECKO · GOOB · AUTOASSOC · PSOUP · SOCIALANIMAL · SAMUELPEPYS · MEMORYHOLE · HRMAP · PSOUPALERT · LAUGHINGHYENA -
Regions [pdf]
A classified GCHQ document listing 63 undersea and terrestrial cables accessible to British intelligence, with landing locations across the UK coastline and capacity figures in gigabits per second, forming part of the Snowden archive documents published by Süddeutsche Zeitung revealing Cable & Wireless/Gerontic's cooperation with GCHQ.
“Vodafone – der lange Arm des britischen Geheimdienstes?” - WDR ↗ (2014-11-20) · “Spy cable revealed: how telecoms firm worked with GCHQ” - Channel 4 ↗ (2014-11-20) · “Der lange Arm des Geheimdienstes?” - NDR ↗ (2014-11-21) · “Snowden-Leaks: How Vodafone-Subsidiary Cable & Wireless Aided GCHQ’s Spying Efforts” - Süddeutsche Zeitung ↗ (2014-11-25) · Restle, Georg · White, Geoff · Strozyk, Jan · Moltke, Henrik · Poitras, Laura · Obermaier, Frederik -
SIGINT Mission Strategic Plan FY2008-2013 [pdf]
This five-year mission plan for the National Security Agency shows how the agency’s leaders view its mission and culture.
-
SKYNET: Applying Advanced Cloud-based Behavior Analytics [pdf]
A classified NSA SIGINT Development presentation detailing SKYNET, a cloud-based machine learning system that analyzes bulk Pakistani call data records to identify suspected couriers and terrorists through automated travel pattern analysis, behavioral profiling, and social network mapping — with the goal of supporting targeting operations against groups including the Haqqani Network.
Programs: SKYNET · DEMONSPIT · TMAC · FASTSCOPE · TUSKATTIRE · GMHALO · GMPLACE · BULLDOZER · MDR2 · CINEPLEX · JEMA · FASCIA · CLOUD · DPS · SEDB · SMARTTRACKER · SORTINGLEAD · ASSOCIATION · BANYAN · RT-RG · SNOWHAZE · MASTERSHAKE · MAINWAY -
Special Source Collection - March 2013 [pdf]
A March 2013 NSA Special Source Operations briefing for the NSA Threat Operations Center outlining SSO's portfolio of surveillance programs, their legal authorities, and operational constraints — revealing that SSO accounts for approximately 60% of NSA content collection and 75% of metadata collection, and mapping all major collection programs across three access categories: foreign partner, corporate, and unilateral.
-
Special Source Operations - The Cryptologic Provider of Intelligence from Global High-Capacity Telecommunications Systems [pdf]
The document outlines NSA special source operations for accessing global telecommunications data, detailing corporate and foreign cable interception programs. It highlights the MUSCULAR program (DS-200B), a joint NSA–GCHQ operation in the UK that taps into high-capacity fiber-optic links to collect and process large volumes of data from private networks.
-
SSO Collection Optimization [pdf]
The documents reveal that the NSA, in collaboration with GCHQ, accessed private internal networks of Google and Yahoo by intercepting data in transit between their data centers, without breaking into user accounts or servers. This involved analyzing proprietary formats and protocols, such as Google’s Gaia authentication system, using specialized tools to extract intelligence while operating primarily under foreign jurisdiction with fewer legal restrictions.
-
SSO GHOSTMACHINE Analytics [pdf]
This document describes the NSA’s cloud analytics platform GHOSTMACHINE, used by Special Source Operations (SSO) to process and analyze massive volumes of data (e.g., from FASCIA) using Hadoop-based infrastructure. It’s essentially the backbone system for querying, correlating, and enriching surveillance data at scale — not just a single analytic, but the analytic/cloud environment powering many programs.
Programs: GHOSTMACHINE · FASCIA · CERF · POPTOP · DRT · SIGDASYS · JUGGERNAUT · SEADIVER · AST128 · TUSKATTIRE · ASDF · BROKENRECORD · REDHAWK · SSO