Tag: metadata analysis 41 documents
Search documents matching “metadata analysis” →Free-text subjects the curators added. Unlike the other axes these are not necessarily from the documents' own vocabulary, so they are the loosest thing to browse by (and the most forgiving).
Graphs
By Banner
By Classification
Refine (combine with another filter, via search)
Sort: Title
Documents
-
MILKWHITE Enrichment Service (MES) Programme [pdf]
This March 2011 GCHQ document details the MILKWHITE program, which is enhancing domestic interception and data analysis capabilities. It outlines expanded funding for advanced analytics, bulk data storage, and service scaling to meet growing demand from multiple law enforcement and intelligence agencies, including SOCA, HMRC, MI5, and SIS, thereby broadening access to large volumes of internet-derived metadata for operational use.
-
New Contact-Chaining Procedures to Allow Better, Faster Analysis SIDtoday [pdf]
The SIGINT Management Directive 424 briefing (Jan 2011) announces a major procedural shift at the NSA, allowing analysts to conduct contact chaining and large-scale metadata analysis from any selector, regardless of nationality or location. This change removes prior restrictions on U.S. data, enabling more powerful social network mapping for foreign intelligence purposes.
Programs: GATEKEEPER -
Next Generation Events [pdf]
A 2009 TOP SECRET briefing outlines the “Next Generation Events” program to scale internet metadata and content analysis across Five Eyes, addressing explosive growth in data volume and the limits of legacy ingest systems. It presents a phased roadmap combining cloud infrastructure, large‑scale event processing, and enhanced profiling to enable bulk analytics and contact chaining.
Programs: NGE · HAUSTORIUM · BLAZINGSADDLES · HARBOURPILOT · MUTANTBROTH · AUTOASSOC · KARMAPOLICE · SOCIALANIMAL · INFINITEMONKEYS · HRMAP · MEMORYHOLE · MARBLEDGECKO · SIMMER · ROCKRIDGE · SAMUELPEPYS · CAFFEINEHIT · LAUGHINGHYENA · SALAMANCA · CLOUD · MONTEVISTA · LOOKINGGLASS · XKEYSCORE · FUMECUPBOARD · IIB · MoaG -
NSA inspector general report on email and internet data collection under Stellar Wind, 24 March 2009 [Working Draft] [pdf]
This 2009 NSA Inspector General draft report details the origins and legal framework of the President’s Surveillance Program (PSP), launched after 9/11 to bypass FISA limits and authorize bulk interception of internet and telephony content and metadata, including communications with one end inside the U.S. It reveals how NSA, with White House backing, expanded surveillance authority to cover “collection gaps” that existing law didn’t allow, effectively legalizing large-scale warrantless spying under a secret presidential directive.
People: Bush, George W. · Hayden, Michael · Tenet, George · Gonzales, Alberto R. · Addington, David S. · Graham, Robert · Deitz, Robert L. · Shelby, Richard · Pelosi, Nancy · Cheney, Dick · Libby, I. Lewis · Lamberth, Royce · Inouye, Daniel · Levin, Daniel · Stevens, Theodore · Kollar-Kotelly, Colleen · Rockefeller, Jay · Alexander, Keith B. · Potenza, Vito -
PINWALE: Transnational DNI Training [pdf]
This presentation is a classified NSA training guide on PINWALE, a SIGINT database used for the storage, retrieval, and analysis of intercepted digital network intelligence (DNI) communications. It outlines system capabilities, query creation, search tools, and interfaces such as NEXUS, UIS, and AGILITY, explaining how analysts search, retrieve, and analyze both content and metadata — including FISA-partitioned data — within the Five Eyes intelligence framework.
-
PRESTON Architecture [pdf]
The PRESTON architecture, centered on GCHQ’s PRESTON Processing system, is detailed across multiple views showing how the agency intercepts, processes, and stores communications. It outlines a sophisticated surveillance framework that integrates lawful interception, data processing, and analyst access, emphasizing covert monitoring and management of intercepted data.
Programs: PRESTON · FAST GROK · BOXSTER · MARMION · GENTIAN · TERRAIN · BLACKNIGHT · CORINTH · FARNDALE · SAMDYCE · B3M · VSLT · SAMBOK · HAUSTORIUM · FUNFAIR · Cheltenham · SALAMANCA · IIB · LOCHNVAR · MONACO · HOTLINE · Benhall · GDR · CE · BRIGHTON · ARSENIDE · SPQR · TACHO · TRAFFIC MASTER · KEYCARD · XKEYSCORE · COURIERSKILL · MONKEYPUZZLE · PILBEAM · GREENHEART · CCDF · GTE · CADENCE · TERRAINS · GCNET · HENCE -
Procedures Used By The National Security Agency For Targeting Non-United States Persons Reasonably Believed To Be Located Outside The United States To Acquire Foreign Intelligence Court Information Pursuant To Section 702 Of The Foreign Intelligence Surveillance Act Of 1978 [pdf]
The NSA’s 2009 Section 702 targeting procedures lay out how analysts decide who can be surveilled without a warrant, supposedly limiting spying to foreigners abroad. In reality, the rules give the agency wide latitude to presume people are foreign, rely on vague “totality of the circumstances” tests, and allow broad technical methods like IP filters and metadata checks. Oversight is mostly internal and after-the-fact, creating significant potential for U.S. communications to be swept up with little accountability.
People: Holder, Eric H. Jr. -
Psychology: A New Kind of SIGDEV [pdf]
This document presents a detailed approach to integrating human sciences—such as psychology, sociology, anthropology, economics, and political science—into intelligence operations. It highlights the use of tools like SIGINT, HUMINT, and covert online monitoring to understand and shape human behavior, influence networks, and assess socio-cultural dynamics. The focus is on using real-time data analytics, including tracking social media interactions, to optimize intelligence gathering and strategic influence.
Programs: JTRIG · HSOC · SQUEAKY DOLPHIN · DISTILLERY · AIRWOLF · ANTICRISISGIRL · SPLUNK · STREAMINGANALYTICS · HOLLOWPOINT · FIREANT · GTE“Snowden Documents Reveal Covert Surveillance and Pressure Tactics Aimed at WikiLeaks and Its Supporters” - The Intercept ↗ (2014-02-18) · “Snowden docs reveal British spies snooped on YouTube and Facebook” - NBC News ↗ (2014-01-27) · Greenwald, Glenn · Gallagher, Ryan · Esposito, Richard · Cole, Matthew · Schone, Mark -
Real-Time Regional Gateway Cloud Analytics for Forward Users [pdf]
This document outlines the NSA’s Real-Time Regional Gateway (RT-RG) program, a global cloud-based signals intelligence system designed to provide near real-time data analysis and targeting support for military and intelligence operations. It details the transition to scalable cloud architectures such as GHOSTMACHINE and Cloudbase, enabling massive data retention, exploitation, advanced geolocation, and automated analytics for tracking high-value targets across multiple theaters, including Afghanistan, Iraq, and the Horn of Africa.
Programs: RT-RG · GOLDMINER · GEOT · AGENTLOGIC · SHARKFINN · SKS · PANOPTICON · DRT · CIDNE · JUGGERNAUT · LOPERS · TALIS · MATTERHORN · AIRHANDLER · LYCANTHROPE · VOICESAIL · RETURNSPRING · SORTINGLEAD · GHOSTMACHINE · GMROYALE · GMREGAL · GMGEIST · GMIDEA · GMTOTE · GMPEN · GMPAC · GMCALM · GMMATE · GMAPPLE · GMCAVE · GMPEACH · GMDEN · GMMOUNT · GMALAMO · GMLIGHT · GMBALTIC · GMPLACE · GMPARK · GMHALO · GMCARBON · DRAGONFIRE · DOMEEXPOSE · GIANTSQUID · MAINWAY · FURIOUSCHAINSAW · SHARKFURY · NIAGARAFILES · SHARKQUERY · SHARKDOCS · OCTAVE · UTT · BEDDOWN · CHEEKYMONKEY · LAYERCAKE · CYBERTRANS · GMSEMINOLE · GMGULF · GMZILLA · HLT · JEMA · NEVIS · SEDB · CLOUDBASE · MAPREDUCE · CASPORT · WAVELEGAL“Mission Creep: How the NSA’s Game-Changing Targeting System Built for Iraq and Afghanistan Ended Up on the Mexico Border” - The Intercept ↗ (2019-05-29) · “Norges hemmelige krig i Afghanistan” - NRK ↗ (2019-05-29) · Moltke, Henrik · Skille, Øyvind Bye · Døvik, Olav · Hansen, Ståle Ellingsgard · Fjørtoft, Martin -
Register for the Worldwide HPCP Conference, 27-31 October SIDtoday [pdf]
Announcement of a top-secret conference on High Powered Cordless Phones. “HPCP communications are increasing exponentially worldwide.”
The Intercept ↗ (2016-08-10) -
ROYAL CONCIERGE [pdf]
This document describes GCHQ’s top‑secret “ROYAL CONCIERGE program, an automated SIGINT system that mines hotel reservation emails and links to them from government (gov.xx) addresses to produce daily tip‑offs about diplomats’ travel and hotel stays. Those alerts enable a range of follow‑on options — from technical attacks (network/hotel room exploitation and XKeyScore fingerprinting) to HUMINT, and even attempts to influence or cancel visits.
-
Running Strategic Analytics Affecting Europe and Africa [pdf]
This Top Secret presentation from the NSA ECC (European Cryptologic Center) outlines the strategic challenges and technical methodology for conducting signal intelligence operations across Europe and Africa. It emphasizes a critical reliance on "Second Party" partnerships, particularly with the UK's GCHQ, and details the use of mass surveillance tools like XKEYSCORE to track terrorism, smuggling, and human trafficking networks.
Programs: XKEYSCORE · RTRG · DISTILLERY · CLOUD · GHOSTMACHINE · NIAGARAFILES · WHIZBANG · MAPREDUCE -
RUSSIA/SIGDEV Holds Bootcamp for Hungarian Partners SIDtoday [pdf]
NSA held a six-day "SIGDEV bootcamp" with the Hungarian Military Intelligence Office (MIO) to collaborate on target development against Russia.
Programs: MUSKETEERThe Intercept ↗ (2019-05-29) -
Sensitive Metadata Analytic Collaboration (SMAC) - Concept of Operations, October 2006 [pdf]
This document outlines the Sensitive Metadata Analytic Collaboration (SMAC) initiative, a system developed by the Five Eyes intelligence agencies to facilitate the sharing and analysis of sensitive metadata to support SIGINT (signals intelligence) missions. While the concept aims to improve collaboration and data sharing, its success is hindered by ongoing policy, legal, and technical barriers, as well as concerns about trust and participation among the agencies involved. The document suggests a complex, somewhat opaque process of metadata sharing that risks undermining privacy protections.
Programs: SMAC -
SHELLTRUMPET Processes its One Trillionth Metadata Record [pdf]
An SSO news item announcing that SHELLTRUMPET processed its one-trillionth metadata record on 21 December 2012 — almost half of that volume in 2012 alone — feeding TRAFFICTHIEF tipping and Real-Time Regional Gateway filtering from CLASSIC, DANCINGOASIS, RAMPART-M, OAKSTAR, MYSTIC, NCSC, MUSKETEER and Second Party systems.
Programs: SHELLTRUMPET · CLASSIC · TRAFFICTHIEF · RTRG · SSO · DANCINGOASIS · RAMPART-M · OAKSTAR · MYSTIC · NCSC · MUSKETEER -
SIGINT Development: the Target Analysis Center (TAC) SIDtoday [pdf]
Description of the Target Analysis Center, whose mission is to "develop corporate-level analytic tools and techniques" against networked systems.
People: Plunkett, DeboraThe Intercept ↗ (2016-12-07) -
SKYNET: Applying Advanced Cloud-based Behavior Analytics [pdf]
A classified NSA SIGINT Development presentation detailing SKYNET, a cloud-based machine learning system that analyzes bulk Pakistani call data records to identify suspected couriers and terrorists through automated travel pattern analysis, behavioral profiling, and social network mapping — with the goal of supporting targeting operations against groups including the Haqqani Network.
Programs: SKYNET · DEMONSPIT · TMAC · FASTSCOPE · TUSKATTIRE · GMHALO · GMPLACE · BULLDOZER · MDR2 · CINEPLEX · JEMA · FASCIA · CLOUD · DPS · SEDB · SMARTTRACKER · SORTINGLEAD · ASSOCIATION · BANYAN · RT-RG · SNOWHAZE · MASTERSHAKE · MAINWAY -
SSO dictionary relevant entries [pdf]
This document provides a detailed list of relevant entries related to NSA's Special Source Operations (SSO), highlighting various NSA surveillance and data collection systems used to monitor global communications. It includes descriptions of programs such as BIGBIRD and DISHFIRE, which focus on cable and SMS interception, as well as tools like LOPERS and COURIERSKILL for processing telephony metadata and refining target selection. The entries emphasize the extensive and automated nature of the NSA's intelligence-gathering operations, particularly in relation to undersea cables and telecommunication networks, and underscore the secretive and expansive scope of SSO's global surveillance activities.
Programs: ARTIFICE · BIGBIRD · FAIRVIEW · SSO · POORWILL · PINECONE · BLACKBELT · CADENCE · BLACKNIGHT · SHIPMASTER · COURIERSKILL · THEORYMASTER · WEALTHYCLUSTER · CLIFFSIDE · DISHFIRE · MARINA · MAINWAY · NUCLEON · PINWALE · FASCIA · BANYAN · LAMPSHADE · ASSOCIATION · HOMEBASE · SEDB · FRIAR · KEYCARD · TURMOIL · TURBULENCE · UTT · IRISHBEAUTY · OCTAVE · KOZYKOVE · LITHIUM · BLARNEY · LOPERS · MAILORDER · LUMBERYARD · MERLIN · NODDY · SAGUARO · NUTHATCH · CONTRAOCTAVE · XKEYSCORE · PLANK · RODEOSTAR · SAGURA · SERENADE · STARGATE · SILVERCOLLAM · SEALION · SEAGULL · SLIVER · SORA-2 · TITANPOINTE · TUBE · VPCS · STONEGATE“NSA Spying Relies on AT&T’s ‘Extreme Willingness to Help’” - ProPublica ↗ (2015-08-15) · “AT&T Helped U.S. Spy on Internet on a Vast Scale” - The New York Times ↗ (2015-08-15) · “The Wiretap Rooms: The NSA’s Hidden Spy Hubs in Eight U.S. Cities” - The Intercept ↗ (2018-06-25) · Angwin, Julia · Larson, Jeff · Savage, Charlie · Moltke, Henrik · Poitras, Laura · Risen, James · Gallagher, Ryan -
SYANPSE Data Model [pdf]
This document outlines the NSA’s SYANPSE Data Model, a classified metadata framework used for mapping relationships between people, organizations, devices, and intelligence designators. It details how various entities (e.g., phone numbers, email addresses, reports, and organizational units) are linked through structured graph relationships to support SIGINT analysis and tasking across the Five Eyes intelligence alliance.
-
The New Meade Operations Center - Organization and Functions SIDtoday [pdf]
A summary of many projects being developed at the Meade Operations Center. “Expanding on its current role as a test bed for new capabilities, the director envisions the MOC as a developmental platform for tactics, techniques, procedures, tools, systems, and ideas that can be exported” to regional offices.
People: Gray, AlThe Intercept ↗ (2016-08-10)