Tag: data processing 29 documents
Search documents matching “data processing” →Free-text subjects the curators added. Unlike the other axes these are not necessarily from the documents' own vocabulary, so they are the loosest thing to browse by (and the most forgiving).
Graphs
By Banner
By Classification
Refine (combine with another filter, via search)
Sort: Title
Documents
-
EAGLEVIEW -- A Data Processing Tool for GSM Cellular Networks SIDtoday [pdf]
A system called "EAGLEVIEW" helps analysts map out GSM cellular phone networks. The system takes information collected in the field and plots out network coverage.
The Intercept ↗ (2017-09-13) -
AURORAGOLD – Target Technology Trends Center support to WPMO [pdf]
This briefing details the technical architecture and data pipeline used to process global cellular network information. It specifically highlights the synergy between data gathering on GSM/UMTS networks and the implementation of "Auto-Minimization" to ensure the resulting SIGINT reporting complies with NSA legal and policy requirements.
-
BLACK HOLE ANALYTICS, ADD/SD briefing, September 2009 [pdf]
This September 2009 GCHQ briefing describes the BLACK HOLE analytics infrastructure and Next Generation Events (NGE) project, which planned to scale from ~8 10-gigabit connections to ~200 10Gs by March 2011, storing intercepted data in massive flat-file repositories at Benhall and Bude before loading it into "Question Focused Dataset" (QFD) databases that power surveillance tools like MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS, and others. The document outlines how "probes" extract Target Detection Identifiers (TDIs) from intercepted data feeds, store them in BLACK HOLE repositories, then enable analysts to profile targets' online activities, create social networks, investigate websites and forums, track web searches and Google Earth usage, and perform large-scale data mining across internet and telephony data.
Programs: BLACKHOLE · MUTANTBROTH · AUTOASSOC · INFINITEMONKEYS · KARMAPOLICE · SOCIALANIMAL · HRMAP · MEMORYHOLE · MARBLEDGECKO · HARDASSOC · SAMUELPEPYS · BLAZINGSADDLES · TELLURIAN · BUFFER · DISTILLERY · LOOKINGGLASS · NGEPeople: bin Laden, Osama -
Communications Capability Development Programme [pdf]
The Communications Capability Development (CCD) Programme (formerly the Intercept Modernisation Programme, IMP) was a Home Office–led effort to maintain the U.K.’s lawful interception and communications data capabilities amid changing technologies, emphasizing that law enforcement faced a “significant decline” without new measures. GCHQ supported CCD with technical expertise and interim services like MILKWHITE Enrichment, while stressing that CCD was politically contentious, technically complex, and required legislation to mandate data retention by U.K. network operators.
-
Data Flow Manager: The Data Fairy? SIDtoday [pdf]
Data flow managers understand data and the processing that ensures that the collected signal is converted to data and intelligence and delivered to customers.
Marking: CONFIDENTIALThe Intercept ↗ (2016-05-16) -
Dealing With a 'Tsunami' of Intercept SIDtoday [pdf]
As part of NSAs efforts to implement Human Language Technology, prototype processors located at collection sites will soon be able to query content at "line speed" based on the language, it is written in.
People: Kulman, Anita H.The Intercept ↗ (2019-05-29) · “Inside NSA, Officials Privately Criticize "Collect It All" Surveillance” - The Intercept ↗ (2015-05-28) · Maass, Peter -
DEEPDIVE Readme [pdf]
This document is a confidential technical guide that provides step-by-step procedures for configuring an XKEYSCORE server to function as a DEEPDIVE server. DEEPDIVE enhances XKEYSCORE by adding a front-end filter and a rolling buffer of all unfiltered data, creating a federated query system that can scan all collection sites with a single query.
-
Digint Imbalance [pdf]
This secret MI5 report highlights a growing imbalance between the massive scale of digital intelligence (DIGINT) collection and the agency’s limited capacity to analyze it, leaving much data unassessed and creating risks of intelligence failure. It identifies major challenges in staffing, tools, and infrastructure, and calls for improved analysis, better integration with other intelligence, and new capabilities to disrupt online threats.
Marking: SECRET -
FASCIA: The NSA's huge trove of location records [pdf]
FASCIA is the National Security Agency's enormous database containing trillions of metadata records, many with device locations, that are collected from a variety of sources. This document shows the volume and types of device-location data collected by the agency.
-
GCHQ access to international internet communications [pdf]
This document describes the scale of the agency's access to international internet communications, stating that GCHQ receives upwards of 50 billion events per day and that this volume is growing.
Programs: GUIDINGLIGHT -
HIMR Data Mining Research Problem Book [pdf]
This document is a research roadmap created by GCHQ for the Heilbronn Institute for Mathematical Research (HIMR). Its purpose is to guide long-term mathematical and data-mining research that could improve signals intelligence (SIGINT) capabilities. It outlines key research problems, datasets, and collaboration plans between intelligence agencies and academic researchers. The main goal is to develop new algorithms and mathematical tools to analyze extremely large communications datasets produced by modern surveillance systems.
Programs: CLOUD · DISTILLERY · DISCOVER · PRESTON · LECKWITH · TERRAIN · BROADOAK · NGE · FIVEALIVE · CHARTBREAKER · FIRSTCONTACT · GEOFUSION · RADONSHARPEN-B · PRIMETIME · HIDDENOTTER · BAKERSDOZEN · CLASP · SALTYOTTER · BIRCH · SOLIDINK · FLUIDINK · MAMBA · GRINNINGROACH · PIRATECAREBEAR · AUTOASSOC · CROUCHINGSQUIRREL · HRMAP · KACHINA · SQUEAL · QUESTA · METEORSHOWER · SUNSTORM · GOLDMINE · HAGERAWEL · WOODY · BUZZ · SILVERLIBRARY · SILVERLINING · VALHALLA · MOUNTMCKINLEY · SEPANG · BLACKHOLE · SALAMANCA · HAGELAWEL · LUCKYSTRIKE · SPIKYROCK · MYOFIBRIL · INJUNCTION · PSYCHICSALMON · RAGINGBULLFROG · ROBOTICFISH · TIMIDTOAD · AURA · CARBONCOPY · CASK · CHORDAL · CRAN · INSTINCT · MADFORGE · SWAMP · VOLSUNGA · WHITERAVEN · MVR · PPF · JTRIG · MAPREDUCE · MAILORDERPeople: Tukey, John -
How the NSA Pinpoints a Mobile Device [pdf]
This document is an internal National Security Agency classification guide outlining the agency's abilities to intercept cellular communications.
Programs: ARCVIEW -
Innovative Super Sized SIGINT Facility Opens in Afghanistan SIDtoday [pdf]
The Afghanistan Regional Operations Cryptologic Center outside Kabul, Afghanistan, is now operational. Located at Bagram airfield, the new "super sized" signals intelligence facility will house 250 personnel from nine countries, 120 of them linguists. Material was flown in on six Boeing 747s or trucked from Pakistan, via the Khyber pass.
“Mission Creep: How the NSA’s Game-Changing Targeting System Built for Iraq and Afghanistan Ended Up on the Mexico Border” - The Intercept ↗ (2019-05-29) · “Norges hemmelige krig i Afghanistan” - NRK ↗ (2019-05-29) · The Intercept ↗ (2019-05-29) · Moltke, Henrik · Skille, Øyvind Bye · Døvik, Olav · Hansen, Ståle Ellingsgard · Fjørtoft, Martin -
LEVITATION and the FFU Hypothesis [pdf]
This document describes LEVITATION, a top-secret Canadian Communications Security Establishment (CSE) project that monitors millions of daily file uploads and downloads from free file-sharing sites to identify potential extremist activity. It details the program’s data collection methods, analytical workflow, use of partner databases, and examples of successful discoveries related to terrorist materials and hostage videos.
-
Mission Assurance Facility Phase 2 Completed SIDtoday [pdf]
A progress report on the construction of the Interim Mission Assurance Facility reveals that the second of four phases is complete. The site will protect SID's assets against catastrophic destruction.
The Intercept ↗ (2017-04-24) -
New Release: The FORNSAT Plan SIDtoday [pdf]
Description of a plan for the modernization of collection and processing of international communications from foreign communications satellites.
Programs: TWDThe Intercept ↗ (2016-05-16) -
NSA's collection processing and dissemination of allied communications [pdf]
A secret 2007 deal between the US and UK allowed the NSA to collect, store, and analyze personal data of British citizens — including those not suspected of wrongdoing — marking a significant expansion of mass surveillance powers and breaking prior Five-Eyes agreements on mutual non-targeting.
-
OPULENT PUP Encryption Attack [pdf]
"OPULENT PUP encryption attack (GCHQ)" is a 2009 technical planning document from the British intelligence agency GCHQ. It outlines a "proof-of-concept demonstrator" for a new capability to intercept and decrypt GSM communications protected by the A5/3 algorithm, requiring a complete overhaul of the existing processing chain used for the older A5/1 standard.
Programs: OPULENTPUP -
PRESTON Architecture [pdf]
The PRESTON architecture, centered on GCHQ’s PRESTON Processing system, is detailed across multiple views showing how the agency intercepts, processes, and stores communications. It outlines a sophisticated surveillance framework that integrates lawful interception, data processing, and analyst access, emphasizing covert monitoring and management of intercepted data.
Programs: PRESTON · FAST GROK · BOXSTER · MARMION · GENTIAN · TERRAIN · BLACKNIGHT · CORINTH · FARNDALE · SAMDYCE · B3M · VSLT · SAMBOK · HAUSTORIUM · FUNFAIR · Cheltenham · SALAMANCA · IIB · LOCHNVAR · MONACO · HOTLINE · Benhall · GDR · CE · BRIGHTON · ARSENIDE · SPQR · TACHO · TRAFFIC MASTER · KEYCARD · XKEYSCORE · COURIERSKILL · MONKEYPUZZLE · PILBEAM · GREENHEART · CCDF · GTE · CADENCE · TERRAINS · GCNET · HENCE -
PRESTON Study [pdf]
This GCHQ document details the PRESTON surveillance program, revealing that over a six-month period, 97% of intercepted communications—including calls, texts, and internet data—were not viewed by analysts despite being fully warranted, highlighting systemic challenges in processing and exploiting the vast volume of collected intelligence. It underscores the reliance on traditional PRESTON sources for events like the London G20, while noting gaps in data feeds from other repositories, reflecting both operational scale and limitations in effective oversight.