Tag: QFD 18 documents
Search documents matching “QFD” →Free-text subjects the curators added. Unlike the other axes these are not necessarily from the documents' own vocabulary, so they are the loosest thing to browse by (and the most forgiving).
Graphs
By Banner
By Classification
Refine (combine with another filter, via search)
Sort: Title
Documents
-
BLACK HOLE ANALYTICS, ADD/SD briefing, September 2009 [pdf]
This September 2009 GCHQ briefing describes the BLACK HOLE analytics infrastructure and Next Generation Events (NGE) project, which planned to scale from ~8 10-gigabit connections to ~200 10Gs by March 2011, storing intercepted data in massive flat-file repositories at Benhall and Bude before loading it into "Question Focused Dataset" (QFD) databases that power surveillance tools like MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS, and others. The document outlines how "probes" extract Target Detection Identifiers (TDIs) from intercepted data feeds, store them in BLACK HOLE repositories, then enable analysts to profile targets' online activities, create social networks, investigate websites and forums, track web searches and Google Earth usage, and perform large-scale data mining across internet and telephony data.
Programs: BLACKHOLE · MUTANTBROTH · AUTOASSOC · INFINITEMONKEYS · KARMAPOLICE · SOCIALANIMAL · HRMAP · MEMORYHOLE · MARBLEDGECKO · HARDASSOC · SAMUELPEPYS · BLAZINGSADDLES · TELLURIAN · BUFFER · DISTILLERY · LOOKINGGLASS · NGEPeople: bin Laden, Osama -
Blazing Saddles [pdf]
This document describes BLAZING SADDLES, a GCHQ program designed to massively scale the storage and analysis of bulk internet surveillance data through multiple specialized databases that track website visits, online identities, communications patterns, and geolocation queries across the global internet population.
Programs: BLAZINGSADDLES · BLACKHOLE · SOCIALANTHROPOID · AUTOASSOC · HRMAP · INFINITEMONKEYS · KARMAPOLICE · MARBLEDGECKO · MUTANTBROTH · MEMORYHOLE · SAMUELPEPYS · SOCIALANIMAL · HAUSTORIUM · GOLDENAXE · NGE -
Demystifying NGE Rock Ridge [pdf]
This presentation outlines the ROCK RIDGE project, which delivers a suite of "Question Focused Databases" (QFDs) like MUTANT BROTH and KARMA POLICE to analyze massive internet event data. These specialized databases are designed to answer specific intelligence questions and are developed using the BLACK HOLE data repository, a key component of the ROUGH DIAMOND program.
Programs: ROCKRIDGE · ROUGHDIAMONG · BLACKHOLE · HAUSTORIUM · HRMAP · MUTANTBROTH · KARMAPOLICE · GOOBZS · INFINITEMONKEYS · AUTOASSOC · SOCIALANIMAL · MARBLEDGECKO · MEMORYHOLE · SAMUELPEPYS · SALAMANCA · MONOPOLY · RADIUS · HARBOURPILOT · LOOKINGGLASS · FIRESTORM · SUPERDRAKE · BLAZINGSADDLES · BLUESHIFT · SUPPORTINGINO · TERRAINS · GEOFUSION · NGEPeople: Lobban, Iain -
Digby geolocation [pdf]
The document reveals that RAF Digby was equipped with advanced profiling and analytics tools to enable (near) real-time co-location of GSM handsets, supporting intelligence operations across the Middle East and North Africa. It highlights the operational use of the RUM PUNCH algorithm on the DISTILLERY framework, which was jointly developed through an NSA–IBM–GCHQ partnership, allowing large-scale stream mining and independent analyst-led tracking operations. Enhancements at Digby extended beyond communications interception to include a new cyber mission, with capabilities to geo-locate emitters linked to IED arming and detonation under the FLAKJACKET mission, expanding the site’s role from surveillance to active battlefield support.
-
Elegant Chaos: collect it all, exploit it all [pdf]
This document outlines a classified project, "Elegant CHAOS," focused on advanced signals intelligence (SIGINT) collection, processing, and analysis using cutting-edge technologies like automated surveys and modems. It highlights goals such as real-time data analysis, collaboration with international partners, and integration with cloud-based platforms to manage massive data. See also the document with notes (by GCHQ): "Elegant-Chaos-collect-it-all-exploit-it-all-plus"
Programs: ELEGANTCHAOS · ASPHALT · ASPHALT-A · ASPHALT-A-PLUS · STORMFORCE · DARKQUEST · TORUS · XKEYSCORE · MHS CLOUD · TURMOIL · TARMAC · ASDF · SLR · POPQUIZ · WEALTHYCLUSTER2 · TRAVELLINGWAVE · BILBOBADGER · DRINKYBIRD · GLOBETROTTER · MASTERSHAKE · REST · TRAFFICTHIEF · VENUSAFFECT · FOGHORN · MATCHMAKER · ROADBED · MOONPENNY · AMULETSTELLAR · CLOUD · WEALTHYCLUSTER · MVR · TINT · IPGEOTRAP · MARINA · PLUS -
Elegant Chaos: collect it all, exploit it all (plus notes) [pdf]
GCHQ/NSA presentation describing ELEGANT CHAOS — an automated system at the Menwith Hill Station satellite interception facility that scores and ranks thousands of intercepted satellite signals by intelligence value, automatically deciding which ones deserve collection resources based on criteria like geographic location, target network hits, technology type and Computer Network Operations interest — built to implement the station's directive to "Sniff it All, Know it All, Collect it All, Process it All, Exploit it All, Partner it All" as collection capacity scaled from 100 signals to a planned 3,000 simultaneous intercepts.
Programs: ELEGANTCHAOS · ASPHALT · ASPHALT-A · ASPHALT-A-PLUS · STORMFORCE · DARKQUEST · TORUS · XKEYSCORE · MHS CLOUD · TURMOIL · TARMAC · ASDF · SLR · POPQUIZ · WEALTHYCLUSTER2 · TRAVELLINGWAVE · BILBOBADGER · DRINKYBIRD · GLOBETROTTER · MASTERSHAKE · REST · TRAFFICTHIEF · VENUSAFFECT · FOGHORN · MATCHMAKER · ROADBED · MOONPENNY · WHIZBANG · CLOUDBASE · APLUS · MAILORDER · CROSSBONES · AMULET STELLAR · MVR · TINT · WEALTHYCLUSTER · IPGEOTRAP · MARINA · CLOUD · PLUS -
Event (SIGINT) [pdf]
5 p.This document explains that within GCHQ and NSA SIGINT operations, an “Event” refers to metadata about observable communications or user actions rather than message content, categorized into types like Communications, Presence, Social, and Geo-reference Events. These events are collected, stored in various databases, and analyzed through convergence and correlation processes, with standardized formats defined by Interface Control Documents such as INTERSTELLAR DUST (IDUST) and Actor Action (AA).
Programs: INTERSTELLAR DUST · HAUSTORIUM · SOCIAL ANTHROPOID · ACTOR-ACTION · SAMREF · GEOFUSION · HARD ASSOC · SALAMANCA · ASSOCIATION · BANYAN · MAINWAY · FASCIA · CULTWEAVE II · VOICESAIL · TEEDALE · PILBEAM · MUTANTBROTH · AUTOASSOC · MARBLEDGECKO · KARMAPOLICE · HRMAP · MEMORYHOLE · INFINITEMONKEYS · SOCIALANIMAL · AUTOTDI · SAMUELPEPYS · Mobile Applications Project · NGE · PPF · SLR -
GCHQ Analytic Cloud Challenges [pdf]
The 2012 GCHQ Analytic Cloud document outlines a sophisticated data processing infrastructure handling approximately 50 billion daily events across multiple sites, utilizing QFD databases and HADOOP clusters for various analytic capabilities targeting intelligence gathering and analysis. It describes architectural components, challenges in scaling and enabling analysts to cope with data complexity, and proposes future technology options including consolidation approaches and new cloud architectures to enhance capabilities.
Programs: EVOLVEDMUTANTBROTH · MUTANTBROTH · HARDASSOC · HRMAP · KARMAPOLICE · SOCIALANTHROPOID · AUTOASSOC · GEOFUSION · MVR · TERRAIN · SALAMANCA · BLACKHOLE · SAMUELPEPYS · BROKER · VAIL · GUIDINGLIGHT · FIRSTCONTACT · CLOUDYCOBRA · RUMOURMILL · EVERYASSOC · EVERYPOLICE · EVERYCREATURE · EVERYEAD · EVERYCIPHER · GREYFOX · SILVERFOX · GOLDENAXE · TRIBALCARNEM · PUBLICANEMONE · EPICFAIL · STERLINGMOTH · FOGHORN · LOOKINGGLASS · FIREENGINE · NGGEO · ALPHACENTAURI · CLOUDBASE · ACCUMULO · ASEM · SEM · HAKIM · STREAMING · HBASE · LAUREL · SKINNYCLOUD · HARDY · GCHQCORE -
GHOSTMACHINE: Identifier Lead Triage with ECHOBASE [pdf]
This June 2012 presentation introduces ECHOBASE, an NSA analytic architecture designed to automate the triage of massive identifier lead lists (ranging from 50 to 10,000+ entries). By applying behavior-based analytics and "yes/no" behavioral queries against raw SIGINT, the system allows analysts to prioritize high-interest targets and discard irrelevant leads at scale.
Programs: GHOSTMACHINE · ECHOBASE · IDENTIFIERSCOREBOARD · FOREMAN · CONTRAOCTAVE · OCTAVE · UTT · WAVELEGAL · CASPORT · PSC -
HOMING PIGEON [pdf]
Slides on a query-focused dataset that would correlate GSM handsets seen on two or more aircraft flights with their subscribers.
“US and Britain refine their ability to tap into airline passenger’s mobile phones while they are in the air” - Le Monde ↗ (2016-12-07) · “American and British Spy Agencies Targeted In-Flight Mobile Phone Use” - The Intercept ↗ (2016-12-07) · “No Place to Hide” - Metropolitan Books ↗ (2014-05-13) · Follorou, Jacques · Greenwald, Glenn -
Menwith initiatives maximizing our access [pdf]
11 p.This document details Menwith Hill Station’s initiatives to maximize and optimize SIGINT collection, processing, and exploitation, including large-scale monitoring of internet and communications traffic, metadata analysis, and cross-platform target tracking. It describes programs and techniques for surveying, processing, and exploiting signals, enabling analysts to identify, geolocate, and track targets across multiple networks and devices.
Programs: ASPHALT · BLACKTOP · TARMAC · GTE · INDEX · ROADBED · DARKQUEST · EVIL EYE · Spider · SHAREDVISION · QUEST · NOSEYPARKER · XKEYSCORE · GRANDPIANO · FOGHORN · BASSQUEST · APPARITION · MUTANT BROTH · KARMAPOLICE · SOCIALANIMAL · MEMORY HOLE · TURBULENCE · TELLURIAN · MAILORDER · AUTO ASSOC · SLR · BLACKHOLE · JTRIG · PPF · LECKWITH · SCEPTRE · CLOUDPeople: Alexander, Keith B.“GCHQ taps fibre-optic cables for secret access to world's communications” - The Guardian ↗ (2013-06-21) · “Inside Menwith Hill: The NSA’s British Base at the Heart of U.S. Targeted Killing” - The Intercept ↗ (2016-09-06) · MacAskill, Ewen · Borger, Julian · Hopkins, Nick · Davies, Nick · Ball, James · Gallagher, Ryan -
Mobile Apps Checkpoint Meeting Archives [pdf]
This GCHQ document tracks project checkpoints from December 2009 to July 2010, focusing on the rollout and operationalization of mobile apps, TERRAIN system upgrades, GTP/GRX access, and converged event processing. Key themes include coordinating multi-team dependencies, addressing technical gaps in handset and app data characterization, and ensuring timely delivery of milestones like MIP-1.
Programs: TERRAIN · PRESTON · XKEYSCORE · BEGAL · TAMING PASTRIES · MILKWHITE · MVR · SOCIAL ANTHROPOID · INTEGER SPIN · MUTANT BROTH · GEO FUSION · MARBLED GEKO · BLACKHOLE · ACTOR ACTION · TRAFFICMASTER · HARD ASSOC · MIRTF · HEARTBEAT · SILVER LINING · GREENHEART · EREPO · MAGLITE · POKERFACE · BOSTROM · MIDDLESEX GREEN · ENCHANTERSS · HAUSTORIUM · SALAMANCA · BROADOAK · MONKEY PUZZLE · TACHO · EREPO GAMMA · NINJA · PEBBLE BED · GOLDEN EYE · STRAPROC · STARPROC · LOOKING GLASS · SORCERER · SOCIAL ANIMAL · GTE · NGE · PPF · IIB · UDAQ · MES · TERRAINS · CCDF · SINO · GDR · OSDS · LECKWITH -
NGE BLACK HOLE ConOp [pdf]
This document outlines the "BLACK HOLE" Concept of Operations (ConOp) for the "Next Generation Events" (NGE) project within "BLAZING SADDLES," detailing a flat file data store designed to house various intelligence feeds. Its primary purpose is to serve as a source for Query Focused Datasets (QFDs) and support research, bulk analysis, and target discovery, with two operational instances at Benhall and Bude handling massively increased data volumes.
Programs: NGE · BLAZINGSADDLES · BLACKHOLE · MUTANTBROTH · KARMAPOLICE · SOCIALANIMAL · TELLURIAN · BUFFER · MAILORDER · MARBLEDGECKO · SALAMANCA · SMOKINGSADDLES · CIRCUIT · BLACKFIND · BLACKCAT · DATAFLOWCAB · CHORDAL · DEBIT CARD · B13B -
NSA/GCHQ/CSEC Network Tradecraft Advancement Team [pdf]
The "NSA/GCHQ/CSEC Network Tradecraft Advancement Team (NTAT) 2010" document describes a collaborative mission between the US, UK, and Canada to synchronize and advance SIGINT techniques. The NTAT focused on "Network Analysis Tradecraft," specifically integrating analysis tools such as IRASCIABLE RABBIT and TOYGRIPPE into CSEC's OLYMPIA and developing standardized "use cases" for tracking mobile networks and virtual private networks (VPNs).
-
Operational Engineering November 2010 [pdf]
This internal GCHQ briefing from late 2010 documents the Events Product Centre's suite of bulk internet and communications surveillance tools — including IMMINGLE, SALAMANCA, SOCIAL ANTHROPOID, MUTANT BROTH, KARMA POLICE, SAMUEL PEPYS and GUIDING LIGHT — covering scaling targets, new data sources, and operational capabilities across telephony, webmail, instant messaging, social media, GPRS and VoIP collection.
Programs: EVENTS · IMMINGLE · SALAMANCA · GUIDINGLIGHT · B3M · HARDASSOC · MAINWAY · THUGGEE · FASCIA · HAUSTORIUM · FIRESTORM · TERRAIN · SHAREOWN · ESCHAR · BRIO · INFINITEMONKEY · KARMAPOLICE · MARBLEDGECKO · MEMORYHOLE · MUTANTBROTH · SAMUELPEPYS · SOCIALANIMAL · SOCIALANTHROPOID · HEARTBEAT · SANTHROPOID · LEAKYGATEWAYS · GTPTUNNEL · LUSTRE · GLASSBACK · COLLATERAL · HRMAP · SWORDPLAY · BROADOAK · REFORMER · ARTEMIS · MERLOT · CONTRAOCTAVE · CRYINGFOWL · CULTWEAVE · SAMDYCE · GCQUERY · BLAZZINGSADDLES · EREPO · JTRIG · SOUNDER -
PROFILING SSL AND ATTRIBUTING PRIVATE NETWORKS [pdf]
A TOP SECRET GCHQ presentation introducing two prototype cloud analytics — FLYING PIG, a knowledge base for investigating TLS/SSL traffic at scale across approximately 20 billion events per week, and HUSH PUPPY, a tool for attributing private network traffic to identifiable organisations — with live interface screenshots demonstrating queries against mail.ru infrastructure and operational successes including identification of the Afghan Ministry of Defence network and a foreign Ministry of Foreign Affairs VSAT network.
Programs: FLYINGPIG · HUSHPUPPY · QUICKANT · GEOFUSION · LEGIONJADE · LEGIONRUBY · MAKERSMARK · XKEYSCORE · NOSEYPARKER“Prying Eyes: Inside the NSA's War on Internet Security” - Der Spiegel ↗ (2014-12-28) · “'Brasil é um grande alvo', diz jornalista sobre vigilância dos EUA” - O Globo ↗ (2013-07-07) · Appelbaum, Jacob · Gibson, Aaron · Grothoff, Christian · Müller-Maguhn, Andy · Poitras, Laura · Sontheimer, Michael · Stöcker, Christian · Greenwald, Glenn · Kaz, Roberto · Casado, José -
Report on Architectural Risk 2012 - Summary [pdf]
This 2012 GCHQ report identifies that a historical under-investment in IT architecture has created significant risk, especially with new challenges like massive data growth, cyber defense, and increased partnering. The document recommends immediate, prioritized investment in big data strategies, integrated analytics frameworks, and security services to address the strain on the system.
Programs: BLACKHOLE -
SIGINT Mission Thread 3 SIDtoday [pdf]
A 2011 NSA SIDtoday article outlining SIGINT Mission Thread 3's strategic initiative to shift the SIGINT Directorate's analytical culture away from data production toward discovery and understanding, introducing tools like the Analytic Body of Knowledge, Analytics Taxonomy, and GAMBIT portal to systematize tradecraft sharing across the Five Eyes enterprise.
Programs: GAMBITPeople: Gladwell, Malcolm