Tag: data ingest 12 documents
Search documents matching “data ingest” →Free-text subjects the curators added. Unlike the other axes these are not necessarily from the documents' own vocabulary, so they are the loosest thing to browse by (and the most forgiving).
Graphs
By Banner
By Classification
Refine (combine with another filter, via search)
Sort: Title
Documents
-
AURORAGOLD – Target Technology Trends Center support to WPMO [pdf]
This briefing details the technical architecture and data pipeline used to process global cellular network information. It specifically highlights the synergy between data gathering on GSM/UMTS networks and the implementation of "Auto-Minimization" to ensure the resulting SIGINT reporting complies with NSA legal and policy requirements.
-
DEEPDIVE Readme [pdf]
This document is a confidential technical guide that provides step-by-step procedures for configuring an XKEYSCORE server to function as a DEEPDIVE server. DEEPDIVE enhances XKEYSCORE by adding a front-end filter and a rolling buffer of all unfiltered data, creating a federated query system that can scan all collection sites with a single query.
-
Elegant Chaos: collect it all, exploit it all [pdf]
This document outlines a classified project, "Elegant CHAOS," focused on advanced signals intelligence (SIGINT) collection, processing, and analysis using cutting-edge technologies like automated surveys and modems. It highlights goals such as real-time data analysis, collaboration with international partners, and integration with cloud-based platforms to manage massive data. See also the document with notes (by GCHQ): "Elegant-Chaos-collect-it-all-exploit-it-all-plus"
Programs: ELEGANTCHAOS · ASPHALT · ASPHALT-A · ASPHALT-A-PLUS · STORMFORCE · DARKQUEST · TORUS · XKEYSCORE · MHS CLOUD · TURMOIL · TARMAC · ASDF · SLR · POPQUIZ · WEALTHYCLUSTER2 · TRAVELLINGWAVE · BILBOBADGER · DRINKYBIRD · GLOBETROTTER · MASTERSHAKE · REST · TRAFFICTHIEF · VENUSAFFECT · FOGHORN · MATCHMAKER · ROADBED · MOONPENNY · AMULETSTELLAR · CLOUD · WEALTHYCLUSTER · MVR · TINT · IPGEOTRAP · MARINA · PLUS -
Elegant Chaos: collect it all, exploit it all (plus notes) [pdf]
GCHQ/NSA presentation describing ELEGANT CHAOS — an automated system at the Menwith Hill Station satellite interception facility that scores and ranks thousands of intercepted satellite signals by intelligence value, automatically deciding which ones deserve collection resources based on criteria like geographic location, target network hits, technology type and Computer Network Operations interest — built to implement the station's directive to "Sniff it All, Know it All, Collect it All, Process it All, Exploit it All, Partner it All" as collection capacity scaled from 100 signals to a planned 3,000 simultaneous intercepts.
Programs: ELEGANTCHAOS · ASPHALT · ASPHALT-A · ASPHALT-A-PLUS · STORMFORCE · DARKQUEST · TORUS · XKEYSCORE · MHS CLOUD · TURMOIL · TARMAC · ASDF · SLR · POPQUIZ · WEALTHYCLUSTER2 · TRAVELLINGWAVE · BILBOBADGER · DRINKYBIRD · GLOBETROTTER · MASTERSHAKE · REST · TRAFFICTHIEF · VENUSAFFECT · FOGHORN · MATCHMAKER · ROADBED · MOONPENNY · WHIZBANG · CLOUDBASE · APLUS · MAILORDER · CROSSBONES · AMULET STELLAR · MVR · TINT · WEALTHYCLUSTER · IPGEOTRAP · MARINA · CLOUD · PLUS -
Excerpt of May 2008 NSA Afghanistan Dataflow Analysis [pdf]
The document outlines how the NSA’s Real Time Regional Gateway (RT-RG) in Afghanistan progressively expanded from late 2007 through 2008 to ingest massive amounts of GSM, VHF, HF, and telephony metadata from dozens of locations across Afghanistan, supplemented by third-party “dirtbox” collection from allied nations such as the Netherlands, Denmark, Norway, Germany, and Spain. It details the specific collection sites, data types, and timelines (SPIN 9, 10, 11, etc.), illustrating how partner contributions and multiple signals sources fed into the RT-RG system to enable real-time intelligence.
Programs: FOURCORNERS · FORTRESS · WARRIOR · ECHIDNA · CASSIEOPIA · KARUS · SOUNDER · SNICK · TEMPOWOLFE · CALYPSO · TATTERAN · AIRBORNE · LEAVEBEHIND · UDEAUS · NELEUS · CROSSHAIR · SSO · DRT · RIVET JOINT · VOICESAIL“Mission Creep: How the NSA’s Game-Changing Targeting System Built for Iraq and Afghanistan Ended Up on the Mexico Border” - The Intercept ↗ (2019-05-29) · “Norges hemmelige krig i Afghanistan” - NRK ↗ (2019-05-29) · Moltke, Henrik · Skille, Øyvind Bye · Døvik, Olav · Hansen, Ståle Ellingsgard · Fjørtoft, Martin -
GCHQ Analytic Cloud Challenges [pdf]
The 2012 GCHQ Analytic Cloud document outlines a sophisticated data processing infrastructure handling approximately 50 billion daily events across multiple sites, utilizing QFD databases and HADOOP clusters for various analytic capabilities targeting intelligence gathering and analysis. It describes architectural components, challenges in scaling and enabling analysts to cope with data complexity, and proposes future technology options including consolidation approaches and new cloud architectures to enhance capabilities.
Programs: EVOLVEDMUTANTBROTH · MUTANTBROTH · HARDASSOC · HRMAP · KARMAPOLICE · SOCIALANTHROPOID · AUTOASSOC · GEOFUSION · MVR · TERRAIN · SALAMANCA · BLACKHOLE · SAMUELPEPYS · BROKER · VAIL · GUIDINGLIGHT · FIRSTCONTACT · CLOUDYCOBRA · RUMOURMILL · EVERYASSOC · EVERYPOLICE · EVERYCREATURE · EVERYEAD · EVERYCIPHER · GREYFOX · SILVERFOX · GOLDENAXE · TRIBALCARNEM · PUBLICANEMONE · EPICFAIL · STERLINGMOTH · FOGHORN · LOOKINGGLASS · FIREENGINE · NGGEO · ALPHACENTAURI · CLOUDBASE · ACCUMULO · ASEM · SEM · HAKIM · STREAMING · HBASE · LAUREL · SKINNYCLOUD · HARDY · GCHQCORE -
HAPPY TRIGGER/LOVELY HORSE/Zool/TWO FACE – Open Source for Cyber Defence/Progress [pdf]
A TOP SECRET STRAP1 COMINT GCWiki page cataloguing GCHQ's open source intelligence gathering for cyber defence, listing approved and pending data sources fed into internal databases HAPPY TRIGGER, LOVELY HORSE, ZooL and TWO FACE, covering malware tracking, vulnerability intelligence, infrastructure mapping and domain monitoring.
Programs: HAPPYTRIGGER · LOVELYHORSE · TWOFACE · ZOOL · XKEYSCORE · POSITIVEPONY · NETPLATE · HIDDENSPOTLIGHT · SNORT · FOXTRAIL · REFRIEDCHICKEN · GEOFUSION · GLOBALSURGE · SHORTFALL · JTRIG · BIRDSEED · BIRDSTRIKE · MARBLEPOLLSPeople: Stewart, Joe -
NSA/GCHQ/CSEC Network Tradecraft Advancement Team [pdf]
The "NSA/GCHQ/CSEC Network Tradecraft Advancement Team (NTAT) 2010" document describes a collaborative mission between the US, UK, and Canada to synchronize and advance SIGINT techniques. The NTAT focused on "Network Analysis Tradecraft," specifically integrating analysis tools such as IRASCIABLE RABBIT and TOYGRIPPE into CSEC's OLYMPIA and developing standardized "use cases" for tracking mobile networks and virtual private networks (VPNs).
-
NWP NSA [pdf]
A January 2011 internal NSA newsletter from Yakima Research Station (YRS) covering the station's early history as the West Coast ECHELON/FROSTING satellite interception facility established in 1966 to collect INTELSAT communications, and its more recent expansion into cloud-based cyber threat discovery using FORNSAT and Special Source Operations data.
Programs: EARLYBIRD · FROSTING · TRANSIENT · ECHELON · JACKKNIFE · SAFEGUARD · STEAMS · CYBERQUEST · GINPENNANT · RUMBUCKET · CROSSBONES · SSO -
QFD BLACKHOLE Technology Behind INOC [pdf]
A UK SECRET STRAP2 COMINT ORCON GCHQ Applied Research presentation from March 2009 documenting the BLACK HOLE bulk data repository's scale and structure, and the suite of analytical tools — MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS and MARBLED GECKO — built on top of it to answer specific intelligence questions about online presence, browsing activity, forum usage and geolocation searches.
Programs: BLACKHOLE · MESME · NETEZZA · CLOUDS · MUTANTBROTH · KARMAPOLICE · INFINITEMONKEYS · MARBLEDGECKO · GOOB · AUTOASSOC · PSOUP · SOCIALANIMAL · SAMUELPEPYS · MEMORYHOLE · HRMAP · PSOUPALERT · LAUGHINGHYENA -
Real-Time Regional Gateway Cloud Analytics for Forward Users [pdf]
This document outlines the NSA’s Real-Time Regional Gateway (RT-RG) program, a global cloud-based signals intelligence system designed to provide near real-time data analysis and targeting support for military and intelligence operations. It details the transition to scalable cloud architectures such as GHOSTMACHINE and Cloudbase, enabling massive data retention, exploitation, advanced geolocation, and automated analytics for tracking high-value targets across multiple theaters, including Afghanistan, Iraq, and the Horn of Africa.
Programs: RT-RG · GOLDMINER · GEOT · AGENTLOGIC · SHARKFINN · SKS · PANOPTICON · DRT · CIDNE · JUGGERNAUT · LOPERS · TALIS · MATTERHORN · AIRHANDLER · LYCANTHROPE · VOICESAIL · RETURNSPRING · SORTINGLEAD · GHOSTMACHINE · GMROYALE · GMREGAL · GMGEIST · GMIDEA · GMTOTE · GMPEN · GMPAC · GMCALM · GMMATE · GMAPPLE · GMCAVE · GMPEACH · GMDEN · GMMOUNT · GMALAMO · GMLIGHT · GMBALTIC · GMPLACE · GMPARK · GMHALO · GMCARBON · DRAGONFIRE · DOMEEXPOSE · GIANTSQUID · MAINWAY · FURIOUSCHAINSAW · SHARKFURY · NIAGARAFILES · SHARKQUERY · SHARKDOCS · OCTAVE · UTT · BEDDOWN · CHEEKYMONKEY · LAYERCAKE · CYBERTRANS · GMSEMINOLE · GMGULF · GMZILLA · HLT · JEMA · NEVIS · SEDB · CLOUDBASE · MAPREDUCE · CASPORT · WAVELEGAL“Mission Creep: How the NSA’s Game-Changing Targeting System Built for Iraq and Afghanistan Ended Up on the Mexico Border” - The Intercept ↗ (2019-05-29) · “Norges hemmelige krig i Afghanistan” - NRK ↗ (2019-05-29) · Moltke, Henrik · Skille, Øyvind Bye · Døvik, Olav · Hansen, Ståle Ellingsgard · Fjørtoft, Martin -
XKeyscoreTabs XKS Development [pdf]
The document outlines the development and operational procedures for XKeyscore (XKS), detailing its types, upgrades, and the role of microplugins in enhancing data collection capabilities. It describes how different configurations manage data ingestion and the criteria used to filter and store communications for intelligence purposes.