Tag: data mining 15 documents
Search documents matching “data mining” →Free-text subjects the curators added. Unlike the other axes these are not necessarily from the documents' own vocabulary, so they are the loosest thing to browse by (and the most forgiving).
Graphs
By Banner
By Classification
Refine (combine with another filter, via search)
Sort: Title
Documents
-
AND THEY SAID TO THE TITANS: « WATCH OUT OLYMPIANS IN THE HOUSE! » [pdf]
This Top Secret CSEC “Olympia” document details a Canadian intelligence program that analyzed and tracked the Brazilian Ministry of Mines and Energy’s telecommunications, mapping IPs, servers, proxies, and mobile devices while automating data collection and network analysis to develop actionable intelligence. It outlines tools, workflows, and contacts used for surveillance, though the presentation does not explain the specific purpose of the operation.
Programs: OLYMPIA · QUOVA · PROMETHEUS · PEITHO · SLINGSHOT · LEVITATE · STRATOS · EVILOLIVE · GCHQ Geofusion · ATLAS · HYPERION · ATHENA · DANAUS · TIDALSURGE · BLACKPEARL · MARINA · STARSEARCH · PEPPERBOX · TRITON · PACKAGEDGOODS · ARK · TOYGRIPPE · FRIARTUCK · MASTERSHAKE · COEUS · STALKER · MAINWAY · FASCIA · SEDB · OCTSKYWARD · FASTBAT · TWINSERPENT · EONBLUE · DISHFIRE · GNDB“Read a CSEC document that was first acquired by Edward Snowden” - The Globe and Mail ↗ (2013-11-30) · “Ministério de Minas e Energia está na mira de espiões americanos e canadenses” - O Globo ↗ (2013-10-06) · “No Place to Hide” - Metropolitan Books ↗ (2014-05-13) · Freeze, Colin · Bridi, Sônia · Greenwald, Glenn -
Cyber Defence Operations Legal Policy [pdf]
GCHQ's internal wiki laying out the legal plumbing for its Cyber Defence Operations (formerly NDIST)—detailing warrant types (RIPA, ISA, CNE), data retention schedules, targeting rules for UK/Five Eyes infrastructure, XKEYSCORE query protocols, and the bureaucratic hoops analysts must jump through to search attacker IPs, fingerprint UK networks, and share malware intelligence without tripping over location/nationality safeguards or commercial sensitivities.
Programs: JTRIG · SPAY · GORDIANKNOT · CHORDAL · DISCOVER · EVENTS · CASPORT · XKEYSCORE · CONTENT CLOUD · SQUEAL · MIRAGE · MUTANTBROTH · ANXIOUS · HALTERHITCH · BROADOAK · DONKEYKONG · ZOOL · PALANTIR · OVERLINK · RT · GATEKEEPER · IRONHAND · HARUSPEX · B3M · LOOKINGGLASS · GOLDENEYE · BLAZINGSADDLES · MOONRAKER · TRITON -
Cyber Network Defence R&D Activities [pdf]
A September 2010 CSEC briefing to the Five Eyes 5IARC conference on cyber defence R&D, giving the full PONY EXPRESS email attachment scanning architecture and the PHOTONIC PRISM sensor, with speaker notes in which the presenter concedes CSEC does little research of its own.
Marking: SECRETPrograms: PHOTONICPRISM · PONYEXPRESS · SLIPSTREAM · POPQUIZ · 8BALL · SNORT · POPEYESEAR · FLOWPONY · SMTPPONY · MAILPONY · METAPONY · SCANPONY · AGGPONY · SYNCPONY · POOLTABLE · POOLCUE · CORNERPOCKET · TONTO · MULASSES · MULVAL · ASSETRANK · JNDMS · ARMOUR · SWAD · MINISSL · SSLFUZZER · N3C · N3 · CTEC · DCITS · 5IARC -
Demystifying NGE Rock Ridge [pdf]
This presentation outlines the ROCK RIDGE project, which delivers a suite of "Question Focused Databases" (QFDs) like MUTANT BROTH and KARMA POLICE to analyze massive internet event data. These specialized databases are designed to answer specific intelligence questions and are developed using the BLACK HOLE data repository, a key component of the ROUGH DIAMOND program.
Programs: ROCKRIDGE · ROUGHDIAMONG · BLACKHOLE · HAUSTORIUM · HRMAP · MUTANTBROTH · KARMAPOLICE · GOOBZS · INFINITEMONKEYS · AUTOASSOC · SOCIALANIMAL · MARBLEDGECKO · MEMORYHOLE · SAMUELPEPYS · SALAMANCA · MONOPOLY · RADIUS · HARBOURPILOT · LOOKINGGLASS · FIRESTORM · SUPERDRAKE · BLAZINGSADDLES · BLUESHIFT · SUPPORTINGINO · TERRAINS · GEOFUSION · NGEPeople: Lobban, Iain -
GCHQ Analytic Cloud Challenges [pdf]
The 2012 GCHQ Analytic Cloud document outlines a sophisticated data processing infrastructure handling approximately 50 billion daily events across multiple sites, utilizing QFD databases and HADOOP clusters for various analytic capabilities targeting intelligence gathering and analysis. It describes architectural components, challenges in scaling and enabling analysts to cope with data complexity, and proposes future technology options including consolidation approaches and new cloud architectures to enhance capabilities.
Programs: EVOLVEDMUTANTBROTH · MUTANTBROTH · HARDASSOC · HRMAP · KARMAPOLICE · SOCIALANTHROPOID · AUTOASSOC · GEOFUSION · MVR · TERRAIN · SALAMANCA · BLACKHOLE · SAMUELPEPYS · BROKER · VAIL · GUIDINGLIGHT · FIRSTCONTACT · CLOUDYCOBRA · RUMOURMILL · EVERYASSOC · EVERYPOLICE · EVERYCREATURE · EVERYEAD · EVERYCIPHER · GREYFOX · SILVERFOX · GOLDENAXE · TRIBALCARNEM · PUBLICANEMONE · EPICFAIL · STERLINGMOTH · FOGHORN · LOOKINGGLASS · FIREENGINE · NGGEO · ALPHACENTAURI · CLOUDBASE · ACCUMULO · ASEM · SEM · HAKIM · STREAMING · HBASE · LAUREL · SKINNYCLOUD · HARDY · GCHQCORE -
GCHQ VisWeek 2008 Conference Report [pdf]
This GCHQ document summarizes key findings from the VisWeek 2008 visualization conference in Ohio, highlighting techniques and tools relevant to SIGINT analysis including collaborative analysis capabilities, graph visualization methods, and the Palantir Government platform. The report recommends incorporating features like analyst notebooks for tracking investigations, collaborative workspaces for multiple analysts, and various graph layout algorithms into GCHQ's MONTE VISTA visualization system.
Programs: MONTE VISTA · MAMBA · CHARTBREAKER · PIGS EAR · KNIME · RAPTOR · BROAD OAK · HAUSTORIUM · SALAMANCA · IIB · INFOVIS · PALANTIR -
Getting Leads: GEO Mining SIDtoday [pdf]
Description of an approach to sifting through large amounts of phone call metadata to find leads for new surveillance target called "GEO Mining." The author gives an example of searching for heroin-production labs in Afghanistan using GEO Mining, by filtering all phone calls in Afghanistan down to calls made near known heroin-processing labs. In this case, "The top 20 handsets were then tasked for collection, and at least one has produced reportable intelligence."
People: Brown, PaulThe Intercept ↗ (2017-04-24) -
ICTR Cloud Efforts [pdf]
A GCHQ presentation details how the agency developed cloud-based big data analytics to achieve "population scale" surveillance—processing trillions of internet events to build geographic and behavioral profiles on every online identifier worldwide, enabling discovery of unknown targets through pattern analysis rather than traditional keyword searches.
Programs: EVERYASSOC · PROBABILITYCLOUD · SILVERLINING · CULTWEAVE · KARMAPOLICE · EVERYPOLICE · AWKWARDTURTLE · GORDIANKNOT · SANDIA · BLOODHOUND · FIVEALIVE · NGE · MAPREDUCE -
LEVITATION and the FFU Hypothesis [pdf]
This document describes LEVITATION, a top-secret Canadian Communications Security Establishment (CSE) project that monitors millions of daily file uploads and downloads from free file-sharing sites to identify potential extremist activity. It details the program’s data collection methods, analytical workflow, use of partner databases, and examples of successful discoveries related to terrorist materials and hostage videos.
-
New Tool Identifies Non-Productive Phone Numbers SIDtoday [pdf]
A new tool that helps weed out phone numbers that are tasked for collection but are providing no intelligence value, based on reports showing how much of "each phone number's traffic has been reviewed, deemed relevant, and reported on." The idea is to avoid needlessly "consuming precious SIGINT resources." The tool is part of a larger project called "RAINMAN" to automatically prioritize traffic scanned by language analysts.
People: Curet, NormanThe Intercept ↗ (2017-09-13) -
New Tool Identifies Non-Productive Phone Numbers (repost) SIDtoday [pdf]
Repost of article on a new tool that helps weed out phone numbers that are tasked for collection but are providing no intelligence value, based on reports showing how much of "each phone number's traffic has been reviewed, deemed relevant, and reported on." The idea is to avoid needlessly "consuming precious SIGINT resources." The tool is part of a larger project called "RAINMAN" to automatically prioritize traffic scanned by language analysts.
People: Curet, NormanThe Intercept ↗ (2017-09-13) -
Operational Legalities – GCHQ Powerpoint Presentation [pdf]
GCHQ’s “Operational Legalities” deck is an internal compliance playbook that translates the UK’s surveillance laws into day‑to‑day SIGINT procedure—drilling staff on authorisation, necessity, and proportionality, and walking through warrants, targeting justifications, database queries, dissemination, and oversight. Half slides and half speaker notes, it shows how analysts are trained to document, audit, and defend operations under ISA, RIPA, HRA, and WTA while navigating edge cases like location, nationality, second‑party constraints, and “indirect targeting.”
-
QFD BLACKHOLE Technology Behind INOC [pdf]
A UK SECRET STRAP2 COMINT ORCON GCHQ Applied Research presentation from March 2009 documenting the BLACK HOLE bulk data repository's scale and structure, and the suite of analytical tools — MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS and MARBLED GECKO — built on top of it to answer specific intelligence questions about online presence, browsing activity, forum usage and geolocation searches.
Programs: BLACKHOLE · MESME · NETEZZA · CLOUDS · MUTANTBROTH · KARMAPOLICE · INFINITEMONKEYS · MARBLEDGECKO · GOOB · AUTOASSOC · PSOUP · SOCIALANIMAL · SAMUELPEPYS · MEMORYHOLE · HRMAP · PSOUPALERT · LAUGHINGHYENA -
Seeing into the Global SIGINT System - A View from the Field SIDtoday [pdf]
The Signals Intelligence Directorate's technical director sums up his thoughts on the visit to the expeditionary signals intelligence units in Iraq, Qatar, and Afghanistan.
People: Hurry, DaveThe Intercept ↗ (2017-09-13) -
Sensitive Targeting Authorisation [pdf]
A GCHQ internal form template for Sensitive Targeting Authorisation (STA), used to obtain approval for surveillance targeting of individuals or organisations with sensitive nationalities or in sensitive locations, covering both real-time tasking and historical database datamining, with separate authorisation tracks for overseas targets and UK-based targets.