Data Stored in BLACK HOLE
Open PDFDocument date: -
Pages: 2
Genre: wiki
Banner: TOP SECRET STRAP1 COMINT
Classification: TOP SECRET STRAP 1 COMINT
Dissemination: -
This document describes BLACK HOLE, a TOP SECRET data repository that stores approximately 200GB per day of internet activity "events" such as webmail, chat, and browsing. It details the system's various datamarts, including MUTANT BROTH and KARMA POLICE, which collectively hold terabytes of data for intelligence analysis.
BLACKHOLE, B13B, MUTANTBROTH, KARMAPOLICE, MEMORYHOLE, AUTOASSOC, SOCIALANIMAL, HRMAP, INFINITEMONKEYS
People mentioned:
-
Countries:
-
Tags:
data repository, surveillance infrastructure, datamart, mass surveillance, internet surveillance, bearers, data retention, metadata repositories, content database, data volume, webmail, chat, FTP, browsing history, social networks, data enrichment, Applied Research, SIGDEV, ICTR
Identifier:
data-stored-in-black-hole
PDF SHA-256:
1065755fb59ae4e88aa10d724865a9d1d045d085752417dfd78bc092ad45b5a6
My notes
TOP SECRET STRAP1 COMINT
The maximum classification allowed on GCWiki is TOP SECRET STRAP1 COMINT. Click to report inappropriate content.
For GCWiki help contact: webteam Support page
Data stored in BLACK HOLE
From GCWiki
Jump to: navigation, search
This page describes the data which is stored in BLACK HOLE, where it comes from and how much of it there is. Some of the information is too highly classified to be contained here, or falls under the Applied Research intellectual property restrictions. Nonetheless, as much as possible is published here.
BLACK HOLE: The Knowledge
- What it is
- Where it came from
- What is in it
- What data it contains
- How much data there is
- How to use it
- How to connect to it
- How to find the file you want
- Who uses it, and why
- Applied Reseach uses
- SIGINT Development uses
- IPT uses
- How it is built
- Logical connectivity
- Physical system
- Security issues
- What could happen to it
- Applied Research enhancements
- SIGINT Development intentions
- Engineering interest
- What we like about it
- Good things
- Things that could be better
- Exploding myths
- Wish list
v · d · e
Contents
- 1 What data BLACK HOLE contains
- 1.1 Types of data
- 1.2 Classification of the data
- 2 Where the data in BLACK HOLE comes from
- 3 How much data there is
- 3.1 Volumes in BLACK HOLE
- 3.2 Volumes in the datamarts
[edit] What data BLACK HOLE contains
[edit] Types of data
The events created cover webmail, email transfers, ftp, chat, internet browsing, website logins, vbulletin web fora, web cams, gaming, social networking -- and the list is growing.
The data is primarily generated by B13B high-rate bearer processing modules, and so the exact types depend on which modules are running at the time.
[edit] Classification of the data
The classification of the data in BLACK HOLE ranges from SECRET STRAP1 to TOP SECRET STRAP2 UK EYES ONLY. Most data should be covered by TOP SECRET STRAP2 and there may, inadvertently, be ECI data (but this is not expected, as we are not actively doing things covered by ECI). The data includes fields which have been produced by enrichment (such as, for example, target details) and this can increase the classification.
[edit] Where the data in BLACK HOLE comes from
Take a look under the covers of BLACK HOLE to find out where the data comes from.
[edit] How much data there is
[edit] Volumes in BLACK HOLE
The total amount of (compressed) data entering BLACK HOLE is approximately 200GB/day.
[edit] Volumes in the datamarts
| Prototype | Current retention time / months | Current database size / TB | Estimated size for six months retention / TB | Current database size / billion rows | Estimated database size for 6 months retention / billion rows |
|---|---|---|---|---|---|
| MUTANT BROTH | 4 | 7.7 | 11.55 | 21.7 | 32.55 |
| KARMA POLICE | 3 | 6.8 | 13.6 | 17.8 | 35.6 |
| MEMORY HOLE | 0.5 | 0.6 | 7.2 | 2.2 | 26.4 |
| AUTOASSOC | 6 | 0.1 | 0.1 | 11 | 11 |
| SOCIAL ANIMAL | 1 | 1.6 | 9.6 | 1 | 6 |
| HRMAP | 3 | 7 | 14 | 16 | 32 |
| INFINITE MONKEYS | 6 | 0.02 | 0.02 | 0.2 | 0.2 |
| Total | 23.82 | 56.07 | 69.9 | 143.75 |
Please note - all of these values are approximate.
POC: [REDACTED] (mail Phonebook Find)
Retrieved from [REDACTED] Data stored in BLACK HOLE"
Category: ICTR BLACK HOLE
Published in the Same Report
(10 of 28 documents)
-
Supporting Internet Operations - Special Source Access [pdf]
In this document GCHQ outlines its plans for expanding internet operations, focusing on increasing access to 10G bearers and enhancing its intercept and egress capabilities by 2011. The agency highlighted that “25 percent of all internet traffic” transits the U.K. through approximately 1,600 different 10-gigabit cables, and it noted that it could “survey the majority of the 1,600” to “select the most valuable to switch into our processing systems.”
-
Event (SIGINT) [pdf]
5 p.This document explains that within GCHQ and NSA SIGINT operations, an “Event” refers to metadata about observable communications or user actions rather than message content, categorized into types like Communications, Presence, Social, and Geo-reference Events. These events are collected, stored in various databases, and analyzed through convergence and correlation processes, with standardized formats defined by Interface Control Documents such as INTERSTELLAR DUST (IDUST) and Actor Action (AA).
Programs: INTERSTELLAR DUST · HAUSTORIUM · SOCIAL ANTHROPOID · ACTOR-ACTION · SAMREF · GEOFUSION · HARD ASSOC · SALAMANCA · ASSOCIATION · BANYAN · MAINWAY · FASCIA · CULTWEAVE II · VOICESAIL · TEEDALE · PILBEAM · MUTANTBROTH · AUTOASSOC · MARBLEDGECKO · KARMAPOLICE · HRMAP · MEMORYHOLE · INFINITEMONKEYS · SOCIALANIMAL · AUTOTDI · SAMUELPEPYS · Mobile Applications Project · NGE · PPF · SLR -
Access: Vision 2013 [pdf]
This July 2010 GCHQ strategic document outlines the agency's vision for 2013 to create "the world's biggest SIGINT engine" with access to 800 10-gigabit connections (up from existing capacity), expanding overseas operations to 60% while reducing UK-based to 40%, and developing new capabilities including computer network exploitation (CNE), exfiltration, beaconry, geolocation, and collection of "Internet Pocket Litter" like cookies and browsing data. The strategy emphasizes "scale and pace," "balance," and "agility" while claiming to operate within legal frameworks of "necessity, proportionality and authorisation," and highlights partnerships with second and third parties to access "intractable and hard targets."
Marking: SECRET -
BLACK HOLE ANALYTICS, ADD/SD briefing, September 2009 [pdf]
This September 2009 GCHQ briefing describes the BLACK HOLE analytics infrastructure and Next Generation Events (NGE) project, which planned to scale from ~8 10-gigabit connections to ~200 10Gs by March 2011, storing intercepted data in massive flat-file repositories at Benhall and Bude before loading it into "Question Focused Dataset" (QFD) databases that power surveillance tools like MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS, and others. The document outlines how "probes" extract Target Detection Identifiers (TDIs) from intercepted data feeds, store them in BLACK HOLE repositories, then enable analysts to profile targets' online activities, create social networks, investigate websites and forums, track web searches and Google Earth usage, and perform large-scale data mining across internet and telephony data.
Programs: BLACKHOLE · MUTANTBROTH · AUTOASSOC · INFINITEMONKEYS · KARMAPOLICE · SOCIALANIMAL · HRMAP · MEMORYHOLE · MARBLEDGECKO · HARDASSOC · SAMUELPEPYS · BLAZINGSADDLES · TELLURIAN · BUFFER · DISTILLERY · LOOKINGGLASS · NGEPeople: bin Laden, Osama -
Report on Architectural Risk 2012 - Summary [pdf]
This 2012 GCHQ report identifies that a historical under-investment in IT architecture has created significant risk, especially with new challenges like massive data growth, cyber defense, and increased partnering. The document recommends immediate, prioritized investment in big data strategies, integrated analytics frameworks, and security services to address the strain on the system.
Programs: BLACKHOLE -
Blazing Saddles [pdf]
This document describes BLAZING SADDLES, a GCHQ program designed to massively scale the storage and analysis of bulk internet surveillance data through multiple specialized databases that track website visits, online identities, communications patterns, and geolocation queries across the global internet population.
Programs: BLAZINGSADDLES · BLACKHOLE · SOCIALANTHROPOID · AUTOASSOC · HRMAP · INFINITEMONKEYS · KARMAPOLICE · MARBLEDGECKO · MUTANTBROTH · MEMORYHOLE · SAMUELPEPYS · SOCIALANIMAL · HAUSTORIUM · GOLDENAXE · NGE -
Broadcast/Internet Radio Exploitation and Analysis, 6th November 2009, GCHQ [pdf]
This GCHQ report analyzes 6.68 million internet radio listening events collected over three months in 2009, detailing global broadcasting patterns, user behaviors, and specifically examining Islamic radio stations and Pakistan-UK connections for potential radicalization intelligence, while recommending integration with bulk surveillance databases like KARMA POLICE for enhanced user profiling.
-
ICTR Cloud Efforts [pdf]
A GCHQ presentation details how the agency developed cloud-based big data analytics to achieve "population scale" surveillance—processing trillions of internet events to build geographic and behavioral profiles on every online identifier worldwide, enabling discovery of unknown targets through pattern analysis rather than traditional keyword searches.
Programs: EVERYASSOC · PROBABILITYCLOUD · SILVERLINING · CULTWEAVE · KARMAPOLICE · EVERYPOLICE · AWKWARDTURTLE · GORDIANKNOT · SANDIA · BLOODHOUND · FIVEALIVE · NGE · MAPREDUCE -
Content-Metadata Matrix [pdf]
GCHQ's internal taxonomy sorts intercepted data into "content" or "metadata" to determine what can be searched without individual warrants—email subject lines, full URLs, and buddy lists fall on opposite sides of the line, enabling analysts to mine massive troves of IP addresses, phone numbers, location data, and browsing history with just a few keystrokes and a brief justification.
-
Cyber Defence Operations Legal Policy [pdf]
GCHQ's internal wiki laying out the legal plumbing for its Cyber Defence Operations (formerly NDIST)—detailing warrant types (RIPA, ISA, CNE), data retention schedules, targeting rules for UK/Five Eyes infrastructure, XKEYSCORE query protocols, and the bureaucratic hoops analysts must jump through to search attacker IPs, fingerprint UK networks, and share malware intelligence without tripping over location/nationality safeguards or commercial sensitivities.
Programs: JTRIG · SPAY · GORDIANKNOT · CHORDAL · DISCOVER · EVENTS · CASPORT · XKEYSCORE · CONTENT CLOUD · SQUEAL · MIRAGE · MUTANTBROTH · ANXIOUS · HALTERHITCH · BROADOAK · DONKEYKONG · ZOOL · PALANTIR · OVERLINK · RT · GATEKEEPER · IRONHAND · HARUSPEX · B3M · LOOKINGGLASS · GOLDENEYE · BLAZINGSADDLES · MOONRAKER · TRITON
Related Documents
-
BLACK HOLE ANALYTICS, ADD/SD briefing, September 2009 [pdf]
This September 2009 GCHQ briefing describes the BLACK HOLE analytics infrastructure and Next Generation Events (NGE) project, which planned to scale from ~8 10-gigabit connections to ~200 10Gs by March 2011, storing intercepted data in massive flat-file repositories at Benhall and Bude before loading it into "Question Focused Dataset" (QFD) databases that power surveillance tools like MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS, and others. The document outlines how "probes" extract Target Detection Identifiers (TDIs) from intercepted data feeds, store them in BLACK HOLE repositories, then enable analysts to profile targets' online activities, create social networks, investigate websites and forums, track web searches and Google Earth usage, and perform large-scale data mining across internet and telephony data.
Programs: BLACKHOLE · MUTANTBROTH · AUTOASSOC · INFINITEMONKEYS · KARMAPOLICE · SOCIALANIMAL · HRMAP · MEMORYHOLE · MARBLEDGECKO · HARDASSOC · SAMUELPEPYS · BLAZINGSADDLES · TELLURIAN · BUFFER · DISTILLERY · LOOKINGGLASS · NGEPeople: bin Laden, Osama -
Blazing Saddles [pdf]
This document describes BLAZING SADDLES, a GCHQ program designed to massively scale the storage and analysis of bulk internet surveillance data through multiple specialized databases that track website visits, online identities, communications patterns, and geolocation queries across the global internet population.
Programs: BLAZINGSADDLES · BLACKHOLE · SOCIALANTHROPOID · AUTOASSOC · HRMAP · INFINITEMONKEYS · KARMAPOLICE · MARBLEDGECKO · MUTANTBROTH · MEMORYHOLE · SAMUELPEPYS · SOCIALANIMAL · HAUSTORIUM · GOLDENAXE · NGE -
QFD BLACKHOLE Technology Behind INOC [pdf]
A UK SECRET STRAP2 COMINT ORCON GCHQ Applied Research presentation from March 2009 documenting the BLACK HOLE bulk data repository's scale and structure, and the suite of analytical tools — MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS and MARBLED GECKO — built on top of it to answer specific intelligence questions about online presence, browsing activity, forum usage and geolocation searches.
Programs: BLACKHOLE · MESME · NETEZZA · CLOUDS · MUTANTBROTH · KARMAPOLICE · INFINITEMONKEYS · MARBLEDGECKO · GOOB · AUTOASSOC · PSOUP · SOCIALANIMAL · SAMUELPEPYS · MEMORYHOLE · HRMAP · PSOUPALERT · LAUGHINGHYENA -
Demystifying NGE Rock Ridge [pdf]
This presentation outlines the ROCK RIDGE project, which delivers a suite of "Question Focused Databases" (QFDs) like MUTANT BROTH and KARMA POLICE to analyze massive internet event data. These specialized databases are designed to answer specific intelligence questions and are developed using the BLACK HOLE data repository, a key component of the ROUGH DIAMOND program.
Programs: ROCKRIDGE · ROUGHDIAMONG · BLACKHOLE · HAUSTORIUM · HRMAP · MUTANTBROTH · KARMAPOLICE · GOOBZS · INFINITEMONKEYS · AUTOASSOC · SOCIALANIMAL · MARBLEDGECKO · MEMORYHOLE · SAMUELPEPYS · SALAMANCA · MONOPOLY · RADIUS · HARBOURPILOT · LOOKINGGLASS · FIRESTORM · SUPERDRAKE · BLAZINGSADDLES · BLUESHIFT · SUPPORTINGINO · TERRAINS · GEOFUSION · NGEPeople: Lobban, Iain -
Target Detection Identifiers [pdf]
A GCHQ presentation from March 2009 introducing the concept of Target Detection Identifiers — the "fundamental atom of the Internet age" — describing how cookies and other unique persistent identifiers extracted from bulk internet traffic at 2,500 per second are used to build the surveillance databases, with a live AutoTDI discovery table showing cookies extracted from Facebook, Yahoo, Google, Reuters, YouPorn, Bebo and other sites.
Programs: MUTANTBROTH · AUTOASSOC · KARMAPOLICE · INFINITEMONKEYS · SOCIALANIMAL · MEMORYHOLE · MARBLEDGECKO · HRMAP · AUTOTDI