Data Stored in BLACK HOLE

Open PDF
Creating agency: GCHQ Government Communications Headquarters (UK)
Document date: -
Pages: 2
Genre: wiki
Banner:
Classification: TOP SECRET STRAP 1 COMINT
Dissemination: -

This document describes BLACK HOLE, a TOP SECRET data repository that stores approximately 200GB per day of internet activity "events" such as webmail, chat, and browsing. It details the system's various datamarts, including MUTANT BROTH and KARMA POLICE, which collectively hold terabytes of data for intelligence analysis.

Long Description:
BLACK HOLE is a TOP SECRET STRAP1 COMINT data storage system that houses a vast collection of internet communication events. The data, primarily generated by B13B high-rate bearer processing modules, covers a wide range of activities including webmail, chat, FTP, internet browsing, website logins, social networking, and more. The classification of the information stored ranges from SECRET STRAP1 to TOP SECRET STRAP2 UK EYES ONLY, and its sensitivity can be increased by data enrichment processes, such as the addition of target details. The system ingests approximately 200GB of compressed data daily. This data feeds into a series of prototype datamarts with codenames like MUTANT BROTH, KARMA POLICE, and MEMORY HOLE. A table illustrates the scale of these databases, showing a current collective size of nearly 24 TB, with projections to grow to over 56 TB to accommodate a six-month retention period. The intended users of this massive data store include internal groups such as Applied Research and SIGINT Development for analysis and exploitation.

TOP SECRET STRAP1 COMINT
The maximum classification allowed on GCWiki is TOP SECRET STRAP1 COMINT. Click to report inappropriate content.
For GCWiki help contact: webteam Support page

Data stored in BLACK HOLE

From GCWiki
Jump to: navigation, search

This page describes the data which is stored in BLACK HOLE, where it comes from and how much of it there is. Some of the information is too highly classified to be contained here, or falls under the Applied Research intellectual property restrictions. Nonetheless, as much as possible is published here.

BLACK HOLE: The Knowledge

  • What it is
  • Where it came from
  • What is in it
    • What data it contains
    • How much data there is
  • How to use it
    • How to connect to it
    • How to find the file you want
  • Who uses it, and why
    • Applied Reseach uses
    • SIGINT Development uses
    • IPT uses
  • How it is built
    • Logical connectivity
    • Physical system
    • Security issues
  • What could happen to it
    • Applied Research enhancements
    • SIGINT Development intentions
    • Engineering interest
  • What we like about it
    • Good things
    • Things that could be better
    • Exploding myths
    • Wish list

v · d · e

Contents

  • 1 What data BLACK HOLE contains
    • 1.1 Types of data
    • 1.2 Classification of the data
  • 2 Where the data in BLACK HOLE comes from
  • 3 How much data there is
    • 3.1 Volumes in BLACK HOLE
    • 3.2 Volumes in the datamarts

[edit] What data BLACK HOLE contains

[edit] Types of data

The events created cover webmail, email transfers, ftp, chat, internet browsing, website logins, vbulletin web fora, web cams, gaming, social networking -- and the list is growing.

The data is primarily generated by B13B high-rate bearer processing modules, and so the exact types depend on which modules are running at the time.

[edit] Classification of the data

The classification of the data in BLACK HOLE ranges from SECRET STRAP1 to TOP SECRET STRAP2 UK EYES ONLY. Most data should be covered by TOP SECRET STRAP2 and there may, inadvertently, be ECI data (but this is not expected, as we are not actively doing things covered by ECI). The data includes fields which have been produced by enrichment (such as, for example, target details) and this can increase the classification.

[edit] Where the data in BLACK HOLE comes from

Take a look under the covers of BLACK HOLE to find out where the data comes from.

[edit] How much data there is

[edit] Volumes in BLACK HOLE

The total amount of (compressed) data entering BLACK HOLE is approximately 200GB/day.

[edit] Volumes in the datamarts

Prototype Current retention time / months Current database size / TB Estimated size for six months retention / TB Current database size / billion rows Estimated database size for 6 months retention / billion rows
MUTANT BROTH 4 7.7 11.55 21.7 32.55
KARMA POLICE 3 6.8 13.6 17.8 35.6
MEMORY HOLE 0.5 0.6 7.2 2.2 26.4
AUTOASSOC 6 0.1 0.1 11 11
SOCIAL ANIMAL 1 1.6 9.6 1 6
HRMAP 3 7 14 16 32
INFINITE MONKEYS 6 0.02 0.02 0.2 0.2
Total 23.82 56.07 69.9 143.75

Please note - all of these values are approximate.

POC: [REDACTED] (mail Phonebook Find)

Retrieved from [REDACTED] Data stored in BLACK HOLE"
Category: ICTR BLACK HOLE