TINT External July 2009

Open PDF
Creating agency: GCHQ Government Communications Headquarters (UK)
Document date: 2009-01-01
Pages: 1
Genre: other
Classification: -
Dissemination: -

A GCHQ TINT external document from July 2009 describing the BLACKHOLE bulk metadata repository — designed to export unselected metadata in SLR format to enable higher-order analytics — noting anticipated very high volumes expected to exceed MAILORDER transport capacity and proposing a direct connection instead.

Long Description:
This is a fragment from a GCHQ TINT external document dated July 2009. The excerpt is a single entry — item 14 — from what appears to be a larger catalogue of data systems or experimental capabilities, describing BLACKHOLE. BLACKHOLE is defined by purpose as a repository for exporting bulk unselected metadata in SLR format, enabling higher-order analytics to be carried out on the collected data. The distinction between "bulk, unselected" and targeted collection is significant — BLACKHOLE explicitly receives data that has not been filtered against specific selectors, meaning it ingests the full firehose of intercepted internet metadata rather than only records matching known targets. The data flow is described as to and from GCHQ, with a direct connection proposed. Anticipated volume is classified as "very high" — high enough that the standard MAILORDER transport mechanism is expected to be insufficient to handle it. This is consistent with the Karma Police article's description of BLACKHOLE storing over 1.1 trillion events between August 2007 and March 2009, growing to 10 billion new entries per day and eventually 50 billion per day by 2012. The dependent experiments section references "SLRs at Scale" — suggesting the SLR format and its scaling properties were an active research area at time of writing. The document connects directly to the TDI Introduction presentation processed immediately before, which describes MUTANT BROTH, KARMA POLICE and the other bulk analytics databases as drawing from the BLACKHOLE repository — confirming BLACKHOLE's role as the central raw data store from which all downstream TDI analytics are derived.
Programs:
BLACKHOLE, MAILORDER

People mentioned:
-

Countries:
-

Tags:
TINT, SLR, bulk metadata, unselected collection, higher-order analytics, internet surveillance, metadata repositories

Identifier:
tint-external-july-2009

PDF SHA-256:
e5ff79eb7ab6b7cfbfbcb9255262b3734adffddc58119872b1597840fa76461b
  1. BLACKHOLE

Purpose
For exporting bulk, unselected metadata (in SLR format) to a repository where higher-order analytics can be carried out.

Data To/From
GCHQ

Connection To/From
GCHQ

Anticipated Volume
Very High

Dependant Experiments

  • SLRs at Scale

Transport
TBD. Volumes expected to be too high for MAILORDER. Direct connection proposed.

Notes