Content-Metadata Matrix
Open PDFDocument date: 2010-01-20
Pages: 3
Genre: guide
Banner: UK SECRET STRAP1 COMINT AUS/CAN/NZ/UK/US EYES ONLY ORCON
Classification: UK SECRET STRAP 1 COMINT
Dissemination: AUS,CAN,NZ,UK,US,ORCON
GCHQ's internal taxonomy sorts intercepted data into "content" or "metadata" to determine what can be searched without individual warrants—email subject lines, full URLs, and buddy lists fall on opposite sides of the line, enabling analysts to mine massive troves of IP addresses, phone numbers, location data, and browsing history with just a few keystrokes and a brief justification.
-
People mentioned:
-
Countries:
-
Tags:
policy, metadata classification, content classification, RIPA, emails, email headers, email subjects, IP addresses, data routing, phone numbers, IMSI, IMEI, cookies, machine IDs, buddy lists, contact lists, keylogger, voice communications, SMS, chat room logs, webchats, instant messaging, authentication, passwords, speaker identification, language identification, attachment metadata, DTMF, call detail records, location data, billing information, bulk collection, browsing history, cryptographic key, draft messages, social networks, address books
Identifier:
content-metadata-matrix
PDF SHA-256:
9f5cd72ee14c30027f8cf32c53803d1d4737657d74a2fd8d2303a2a049d8b8b4
My notes
UK SECRET STRAP1
COMINT AUS/CAN/NZ/UK/US EYES ONLY ORCON
CONTENT OR METADATA?
Categorisation of aspects of intercepted communications - GCHQ policy guidance.
POCS: [REDACTED] of GCHQ OPP-LEG [REDACTED] @gchq.gov.uk) and [REDACTED] of GCHQ OPP-HQ [REDACTED] @gchq.gov.uk).
Date last verified: 20 January 2010 by [REDACTED] Version 2.
| Data type sorted by data class (column B; defined below) | Class (see below) | Content (c) or Metadata (m) | Note: For convenience, the terms "Metadata", "Events" and "Communications Data" are often used interchangeably, although only the last is defined in UK law (RIPA). |
|---|---|---|---|
| attachment to e-mail, eg routing diagram, picture, video | a | c | |
| e-mail address in the body of a message | a | c | |
| name of a file attached to an e-mail | a | c | |
| authentication data to a communications service: login ID, userid, password | ac | m | unless sent inside the body of a communication |
| an e-mail inside a message | c | c | |
| bulletin board posting | c | c | |
| chat-room discussions | c | c | |
| content of a voice call | c | c | |
| content of an e-mail | c | c | |
| cookie as a whole (some elements are listed separately below) | c | c | |
| DTMF data, as opposed to dialling | c | c | |
| keystroke logs | c | c | |
| search results | c | c | |
| search strings | c | c | |
| SMS or IM text | c | c | |
| video | c | c | |
| voice mail boxes | c | c | |
| web cam transmissions | c | c | |
| web forms filled in by people | c | c | |
| 'to' 'from', 'cc', 'bcc', and 'fwd' lines within e-mail header | ca | m | |
| chat aliases, chat handles and other related or similar identifiers | ca | m | |
| e-mail address from a cookie sent to set up a communication channel | ca | m | |
| IMEI data | ca | m | |
| IMSI caller ID | ca | m | |
| IP addresses of the computers sending and receiving the message | ca | m | |
| machine ID extracted from cookies (eg Yahoo B cookies) | ca | m |
UK SECRET STRAP1
COMINT AUS/CAN/NZ/UK/US EYES ONLY ORCON
UK SECRET STRAP1
COMINT AUS/CAN/NZ/UK/US EYES ONLY ORCON
| Data type sorted by data class (column B; defined below) | Class (see below) | Content (c) or Metadata (m) | Note: For convenience, the terms "Metadata", "Events" and "Communications Data" are often used interchangeably, although only the last is defined in UK law (RIPA). |
|---|---|---|---|
| personal IDs extracted from cookies, mail headers, chat sessions or buddy lists | ca | m | |
| telephone number | ca | m | |
| content summarization | cdd | cdd | |
| file type of an attachment to an email | cdd | cdd | |
| language or language fingerprinting | cdd | cdd | |
| speaker's gender | cdd | cdd | |
| speaker's ID | cdd | cdd | |
| e-mail subject line | ch | c | |
| dialling, signalling, routing, addressing or signalling information | d | m | |
| call charge records, including info about the time and length of a call | d | m | |
| Calling Line Identification, including numbers dialled | d | m | |
| details of routers and IP addresses that have handled the message | d | m | |
| DTMF dialling (not data) | d | m | |
| location of parties to a communication, not derived from content | d | m | |
| logs of visitors to chat rooms including how often they have visited/posted | d | m | |
| network management, eg billing, authentication or tracking of communicants | d | m | |
| start and finish time of an internet session or phone call | d | m | |
| status of chat sites, ie whether they are active and how many participants | d | m | |
| URLs up to and including the domain name | d | m | |
| session initiation protocols | d | m | |
| website registers including owner details; assume not UKUSA owned/registered | d | m | |
| creation of/access to draft message | m | m | N.B. the contents of a draft message are Content. |
| history of websites browsed (full URLs) | m | c | |
| buddy lists for web mail, instant messenger or social networking | m | m | |
| folders used to organise e-mails | m | m | |
| address books or contacts lists for web mail etc | m | c | |
| crypto keys | m | c | |
| password to internet or telephony services other than communications services | m | c | |
| URLs beyond the domain name, ie one that may include search terms | m | c |
UK SECRET STRAP1
COMINT AUS/CAN/NZ/UK/US EYES ONLY ORCON
UK SECRET STRAP1
COMINT AUS/CAN/NZ/UK/US EYES ONLY ORCON
| Data type sorted by data class (column B; defined below) | Class (see below) | Content (c) or Metadata (m) | Note: For convenience, the terms "Metadata", "Events" and "Communications Data" are often used interchangeably, although only the last is defined in UK law (RIPA). |
|---|
Coding of data classes
- a - attachment
- ac - authentication of communications
- c - content
- ca - communications address
- cdd - content derived data; some types in this class may be able to be treated as communications data due to low level of intrusion
- ch - content header
- d - traffic data including network management (excludes any such data sent inside the body of a communication, eg by a CSP)
- m - miscellaneous; some types in this class may be able to be treated as communications data
- bulk unselected: as taken from bearers without filtering or selection, save for national sensitivities
UK SECRET STRAP1
COMINT AUS/CAN/NZ/UK/US EYES ONLY ORCON
Published in the Same Report
(10 of 28 documents)
-
Supporting Internet Operations - Special Source Access [pdf]
In this document GCHQ outlines its plans for expanding internet operations, focusing on increasing access to 10G bearers and enhancing its intercept and egress capabilities by 2011. The agency highlighted that “25 percent of all internet traffic” transits the U.K. through approximately 1,600 different 10-gigabit cables, and it noted that it could “survey the majority of the 1,600” to “select the most valuable to switch into our processing systems.”
-
Event (SIGINT) [pdf]
5 p.This document explains that within GCHQ and NSA SIGINT operations, an “Event” refers to metadata about observable communications or user actions rather than message content, categorized into types like Communications, Presence, Social, and Geo-reference Events. These events are collected, stored in various databases, and analyzed through convergence and correlation processes, with standardized formats defined by Interface Control Documents such as INTERSTELLAR DUST (IDUST) and Actor Action (AA).
Programs: INTERSTELLAR DUST · HAUSTORIUM · SOCIAL ANTHROPOID · ACTOR-ACTION · SAMREF · GEOFUSION · HARD ASSOC · SALAMANCA · ASSOCIATION · BANYAN · MAINWAY · FASCIA · CULTWEAVE II · VOICESAIL · TEEDALE · PILBEAM · MUTANTBROTH · AUTOASSOC · MARBLEDGECKO · KARMAPOLICE · HRMAP · MEMORYHOLE · INFINITEMONKEYS · SOCIALANIMAL · AUTOTDI · SAMUELPEPYS · Mobile Applications Project · NGE · PPF · SLR -
Access: Vision 2013 [pdf]
This July 2010 GCHQ strategic document outlines the agency's vision for 2013 to create "the world's biggest SIGINT engine" with access to 800 10-gigabit connections (up from existing capacity), expanding overseas operations to 60% while reducing UK-based to 40%, and developing new capabilities including computer network exploitation (CNE), exfiltration, beaconry, geolocation, and collection of "Internet Pocket Litter" like cookies and browsing data. The strategy emphasizes "scale and pace," "balance," and "agility" while claiming to operate within legal frameworks of "necessity, proportionality and authorisation," and highlights partnerships with second and third parties to access "intractable and hard targets."
Marking: SECRET -
BLACK HOLE ANALYTICS, ADD/SD briefing, September 2009 [pdf]
This September 2009 GCHQ briefing describes the BLACK HOLE analytics infrastructure and Next Generation Events (NGE) project, which planned to scale from ~8 10-gigabit connections to ~200 10Gs by March 2011, storing intercepted data in massive flat-file repositories at Benhall and Bude before loading it into "Question Focused Dataset" (QFD) databases that power surveillance tools like MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS, and others. The document outlines how "probes" extract Target Detection Identifiers (TDIs) from intercepted data feeds, store them in BLACK HOLE repositories, then enable analysts to profile targets' online activities, create social networks, investigate websites and forums, track web searches and Google Earth usage, and perform large-scale data mining across internet and telephony data.
Programs: BLACKHOLE · MUTANTBROTH · AUTOASSOC · INFINITEMONKEYS · KARMAPOLICE · SOCIALANIMAL · HRMAP · MEMORYHOLE · MARBLEDGECKO · HARDASSOC · SAMUELPEPYS · BLAZINGSADDLES · TELLURIAN · BUFFER · DISTILLERY · LOOKINGGLASS · NGEPeople: bin Laden, Osama -
Report on Architectural Risk 2012 - Summary [pdf]
This 2012 GCHQ report identifies that a historical under-investment in IT architecture has created significant risk, especially with new challenges like massive data growth, cyber defense, and increased partnering. The document recommends immediate, prioritized investment in big data strategies, integrated analytics frameworks, and security services to address the strain on the system.
Programs: BLACKHOLE -
Blazing Saddles [pdf]
This document describes BLAZING SADDLES, a GCHQ program designed to massively scale the storage and analysis of bulk internet surveillance data through multiple specialized databases that track website visits, online identities, communications patterns, and geolocation queries across the global internet population.
Programs: BLAZINGSADDLES · BLACKHOLE · SOCIALANTHROPOID · AUTOASSOC · HRMAP · INFINITEMONKEYS · KARMAPOLICE · MARBLEDGECKO · MUTANTBROTH · MEMORYHOLE · SAMUELPEPYS · SOCIALANIMAL · HAUSTORIUM · GOLDENAXE · NGE -
Broadcast/Internet Radio Exploitation and Analysis, 6th November 2009, GCHQ [pdf]
This GCHQ report analyzes 6.68 million internet radio listening events collected over three months in 2009, detailing global broadcasting patterns, user behaviors, and specifically examining Islamic radio stations and Pakistan-UK connections for potential radicalization intelligence, while recommending integration with bulk surveillance databases like KARMA POLICE for enhanced user profiling.
-
ICTR Cloud Efforts [pdf]
A GCHQ presentation details how the agency developed cloud-based big data analytics to achieve "population scale" surveillance—processing trillions of internet events to build geographic and behavioral profiles on every online identifier worldwide, enabling discovery of unknown targets through pattern analysis rather than traditional keyword searches.
Programs: EVERYASSOC · PROBABILITYCLOUD · SILVERLINING · CULTWEAVE · KARMAPOLICE · EVERYPOLICE · AWKWARDTURTLE · GORDIANKNOT · SANDIA · BLOODHOUND · FIVEALIVE · NGE · MAPREDUCE -
Cyber Defence Operations Legal Policy [pdf]
GCHQ's internal wiki laying out the legal plumbing for its Cyber Defence Operations (formerly NDIST)—detailing warrant types (RIPA, ISA, CNE), data retention schedules, targeting rules for UK/Five Eyes infrastructure, XKEYSCORE query protocols, and the bureaucratic hoops analysts must jump through to search attacker IPs, fingerprint UK networks, and share malware intelligence without tripping over location/nationality safeguards or commercial sensitivities.
Programs: JTRIG · SPAY · GORDIANKNOT · CHORDAL · DISCOVER · EVENTS · CASPORT · XKEYSCORE · CONTENT CLOUD · SQUEAL · MIRAGE · MUTANTBROTH · ANXIOUS · HALTERHITCH · BROADOAK · DONKEYKONG · ZOOL · PALANTIR · OVERLINK · RT · GATEKEEPER · IRONHAND · HARUSPEX · B3M · LOOKINGGLASS · GOLDENEYE · BLAZINGSADDLES · MOONRAKER · TRITON -
Data Stored in BLACK HOLE [pdf]
This document describes BLACK HOLE, a TOP SECRET data repository that stores approximately 200GB per day of internet activity "events" such as webmail, chat, and browsing. It details the system's various datamarts, including MUTANT BROTH and KARMA POLICE, which collectively hold terabytes of data for intelligence analysis.
Programs: BLACKHOLE · B13B · MUTANTBROTH · KARMAPOLICE · MEMORYHOLE · AUTOASSOC · SOCIALANIMAL · HRMAP · INFINITEMONKEYS
Related Documents
-
CNE End Point Requirements [pdf]
This NSA/GCHQ document outlines comprehensive requirements for an endpoint surveillance systems that enables analysts to systematically identify intelligence collection needs and discover appropriate tools for data extraction from compromised computers. The framework covers capabilities for remote data collection, cross-platform intelligence integration, query functions, tasking procedures, and visualization tools across implanted networks.
Programs: STARGATE · LOOKING GLASS · QUANTUM · GOLDENEYE · XKEYSCORE · MARINA · JTRIG · BROADOAK · UTT · GLOBALREACH · GLOBALSURGE · THUGGEE · EREPO · DANCINGBEAR · HAUSTORIUM · SHAREOWN · MUTANTBROTH · MOONRAKER · CARBONROD · RUFFLE · ROCKOPERA · QUINCEY · ECLIPSE · MUGSHOT · HIGHNOTE · UDAQ2 · EKB · TURBINE · ACNO · FLAMECARPET · FUMECUPBOARD · JACKPOT · SLIPSTREAM · B13B · PSC · QUINCY · DAREDEVIL · ROADBED -
BLACK HOLE ANALYTICS, ADD/SD briefing, September 2009 [pdf]
This September 2009 GCHQ briefing describes the BLACK HOLE analytics infrastructure and Next Generation Events (NGE) project, which planned to scale from ~8 10-gigabit connections to ~200 10Gs by March 2011, storing intercepted data in massive flat-file repositories at Benhall and Bude before loading it into "Question Focused Dataset" (QFD) databases that power surveillance tools like MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS, and others. The document outlines how "probes" extract Target Detection Identifiers (TDIs) from intercepted data feeds, store them in BLACK HOLE repositories, then enable analysts to profile targets' online activities, create social networks, investigate websites and forums, track web searches and Google Earth usage, and perform large-scale data mining across internet and telephony data.
Programs: BLACKHOLE · MUTANTBROTH · AUTOASSOC · INFINITEMONKEYS · KARMAPOLICE · SOCIALANIMAL · HRMAP · MEMORYHOLE · MARBLEDGECKO · HARDASSOC · SAMUELPEPYS · BLAZINGSADDLES · TELLURIAN · BUFFER · DISTILLERY · LOOKINGGLASS · NGEPeople: bin Laden, Osama -
PHANTOM PARROT [pdf]
This document discusses PHANTOM PARROT, a GTAC tool used for querying mobile phone data collected from individuals stopped at UK ports as part of the ITT Op WILDWAY initiative. The tool allows intelligence analysts to access phonebook, SMS, and call log data from these stops, which can be used for counter-terrorism efforts. Data is obtained under the Terrorism Act 2000, and the tool is part of a broader intelligence-gathering effort, eventually to be integrated into the LUCKY STRIKE project.
Programs: PHANTOM PARROT · WILDWAY · RANNOCH MOOR · LUCKY STRIKE · LOOKING GLASS · DISHFIRE · CORINTH · MOONRAKER · RUSSETT -
FASCIA: The NSA's huge trove of location records [pdf]
FASCIA is the National Security Agency's enormous database containing trillions of metadata records, many with device locations, that are collected from a variety of sources. This document shows the volume and types of device-location data collected by the agency.
-
QFD BLACKHOLE Technology Behind INOC [pdf]
A UK SECRET STRAP2 COMINT ORCON GCHQ Applied Research presentation from March 2009 documenting the BLACK HOLE bulk data repository's scale and structure, and the suite of analytical tools — MUTANT BROTH, KARMA POLICE, INFINITE MONKEYS and MARBLED GECKO — built on top of it to answer specific intelligence questions about online presence, browsing activity, forum usage and geolocation searches.
Programs: BLACKHOLE · MESME · NETEZZA · CLOUDS · MUTANTBROTH · KARMAPOLICE · INFINITEMONKEYS · MARBLEDGECKO · GOOB · AUTOASSOC · PSOUP · SOCIALANIMAL · SAMUELPEPYS · MEMORYHOLE · HRMAP · PSOUPALERT · LAUGHINGHYENA