Demystifying NGE Rock Ridge

Open PDF
Creating agency: GCHQ Government Communications Headquarters (UK)
Document date: -
Pages: 9
Genre: slides
Banner:
Classification: SECRET STRAP 1
Dissemination: -

This presentation outlines the ROCK RIDGE project, which delivers a suite of "Question Focused Databases" (QFDs) like MUTANT BROTH and KARMA POLICE to analyze massive internet event data. These specialized databases are designed to answer specific intelligence questions and are developed using the BLACK HOLE data repository, a key component of the ROUGH DIAMOND program.

Long Description:
The ROCKRIDGE project is a "Next Generation events" initiative focused on developing and deploying a suite of simple, powerful databases called Question Focused Databases (QFDs). Each QFD is designed to answer a single, specific analytic question, such as "Where was my target online?" (MUTANT BROTH) or "What websites has my target visited?" (KARMA POLICE). Other QFDs detailed include SOCIAL ANIMAL for social media analysis and SAMUEL PEPYS for near-real-time activity logging. This QFD development is facilitated by the ROUGH DIAMOND program's BLACK HOLE, a massive flat-file data repository with a six-month rolling buffer that enables rapid prototyping and bulk analysis. A major goal of the project is "Convergence," which aims to evolve QFDs to store and relate diverse event types, linking mobile device identifiers with internet identifiers like email addresses. The initiative plans to ingest a wide array of new data feeds, including telephony events (SALAMANCA), converged events (TERRAINS), and data from mobile phone "tunnels," eventually replacing older, more complex databases like HAUSTORIUM. The roadmap from early 2010 outlines a rapid series of trials to operationalize the collection of data from sources like webmail, instant messaging, and 10G internet bearers at sites including RPC1 (Bude).
Programs:
ROCKRIDGE, ROUGHDIAMONG, BLACKHOLE, HAUSTORIUM, HRMAP, MUTANTBROTH, KARMAPOLICE, GOOBZS, INFINITEMONKEYS, AUTOASSOC, SOCIALANIMAL, MARBLEDGECKO, MEMORYHOLE, SAMUELPEPYS, SALAMANCA, MONOPOLY, RADIUS, HARBOURPILOT, LOOKINGGLASS, FIRESTORM, SUPERDRAKE, BLAZINGSADDLES, BLUESHIFT, SUPPORTINGINO, TERRAINS, GEOFUSION, NGE

People mentioned:
Lobban, Iain

Countries:
Sweden

Tags:
QFD, Next Generation Events, convergence, data mining, federated query, events, event processing, CDMA2000, 10G, mobile tunnels, internet bearers, target development, ICTR, GTE, GTAC, TDB, cyber operations, SMO, Bude, ingest, data feeds, webmail, web search, vBulletin, web forums, social networks, MMS, Google Maps, Hotmail, Gmail, Yahoo, Windows Live Messenger, SIP, diarisation, IP geolocation, IP addresses, User-Agent, cookies

Identifier:
demystifying-nge-rock-ridge

PDF SHA-256:
fe8414bac2ac47d637543ab3652a9fa0be5291dfad004af85f378622c754b852

ROCKRIDGE

SECRET STRAP1

NEXT GENERATION events


Scope and Aims

Ingest more events feeds as new accesses come online

Increase maturity and availability of QFDs

Pull through more QFDs based on Ops priority

Deliver QFDs capable of holding 'Convergence' data and wider event types

ROUGH DIAMOND

Provide a data mining and collaborative QFD development facility (BLACK HOLE - part of ROUGH DIAMOND)

Enable sharing of QFD data with 2nd and 3rd Parties

Interface with visualisation services in FIRE STORM

SECRET STRAP1
NEXT GENERATION events


What is a QFD?

Designed to answer single analytic question (e.g. 'where is my target?')

Simple table structure compared to traditional multi-function databases (e.g. HAUSTORIUM)

Pioneered by ICTR, now developed by a community including Next Gen Events, ICTR, SD, GTE, ...

Question Focused Database

No specialised database technologies so simpler to develop and maintain

Additional instances can easily be deployed at new locations or to increase capacity

Smaller size and lower complexity means easier and quicker to develop and change

SECRET STRAP1
NEXT GENERATION events


What does each QFD answer?

Mutant Broth

  • When was my target on line?
  • Where was my target on line?

HRMap

  • What web pages was my target looking at before going to this dodgy website?

Karma Police

  • What websites has my target visited?
  • Who's been visiting this dodgy websites?

Social Animal

  • Who is my target interacting with on social networking sites?
  • What files have my target been uploading/downloading?

GooBzs
(QFD Query Federator)

AutoAssoc

  • What alternative identifiers can I use to search for my target?

Marbled Gecko

  • Who's been looking at this suspicious part of the world?
  • What part of the world has my target been looking at?

Samuel Pepys (Coming soon!)

  • What is my target doing on-line right now?!

Infinite Monkeys

  • Who's been posting (vBulletin boards) to this forum?
  • What posting (vBulletin boards) activity has my target been up to?

Memory Hole

  • Who's been searching for these suspicious things on-line?
  • What has my target been searching for on-line?

SECRET STRAP1
NEXT GENERATION events


Ingest roadmap

Internet Presence Events from 10G bearers
eAD events

SALAMANCA
Telephony Events

Initial Converged Events from TERRAINS

MONOPOLY Special Source Events

Broadband RADIUS events

CDMA2000 test events

Further 10G bearers at RPC1 (Bude)

Feb 10
Mar 10
Apr 10
May 10
Jun 10
Jul 10
Aug 10

Trial part 1 - MUTANT BROTH, INFINITE MONKEYS, HRMAP, MEMORY HOLE from mobile tunnels

Experiment Explore Deployed across CPC and RPC1

Trial part 2 - MMS, Blackberry, Google Maps, mobile Hotmail, mobile Gmail from mobile tunnels

Explore Deployed across CPC and RPC1

TPS are working with the NGE Project and SMO Mobile theme to produce internet presence and application usage events from within mobile phone 'tunnels' in internet bearers. These will be trialed before full operational rollout

Trial part 3 - Hotmail, Gmail, mail RU, Yahoo webmail from internet bearers

Experiment Explore Deployed across CPC and RPC1

Trial part 4 - Windows Live IM, Yahoo Mail, SIP from internet bearers

'QFD style' events will also be produced for types of event traditionally fed into the older HAUSTORIUM and HARBOUR PILOT databases

Explore Deployed across CPC and RPC1

SECRET STRAP1

NEXT GENERATION events


Convergence QFDs

Evolved MUTANT BROTH
Search by geographical area
Search Results

Google

Screenshots from evolved MUTANT BROTH web interface, and an export of it's data to Google Earth

This major thread of work will:

  • Store events where internet applications are accessed from a mobile device
  • Allow analysts to relate mobile device identifiers to internet identifiers such as email addresses
  • Enable QFDs to store other more diverse event types, such as telephony events (currently SALAMANCA), and email events (currently HAUSTORIUM / HARBOUR PILOT)
  • Interface to LOOKING GLASS visualisation coming soon (in FIRE STORM work package)

SECRET STRAP1
NEXT GENERATION events


SAMUEL PEPYS QFD

Purpose: Provide a near-real-time diarisation of any IP address

Results
GeoFusion reports IP address 90.237.[REDACTED] as MALMO (low confidence), SE (medlum confidence).

Date Time (UTC) Source Destination Type Description
02/02/10 00:08:01 90.237.[REDACTED] 205.178.145.65 Websearch Visited cryptome.org/eyeball/gchq-eyeball.htm (after se
Bearer              GWUSC503
Connection          TCP: 90.237[REDACTED] port 51475 to 205.178.145.65 port 80
Normalised query    gchq cotswold
Search Term         gchq cotswold
Search Host         www.google.se
Search-Clicked-Host cryptome.org
Search-Clicked-URI  /eyeball/gchq-eyeball.htm
Accept-Language     SV-SC
User-Agent          Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; WOW64; Trident/1.0; GTB6.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; eSobiSubscriber 2.0.4.16)
Cookie              nginx cached-1
Geo-IP-Src          55.60;13.00; MALMO;SE;BIVV
Geo-IP-Dst          39.0062;-//-77.4288;STERLING;US;MMM
Date Time (UTC) Source Destination Type Description
02/02/10 00:07:59 90.237.[REDACTED] 205.178.145.65 HTTP GET cryptome.org/eyeball/gchq-eyeball.htm
01/02/10 23:41:12 90.237.[REDACTED] 205.178.145.65 HTTP GET cryptome.org/site-r/site-r.htm
01/02/10 23:41:08 90.237.[REDACTED] 205.178.145.65 HTTP GET eyeball-series.org/site-r/site-r.htm
01/02/10 23:38:28 90.237.[REDACTED] 205.178.145.65 Websearch Visited cryptome.org/eyeball/siter-birdseye/siter-birdsey
01/02/10 23:38:26 90.237.[REDACTED] 205.178.145.65 HTTP GET cryptome.org/eyeball/siter-birdseye/siter-birdeye.h

Expand all Collapse all Export CSV Export raw 1/1 6 Row(s)

Prototyped by ICTR – Currently being pulled through by ROCK RIDGE, will be scaled to full 10G volumes by May 2010

SECRET STRAP1
NEXT GENERATION events


BLACK HOLE

What is BLACK HOLE?

  • A flat file store housing all data from a wide range of feeds (events and content)
  • Provides a set of tools for accessing that data.
  • Intended to be the source of events (and limited content) for the development of new QFDs and analytics.
  • Contains a rolling 6 months retention Part of ROUGH DIAMOND

What does it enable?

  • New QFDs to be rapidly prototyped, then to be added to the operational QFD suite
  • Trialling of new bulk analysis ideas
  • New sources of data to be introduced quickly into existing QFDs.
  • Users to look for particular patterns and behaviours (target discovery)
  • TR, GTAC and GTE access to more data for research purposes, which may not be QFD related.

SECRET STRAP1
NEXT GENERATION events


User Feedback

'Absolutely FABULOUS well done '
(Iain Lobban, ref SUPERDRAKE reporting)

'its amazing to see how the pace of delivery in TDB has increased and I have been impressed by your responsiveness to customer needs. '
([REDACTED], Senior User)

'Almost exactly a year ago I set you the challenge of delivering an upscaled massive events capability in order to support Internet Operations being conducted by GCHQ. Through your stripy team working on BLAZING SADDLES,
BLUESHIFT and SUPPORTING INO you successfully met this challenge and delivered us a significant new capability in July.'
([REDACTED], Deputy Director Cyber Operations)

'It's working flawlessly'
(analyst, ref BLACK HOLE)

'Bloody awesome'
(analyst, ref SUPERDRAKE QFD)

SECRET STRAP1
NEXT GENERATION events