Demystifying NGE Rock Ridge
Open PDFDocument date: -
Pages: 9
Genre: slides
Banner: SECRET STRAP1
Classification: SECRET STRAP 1
Dissemination: -
This presentation outlines the ROCK RIDGE project, which delivers a suite of "Question Focused Databases" (QFDs) like MUTANT BROTH and KARMA POLICE to analyze massive internet event data. These specialized databases are designed to answer specific intelligence questions and are developed using the BLACK HOLE data repository, a key component of the ROUGH DIAMOND program.
My notes
ROCKRIDGE
SECRET STRAP1
NEXT GENERATION events
Scope and Aims
Ingest more events feeds as new accesses come online
Increase maturity and availability of QFDs
Pull through more QFDs based on Ops priority
Deliver QFDs capable of holding 'Convergence' data and wider event types
ROUGH DIAMOND
Provide a data mining and collaborative QFD development facility (BLACK HOLE - part of ROUGH DIAMOND)
Enable sharing of QFD data with 2nd and 3rd Parties
Interface with visualisation services in FIRE STORM
SECRET STRAP1
NEXT GENERATION events
What is a QFD?
Designed to answer single analytic question (e.g. 'where is my target?')
Simple table structure compared to traditional multi-function databases (e.g. HAUSTORIUM)
Pioneered by ICTR, now developed by a community including Next Gen Events, ICTR, SD, GTE, ...
Question Focused Database
No specialised database technologies so simpler to develop and maintain
Additional instances can easily be deployed at new locations or to increase capacity
Smaller size and lower complexity means easier and quicker to develop and change
SECRET STRAP1
NEXT GENERATION events
What does each QFD answer?
Mutant Broth
- When was my target on line?
- Where was my target on line?
HRMap
- What web pages was my target looking at before going to this dodgy website?
Karma Police
- What websites has my target visited?
- Who's been visiting this dodgy websites?
Social Animal
- Who is my target interacting with on social networking sites?
- What files have my target been uploading/downloading?
GooBzs
(QFD Query Federator)
AutoAssoc
- What alternative identifiers can I use to search for my target?
Marbled Gecko
- Who's been looking at this suspicious part of the world?
- What part of the world has my target been looking at?
Samuel Pepys (Coming soon!)
- What is my target doing on-line right now?!
Infinite Monkeys
- Who's been posting (vBulletin boards) to this forum?
- What posting (vBulletin boards) activity has my target been up to?
Memory Hole
- Who's been searching for these suspicious things on-line?
- What has my target been searching for on-line?
SECRET STRAP1
NEXT GENERATION events
Ingest roadmap
Internet Presence Events from 10G bearers
eAD events
SALAMANCA
Telephony Events
Initial Converged Events from TERRAINS
MONOPOLY Special Source Events
Broadband RADIUS events
CDMA2000 test events
Further 10G bearers at RPC1 (Bude)
Feb 10
Mar 10
Apr 10
May 10
Jun 10
Jul 10
Aug 10
Trial part 1 - MUTANT BROTH, INFINITE MONKEYS, HRMAP, MEMORY HOLE from mobile tunnels
Experiment Explore Deployed across CPC and RPC1
Trial part 2 - MMS, Blackberry, Google Maps, mobile Hotmail, mobile Gmail from mobile tunnels
Explore Deployed across CPC and RPC1
TPS are working with the NGE Project and SMO Mobile theme to produce internet presence and application usage events from within mobile phone 'tunnels' in internet bearers. These will be trialed before full operational rollout
Trial part 3 - Hotmail, Gmail, mail RU, Yahoo webmail from internet bearers
Experiment Explore Deployed across CPC and RPC1
Trial part 4 - Windows Live IM, Yahoo Mail, SIP from internet bearers
'QFD style' events will also be produced for types of event traditionally fed into the older HAUSTORIUM and HARBOUR PILOT databases
Explore Deployed across CPC and RPC1
SECRET STRAP1
NEXT GENERATION events
Convergence QFDs
Evolved MUTANT BROTH
Search by geographical area
Search Results
Screenshots from evolved MUTANT BROTH web interface, and an export of it's data to Google Earth
This major thread of work will:
- Store events where internet applications are accessed from a mobile device
- Allow analysts to relate mobile device identifiers to internet identifiers such as email addresses
- Enable QFDs to store other more diverse event types, such as telephony events (currently SALAMANCA), and email events (currently HAUSTORIUM / HARBOUR PILOT)
- Interface to LOOKING GLASS visualisation coming soon (in FIRE STORM work package)
SECRET STRAP1
NEXT GENERATION events
SAMUEL PEPYS QFD
Purpose: Provide a near-real-time diarisation of any IP address
Results
GeoFusion reports IP address 90.237.[REDACTED] as MALMO (low confidence), SE (medlum confidence).
| Date | Time (UTC) | Source | Destination | Type | Description |
|---|---|---|---|---|---|
| 02/02/10 | 00:08:01 | 90.237.[REDACTED] | 205.178.145.65 | Websearch | Visited cryptome.org/eyeball/gchq-eyeball.htm (after se |
Bearer GWUSC503
Connection TCP: 90.237[REDACTED] port 51475 to 205.178.145.65 port 80
Normalised query gchq cotswold
Search Term gchq cotswold
Search Host www.google.se
Search-Clicked-Host cryptome.org
Search-Clicked-URI /eyeball/gchq-eyeball.htm
Accept-Language SV-SC
User-Agent Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; WOW64; Trident/1.0; GTB6.1; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; eSobiSubscriber 2.0.4.16)
Cookie nginx cached-1
Geo-IP-Src 55.60;13.00; MALMO;SE;BIVV
Geo-IP-Dst 39.0062;-//-77.4288;STERLING;US;MMM
| Date | Time (UTC) | Source | Destination | Type | Description |
|---|---|---|---|---|---|
| 02/02/10 | 00:07:59 | 90.237.[REDACTED] | 205.178.145.65 | HTTP | GET cryptome.org/eyeball/gchq-eyeball.htm |
| 01/02/10 | 23:41:12 | 90.237.[REDACTED] | 205.178.145.65 | HTTP | GET cryptome.org/site-r/site-r.htm |
| 01/02/10 | 23:41:08 | 90.237.[REDACTED] | 205.178.145.65 | HTTP | GET eyeball-series.org/site-r/site-r.htm |
| 01/02/10 | 23:38:28 | 90.237.[REDACTED] | 205.178.145.65 | Websearch | Visited cryptome.org/eyeball/siter-birdseye/siter-birdsey |
| 01/02/10 | 23:38:26 | 90.237.[REDACTED] | 205.178.145.65 | HTTP | GET cryptome.org/eyeball/siter-birdseye/siter-birdeye.h |
Expand all Collapse all Export CSV Export raw 1/1 6 Row(s)
Prototyped by ICTR – Currently being pulled through by ROCK RIDGE, will be scaled to full 10G volumes by May 2010
SECRET STRAP1
NEXT GENERATION events
BLACK HOLE
What is BLACK HOLE?
- A flat file store housing all data from a wide range of feeds (events and content)
- Provides a set of tools for accessing that data.
- Intended to be the source of events (and limited content) for the development of new QFDs and analytics.
- Contains a rolling 6 months retention Part of ROUGH DIAMOND
What does it enable?
- New QFDs to be rapidly prototyped, then to be added to the operational QFD suite
- Trialling of new bulk analysis ideas
- New sources of data to be introduced quickly into existing QFDs.
- Users to look for particular patterns and behaviours (target discovery)
- TR, GTAC and GTE access to more data for research purposes, which may not be QFD related.
SECRET STRAP1
NEXT GENERATION events
User Feedback
'Absolutely FABULOUS well done '
(Iain Lobban, ref SUPERDRAKE reporting)
'its amazing to see how the pace of delivery in TDB has increased and I have been impressed by your responsiveness to customer needs. '
([REDACTED], Senior User)
'Almost exactly a year ago I set you the challenge of delivering an upscaled massive events capability in order to support Internet Operations being conducted by GCHQ. Through your stripy team working on BLAZING SADDLES,
BLUESHIFT and SUPPORTING INO you successfully met this challenge and delivered us a significant new capability in July.'
([REDACTED], Deputy Director Cyber Operations)
'It's working flawlessly'
(analyst, ref BLACK HOLE)
'Bloody awesome'
(analyst, ref SUPERDRAKE QFD)
SECRET STRAP1
NEXT GENERATION events