Operational Engineering November 2010

Open PDF
Creating agency: GCHQ Government Communications Headquarters (UK)
Document date: 2010-11-01
Pages: 37
Genre: slides
Banner:
Classification: SECRET STRAP 1
Dissemination: -

This internal GCHQ briefing from late 2010 documents the Events Product Centre's suite of bulk internet and communications surveillance tools — including IMMINGLE, SALAMANCA, SOCIAL ANTHROPOID, MUTANT BROTH, KARMA POLICE, SAMUEL PEPYS and GUIDING LIGHT — covering scaling targets, new data sources, and operational capabilities across telephony, webmail, instant messaging, social media, GPRS and VoIP collection.

Long Description:
This is a SECRET STRAP1 GCHQ internal presentation from November 2010, delivered under the "Blazing Saddles" operational framework by the Events Product Centre, providing a comprehensive overview of GCHQ's bulk internet and communications surveillance infrastructure at a moment of significant scaling and capability expansion. The document covers several major surveillance systems in operational detail. IMMINGLE is described as a contact chaining and event aggregation tool drawing on multiple data sources simultaneously: SALAMANCA (telephony metadata), B3M, HARD ASSOC, FASCIA (mobile location and metadata), SAMDYCE, HAUSTORIUM, CULTWEAVE and MAINWAY II (NSA's telephony metadata database). A sample query interface shows analysts querying multiple phone numbers simultaneously with enrichment from BROAD OAK, CONTRAOCTAVE and CRYING FOWL. A live example shows tracking a Saudi Aramco senior vice president across Malaysian and Saudi Arabian GPRS events. SALAMANCA is described as receiving additional feeds from TERRAINs at BUDE and SOUNDER — GCHQ's Cornwall and Cyprus cable tapping stations — with second party (GCHQ partner, likely NSA) usage now routed through SHAREOWN replacing ESCHAR. SOCIAL ANTHROPOID is presented as the converged communications database consolidating telephony, instant messenger, webmail, SIP/H323 VoIP, Yahoo Voice, BlackBerry, MMS and SMS data, replacing legacy tools SOCIAL ANIMAL and HAUSTORIUM. Live interface screenshots show chat events between redacted Hotmail and Linuxmail accounts, telephony call records, GTP tunnel creation events linking IMSI, telephone numbers and BlackBerry PIN identifiers, and Yahoo webmail events with Huawei IMSI identifiers leaking through what the document calls "leaky gateways." The scaling section is particularly revealing: MUTANT BROTH was ingesting over 10.5 billion events per day with six months data retention requiring 400 terabytes, with total storage exceeding one petabyte. Plans called for expansion to 20 billion events per day requiring one petabyte for MUTANT BROTH alone, with overall storage growing to 2.5 petabytes, ultimately scoping to 400 bearers. The TDI listing section documents the taxonomy of Target Detection Identifiers including Yahoo-Y-Cookie, Yahoo-B-Cookie, Google-PREFID-Cookie, Facebook cookies, SIP-INVITE-From, and dozens of other web identifiers used to track individuals across platforms. GUIDING LIGHT is presented as a new QFD (Query Focus Dataset) tool for understanding traffic on Next Generation Events bearers, with a full profile query result table showing millions of events by source type, event type, origination and destination country — revealing collection concentrated on Pakistan, the Falkland Islands (FK), Iran (IR) and other target countries, with large volumes of Yahoo cookie, SIP and Yahoo Messenger events. New data sources noted include LUSTRE (good for North Africa), and CNE, JTRIG GLASSBACK and COLLATERAL data feeds being added to MUTANT BROTH.
Programs:
EVENTS, IMMINGLE, SALAMANCA, GUIDINGLIGHT, B3M, HARDASSOC, MAINWAY, THUGGEE, FASCIA, HAUSTORIUM, FIRESTORM, TERRAIN, SHAREOWN, ESCHAR, BRIO, INFINITEMONKEY, KARMAPOLICE, MARBLEDGECKO, MEMORYHOLE, MUTANTBROTH, SAMUELPEPYS, SOCIALANIMAL, SOCIALANTHROPOID, HEARTBEAT, SANTHROPOID, LEAKYGATEWAYS, GTPTUNNEL, LUSTRE, GLASSBACK, COLLATERAL, HRMAP, SWORDPLAY, BROADOAK, REFORMER, ARTEMIS, MERLOT, CONTRAOCTAVE, CRYINGFOWL, CULTWEAVE, SAMDYCE, GCQUERY, BLAZZINGSADDLES, EREPO, JTRIG, SOUNDER

People mentioned:
-

Countries:
Saudi Arabia, Iran, Germany, Pakistan, China, France, Malaysia

Tags:
Events Product Centre, JTRIG, Bude, SOUNDER, TEMPORA, GTP tunnels, BlackBerry, Yahoo cookies, Facebook cookies, cookies, SIP, VoIP, GPRS, telephony metadata, contact chaining, bulk collection, internet surveillance, cookie tracking, TDI, QFD, bearers, petabyte storage, Saudi Aramco, second party, population-scale surveillance, Next Generation Events

Identifier:
operational-engineering-nov-2010

PDF SHA-256:
1d534f7681d0d63226e228e64c44273ee45cef5982b2b3769eb3cca2cc354c44

events

Events Product Centre

SECRET STRAP1
GCHQ


events

Agenda

  • Welcome
  • Immingle
  • Salamanca
  • QFDs
  • Guiding Light
  • Questions

SECRET STRAP1
GCHQ


events

IMMINGLE

Key changes July 2010 to present:

  • Inferred data from B3M now flagged
  • Updates to handle identifiers from HARD ASSOC and B3M correctly
  • MAINWAY: MSRNs now grouped and flagged in same way as SALAMANCA
  • MAINWAY: direct access to event details provided
  • GPRS flagging – THUGGEE rules applied to SALAMANCA events

SECRET STRAP1
GCHQ


events

IMMINGLE

Event Summary (First Circle) Query

Query Name IMM 14-10-2010 103234

Legalities Justification
MIRANDA 20140
HRA Justification Dev testing
JIC Priority 7NS

Date Range
Start Date 1 Month Ago 14/09/2010 00:00:00
End Date Today 14/10/2010 23:59:59

Event Sources
SALAMANCA
B3M
HARD ASSOC
FASCIA
SAMDYCE
HAUSTORIUM
CULTWEAVE
MAINWAY II
Event details
Precomputed Pairs
Used to find external SEED identifiers alternatives will be the supplied seed

MAINWAY options and Help pages

FREEFORM MERLOT Add Term
Data File Location | Source | Custom Template
Add file | Remove Selected | Remove All

Selector Parameters
Before using C2C for event analysis, please read this section of the user guide

Seed Entry
Add All ->
<- Return Type
<- Return Selected
<- Return All
List of seeds to be queried
[REDACTED] phone
[REDACTED] phone
[REDACTED] phone
[REDACTED] phone
[REDACTED] phone

PHONE IMSI IMEI FHEX
C2C Machine
Remove All

Enrichment Sources
BROAD OAK
CONTRAOCTAVE
CRYING FOWL
Select All | Deselect All

Visualisation Options
Display Associations
Perform Dedupe
Flag Seed identifier
Include Unknown Identifiers
Include inferred identifiers
Summary Count for High Rates Report 1000

C2C Only
These options apply to HAUSTORIUM only
Relationships: Strong All
DO ACTIVITY SUMMARY
Maximum Event Participant Count 200

Query Cancel

SECRET STRAP1
GCHQ


events

IMMINGLE

GPRS flag and count

MALAYSIA :26 MAINWAY
AKABIA
MALAYSIA
07/10/2010 06:57 5
11
07/11/2010 05-14/09/2010 07:51 14
19/09/2010-08:20
MAINW 02/10/2010 03:37 SALAMANCA
29/09/2010 01:21
MAINWAY II: 8, FASCIA: 151 FASCIA: 3WAY II: 10
FASCIA: 24
ASCIA: 21
30123/09/2010 14:19:39
MAINWAY II: 1, FASCIA: 3
FASCIA: 11 1/4, FASCIA. 111
60122674959. 26 MAINWAY II: 4, FASCIA:
FASCIA: 0 0 11:06
18/09/2010 07:2012
FASCIA: 19
02/10/2010 11:263/2010 03:34 6
MAINWAY II: 17, FASCIA: 33

GPRS Events: 8
SAUDI ARABIA
SEED IDENTIFIER

MAINWAY II, FASCIA: 9
01/10/2010 10:3710 12:52
01/10/2010 10:40 15,4ASCIA: 1
MAINWAY II: 8, SALAMANCA: B, F
19/09/20 10 12:15
MAINWAY II: 3, SALAMANCA: 3
Datasource Text MAINWAY II, SALAMANCA, OCT...
BROAD OAK Is N... Text No
GPRS Last Seen Text 26-09-2010 07:43:02
GPRS First Seen Text 24-09-2010 06:31:07
BROAD OAK Desi... Text [REDACTED]
BROAD OAK Sher... Text 1
BROAD ОАК Неа... Text Saudi Aramco SVP Special Assign...

SECRET STRAP1
GCHQ


events

IMMINGLE

What next?

  • FASCIA GPRS flagging
  • HAUSTORIUM decommissioning
  • Next Gen Contact Chaining trial....

NEXT GENERATION
events

FIRE STORM

SECRET STRAP1
GCHQ


events

BRIO and SALAMANCA

Key changes since July:

  • NRT (Near Real Time) Storage = 3 days
  • Extra feeds from TERRAINs at BUDE and SOUNDER
  • 2nd Party usage of SALAMANCA: SHAREOWN replaces ESCHAR
  • CallAnsweredState and CallEndState added to TERRAIN-SALAMANCA feed

SECRET STRAP1
GCHQ


events

BRIO/SALAMANCA (cont.)

Pakistan NGN inferencing errors

GCQuery (Operational Server) - Microsoft Internet Explorer
File Edit View Favorites Tools Help
Back Search Favorites
Address: [REDACTED]

GCQuery (Operational Server)
HELP

HPSu 8.3

Outage Status
There are no planned outages.

WARNING - Incorrect data in SALAMANCA!
Inferred events on Source FDDG-QY, SIGAD UKC-3071 between 18 June 2010 and 19 October 2010 should not be used.
More Information

Problems?
Please report all problems or queries via the ITS2.

Need an account?
You can get a GCQuery account via the ITSO's Online Forms:
*   SALAMANCA ARTEMIS GREEN
*   All Others

re-update. The Presidential Britain above Feb 2000 and may be
rmation Regulation. Refer-updates are requests to GCHQ at

[REDACTED] Last Updated: 7th Mar 2010

SECRET STRAP1
GCHQ


events

QFDs

Blazing Saddles

[REDACTED]

SECRET STRAP1
GCHQ


events

Scaling

Blazing Saddles

Current scale

  • There are 100 unique bearers feeding the BzS tools.
  • Consistently averaging over 30 billion events per-day into the input buffer.
    • MB is loading over 10.5 billion
    • 6 months data retention for MB = 1,890,000,000,000 records and requires 400 TB.
    • Total storage of over one petabyte.

SECRET STRAP1
GCHQ


events

Scaling

Blazing Saddles

Future Scale

  • Further 58 bearers by end of 2010
  • An additional 40 bearers in Q1 2011.
    • MB will ingest over 20 billion events per day requiring one petabyte of storage.
    • Overall storage will increase to 2.5 petabytes.
  • Scope scaling to 400 bearers.

SECRET STRAP1
GCHQ


events

TDI listing

HR Ma
INFINITE MON
KARMA POLIC
MARBLED GEC
MEMORY HOL
MUTANT BROT
SAMUEL PEPY
SOCIAL ANIMAL
SOCIAL ANTHROPOID user guide
TDI Database login using Corporate Directory password

TDI listing

Browse by technology
Rows 15 Go
View Chart

Or, type in the TDI type you are interested in
TDI type Yahoo-Y-Cookie Go

Technologies associated with Yahoo-Y-Cookie
Tech Description
Yahoo Yahoo provides various free web services including webmail, a web portal, a web directory, news, and mapping.
1-1

Yahoo-Y-Cookie details

TDI Name Yahoo-Y-Cookie
Type TDI
Scope User
Description This is the username of the Yahoo! user who is logged in. The username Is the first bit of the Yahoo! e-mail address (everything before the "@" if the user is on a primary domain-or the full email address if the user is on a secondary domain (see <a href=[REDACTED]>here</a> for more info on primary/secondary domains). In raw traffic, the username appears as part of the "Y" cookie string, with "I" before it and "/o" after it, obfuscated using a simple substitution code called ROT13. The QFDs automatically break out the relevant part of the cookie and turn it back into plain text for use as a TDI. If a Yahoo-Y-Cookie is seen with user agent Mozilla/4.0 (compatible; MSIE 5.5...) then that is probably the Yahoo! instant messaging client rather than a web browser.
Further info If the information above is blank, incomplete, or unhelpful, please email OTE Tech Tracking and demand answers.

BEGAL rules generating Yahoo-Y-Cookie events
Name                  ID
EXP_Yahoo-Y-Cookie_0  11113
1-1

[Full Feed]
Subscribe to the blog feed on the GCWeb dynamic home page.

SECRET STRAP1
GCHQ


events

Samuel Pepys

Blazing Saddles

Pull through and upscaling of TR SPs.

  • Currently 43 bearers.
    • 14 from TR SP
    • 29 additional bearers from TPS (generating HTTP, TDI, Websearch, FTP and Squeal).
    • Circa 40 additional bearers just generating Squeal.
  • Approval to increase aperture to 100 bearers for all data-types.
  • Approval to increase user numbers to 200.

SECRET STRAP1
GCHQ


events

Blazing Saddles

SOCIAL ANTHROPOID

SECRET STRAP1
GCHQ


events

Blazing Saddles

What is Social Anthropoid?

  • SOCIAL ANTHROPOID is a converged comms database. It will allow you to see when your targets have communicated via phone, over the internet, or using converged channels (e.g., sending e-mails from a phone or making voice calls over the internet).

SECRET STRAP1
GCHQ


events

Blazing Saddles

What about the existing comms data-bases?

  • When SOCIAL ANTHROPOID contains all the necessary data and has all the core functionality of the legacy tools Social animal, HAUSTORIUM and SALAMANCA will be de-commissioned.

SECRET STRAP1
GCHQ


events

Blazing Saddles

What data is in Social Anthropoid??

  • All of Salamanca data (telephony)
    • Social animal data.
    • Instant Messenger.
    • Webmail. - SIP & H323 VOIP
      • Yahoo Voice
      • Blackberry
      • MMS
  • SMS (from Salamanca and other sources)
    • GTP (GPRS session set-ups)
      • And more..

SECRET STRAP1
GCHQ


events

Blazing Saddles

What about SMTP, POP3 and IMAP?

  • Starting to receive these data types now.
  • Capability deployed as part of HeartBeat 11.

SECRET STRAP1
GCHQ


events

Blazing Saddles

Query
Saved queries
You have 1 saved queries.
Test:

*   Queries will be automatically submitted to all instances of SOCIAL ANTHROPOID, SOCIAL ANIMAL and Converge SOCIAL  ANIMAL.
*   For bulk queries, enter multiple selectors (one per line).
*   If allow wildcards is ticked, it is treated as a multi-character wildcard (e.g., pauls* ill match paul123, paul456, paul*@yahoo.com will match paul123@yahoo.com but not paul123@hotmail.com). Unlike other QFDs, * and \ have no special meaning (to query for a literal string, uncheck allow wildcards rather than 'escaping' the wildcard.
*   By default, results will be returned in which your input selector appears in either the User A or User B column (in SOCIAL ANIMAL terms is the 'actor' or the 'subject' with in the event). To return results in which your selector appears only as the active user, tick the Query active users only checkbox.
*   Front-end processing normalises C2C selectors in various ways, including the removal of dots from the usernames of Email addresses. To get Gmail results, you will need to normalise your queries in the same way (e.g., search for badguy@gmail.com instead of bad.guy@gmail.com). Gmail itself ignores the dots so there is no danger of getting events for the wrong account. If in doubt, consult your local C2C tech ex.

Miranda 2014C
JIC Priority & Purpose 1NS
HRA Justification testing

Search period (optional): to
Filter results by matched selectors prior to display
Allow wildcards
Query active users only

Save Query
Submit query

SECRET STRAP1
GCHQ


events

C2C

Blazing Saddles

Query Input | Results [ REDACTED] x | Results [REDACTED] | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x

User A role
User A type
User A
User A row value
User A display name
User B type
User B
User B row value
User B display name

Filter criteria
To filter the result set choose some criteria

28Oct-
Active e
Action:
:ha:
message
Locato
Source
More
24-Oct-
Active e
Action:
:ha:
message
Locat
Source
More
24-Oct-
Active e
Action:
:ha:
message
Locat
Source
more
09-Oct-2010 10:26:12 – chat event (windowslive), 2 selectors
Page 1 of 1 First Previous 1 Next Last (1) Change Sort Order Display Summary CSV Export
Temporal filtering
TDI filtering
Participant count filtering

Results summary

Action Action Type User A Role User A User A Type User B Role User B User B Type First Seen Last Seen Count
chat message chat message [REDACTED]@hotmail.com email [REDACTED]@linuxmail.org email 02-Jul-2010 14:01:19 02-Jul-2010 14:01:19 1
chat message chat message [REDACTED]@hotmail.com Sender Recipient [REDACTED]@linuxmail.org Unknown 23-Aug-2010 18:21:49 23-Aug-2010 18:23:13 2
chat message chat message [REDACTED]@linuxmail.org Sender Recipient [REDACTED]@hotmail.com Unknown 24-Oct-2010 15:34:48 24-Oct-2010 15:34:48 1
chat message chat message [REDACTED]@linuxmail.org Sender Recipient [REDACTED]@hotmail.com Unknown 24-Oct-2010 15:57:43 28-Oct-2010 04:00:23 2
chat message chat message [REDACTED]@hotmail.fr email [REDACTED]@linuxmail.org email 12-Jun-2010 18:16:14 12-Jun-2010 18:16:33 2
chat message chat message [REDACTED]@hotmail.it Sender Recipient [REDACTED]@linuxmail.org Unknown 09-Oct-2010 18:26:08 09-Oct-2010 18:26:12 2
chat message chat message [REDACTED]@linuxmail.org email [REDACTED]@linuxmail.org email 23-Aug-2010 18:40:56 23-Aug-2010 18:41:08 6

SECRET STRAP1
GCHQ


events

Telephony in Santhropoid

Blazing Saddles

Query Input | Results [ REDACTED] x | Results [REDACTED] | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x

User A role
User A type
User A
User A row value
User A display name
User B type
User B
User B row value
User B display name

03-Nov-2010 17:34:20 - telephony event (global), 2 selectors, duration: 00:00:06
Active user: [REDACTED](tel_number)
Action: unknown Action type: call
unknown call tel_number [REDACTED] dialed tel_number [REDACTED]
Locators:
Source Point-Code: 60741 Destination Point-Code: 20082
More

03-Nov-2010 17:34:19 - telephony event (global), 2 selectors, duration: 00:00:06
Active user: [REDACTED]el_number)
Action: unknown Action type: call
unknown call tel_number [REDACTED] dialed tel_number [REDACTED]
Locators:
Source Point-Code: 60741 Destination Point-Code: 20082
More

01-Nov-2010 17:17:39 telephony event (global), 2 selectors, duration: 00:00:05
Active user: [REDACTED](tel_number)
Action: unknown Action type: call
unknown call tel_number [REDACTED] dialed tel_number [REDACTED]
Locators:
Source Point Code: 60741 Destination Point Code: 20082
More

SECRET STRAP1
GCHQ


events

Convergence - GTP tunnel

Blazing Saddles

Query Input | Results [ REDACTED] x | Results [REDACTED] | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x

User A role
User A type
User A
User A row value
User A display name
User B type
User B
User B row value
User B display name

01-Nov-2010 07:16:06 - tunnel event (gtp), 3 selectors
Active user: [REDACTED] (imsi) [REDACTED] (tel_number) [REDACTED] (blackberry_mid_pin)
Action: create Action type: tunnel
create tunnel imsi [REDACTED] not_available not_available
create tunnel tel_number [REDACTED] not_available not_available
create tunnel blackberry_mid_pin [REDACTED] not_available not_available
Locators:
Source IPV4: [REDACTED] Source SgsnAddress: [REDACTED] Destination IPV4: [REDACTED]
GTP Convergence Data:
This event represents the creation of GTP tunnel [REDACTED]
View all events from this GTP tunnel
More

01-Nov-2010 04:20:10 - tunnel event (gtp), 3 selectors
Active user: [REDACTED] imsi) [REDACTED] tel_number) [REDACTED] blackberry_mid_pin)
Action: create Action type: tunnel
create tunnel imsi [REDACTED] not_available not_available
create tunnel tel_number [REDACTED] not_available not_available
create tunnel blackberry_mid_pin [REDACTED] not_available not_available
Locators:
Source IPV4: [REDACTED] Source SgsnAddress: [REDACTED] Destination IPV4: [REDACTED]
GTP Convergence Data:
This event represents the creation of GTP tunnel [REDACTED]
View all events from this GTP tunnel

SECRET STRAP1
GCHQ


events

Convergence - Leaky Gateways

Blazing Saddles

Query Input | Results [ REDACTED] x | Results [REDACTED] | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x

User A role
User A type
User A
User A row value
User A display name
User B type
User B
User B row value
User B display name

24-Oct-2010 06:11:57 - webmail event (yahoo), 5 selectors
Active user: [REDACTED] (UserId) [REDACTED] AccountOwner [REDACTED] (MachineId) [REDACTED] (X-huawei-IMSI)
Action: send Action type: message
From Unknown [REDACTED]@yahoo.com From [REDACTED] Unknown [REDACTED]@yahoo.com
Locators:
Source IPV4: [REDACTED] Destination IPV4: [REDACTED]
More

24-Oct-2010 06:11:57 webmail event (yahoo), 5 selectors
Active user: [REDACTED] (UserId) [REDACTED] AccountOwner [REDACTED] (MachineId) [REDACTED] (X-huawei-IMSI)
Action: send Action type: message
From Unknown [REDACTED]@yahoo.com From Unknown [REDACTED]@yahoo.com
Locators:
Source IPV4: [REDACTED] Destination IPV4: [REDACTED]
More

24-Oct-2010 06:10:16 - webmail event (yahoo), 5 selectors
Active user: [REDACTED] (UserId) [REDACTED] AccountOwner [REDACTED] (MachineId) [REDACTED] (X-huawei-IMSI)
Action: send Action type: message
From Unknown [REDACTED]@yahoo.com From Unknown [REDACTED]
Locators:
Source IPV4: [REDACTED] Destination IPV4: [REDACTED]
More

24-Oct-2010 06:10:16 webmail event (yahoo), 5 selectors
Active user: [REDACTED] (UserId) [REDACTED] AccountOwner [REDACTED] (MachineId) [REDACTED] (X-huawei-IMSI)

SECRET STRAP1
GCHQ


events

OSN

Blazing Saddles

Query Input | Results [ REDACTED] x | Results [REDACTED] | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x | Results [REDACTED] x

User A role
User A type
User A
User A row value
User A display name
User B type
User B
User B row value
User B display name

15-Jul-2010 21:41:42 - SOCIAL ANIMAL event (Facebook), 2 selectors
Active user: [REDACTED] (Facebook-uid)
Action: chat Action type: message
chat message Facebook-uid [REDACTED] Facebook-uid [REDACTED]
Locators:
Source IPV4: [REDACTED]
More

15-Jul-2010 21:41:42 - SOCIAL ANIMAL event (Facebook), 2 selectors
Active user: [REDACTED] (Facebook-uid)
Action: alias Action type: user
alias user Facebook-uid [REDACTED] email [REDACTED]
Locators:
More

15-Jul-2010 21:41:37 - SOCIAL ANIMAL event (Facebook), 2 selectors
Active user: [REDACTED] (Facebook-uid)
Action: chat Action type: message
chat message Facebook-uid [REDACTED] Facebook-uid [REDACTED]
Locators:
Source IPV4: [REDACTED]
More

15-Jul-2010 21:41:33 - SOCIAL ANIMAL event (Facebook), 2 selectors
Active user: [REDACTED] (Facebook-uid)
Action: chat Action type: message

SECRET STRAP1
GCHQ


events

Blazing Saddles

Looks good, When can I have an account?

  • Santhropoid is currently in the second stage of UAT.
  • We currently have 200 users representing all areas of the business.
  • Aiming to be in a position to release Santhropoid to the masses in early January.

SECRET STRAP1
GCHQ


events

New data sources

Blazing Saddles

  • LUSTRE – new data-source available in MB. Good for North Africa.
  • Source field – This will enable new non-routine data-sources to be added to the QFD's.
    • CNE
    • JTRIG – GLASSBACK data used for test case.
    • COLLATERAL

SECRET STRAP1
GCHQ


events

New Loaders

Blazing Saddles

New loaders deployed to MB and HR Map, improvements to KP.

  • Latency of the data in the QFDs has been greatly reduced, now around 12 hours.
    • Each instance of MB can now ingest 8 billion events per-day (total 32 billion)
  • Some QFDs were previously 1-5 days behind.
  • Query performance during loading has also been improved.

SECRET STRAP1
GCHQ


events

GUIDING LIGHT QFD

Presented by [REDACTED]
(Guiding Light SU)

SECRET STRAP1
GCHQ


events

What is GUIDING LIGHT?

  • New QFD developed in August 2010 by TDB-Events.
  • Primary objective:

"To understand the traffic seen on the Next Gen Events bearers."

SECRET STRAP1
GCHQ


events

What can it do for me?

General Questions:

  • Given a case notation, what are the TDI types that are found on it?
  • Given a TDI type/subset, which bearers produce the highest number of events?
  • What type of traffic is on which bearers and where is it coming from?
  • Which bearers provide the most amount of traffic type x from place y?

SECRET STRAP1
GCHQ


events

Front End Interface

GUIDING LIGHT

From Date To Date
Custom 01-NOV-2010 Custom 01-NOV-2010

Bearer e.g. GWUKC151% 
Event Type e.g. %facebook%

Country Digraphs (using ISO standard)

Query Type
From A to B

Country A
Country B
Min Event Count

Note: The % wildcard character represents 0 or more characters.

Daily Counts Source Types Bearers Countries Event Types Full Profile

SECRET STRAP1
GCHQ


events

Results - Full Profile Query

Results

Sections
  Parameters
  Results
  Event Types
  Bearers
  Countries (From)
  Countries (To)
  Event Type, Country
  Country, Bearer
  Bearer, Event Type
  Daily Counts
  Bearer Counts
  Bearer Pivot
  EventType Count
  EventType Pivot
  Country Counts
  Country Pivot
  SourceType Count
  Source Type Pivot
# Source Type Source Site Descriptor Event Type Origination Country Destination Country Event Count
1 vol Cheltenham EREPO WEB_CSS 2,613,571
2 s3c_presence Cheltenham EREPO Yahoo-B-Cookie US FK 1,817,599
3 s3c_presence Cheltenham EREPO Yahoo-Y-Cookie US FK 1,002,176
4 s3c_presence Cheltenham EREPO Yahoo-B-Cookie US IR 1,281,351
5 s3c_presence Cheltenham EREPO Yahoo-Y-Cookie US IR 82,175
6 host_referer Cheltenham LADPO HOST_AL_URL_S FK 25,236
7 s3c_presence Cheltenham EREPO SIP-INVITE-From PK FK 212,000
8 s3c_presence Cheltenham FSTDO SIP-INVITE-From US FK 197,752
9 Cheltenham EREPO YdJS PK US 167,713
10 s3c_presence Cheltenham EREPO Yahoo Messenger US FK 167,520
11 Cheltenham EREPO YdJS US FK 162,533
12 s3c_presence Cheltenham EREPO SIP-INVITE-From GB FK 145,339
13 host_referer Cheltenham EREPO HOST_REFERER IR 145,000
14 s3c_presence Cheltenham EREPO Yahoo-Messenger PK US 132,238
15 s3c_s Cheltenham EREPO s p US FK 112,434
16 s3c_presence Cheltenham LADPO SIP-INVITE-From PK US 91,239
17 s3c_presence Cheltenham EREPO Yahoo-D-Cookie US XX 92,217
18 s3c_s Cheltenham FSTDO s p C FK 73,711
19 s3c_presence Cheltenham EREPO SIP-INVITE-From PK DE 91,772
20 s3c_presence Cheltenham EREPO SIP-INVITE-From CH FK 82,173
21 s3c_presence Cheltenham EREPO Yahoo-B-Cookie US II 76,272
22 s3c_presence Cheltenham LADPO Yahoo-Y-Cookie US XX 67,226
23 s3c_s Cheltenham EREPO s p PK FK 52,330
24 s3c_presence Cheltenham FSTDO Google-DPFC-Cookie PK FK 54,716
25 s3c_presence Cheltenham EREPO Yahoo Y C US II 43,336
26 s3c_presence Cheltenham EREPO Yahoo B Cookie US GB 43,771
27 s3c_presence Cheltenham EREPO Yahoo-Y-Cookie US GB 41,333
28 search Cheltenham EREPO www.google.com.pk PK FK 62,333
29 s3c_presence Cheltenham EREPO Simbar-SIMBAR-User-Agent US IR 32,252
30 s3c_presence Cheltenham EREPO SIP-INVITE-From M FK 34,232
31 search Cheltenham LADPO www.google.com.pk FK 24,234
32 s3c_presence Cheltenham EREPO DXT_Shopreports-CRC_T-User-Agent US FK 21,100
33 s3c_s Cheltenham FSTDO s p PK US 25,715
34 search Cheltenham EREPO www.jungle.com US IP 25,127
35 Cheltenham EREPO s l CH FK 94,434
36 s3c_s Cheltenham EREPO s l PK DE 31,257
37 s3c_presence Cheltenham EREPO SIP-INVITE-From FR FK 12,376
38 s3c_presence Cheltenham EREPO Facebook User-Cookie US FK 12,246
39 host_referer Cheltenham EREPO HOST_REFERER XX 12,449
40 s3c_presence Cheltenham LADPO Simbar-SIMBAR-User-Agent US FK 14,432
41 host_referer Cheltenham EREPO HOST_REFERER II 12,104
42 s3c_s Cheltenham FSTDO s p NY FK 11,740
43 s3c_presence Cheltenham EREPO Yahoo S P REGISTER From US II 11,510
44 search Cheltenham EREPO clinilab.udum.or.uk PK FK 11,558

SECRET STRAP1
GCHQ


events

Results Pivot: Event Types

Sections
  Parameters
  Results
  Event Types
  Bearers
  Countries (From)
  Countries (To)
  Event Type, Country
  Country, Bearer
  Bearer, Event Type
  Daily Counts
  Bearer Counts
  Bearer Pivot
  EventType Count
  EventType Pivot
  Country Counts
  Country Pivot
  SourceType Count
  Source Type Pivot
Event Type Event Count
Yahoo-B-Cookie 3,370,673
Yahoo-Y-Cookie 2,682,201
WEBLOG 2,519,571
SIP-INVITE-From 874,568
HOST_REFERER 439,609
YMSG 364,400
sip 345,876
Yahoo-Messenger 337,035
www.google.com.pk 72,419
Google-PREFID-Cookie 71,337
Simbar-SIMBAR-User-Agent 58,653
www.google.com 38,747
EXP_Shopperreports-SRS_IT-User-Agent 33,847
Facebook-c_user-Cookie 21,664
Yahoo-SIP-REGISTER-From 16,444
www.bing.com 14,485
clients1.google.com.pk 14,093
SIP-REGISTER-From 14,020
Yahoo-B-Set-Cookie 11,720
DoubleClick-id-Cookie 10,107
Yahoo-Y-Set-Cookie 8,631
MS-MUID-Cookie 7,222
Yahoo-login-Method-Body 6,964
Google-Earth-Tile 6,905

SECRET STRAP1
GCHQ


events

Results Pivot: Countries (From)

Sections
  Parameters
  Results
  Event Types
  Bearers
  Countries (From)
  Countries (To)
  Event Type, Country
  Country, Bearer
  Bearer, Event Type
  Daily Counts
  Bearer Counts
  Bearer Pivot
  EventType Count
  EventType Pivot
  Country Counts
  Country Pivot
  SourceType Count
  Source Type Pivot
Origination Country Destination Country Bearer Event Count
PK US EREPO 450,341
FK EREPO 341,767
DE EREPO 103,742
MY EREPO 3,482
KS EREPO 2,102
IN EREPO 1,704
CN EREPO 779
CH EREPO 668
IL EREPO 245
GF EREPO 133
GB EREPO 128
HK EREPO 73
FR EREPO 47
IP EREPO 18
EU EREPO 14
JF EREPO 12
NO EREPO 8
FS EREPO 8
DE EREPO 6
CA EREPO 4
FD EREPO 2
NK EREPO 2
II EREPO 1
IP EREPO 1
XX EREPO 1
GB FK EREPO 257,239
IR EREPO 2,242
II EREPO 179
XX EREPO 56
US EREPO 13
DE EREPO 1
CH FK EREPO 128,305
IR EREPO 7,9
II EREPO 17
XX EREPO 12
MY FK EREPO 48,190
XX EREPO 27

SECRET STRAP1
GCHQ


events

Recent Enhancements

Data from Bude (RPC)

  • Including data from SWORDPLAY

New fields

  • PDDG
  • SIGAD
  • SSDG

SECRET STRAP1
GCHQ


events

Future Enhancements

Near future:

  • Adding BROAD OAK Targeting data
  • Incorporating MI functionality from REFORMER (where appropriate!)
  • Adding more feeds. (Ongoing)

Longer term:

  • Adding Cipher and eAD MI information
  • Linkage into ARTEMIS (or its successor)

SECRET STRAP1
GCHQ


events

Any Questions

?

SECRET STRAP1
GCHQ