ICTR Cloud Efforts

Open PDF
Creating agency: GCHQ Government Communications Headquarters (UK)
Document date: 2011-07-01
Pages: 29
Genre: slides
Banner:
Classification: TOP SECRET STRAP 1
Dissemination: 5EYES

A GCHQ presentation details how the agency developed cloud-based big data analytics to achieve "population scale" surveillance—processing trillions of internet events to build geographic and behavioral profiles on every online identifier worldwide, enabling discovery of unknown targets through pattern analysis rather than traditional keyword searches.

Long Description:
This GCHQ presentation from their ICT Research division describes a sophisticated suite of big data surveillance tools built on cloud computing and MapReduce frameworks to analyze internet activity at planetary scale. The system creates "Geo-Time profiles for every Internet identifier" the agency observes, condensing trillions of surveillance events into billions of individual profiles tracking when and where each email address, IP address, or phone number appears globally. Using techniques borrowed from commercial recommender systems—like Netflix's movie suggestions or Amazon's book recommendations—GCHQ applies the same algorithms to discover "terrorists who like website X also like website Y," building association graphs that reveal communities and patterns invisible to traditional intelligence analysis. The presentation reveals multiple codename programs working in concert: KARMA POLICE correlates IP addresses with website visits, SILVER LINING provides the bulk data infrastructure, and tools like AWKWARD TURTLE create website recommendation graphs based on target browsing behavior. GCHQ emphasizes this represents "target discovery at population scale" using "bulk unselected events—all events for country or world," meaning they're analyzing complete internet traffic for entire nations without pre-selecting targets. The agency explicitly states they process data on individuals not suspected of wrongdoing, using graph analysis and machine learning to identify "unknown targets" by detecting unusual behavioral patterns in the noise of global communications. The presentation demonstrates sophisticated use of graph theory and network analysis, treating the internet as a massive interconnected graph where nodes represent people, devices, websites, or search terms, and edges represent relationships or communications between them. Using open-source academic tools like Apache Spark's Bagel implementation and Carnegie Mellon's Pegasus graph mining system, GCHQ performs what they call "graph componentization"—breaking down massive networks into smaller communities and clusters. They explicitly reference research showing that large networks are "dominated by Giant Connected Component" with a "loosely connected periphery," meaning most internet users form one massive interconnected web, but the interesting intelligence targets often exist in smaller, isolated components at the edges. By identifying these disconnected or loosely connected groups, analysts can focus on communities exhibiting unusual isolation or communication patterns that might indicate operational security by adversary groups. One particularly revealing section describes building "query term graphs" from search engine data, where nodes represent search queries and weighted directed edges connect queries that the same user searched for within five minutes of each other. Using PageRank and Personalized PageRank algorithms (the same mathematics Google uses to rank web pages), GCHQ analyzes these graphs to understand how search behavior clusters around topics of interest. The presentation shows an example analysis seeded on the term "GCHQ" itself, revealing that users who search for GCHQ also frequently search for terms like "Free People Check," "MI6," "MI5," "Foreign & Commonwealth Office," "KGB," and intriguingly, "Wikileaks" (ranking 110th). This meta-analysis of their own agency's search footprint demonstrates how the system can map the conceptual landscape around any topic by analyzing collective search behavior, potentially identifying radicalization pathways, research patterns, or investigative journalism into intelligence agencies. The document is remarkably candid about the experimental nature of these techniques and their limitations. Unlike the confident certainty often projected in intelligence presentations, GCHQ acknowledges these are "fragile" methods requiring "a critical mass of suitably skilled people" and close collaboration between data scientists and operational analysts. They emphasize "it's not about tool development but the development of new (and fragile) data mining techniques," suggesting these systems require constant human interpretation and refinement. The agency notes that automated target discovery "can only be used to effect if it is tied in with analyst knowledge of other patterns of behaviour, possibly geo-related," meaning the algorithms identify candidates but human expertise remains essential for validation. They describe discovering "phone groups of interest undiscoverable by traditional analysis" through behavioral pattern matching, representing genuine operational successes in finding targets who successfully evaded keyword-based surveillance. The presentation's discussion of "Critical Success Factors" reveals the organizational challenges of implementing mass surveillance at this scale. GCHQ identifies four requirements: technical expertise in data mining, analyst understanding of target modus operandi and ability to follow up leads, supporting IT infrastructure (provided by SILVER LINING), and crucially, "bulk access to relevant data sets." This last requirement appears to be a point of internal friction—the repeated mentions of lacking access to CULT WEAVE and the inclusion of a Dilbert cartoon mocking cloud computing buzzwords suggest political and bureaucratic obstacles within the agency. The cartoon, showing a pointy-haired boss demanding cloud computing implementation "so I have something to talk about at the executive meeting," hints at pressure to adopt trendy technologies without necessarily having clear operational justification, though the presentation then pivots to claiming genuine "operational benefit" from their cloud initiatives. The final sections explore what GCHQ calls "Exploratory Data Analysis of Internet/Cyber Events," asking fundamental questions about internet behavior at scale: What is statistically normal? Can they identify regional patterns in internet traffic? Can they fingerprint users based purely on browsing habits? Can they detect malicious sites by analyzing the graph structure of how users reach them? These research questions reveal an agency grappling with information overload, trying to develop mathematical frameworks for understanding global internet behavior so they can identify meaningful anomalies. They propose using techniques like "BadRank" (a play on PageRank) to identify malicious websites by analyzing link patterns from known-bad sites, and discuss analyzing the complete netflow dataset to understand IP-level traffic patterns. The document acknowledges a "gap in understanding of events at Internet Scale," suggesting that even with access to vast surveillance feeds, making sense of the data remains a fundamental challenge. The slides discuss "trillions of events," "population scale," "bulk unselected events," and "all events for country or world" in clinical, technical language, treating mass surveillance as primarily a data engineering challenge. There's no discussion of legal frameworks, privacy implications, or the ethics of building profiles on every internet user globally. The agency frames this as target discovery—finding terrorists and adversaries—but the methodology requires first surveilling everyone to identify the few. The document's matter-of-fact tone about processing "all the events we have for a country" and building profiles for "every Internet identifier we see" reveals an intelligence culture of comprehensive population monitoring where everyone's data feeds the machine learning systems that might, eventually, identify someone worth investigating more closely.
Programs:
EVERYASSOC, PROBABILITYCLOUD, SILVERLINING, CULTWEAVE, KARMAPOLICE, EVERYPOLICE, AWKWARDTURTLE, GORDIANKNOT, SANDIA, BLOODHOUND, FIVEALIVE, NGE, MAPREDUCE

People mentioned:
Adams, Scott, Wright, Peter, Dean, Jeffrey, Ghemawat, Sanjay, Leskovec, Jure, Lang, Kevin J., Dasgupta, Anirban, Mahoney, Michael W., Boldi, Paolo, Bonchi, Francesco, Castillo, Carlos, Donato, Debora, Gionis, Aristides, Vigna, Sebastiano

Countries:
-

Tags:
Dilbert comic, pattern-of-life, complete surveillance, ICTR, ICT Research, big data, cloud computing, MapReduce, population-scale surveillance, bulk collection, unselected collection, target discovery, graph analysis, Geo-Time profiles, behavioral analysis, modus operandi, internet metadata, mass surveillance, data mining, machine learning, graph componentization, community detection, recommender systems, social network analysis, trillions of events, behavioral profiling, exploratory data analysis, cyber intelligence, geolocation, IP correlation, website profiling, search query monitoring, personalized PageRank, network analysis, contact chaining, association scoring, bi-partite graphs, weighted graphs, giant connected component, alternate identifiers, relationship mapping, query flow graphs, browser fingerprinting, malicious site detection, botnets, distributed attack detection, netflow analysis

Identifier:
cloud-developers-exchange-july-2011

PDF SHA-256:
dc12ef88cb5c97cce978357e1c6bdcce4cd4a5768bb51d13dc0a7c095cf5b864

"ICTR Cloud Efforts”

developing "canonical” SIGINT analytics, finding hard targets and exploratory data analysis at scale

Data Mining Research – ICTR, GCHQ
Dr [REDACTED]

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Building a SIGINT toolbox for BIG DATA

  • Cloud analytics for SIGINT canonical operations
    • Aggregation - building Geo-Time profiles for Internet Presence
    • All pairs association – alternate identifiers and Geo Associates
    • Componentisation – identify interesting small or large groups
  • Target discovery at population scale
    • target discovery – discover unknown targets
    • known target communications behaviour – modus operandi (MO)
    • population scale bulk unselected events - all events for country or world
  • Exploratory Data Analysis of Internet / Cyber Events

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


GCHQ Cloud Analytic Development

LET'S IMPLEMENT CLOUD COMPUTING SO I HAVE SOMETHING TO TALK ABOUT AT THE EXECUTIVE MEETING.
TELL THEM WE'RE EVALUATING IT. THAT WAY NEITHER OF US NEEDS TO DO ANY REAL WORK.
Dilbert.com DilbertCartoonist@gmail.com
11-18-09 2009 Scott Adams, Inc./Dist. by UFS, inc.
www.dilbert.com/strips/comic/2009-11-18/

In last few years Data Mining Research at GCHQ have:

  • developed new population scale analytics for multi-petabyte cluster
  • evaluated cloud for data marting, bulk association, graph analytics
  • delivered operational benefit – population scale target discovery

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Geo-Summaries for all Internet presence

  • Building Geo-Time profiles for every Internet identifier we see
  • Discovering targets using Modus Operandi
  • Summarisation of "Geo Pattern of Life" for every Internet identifier
    • Summarises how often each identifier seen in every country per week
    • Massively reduces data volumes (trillions of events to billions of profiles)

Email= [REDACTED]
Seen in: PK 17 times, UK 2 times

Week commencing Seen
29/06/2009 UK,1
06/07/2009 UK,1
12/10/2009 PK,9
19/10/2009 PK,8

Perfect for MapReduce

IP-Geo for all Internet presence

Note scale of resulting profiles ...

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Geo-profiling over all presence events

  • Perfect for MapReduce
    • counting the number of occurrences in a large collection of records
    • "MapReduce: A Flexible Data Processing Tool” Dean and Ghemawat Comms of the ACM January 2010 53(1) pages 72-77
  • The Geo-Time summaries for all target identifiers can be used to answer a number of questions:
    • Where has this target identifier been?
    • Which target identifiers match the following country travel pattern?
    • Do anomalous Geo sightings indicate coordinated activity?
  • When combined with domain knowledge, can be extremely powerful if aggregated over all the data

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


EVERY ASSOC & BotGraph:

bulk pairwise associations
and graph componentization

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Large-scale community detection toolbox

  • All pairwise correlation/association – build your graph
    • EVERY ASSOC for TDI alternate identifier scoring
    • BotGraph for webmail spam – Zhao et al Botgraph [NSDI 09]
    • PROBABILITY CLOUD for handset Geo-Association scoring
  • Graph Componentisation
    • GCHQ MapReduce or Bagel implementation
    • Open source MapReduce implementations (CMU Pegasus)
  • Analysis pattern to identify sub-sets for deeper analysis
    • Simple approach to make sense of huge datasets
    • Detect communities of potential interest from massive datasets
    • Rarely sufficient but essential first step in data volume reduction

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Large networks are dominated by Giant

Connected Component: this can help you

Leskovec, Lang, Dasgupta and Mahoney Community Structure in large networks: Natural cluster sizes and the absence of large well-defined clusters arXiv:0810.1355 (2008)

  • Giant connected component dominates large networks
  • Loosely connected periphery
  • Relatively small number of disconnected small components

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Target Discovery at Population Scale

  • We are describing a target discovery technique based on known target communications behaviour applied to population scale bulk unselected events
  • target discovery – discover unknown targets
  • known target communications behaviour – modus operandi (MO)
  • population scale – all the events we have for a country
  • unselected events - not seeded on targets

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Caveat Emptor

  • Method has shown promise to discover phone groups of interest
    undiscoverable by traditional analysis.
  • “Find adversaries through their behaviour”
  • Initial identification of candidates is pure target discovery
    • not seeded on targets
    • search for behaviour in massive events
  • BUT it can only be used to effect if it is tied in with analyst knowledge of other patterns of behaviour, possibly geo-related.

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Critical Success Factors

  • Technical expertise in data mining (ICTR)
  • Good understanding of target MO and ability to follow up new leads which are generated (Ops CT Analysts)
  • Supporting IT infrastructure (SILVER LINING)
  • Bulk access to relevant data sets (SILVER LINING)
    • ICTR lacks bulk access to CULT WEAVE – had snapshot in 2007
    • There were promising research lines: see SD conference 2007

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Operational Data Mining - Key message

  • A combination of technical data mining experts, SIGINT developers, Operations analysts, appropriate data access and suitable IT is needed to make target discovery happen
  • In our experience to date, it's not about tool development but the development of new (and fragile) data mining techniques by a critical mass of suitably skilled people!
  • There are a set of cloud analytics that should form part of a toolbox but even then their successful application is likely to be as a result of collaboration with analysts

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


KARMA POLICE – correlation between websites and internet IDs

Ian TDI
Graham TDI
Simon TDI
www.goodmaths.com
www.badmaths.com
www.bbc.co.uk
www.google.com
www.hotmail.com
0.6 0.9 0.4 0.8 0.8 0.9 0.5 0.9 0.9 0.4 0.9 0.3

  • Internet ID – IP – Web address: correlation scored on statistics of IP
    • KARMA POLICE QFD from ICTR
    • EVERY POLICE QFD on cloud
  • Internet ID-website correlations form a weighted bi-partite graph
    • Links are weighted by KARMA POLICE correlation scores
    • Example graph showing correlations between Internet IDs and websites

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


AWKWARD TURTLE – Cloud QFD

  • What is a recommender system?
    • Netflix – subscribers who like film X also like film Y
    • Amazon – customers who like book X also like book Y
    • GCHQ – Terrorists who like website X also like website Y
  • MapReduce – vector of TDI scores for every website
    • Vector dot product – “cosine similarity” measure
    • Maximum degree TDI cut-off
    • Target activity is being used as similarity measure
  • Website-website correlations – found previously unknown file hosting

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Recommender Systems

  • We have currently only used very simple techniques
  • Body of active research
    • Netflix prize stimulated :)
  • Interested in seeing more statistical inference and large-scale modelling
    • Potential for long term research
  • Behavioural targeting
    • Cf Google and Yahoo ad serving to subscriber profile

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Query term graph

  • Given a search term, which other search terms are related?
  • Build Query term graph (MapReduce):
    • Nodes are queries
    • Directed edges between nodes if a machine searches for one term then the other within 5 minutes
    • Edge weighted according to frequency of search pattern
  • Boldi, Bonchi, Castillo, Donato, Gionis and Vigna The query-flow graph: Model and applications CIKM 08
  • Gionis Efficient Tools for Mining Large Graphs MLG 10

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Ranking in Query Term graph - PageRank

  • Small component from full query term graph
  • All terms to do with different types of antiques
  • Red nodes are top 5 PageRank scores

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Personalised PageRank (PPR)

  • Red node is seed node – Victorian Card Tables
  • Yellow nodes are top 5 Personalised PageRank scores
  • Nodes with high PR score also score highly with PPR

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Normalised PageRank

  • Want to find nodes with high Personalised PageRank score, q, compared to its PageRank score, p
  • p and q are both (stationary) probability distributions on the same set so KL-divergence comes to mind

$$ KL(q||p) = \sum_i q_i \log \frac{q_i}{p_i} $$

  • We can rank the nodes based on their contribution to this sum, $q_i \log \frac{q_i}{p_i}$
  • This is the Normalised PageRank score

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Normalised PageRank score

  • Red node same seed
  • Yellow nodes are top 5 Normalised PageRank scores
  • Nodes with very high PageRank scores no longer dominate

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Comments on Normalised PageRank

  • Could go N-hops from seed node
    • Have to set pizza node degree limit
    • N-hop with pizza limit is standard contact chaining method
  • Normalised PageRank deals with high degree nodes
    • High degree nodes tend to have high PageRank
    • Must score very highly on PPR to score well in Normalised PR
  • Shown results within small component
  • Evaluate Normalised PR for seed term in Giant Connected Component of Query Term Graph using Bagel

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


“GCHQ” seed query term

Rank Query NPR
1 Free People Check 0.791
2= Jobs At Chanel 0.721
2= Peter Wright (Arabic) 0.721
4 GCSE Bitesize Science 0.670
5 MI6 0.652
20 SKS 0.038
22 Foreign & Commonwealth Office 0.034
37 MI5 0.010
47= MI6 James Bond 0.009
47= MI^ 0.009
47= MI8 0.009
72 KGB 0.008
110 Wikileaks 0.003

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Comments on Query Term Graph

  • Query term graph is very noisy, as are all our Internet Events meta-data graphs
  • Some promising results in finding similar queries but essential that results are interpreted by analysts
  • Large amount of research to do
    • Clustering / Sessionising / ... [lots of commercially motivated work]
    • Query chains – Banana -> Apple different intent to iPod -> Apple
    • Understanding the search behaviour of targets
  • Normalised PageRank insights may be generally useful

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Exploratory Data Analysis of Large-Scale

Internet Events – gap in understanding

  • Relevance to Cyber and SIGINT – what is normal in the statistics of internet behaviour at large scale?
    • Can we measure or model the salient features of large-scale internet communications meta-data?
    • Can we identify behaviours associated with target activity (be that human, machine or collective BotNet activity) that are detectable?
  • GORDIAN KNOT (Network Defence) vs SIGINT feeds
    • Understand the potential of GORDIAN KNOT for Cyber EDA
    • What's the gap between GORDIAN KNOT and SIGINT data?

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Internet/Cyber EDA – FY 11/12

  • Fingerprint web browsing sessions
    • Can we ID a user based on their browsing habits?
  • Is the Internet Regional?
    • Hypothesis: “Internet is becoming more regionalised. Any machines communicating over long distances are of greater interest"
    • Does the data support this?
    • Can we characterise the activity and significance of long distance communications?
    • Applications to Cyber, but also potentially to other Intelligence questions

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Internet/Cyber EDA – FY 11/12

  • Attempt to identify malicious sites in the HTTP graph
  • "BadRank – given set of known "bad" web sites, can we identify associated sites that either point in same direction, or are reached from initial sites
    • Identify loosely connected components – bits that aren't closely tied in by association with Google et al.
    • Subgraph detection - if we have an approximate idea of how a user reaches a malicious web site, can we identify this pattern and similar others in the HTTP graph? [[REDACTED] SANDIA work]

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Internet/Cyber EDA – FY 11/12

  • FIVE ALIVE - carry out EDA on the netflow dataset created by TR-FSP
    • FIVE ALIVE is a bulk store of IP flow records, coupled with some very simple analytics that summarize and visualize IP activity
    • The main challenge here is to deal with the size of the dataset; current work in TR-FSP has revolved around looking at subsets of the data but it would be interesting to work on the dataset as a whole

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


BLOOD HOUND – ICTR-NE

  • Detect electronic attack – aim to detect distributed and automated behaviour
  • Idea from IDA/CCS SCAMP 2009
    • 'Using degree distributions to detect internet traffic anomalies' Scheinerman
  • Detect multiple IPs with same degree:
    • in-degree (distributed hacking/port scanning)
    • out-degree (DDOS/bot tasking)
  • Graph: peak at in-degree ~ 10^1.8 = 63
    • Appears to be some sort of hacking activity
    • Dictionary attack: cycling through range of IPs on network, making 63 GET requests to each
    • Trying 63 combinations of URI, with the intent of getting a MySQL setup script (basic exhaust)

Log10 number of IPs with given in-degree
Log10 in-degree
6 5 4 3 2 1 0
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES


Summary

  • Pattern-based data mining – unknown target discovery
    • Bulk unselected events – population scale – all events for country
    • Operational data mining – hard target discovery – real results
    • Target modus operandi – behavioural based discovery
  • Selector-based data mining - unprecedented scale
    • Relationship scoring within multi-modal communications network
  • Exploratory Data Analysis of Large-Scale Internet Events
    • Gap in understanding of events at Internet Scale
    • How can BIG DATA analytics contribute to Cyber target discovery?

ICT RESEARCH

This information is exempt from disclosure under the Freedom of Information Act 2000 and may be subject to exemption under other UK information legislation. Refer disclosure requests to GCHQ on [REDACTED]

NEXT GENERATION events

UK TOP SECRET STRAP1 5EYES