SKYNET: Courier Detection via Machine Learning

Open PDF
Creating agency: NSA National Security Agency (US)
Document date: 2012-06-02
Pages: 20
Genre: slides
Banner:
Classification: TOP SECRET COMINT
Dissemination: REL TO USA, FVEY

A June 2012 NSA research presentation detailing the machine learning methodology behind SKYNET's courier detection system, which applies a Random Forest classifier to 55 million Pakistani phone records to identify individuals exhibiting courier-like behavioral patterns — with one of the highest-scoring results being Ahmed Zaidan, Al Jazeera journalist flagged as a suspected al-Qaeda courier.

Long Description:
This top secret NSA research presentation, dated June 5, 2012 and produced collaboratively by analysts from S2I5, R66F, and T1211, describes the technical methodology underlying SKYNET's automated courier detection capability. The core research question is whether a machine learning classifier trained on a small set of known al-Qaeda Senior Leadership (AQSL) courier selectors can identify behaviorally similar individuals from a population of approximately 55 million Pakistani phone subscribers — without relying on call chaining from known targets. The system extracts up to 80 behavioral features from GSM metadata including unique cell tower visit counts, travel frequency, incoming-to-outgoing call ratios, SIM and handset swapping behavior, and social network characteristics, then uses these features to distinguish couriers from the general Pakistani population. The presentation documents a progression of classifier performance improvements. An initial centroid-based detector achieved an equal error rate of 10-20% in a cross-validation experiment on seven known courier selector pairs. Switching to a Random Forest classifier reduced the false alarm rate to 0.18% at 50% miss rate — a sevenfold improvement over random performance. Adding training data scraped from Anchory intelligence reports containing the keyword "courier" further reduced the false alarm rate to 0.008% (at a 50% miss rate) and gave a 46-fold improvement over random in tasked precision, with 21 of the top 500 scoring selectors being already-tasked known targets. The system was then run via MapReduce against the full 55 million selector dataset, with results assessed both quantitatively and through manual SIGINT review. The presentation's most significant and controversial disclosure is the identification of Ahmed Zaidan — described in the document as a member of al-Qaeda, a member of the Muslim Brotherhood, and a journalist working for Al Jazeera — as the highest-scoring selector among those who traveled between Peshawar and Lahore. Zaidan was at the time Al Jazeera's Islamabad bureau chief, and his travel patterns to conflict zones as a journalist were apparently sufficient for SKYNET's algorithm to flag him as a probable courier. The document does not indicate any corroborating evidence beyond behavioral metadata, raising significant questions about the reliability of algorithmic targeting against a population of 55 million people at a 0.008% false alarm rate — which would still produce thousands of false positives at population scale.
Programs:
SMARTTRACKER, MAPREDUCE, SKYNET

People mentioned:
Zaidan, Ahmad Muaffaq

Countries:
Pakistan, Afghanistan

Tags:
journalism, Al Jazeera, Al-Qaeda, machine learning, courier tracking, behavioral profiling, pattern-of-life, GSM, call detail records, AQSL, courier network, journalist surveillance, IMSI, MSISDN, cell towers, travel analytics, social network analysis, Anchory, targeting

Identifier:
skynet-courier-detection-via-machine-learning

PDF SHA-256:
d64b2a68973e894fb1564d7ee8b691362a97f8812508341464d3d2ded20fa34c

TOP SECRET//COMINT//REL TO USA, FVEY

SKYNET:

Courier Detection via Machine Learning

  • [REDACTED], R66F/JHU
  • [REDACTED], R66F
  • [REDACTED], R66F
  • [REDACTED], T1211
  • [REDACTED], T1211
  • [REDACTED], S2I5
  • [REDACTED], S2I5/TD

June 5, 2012

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Given a handful of courier selectors, can we find others that “behave similarly” by analyzing GSM metadata?

OPR249(I)
SAIDU SHARIF
NDB
JALALABAD
15600
NDB
ARDEZ
640-2°E
Khost
BANNU
Miram Shah, North Waziristan
11770
Wana, South Waziristan
DERA ISMAIL KHAN
13000
OPR217
OPR203
NDB
CHARSADDA
PESHAWAR
OPR205
TARBELA
DAM
HARIPUR
NHAS
OPR210
OPR202
KOHAT
MARI INDUS EAS
OPR217
NDB
DARWALI
OPR203
4992
OPR216
SARGODHA
OPR230
DERA ISMAIL KHAN
OPR218
JHANG SADR
OPR253
7980
OPR251
RANQUI
MULTAN
VEHARI
Abbottabad
Islamabad
Rawalpindi
OPR277
RAWALPINDI
TAXILA
CHAKIALA
NDB
BANIHA
OPR235
ABANDONED
RAHWALI
NDB
SHEIKRUF
Faisalabad
DME
LAHORE
NDB
FAISALABAD
OKARA
Lahore
Sahiwal
Miles 20  40  60  80
Km         100
CEASE

Paths Legend

It's worth noting that:
* we are looking for different people using phones in similar ways
* without using any call chaining techniques from known selectors
* by scanning through all selectors seen in Pakistan that have not left Af/Pak (~55M)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

From GSM metadata, we can measure aspects of each selector's pattern-of-life, social network, and travel behavior

Path Legend
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
Sunday
Monday
Tuesday
Wednesday
Thursday
Friday
Saturday

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

This presentation describes our search for AQSL couriers using behavioral profiling

  • Behavioral Feature Extraction
  • Cross Validation Experiment on AQSL Couriers
  • Preliminary SIGINT Findings

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Counting unique UCELLIDs shows that couriers travel more often than typical Pakistani selectors

density
0.06
0.05
0.04
0.03
0.02
0.01
0.00

0 20 40 60 80 100 120
Number of Unique CellIDs

Group
AQSL Local Comms
AQSL Remote Comms
Seen in Pakistan

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

By examining multiple features at once, we can see some indicative behaviors of our courier selectors

Average Daily Outgoing Voice
10
8
6
4
2
0

Average Daily Incoming Voice
0 2 4 6 8 10

numCell
0
20
40
60
80
100
120

Group
Seen in Pakistan
AQSL Local Comms
AQSL Remote Comms

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Looking at a hierarchical clustering derived from all 80 features, the AQSL groups mostly stay together

Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
AQSL Remote Comms : imsi
AQSL Remote Comms : imsi
AQSL Remote Comms : imsi
AQSL Remote Comms : imsi
Gen Pop: msi
AQSL Local Comms : imsi
AQSL Local Comms : imsi
Gen Pop : imsi
Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : msi
Gen Pop : msi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
Gen Pop : imsi
4 days of collect   AQSL Local Comms : imsi
Gen Pop : imsi

1.0
0.8
0.6
0.4
0.2
0.0

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Now, we'll describe a cross validation experiment on the AQSL selectors that we were provided

  • Behavioral Feature Extraction
  • Cross Validation Experiment on AQSL Couriers
  • Preliminary SIGINT Findings

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Our initial detector uses the centroid of the AQSL couriers to “find other selectors like these”

AQSL Cross-Validation Experiment

  • 7 MSISDN/IMSI pairs
  • Hold each pair out and score them when training the centroid on the rest
  • Assume that random draws of Pakistani selectors are nontargets
  • How well do we do?
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)

* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Our initial detector uses the centroid of the AQSL couriers to “find other selectors like these”

AQSL Cross-Validation Experiment

  • Initial experiments showed EER in 10-20% range
  • Here, performance is much worse against these nontargets:
    • Seen in Pakistan
    • Not seen outside of Af/Pak
    • Not FVEY selectors
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)

* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Statistical algorithms are able to find the couriers at very low false alarm rates, if we're allowed to miss half of them

Random Forest Classifier

  • 7 MSISDN/IMSI pairs
  • Hold each pair out and then try to find them after learning how to distinguish remaining couriers from other Pakistanis (using 100k random selectors here)
  • Assume that random draws of Pakistani selectors are nontargets
  • 0.18% False Alarm Rate at 50% Miss Rate
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)

better

* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)
* Random Forest(All Raw Features)
* Random Forest (Outgoing Raw Features)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

We've been experimenting with several error metrics on both small and large test sets

100k Test Selectors 55M Test Selectors
Training Data Classifier Features False Alarm Rate at 50% Miss Rate Mean Reciprocal Rank Tasked Selectors in Top 500 Tasked Selectors in Top 100
None Random None 50% 1/23k (simulated) 0.64 (active/Pak) 0.13 (active/Pak)
Known Couriers Centroid All 20% 1/18k
Outgoing 43% 1/27k
Random Forest Outgoing 0.18% 1/9.9 5 1
+ Anchory Selectors Random Forest Outgoing

Random Forest:

  • 0.18% false alarm rate at 50% miss rate
  • 7x improvement over random performance when evaluating its tasked precision at 100

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

To get more training data we scraped selectors from S2I11 Anchory reports containing keyword "courier”

Anchory Selectors

  • Searched for reports containing "S2I11” AND "courier"
  • Filtered out non-mobile numbers and kept selectors with "interesting” travel patterns seen in SmartTracker
Quetta
bad
Borth V tan
ahore

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Adding selectors from Anchory reports to the training data reduced the false alarm rates even further

Anchory Selectors

  • Searched for reports containing "S2I11” AND "courier"
  • Filtered out non-mobile numbers and kept selectors with "interesting” travel patterns seen in SmartTracker
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)

* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)
* Random Forest(All Raw Features)
* Random Forest (Outgoing Raw Features)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

We've been experimenting with several error metrics on both small and large test sets

100k Test Selectors 55M Test Selectors
Training Data Classifier Features False Alarm Rate at 50% Miss Rate Mean Reciprocal Rank Tasked Selectors in Top 500 Tasked Selectors in Top 100
None Random None 50% 1/23k (simulated) 0.64 (active/Pak) 0.13 (active/Pak)
Known Couriers Centroid All 20% 1/18k
Outgoing 43% 1/27k
Random Forest Outgoing 0.18% 1/9.9 5 1
+ Anchory Selectors Random Forest Outgoing 0.008% 1/14 21 6

Random Forest trained on Known Couriers + Anchory Selectors:

  • 0.008% false alarm rate at 50% miss rate
  • 46x improvement over random performance when evaluating its tasked precision at 100

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Now, we'll investigate some findings after running these classifiers on +55M Pakistani selectors via MapReduce

  • Behavioral Feature Extraction
  • Cross Validation Experiment on AQSL Couriers
  • Preliminary SIGINT Findings

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

The highest scoring selector that traveled to Peshawar and Lahore is PROB AHMED ZAIDAN

amabad
walpind
Khost
Miram Shah, North Wazirist
Wana, South Waziristan
Faisalaba
hore

Miles 100
Km 100 200 300

Path Legend
PROB AHMED MUWAFAK ZAIDAN
TIDE Person Number: [REDACTED]
* MEMBER OF AL-QA'IDA
* MEMBER OF MUSLIM BROTHERHOOD
* WORKS FOR AL JAZEERA

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

In the top 500 scoring selectors, 21 are tasked leading us to believe that we're on the right track

amabad
walpind
Shah North
Wana, South Waz n
hore
uetta

Miles 100
Km 100 200 300

Path Legend
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

We have also discovered many untasked selectors with interesting travel patterns

ost
iram Shah, North Waziristan
Wana, South Waziristan
ahore
Quetta
Karachi

Path Legend
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Preliminary results indicate that we're on the right track, but much remains to be done

Cross Validation Experiment:

  • Random Forest classifier operating at 0.18% false alarm rate at 50% miss
  • Enhancing training data with Anchory selectors reduced that to 0.008%
  • Mean Reciprocal Rank is ~1/10

Preliminary SIGINT Findings:

  • Behavioral features helped discover similar selectors with “courier-like" travel patterns
  • High number of tasked selectors at the top is hopefully indicative of the detector performing well “in the wild”
miss probability (%)
99
95
80
60
40
20
5
1
0.1
0.01
1e-04
1e-04 0.01 0.1 1 5 20 40 60 80 95 99
false alarm probability (%)

* Centroid(All Raw Features)
* Centroid(All Normalized Features)
* Centroid(Outgoing Raw Features)
* Centroid(Outgoing Normalized Features)
* Random Forest(All Raw Features)
* Random Forest (Outgoing Raw Features)
amabad
Wana, South Waz
quetta
Miles 100
Km 100 200 300

TOP SECRET//COMINT//REL TO USA, FVEY