Real-Time Regional Gateway Cloud Analytics for Forward Users

Open PDF
Creating agency: NSA National Security Agency (US)
Document date: 2012-05-10
Pages: 33
Genre: slides
Banner:
Classification: TOP SECRET SI
Dissemination: REL USA, FVEY

This document outlines the NSA’s Real-Time Regional Gateway (RT-RG) program, a global cloud-based signals intelligence system designed to provide near real-time data analysis and targeting support for military and intelligence operations. It details the transition to scalable cloud architectures such as GHOSTMACHINE and Cloudbase, enabling massive data retention, exploitation, advanced geolocation, and automated analytics for tracking high-value targets across multiple theaters, including Afghanistan, Iraq, and the Horn of Africa.

Long Description:
This document provides a comprehensive overview of the National Security Agency’s (NSA) Real-Time Regional Gateway (RT-RG) program, a highly classified signals intelligence (SIGINT) platform designed to deliver near real-time intelligence to forward-deployed military and intelligence users. Originating during operations in Iraq and Afghanistan, RT-RG evolved into a global system integrating tactical collection, data analytics, and rapid dissemination of intelligence for counterterrorism (CT), counterinsurgency (COIN), and counter–improvised explosive device (CIED) missions. It emphasizes the system’s operational success in Afghanistan, where it reportedly contributed to thousands of capture/kill operations and provided critical insights for U.S. and allied forces. RT-RG’s core mission is to enhance the speed and precision of SIGINT exploitation, allowing analysts and warfighters to make time-sensitive decisions using live and historical data feeds. The document further explains how RT-RG leverages cloud-based architectures, including NSA’s GHOSTMACHINE and Cloudbase platforms, to handle the exponential growth of intelligence data. It describes a transition from traditional relational databases to scalable distributed computing environments using Apache Hadoop and MapReduce, enabling the ingestion, storage, and analysis of hundreds of terabytes of communications and geospatial metadata. These systems allow for powerful data mining operations such as “patterns of life” analysis, network graphing, co-traveler detection, and automated geolocation of targets across multiple theaters of operation. The architecture supports integration with numerous data feeds (e.g., DRT, JUGGERNAUT, VOICESAIL) and analytic tools like Sharkfinn, SKS, and Panopticon, which provide visualization, text exploitation, and entity extraction capabilities for intelligence analysts. In its later sections, the presentation details RTRG’s global expansion and operational footprint, listing deployments across Afghanistan, Iraq, Bahrain, Germany, Korea, and Africa, as well as partnerships with military and intelligence units such as the U.S. Marine Corps, U.S. Army SIGINT teams, and special operations task forces. The system’s architecture aims to support both tactical and strategic intelligence needs by combining target-centric and network-centric analytics within a unified cloud environment. The document concludes by highlighting RTRG’s future direction — increased integration of human language technology (HLT), content analysis, and convergence with active SIGINT capabilities — underscoring the NSA’s ambition to create a globally connected, data-intensive surveillance infrastructure.
Programs:
RT-RG, GOLDMINER, GEOT, AGENTLOGIC, SHARKFINN, SKS, PANOPTICON, DRT, CIDNE, JUGGERNAUT, LOPERS, TALIS, MATTERHORN, AIRHANDLER, LYCANTHROPE, VOICESAIL, RETURNSPRING, SORTINGLEAD, GHOSTMACHINE, GMROYALE, GMREGAL, GMGEIST, GMIDEA, GMTOTE, GMPEN, GMPAC, GMCALM, GMMATE, GMAPPLE, GMCAVE, GMPEACH, GMDEN, GMMOUNT, GMALAMO, GMLIGHT, GMBALTIC, GMPLACE, GMPARK, GMHALO, GMCARBON, DRAGONFIRE, DOMEEXPOSE, GIANTSQUID, MAINWAY, FURIOUSCHAINSAW, SHARKFURY, NIAGARAFILES, SHARKQUERY, SHARKDOCS, OCTAVE, UTT, BEDDOWN, CHEEKYMONKEY, LAYERCAKE, CYBERTRANS, GMSEMINOLE, GMGULF, GMZILLA, HLT, JEMA, NEVIS, SEDB, CLOUDBASE, MAPREDUCE, CASPORT, WAVELEGAL

People mentioned:
Petraeus, David, McRaven, William

Countries:
Iraq, Afghanistan, Iran, Yemen, Bahrain, Mexico, Germany, North Korea, South Korea, Japan, Australia, Pakistan, Canada, Denmark, France, Great Britain, Netherlands, Norway, New Zealand, Sweden

Tags:
pattern-of-life, open source, SIGDEV, Real-Time Regional Gateway, RT-RG, Cloudbase, Hadoop, BigTable, MapReduce, cloud computing, surveillance architecture, data analysis, metadata analysis, geolocation, geospatial intelligence, GEOINT, target development, co-travelers, meetings analytics, contact chaining, selector development, Structured Knowledge Space, text processing, SMS, SMS translation, high-value targets, counterinsurgency, COIN, Counter-IED, CIED, counterterrorism, Horn of Africa, Persian Gulf, CENTCOM, NSA Georgia, NSA Texas, NSA Colorado, NSA Bahrain, NIOC-Bahrain, Bagram, Kabul, Jalalabad, USS Georgia, USS Florida, U.S. Marine Corps, U.S. Army, U.S. Air Force, SOCOM, Cryptologic Support Team, Cryptologic Services Group, Fusion Cell, tactical SIGINT, forward data centers, data retention, metadata storage, relational databases, Oracle, Publish-Subscribe Messaging, DNR, DNI, data ingest, data enrichment, spatial visualization, heatmaps, event aggregation, storage infrastructure, OpenStack, Red Hat, Java, Google Earth, Near Real-Time Intelligence, maritime, naval intelligence, Straits of Hormuz, GSM, speaker identification, data pipelines, analytic tools, exploitation, data exploitation

Identifier:
Readfiles-042

PDF SHA-256:
a886261d8580fe0cb49995a11adf4fd362924804289305df6e702500c00c84ac

TOP SECRET //SI// REL USA,FVEY

Real-Time Regional Gateway Cloud Analytics for Forward Users

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

[REDACTED]

TOP SECRET//SI//REL USA, FVEY

REAL-TIME REGIONAL GATEWAY


SECRET//REL USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

BLUF

RTRG:

...brings near real-time intelligence to the warfighter

..."grew up" supporting operations in Iraq

...RTRG is now a global architecture

...leveraging the emerging cloud architecture to answer questions we have not been able to do before

2
SECRET//REL USA, FVEY


TOP SECRET//SI//REL USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

RTRG in Afghanistan

(U)

(S//REL)

(S//REL) Area 82 – Bagram Air Base Home of RTRG AF1 & gmTote

  • 20 days data retention from AFPAK
  • 200-600 daily users
  • New upgrades include two systems in Kabul

(TS//REL)

Mission Areas: Tracking high-value targets (HVT), Counter-Insurgency (COIN), Counter-IED (CIED)

Organizations Using RTRG:

  • CSG* Afghanistan
  • U.S. Marine Corps (USMC) 1st and 2nd Radio Battalions
  • U.S. Army SIGINT analysts at BCT* level
  • U.S. Air Force National Tactical Integration
  • Jalalabad Fusion Cell (USMC)
  • S2 TOPI
  • NSA-G SWAN Counternarcotics Team
  • All special operations task forces

(TS//SI//REL)

“RTRG is the most significant SIGINT support to the war fighter in the last decade"

  • General David Petraeus

"USSOCOM has enduring and critical needs for the tools and data that RT-RG provides"

  • Admiral William McRaven

*CST- Cryptologic Support Team
*BCT - Brigade Combat Team
*CSG - Cryptologic Support Group

TOP SECRET//SI//REL USA, FVEY
3


TOP SECRET//SI//REL USA,FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

CSG Afghanistan Statistics

  • In 2011, RTRG in Afghanistan
    • Played a key role in 90% of all SIGINT developed operations
    • Leading to 2270 capture/kill operations
    • 6534 enemies killed in action
    • 1117 detainees

4
TOP SECRET//SI//REL USA, FVEY


TOP SECRET//SI//REL USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

RTRG in the Gulf and Horn of Africa

Monitoring Iranian Navy (IRIN) in Straits of Hormuz

QESHM ISLAND
HIDARABI ISLAND
HENGAM ISLAND
BAR-E-LANGEH
FARUR ISLAND
GREATER TUNB
BANI FARUR
ABU MUSA
JASH
USS GEORGIA
SSGN 729

Photo of IRIN vessel from US Navy aircraft, located by RTRG

Supporting CENTCOM Maritime (NAVCENT)
Navy Information Operations Command- Bahrain (NIOC-B)

Missions supported:

  • Iran, Yemen, Persian Gulf
  • Recent successes include monitoring of Iranian naval assets

(TS//SI//REL)

RTRG Afloat on subsurface platform USS Georgia (SSGN-729)

(TS//SI//REL)
Missions supported:

  • Horn of Africa: In first week of mission, system received 31 million GSM events, leading to 10 high-value target voice ID, and 90 tactical tip-offs

TOP SECRET//SI//REL USA, FVEY
5


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

RTRG Global Operations

NSA/CSS COLORADO
TECHNICAL SIGINT
(U//FOUO)

(U//FOUO)

US-2 - Denver
Global Maritime & ELINT

NATIONAL SECURITY AGENCY
UNITED STATES OF AMERICA

US-1 Fort Meade
Mission Assurance

IQ2 Iraq*
COIN, CIED

AF-1,5 Bagram & Kabul
COIN,CIED

(S//REL)

USS Georgia & USS Florida

US-3 JFCOM
Joint Forces Command

BH-1 NIOC** Bahrain
AFRICOM, EUCOM, PACOM

GE-3 Germany**
AFPAK
Continuity-of-Operations

KO-1,2 USFK** Pangyo
PACOM & North Korea

US-8 NSA-Texas
Counternarcotics, Maritime, Mexico & SOUTHCOM Support

Texas Cryptologic Center
NSA/CSS TEXAS

* In draw-down
** ECC - NSA European Technical Center
USFK-U.S. Forces, Korea
NIOC - Navy Information Operations Command

TOP SECRET//SI//REL TO USA, FVEY
6


UNCLASSIFIED//FOR OFFICIAL USE ONLY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Outline

  • RTRG Mission Overview
  • RTRG System: Today and Tomorrow
  • Target-Centric and Network-Centric Cloud Analytics
  • Future Work

7
UNCLASSIFIED//FOR OFFICIAL USE ONLY


SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Current RTRG Architecture

RTRG System

Goldminer
Metadata Search

GeoT
Geospatial Tools

Agent Logic
Alerting Tools

Sharkfinn
Selector Enrichment

SKS
Report / Doc Manager

Panopticon
Target Management

Supporting Tactical Users

Forward Data Centers

Services Layer: Web services, authentication, auditing

Publish and Subscribe Messaging

Oracle relational database & dimensional data model

Ingest and Enrichment Pipeline:
Flexible, high-speed architecture for parser and data processors

Data Feeds:

  • DRT
  • JUGGERNAUT
  • CIDNE
  • LOPERS
  • TALIS/MATTERHORN
  • AIRHANDLER
  • LYCANTHROPE
  • KL
  • VOICESAIL
  • RETURNSPRING
    ... and a growing list of others*

(DNR, DNI collect, tipping and reporting)

A successful architecture for several years. Demand for more data feeds, longer retention, and data-intensive analytics has driven RTRG to seek new solutions

*Based on Afghanistan RTRG data flow

SECRET//SI//REL TO USA, FVEY


SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

RTRG Data Challenges

Current Challenges

  • Data Storage & Retention
    • "Patterns of Life” analysis needs require 6+ months of data from world-wide collection
    • A typical system has capacity for only 4-6 weeks of regional data (~90% user queries are within seven days of “now”)
    • Data Use & Computation
    • Analytic processes should make maximum use of all available data to find small signals
    • Relational databases are unsuited to sophisticated analytics such as correlation and matching
  • Data & Technology Heterogeneity
    • New types of data must be added to the system continually
    • With traditional databases, schema modifications are difficult
    • Exotic data management solutions are difficult to adopt due to limited expertise

SECRET//SI//REL TO USA, FVEY


UNCLASSIFIED//FOR OFFICIAL USE ONLY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Cloud Architectures for Analytics

Google

NATIONAL SECURITY AGENCY
UNITED STATES OF AMERICA

hadoop

Apache

Cloudbase
Scalable BigTable Implementation with Security

Emerging NSA Cloud Reference Architecture is well-suited for developing analytics on intelligence data

Scalable: Distributed file systems and databases are built on clusters of commodity hardware, leveraging open source projects and industrial solutions

Computable: The MapReduce programming model simplifies writing efficient parallel computations that operate over large volumes of data

Flexible: Cloud technologies enable flexible schema and leverage large open-source efforts

UNCLASSIFIED//FOR OFFICIAL USE ONLY


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Scalability & Computation

Data Challenges in AFPAK

Current RTRG (AF1)

  • Current database is 27 terabytes (TB)
  • Retention is ~30 days

Future Cloud enabled system

  • Even a modest cloud system (3 rack) for storage will be at least 125 TB of storage
  • 5x increase in available space
  • Actual retention improvement depends on how the space resources are allocated

Cloud supports more data feeds & more days of historical data

Analytic Challenges from Iraq & AFPAK

  • Many analytics used by RTRG are based on R6 SORTINGLEAD event summaries
  • Event summaries were originally created on relational databases
  • Collection increased dramatically, and a mapreduce implementation was needed
  • For new analytics with present day collection volumes, a practical parallel execution model is crucial

Cloud supports large-scale analytics

TOP SECRET//SI//REL TO USA, FVEY
11


TOP SECRET//SI//REL USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

NSA Cloud Computing Enterprise

  • 5-12 racks commodity hardware
    • 150+ data nodes
    • 16 GB RAM each
  • Apache Hadoop
    • 100s of terabytes of storage
    • Stores 10s to 100s of billions of events
  • NSA Cloudbase
    • 100s of nodes serving BigTable implementation
    • Stores 100s of billions of entries

gmAPPLE
Yakima

gmCAVE
gmPEACH
Utah Language Center

gmDEN
ADF-C

gmMOUNT

gmLIGHT
gmBALTIC
gmPLACE
gmPARK
gmHALO
gmALAMO
gmCARBON

Europe

gmROYALE
gmREGAL
gmGEIST

Iraq

gmIDEA (Camp Victory)

Afghanistan

gmTOTE (Bagram)

Korea & Japan
gmPEN
S. Korea

gmPAC
Misawa, Japan

gmCALM
Camp Humphreys
S. Korea

Australia

gmMATE
(S//REL) (Alice Springs)

GHOSTMACHINE is a data-intensive cloud system with many fielded instances worldwide

Hadoop clusters have been demonstrated with 10s of petabytes and over 10,000 cores

12
Map does not include all GHOSTMACHINE/SiteStore systems
TOP SECRET//SI//REL USA, FVEY


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Today's "Hybrid” Architecture

GTM
GeoT
SKS
PANOPTICON
Applications

Services Layer

Publish and Subscribe Messaging

Relational Database

Ingest and Enrichment

hadoop
Large scale data preprocessing and analysis

Cloudbase
Scalable BigTable Implementation with Security
Massive storage and fast record lookup

GHOSTMACHINE

RTRG and GHOSTMACHINE systems are paired with one another:
MapReduce analytic results are fed back to RTRG relational database

TOP SECRET//SI//REL TO USA, FVEY
13


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Tomorrow's Architecture

User interfaces

  • W WidgetFramework
  • GTM
  • GeoT
  • CHET
  • H&S
  • SKS
  • PANOPTICON
  • Web Tier

Services

  • Application Services
  • Alerting
  • Ingest
    • Real time Enrichment
  • Analytics
    • Streaming-based
  • Analytics
    • Map Reduce-based

Data

  • Data Storage (NFS)
  • Secure Data Access Services
    • Casport, Wavelegal, etc.
  • Relational Store
    • Enrichment, User, & Auxiliary Data
  • Cloudbase
    • Event Data and Analytic Results

Operating System

  • Java VM
  • Java VM
  • Operating System
    • Hardened RH Linux
  • Operating System
    • Hardened RH Linux
  • OpenStack VM
  • Utility Cloud
    • Hardware Layer
  • Data/Analytic Cloud
    • Hardware Layer

Data Storage (HDFS)

The Cloud will bring new data-intensive capabilities, support existing missions, and align RTRG installations with emerging NSA and IC standards

TOP SECRET//SI//REL TO USA, FVEY


UNCLASSIFIED//FOR OFFICIAL USE ONLY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Outline

  • RTRG Mission Overview
  • RTRG System: Today and Tomorrow
  • Target-Centric and Network-Centric Cloud Analytics
  • Future Work

15
UNCLASSIFIED//FOR OFFICIAL USE ONLY


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Analytics Overview

Target Development

Co-Travelers & Meetings
Contact Similarity
High Priority
Untasked Numbers

Graph & Network

Graph Triage
Handset Swap
Target Enrichment & Disambiguation

Beddown
Layercake
Heatmaps

Geo-Spatial

The data-intensive computing capabilities in the system enables a set of graph/network analytics and target development analytics

TOP SECRET//SI//REL TO USA, FVEY
16


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Meeting Target Development Challenges

Past Analysts manually queried multiple, independent repositories, aggregating results in Excel, taking hours or work for search and refinement

Now RT-RG provides a streamlined, integrated workflow saving analyst effort

Tips

Selectors

GIST Report

Detainee Reports

Target Development

Event Type comms.gsm
Event Subtype call.*
Selectors IMSI:12345

NAI Kandahar
?

Submit Query Cancel Query Export To File...

Metadata Search

[REDACTED]
Profile Reports Network Changelog Recommendations
OBJ DEWEY BEACH ASSOCIATE
Bottom Line Pech Valley INS possibly in charge of a "company" sized element. Led the attack
Last Active: (TS//SI//REL TO USA, FVEY) (IMEI)
9 hours ago
Last Location: (TS//SI//REL TO USA, FVEY) (IMEI)
RTRG-Afghanistan Targets 9 hours ago Geo
Processor: AROCC
Last Report: (TS//SI//REL TO USA, FVEY) (MSISDN) view report
5 months ago
VID/SID: 1 year ago

Collaboration

TOP SECRET//COMINT//REL TO U
CST 17 FOB FENTY (PAN): AF IN
(Khaksar)
Thu, 10 May 2012 15:55:20 Zu
TOP SECRET//COMINT//REL TO
NZL//20320108
ASSET: JUGGERNAUT

Geospatial Alerting

Afghanistan/Insurgency:
Taliban Leaders in
Chaman, Pakistan,
Command Insurgent
Operations throughout
Southern Afghanistan-A
SIGINT Assessment (S//
SI//REL TO USA, FVEY)

Afghanistan: Influence of
Chamay on RC South,
Juge-July2010

IIR 6 223 0164 12/DOME
EXPOSE/DRAGON FIRE-
EXPLOITATION OFPAGE 3

Report Text for:
02992282
2017302 AUG 10
FM USJ-800
T O P S E C R E T UMBRA US/UK/CAN/AUS/NZ EYES ONLY
SERIAL: 3/CY/507710-10
SUBJ: Afghanistan/Insurgency: Taliban Leaders in Chaman,
Pakistan, Command Insurgent Operations throughout
Southem
Afghanistan A SIGINT Assessment (S//SI//REL TO
USA, FVEY)

20100119 08) BIMINI
(10020-3AFG)
_Nokia1650

kc taliban ircurg

Enrichment

Outcomes

Geolocation

Capture

TOP SECRET//SI//REL TO USA, FVEY
17


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Target Development with Meetings

Who is at the same UCELLID at the same time?

Manual Process

  • Take your selector and query for every unique location he has been and at what time
  • Query for other selectors who have been at the same places at the same times (impossible or painful)
  • OR compare to another known set of selectors to find overlap (excel / ArcGIS / JEMA) (limiting to what you know)
  • Summary statistics on the matching IMSIs using excel or ArcGIS

Cloud Process

  • Pre-calculates all UCELLID overlaps between tasked selectors
  • Simply query your selector in cloud-generated QFD and view summary statistics

Counts

Meetings
Different Dates
Different Locations

Count

Selectors

TOP SECRET//SI//REL TO USA, FVEY


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Target Development with Co-Travelers*

Is there a pair traveling together?

Manual Process

  • You could use the same manual process from Meetings, however, this would not find co-travelers on different networks
  • Manual comparison of pairs of known selectors is possible with ArcGIS or similar spatial tools - You must know the pairs up front

Cloud Process

  • Measures miles-per-hour (MPH) between tasked selectors as they move around.
  • Low average MPH = co-traveling.
  • Simply query for your selector to view statistics on average MPH, days calculated, etc.

*Also known as "Sidekicks"

TOP SECRET//SI//REL TO USA, FVEY


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Meeting Network Challenges

Past Manually query multiple repositories and build network with Analyst Notebook (ANB) - amount of labor can be prohibitive

Now RT-RG tools exist for contact chaining for selector-to-selector & selector-to-report graphs, with more analytics and tools to come

Southwest Asia/Narcotics:
Senior Traffickers from the AbdulAlim Ishaqzai Narcotics Network Located in HelmandProvince, Afghanistan, After Bahram Chah Raid, 5 November2010 (S//REL TO USA, FVEY)

UIM[REDACTED]
ASKS[REDACTED]
[REDACTED]IF HE
HAS HIS MISSING BAG.

[REDACTED]

[REDACTED]

[REDACTED] GETS
PASSED[REDACTED]
PHONE NUMBER.

Selector-to-Report Graph from Enrichment

Search graph...

Calls:2
Calls:3

[REDACTED] IMSI)

// 2040-06-18 // RC EAST
TIPPER // OBX GIANT
SQUID //

selectors

VF // 2010-06-16 // VHF
N2K TIPPER // OBJ GIANT
SQUID //

VF//2010-06-15//CSG
BAGRAM SELECTORS
TIPPER//OBJ GIANT
SQUID//

calls: 718
PREPAID PREPAID
PREPAID

THIS TACREP IS REL
USA AUS,CAN,DEU,DNK,F
RA,GBR,NLD,NO,NZL,SW
E

OBJ COOPER (OMEGA 10)
[REDACTED]

targets

[REDACTED]IMEI)

[REDACTED]

MAINWAY graph in RTRG UI

TOP SECRET//SI//REL TO USA, FVEY
20


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Graph Analysis with Furious Chainsaw

  • Graph representation is natural for DNR
  • Result is Furious Chainsaw
    • Prototype on Cloudbase
    • Now supports contact chains and trends
    • Will support other graph analytics in the future
  • Triage capability for forward users to complement Enterprise databases
  • Enables chaining and other analytics, provides foundation for graph algorithms

Metadata matrix in Cloudbase supports fast graph traversal

Graph View in Renoir – but many other analytics are possible

TOP SECRET//SI//REL TO USA, FVEY
21


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Graph Triage: Multiple Views

Contact Neighborhood
Ozone Widget

Friends and Family
IMS[REDACTED]
test
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
ANB

Daily Call Trends in Panopticon
PANOPTICON Create Profiles Partners
Details
Relationships
Discussions

Bar
Event Count
100
80
60
40
20
0
0 1 2 3 4 5 6 7 8 9 1011121314151617181920212223
Hour of Day (Zulu)

Top Associates
Ozone Widget

Top 100 Associations

Top 100 Associations | Event Count
[REDACTED] | 81
[REDACTED] | 63
[REDACTED] | 61
[REDACTED] | 59
[REDACTED] | 49
[REDACTED] | 43
[REDACTED] | 37
[REDACTED] | 34
[REDACTED] | 27
[REDACTED] | 24
Showing 1 to 10 of 100 entries
Previous Next

Weekly Call Trends
Ozone Widget

Day of Week
Bar
Event Count
200
150
100
50
0
Mon
Tue
Wed
Thu
Fri
Sat
Sun
Day of Week

DNR graphs in Furious Chainsaw tables in Cloudbase support a wide range of fast queries and analytics

TOP SECRET//SI//REL TO USA, FVEY
22


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Unstructured Data Exploitation

SHARKFINN

  • Selector extraction, normalization, and enrichment
  • Flexible free-text query interface
  • Graph, text, and spreadsheet output formats

SKS Structured Knowledge Space

  • Entity extraction (people, organizations, times, geos)
  • Keyword, faceted, and people search
  • Document clustering
  • Arabic name expansion

sharkfury

  • Integrated data ingest using Niagara Files (NiFi)
  • SharkQuery: search by selector, entity, location, and keyword
  • SharkDocs: query, sharing, and collaboration on user uploaded documents
  • Visualization of results in query overview, table, graph, and map
  • Cloudbase and HDFS for scalable text analytics platform

TOP SECRET//SI//REL TO USA, FVEY


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Target Development - SharkFinn / SKS

SKS

"OBJ SMITHERS" AND OBJECTIVES: OBJ TOPPER
Previous 30 Days

1.Perform keyword search with seed OBJ on report library

OBJECTIVES
OBJ SMITHERS (19)
OBJ TOPPER (19)
OBJ SPRINGFIELD (12)
OBJ APU (6)
OBJ GROUNDSMAN WILLY (6)
OBJ WIGGUM (4)
OBJ MAYFIELD (3)
OBJ NEVIS (1)
OBJ TRAJAN (1)

  1. Finding co-occurring objectives

SELECTORS
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]
[REDACTED]

3.Selector co-occurrence

4.Additional Search & Filtering

SELECTOR
1 [REDACTED]
2 [REDACTED]
3 [REDACTED]
4 [REDACTED]
5 [REDACTED]
6 [REDACTED]
7 [REDACTED]
8 [REDACTED]
9 [REDACTED]
10 [REDACTED]
11 [REDACTED]
12 [REDACTED]
13 [REDACTED]

  1. Correlate found selectors with Octave/UTT and Panopticon
  1. Export selector & Report Graph

TOP SECRET//SI//REL TO USA, FVEY


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Meeting Geolocation Challenges

Past Analysts manually correlated locations using map viewers or spreadsheets, aggregating data from multiple sources

Now Analytics and alerts push target information by subscription

412.001.00078.27161
UCELLIDS
412.001.00076.11402
/01
/10
/16
/22
/23
/24
/25
/26
/27
/28
/29
3/04
3/05
3/06
3/07
3/08
3/09
3/10
3/11
3/12
3/13
3/14
3/15
Day
RT-RG pattern of life analytic for target location cues

tio field
First Event
Last Event

MAHIM
Alükh
Qal'eh-ye
Adam Khan
Vazil
Niazi
Shiwak
Shiv

Analysts are notified by alerts, based on:

  • Geospatial NAI*
  • Tasking status
  • SMS content
  • Selectors, callsigns, frequencies
  • ...

Op ATTENTION INS RC-C Kabul
(PAN):UNID ASW SUICIDE VEST KABUL 3
Thu, 10 May 2012 17:48:12 Zulu
TOP SECRET//COMINT//REL TO USA, AUS,
CAN, GBR, NZL//20320108
ASSET: LOPERS
EVENT TYPE: comms.gsm/call.orig
SIGAD: US-3309
PDDG: K2
SSD: L2
POEI: (Missing)
CASN: GCCA0153L5C0001
SELECTION MODE: STANDARD
COLLECTION AUTHORITY: EO12333
Target Location
Approx. Geo: 42S WD 21111 18789
UCELLID: [REDACTED]
Geo Data Source(s): GSM Geo-Sigdev
4 Item(s)

25
TOP SECRET//SI//REL TO USA, FVEY

  • NAI = Named Area of Interest

TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Target Geolocation with Bed Down

Find the most consistent location of the day's first/last event

First Events
+
Last Events
=
Bed Down

Manual Process - One Selector At a Time

  • Query all events for your selector.
  • Mark first and last events manually.
  • OR
  • Enter in a tool like CheekyMonkey to view gaps in activity.
  • Slow process to do one selector at a time

Cloud Process- All Tasked Selectors

  • Pre-calculates first and last events in local time for ALL selectors.
  • Will calculate estimated Bed Down at query time.
  • Can query multiple selectors in seconds, find common overlap.

TOP SECRET//SI//REL TO USA, FVEY


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Target Geolocation with LayerCake

LayerCake - find geospatial overlap of a set of targets.

Manual Process

  • Query all events for your selectors.
  • Display events spatially on mapping software (impossible to view polygon overlaps).
  • Rasterize and do "raster math” to determine max overlap. (very complex and expensive task in most GIS tools).

Cloud Process

  • Pre-calculates unique locations visited for ALL selectors.
  • Raster heat maps drawn at query time.

TOP SECRET//SI//REL TO USA, FVEY


TOP SECRET//SI// REL USA,FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Spatial Visualization of Data

Using MapReduce, analytics can count and aggregate over large number of daily events to give a multi-resolution spatial visualization of the data

Google Earth

Cloudbase Tables for Geo Data

MapReduce for Binning, Counting

Event Data

Analytic Data Flow

Synoptic View

TS//SI//REL

Call volume map of RTRG (AF1) collection (A day in Dec, 2011)

Fewer DNR events

Fewer events

28
TOP SECRET//SI// REL USA, FVEY


TOP SECRET//SI // REL USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Spatial Visualization of Data

Cloud enabled analytic allows viewing the spatial data at many levels-of-detail

“Meso-scale” View

Detail View – “Mashup” with heatmap

Kabul
Peshawar
Rawalpindi
Hyderabad
Karachi
SEDB
Faisalabad
Amritsar
Lahore
Srinagar

Heatmap of activity of a set of targets

[REDACTED]
ASSOC OF CH
[REDACTED]
NDS D124 DEP CHIEF[REDACTED]
ΜΟΙ ΖΟΝΕ 101 AS
KARZAI
AIDE TO MO
OCT 829
UPS AND INDIV
MOI CT OFFI

وز محشر شفا خواهی تمام مومنین است.
We will arrive on saturday
محمد در عرب سردار دین است چراغ آسمان
شمع زمین است. به فردای قیامت و

And Z the day of judgment
healing desiring all believers is.
We+will+arrive+on+saturday+
[REDACTED]
commander religion is: light
heaven/sky candle land is: to the
resurrection-day R
Directions: To here - From here

MFA
MOD SECRETARIAT
MP
DEVELOPMENT ICW IRGC QF DEPT 409
KHALEQ
MP

Using Cybertrans translation service on SMS messages, integrated with heatmaps

Fewer calls
More calls

TOP SECRET//SI // REL USA, FVEY
29


UNCLASSIFIED//FOR OFFICIAL USE ONLY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Outline

  • RTRG Mission Overview
  • RTRG System: Today and Tomorrow
  • Target-Centric and Network-Centric Cloud Analytics
  • Future Work

30
UNCLASSIFIED//FOR OFFICIAL USE ONLY


TOP SECRET//SI//REL TO USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

RTRG Tomorrow

  • Improved DNI capabilities; focus on convergence
  • Integrating active SIGINT capabilities
  • Increased CT and expeditionary capabilities
  • Better tools for faster analytic development
  • Incorporation of content analysis and HLT capabilities
  • Improved integration between target and population analytics

31
TOP SECRET//SI//REL TO USA, FVEY


TOP SECRET//SI// REL USA, FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

RTRG Planned Cloud Instances

NSA/CSS GEORGIA
FORT GORDON

gmSeminole
NSA Georgia

NAVY INFORMATION
OPERATIONS COMMAND BAHRAIN
NETWARCOM

gmGulf
NIOC-Bahrain

NCR AFGHANISTAN
SEMPER VIGILANS SEMPER PARATOS SEMPER VOLENS

gmZilla
Kabul

TOP SECRET//SI// REL USA, FVEY
32


TOP SECRET//SI//REL USA,FVEY

NATIONAL SECURITY AGENCY
SIGDEV
SIGINT DEVELOPMENT

Summary

  • RTRG has been a successful regional data store and exploitation system for COIN, CIED and other missions
  • Moving to NSA Cloud infrastructure
    • More historical data
    • Deeper analysis using parallel programs
    • Allows for more flexible deployments to IC, DoD service installations
  • Continuing to support advanced analytics for current and future operations

TOP SECRET//SI//REL USA, FVEY
33