XKEYSCORE intro

Open PDF +3 other copies
Creating agency: NSA National Security Agency (US)
Document date: 2008-02-25
Pages: 32
Genre: slides
Banner:
Classification: TOP SECRET COMINT
Dissemination: REL TO USA, AUS, CAN, GBR, NZL
Part of an assembly: XKEYSCORE intro, 25 February 2008
O Globo ↗2013-07-07
O Globo published three of the slides (6, 11, 20) as images, with slide 6 cropped, and the Google Maps screenshot from slide 20 as a separate image. The Guardian's copy does not show that screenshot; The Intercept's 2015 copy has it laid over the slide text.
The Guardian published two slides ("Discovery of new target web services" and "Entity Extraction") The Intercept didn't publish. The Guardian's version is overall more redacted.
the copy shown on this pagedocument page ↗original PDF ↗
The pages reprinted in the document appendix of No Place to Hide (appendix pages 68-69).
The Intercept's version is the least censored, but it's missing two slides that the Guardian published. Some of the slides have screenshots on top of text, but the texts are visible in the Guardian's version.

A TOP SECRET NSA presentation from 2008 introducing XKeyscore — one of the earliest overview documents in the archive — describing the system as a globally distributed Linux cluster across approximately 150 sites and 700+ servers that stores a ~3-day rolling buffer of all unfiltered internet traffic, and demonstrating its core capabilities for finding previously unknown targets by mining metadata, tracking documents by embedded image hashes, detecting encryption and VPN use, and language tracking.

Long Description:
This is a TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL NSA presentation dated 25 February 2008 from the xkeyscore@nsa team. The presentation defines XKeyscore as a DNI (Digital Network Intelligence) exploitation system and analytic framework performing both strong selection (email addresses) and soft selection (content), providing real-time target activity tipping, and maintaining a "rolling buffer" of approximately 3 days of all unfiltered data it sees — full-take data stored at the collection site and indexed by metadata. The federated query system allows one query to scan all sites simultaneously, and performing full-take collection lets analysts find previously unknown targets by mining the metadata. The system details describe a massive distributed Linux cluster of over 500 servers (later in the same presentation stated as over 700 servers across approximately 150 sites) that scales linearly by adding servers. The query hierarchy flows from user queries through the XKeyscore web server to F6 HQS (Special Collection Service), FORNSAT sites and SSO sites, and onward to individual F6 sites. The presentation's conceptual core is the "why go deep" argument. Strong selection alone provides limited capability because much web activity is anonymous, but this anonymous traffic can be used to detect anomalies that lead to intelligence directly or to strong selectors for traditional tasking. The plug-ins that extract and index metadata include Email Addresses, Extracted Files, Full Log, HTTP Parser, Phone Number and User Activity. An HTTP Parser example shows a Google search for "islamabad" captured from a Pakistani IP with no username or strong selector present — illustrating how the system captures anonymous web activity. The capabilities demonstration is extensive and reveals the breadth of XKeyscore's discovery functions. Finding targets works by looking for anomalous events — someone whose language is out of place for their region, someone using encryption, or someone searching the web for suspicious material. The encryption slide shows capturing password-protected documents. Technology detection allows queries like "show me all the VPN startups in country X, and give me the data so I can decrypt and discover the users" — with the note that no other system performs this on raw unselected bulk traffic because data volumes prohibit forwarding. Persona Session Collection is described in both traditional form (triggered by a strong selector) and "Reverse PSC" (working from an anomalous event back to a strong selector), which is impossible when data has already been strong-selected. Language tracking answers "my target speaks German but is in Pakistan" by extracting and storing all HTML language tags. The Google Maps capability captures web-based searches and map locations. The document tracking capabilities are especially significant. XKeyscore extracts and stores document authoring metadata (Author, Organization, dates) for retrospective surveys tracing document origins, with metadata typically kept up to 30 days. All images inside documents are hashed so analysts can search for anyone who received or transmitted a particular document — noted as "really useful for company logos." The worked examples show intercepted Iraqi documents including a Biographical Data Sheet with an image hash highlighted, an Iraqi government administrative document about penalties for officials, and a PROACTIVE Communications Inc. ICZN closeout document listing network equipment (Cisco PIX 501 firewalls, BorderGuard devices, Dell equipment) at what appear to be Iraqi installation sites. The TAO Detection slide shows that TAO fingerprints are loaded into XKeyscore's fingerprint engine to answer "show me all the exploitable machines in country X." The success stories section makes notable claims. The headline claim is "over 300 terrorists captured using intelligence generated from XKeyscore." The TRAFFICTHIEF integration provided near-real-time tips to counterterrorism operations in coordination with coalition forces in Iraq 24 hours a day, producing hundreds of confirmed alerts per day on over 3,000 user accounts, with an example of coalition forces detaining individuals in 2004. A May 2006 SIDToday-sourced success story describes WealthyCluster2 and XKeyscore installation at a redacted location connected to MOONSHINE, enabling processing of wireless collection, near-real-time tipping and full-take SIGDEV — geolocating previously un-locatable cafés (described as "a Goldmine"), acquiring targets, providing NSA/Georgia tips with precise locations to JSOC, and leading to terrorist captures. The innovation and future slides list high-speed selection, MARINA integration, GPRS/WLAN integration, SSO CRDB, workflows and multi-level dictionaries as recent innovations, with future plans including Cell Processor (R4) acceleration, entity extraction, VoIP, EXIF tag extraction, expanded Google Earth capability and integration of all CES-AppProcs. The document is significant as the foundational statement of XKeyscore's purpose and the earliest dated articulation of its "full-take" philosophy — capturing everything and using metadata mining to find targets who left no strong selector. The document tracking via embedded image hashing, the reverse persona session collection concept, and the language-out-of-place detection technique are all capabilities that distinguish XKeyscore from selector-based collection systems. The operational success claims tie XKeyscore directly to detentions in Iraq, connecting the bulk collection system to kinetic counterterrorism outcomes.
Programs:
XKEYSCORE, TURMOIL, TURBULENCE, TRAFFICTHIEF, WEALTHYCLUSTER, MOONSHINE, SEARCHLIGHT, MARINA, PSC, SSO, TIMBERLINE

People mentioned:
bin Laden, Osama

Countries:
Germany, Pakistan, Iraq, Iran, China, Poland

Tags:
DNI, full-take collection, rolling buffer, federated query, strong selectors, soft selection, persona session collection, anomaly detection, encryption detection, VPNs, language identification, document tracking, hashes, EXIF, Google Maps, phone numbers, email addresses, user activity, extracted files, Tailored Access Operations, FORNSAT, SSO, Linux, Islamabad, coalition forces, JSOC, Cisco, BorderGuard, biographical data, AppProc, SIGDEV, IAEO

Identifier:
EFF_-_20130731-guard-xkeyscore_training_slides
Also known as:
nsa-xkeyscore
NSA_XKeyscore_Powerpoint
ACLU_-_NSA_XKeyscore_Powerpoint
nsa-pdfs-redacted-ed

PDF SHA-256:
f9148393957a3efb8bbe7fefe66eb94e06ea8df977a26407f48dfab186fca9e3

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKEYSCORE

25 Feb 2008
xkeyscore@nsa

DERIVED FROM: NSA/CSSM 1-52
DATED: 20070108
DECLASSIFY ON: 20320108

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What is XKEYSCORE?

  1. DNI Exploitation System/Analytic Framework
  2. Performs strong (e.g. email) and soft (content) selection
  3. Provides real-time target activity (tipping)
  4. "Rolling Buffer" of ~3 days of ALL unfiltered data seen by XKEYSCORE:
    • Stores full-take data at the collection site - indexed by meta-data
    • Provides a series of viewers for common data types
  5. Federated Query system – one query scans all sites
    • Performing full-take allows analysts to find targets that were previously unknown by mining the meta-data

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Methodology

  • Small, focused team
  • Work closely with the analysts
  • Evolutionary development cycle (deploy early, deploy often)
  • React to mission requirements
  • Support staff integrated with developers
  • Sometimes a delicate balance of mission and research

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

System Details

  • Massive distributed Linux cluster
  • Over 500 servers distributed around the world
  • System can scale linearly - simply add a new server to the cluster
  • Federated Query Mechanism

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Query Hierarchy

User Queries
Query

XKEYSCORE web Server
Query Query Query

F6 HQS

Query Query

F6 Site 1
F6 Site 2

FORNSAT site

SSO site

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Where is X-KEYSCORE?

Approximately 150 sites
Over 700 servers

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What is unique about XKEYSCORE?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

General Capability

Processing Speed
Processing Depth

TURMOIL/TURBULENCE
XKEYSCORE

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Why do shallow

  • Can look at more data
  • XKEYSCORE can also be configured to go shallow if the data rate is too high

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Why go deep

  • Strong Selection itself give us only a very limited capability
  • A large amount of time spent on the web is performing actions that are anonymous
  • We can use this traffic to detect anomalies which can lead us to intelligence by itself, or strong selectors for traditional tasking

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What XKS does with the Sessions

Plug-ins extract and index metadata into tables

[sessions] -> [processing engine] -> (database) -> (user queries)

Session
phone numbers
email addresses
log ins
user activity

Database
metadata tables
full log

QUERY

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Plug-ins

Plug-in DESCRIPTION
E-mail Addresses Indexes every E-mail address seen in a session by both username and domain
Extracted Files Indexes every file seen in a session by both filename and extension
Full Log Indexes every DNI session collected. Data is indexed by the standard N-tupple (IP, Port, Casenotation etc.)
HTTP Parser Indexes the client-side HTTP traffic (examples to follow)
Phone Number Indexes every phone number seen in a session (e.g. address book entries or signature block)
User Activity Indexes the Webmail and Chat activity to include username, buddylist, machine specific cookies etc.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What Can Be Stored?

  • Anything you wish to extract
    • Choose your metadata
    • Customizable storage times
    • Ex: HTTP Parser
[REDACTED]

GET /search?hl=en&q=islamabad&meta= HTTP/1.0
Accept: image/git, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-
application/msword, application/x-shockwave-flash, */*
Referer: http://www.google.com.pk/
No username/strong selector
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Host: www.google.com.pl
[REDACTED]
Connection: keep-alive

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What can you do with XKEYSCORE?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Finding Targets

  • How do I find a strong-selector for a known target?
  • How do I find a cell of terrorists that has no connection to known strong-selectors?
  • Answer: Look for anomalous events
    • E.g. Someone whose language is out of place for the region they are in
    • Someone who is using encryption
    • Someone searching the web for suspicious stuff

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Encryption

  • Show me all the encrypted word documents from Iran
  • Show me all PGP usage in Iran
    • Once again – data volume too high so forwarding these back is not possible
    • No strong-selector
    • Can perform this kind of retrospective query, then simply pull content of interest from site as required

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Technology Detection

  • Show me all the VPN startups in country X, and give me the data so I can decrypt and discover the users
    • These events are easily browsable in XKEYSCORE
      • No strong-selector
    • XKEYSCORE extracts and stores authoring information for many major document types – can perform a retrospective survey to trace the document origin since metadata is typically kept for up to 30 days
  • No other system performs this on raw unselected bulk traffic, data volumes prohibit forwarding

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Persona Session Collection

  • Traditionally triggered by a strong-selector event, but it doesn't have to be this way
  • Reverse PSC - from anomalous event back to a strong selector. You cannot perform this kind of analysis when the data has first been strong selected.
  • Tie in with Marina - allow PSC collection after the event

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Language Tracking

  • My target speaks German but is in Pakistan – how can I find him?
    • XKEYSCORE's HTTP Activity plugin extracts and stores all HTML language tags which can then be searched
    • Not possible in any other system but XKEYSCORE, nor could it be -
      • volumes are too great to forward
      • No strong-selector

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Google Maps

  • My target uses Google Maps to scope target locations – can I use this information to determine his email address? What about the web-searches – do any stand out and look suspicious?
    • XKEYSCORE extracts and databases these events including all web-based searches which can be retrospectively quried
    • No strong-selector
    • Data volume too high to forward

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Document Tracking

  • I have a Jihadist document that has been passed around through numerous people, who wrote this and where were they?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


[REDACTED]


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Interesting Document Discovery

  • Show me all the Microsoft Excel spreadsheets containing MAC addresses coming out of Iraq so I can perform network mapping.
    • New extractor allows different dictionaries to run on document/email bodies – these more complex dictionaries can generate and database this information.
    • No strong-selector
    • Data volume is high
    • Multiple dictionaries targeted at specific data types

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

TAO

  • Show me all the exploitable machines in country X
    • Fingerprints from TAO are loaded into XKEYSCORE's application/fingerprintID engine
    • Data is tagged and databased
    • No strong-selector
    • Complex boolean tasking and regular expressions required

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Discovery of new target web services

  • New web services every day
  • Scanning content for the userid rather than performing strong selection means we may detect activity for applications we previously had no idea about

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Entity Extraction

  • Have technology (thanks to R6) – for English, Arabic and Chinese
  • Allow queries like:
  • Show me all the word documents with references to IAEO
  • Show me all documents that reference Osama Bin Laden
  • Will allow a 'show me more like this' capability

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKEYSCORE Success Stories

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Over 300 terrorists captured using intelligence generated from XKEYSCORE

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


[REDACTED]


[REDACTED]


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Innovation

  • High Speed Selection
  • Toolbar
  • Integration with Marina
  • GPRS, WLAN integration
  • SSO CRDB
  • Workflows
  • Multi-level Dictionaries

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Future

  • High speeds yet again (algorithmic and Cell Processor (R4))
  • Better presentation
  • Entity Extraction
  • VoIP
  • More networking protocols
  • Additional metadata
    • Expand on google-earth capability
    • EXIF tags
    • Integration of all CES-AppProcs
  • Easier to install/maintain/upgrade

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL