← XKEYSCORE intro

XKEYSCORE intro copy: xks-intro

Another published copy of this document that differs from the one on the document page - for example less redacted, with extra pages, or a different scan.

Download this copy (PDF) All pages (30)

Pages that differ from the main document: 13, 16, 20, 21, 22, 23, 27, 28

The Intercept 2015-07-01
XKEYSCORE: NSA’s Google for the World’s Private Communications article ↗
The Intercept's version is the least censored, but it's missing two slides that the Guardian published. Some of the slides have screenshots on top of text, but the texts are visible in the Guardian's version.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKEYSCORE

25 Feb 2008
xkeyscore@nsa

DERIVED FROM: NSA/CSSM 1-52
DATED: 20070108
DECLASSIFY ON: 20320108

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What is XKEYSCORE?

  1. DNI Exploitation System/Analytic Framework
  2. Performs strong (e.g. email) and soft (content) selection
  3. Provides real-time target activity (tipping)
  4. "Rolling Buffer" of ~3 days of ALL unfiltered data seen by XKEYSCORE:
    • Stores full-take data at the collection site - indexed by meta-data
    • Provides a series of viewers for common data types
  5. Federated Query system – one query scans all sites
    • Performing full-take allows analysts to find targets that were previously unknown by mining the meta-data

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Methodology

  • Small, focused team
  • Work closely with the analysts
  • Evolutionary development cycle (deploy early, deploy often)
  • React to mission requirements
  • Support staff integrated with developers
  • Sometimes a delicate balance of mission and research

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

System Details

  • Massive distributed Linux cluster
  • Over 500 servers distributed around the world
  • System can scale linearly - simply add a new server to the cluster
  • Federated Query Mechanism

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Query Hierarchy

User Queries
Query

XKEYSCORE web Server
Query Query Query

F6 HQS

Query Query

F6 Site 1
F6 Site 2

FORNSAT site

SSO site

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Where is X-KEYSCORE?

Approximately 150 sites
Over 700 servers

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What is unique about XKEYSCORE?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

General Capability

Processing Speed
TURMOIL/TURBULENCE
XKEYSCORE

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Why do shallow

  • Can look at more data
  • XKEYSCORE can also be configured to go shallow if the data rate is too high

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Why go deep

  • Strong Selection itself give us only a very limited capability
  • A large amount of time spent on the web is performing actions that are anonymous
  • We can use this traffic to detect anomalies which can lead us to intelligence by itself, or strong selectors for traditional tasking

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What XKS does with the Sessions

Plug-ins extract and index metadata into tables

[sessions] -> [processing engine] -> (database) -> (user queries)

Session
phone numbers
email addresses
log ins
user activity

Database
metadata tables
full log

QUERY

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Plug-ins

Plug-in DESCRIPTION
E-mail Addresses Indexes every E-mail address seen in a session by both username and domain
Extracted Files Indexes every file seen in a session by both filename and extension
Full Log Indexes every DNI session collected. Data is indexed by the standard N-tupple (IP, Port, Casenotation etc.)
HTTP Parser Indexes the client-side HTTP traffic (examples to follow)
Phone Number Indexes every phone number seen in a session (e.g. address book entries or signature block)
User Activity Indexes the Webmail and Chat activity to include username, buddylist, machine specific cookies etc.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What Can Be Stored?

  • Anything you wish to extract
    • Choose your metadata
    • Customizable storage times
    • Ex: HTTP Parser
FM IP 58.[REDACTED] TO IP 64.[REDACTED]

GET /search?hl=en&q=islamabad&meta= HTTP/1.0
Accept: image/git, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-
application/msword, application/x-shockwave-flash, */*
Referer: http://www.google.com.pk/
No username/strong selector
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Host: www.google.com.pl
Cookie: PREF=ID=678Tb0a34384e2f6:TM=1168503483:LM=1168503483:s=KKzZb3kPcw4vNxGt
Via: 1.0 proxy: [REDACTED]:8080 (squid/2.5.STABLE13)
X-Forwarded-For: 58.[REDACTED]
Cache-control: max-age=239200
Connection: keep-alive

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What can you do with XKEYSCORE?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Finding Targets

  • How do I find a strong-selector for a known target?
  • How do I find a cell of terrorists that has no connection to known strong-selectors?
  • Answer: Look for anomalous events
    • E.g. Someone whose language is out of place for the region they are in
    • Someone who is using encryption
    • Someone searching the web for suspicious stuff

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Encryption

  • Show me all the encrypted word documents from Iran
  • Show me all PGP usage in Iran
    • Once again – data volume too high so forwarding these back is not possible
    • No strong-selector
    • Can perform this kind of retrospective query, then simply pull content of interest from site as required
Enter Password
Password
Confirm
OK
Cancel
Help

بسم الله الرحمن الرحيم
لسيد نایب رییس لوزراء ان ه ترم
لسید هدی و ل م ک ت اب ان محترم
انا موطن و الصال اي ان ان اداری و و الام التي و
ال کتب من الدرجة ان اولو فم دین.
ایف لا ان م و نقي ن د جلال ع راق وفق القانون السفر و ال یف اد
الن ل ذ ومن الدرجة ال اولو ف م ادون.
م ع القلب: از مو کی ن ب ال عقوبات ال لا ربية المنصوص علیه فی
قانون ريض ب الموظف فی الدولة ل م رق و 14 لسنة 1991 وب ع د
م و فن العید لن اب .
ا . عقوبة لفت النار
ب عقوب: ال ان ذار
ج عقوب: قطع الراتب لمدة لا ترید عن ثل الثلثة في ام
التوقيع علی ان او ام و از ادرية ان هر آب از ببشرة و ل ارفك الله
وبراءة القمة لقص آ ب ان دي في ن م ن ان درجة ال اولی نام ادون.
الهاتان لي ان انتهم ان ي ة :
التي منح المك في ان وال د لي ب ان م السید ال نایب ال می کرم
ب م ال اي نتج او ز حدود م ب ال غد | (150000) م ی ة ی ص ون الف دینار
نا م ن ك بي ال مكتب .
التوقيع علی ق و ا م ل مشتريات و فی انتي ر و الغرب ت التي

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Technology Detection

  • Show me all the VPN startups in country X, and give me the data so I can decrypt and discover the users
    • These events are easily browsable in XKEYSCORE
      • No strong-selector
    • XKEYSCORE extracts and stores authoring information for many major document types – can perform a retrospective survey to trace the document origin since metadata is typically kept for up to 30 days
  • No other system performs this on raw unselected bulk traffic, data volumes prohibit forwarding

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Persona Session Collection

  • Traditionally triggered by a strong-selector event, but it doesn't have to be this way
  • Reverse PSC - from anomalous event back to a strong selector. You cannot perform this kind of analysis when the data has first been strong selected.
  • Tie in with Marina - allow PSC collection after the event

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Language Tracking

  • My target speaks German but is in Pakistan – how can I find him?
    • XKEYSCORE's HTTP Activity plugin extracts and stores all HTML language tags which can then be searched
    • Not possible in any other system but XKEYSCORE, nor could it be -
      • volumes are too great to forward
      • No strong-selector

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Google Maps

  • My target uses Google Maps to scope target locations – can I use this information to determine his email address? What about the web-searches – do any stand out and look suspicious?
    • XKEYSCORE extracts and databases these events including all web-based searches which can be retrospectively quried
    • No strong-selector
    • Data volume too high to forward

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Document Tracking

  • I have a Jihadist document that has been passed around through numerous people, who wrote this and where were they?
General | Description | User Defined | Internet | Statistics
Title بسم الله الرحمن الرحيم
Subject:
Keywords
Comments
Type:** MS Word document
Location:
Size:
Created:
Modified:
Digitally signe
Last printed:
Total editing t
Revision numb
Apply use
Template:

عليه افي
199 و ب ع د
... و الى انفكاك
م ادون.
الي منح المك ف ي ت و ال لي اب امن م السيد الان ای ب ال محترم
ب م ال اي تج او زحدود م ل غ ها (150000) م مئة وخص ون لف في نار
ل م ن س بي ال مكتب .
نتي التوقيع على قوائم المشتريات و الفواتي رو نطلب ت

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Document Tracking

  • All images are hashed in the metadata so that you can search for anyone who has received or transmitted this document.
  • This is really useful for company logos.
Biographical Data Sheet
معلومات شخصية
Note: Fill the blocks in English & Type - Submit electronically via e-mail.
ملاحظة : ملأ الاستمارة الانكليزية باستخدام نامج الوورد وارسلها بالبريد الالکتروني
First Name الاسم الاء [REDACTED
1st Middle (father's) Name*اسم الاب [REDACTED]
2nd Middle (Grandfather) Name اسم الجد [REDACTED]
Family / Tribal Name [REDACTED]
Aliases/Nickname [REDACTED]
Mother's Name اسم الام [REDACTED]
Mother's Tribal Name [REDACTED]
<hash> 5c24dabb15f0c5d64dcf7[REDACTED]<hash>
Birth Date تاريخ الميلاد
DD/يوم 17
MM/شهر [REDACTED]
YY/سنة [REDACTED]
City/Country of Birth مكان الولادة الدولة - المدينة البلدة
Iraq-kut-dor almoalmean
Nationality الجنسية
Iraqi
Race القومية
Arab
Religion الديانة
Muslim
Gender (Male/Female) الجنس
male
Marital Status (Single/Married/Divorced) الحالة الزوجية
single
Number of children عدد الاطفال
non
<lastauthor>[REDACTED]</
<title>Biographical Da
<language>Arabic</la
– <comment>
    <METADATA_RO
    <Language>Arabic<
    </DateItem> <Proce
    <Processing> >Nam
    </METADATA_RO
  </document_metadata>
</xks_meta>

from

  • No s

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Interesting Document Discovery

  • Show me all the Microsoft Excel spreadsheets containing MAC addresses coming out of Iraq so I can perform network mapping.
    • New extractor allows different dictionaries to run on document/email bodies – these more complex dictionaries can generate and database this information.
    • No strong-selector
    • Data volume is high
    • Multiple dictionaries targeted at specific data types
PROACTIVE Communications, Inc.
ICZN Closeout Document
Site Information
Site Name/City: [REDACTED] عرفی مدل تاج ول عام
Function (IPS, PJOC, POL, ...): [REDACTED] العملیة الشارقة
GPS Coordonates: [REDACTED]
Site Pre-Commissioned Date: [REDACTED]
Site Commissioned Date: [REDACTED]
Meters of CAT 5 Used: 300 Meters

PCI Deliverable Equipment
Switch: [REDACTED] دواعي التحول
NetModem: [REDACTED]
Pix 501 Firewall: [REDACTED]

Ethernet Switch Serial Number: [REDACTED] المحول
# of Ports / Make / Model: [REDACTED]

Serial Number: [REDACTED] المحول
Model: [REDACTED]
Telephone Number: [REDACTED] رقم الهاتف

Government Furnished Equipment
BorderGuard Serial Number: [REDACTED] رقم المسلسل
Model: [REDACTED]

Dell Model Type: Serial Number: [REDACTED]
Monitor Type: Serial Number: [REDACTED]

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

TAO

  • Show me all the exploitable machines in country X
    • Fingerprints from TAO are loaded into XKEYSCORE's application/fingerprintID engine
    • Data is tagged and databased
    • No strong-selector
    • Complex boolean tasking and regular expressions required

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKEYSCORE Success Stories

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Over 300 terrorists captured using intelligence generated from XKEYSCORE

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKEYSCORE and TRAFFICTHIEF

  • Customer: CounterTerrorism (CT)
  • Provides near real-time tips to TRAFFICTHIEF server in operations in coordination with coalition forces in Iraq 24 hours a day
  • Currently producing hundreds of confirmed alerts per day on over 3000 user accounts

Afternoon of [REDACTED] 2004 – coalition detained individuals below:

[REDACTED] [REDACTED]

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKEYSCORE Success

May 2006, WealthyCluster2 and X-KEYSCORE Installed at [REDACTED]

  • Connected to Moonshine
  • Enabled processing of wireless collection
  • Enabled near-real-time tipping
  • Enabled full-take SIGDEV

Un-locatable cafés were geolocated:

  • [REDACTED] – “A Goldmine”
  • Four Other Cafés Being Developed

Acquired important targets:

  • NSA/Georgia Tips With Precise Locations
  • JSOC Tools In New [REDACTED]
  • Reacquired [REDACTED] Lost When Zarkanet Went Down

Terrorists were captured:

  • Members of the [REDACTED]
  • Members of the [REDACTED]
SIDtoday - Microsoft Internet Explorer

SIDtoday
THE INSIDE SCOOP
30 2006
ARCHIVES    FEEDBACK    SIDTODAY - HOME

SIDtoday

(TS//SI) Breakthrough: Previously 'Unfindable' Internet Cafes in
Iraq Can Now Be Located

FROM: SIGINT Communications (S02C) and
[REDACTED] Iraq Wireless Team (SSG211)

Wednesday, July 23, 2006

(TS//SI) Terrorists who visit Internet cafes associated with the [REDACTED] can now
potentially be located and arrested.

(TS//SI) There are a number of terrorists now in prison whose last moments of freedom went
something like this: Each terrorist woke up in the middle of the night, finding his bed
surrounded by U.S. troops. As he was detained and dragged out the door of his safehouse, he
may have wondered, "How did they know I was here?"

(TS//SI) Answer: SIGINT had detected the terrorist's activity in an Internet café earlier that day
and tipped his location to the appropriate forces in Iraq within minutes of his activity. Then,
through various means, the target was tracked from the café to his safehouse, leading our
forces not only to him, but also to other members of his terrorist network staying with him.
(See an earlier SIDtoday article on this topic.)

(TS//SI) We've had success in targeting cafes over the past year, but until recently there was a
major gap in our capabilities. Namely, we could not precisely locate cafes that were served by
the Al [REDACTED], a large, entirely wireless operation. Because of Al Mehtab's complex
network structure, use of dynamic ip addresses, etc., we could narrow down a user's location to
which city he was in - but nothing more precise than that.

(TS//SI) The picture changed this spring, however, with two important developments. First, in
late April, we intercepted (via Timberline FORNSAT and TAO collection) the customer database
for [REDACTED]. Next, in May, two new systems (WealthyCluster2 and X-KEYSCORE) were
installed at SCS_Mosul that made it possible to process wireless collection in the region. SIGINT
now had the capability to intercept signals passing between [REDACTED] neighborhood
communications towers and the users.

(TS//SI) Putting these pieces of information together, the SIGINT System now had the ability to
detect terrorist use of the [REDACTED] network and to even find the location of the café. As the
SIGINT started coming in, the Network Analysis Center (NAC), in coordination with
Counterterrorism (CT), began issuing tippers with precise locational information to other
SIGINTers and military customers.

(TS//SI) The outcome so far has been this: previously un-locatable cafes have been found and at
least four "wanted" terrorists have been captured. The NAC is working closely with CT to share
their techniques, and CT is in the process of taking over future responsibility for tipping [REDACTED]
café locations being used by wanted terrorists.

LINKS
Email

Editorial Corrections
Organization
Projects
Searchlight

SIDtoday blog

Click here to

SUBMIT AN ITEM
please choose -

TOP S T//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Innovation

  • High Speed Selection
  • Toolbar
  • Integration with Marina
  • GPRS, WLAN integration
  • SSO CRDB
  • Workflows
  • Multi-level Dictionaries

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Future

  • High speeds yet again (algorithmic and Cell Processor (R4))
  • Better presentation
  • Entity Extraction
  • VoIP
  • More networking protocols
  • Additional metadata
    • Expand on google-earth capability
    • EXIF tags
    • Integration of all CES-AppProcs
  • Easier to install/maintain/upgrade

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL