XKEYSCORE intro copy: xks-intro
Another published copy of this document that differs from the one on the document page - for example less redacted, with extra pages, or a different scan.
Pages that differ from the main document: 13, 16, 20, 21, 22, 23, 27, 28
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
XKEYSCORE
25 Feb 2008
xkeyscore@nsa
DERIVED FROM: NSA/CSSM 1-52
DATED: 20070108
DECLASSIFY ON: 20320108
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
What is XKEYSCORE?
- DNI Exploitation System/Analytic Framework
- Performs strong (e.g. email) and soft (content) selection
- Provides real-time target activity (tipping)
- "Rolling Buffer" of ~3 days of ALL unfiltered data seen by XKEYSCORE:
- Stores full-take data at the collection site - indexed by meta-data
- Provides a series of viewers for common data types
- Federated Query system – one query scans all sites
- Performing full-take allows analysts to find targets that were previously unknown by mining the meta-data
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Methodology
- Small, focused team
- Work closely with the analysts
- Evolutionary development cycle (deploy early, deploy often)
- React to mission requirements
- Support staff integrated with developers
- Sometimes a delicate balance of mission and research
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
System Details
- Massive distributed Linux cluster
- Over 500 servers distributed around the world
- System can scale linearly - simply add a new server to the cluster
- Federated Query Mechanism
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Query Hierarchy
User Queries
Query
XKEYSCORE web Server
Query Query Query
F6 HQS
Query Query
F6 Site 1
F6 Site 2
FORNSAT site
SSO site
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Where is X-KEYSCORE?
Approximately 150 sites
Over 700 servers
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
What is unique about XKEYSCORE?
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
General Capability
Processing Speed
TURMOIL/TURBULENCE
XKEYSCORE
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Why do shallow
- Can look at more data
- XKEYSCORE can also be configured to go shallow if the data rate is too high
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Why go deep
- Strong Selection itself give us only a very limited capability
- A large amount of time spent on the web is performing actions that are anonymous
- We can use this traffic to detect anomalies which can lead us to intelligence by itself, or strong selectors for traditional tasking
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
What XKS does with the Sessions
Plug-ins extract and index metadata into tables
[sessions] -> [processing engine] -> (database) -> (user queries)
Session
phone numbers
email addresses
log ins
user activity
Database
metadata tables
full log
QUERY
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Plug-ins
| Plug-in | DESCRIPTION |
|---|---|
| E-mail Addresses | Indexes every E-mail address seen in a session by both username and domain |
| Extracted Files | Indexes every file seen in a session by both filename and extension |
| Full Log | Indexes every DNI session collected. Data is indexed by the standard N-tupple (IP, Port, Casenotation etc.) |
| HTTP Parser | Indexes the client-side HTTP traffic (examples to follow) |
| Phone Number | Indexes every phone number seen in a session (e.g. address book entries or signature block) |
| User Activity | Indexes the Webmail and Chat activity to include username, buddylist, machine specific cookies etc. |
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
What Can Be Stored?
- Anything you wish to extract
- Choose your metadata
- Customizable storage times
- Ex: HTTP Parser
FM IP 58.[REDACTED] TO IP 64.[REDACTED]
GET /search?hl=en&q=islamabad&meta= HTTP/1.0
Accept: image/git, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-
application/msword, application/x-shockwave-flash, */*
Referer: http://www.google.com.pk/
No username/strong selector
Accept-Language: en-us
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Host: www.google.com.pl
Cookie: PREF=ID=678Tb0a34384e2f6:TM=1168503483:LM=1168503483:s=KKzZb3kPcw4vNxGt
Via: 1.0 proxy: [REDACTED]:8080 (squid/2.5.STABLE13)
X-Forwarded-For: 58.[REDACTED]
Cache-control: max-age=239200
Connection: keep-alive
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
What can you do with XKEYSCORE?
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Finding Targets
- How do I find a strong-selector for a known target?
- How do I find a cell of terrorists that has no connection to known strong-selectors?
- Answer: Look for anomalous events
- E.g. Someone whose language is out of place for the region they are in
- Someone who is using encryption
- Someone searching the web for suspicious stuff
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Encryption
- Show me all the encrypted word documents from Iran
- Show me all PGP usage in Iran
- Once again – data volume too high so forwarding these back is not possible
- No strong-selector
- Can perform this kind of retrospective query, then simply pull content of interest from site as required
Enter Password
Password
Confirm
OK
Cancel
Help
بسم الله الرحمن الرحيم
لسيد نایب رییس لوزراء ان ه ترم
لسید هدی و ل م ک ت اب ان محترم
انا موطن و الصال اي ان ان اداری و و الام التي و
ال کتب من الدرجة ان اولو فم دین.
ایف لا ان م و نقي ن د جلال ع راق وفق القانون السفر و ال یف اد
الن ل ذ ومن الدرجة ال اولو ف م ادون.
م ع القلب: از مو کی ن ب ال عقوبات ال لا ربية المنصوص علیه فی
قانون ريض ب الموظف فی الدولة ل م رق و 14 لسنة 1991 وب ع د
م و فن العید لن اب .
ا . عقوبة لفت النار
ب عقوب: ال ان ذار
ج عقوب: قطع الراتب لمدة لا ترید عن ثل الثلثة في ام
التوقيع علی ان او ام و از ادرية ان هر آب از ببشرة و ل ارفك الله
وبراءة القمة لقص آ ب ان دي في ن م ن ان درجة ال اولی نام ادون.
الهاتان لي ان انتهم ان ي ة :
التي منح المك في ان وال د لي ب ان م السید ال نایب ال می کرم
ب م ال اي نتج او ز حدود م ب ال غد | (150000) م ی ة ی ص ون الف دینار
نا م ن ك بي ال مكتب .
التوقيع علی ق و ا م ل مشتريات و فی انتي ر و الغرب ت التي
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Technology Detection
- Show me all the VPN startups in country X, and give me the data so I can decrypt and discover the users
- These events are easily browsable in XKEYSCORE
- No strong-selector
- XKEYSCORE extracts and stores authoring information for many major document types – can perform a retrospective survey to trace the document origin since metadata is typically kept for up to 30 days
- These events are easily browsable in XKEYSCORE
- No other system performs this on raw unselected bulk traffic, data volumes prohibit forwarding
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Persona Session Collection
- Traditionally triggered by a strong-selector event, but it doesn't have to be this way
- Reverse PSC - from anomalous event back to a strong selector. You cannot perform this kind of analysis when the data has first been strong selected.
- Tie in with Marina - allow PSC collection after the event
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Language Tracking
- My target speaks German but is in Pakistan – how can I find him?
- XKEYSCORE's HTTP Activity plugin extracts and stores all HTML language tags which can then be searched
- Not possible in any other system but XKEYSCORE, nor could it be -
- volumes are too great to forward
- No strong-selector
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Google Maps
- My target uses Google Maps to scope target locations – can I use this information to determine his email address? What about the web-searches – do any stand out and look suspicious?
- XKEYSCORE extracts and databases these events including all web-based searches which can be retrospectively quried
- No strong-selector
- Data volume too high to forward
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Document Tracking
- I have a Jihadist document that has been passed around through numerous people, who wrote this and where were they?
General | Description | User Defined | Internet | Statistics
Title بسم الله الرحمن الرحيم
Subject:
Keywords
Comments
Type:** MS Word document
Location:
Size:
Created:
Modified:
Digitally signe
Last printed:
Total editing t
Revision numb
Apply use
Template:
عليه افي
199 و ب ع د
... و الى انفكاك
م ادون.
الي منح المك ف ي ت و ال لي اب امن م السيد الان ای ب ال محترم
ب م ال اي تج او زحدود م ل غ ها (150000) م مئة وخص ون لف في نار
ل م ن س بي ال مكتب .
نتي التوقيع على قوائم المشتريات و الفواتي رو نطلب ت
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Document Tracking
- All images are hashed in the metadata so that you can search for anyone who has received or transmitted this document.
- This is really useful for company logos.
Biographical Data Sheet
معلومات شخصية
Note: Fill the blocks in English & Type - Submit electronically via e-mail.
ملاحظة : ملأ الاستمارة الانكليزية باستخدام نامج الوورد وارسلها بالبريد الالکتروني
First Name الاسم الاء [REDACTED
1st Middle (father's) Name*اسم الاب [REDACTED]
2nd Middle (Grandfather) Name اسم الجد [REDACTED]
Family / Tribal Name [REDACTED]
Aliases/Nickname [REDACTED]
Mother's Name اسم الام [REDACTED]
Mother's Tribal Name [REDACTED]
<hash> 5c24dabb15f0c5d64dcf7[REDACTED]<hash>
Birth Date تاريخ الميلاد
DD/يوم 17
MM/شهر [REDACTED]
YY/سنة [REDACTED]
City/Country of Birth مكان الولادة الدولة - المدينة البلدة
Iraq-kut-dor almoalmean
Nationality الجنسية
Iraqi
Race القومية
Arab
Religion الديانة
Muslim
Gender (Male/Female) الجنس
male
Marital Status (Single/Married/Divorced) الحالة الزوجية
single
Number of children عدد الاطفال
non
<lastauthor>[REDACTED]</
<title>Biographical Da
<language>Arabic</la
– <comment>
<METADATA_RO
<Language>Arabic<
</DateItem> <Proce
<Processing> >Nam
</METADATA_RO
</document_metadata>
</xks_meta>
from
- No s
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Interesting Document Discovery
- Show me all the Microsoft Excel spreadsheets containing MAC addresses coming out of Iraq so I can perform network mapping.
- New extractor allows different dictionaries to run on document/email bodies – these more complex dictionaries can generate and database this information.
- No strong-selector
- Data volume is high
- Multiple dictionaries targeted at specific data types
PROACTIVE Communications, Inc.
ICZN Closeout Document
Site Information
Site Name/City: [REDACTED] عرفی مدل تاج ول عام
Function (IPS, PJOC, POL, ...): [REDACTED] العملیة الشارقة
GPS Coordonates: [REDACTED]
Site Pre-Commissioned Date: [REDACTED]
Site Commissioned Date: [REDACTED]
Meters of CAT 5 Used: 300 Meters
PCI Deliverable Equipment
Switch: [REDACTED] دواعي التحول
NetModem: [REDACTED]
Pix 501 Firewall: [REDACTED]
Ethernet Switch Serial Number: [REDACTED] المحول
# of Ports / Make / Model: [REDACTED]
Serial Number: [REDACTED] المحول
Model: [REDACTED]
Telephone Number: [REDACTED] رقم الهاتف
Government Furnished Equipment
BorderGuard Serial Number: [REDACTED] رقم المسلسل
Model: [REDACTED]
Dell Model Type: Serial Number: [REDACTED]
Monitor Type: Serial Number: [REDACTED]
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TAO
- Show me all the exploitable machines in country X
- Fingerprints from TAO are loaded into XKEYSCORE's application/fingerprintID engine
- Data is tagged and databased
- No strong-selector
- Complex boolean tasking and regular expressions required
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
XKEYSCORE Success Stories
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Over 300 terrorists captured using intelligence generated from XKEYSCORE
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
XKEYSCORE and TRAFFICTHIEF
- Customer: CounterTerrorism (CT)
- Provides near real-time tips to TRAFFICTHIEF server in operations in coordination with coalition forces in Iraq 24 hours a day
- Currently producing hundreds of confirmed alerts per day on over 3000 user accounts
Afternoon of [REDACTED] 2004 – coalition detained individuals below:
[REDACTED] [REDACTED]
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
XKEYSCORE Success
May 2006, WealthyCluster2 and X-KEYSCORE Installed at [REDACTED]
- Connected to Moonshine
- Enabled processing of wireless collection
- Enabled near-real-time tipping
- Enabled full-take SIGDEV
Un-locatable cafés were geolocated:
- [REDACTED] – “A Goldmine”
- Four Other Cafés Being Developed
Acquired important targets:
- NSA/Georgia Tips With Precise Locations
- JSOC Tools In New [REDACTED]
- Reacquired [REDACTED] Lost When Zarkanet Went Down
Terrorists were captured:
- Members of the [REDACTED]
- Members of the [REDACTED]
SIDtoday - Microsoft Internet Explorer
SIDtoday
THE INSIDE SCOOP
30 2006
ARCHIVES FEEDBACK SIDTODAY - HOME
SIDtoday
(TS//SI) Breakthrough: Previously 'Unfindable' Internet Cafes in
Iraq Can Now Be Located
FROM: SIGINT Communications (S02C) and
[REDACTED] Iraq Wireless Team (SSG211)
Wednesday, July 23, 2006
(TS//SI) Terrorists who visit Internet cafes associated with the [REDACTED] can now
potentially be located and arrested.
(TS//SI) There are a number of terrorists now in prison whose last moments of freedom went
something like this: Each terrorist woke up in the middle of the night, finding his bed
surrounded by U.S. troops. As he was detained and dragged out the door of his safehouse, he
may have wondered, "How did they know I was here?"
(TS//SI) Answer: SIGINT had detected the terrorist's activity in an Internet café earlier that day
and tipped his location to the appropriate forces in Iraq within minutes of his activity. Then,
through various means, the target was tracked from the café to his safehouse, leading our
forces not only to him, but also to other members of his terrorist network staying with him.
(See an earlier SIDtoday article on this topic.)
(TS//SI) We've had success in targeting cafes over the past year, but until recently there was a
major gap in our capabilities. Namely, we could not precisely locate cafes that were served by
the Al [REDACTED], a large, entirely wireless operation. Because of Al Mehtab's complex
network structure, use of dynamic ip addresses, etc., we could narrow down a user's location to
which city he was in - but nothing more precise than that.
(TS//SI) The picture changed this spring, however, with two important developments. First, in
late April, we intercepted (via Timberline FORNSAT and TAO collection) the customer database
for [REDACTED]. Next, in May, two new systems (WealthyCluster2 and X-KEYSCORE) were
installed at SCS_Mosul that made it possible to process wireless collection in the region. SIGINT
now had the capability to intercept signals passing between [REDACTED] neighborhood
communications towers and the users.
(TS//SI) Putting these pieces of information together, the SIGINT System now had the ability to
detect terrorist use of the [REDACTED] network and to even find the location of the café. As the
SIGINT started coming in, the Network Analysis Center (NAC), in coordination with
Counterterrorism (CT), began issuing tippers with precise locational information to other
SIGINTers and military customers.
(TS//SI) The outcome so far has been this: previously un-locatable cafes have been found and at
least four "wanted" terrorists have been captured. The NAC is working closely with CT to share
their techniques, and CT is in the process of taking over future responsibility for tipping [REDACTED]
café locations being used by wanted terrorists.
LINKS
Email
Editorial Corrections
Organization
Projects
Searchlight
SIDtoday blog
Click here to
SUBMIT AN ITEM
please choose -
TOP S T//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Innovation
- High Speed Selection
- Toolbar
- Integration with Marina
- GPRS, WLAN integration
- SSO CRDB
- Workflows
- Multi-level Dictionaries
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
Future
- High speeds yet again (algorithmic and Cell Processor (R4))
- Better presentation
- Entity Extraction
- VoIP
- More networking protocols
- Additional metadata
- Expand on google-earth capability
- EXIF tags
- Integration of all CES-AppProcs
- Easier to install/maintain/upgrade
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL







