XKEYSCORE, Cipher Detection, and You!
Open PDF +1 other copyDocument date: 2008-08-21
Pages: 17
Genre: slides
Banner: TOP SECRET//COMINT//REL TO USA, FVEY
Classification: TOP SECRET COMINT
Dissemination: REL TO USA, FVEY
A TOP SECRET presentation from NSA's Mathematics Research Group dated 21 August 2008, explaining how XKeyscore detects encrypted and steganographic communications — describing the fingerprint and C++ plug-in systems used to flag sessions using particular ciphers, VPNs, encrypted archives and hidden-data tools, with worked examples detecting RAR archives, OpenVPN, digital certificates and various encryption products, plus a revealing note that detection of encryption is itself used to identify communications worth targeting.
My notes
TOP SECRET//COMINT//REL TO USA, FVEY
XKEYSCORE, Cipher Detection, and You!
[REDACTED]
Mathematics Research Group
21 August 2008
TOP SECRET//COMINT//REL TO USA, FVEY
UNCLASSIFIED
The Protocol Stack
- Application Layer (HTTP, FTP, etc.)
- Transport Layer (TCP, UDP)
- Network Layer (IPv4, IPv6)
- Data Link Layer (PPP)
- Physical Layer (Copper, Fiber)
UNCLASSIFIED
TOP SECRET//COMINT//REL TO USA, FVEY
Data Sources
- FORNSAT (downlink)
- Overhead (uplink)
- Special Source
- Tailored Access
- F6
- FISA (limited)
- 3rd party
TOP SECRET//COMINT//REL TO USA, FVEY
SECRET//COMINT//REL TO USA, FVEY
Front-end Processing
WEALTHYCLUSTER2/TURMOIL
Demodulate Demultiplex Packetize Sessionize
Decrypt?
SECRET//COMINT//REL TO USA, FVEY
SECRET//COMINT//REL TO USA, FVEY
What does XEYSCORE do?
- Selection of tasked CADENCE/UTT terms.
- Send hits to PINWALE/PRESSUREWAVE.
- Tipping to TRAFFICTHIEF.
- Fingerprinting.
- SIGINT development using two rolling buffers:
- Metadata
- Content (data)
SECRET//COMINT//REL TO USA, FVEY
TOP SECRET//COMINT//REL TO USA, FVEY
Retrospective Searching
XKEYSCORE
- All data are stored, not just hits.
- Queries are distributed to entire network of sites.
Metadata Buffer
- ~30 days
- Searchable
- MySQL database
Content Buffer
- ~7 days
- Retrievable
- Archived on disk
TOP SECRET//COMINT//REL TO USA, FVEY
TOP SECRET//COMINT//REL TO USA, FVEY
Fingerprinting
- Pattern matching against the data.
- Session is marked, but not sent to PINWALE.
- Fingerprint stored as metadata.
- Have to search for it.
- Rich set of patterns
- Strings have a minimum of three* anchors (fixed bytes).
- [Exception: Two bytes at the beginning of a session]
- Regular expressions allowed (require nonoptional string of three* bytes within regex)
- Context-dependent terms.
- *XKS reserves the right to increase this to four.
- Strings have a minimum of three* anchors (fixed bytes).
TOP SECRET//COMINT//REL TO USA, FVEY
TOP SECRET//COMINT//REL TO USA, FVEY
Examples
7.0) = 'helix stronghold encrypted file';
'\xd6\x56\x34\xb7\x80\x05\xfe\x8b'c and
'\xaf\x52\x72\x60\xdd\xfe\x72\xc2'c and
(port(443) or port(80));
fingerprint('encryption/the_algorithm',
3.0)=
/-XYZ-.{0,30}mp[eg]/;
TOP SECRET//COMINT//REL TO USA, FVEY
TOP SECRET//COMINT//REL TO USA, FVEY
Syntax Features
- Case Sensitivity
fingerprint('certificate/digital_id') =
'-BEGIN CERTIFICATE-'c;
- Full Boolean logic
- Grouping with parentheses
- Operators:
and,or,not
- Variables
$udp = protocol('udp');
fingerprint('vpn/openvpn/x509/wera') =
$udp and 'openvpn_wera'c;
TOP SECRET//COMINT//REL TO USA, FVEY
TOP SECRET//COMINT//REL TO USA, FVEY
Available Functions
- port
- first
- hex
fingerprint('encryption/kryptel') =
hex('E8E2454300040004635C4EE9A2F9D111A
489E498F70C0B43404F4BFA50F2D111A4898E6
30458E285');
- pos
fingerprint('encryption/cipherpad') =
pos('CPAD1'c) < 4000;
TOP SECRET//COMINT//REL TO USA, FVEY
TOP SECRET//COMINT//REL TO USA, FVEY
- Distance (similar to pos, but for distance between tokens)
- Lpos
Spop_basic = lpos('+OK 'c) or '\nQUIT';
- First
appid('mail/smtp/...) = first('ehlo') and ... ;
- Last (similar to first)
- Follows (one token after another)
- Between (one token between two others)
- Order
TOP SECRET//COMINT//REL TO USA, FVEY
TOP SECRET//COMINT//REL TO USA, FVEY
Other Features
- Fingerprint definitions updated hourly throughout the entire enterprise.
- Workflows
- Submit through user interface.
- Standing queries that run like cron jobs.
- Limited follow-on processing.
- User interface for fingerprint submission (coming soon).
- Currently done by XKS personnel.
TOP SECRET//COMINT//REL TO USA, FVEY
TOP SECRET//COMINT//REL TO USA, FVEY
Plug-ins
- Full power of C++ for when pattern matching does not suffice.
- Usually limited to certain file types
- Huge JPEG volume from web surfing
- Current steg/encryption plugins that fingerprint sessions:
- PHOSPHORESSENCE library of steg detectors
- SHELLLOCK steg detection
- SEDENA indigenous encryption software
- Drawback: Must wait for site upgrade to deploy.
TOP SECRET//COMINT//REL TO USA, FVEY
TOP SECRET//COMINT//REL TO USA, FVEY
Trade-off
- Fingerprints easily deployed, but limited to pattern matching.
- Plug-ins slow to deploy, but allow for complex testing.
- New compromise:
- Snippets of C++ code in fingerprint
- Deployed hourly like fingerprint with most of the flexibility of a full plug-in.
- Very complicated tests probably still need to be plug-ins.
- Currently stood up at only a few sites.
TOP SECRET//COMINT//REL TO USA, FVEY
TOP SECRET//COMINT//REL TO USA, FVEY
Example
fingerprint('encryption/archive/rar') =
'\x52\x61\x72\x21\x1a\x07\x00'c
: c++ {{
const uint8_t *ptr =
find_first("\x52\x61\x72\x21\x1a\x07\x00");
if (ptr == NULL)
return false;
if (end()-ptr < 64)
return false;
if ((ptr[23] &0x04) != 0x04)
return false;
if ((ptr[10] &0x80) != 0x80)
return false;
return true;
}};
TOP SECRET//COMINT//REL TO USA, FVEY
TOP SECRET//COMINT//REL TO USA, FVEY
Advanced Feature
- Follow-on check with anchorless regexes:
%dhcp_check = regex {{
^[\x01\x02][\x01- ]\x06.*c\x82sc
}};
appid('netmanagement/dhcp/client_to_server',
3.0) =
from_port(68) and to_port(67)
: %dhcp_check;
TOP SECRET//COMINT//REL TO USA, FVEY
TOP SECRET//COMINT//REL TO USA, FVEY
Releasability Issues
- Nearly all XKS personnel have PICARESQUE!
- Those that don't have PRIVAC.
- XKS distribution comes in two flavors
- 1st & 2nd party
- 3rd party
- No NOFORN capabilities permitted.
- Special dispensation from [REDACTED] for some capabilities to SMOKYSINK.
- Can keep PICARESQUE code running on R1's rednet if absolutely necessary.
TOP SECRET//COMINT//REL TO USA, FVEY