X-KEYSCORE Application IDs

Open PDF
Creating agency: NSA National Security Agency (US)
Document date: -
Pages: 21
Genre: slides
Banner:
Classification: TOP SECRET COMINT ORCON
Dissemination: REL TO USA, FVEY

A TOP SECRET XKeyscore developer training presentation teaching how to write the pattern-matching rules that automatically classify every intercepted internet session by application type, with working examples for Gmail detection, Yahoo chat, ICQ, VoIP protocols and a Mujahideen Secrets encryption fingerprint, plus a side-by-side comparison showing the migration from legacy XML-based pattern definitions to the modern C-like AppID syntax.

Long Description:
This is a TOP SECRET//COMINT//ORCON//REL TO USA, FVEY XKeyscore Application IDs training presentation from the xkeyscore@nsa development team. It covers substantially the same material as the companion "AppIDs & Fingerprints" presentation but appears to be an earlier or parallel version, distinguished by its ORCON (Originator Controlled) marking and several content differences. The core syntax, naming conventions, levels system, Boolean/regex/binary patterns, chainwords, built-in functions and predefined chainwords are identical to the companion document. The key differences are in the examples and the absence of the advanced C++ code-based AppID, microplugin and context-sensitive scanning sections that appear in the later version — suggesting this presentation predates those capabilities or was targeted at a less technical audience. The most notable unique content is the Gmail detection chainword ($gmail), which reveals the specific strings XKeyscore uses to identify Gmail traffic: JavaScript function calls from Gmail's interface, POST/GET requests to /gmail paths, cookie patterns including SID cookies with specific character lengths on the .google.com domain, Gmail-specific cookies (GMAIL_AT, GMAIL_STAT, GMAIL_RTT, GMAIL_LOGIN), HTML title tags containing "Gmail" and the Google Front End server header (Server: GFE/). This provides a concrete example of how XKeyscore fingerprints a major webmail provider by matching against dozens of internal application-specific strings that ordinary users would never see. The Mujahideen Secrets fingerprint appears in a slightly different form from the companion document — fingerprint('encryption/moujahedeen') matching "begin+gimf+asrar+el+moujahedeen" or "begin gimf asrar el moujahedeen" — representing the GIMF (Global Islamic Media Front) Asrar al-Mujahideen encryption tool header in both URL-encoded and plaintext forms. The final slide shows a legacy XML-based pattern definition for IRC chat detection alongside its equivalent modern AppID syntax, demonstrating XKeyscore's evolution from a verbose XML configuration format to the streamlined C-like language. The XML format used keyword (KW) entries with positional constraints (POS=C for line-beginning matching equivalent to the modern first() function) and explicit select/deselect blocks, while the modern syntax expresses the same logic far more concisely using first() and Boolean operators. The AppID validation output confirms the same configuration file structure as the companion document — anonymizer.appid, bulletin_board.appid, tao_vpn.appid, tdmoip.appid, terminal.appid and voip.appid loaded from the dictionaries directory.
Programs:
XKEYSCORE, DECODEORDAIN

People mentioned:
-

Countries:
-

Tags:
AppID, fingerprints, pattern matching, chainword, Gmail, Yahoo Messenger, ICQ, IRC, SIP, Skinny, SMTP, VoIP, IMS, Mujahideen Secrets, GIMF, Asrar al-Mujahideen, Google Front End, GFE, legacy XML, regex, vBulletin

Identifier:
xks-application-ids-brief

PDF SHA-256:
734330b44f5311dc59fb0ff84fb51013e7adc597ca383e17ab1ef0bd6154935f

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

KEYSCORE

X-KEYSCORE Application IDs

xkeyscore@nsa

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Basic Syntax

Syntax:

  • Very C-like.

function('name', level, <optional info> ) = 'search terms and pattern';

Two valid search functions appid and fingerprint:

  • appid('chat/icq', 8.5, wireshark='icq', chatproc='ICQ') = /[^o]icq/c and $icq;
  • fingerprint('fingerprint/phone/nokia/generic', 7.0) = 'user-agent: nokia' or 'profile: http://nds.nokia.com/uaprof/n';

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Naming Conventions

XKS Appid's are named using a
pseudo directory convention.

/application_type/sub_type/name

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Levels

Levels are 1.0 – 9.9 with lower numbers having a higher priority. This
allows multiple signatures to match a piece of traffic and only the
most specific appid will be applied. An example might be:

9.9 Yahoo
9.8 Yahoo/chat
9.7 Yahoo/chat/incoming

Since the Yahoo/chat/incoming has the lowest level, the traffic will be
labeled as yahoo/chat/incoming

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Basic Search Patterns

XKEYSCORE supports Boolean operations and regular
expressions

Raw text must be encapsulated between single quotes

  • 'search term'

Terms can be combined with Boolean logic

  • 'search term' and 'another term'
  • 'search term' or 'another term'

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Binary and Regex Patterns

Binary patterns can be represented by putting a \x in front of each binary
value

  • '\xff\xff\x00\x02'

Note: Unlike C, no double back slashing required

/regex/

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

CHAINWORDS

You can assign a pattern to a variable (CHAINWORD) and reuse
the variable in many patterns.

  • $sip = 'via: sip' and 'cseq:' and 'SIP/2'c;

Now we can use this variable in future definitions:

  • appid('voip/sip', 7.2 ) = $sip;
  • appid('voip/sip/invite', 6.9) = $sip and 'INVITE';

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Built in functions

ip(expr) Matches against an IP Address looks in to address and from address in the session headere
• ip('10.10.10.1');
toport( expr) Matches against the Destination/To port. Note this must be a numeric representation of a port.
• toport( 1920 );
fromport( expr) Matches against the Source/From port. Note this must be a numeric representation of a port.
• fromport(80);
port(expr) Matches against the either port. Note this must be a numeric representation of a port.
•port(6667);
next_protocol(expr) Matches against the integer version of the next protocol.
• next_protocol(250);
protocol ('text') Will only work for IP next protocol names as defined in the IANA next protocol numbers document
• protocol('tcp');

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Built in functions

email_address(sel) permutes just like strong_selector (just like DECODEORDAIN
mac_address(addr) Tasks a mac address
smac(addr)
dmac(addr)
ip(addr) tasks this IP address (either to or from)
from_ip(addr) tasks this IP address only when it is the originator
to_ip(addr) tasks this IP address only when it is the destination

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

More built in functions

first(expr) Matches against a pattern at the beginning of the session
lpos(expr) Matches against a pattern at the beginning of each line (\n)
pos( expr ) expression occurs at offset X in the session
• pos('Hello') == 5,
• pos(/Good.*Grief/) <= 10
between(expr) • between('Hello', 'World', 10, 100)
Separation between 'Hello' and 'World' is greater than or equal to 10 bytes and less than or equal to 100 bytes This is the same as using the following regular expression:
• /Hello.{10,100}World/
'term'c Does a case sensitive match of the term
'term'u Treats the term as UTF-16

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Predefined Chainwords

There are a number of chainwords predefined for convenience:

  • $tcp
  • $udp
  • $icmp
  • $sctp
  • $rpc
  • $arp
  • $ssl
  • $http_cmd
  • $http
  • $http_get
  • $http_put
  • $http_post
  • $http_delete
  • $http_trace
  • $http_head
  • $http_options
  • $http_partial
  • $vbulletin
  • $mime_type
  • $user_agent

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Example

appid('voip/sip/IMS', 6.0, wireshark='sip') =
  ('via: sip' or 'v: sip') and 'cseq:' and (
    'p-access-network-info:' or
    'p-called-party-id:' or
    'p-charging-vector:' or
    'p-charging-vector-addresses:' or
    'p-media-authorization:' or
    'security-verify:' or
    'security-server:' or
    'security-client:' or
    'service-route:' or
    'record-route:' and 'pcscf' or
    'record-route:' and 'scscf' or
    'contact:' and 'pcscf' or
    'contact:' and 'scscf' or
    'proxy-authorization:' and 'pcscf' or
    'proxy-authorization:' and 'scscf' or
    'path:' and 'pcscf' or
    'path:' and 'scscf'
);

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Example

appid('voip/skinny/keep-alive', 3.0, wireshark='skinny') =
       toport(2000) and
first('\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
;

appid('voip/skinny/keep-alive-ack', 3.0, wireshark='skinny') =
       fromport(2000) and
first('\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00')
;

appid('voip/skinny(port2000)', 9.9, wireshark='skinny') =
      port(2000);

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Example

appid('chat/yahoo', 6.0, chatproc='Yahoo') =
    (('YCHT'c and $yahoo_chat) or first('YCHT'c)) and not
  port(5050);

appid('chat/icq', 8.5, wireshark='icq', chatproc='ICQ') =
    /[^o]icq/c and $icq;

appid('chat/icq', 9.0, wireshark='icq', chatproc='ICQ') =
    first('icq') and not port(25);

fingerprint('encryption/moujahedeen', 7.0) =
    'begin+gimf+asrar+el+moujahedeen' or
    'begin gimf asrar el moujahedeen';

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Example

appid('mail/smtp/to_server', 8.5, direction=$from_server,
  wireshark='smtp') =
    toport(25) and
    ( first('helo') or
      first('ehlo') or
      first('data') or
      (lpos('To: 'c) and lpos('From:'c)) or
      lpos('QUIT'c) or
      lpos('mail from:') or
      lpos('rcpt to:') );

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Example

$gmail =     '<script>D=(top.js&&top.js.init)?function(d){top.js.P(window'c or
           first('POST/gmail'c) or
           first('GET /gmail'c) or
           'GMAIL AT='c or
           /SID=[A-Za-z0-9\-\__]{87}=;Domain=\.google\.com/c or
           'GMAIL_STAT='c or
           '[[\\"ct\\"'c or
           'S=gmail='c or
           'ain=\'mail.google.com'c or
           '<title>Gmail'c or
           'GMAIL RTT='c or
           'GMAIL_LOGIN='c or
           '\
           '\nServer: GFE/'c;
       appid('mail/webmail/gmail', 8.0, webproc='Gmail') =
           $gmail;

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Append Option

# append the mime_type and HTML title to any of these appids..
PARAMS = append=$mime_type, append2=$http_info,
   append3=$doc_title;

$web = "web";

appid('http/proxy_to_server', 9.1, $web, direction=$proxy_to_server) =
    $webproxy_to_server;

appid('http/proxy_to_client', 9.1, $web, direction=$proxy_to_client) =
    $webproxy_to_client;

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Type Option

Third parameter is the type; if missing, it takes up to the first slash as
the type

appid('http/response', 9.2, $web) =
    $http and
    not ('x-cache' or 'x-forward' or 'get /' or
        'post /' or 'get http' or 'post http');

appid('http/response/partial', 9.1, $web) =
    $http and $http_partial;

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Appid utility

appid options:
--help                  this help message
--list-all              list all the application/fingerprint names and
                        levels
--list-appids           list all the application names (no fingerprints)
--list-fingerprints     list all the application names (no appids)
--list-types            list all the application types
--list-levels           list all the application levels
--unit-test             perform unit tests with data in the heirachy
                        'datadir', with files matching 'filespec'
--quiet                 don't print any load messages
--appid_fname arg       location of appid.cfg
--input-file arg        input file to test
--datadir arg           The test data directory. Defaults to
                        $(XSCORE_TEST_DATA_DIR)/appids
--filespec arg (=.*\.u124) A regular expression to match against files to
                        check
--noexit arg (=0)       do not stop on the first error

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

Appid Validation

appid sample.u124
Loading appids
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/appid_definitions.cfg
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/anonymizer.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/bulletin_board.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/tao_vpn.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/tdmoip.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/terminal.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/voip.appid
->Loading : /home/oper/xkeyscore/config/dictionaries/appid/appid_definitions.cfg
Finished loading appids
Filename: sample.u124

Appid: encryption/https

Total Size: 19.36Kbits
Total Time: 0.01secs
Rate: 1.936Mbits/s
Overall performance:
Total Time: 0.01secs
Total Bits: 0.01936Mbits
Overall Rate: 1.936Mbits/s

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123


Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123

<entry>
  <type> boolean </type>
  <name> chat/irc </name>
  <category> application i
  <information>
    <application type> chat
    <level> 8.1 </level>
    <action> application id
  </information>
  <select>
    KW='<POS=C>nick'+
    KW='<POS=C>ison' +
    KW='<POS=C>whois'+
    KW-'<POS-C>isirc'+
    KW='<POS=C>irc'!
    KW='<POS=C>join'+
    KW-'<POS-C>auth'+
    KW-'<POS-C>crypt'+
    KW='<POS=C>ping'!
    KW='<POS=C>pong'+KW-'<POS-C>privmsg'+
    KW='<POS=C>notice auth'+
    KW='irc'&KW='privmsg'&KW='notice'+
    KW='error : you'
  </select>
  <deselect> KW='ap 00120'&KW='user' </deselect>
</entry>

appid('chat/irc', 8.5, wireshark-'irc', chatproc-'IRC')
  'privmsg ';
  
appid('chat/irc', 8.1, wireshark='irc', chatproc='1RC')
  not (por=(1-0) and 'user') and
  ( first('nick') or first('ison ') or
  first('whois ') or first('isirc') or
  first('irc ') or first('join ') or
  first('auch ') or first('crypt ') or
  first('ping ') or first('pong ') or
  first('privmsg ') or
  first('notice auth') or
  first('irc' and 'privmsg' and 'notice')
  or 'error : you'
);

Classification: TOP SECRET//COMINT//ORCON//REL TO USA, FVEY//20291123