Introduction to XKS Application IDs and Fingerprints

Open PDF
Creating agency: NSA National Security Agency (US)
Document date: 2009-08-27
Pages: 60
Genre: slides
Banner:
Classification: TOP SECRET COMINT
Dissemination: REL TO USA, AUS, CAN, GBR, NZL

This August 2009 NSA training document introduces XKEYSCORE's Application IDs (AppIDs) and Fingerprints system, explaining how metadata tags are automatically applied to intercepted sessions to identify and categorize different types of Internet traffic, applications, and behaviors, evolving through four generations from simple keyword matching to sophisticated code-based metadata extraction.

Long Description:
The presentation provides comprehensive training on XKEYSCORE's AppID and Fingerprint classification system, which serves as the foundation for organizing and analyzing massive volumes of intercepted Internet traffic. AppIDs are unique metadata tags that identify what application is being used in each network session (like Yahoo webmail, MSN Messenger, or HTTP GET requests), while Fingerprints tag additional attributes that can apply to multiple applications simultaneously (like encryption methods, device types, or content characteristics). The document emphasizes that each session receives only one AppID—the most specific applicable definition based on a numeric priority system where lower numbers indicate more granular classifications—while sessions can have multiple fingerprints describing various attributes. The training walks through four generations of increasingly sophisticated detection capabilities. First-generation AppIDs used simple keyword scanning and regular expressions to identify traffic patterns. Second-generation introduced context-sensitive scanning with contexts (like http_host, html_title, user_agent, IP addresses, and port numbers), allowing analysts to target specific parts of network traffic rather than scanning entire sessions, dramatically improving accuracy and performance. Third-generation added code-based scanning using C++ algorithms to perform statistical analysis and pattern recognition beyond what keywords alone could achieve. Fourth-generation, also called "Microplugins," enables AppIDs to automatically extract and database specific metadata fields from identified traffic, allowing new intelligence requirements to be deployed across all 130+ field sites within an hour rather than months required for traditional plugin upgrades. The document provides extensive practical examples demonstrating real-world applications: identifying Yahoo webmail through "Host: mail.yahoo" headers, detecting Mujahideen Secrets encryption in extremist forums, tracking Facebook chat sessions and extracting sender/message metadata, monitoring file uploads to free hosting sites like zShare and capturing filenames/URLs/deletion links, fingerprinting mobile devices (iPhone, Motorola, BlackBerry, Sony Ericsson), and identifying specific applications through cookies, user agents, and HTML titles. Throughout the training, USSID 18 compliance is repeatedly addressed—analysts are warned that querying solely on technology fingerprints like "iPhone" would violate regulations, but combining such fingerprints with foreign intelligence targets (like Afghan IP addresses or Iranian proxies) creates compliant queries. The presentation demonstrates how analysts use histogram analysis of AppIDs to quickly understand what applications a target is using without manually reviewing each session, and how the directory-like structure (mail/webmail/yahoo/login) allows flexible searching at different levels of specificity.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL//20291123

KEYSCORE

Introduction to XKS Application IDs and Fingerprints

27 August 2009

DERIVED FROM: NSA/CSSM 1-52
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL//20291123


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Agenda

  • Overview of Application IDs and Fingerprints
  • Background of the 4 generations of AppIDs+Fingerprints
  • Examples of how they are used for target development SIGDEV

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

What is an AppID?

  • An Application ID (AppID) is a meta-data tag given to a session to help describe what application is being seen in the traffic
  • Examples:
    • mail/webmail/yahoo indicates that the traffic was Yahoo Webmail
    • chat/msn_messenger indicates the traffic was MSN Messenger
    • http/get indicates that the traffic was an HTTP Get

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Why even have AppIDs/Fingerprints?

  • What's the point of AppIDs/Fingerprints?
  • For one, they give you a powerful tool for the quick analysis of what applications are being seen in your traffic.
  • A simple histogram on AppID allows you to quickly identify all of the applications seen for a given result set, without needing to view each piece of content

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Why even have AppIDs/Fingerprints?

  • Ex: Histogram the applications used during Target activity:
Histogram Grid
Page 1 of 1 Clear Selection Export

| Filter | Application                             | Count  |
| :----- | :-------------------------------------- | :----- |
|        | http/get                                | 92     |
|        | update service/windows                  | 47     |
|        | unknown/port80/http www                 | 25     |
|        | mail/webmail/gawab                      | 11     |
|        | http/response                           | 10     |
|        | mail/webmail/mailru                     | 8      |
|        | photo sharing/i494.photobucket.com      | 8      |
|        | http/post/x-www-form-urlencoded         | 6      |
|        | http/response/gif                       | 6      |
|        | mail/webmail/qmail                      | 5      |
|        | http/response/400 bad request/html      | 4      |
|        | http/response/not found/html            | 4      |
|        | filetransfer/web/archive.org/download/request | 3      |

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Why even have AppIDs/Fingerprints?

  • Secondly, they provide an additional criteria that you can use in your query.
  • NOTE: It's important to point out that since most AppIDs + Fingerprints are tagging technology and/or applications, they SHOULD NOT be the sole criteria for your queries in X-KEYSCORE!

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Why even have AppIDs/Fingerprints?

  • EX: I'm looking for targets using mail.ru from behind a large Iranian proxy:
IP Address: 78.[REDACTED] Either

AppID
(+Fingerprints) [fulltext]:

Field Builder
AppID (+Fingerprints)
mail/webmail/mailru
mail/webmail/mailru
mail/webmail/mailru/attachment
mail/webmail/mailru/post

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Why even have AppIDs/Fingerprints?

*   EX: I'm looking for targets using mail.ru from behind a large Iranian proxy:

IP Address: 78.[REDACTED] Either

AppID
(+Fingerprints) [fulltext]:

Field Builder
AppID (+Fingerprints)
mail/webmail/mailru
mail/webmail/mailru
mail/webmail/mailru/attachment
mail/webmail/mailru/post

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Why even have AppIDs/Fingerprints?

  • EX: I'm looking for Mojaheden Secrets 2 use in extremist web forums:
Field Builder
AppID (+Fingerprints)
forum/extremist/
forum/extremist/al-firdawsArabic
forum/extremist/al-firdawsEnglish
forum/extremist/al-hisbah
forum/extremist/al-hisbahWorkshop
forum/extremist/al-ikhlas
forum/extremist/al-nukhbah
forum/extremist/al-nusrah
forum/extremist/al-qimmah
forum/extremist/al-shura
forum/extremist/al-tawhid
forum/extremist/aljazeeratalk
forum/extremist/alm3refh
forum/extremist/amb
forum/extremist/ashiyane

Field Builder
AppID (+Fingerprints)
moj
encryption/mojaheden2
encryption/mojaheden2/encodedheader
encryption/mojaheden2/hidden
encryption/mojaheden2/hidden2
encryption/mojaheden2/keyids
encryption/mojaheden2/securefile

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

How do AppIDs work?

  • AppID's are effectively looking for keywords in order to assign the AppID tag.
  • Example, let's say that this is the definition for mail/webmail/yahoo:
appid('mail/webmail/yahoo', 9.0) = 'Host: mail.yahoo';

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Example

  • Here is a client side Yahoo session:
GET /login.html HTTP/1.1
Referer: http://us.f359.mail.yahoo.com/ym/ShowLetter
Accept-Language: ar
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: mail.yahoo.com
Connection: Keep-Alive
Cookie: B=fn50ehd2612o2&b=3&s=rp; YMBM=d=&v=1;

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Example

appid('mail/webmail/yahoo', 9.0) = 'Host: mail.yahoo';
GET /login.html HTTP/1.1
Referer: http://us.f359.mail.yahoo.com/ym/ShowLetter
Accept-Language: ar
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: mail.yahoo.com
Connection: Keep-Alive
Cookie: B=fn50ehd2612o2&b=3&s=rp; YMBM=d=&v=1;

Application: mail/webmail/yahoo

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

How AppIDs work

  • What does the number in the AppID mean?
    appid('mail/webmail/yahoo', 9.0)=
  • Each session can have only one AppID
  • The goal is for the AppID to be as descriptive as possible
  • Any given session might qualify under multiple AppIDs definitions, but only the most specific AppID that applies to the session is assigned
  • Lowest number wins, so the lower the number, the more specific the AppID definition

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

How do AppIDs work?

  • Let's say there's another more descriptive appid for mail/webmail/yahoo/login:
appid('mail/webmail/yahoo/login, 8.0) = 'Host: mail.yahoo' and '/login';
  • It has a lower number than mail/webmail/yahoo, so if it “hits” it will be applied

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Example

appid('mail/webmail/yahoo', 9.0) = 'Host: mail.yahoo';
appid('mail/webmail/yahoo/login, 8.0) = 'Host: mail.yahoo' and '/login';
GET /login.html HTTP/1.1
Referer: http://us.f359.mail.yahoo.com/ym/ShowLetter
Accept-Language: ar
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: mail.yahoo.com
Connection: Keep-Alive
Cookie: B=fn50ehd2612o2&b=3&s=rp; YMBM=d=&v=1;

Application: mail/webmail/yahoo/login

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

AppID Structure

  • Note that the AppIDs have a directory-like structure:
  • mail/webmail/yahoo and mail/webmail/yahoo/login
  • If you wanted to search for all webmail activity you could search for mail/webmail/*
  • If you wanted to search for all Yahoo mail activity you could search for mail/webmail/yahoo/*
  • etc

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

How AppIDs work

  • Some session can hit on many AppIDs.
  • For example a single session might hit on:
    • appid('http/response', 9.2)
    • appid('mail/webmail', 8.9)
    • appid('mail/webmail/yahoo', 6.0)
    • appid('mail/webmail/yahoo/attachment', 5.0)
  • Which one will be assigned as the winning AppID?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

How AppIDs work

  • When you see an AppID how do you know what was used to define that AppID?
  • Through the XKS AppID signature page available through "go xkeyscore"
  • Or by simply clicking on the hyperlink AppID from the new GUI!

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

What is a fingerprint?

  • AppIDs were built to describe applications, of which there should only be one application seen per session.
  • How do we describe other attributes of a session that aren't necessarily tied to a particular application?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

What is a fingerprint?

  • One great example is encryption
  • A particular type of encryption could be used in Yahoo Email, Gmail Email, SMTP Email.
  • It could be used inside of a Word Document being uploaded to a free file website.
  • It could be used inside of a private message sent through Facebook.
  • Etc.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

What is a fingerprint?

  • How can we tag anytime we see that type of encryption regardless of the application we saw it in?
  • Answer - Fingerprints
  • Think of Fingerprints as "attributes" of a session.
  • A session can have as many fingerprints as is needed to best describe it.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Example

appid('mail/webmail/yahoo', 9.0) = 'Host: mail.yahoo';
appid('mail/yahoo/login, 8.0) = 'Host: mail.yahoo' and '/login';
fingerprint( 'mail/arabic') = 'mail' and /language[:=] ?ar/;
GET /login.html HTTP/1.1
Referer: http://us.f359.mail.yahoo.com/ym/ShowLetter
Accept-Language: ar
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: mail.yahoo.com
Connection: Keep-Alive
Cookie: B=fn50ehd2612o2&b=3&s=rp; YMBM=d=&v=1;

Application: mail/webmail/yahoo/login
Fingerprint: mail/webmail/yahoo/login mail/arabic

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Appid vs Fingerprint

Each session gets one appid -- lowest level wins. It gets databased in the 'application' field.

All matching fingerprints are stored in the 'fingerprint' field.

Application Type:
Application Info:
Application:
AppID
(+Fingerprints) [fulltext]:
[Populate with Field Builder]
[Populate with Tree Field Builder]

Winning appid
Winning appid + all fingerprints

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Fingerprint Examples

  • Ex: E-Mails with encryption
From: "Launchpad OpenPGP Key Confirmation" <noreply@launchpad.net> [Save Address] [Block Sender]
To:
Cc:
Subject: Launchpad: Confirm your OpenPGP Key
Date: Wed, 31 Dec 2008 10:04:16 -0000

Application: mail/webmail/outblaze
AppID (+Fingerprints): mail/webmail/outblaze has_fingerprint encryption/pgp encryption/pgp/message


-----BEGIN PGP MESSAGE-----
Version: GnuPG v1.4.6 (GNU/Linux)
spflvtVPZsl1vpg67VdHFUprgwOJpmjQlb73gWmhbOURzzyGddDRIa9CcFzJA7OIL
3XyCrlniniJ4/c98+khDazh1XY/S7yNi38WrIkd3GOz9DFFI1Nu31nwjh3+nc0pv
OlyztsQzLFB8/+qJrPvmK8fzz7tWp2djxyfMGoAYNAf/QOohROBjqTgOUlqLRVrE
eEFivrMOnBx160SHIFra7LpZlstTUFPBJNAkgguk7m8fJ0dMmdUOV5MeM1x8GuWw5+
Uk4bBwwwZ1VpEVHCyGuv8ux+V+KpSkQtDwdhlp12SZ2SUm1upnVB9lfcnlhWvxZp
LaY3mXqNWhyhzFPFxkhUwqzd/rMxrcJucfXGaeisSizZDIQOWxTSwe7BwwG8Bvnr
QEQVKY30WwVg+2pDTPRkQq3uEqOwvj9JY7KTPMrf2gZLNABDuCJm5IRALZqqETTg4dh
xV0r9+2ZLtYGDXQhLMyBEIYns4+jiP1rd3E+TW7JVUe/dPluyC4DWOUPklwuHcC+
StLAuQHMS6RkB4aADndI6QG9KEWwwjq2PvfuMiBWo5jU8RFoDSx8q5t1ukgeCxr6xr
Q4eTmOFTIA71G312Xa7ZniOzyixiWZ4CAhhHLF+3baFD3lb4/EFmRvPBdqy6wUyHD
Z5EXyHDzI4XIDyEe/aomEqAsUqPs8MZirHHzpbAS3LBG5bVKAKU59bENpf/KOgT
a3lUAeQ1t6xLzgToVdfhEkPj5bxODrWcZiHeTET1nV+3pc2P58+QlCDOETiDCA/j
dhG2brUwbxny6Ap7fU5e1ALU3ryoXKvt9eCXZHooY/p9QlC3koHCWptGD6gKCxIt
KWK5M+HkxhHy4V7Wb137CStzeLda8BdU43Kh0ZQWwWjK7pDXKKHHLYlGlawRScQa
e6J+y4JR1KKyXiXY94Erxa/PDFzuYV/QCJUDppqWFR22bXuy4FhkosLWwM8G+UBHVt
UfgRxq8as60DhBDWVy08eLEAdE92TVffJgXOVwAOzTqBrP7uZV/Q7ABFFGTQ9n
=N4CJ
-----END PGP MESSAGE-----
Thanks.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Fingerprint Examples

What caused those fingerprints to hit?

Application AppID (+Fingerprints)
mail/webmail/outblaze mail/webmail/outblaze has_fingerprint encryption/pgp encryption/pgp/message

Look at the definitions (notice any overlap?):

fingerprint('encryption/pgp') =
'begin pgp message' or 'begin+pgp+message';

fingerprint('encryption/pgp/message')=
/(?:BEGIN|END) PGP MESSAGE/;

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Ex: Extremist Forum Private Messages

HTTP Header Information      Content Type: HTTP/POST/Form-Data
POST                         /vb/private.php?do=insertpm &pmid= HTTP/1.1
Accept:                      image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, */*
Referer:                     http://al-faloja.info/vb/private.php?do=newpm&u=9692
Accept-Language:             en-gb
Content-Type:                application/x-www-form-urlencoded
UA-CPU:                      x86
Accept-Encoding:             gzip, deflate
User-Agent:                  Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; FDM)

Application: mail/webmail/vbulletin/private_message/insert
AppID (+Fingerprints): mail/webmail/vbulletin/private_message/insert has_fingerprint forum/extremist/al-faloja

recipients [REDACTED]
bccrecipients
title خبر مهم
message سنت كتيبة المواجهات التابعة لحركة الشباب المجاهدين بفضل الله مساء يوم الإثنين 08 محرم 1430 هـ الموافق لـ 05-01-2009م هجوما مباشرا وعنيفا على مصنع الباستا للقوات الصليبية الإثيوبية في مقديشو، وشاركت كتيبة المدفعيات في العملية المباركة حيث قامت بقصف المصنع توابل من الصواريخ والمدفعيات

واستخدم المجاهدون في الهجوم أساليب قتالية غير مسبوقة مما أرغم على قوات العدو التراجع من دفاعاتها في الشارع العام المؤدي إلى المصنع، وحينما اجتمعوا على جحرهم فاجأتهم كتيبة المدفعيات بقصف عنيف ودقيق ويتوقع خسائر بشرية جسيمة في صفوف القوات الصليبية ولله الحمد والمنة

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

AppID vs Fingerprint

  • AppIDs and Fingerprints use the exact same language inside of XKS.
  • You can tell which one it is by the definition:
appid (mail/webmail/yahoo)
fingerprint (encryption/pgp)

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

AppID/Fingerprint Language Evolution

  • There have been 4 generations of XKS AppID/Fingerprint languages
  • 1st Generation: Simple Keyword Scanning
  • 2nd Generation: Context Aware Keyword Scanning
  • 3rd Generation: Code based AppIDs/Fingerprints
  • 4th Generation: Code based AppIDs that can extract meta-data (also known as Micro Plugins)

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

1st Generation AppIDs/Fingerprints

  • In the beginning, AppIDs and Fingerprints were just keyword scanning similar to CADENCE tasking Ex:
appid('mail/webmail/yahoo', 9.0) =
'Host: mail.yahoo';
appid('mail/yahoo/login, 8.0) =
'Host: mail.yahoo' and '/login';

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

1st Generation AppIDs/Fingerprints

  • 1st Generation would also support Regular Expression (REGEX's):
fingerprint('encryption/pgp/message')=
/(?:BEGIN|END) PGP MESSAGE/;

(instead of quotes REGEX's are enclosed by forward slashes)

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

1st Generation AppIDs/Fingerprints

  • As well as Hex scanning:
appid('database/ms_sql_server(tds)/login', 7.5)=
'\x06\x83\xf2\xf8\xff\x00\x00\x00\x00\xe0\
x03\x00\x00\x88\xff\xff\xff\x36\x04\x00\x00';

(Hex characters are prefaced by \x)

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

2nd Generation AppIDs/Fingerprints

  • 2nd Generation AppIDs/Fingerprints introduced XKS's context sensitive scanning engine.
  • For example, rather than scanning an entire session top to bottom to look for 'facebook.com' we can just use the dictionary context http_host to target the scan for the host field only.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

How do AppIDs work?

  • AppID's are effectively looking for keywords in order to assign the AppID tag.
  • Example, this is the definition for Hi5
appid('mail/webmail/hi5', 6.0')=
          'hi5loggedIn'c or
          http_host('hi5.com') or
          html_title('hi5');

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

What do AppID's look like?

  • If you look at the raw text of this traffic, one of the definitions for the mail/webmail/hi5 will hit:
Session Header (3) Meta (5) Attachments (2)
Formatter: ASCII Send to: Download Session Mode: Snippet | Options

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
html_title('hi5');
<title>hi5 | Your Friends. Your World.</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

Registration is quick and easy!
Register

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

2nd Generation AppIDs/Fingerprints

  • Example:
$facebook =
                html_title('Facebook') or
                http_host('.facebook.com'};
appid('social/facebook', 3.0, webproc='Facebook'} =
                 $facebook;

Note the use of the chain word $facebook in the AppID definition

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

2nd Generation AppIDs/Fingerprints

$facebook =
                html_title('Facebook') or
                http_host('.facebook.com'};
appid('social/facebook', 3.0, webproc='Facebook'} =
                $facebook;
GET /yoville/view_gifts.php?giftskip=1 &ist=1 HTTP/1.1
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash
Accept-Language: en-us
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Host: apps.facebook.com
Connection: Keep-Alive
Cookie: datr=1251060871-982d5658affe4152e8816a7958b9b95031b60aea9fffaecd04f34

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

2nd Generation AppIDs/Fingerprints

$facebook =
                html_title('Facebook') or
                http_host('.facebook.com'};
appid('social/facebook', 3.0, webproc='Facebook'} =
                $facebook;

All of these hosts would match this AppID:

Host
platform.ak.facebook.com
vthumb.ak.facebook.com
creative.ak.facebook.com
www.facebook.com
02959290782.channel32.facebook.com
apps.facebook.com
facebook.com
03458988995.channel32.facebook.com
static.ak.facebook.com
b.static.ak.facebook.com
03881417000.channel32.facebook.com
badge.facebook.com

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

2nd Generation AppIDs/Fingerprints

  • Example:
$kaspersky_ip =
        ip('80.239.144.72') or
        ip('80.239.144.73') or
        ip('80.239.144.74') or
        ip('80.239.144.75') or
        ip('80.239.144.76') or
        ip('80.239.144.77') or
        ip('80.239.144.78') or
        ip('80.239.144.79');


appid('antivirus/kaspersky', 1.0) =
        $kaspersky_ip;
appid('antivirus/kaspersky/updater', 5.0) =
        port(21) and $kaspersky_ip;

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

2nd Generation AppIDs/Fingerprints

  • Can you tell what's going on here?
appid('mail/webmail/netlog', 8.0, webproc='Netlog'} =
                html_title('Netlog'c) or
                http_host('.netlog.com') or
                http_cookie(/domain=.{3,10}\.netlog\.com/};

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

2nd Generation AppIDs/Fingerprints

  • Mobile User Agent fingerprints:
fingerprint('browser/cellphone/iphone'} =
        browser('iPhone');

fingerprint('browser/cellphone/motorola'} =
        browser('MOT-'c or 'motorola'};

fingerprint('browser/cellphone/sony_ericsson'} =
        browser('SonyErricsson'};

fingerprint('browser/cellphone/blackberry') =
        browser('BlackBerry');

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

USSID18 Considerations!

  • If you were to query on any of these fingerprints by themselves, would your auditor be happy?
fingerprint('browser/cellphone/iphone'} =
        browser('iPhone');

fingerprint('browser/cellphone/motorola'} =
        browser('MOT-'c or 'motorola'};

fingerprint('browser/cellphone/sony_ericsson'} =
        browser('SonyErricsson'};

fingerprint('browser/cellphone/blackberry') =
        browser('BlackBerry');

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

USSID18 Considerations!

  • But if you were to query on an Afghan IP address that was a valid foreign intel target, and then "AND" it with those fingerprints, that would be a USSID18 compliant query (and your auditor would be happy)

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

3rd Generation AppIDs/Fingerprints

  • 3rd Generation AppIDs/Fingerprints introduced the ability to have code-based scanning
  • Why is this important? Because scanning sessions for keywords, hex values and regular expression can only take you so far.
  • Using Code-based AppIDs, we can run statistical tests of the data that can help determine what type of data it is when keyword scanning can't give us a result.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

4th Generation AppIDs/Fingerprints

  • 4th Generation AppIDs/Fingerprints introduce the ability to extract and database meta-data from Appid/Fingerprints
  • Why is this important?
  • With the dynamic nature of DNI applications, we need the ability to quickly react and deploy solutions to extract new fields of meta-data that are important to analysts

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

4th Generation AppIDs/Fingerprints

  • Previously, if we identified a new protocol or a new field that we wanted to extract meta-data, we would need to upgrade a "core" plug-in and wait until we could upgrade the field sites.
  • With 130 field sites, each on their own upgrade schedule, this could take months for a simple change to get out in the field

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

4th Generation AppIDs/Fingerprints

  • With 4th generation AppIDs, a new protocol, meta-data value, can be properly processed within an hour of updating the AppID/Fingerprint.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

4th Generation AppIDs/Fingerprints

  • Examples:
appid('social/facebook/chat/to_server', 1.0} =
        http_host('facebook.com') and
        $http_post and
        url('/ajax/chat/send.php'}
        : c++
        extractors = {{
          login_email = /login_x=.*([a-z0-9_\-\.]{30}%40[a-z0-9_\-\.]{30}}/;
          text = /msg_text=([^&\n\r]+}/;
        }}
        main = {{
          if (login_email} {
            xks::user_activity_t ua("chat", "facebook");
            ua.client.add(xks::urldecode (login_email[0]), "facebook");
            ua.apply();
          }
          if (text) {
            xks::chat_body(xks::urldecode(text[0]}};
          }

          return true;
        }};

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Facebook Chat V4 Appid Example

  • Let's take a closer look:
  • First a V4 AppID needs to be "anchored". The anchor is the beginning part of the AppID
appid('social/facebook/chat/to_server', 1.0) =
        http_host('facebook.com') and
        $http_post and
        url('/ajax/chat/send.php'}

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Facebook Chat V4 Appid Example

  • DNI Presenter Display:
Session Header (3) Attachments (3) Meta (9)
Formatter: DNI_PRESENTER Send to: Download Session Mode: Snippet | Options

UIS Web Form Display

Form Fields
msg_id: [REDACTED]
client_time: 1250642180342
to: [REDACTED]
num_tabs: 1
pvs_time: 1250642145719
msg_text: dont u still recognize me?
post_form_id: ecba326db1d050497f8a18f8924fa8fd
fb_dtsg: GMFF9ISWX8AX_L7ID-kiN7cL38E
post_form_id_source: AsyncRequest
__a: 1
nctr[id]: c3455f163d438fb1ec7c5a5430fa9432
nctr[nid]: 46fcef7f8c1f286b4d1e0246c2d734a0
nctr[ct]: 1250642184720

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Facebook Chat V4 Appid Example

  • Lets look at the raw:
Session Header (3) Attachments (3) Meta (9)
Formatter: ASCII Send to: Download Session Mode: Snippet | Options Search Content: Enter text to searc

POST http://www.facebook.com/ajax/chat/send.php HTTP/1.1
Host: www.facebook.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.13) Gecko/2009073022 Firefox/3.0.13
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Proxy-Connection: keep-alive
X-SVN-Rev: 181721
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: http://www.facebook.com/editpicture.php?success=1
Content-Length: 366
Cookie: datr=1248211999-a94dd86bl16554d2b5fd014801005bb7e7b6b886c627c920a4e03; s_vsn_facebookpoc_1=1640694104
Pragma: no-cache
Cache-Control: no-cache
msg_id=[REDACTED]client_time=1250642180342&to=[REDACTED]num_tabs=1&pvs_time=1250642145719&msg_text=dont%20u

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Facebook Chat V4 Appid Example

  • The "anchor” of this V4 AppID was present:
appid('social/facebook/chat/to_server', 1.0) =
        http_host('facebook.com') and
        $http_post and
        url('/ajax/chat/send.php'}
POST http://www.facebook.com/ajax/chat/send.php HTTP/1.1
Host: www.facebook.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.13)
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Facebook Chat V4 Appid Example

  • Once the "anchor” hits, the rest of the code executes. In this case, we're looking for these two REGEX's from the "Extractors” section:
extractors = {{
  login_email = /login_x=.*([a-z0-9_\-\.]{30}%40[a-z0-9_\-\.]{30}}/;
  text = /msg_text=([^&\n\r]+}/;
}}

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Facebook Chat V4 Appid Example

This REGEX hits within the large cookie string

login_email = /login_x=.*([a-z0-9_\-\.]{30}%40[a-z0-9_\-\.]{30}}/;

Cookie: [REDACTED]

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Facebook Chat V4 Appid Example

A close look

login_email = /login_x=.*([a-z0-9_\-\.]{30}%40[a-z0-9_\-\.]{30}}/;

login_x=a%3A2%3A%7Bs%3A5%3A%22emai
l%22%3Bs%3A26%3A%22[REDACTED]%40yahoo.com%22%3Bs%3A19%3A%22
remember_me_default%22%3Bb%3A1%3B
%7D;

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Facebook Chat V4 Appid Example

  • The other REGEX:
text = /msg_text=([^&\n\r]+}/;

*   msg_text=dont%20u%20still%20recognize%
    20me%3F&post_form_id

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

Facebook Chat V4 Appid Example

  • Finally, in the "Main" section, if those REGEX's found the data they were looking for, they get databased
main = {{
  if (login_email} {
    xks::user_activity_t ua("chat", "facebook");
    ua.client.add(xks::urldecode(login_email[0]), "facebook");
    ua.apply();
  }
  if (text) {
    xks::chat_body(xks::urldecode(text[0]}};
  }
  return true;

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

4th Generation AppIDs/Fingerprints

  • Another example:
appid('filetransfer/web/zshare.net/upload/response', 5.0}=
        http_title('zSHARE') and 'zshare.net/delete.html'
        : c++
  extractors : {{
    wft_file_name = /The\sfile\s<strong><font\scolor=\"#333333\">([^<]{1,300})\s</;
    wft_delete_url = /zshare.net\/delete.html\?([0-9]+)-([0-9a-zA-Z]{32}}\"/;
    wft_upload_id = /<font color=\"#666666\"><a href=\"http:\/\/www\.zshare\.net\/[^\/]+\/([^/]+)/;
    wft_url = /<font color=\"#666666\"><a href=\"(http:\/\/www.zshare.net\/[^\/]+\/[^\/]+)/;
    wft_uploader_username = /<small>Logged in as: ([^<]+) <\/small>/;
  }}
  main = {{
    if (wft_delete_url ) {
        DB["web_file_transfer"]["wft_upload_id"] = wft_upload_id[0];
        DB["web_file_transfer"]["wft_delete"] = wft_delete_url[0]+"-"+wft_delete_url[1];

        DB["web_file_transfer"]["wft_site_name"] = "zshare.net";
        DB["web_file_transfer"]["transfer_type"] = "upload";

        if (wft_file_name) {
          DB["web_file_transfer"]["wft_filename"] = wft_file_name[0];
        }

        if (wft_url} {
          DB["web_file_transfer"]["wft_url"] = wft_url[0];
        }
        if (wft_uploader_username) {
          DB["web_file_transfer"]["uploader_username"] = wft_uploader_username[0];
        }
        DB.apply();
    } else {
        logger.debug("filetransfer/web/zshare.net/upload/response: Host regexs didn't match");
    }
    return true;
}};

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

FFU Successful Upload Pages

Welcome to zSHARE

With zSHARE you can upload files, images, videos, audio and flash for free. Simply use the upload form below and start sharing! You can also use zSHARE as your personal file storage: backup your data and protect your files. First Time? Read our FAQ!

*   Upload now
*   Login
*   Create Free Account
*   Premium
*   FAQ

File Uploaded

The file wok.rm was successfully uploaded! (18.48MB). You're now ready to share it with unlimited people or keep it as a backup.

Download Link:
http://www.zshare.net/download/6438345621f08561/

Link for forums: [URL=http://www.zshare.net/download/6438345621f085
Direct Link:     http://www.zshare.net/download/6438345621f08561/
Delete Link:     http://www.zshare.net/delete.html?64383456-779939356

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

FFU Successful Upload Pages

  • Again look for the anchor to hit in the raw traffic
appid('filetransfer/web/zshare.net/upload/response', 5.0}=
        http_title('zSHARE') and 'zshare.net/delete.html'
<title>zSHARE - Free File, Image and Video Hosting</title>
value="http://www.zshare.net/delete.html?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

KEYSCORE

FFU Successful Upload Pages

  • Next look for the extractor REGEX's to match
extractors : {{
  wft_file_name = /The\sfile\s<strong><font\scolor=\"#333333\">([^<]{1,300})\s</;
class="textl">The file <strong><font color="#333333">wok.rm </font></strong>
  • Then database what was extracted
main = {{
       if (wft_file_name) {
         DB["web_file_transfer"]["wft_filename"] = wft_file_name[0];

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL