Advanced HTTP Activity Analysis

Open PDF +1 other copy
Creating agency: NSA National Security Agency (US)
Document date: 2009-01-01
Pages: 114
Genre: slides
Banner:
Classification: TOP SECRET COMINT
Dissemination: REL TO USA, AUS, CAN, GBR, NZL
the copy shown on this page
The pages reprinted in the document appendix of No Place to Hide (appendix pages 71-74).

This NSA training document from 2009 teaches intelligence analysts how to analyze HTTP internet traffic and exploit it using the XKEYSCORE surveillance system to monitor and identify internet users globally.

Long Description:
This 114-page NSA training PowerPoint manual provides comprehensive instruction on monitoring and exploiting HTTP web traffic using XKEYSCORE (XKS), one of the NSA's most powerful surveillance tools. It systematically explains HTTP protocol fundamentals, the difference between client-to-server requests and server-to-client responses, and how to extract intelligence value from various HTTP metadata fields including cookies, user agents, referer headers, search queries, and proxy information. The training covers practical targeting techniques including "persona analysis" to build profiles of internet users, methods to identify individuals behind proxy servers using techniques like HTTP Header Fingerprinting (HHFP), and ways to track user activity across webmail services (Yahoo, Gmail, Hotmail), social networks (Facebook, MySpace), search engines (Google, Bing), and other web applications. Special attention is given to cookie exploitation, which can reveal user authentication, demographics, language preferences, and serve as "machine specific" identifiers. The document demonstrates how analysts can reconstruct complete user browsing sessions, identify search terms (even from referer fields when direct collection is unavailable), and distinguish individual users in multi-user proxy environments—particularly relevant for targeting internet users in countries like Iran where ISP-level proxies are common.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Advanced HTTP Activity Analysis

2009

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Goal

The goal of this training is to get you familiar with basic HTTP traffic and understand how to target and expliot it using X-KEYSCORE

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Agenda

  • ?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What is HTTP?

HTTP stands for Hypertext Transfer Protocol and it’s the primary protocol for transferring data on the World Wide Web

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Why are we interested in HTTP?

  • facebook
  • Yahoo!
  • twitter
  • myspace.com a place for friends
  • CNN.com
  • Google Earth
  • mail.ru
  • Wikipedia The Free Encyclopedia
  • Gmail by Google BETA

Because nearly everything a typical user does on the Internet uses HTTP

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Why are we interested in HTTP?

  • Almost all web-browsing uses HTTP:
    • Internet surfing
    • Webmail (Yahoo/Hotmail/Gmail/etc.)
    • OSN (Facebook/MySpace/etc.)
    • Internet Searching (Google/Bing/etc.)
    • Online Mapping (Google Maps/Mapquest/etc.)

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

How does HTTP work?

  • HTTP is comprised of requests from clients to servers and their corresponding responses
  • Many analysts are already familiar with the terms “client-to-server” or “server-to-client” collection (also referred to as “client side” or “server side” collection).

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

How does HTTP work?

  • A “Client” is usually referring to a Browser (like Firefox or IE) which is also referred to as the “User Agent”
  • The “Server” can also be referred to as the “web-server” or “origin-server” which is the machine that is storing the data that is being accessed (like a web-page, a map, an inbox, etc)

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity

  • HTTP activity comes in two types:

Website.com
Client-to-Server “requests”
Server
Server-to-Client “responses”
Client
User

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity

  • HTTP activity comes in two types:

Website.com
Server
Internet
Client
User

While there may be a variety of Proxies, Gateways or Tunnels in between the client and the server, traffic is always going in one direction or the other.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Client vs. Server Side Traffic

  • How do you know which side you’re looking at?
  • Client-to-Server requests are generally small in size and are computers talking to other computers
  • They contain standard HTTP header fields like “Host:” “Accept:” “Connection” etc.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity Examples

Client-to-Server request:

TOP SECRET//COMINT//20320108
ID: sess_orig_proc
Type: HTTP-GET
Printer Friendly Version
DNI Display | Raw Data | DNI Format
Services

GET /Hezbollah-Terrorism-Judith-Palmer-Harik/dp/1860648932 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/1.0.154.48 Safari/525.19
Referer: http://www.google.com.pk/search?hl=en&q=written books on hizbollah&btnG=Google Search&meta=
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: gzip, deflate, bzip2, sdch
Cookie: ubid-main=185-5525816-8765531
apn-user-id=P1YXY7QF1PUYQ5
Accept-Language: en-US,en
Accept-Charset: ISO-8859-1,*,utf-8
Host: www.amazon.com
Connection: Keep-Alive

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Client vs. Server Side Traffic

  • Server-to-Client responses are generally larger in size and are what web-pages look like at the internet.
  • When you’re at a computer accessing the Internet, you’re only seeing Server-to-Client traffic.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity Examples

Server-to-Client Response:

ID: sess_orig_proc
+ Document information
Type: HTTP
Printer Friendly Version
DNI Display Raw Data DNI Forma
+ HTTP Header Information
Services
Barca reinstates 6-point lead over Real
Isfahan to exhibit expressionist art
Home Page
Iran
Middle East
Iraq
Palestine
Lebanon
Turkey
Persian Gulf
Others
US
Asia/Pacific
Africa
Europe
Americas
Sci/Tech
Health
Kuwait government 'resigns' over economy
Mon, 16 Mar 2009 19:07:16 GMT
The Kuwaiti government has submitted its resignation to the county's emir amid a row over the premier's handling of the economic crisis. "The resignation has been submitted formally and it's up to the emir (ruler) to decide," Reuters quoted Nasser al-Duwailah, a parliamentarian, as saying on Monday. The resignation would further delay the approval of 5 billion dinars (USD 5.11 billion) rescue package which is to be injected to the Persian Gulf nation's economy to ease the impact of global financial crisis. The government has not commented on the report.
Go
Latest News
Kuwait govern economy
Childhood diet risk
'US-Russian pa shield row'
Judges want M confiscated
Leader pardon
Ancient book
Lieberman eyes ally
Intelligent peop

Bonus question: Why are the images in this web-page missing?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity

  • XKS HTTP Activity Meta-data differs greatly depending on which side of traffic we’re collecting
  • In nearly all cases it’s better to have client-to-server traffic

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity Client-to-Server

GET /search?tab=urdu&order=sortboth&q=musharraf&start=3&scope=urdu&link=next HTTP/1.1
Accept: */*
Referer: http://search.bbc.co.uk/search?tab=urdu&order=sortboth&q=musharraf&start=2&scope=urdu
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: search.bbc.co.uk
Cookie: BBC-UID=b479a5f4ad230a53063d513630203acb22584634a0e0b164c45f96efc054cf950Mozilla%2F4%2E0%20%28comp
Cache-Control: max-stale=0
Connection: Keep-Alive
X-BlueCoat-Via: 66808702E9A98546
Host
search.bbc.co.uk
URL Path
/search
URL Args
tab=urdu&order=sortboth&q=musharraf&start=3&scope=urdu&link=next
Search Terms
musharraf
Language
en
Browser
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Via
66808702E9A98546
Referer
http://search.bbc.co.uk/search?tab=urdu&order=sortboth&q=musharraf&start=2&scope=urdu
Cookie
BBC-UID=b479a5f4ad230a5306d351630023cb2260463a40e0b164c45f9e6cf054cf950Mozilla%2F4%2E0%20%28comp

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity Server-to-Client

Application Info
Press TV - Kuwait government 'resigns' over economy
HTTP Type
response

ID: sess_orig_proc
+ Document information
Type: HTTP
Printer Friendly Version
DNI Display Raw Data DNI Forma
+ HTTP Header Information
Services
Barca reinstates 6-point lead over Real
Isfahan to exhibit expressionist art
Home Page
Iran
Middle East
Iraq
Palestine
Lebanon
Turkey
Persian Gulf
Others
US
Asia/Pacific
Africa
Europe
Americas
Sci/Tech
Health
Kuwait government 'resigns' over economy
Mon, 16 Mar 2009 19:07:16 GMT
The Kuwaiti government has submitted its resignation to the country's emir amid a row over the premier's handling of the economic crisis. "The resignation has been submitted formally and it's up to the amir (ruler) to decide," Reuters quoted Nasser al-Duwailah, a parliamentarian, as saying on Monday. The resignation would further delay the approval of 5 billion dinars (USD 5.11 billion) rescue package which is to be injected to the Persian Gulf nation's economy to ease the impact of the global financial crisis. The government has not commented on the report.
Go
Latest News
Kuwait govern economy
Childhood diet risk
'US-Russian pa shield row'
Judges want M confiscated
Leader pardon
Ancient book
Lieberman eyes ally
Intelligent peop

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity – HTTP Types

  • Meta-data will also tell you which side of traffic you’re looking at
  • Client-to-server has two main types:

HTTP Type
get
HTTP Type
post

  • Server-to-client has only one:

HTTP Type
response

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity – Get vs Post

  • A ‘GET’ is you requesting data from the server (most web surfing)
  • A ‘POST’ is you sending data to the server (i.e. signing in, filling out a form, composing an E-mail, uploading a file etc.)

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL

Let’s break down the important parts of a client-to-server request

TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Client-to-Server

GET /home.html
Host: sample.website.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

First thing to note is the Host: line which tells you the name of the server that the client is requesting data from

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Host Field

It’s important to note, that in many cases users think they’re at websites like www.yahoo.com, but behind the scenes data is coming from a number of different servers without the user knowing it:

GET /mc/modules/mr/abContacts?mcnumb=RIIDb59jm & jsrand=98037807 & rand=2127033459 HTTP/1.0
Accept: */*
Accept-Language: fa
Referer: http://us.mc575.mail.yahoo.com/mc/showFolder?.ylc=X3oDMTBucmNobGR0BFSTA2M5ODMwMTAyNwRhYWNkZxNx2dz?mid=1_21857_AERkxEIAANvjSi6wUQ7fiLz4fY&fid=Inbox&sort=date&order=up&startMid=36&filterBy=
x-requested-with: XMLHttpRequest
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)
Host: us.mc575.mail.yahoo.com
Cookie: MG d=IvAXlFvaYnFGmmIfzw3zBCVVRe2jUKZLwwyoKSrjxxG0XVYaJhF95dLsZ5C0x1eDlcTcaHS_vpi
Y

Bonus question: What would the impact of this be in how you formulate your X-KEYSCORE queries using the Host field?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Client-to-Server

GET /home.html
Host: sample.website.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

Second the GET line tells you which files the user is requesting from the server.
If you simply take that line and append it to the Host line you have the live public URL that the user is requesting:
http://sample.website.com/home.html

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Client-to-Server

GET /example.php?region=iraq
Host: sample.website.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

When the GET line has a ? mark in it, then the GET request is also passing information to the server.
So in this case the client is requesting the file example.php but it’s also passing along a value that could have been entered by the user.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

URL Lines

When there is a ? mark in the URL line, then X-KEYSCORE is breaking it up into two parts. The first part is called the URL Path and the second part is called the URL Argument.

URL Path URL Args
/search tab=urdu&order=sortboth&q=musharraf&start=3&scope=urdu&link=next

Notice all of the “arguments” (each separated by &’s) in this URL:

GET /search?tab=urdu&order=sortboth&q=musharraf&start=3&scope=urdu&link=next HTTP/1.1
Accept: */*
Referer: http://search.bbc.co.uk/search?tab=urdu&order=sortboth&q=musharraf&start=2&scope=urdu
Accept-Language: en-us
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: search.bbc.co.uk
Cookie: BBC-UID=b479a5f4ad230a53063d513630203acb22b84634a0e0b164c45f96efc054c950Mozilla%2f4%2e0%20%2bcom
Cache-Control: max-stale=0
Connection: Keep-Alive
X-BlueCoat-Via: 66808702E9A98546

Bonus question: Any idea what the information that is being passed in the URL Argument in this example are for?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Client-to-Server

GET /home.html
Host: sample.website.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

The User-Agent line gives you information on what type of client is requesting the data. In this case, we can see that it was a Firefox 3.0 browser from a Windows NT 5.1 (XP) machine.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL

User Agents

TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

User Agents

The User Agent (also known as the “browser”) can be very valuable.
While it can not be trusted to be absolutely unique, in many cases you can use it to unwind a proxy or multi-user environment.
It can also help provide hints if the origins of the request came from a mobile device:

User-Agent: Mozilla/5.0 (SymbianOS/9.2; U; Series60/3.1 NokiaE63-1/100.21.110; Profile/MIDP-2.0 Configurat like Gecko) Safari/413
User-Agent: NokiaN72/5.0706.4.0.1 Series60/2.8 Profile/MIDP-2.0 Configuration/CLDC-1.1
User-Agent: iPhone Mail (5H11)

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Client-to-Server

GET /home.html
Host: sample.website.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

The various “Accept” lines instruct the server on the types of responses the client can accept back.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL

Let’s look at a simplified version of a HTTP request and response

TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL


What is Web (HTTP) Activity

This shows how a person logs on to a webpage

From Port 3434* (client)
Click on http://www.hotmail.com GET Request
To Port 80 (Server)
MSN

The client’s port can be any high-numbered port, 3434 is just an example

What is Web (HTTP) Activity


This shows how a person logs on to a webpage

From Port 3434* (client)
Click on http://www.hotmail.com GET Request
To Port 80 (Server)

To Port 3434* (client)
"Welcome to Hotmail" HTTP Response
From Port 80 (Server)
MSN

The client's port can be any high-numbered port, 3434 is just an example

What is Web (HTTP) Activity


This shows how a person logs on to a webpage

From Port 3434* (client)
Click on http://www.hotmail.com GET Request
To Port 80 (Server)

To Port 3434* (client)
"Welcome to Hotmail" HTTP Response
From Port 80 (Server)

From Port 3434* (client)
Email Address: me@hotmail.com
Password: Admin123
POST to the Web server
To Port 80 (Server)
MSN

The client’s port can be any high-numbered port, 3434 is just an example


What is Web (HTTP) Activity

This shows how a person logs on to a webpage

From Port 3434* (client)
Click on http://www.hotmail.com GET Request
To Port 80 (Server)

To Port 3434* (client)
"Welcome to Hotmail" HTTP Response
From Port 80 (Server)

From Port 3434* (client)
Email Address: me@hotmail.com
Password: Admin123
POST to the Web server
To Port 80 (Server)

To Port 3434* (client)
"Welcome to your Inbox/homepage" HTTP Response
From Port 80 (Server)
MSN

The client's port can be any high-numbered port, 3434 is just an example


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity

  • Real traffic, however, can be a little more complicated.
  • Almost all web pages are built from multiple files.
  • For example, every single image or banner ad on a web page is a separate file that needs to be individually requested before the server that has the file can respond

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity – Real World

  • Let’s look at the “NSA Today” home page.
Dynamic Page -- Highest Possible Classification is
TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL
NATIONAL SECURITY AGENCY
UNITED STATES OF AMERICA
NSA DAILY
... the best minute you'll spend today
Current Conditions
Weather Information

Daily's Hot Links Search | Searchlight | Ext. Connections | Orgs | Projects | EA Outages

3 Oct 2009
TAKE 2
XmedMedia
Agency Mass Mailers
Today's EVENTS
Multivision Ads
Technology Expo
Weight Watchers @
NSA Daily "Need to Know"
Past News »
(U//FOUO) NSAG Hosts the CENTCOM Senior Enlisted Leader
(U//FOUO) On 9 September 2009, NSA/CSS Georgia (NSAG) had the privilege of hosting a visit from Command Sergeant Major (CSM) Marvin Hill, Command Senior Enlisted Leader for United States Central Command (CENTCOM).
FOR OFFICIAL USE ONLY
(U//FOUO) Pictured: NSA Georgia CSM Lori Brown welcomes Command Sergeant Major Marvin Hill, Command Senior Enlisted Leader for United States Central Command.
(U//FOUO) This was CSM Hill's first visit to NSAG. The first stop during his visit was with ITD at the helpdesk. SGM Jacobs, of ITD, provided the CSM an overview of our maintenance operations and communications hub. CSM Hill told the ITD personnel, "you keep doing what you do, my comms run through here....and I need my comms"
NSA IN THE NEWS
From the DIRECTOR
LTG Keith B. Alexander; United States Army
Mission Messages
Top IAD News
(U) Securing the

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity – Real World

  • It looks like one page, but each of the images and banners are separate data files that your browser pieces back together

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity – Real World

  • In fact, to build the NSA Today home page it takes 34 separate files from 4 different servers
  • However, most people probably don’t notice, because the entire page loads in <300 milliseconds.
  • If we had a slow internet connection, we’d notice the images would initially be missing.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity Real-Word

+ Document information
Type: HTTP
Printer Friendly Version
DNI Display Raw Data DNI Forma
+ HTTP Header Information
Services
Barca reinstates 6-point lead over Real
Isfahan to exhibit expressionist art
Home Page
Iran
Middle East
Iraq
Palestine
Lebanon
Turkey
Persian Gulf
Others
US
Asia/Pacific
Africa
Europe
Americas
Sci/Tech
Health
Kuwait government 'resigns' over economy
Mon, 16 Mar 2009 19:07:16 GMT
The Kuwaiti government has submitted its resignation to the country's emir amid a row over the premier's handling of the economic crisis. "The resignation has been submitted formally and it's up to the amir (ruler) to decide," Reuters quoted Nasser al-Duwailah, a parlametarian, as saying on Monday. The resignation would further delay the approval of 5 billion dinars (USD 5.11 billion) rescue package which is to be injected to the Persian Gulf nation's economy to ease the impact of the global financial crisis. The government has not commented on the report.
Go
Latest News
Kuwait govern economy
Childhood diet risk
'US-Russian pa shield row'
Judges want M confiscated
Leader pardon
Ancient book
Lieberman eyes ally
Intelligent peop

Notice that all of the images are missing. They are all separate server-to-client responses and therefore completely separate “sessions” in X-KEYSCORE or PINWALE

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity – Real World

  • It’s important to note that not all of the data on one web-page came from the same server.
  • For example, most of the NSA Today home page come from home.www.nsa, but the image of the current weather conditions came from wk-admiral208.corp.nsa.ic.gov

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity – Real World

  • This happens all the time on the Internet.
  • The cnn.com home page, may have an ad on it that was from the Google ad server and etc.
  • And this does have an impact on our collection!

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity

  • This is the traffic path for building the NSA today home page

home.www.nsa
corpweb1.nsa
siteworks.nsa
wk-admiral208.corp.nsa.ic.gov
User

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

HTTP Activity

  • What happens if we only have collection on one of the paths?

home.www.nsa
corpweb1.nsa
siteworks.nsa
wk-admiral208.corp.nsa.ic.gov
User

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What would that traffic look like?

GET /current.jpg
Host: wk-admiral208.corp.nsa.ic.gov
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10
Accept: image/png,image/*;q=0.8,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://home.www.nsa/
If-Modified-Since: Thu, 08 Oct 2009 19:31:56 GMT
If-None-Match: "d945-16c1-842db643"
Cache-Control: max-age=0

If we only saw this one GET request and not the other 33 required to build the NSA Today home page, would we be able to determine what the user was actually doing?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What exactly is that telling us?

  • First off, we know what file they are requesting.
  • They want current.jpg from the wk-admiral208.corp.nsa.ic.gov server.
  • That’s actually a live public URL (http://wk-admiral208.corp.nsa.ic.gov/current.jpg)
  • Do we have any indication why they wanted that image? Answer is yes! Look at the referer field.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What exactly is that telling us?

  • They were referred from http://home.www.nsa/
  • The referer is in essence, telling you what site was “linking” to the new site.
  • Warning! The referer can act in misleading ways.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Referer Field

  • The referer field is the address of the page that links to new GET request.
  • However, this link could have been automatic to the user.
  • I.e. in the case of the current weather image, the link was automatic and the user wasn’t even aware of the action

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Referer Field

  • The referer field could also indicate a user action.
  • For example, imagine we were on the NSA Today webpage and clicked the link to the SID Today page.
  • What would that traffic look like?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Referer Field

GET /
Host: sidtoday.nsa
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://home.www.nsa/
Cookie: CFID=565238; CFTOKEN=66534796;
CFGLOBALS=urltoken%3DCFID%23%3D565238%26CFTOKEN%23%3D66534796%26jsessionid%23%3Da830dba3a04b67ae6e351b7463444f72496d%23lastvisit%3D%7Bts%20%272009%2D10%2D09%2015%3A38%3A04%27%7D%23timecreated%3D%7Bts%20%272009%2D06%2D19%2010%3A27%3A23%27%7D%23hitcount%3D13%23cftoken%3D66534796%23cfid%3D565238%23;
JSESSIONID=a830dba3a04b67ae6e351b7463444f72496d

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Referer Field

  • Now we’re seeing a request go to host “sidtoday.nsa” with the referer from http://home.www.nsa
  • How can we tell from the traffic that the first automatic referer we saw for the current weather was any different from the user-generated referer we saw for the SID Today article?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL

Cookies!

TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Cookies

  • Cookies are small pieces of text-based data stored on your machine by your web browser.
  • Almost all websites have cookies enabled and they have a variety of uses, including to help the web-site track the activities of their users.
  • Most analysts are probably familiar with “machine specific cookies” like the Yahoo B cookie
  • However cookies are used for a variety of reasons

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What can cookies be used for?

  • Cookies can be used to authenticate a user.
  • For example in many cases, the “active user” for Yahoo web-mail traffic is seen encoded in the I= part of the cookie string.
Y
v=1
n=4edg46n653aef
l=eibj0d_10o0p838to (Yahoo login id: [REDACTED])
p=f2kvvsy0120000000 (Gender: female, Birth year: 1984, Postal code: [REDACTED])
iz=
r=jb
lg=en-US ( Language/content: English )
intl=us ( Country: United States )

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What can cookies be used for?

  • Cookies can be used to store information about the user that the website is interested in
  • Look at how the p= value below tells the website information about the user of this account:
v=1
n=4edg46n653aef
l=eibj0d_10o0p838to (Yahoo login id: [REDACTED])
p=f2kvvsy0120000000 (Gender: female, Birth year: 1984, Postal code: [REDACTED])
iz=
r=jb
lg=en-US ( Language/content: English )
intl=us ( Country: United States )

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What can cookies be used for?

  • Cookies can be used to identify a single machine from hundreds of other users on the same proxy IP address
  • The Yahoo B cookie is a “machine specific cookie”
B
f6146fh596u4b
b=4
d=GtdlgXBpYEQvWQEGtVnWhaxIPNw-
s=9e
i=8OSsR4OwqEO5oGGF2kJh

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What can cookies be used for?

  • Important note: All three of those examples are just subsets of the full Yahoo cookie string

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

How do we know what each cookie value is used for?

  • Nearly every web-site uses cookies that in most cases they designed for their own uses, so how do we know what they all mean?
  • Protocol Exploitation can examine the traffic to try to determine if there is any information contained in cookie strings that we might be interested, for example we’d like to know if any part of the cookie acts like a “machine specific cookie.”

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

How do we know what each cookie value is used for?

  • However, there are far more cookie options out in the wild than PE can possible examine.
  • So even if they aren’t aware of a machine specific cookie, it doesn’t mean that it doesn’t exist.
  • X-KEYSCORE gives you access to the full cookie string, so if you’re adventurous enough you can do your own protocol exploitation.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Remember: Cookies are there for a reason!

  • Websites put cookies on people’s computers for a reason.
  • If the data is valuable for a website, it may be valuable to us as well.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

How long do cookies live for?

  • Cookies, like any other file on a computer, can be deleted by the user.
  • Almost all browsers give you the option to view, manage and delete your cookies
Cookies
Search:
The following cookies are stored on your computer:
Site Cookie Name
bblux110.gp.proj.nsa.ic.gov
brain.r6.r.nsa
calendar.ops.s.nsa
centerjohn.hqs.do.nsa
csrc.do.nsa
cybertrans-secure.eitc.nsa.ic.gov
journal.ibd.nsa
Name: <no cookie selected>
Content: <no cookie selected>
Host: <no cookie selected>
Path: <no cookie selected>
Send For: <no cookie selected>
Expires: <no cookie selected>
Remove Cookie Remove All Cookies Close

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Cookies

You can see what cookies have been stored on your machine by going into the "options" window of your browser and selecting ‘show cookies”

Options
Main Tabs Content Applications Privacy Security Advanced
History
Keep my history for at least 90 days
Remember what I enter in forms and the search bar
Remember what I've downloaded
Cookies
Accept cookies from sites
Accept third-party cookies
Keep until: they expire
Exceptions...
Show Cookies...
Private Data
Always clear my private data when I close Firefox
Ask me before clearing private data
Settings...
Clear Now
OK Cancel Help

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL

Searches

TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Searching the Internet

When a user searches the Internet from one of the many web-based search engines (Google, Bing, etc.) what does the traffic look like?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Searching the Internet: Client-to-Server

  • In most cases, the client-to-server traffic is a GET request where the search term is passed in the URL Arguments:
GET /search?hl=fr&q=iran&lr= HTTP/1.1
Host: www.google.com
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/msword, */*
Cookie: PREF=ID=74f6d7addf51ccd4:U=ccbee9ee665a7dde:TB=2:TM=1255354439:LM=1255433264:S=_ M1i4RfO2ohI81maNID=27=cMFLkpovJCIWI0FC5E3Pu2C6-8_nsMS2zztfvOew9-QYDPWUza4AscyoglQRGNSkDZsi2jL65 flM-R4HgovMBEa66bfiTXn8TH3Ukm-X5hp45rLAb_Y3rNZ42HGIzyne
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Connection: Keep-Alive
Cache-Control: no-cache

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Searching the Internet: Client-to-Server

  • Notice how the URL Path is /search and one part of the URL argument is q=iran
  • Each website can configure their URL’s differently, so while with Google the search term is contained in the q= part of the URL, a different search form might have it as query= or search_term= etc.

http://www.youtube.com/results?search_query=iran&search_type=&aq=o

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Searching the Internet: Client-to-Server

  • X-KEYSCORE tries to account for all the variations of search terms contained in the URL Argument for what it extracts for the “Search Term” column.
  • However, there are always other varieties out there that we haven’t built it hooks for yet, so anytime you see something that you think should be extracted, please contact the team ([REDACTED])

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

“Referer Searches”

  • What happens when a user clicks on a search result?
  • Let’s start by showing the query itself, in this example, we’re going to query the NSANet Google for “XKEYSCORE”

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What does that GET request look like?

GET /search?q=xkeyscore&btnG=Google
Host: google4.q.nsa
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive

We know from this session that the client is requesting the data from the host ‘google4.q.nsa” and we see the search term in the URL Argument

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

“Referer Searches”

  • What happens when a user clicks on a search result?
GET /redmine
Host: xkeyscore.r1.r.nsa
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.10) Gecko/2009042316 (USG-25) Firefox/3.0.10
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Cookie: _session_id=ffd87ac8682e8fa8f421b4ffdf9693ae
Referer: http://google4.q.nsa/search?q=xkeyscore&btnG=Google+Search

First, we can determine the full URL by adding the GET line to the host: http://xkeyscore.r1.r.nsa/redmine/

xkeyscore.r1.r.nsa/redmine
XKEYSCORE - Overview - XKEYSCO...

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

“Referer Searches”

  • Secondly, we get some hints as to why the user was requesting that page from the Referer line:

Referer: http://google4.q.nsa/search?q=xkeyscore&btnG=Google+Search

  • Note that it was the same URL that we were at immediately before we clicked the “result” link

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

“Referer Searches”

  • Let’s look at that process again:

google4.q.nsa
xkeyscore.r1.r.nsa
First, a client-to-server request is sent that contains the query on "xkeyscore"

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Referer Searches

  • Let’s look at that process again:

google4.q.nsa
xkeyscore.r1.r.nsa
Second, the server responds back with the search results

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

“Referer Searches”

  • Let’s look at that process again:

google4.q.nsa
xkeyscore.r1.r.nsa
Third, by clicking on one of the results, a new GET request is issued to retrieve the XKEYSCORE home page. In this request, the location of the original search is listed as the "referer"

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

google4.q.nsa
xkeyscore.r1.r.nsa

What will happen if we only have collection on this link?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

“Referer Searches”

  • When XKEYSCORE sees a search contained in the “referer” field, we still extract it out as meta-data into the “search terms” but we append it with (referer) to denote where it was originally found:
Search Terms
(referer) the legal status of the caspian sea
HTTP Type Host URL Path URL Args
get www.parstimes.com /law/caspian_status.html
Referer: http://www.google.com/search?hl=fa&source=hp&q=the+legal+status+of+the+caspian+sea&lr=

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

“Referer Searches”

GET /law/caspian_status.html HTTP/1.1
Accept: */*
Host: www.parstimes.com
Referer: http://www.google.com/search?hl=fa&source=hp&q=the+legal+status+of+the+caspian+sea&lr=
Accept-Language: fa
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; InfoPath.2)
Cache-Control: max-stale=0
Connection: close
X-BlueCoat-Via: 0A6F53530F3F63EE

Can we guess what happened here?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Another example

TOP SECRET//COMINT//20320108
ID: sess_orig_proc
Type: HTTP-GET
Printer Friendly Version
DNI Display | Raw Data | DNI Format
Services

GET /Hezbollah-Terrorism-Judith-Palmer-Harik/dp/1860648932 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/1.0.154.48 Safari/525.19
Referer: http://www.google.com.pk/search?hl=en&q=written books on hizbollah&btnG=Google Search&meta=
Accept: text/xml,application/xml,application/xhtml xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: gzip, deflate, bzip2, sdch
Cookie: ubid-main=185-5525816-8765531
apn-user-id=P1YXY7QF1PUYQ5
Accept-Language: en-US,en
Accept-Charset: ISO-8859-1,*,utf-8
Host: www.amazon.com
Connection: Keep-Alive

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL

Proxy Information

TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Proxy Information

  • In a lot of cases we’re going to see HTTP Activity from behind a proxy or proxies.
  • What is a proxy?
    • A proxy is a server that is acting as an intermediary for HTTP requests from clients
  • Why do proxies exists?
  • Performance: Proxy can cache responses for static pages
  • Censorship: Proxy can filter traffic
  • Security: Proxy can look for malware
  • Access-Control: Proxy can control access to restricted content

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Proxy Information

  • Routinely, we’re going to see ISP level proxies.
  • That is, instead of having each individual user request web pages directly from the web servers, the ISP is going to collect all of those requests first, and then proxy them out through a handful of proxy IP addresses.
  • When the response is returned, the proxy passes it on to the appriopriate user

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Proxy Information

  • Why would the ISP want to proxy traffic?
  • In many cases the ISP won’t have to supply public IP addresses to all it’s users
  • It can simply give them a private IP address, and then use a handful of public IP addresses for its proxies which are the machines actually requesting the traffic from the web-servers

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Proxies on the Internet

Web-Server
Short-lived connections Single-user
Web-Servers
Short-lived connections Multiple-users multiplexed
Web-Servers
Long-lived connections Multiple-users multiplexed
The Internet
Direct-Connect
Mixed-Gateway
National-Level Proxy
Cache
Proxy-to-Proxy
User-to-Proxy

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Identifying a Proxy

  • How do you know that the IP address that you think is your target is really a proxy?
  • First step, check NKB.
  • They have services that attempt* to automatically detect proxies
  • These services are in no way 100% accurate so this is only the first step in checking to see if the IP Address is a proxy

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Identifying a Proxy: NKB

Query: IP Address [REDACTED]
Date: 2009-10-27@09:21:50
Description Value Confidence Classification
Location Add Analyst Input
IP Range [REDACTED] [REDACTED] (TS//SI//REL TO USA, FVEY)
Lat/Long (precision) (none found) (TS//SI//REL TO USA, FVEY)
City ZAHEDAN 20 (TS//SI//REL TO USA, FVEY)
Country (IR) IRAN 91 (TS//SI//REL TO USA, FVEY)
Provider Add Analyst Input
IP Owner RAYANE FARAZ IRANSHAHR COMPANY, INTERNET SERVICE PROVIDER 82 (U//FOUO)
Autonomous System Route Prefix [REDACTED]0.0/17 59 (U//FOUO)
Autonomous System Number 12880 95 (U//FOUO)
Autonomous System Name DCI-AS DCI Autonomous System 95 (U//FOUO)
FQDN (none found) (U//FOUO)
Domain [REDACTED] 30 (U//FOUO)
Service PROXY (U//FOUO)
Service TRANSPARENT PROXY (U//FOUO)

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Identifying a Proxy

  • Other things to be on the look out for:
  • X-Forwarded-For IP Address
    • What is it?
    • An X-Forwarded-For IP address the proxy passing on to the server what it thinks is the IP address of the user
    • Think of it as the proxy telling the server “this is who I think this request came from”
    • It’s important to note that multiple proxies can, and often, are present, so one proxy might just be reporting the IP address of another proxy

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Identifying a Proxy

  • X-Forwarded-For IP Address as seen in traffic:
GET /HTTP/1.0
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: www.ebay.com
Pragma: no-cache
Via: 1.0 s.jonoobnet.com (squid/3.0.STABLE10)
X-Forwarded-For: [REDACTED]
Cache-Control: max-age=259200
Connection: keep-alive

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Some Examples of X-Forwarded-For headers:

X-Forwarded-For: [REDACTED]
X-Forwarded-For: [REDACTED]
X-Forwarded-For: [REDACTED]
X-Forwarded-For: [REDACTED]
X-Forwarded-For: [REDACTED]
X-Forwarded-For: 192.168.1.10, 10.0.0.22,[REDACTED]
X-Forwarded-For: [REDACTED]
X-Forwarded-For: 127.0.0.1,[REDACTED]
X-Forwarded-For: [REDACTED]google.com,[REDACTED]

Multiple-Layers of Proxies!

In-general, the first IP is the one closet to the original requestor
Keep in mind – these can be totally fake

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Identifying a Proxy

  • Similar to the X-Forwarded-For Tag is the “VIA tag”
  • The VIA tag is the proxy identify itself
GET /HTTP/1.0
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: www.ebay.com
Pragma: no-cache
Via: [REDACTED] (squid/3.0.STABLE10)
X-Forwarded-For: 217.219.95.135
Cache-Control: max-age=259200
Connection: keep-alive

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Identifying a Proxy

  • The Via: tag may even contain some good information about the proxy
  • Be careful though because this information could be falsified:

Via: 1.0 tehran-proxy-srv:3128 (squid/2.5.STABLE1)

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Identifying a Proxy

  • Remember though that the X-Forwarded-For and VIA lines can be falsified and don’t have to be present!
  • If they’re not present, how can you tell the IP address is a proxy?
  • Test it in MARINA!

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Testing IP Addresses in MARINA

  • The primary side effect of a proxy is too many users online at the same time
  • So if all else fails, try querying on the IP address (assuming its USSID18 compliant of course!) in MARINA to see how many users were active within an hour time frame
  • It’s not scientific but generally it will help

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Testing IP Addresses in MARINA

For example look at these results:

Specify Date Range (YYYYMMDD [hhmmss]): 200910260100
to 200910260200
<Select>
Data available back to 1 Dec 2008
Search for User Activity by... Strong Selectors (Emads, IDs, Cookies, Mail Tokens, Phone Numbers, AppProcIPs, AppProcMacs)
that... exactly match
the value(s)... [REDACTED]
DecodeOrdain

3178 Records 1-500

There were 274 unique “Active Users” in that hour, think it’s a proxy?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL

HTTP Header Fingerprint (HHFP)

DERIVED FROM: NSA/CSSM 1-52
08
TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What is the HHFP?

  • GCHQ created the HHFP to help identify individual users behind a single proxy IP address
  • The HHFP is a hash of multiple header fields that can be used to identify a single user behind a proxy

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What is the HHFP?

  • At least one of these values must be present:
    • X-Forwarded-For IP Address
    • Via
    • Client IP address
  • If so, the HHFP is a hash of those values combined with the User Agent string

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

What is the HHFP?

  • EX: Here’s an Iranian proxy IP Address that has multiple HHFP’s underneath it.
  • NOTE: There’s no guarantee that an HHFP is identifying a single unique user, it’s entirely possible that more than one user will have the same HHFP
[REDACTED] (32) 0%
  008b6e2c (1) 3%
  018a707f (1) 3%
  0932e653 (1) 3%
  0aee1edd (1) 3%
  0ba2b5e1 (1) 3%
  0ced7c48 (1) 3%
  13312787 (1) 3%
  135c8dc3 (1) 3%
  19429340 (1) 3%
  18dda1fa (1) 3%
  1a171e21 (1) 3%
  1dd33d95 (1) 3%
  1f661ca6 (1) 3%
  20f8c73f (1) 3%
  219109f0 (1) 3%
  225c2cfb (1) 3%
  23e57929 (1) 3%
  2a0150bd (1) 3%
  2d504fe1 (1) 3%
  2f8bad21 (1) 3%
  31b545bd (1) 3%
  3a07f515 (1) 3%
  3c785e51 (1) 3%
  45340ef9 (1) 3%
  5470cbdc (1) 3%
  73138ecd (1) 3%
  941f97d9 (1) 3%
  9b3392a2 (1) 3%
  a011fc14 (1) 3%
  ac062e81 (1) 3%

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Pros and Cons of HHFP

  • On the positive side, the HHFP is a single 8 digit value which can help identify a single user behind a proxy
  • On the negative side, it requires an XFF IP address, Via string or Client IP Address and since many sessions do not contain all three, they’ll have no HHFP string
  • Also even with the HHFP, all of the fields that are used to build it are available in the XKS HTTP Activity query so it’s not providing you with any data you don’t already have access to

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL

XKS’s HTTP Activity Search

DERIVED FROM: NSA/CSSM 1-52
08
TOP SECRET//COMINT//ORCON/REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKS HTTP Activity Search

After that overview of how HTTP Activity works, let’s look into how to effectively target it through XKS queries

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKS HTTP Activity Search

  • HTTP Activity indexes every HTTP session
  • Client-to-server and server-to-client
  • Can be queried on any of the unique HTTP meta-data fields or any of the “standard” DNI fields (IP Address, SIGAD, CASENOTATION etc).

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKS HTTP Activity Search

  • Unique Meta-data fields of this search include:

Fields already covered in this training:

HTTP Type:
Host:
URL Path:
URL Args:
Search Terms:
Language:
Active User:
TDI Type:
TDI:
Character Encoding:
Content Start:
Content Stop:
Content Total:
Referer:
X Forwarded For:
Via:
Proxy Hash (HHFP):
Cookie:
Browser:
Attachment Filename:
Server Type:
Geo Info [fulltext]
Misc Info [fulltext]:
Links of Interest:

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKS HTTP Activity Search

In addition to all of the common fields like:

IP Address: From
IP Address: To
Port: From
Port: To
Country: From
Country: To
City (IP): From
City (IP): To
SIGAD:
Casenotation:
Session ID (UUID):
Application Type:
Application Info:
Application:
AppID (+Fingerprints) [fulltext]:
Data Length:
Session Length:
DVBS MAC:
DVBS PID:
SMAC:
DMAC:

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKS HTTP Activity Search

  • Most commonly HTTP Activity query searches in XKS will be to enable “persona analysis”
  • Based on MARINA, TRAFFICTHIEF or PINWALE, we’ll want to query XKS to discover all of the HTTP Activity that occurred around the targets session of interest

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Simple HTTP Searches

  • In order to do a “persona analysis” type search, all we’ll need to fill in is the IP of the target (assuming it’s USSID18 compliant) and a short time range “around” the time of the activity:
Datetime: Custom Start: 2009-10-25 09:30 Stop: 2009-10-27 11:00

IP Address: 1.1.1.1 From Either
IP Address:

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKS HTTP Activity Search

Another common query is analysts who want to see all traffic from a given IP address (or IP addresses) to a specific website.

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKS HTTP Activity Search

  • For example let’s say we want to see all traffic from IP Address 1.2.3.4 to the website www.website.com
  • While we can just put the IP address and the “host” into the search form, remember what we saw before about the various host names for a given website

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Host Field

It’s important to note, that in many cases users think they’re at websites like www.yahoo.com, but behind the scenes data is coming from a number of different servers without the user knowing it:

GET /mc/modules/im/abContacts?mcnumb=RIIDb659jm & jsrand=98037807 & rand=2127033459 HTTP/1.0
Accept: */*
Accept-Language: fa
Referer: http://us.mc575.mail.yahoo.com/mc/showFolder?.ylc=X3oDMTBucmNobGR0BFSTA2M5ODMwMTAyNwRhYWNkZxNx2dz?mid=1_21857_AERkEkAANrjSi6wUQ7fiLz4fY&fid=Inbox&sort=date&order=up&startMid=36&filterBy=
x-requested-with: XMLHttpRequest
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)
Host: us.mc575.mail.yahoo.com
Cookie: MG d=IvAXlFvaYnFGmmIfzw3zBCVVRe2jUKZLwwyoKSrjxxG0XVYaJhF95dLsZ5C0x1eDlcTcaHS_vpi
ad9XvB0emj5Rr1
v=l
Y
v=l
n=66k3gh6ns55lf
l=cc70cc03_01sqqx/o (Yahoo login id: [REDACTED])
p=m2g265i013000000 (Gender: male, Birth year: 1980, Postal code: [REDACTED])
r=hq
lg=en-US (Language/content: English)

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

XKS HTTP Activity Search

  • In order to account for all of the possible host names, we must front-wildcard the host name.
  • Be careful when front-wildcarding because beyond being resource intensive for XKS, it can be dangerous from a USSID18 perspective

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Hints for wildcarding a host name

  • If you’re trying to query for traffic to the website www.website.com the best way to wildcard it is:
  • *.website.com
  • Notice that the . before the hostname website is still there, that way we will properly hit on ads.website.com images.website.com but avoid the false hits on www.anotherwebsite.com

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Hints for wildcarding a host name

IP Address: 1.2.3.4 From

Host: *.website.com

Why are we only interested in traffic coming from our IP of interest going to our website of interest?

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Helpful GUI Shortcuts

  • Earlier we talked about how XKS broke a GET request into the URL Path and URL Argument (separated by a ?)
  • Ex: http://forum.[REDACTED]/showthread.php?t=131485
  • Get’s broken out to:
Host URL Path URL Args
forum.[REDACTED] /showthread.php t=131485

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Helpful GUI Shortcuts

  • So if we were to query for this URL we would need to enter those fields in separately:
Host URL Path URL Args
forum.[REDACTED] /showthread.php t=131485
Host: forum[REDACTED]
URL Path: /showthread.php
URL Args: t=131485

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Helpful GUI Shortcuts

  • Or we could use the “URL Field Builder” to simply copy and paste the full URL and let XKS break it into its appropriate parts:
Host: [Populate with URL Field Builder]
URL Path:
URL Args:

URL Field Builder
Enter a URL that will be automatically parsed to populate the host, path, and argument fields:
Enter Cancel

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL


TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL

Helpful GUI Shortcuts

URL Field Builder
Enter a URL that will be automatically parsed to populate the host, path, and argument fields:
http://forum[REDACTED]/showthread.php?t=131485
Enter Cancel

Host: forum[REDACTED]
URL Path: /showthread.php
URL Args: t=131485

TOP SECRET//COMINT//REL TO USA, AUS, CAN, GBR, NZL