User-agents: Why and How and How to

Open PDF
Creating agency: NSA National Security Agency (US)
Document date: 2010-07-01
Pages: 42
Genre: slides
Banner:
Classification: TOP SECRET COMINT
Dissemination: REL TO USA, FVEY

A TOP SECRET NSA training presentation from 2010 teaching analysts how to use browser user-agent strings — the identifying information your web browser automatically sends to every website you visit — to track surveillance targets across the internet, link their known email activity to their anonymous web browsing, and identify opportunities for hacking their computers based on their operating system and software versions.

Long Description:
This is a TOP SECRET//COMINT//REL TO USA, FVEY NSA training presentation from S2I61 dated July 2010. The presentation teaches analysts how to exploit HTTP user-agent strings for surveillance purposes. User-agent strings are automatically transmitted by web browsers to every website visited, containing the browser type and version, operating system, security settings, installed software frameworks and language preferences. The presentation explains that user-agents can link a target's "selected" activity (communications already being collected under tasking) to their "unselected" web browsing (untasked internet activity captured in bulk), create tentative links between targets sharing the same user-agent configuration, and identify CNE (hacking) opportunities by revealing a target's operating system version and software configuration. The presentation includes a lighthearted history of browser user-agent spoofing dating back to Internet Explorer pretending to be Mozilla/Netscape to receive frames-enabled web content. The practical section demonstrates querying user-agents in XKeyscore and PINWALE. The key analytical judgment taught is assessing user-agent reliability — when an IP address shows only two user-agent strings with low traffic volume, the correlation between user-agent and target is "probably reliable." When the same IP shows many different user-agent variations (the example shows 17 distinct strings including variations in .NET CLR versions, Google Toolbar versions and InfoPath installations), reliability "plummets" and cookie/TDI correlation becomes necessary. The presentation warns against assuming complex user-agents are unique — a standard Chrome 6 on Windows 7 string applies to every user with that configuration. Mobile user-agents are highlighted as particularly valuable because they typically include the phone model, creating "IMEI correlation opportunities" for linking web activity to specific physical handsets. The presentation concludes with the operational caveat that user-agents can be manipulated or removed entirely — via Firefox plugins, Opera browser settings or the TOR Button — allowing targets to present a different user-agent for each browsing session. The final slides advertise S2I61's broader services including help with XKeyscore fingerprint development, target technology analysis and collateral description development, positioning the unit as an internal SIGDEV technical support team.
Programs:
XKEYSCORE, PINWALE, AGILITY

People mentioned:
-

Countries:
-

Tags:
User-Agent, HTTP, browser fingerprinting, TDI, cookies, Computer Network Exploitation, operating system, Windows, Firefox, Chrome, Safari, Opera, Internet Explorer, mobile user-agent, iPhone, Android, BlackBerry, IMEI, Gecko, AppleWebKit, Presto, encryption, USSID-18, compliance, web browsing, SIGDEV, target tracking

Identifier:
user-agents

PDF SHA-256:
0177d72ec9aa3f502ed8c042c1c778fd558894c8757f18610bf84ca07a63790c

TOP SECRET//COMINT//REL TO USA, FVEY

(S//SI//REL) User-agents: Why and How and How to

NATIONAL SECURITY AGENCY
UNITED STATES OF AMERICA

The overall classification of this briefing to TOP SECRET//COMINT//REL TO USA, FVEY

[REDACTED] – S2I61
July 2010

Derived From
NSA/CSSM 1-52
Dated 20070108
Declassify on: 20320108

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Agenda

  • WHY: Learn why we care about user agents (UAs)
  • HOW: Learn how to read a user-agent
  • HOW TO: (get it?) Learn how to use user-agents in our tools

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

WHY?

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

What is a User-Agent?

A user-agent is a string which lets websites
know your:

  • type of web-browser or application
  • Operating System
  • Security settings or permissions
  • Versions of relevant programs (media, java, etc.)
  • Etc. (Language settings, ad-ware)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Why would I want to give a website that?

  • Compatibility
  • Specific Website Features
  • Security permissions

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

User-Agents Can

  • Link a target's “selected” activity to their unselected web-browsing
  • Create a tentative link between targets that have the same user-agent
  • Identify CNE opportunities

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

BUT...

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

User-Agents Also

  • Can vary from very unique to extremely common
  • Change with software updates
  • Only identify the web-browser
    • 2 web-browsers = 2 user-agents
  • Can't be trusted...

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Who started this nonsense?

A History Lesson

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

The Great Browser Wars

Back...in the 20th Century!

Before Now but After What is Below

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Remember Frames?

  • Netscape's new fancy web-browser support them!
  • The original “web-browser” Mosaic did not
  • And so began browser “sniffing”

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Worthless Trivia!

  • Mosaic was the first web-browser to embed images with text
  • It supported FTP, Usenet, and Gopher!
  • Its web-browser competitors at the time were Erwise and ViolaWWW

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

The Great Internet Explorer Hoax

Ever wonder why so many user agents start with “Mozilla” but aren't Firefox?

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Internet Explorer fools us all

  • When Internet Explorer was released it did frames too!
  • But since its user-agent didn't say so, no websites would send their super cool frames version to the IE users

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

What to do?

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

LIE, of course!

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Internet Explorer starts to spoof

  • Internet Explorer changed its user agent starting their user agent with Mozilla/1.22
  • Mozilla/1.22 (compatible; MSIE 2.0; Windows 95)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

And it continues to this very day...

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

The How...

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Three Basic Pieces of the UA

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Mozilla/1.22 (compatible; MSIE 2.0; Windows 95)
Mozilla/1.22

Part 1: The Netscape Historical Token

  • Appears in primarily Mozilla Firefox, Google Chrome, and MSIE browsers
    • Modern Version: Mozilla 4.0 or Mozilla 5.0
    • Does not indicate a target uses “Mozilla Firefox”

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Mozilla/1.22 (compatible; MSIE 2.0; Windows 95)
MSIE 2.0

Part 2: The Web Browser Identifier

  • Appears in generally all user-agents
  • Not always in the same place, but usually self explanatory
    • Opera X.X = Opera
    • Firefox X.X = Firefox
    • Safari X.X= Safari
    • Chrome X.X Safari X.X = Google Chrome

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Mozilla/1.22 (compatible; MSIE 2.0; Windows 95)
Windows 95

Part 3: The Operating System Token

  • Appears in basically all HTTP user-agent strings
  • Examples:
    • Windows NT 6.1 = Windows Version 7
    • Windows NT 6.0 = Windows Vista
    • Windows NT 5.2 = Windows XP 64bit
    • Windows NT 5.1 = Windows XP
    • Windows NT 4.0 actually equals Windows NT 4.0

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Other operating systems UAs

  • Mobile
    • MAC OS X
  • Linux
    • Linux i686
    • Free BSD

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Game Consoles

  • Opera/9.30 (Nintendo Wii; U; 2047-7; en)
  • Mozilla/5.0 (Playstation 3; 2.00)
  • PSP (PlayStation Portable); 2.00

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Mobile User Agents

  • Usually self-explanatory
    • Iphone
    • Ipad
    • Blackberry
    • Android
  • Mobile user agents also usually give you the phone model (Read: IMEI correlation opportunities)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Ever wonder what that was?

  • Gecko: a rendering engine used by Firefox and others
  • AppleWebkit: Apple's version of KHTML rendering engine used in Safari and Chrome most commonly
  • Presto : the “core” of the Opera platform suite

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Ever Wonder Contd.

  • .NET CLR is the .NET Framework version
  • SV1 is an artifact created by MSIE 6.0 to make its security better
  • Win64 can indicate that the system is running a 64 bit processor

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Ever Wonder... One more

Many web browsers will also have an “encryption strength” marker

U = USA (128 bit encryption)
I = International (40 bit encryption)
N = No encryption (Woo!)

Most Browsers nowadays come with a U
Since the USG no longer requires encryption changes for international usage.

TOP SECRET//COMINT//REL TO USA, FVEY

--

TOP SECRET//COMINT//REL TO USA, FVEY

Your target's user-agents can shorten and lengthen!

  • Each website may require different information
  • Longer user-agents may have various rendering engine, java versions, and language settings
  • If you see a shortened or longer version of a UA close to your targets logins. Check it out! Carefully...

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

The How To...

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Querying in Xkeyscore

Navigation Menu
*   Classic A-M
*   Alert
*   BlackBerry
*   Call Logs
*   Category DNI
*   Cellular DNI
*   Cisco Passwords
*   Clarent
*   Crypt Scanner
*   DNS
*   Document Metadata
*   Document Tagging
*   Email Addresses
*   Extracted Files
*   Full Log DNI
*   Gen Info
*   HTTP Activity
*   IKE Parser
Browser: Mozilla 5.0*

Remember:
Since Xkeyscore no longer supports leading wildcards you need to be specific with your User-Agent

A User-agent alone is not a strong query
Time Frame, Active IP,
Country, etc. all will help make
your query compliant

Browser
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.375.99 Safari/533.4

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Uniqueness – When to use a User-Agent

Reports
VIEW
FILTERS

Browser
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)

Geo Info
Sort Ascending
Sort Descending
Filters
Color By
Group By
Histogram
Pivot Data
Histogram Grid
Show/Hide...
AutoFit Column Width

Results of an IP-based search for a target

Histogram Grid
Page 1 of 1
Clear Selection Export

Filter Browser
Count
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
6
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)
6

Low Number of users and traffic volume as well as only 2 User-agents
Verdict: Probably reliable

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

More User = More Problems

Browser Count
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) 6
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; 1
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; 1
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB0.0; .NET CLR 2.0.50727; AskTB5.6) 1
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6.5) 1
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6.5; .NET CLR 2.0.50727) 1
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6.5; InfoPath.2) 2
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; GTB6.5; InfoPath.2; .NET CLR 2.0.50727; .NET 1
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1; .NET CLR 2.0.50727; .NET CLR 3.0. 1
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322) 1
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) 6
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322) 1
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; FunWebProducts; AskTbPTV/5.8.0.12304) 1
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6; .NET CLR 1.1.4322) 1
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; IEMB3; IEMB3) 1
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727 1
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; InfoPath.2; .NET CLF 2

Reliability of correlating your target's browser to his web-activity plummets due to User-agent variations and the threat of multiple users.

At this point, cookie and TDI correlation are necessary. User-agents can not be used by themselves.

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Common Sense Helps

Never assume because a User-agent is complicated that it is unique

  • Example:
    Mozilla 5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.3(KHTML, like Gecko) Chrome/6.0.464.0 Safari/534.3

This is the standard user agent for EVERYONE with an updated Chrome browser using Windows 7.

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Xkeyscore Storage

Push to Pinwale or Archive Results

Download Session Mode: S
   AGILITY-RealTime
   Agility
   DNI Presenter
   PINWALE
   PINWALE and Decrypt
   Download SOTF Session
   Download Session
   Download D124 Session

   finished
   Archive Results
   finished

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Pinwale

Fields to add to Metadata View

AppProc Active User

User Agent
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) ...

Generally speaking, the User Agent listed belongs to the selector in the Active User column (if populated)

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Add a User-Agent to a compliant query

Fielded Search Form

1.  Put in the User Agent that is being search for into the CONTENT\_META field under the fielded search form.

KEYWORDS and
cs=ISO8859_1 ( ( ("Opera Mini") ) )
CONTENT_META
Opera Mini
between tags UserAgent and

Smart Form/Native Query

Native, cs=ISO8859_1 ( (content_meta=(((("Opera Mini")) between \<UserAgent and \/UserAgent))))
(content_meta=(((("Opera Mini")) between \<UserAgent and \/UserAgent )))

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Not sure the specific User-Agent?

1.  Add the following syntax to your query:
    \TERM \<yourfield here (E.G. \TERM \<useragent )

2.  Apply Native to the field

3.  Apply the content_meta document zone to the field.

Free Form Query Smart Form
cs=ISO8859_1(((["selectorl"]))
selector1.
or
cs=ISO8859_1( ("selector2")))
selector2
or
) and
CONTENT_META:Native, cs=ISO8859_1 [[TERM \<useragent ]]
TERM \<useragent
or or and

(TS//SI//REL) This query basically ensures that a certain field exists in each result thus removing all the content not relevant to your query.

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

User-Agent Manipulation

  • The best for last
    • User-agents can completely be changed or not included by the user!
      • By Firefox Plugin
      • By Browser settings (Opera)
      • Outside programs (TOR Button)
    • These programs allow users to have a different user agent for each session!

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Questions??

  • Contact Info: [REDACTED]@nsa.ic.gov
  • Website on the High-Side:
  • [REDACTED]
  • [REDACTED]
  • [REDACTED]
  • [REDACTED]
  • Lots of great stuff in open source as well!
    http://www-archive.mozilla.org/build/user-agents-strings.html

TOP SECRET//COMINT//REL TO USA, FVEY


TOP SECRET//COMINT//REL TO USA, FVEY

Got a Tech Problem?

Have a random SIGDEV question?

Need help with a target using new tech to communicate?

Need help developing an accurate collateral description of a technology?

Want help developing Xkeyscore fingerprints for a weird target behavior?

Let us know : DL S2I61_all

TOP SECRET//COMINT//REL TO USA, FVEY