XKeyscore rules

Open PDF
Creating agency: NSA National Security Agency (US)
Document date: -
Pages: 4
Genre: other
Classification: -
Dissemination: -

XKeyscore fingerprint and plug-in source code targeting Tor and other anonymity tools — the rules NSA used to flag anyone connecting to Tor directory servers, requesting Tor bridges, visiting the Tor Project or Tails websites, or researching the Tails operating system — including code that specifically excludes Five Eyes users while fingerprinting everyone else, and a note describing Tails as "a comsec mechanism advocated by extremists on extremist forums."

Long Description:
This is a collection of XKeyscore fingerprint, variable and mapreduce plug-in definitions written in the GENESIS rule language, targeting Tor and other anonymity/privacy tools. The code was published by German public broadcaster ARD (Tagesschau/Panorama, NDR/WDR) in July 2014 and is notable for being sourced separately from the main Snowden document tranche. It caused substantial public controversy because it demonstrated that NSA specifically fingerprinted users of privacy-protecting software, effectively treating the use of anonymity tools as a targeting signal. The code contains several distinct components. The Tor directory server tracking defines two IP variables: $tor_foreign_directory_ip lists five non-Five Eyes Tor directory authority servers (in Germany, the Netherlands, Sweden and Austria) and $tor_fvey_directory_ip lists three Tor directory servers hosted in Five Eyes countries (including 128.31.0.39, the MIT-hosted "moria1" authority). The stated goal is to find potential Tor clients connecting to these directory servers on ports 80 and 443. A pointed code comment instructs analysts not to add non-Five Eyes IPs to the FVEY variable, keeping the two categories rigidly separated. The Tor bridge tracking is among the most significant elements. Tor bridges are unlisted entry points designed specifically to help users in censored countries reach the Tor network without detection. The anonymizer/tor/bridge/tls fingerprint identifies clients accessing bridges.torproject.org via SSL, and the anonymizer/tor/bridge/email fingerprint parses the confirmation emails sent by bridges@torproject.org, extracting the bridge IP addresses and ports directly from the email body using a C++ regex extractor and databasing them into tor_bridges and tor_routers schemas. This means NSA was harvesting the secret bridge addresses that Tor distributes to users precisely to evade surveillance — defeating a core censorship-circumvention mechanism. The website-visit fingerprints reveal the targeting of privacy-interested users. anonymizer/tor/torpoject_visit fingerprints anyone visiting www.torproject.org from a non-Five Eyes country — explicitly excluding US, GB, CA, AU and NZ visitors via the X-Forwarded-For country code. The Tails-related definitions target The Amnesic Incognito Live System, a privacy-focused live operating system. The $TAILS_terms and $TAILS_websites variables and the ct_mo/TAILS fingerprint flag users searching for Tails, viewing documents about it, or visiting tails.boum.org or a Linux Journal article about it. The accompanying comment describes Tails as "a comsec mechanism advocated by extremists on extremist forums" — the characterisation that drew particular criticism, since Tails is used by journalists, activists and privacy-conscious individuals worldwide, not only extremists. The Linux Journal reference led to reports that readers of that publication may have been flagged. The Tor hidden service harvesting is implemented as a mapreduce plug-in (anonymizer/tor/plugin/onion) that scans raw traffic for .onion addresses using a regex matching the 16-character base32 hidden service address format, extracting the address, scheme and port and aggregating them into a tor_onion_survey database with occurrence counts — building a survey of hidden services observed across collection. The final definition fingerprints Mixminion, an anonymous remailer system, by its host name and the IP 128.31.0.34 (also MIT-hosted). The code demonstrates that merely visiting the Tor Project website, requesting a Tor bridge, or researching the Tails operating system was sufficient to be fingerprinted and databased by XKeyscore — with only the Tor Project website rule excluding the Five Eyes countries. The harvesting of Tor bridge addresses from confirmation emails and the survey of .onion hidden services showed active efforts to map and undermine the Tor network's censorship-resistance and anonymity guarantees. The "extremists on extremist forums" framing of Tails became emblematic of criticism that NSA conflated privacy-seeking behaviour with threat.
Programs:
XKEYSCORE, MAPREDUCE

People mentioned:
-

Countries:
-

Tags:
Tor, Tails, anonymizers, fingerprints, privacy tools, hidden services, onion address, MapReduce, Mixminion, remailer, COMSEC, censorship circumvention, extremist forums, Linux Journal, moria1, MIT, deep packet inspection, ARD, Tagesschau, Panorama, NDR, WDR, Das Erste

Identifier:
xkeyscorerules100.txt

PDF SHA-256:
04bdd75ef188e17e8836d04468fa97f0a4af5234d9abbc7c8715999e137e91db
// START_DEFINITION
/**
 * Fingerprint Tor authoritative directories enacting the directory protocol.
 */
fingerprint('anonymizer/tor/node/authority') = $tor_authority
  and ($tor_directory or preappid(/anonymizer\/tor\/directory/));
// END_DEFINITION

// START_DEFINITION
/*
Global Variable for Tor foreign directory servers. Searching for potential Tor
clients connecting to the Tor foreign directory servers on ports 80 and 443.
*/

$tor_foreign_directory_ip = ip('193.23.244.244' or '194.109.206.212' or
'86.59.21.38' or '213.115.239.118' or '212.112.245.170') and port ('80' or
'443');
// END_DEFINITION

// START_DEFINITION
/*
this variable contains the 3 Tor directory servers hosted in FVEY countries.
Please do not update this variable with non-FVEY IPs. These are held in a
separate variable called $tor_foreign_directory_ip. Goal is to find potential
Tor clients connecting to the Tor directory servers.
*/
$tor_fvey_directory_ip = ip('128.31.0.39' or '216.224.124.114' or
'208.83.223.34') and port ('80' or '443');
// END_DEFINITION


// START_DEFINITION
requires grammar version 5
/**
 * Identify clients accessing Tor bridge information.
 */
fingerprint('anonymizer/tor/bridge/tls') =
ssl_x509_subject('bridges.torproject.org') or
ssl_dns_name('bridges.torproject.org');

/**
 * Database Tor bridge information extracted from confirmation emails.
 */
fingerprint('anonymizer/tor/bridge/email') =
email_address('bridges@torproject.org')
  and email_body('https://bridges.torproject.org/' : c++
  extractors: {{
    bridges[] = /bridge\s([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}):?([0-9]{2,4}?[^0-9])/;
  }}
  init: {{
    xks::undefine_name("anonymizer/tor/torbridges/emailconfirmation");
  }}
  main: {{
    static const std::string SCHEMA_OLD = "tor_bridges";
    static const std::string SCHEMA_NEW = "tor_routers";
    static const std::string FLAGS = "Bridge";
    if (bridges) {
      for (size_t i=0; i < bridges.size(); ++i) {
        std::string address = bridges[i][0] + ":" + bridges[i][1];
        DB[SCHEMA_OLD]["tor_bridge"] = address;
        DB.apply();
        DB[SCHEMA_NEW]["tor_ip"] = bridges[i][0];
        DB[SCHEMA_NEW]["tor_port_or"] = bridges[i][1];
        DB[SCHEMA_NEW]["tor_flags"] = FLAGS;
        DB.apply();
      }
      xks::fire_fingerprint("anonymizer/tor/directory/bridge");
    }
    return true;
  }});
// END_DEFINITION


// START_DEFINITION
/*
The fingerprint identifies sessions visiting the Tor Project website from
non-fvey countries.
*/
fingerprint('anonymizer/tor/torpoject_visit')=http_host('www.torproject.org')
and not(xff_cc('US' OR 'GB' OR 'CA' OR 'AU' OR 'NZ'));
// END_DEFINITION


// START_DEFINITION
/*
These variables define terms and websites relating to the TAILs (The Amnesic
Incognito Live System) software program, a comsec mechanism advocated by
extremists on extremist forums.
*/

$TAILS_terms=word('tails' or 'Amnesiac Incognito Live System') and word('linux'
or ' USB ' or ' CD ' or 'secure desktop' or ' IRC ' or 'truecrypt' or ' tor ');
$TAILS_websites=('tails.boum.org/') or ('linuxjournal.com/content/linux*');
// END_DEFINITION

// START_DEFINITION
/*
This fingerprint identifies users searching for the TAILs (The Amnesic
Incognito Live System) software program, viewing documents relating to TAILs,
or viewing websites that detail TAILs.
*/
fingerprint('ct_mo/TAILS')=
fingerprint('documents/comsec/tails_doc') or web_search($TAILS_terms) or
url($TAILS_websites) or html_title($TAILS_websites);
// END_DEFINITION


// START_DEFINITION
requires grammar version 5
/**
 * Aggregate Tor hidden service addresses seen in raw traffic.
 */
mapreduce::plugin('anonymizer/tor/plugin/onion') =
  immediate_keyword(/(?:([a-z]+):\/\/){0,1}([a-z2-7]{16})\.onion(?::(\d+)){0,1}/c : c++
    includes: {{
      #include <boost/lexical_cast.hpp>
    }}
    proto: {{
      message onion_t {
        required string address = 1;
        optional string scheme = 2;
        optional string port = 3;
      }
    }}
    mapper<onion_t>: {{
      static const std::string prefix = "anonymizer/tor/hiddenservice/address/";

      onion_t onion;
      size_t matches = cur_args()->matches.size();
      for (size_t pos=0; pos < matches; ++pos) {
        const std::string &value = match(pos);
        if (value.size() == 16)
          onion.set_address(value);
        else if(!onion.has_scheme())
          onion.set_scheme(value);
        else
          onion.set_port(value);
      }

      if (!onion.has_address())
        return false;

      MAPPER.map(onion.address(), onion);
      xks::fire_fingerprint(prefix + onion.address());
      return true;
    }}
    reducer<onion_t>: {{
      for (values_t::const_iterator iter = VALUES.begin();
          iter != VALUES.end();
          ++iter) {
        DB["tor_onion_survey"]["onion_address"] = iter->address() + ".onion";
        if (iter->has_scheme())
          DB["tor_onion_survey"]["onion_scheme"] = iter->scheme();
        if (iter->has_port())
          DB["tor_onion_survey"]["onion_port"] = iter->port();
        DB["tor_onion_survey"]["onion_count"] = boost::lexical_cast<std::string>(TOTAL_VALUE_COUNT);
        DB.apply();
        DB.clear();
      }
      return true;
    }});

/**
 * Placeholder fingerprint for Tor hidden service addresses.
 * Real fingerpritns will be fired by the plugins
 *   'anonymizer/tor/plugin/onion/*'
 */
fingerprint('anonymizer/tor/hiddenservice/address') = nil;
// END_DEFINITION


// START_DEFINITION
appid('anonymizer/mailer/mixminion', 3.0, viewer=$ascii_viewer) =
        http_host('mixminion') or
        ip('128.31.0.34');
// END_DEFINITION