Forwarding Selected MAILORDER Files Read Me
Open PDFDocument date: -
Pages: 2
Genre: manual
Banner: UNCLASSIFED//FOR OFFICIAL USE ONLY
Classification: UNCLASSIFIED
Dissemination: FOUO
An UNCLASSIFIED//FOR OFFICIAL USE ONLY XKEYSCORE administrator guide explaining how to configure an XKS cluster to forward selected intercepted sessions via MAILORDER, including category throttling limits, output directory configuration and the process setup required to switch all Digital Network Intelligence filtering and selection to the XKEYSCORE collection system.
MAILORDER, XKEYSCORE
People mentioned:
-
Countries:
-
Tags:
session forwarding, category throttling, DNI, Digital Network Intelligence, XKS, master server, slave nodes, collection infrastructure, surveillance infrastructure, quota management, bulk collection, sessionization, administrator guide, site configuration, rsync, process management
Identifier:
selection-forwarding-readme
PDF SHA-256:
4c3cc521bc519789e3ea6a0b0f7ac70e722e2753563bff94bd5897685faf16d5
My notes
UNCLASSIFED//FOR OFFICIAL USE ONLY
(U//FOUO) Forwarding Selected MAILORDER Files Read Me
(U//FOUO) This document explains how to configure an XKS cluster to forward selected sessions via MAILORDER. Sessions are still scanned, tagged, and queryable in XKEYSCORE, even if the quota limit (category throttling) has been exceeded for the day. Use this document to learn the procedures for switching all Digital Network Intelligence (DNI) filtering and selection to the XKEYSCORE collection system.
(U//FOUO) Configuring DNI Selection in xks.config
(U//FOUO) Follow these steps to configure xks.config to use the XKS collection system for DNI filtering and selection:
- (U//FOUO) Logon as the user
oper. - (U//FOUO) At the command line from within any directory, type
viconfigand then press Enter. Thexks.configfile will open. - (U//FOUO) In the
inputssection ofxks.configset the following configurations:
a.forward sessions = yes: This enables session forwarding on the selected output plugin. You can verify that session forwarding is set toyesin thedefault.xmlfile located in$XSCORE_DIR/config/plugins
Note: Only after running thexks setup pluginsprocess, described on page 2, will you see theselected_outputplugin enabled indefault.xml.
b.category_throttle = true(default). This sets the maximum number of sessions per category to 10,000 and the total size forwarded for any given category to be no greater than 1GB. If either of these limits is reached, XKEYSCORE will no longer forward sessions that hit on that particular category for the rest of the day. It will, however, continue to process data for other categories. These limits are reset every 24 hours.
c. All MAILORDER files will be stored on the Master server in:
/export/data/xkeyscore/outputs/mailorder
1 |
UNCLASSIFED//FOR OFFICIAL USE ONLY
UNCLASSIFIED//FOR OFFICIAL USE ONLY
You can check the status of the category throttle using one of the following commands:
mysqls quota status current
Thequota_limitsforwarding status for all categories.mysqls quota status yesterday
Thequota_limitsforwarding status for the previous day.mysqls quota_throttled current
The top 20 categories for the current day (The phrase "negative remaining" indicates the category was throttled).
d.output_location = master: All MAILORDER files will be generated in:
/xks_data/<masterHostname>/outputs/mailorder_working
Then the mailorder_proc process on the Master will rename and move them to:
/xks_data/<masterHostname>/outputs/mailorder
e. mailorder = yes (default) : On the Master, this creates the mailorder_proc process that polls the mailorder_working directory and then moves/renames any MAILORDER files to $XSCORE_DATA_DIR/outputs/mailorder.
4. (U//FOUO) Type: :wq! and then press Enter to save and exit xks.config.
(U//FOUO) Additional Processes
(U//FOUO) In addition to editing the configurations in xks.config, it is important to run several set-up processes.
- (U//FOUO) As the
operuser, at the command prompt, typexks setup pluginsand press Enter. This ensures any applicable changes to plugin configurations will take effect. - (U//FOUO) As the
operuser, at the command prompt, typexks setup processesand press Enter. This will create themailorder_procprocess on the Master. - (U//FOUO) At the command prompt, type
rsync push_configand press Enter. This pushes the latest configuration changes to the slaves. - (U//FOUO) As the
operuser, at the command prompt, typexks proc startand press Enter. This will ensure all of the running processes pick up any configuration changes. - (U//FOUO) At the command prompt, type
xks proc saferestartand press Enter. After setup is complete, this restartsprocess_datasfor the new configurations to take effect.
2 |
UNCLASSIFIED//FOR OFFICIAL USE ONLY
Published in the Same Report
(10 of 46 documents)
-
XKEYSCORE intro [pdf]
A TOP SECRET NSA presentation from 2008 introducing XKeyscore — one of the earliest overview documents in the archive — describing the system as a globally distributed Linux cluster across approximately 150 sites and 700+ servers that stores a ~3-day rolling buffer of all unfiltered internet traffic, and demonstrating its core capabilities for finding previously unknown targets by mining metadata, tracking documents by embedded image hashes, detecting encryption and VPN use, and language tracking.
Programs: XKEYSCORE · TURMOIL · TURBULENCE · TRAFFICTHIEF · WEALTHYCLUSTER · MOONSHINE · SEARCHLIGHT · MARINA · PSC · SSO · TIMBERLINEPeople: bin Laden, Osama“'Brasil é um grande alvo', diz jornalista sobre vigilância dos EUA” - O Globo ↗ (2013-07-07) · “XKeyscore: NSA tool collects 'nearly everything a user does on the internet'” - The Guardian ↗ (2013-07-31) · “No Place to Hide” - Metropolitan Books ↗ (2014-05-13) · “XKEYSCORE: NSA’s Google for the World’s Private Communications” - The Intercept ↗ (2015-07-01) · Greenwald, Glenn · Bridi, Sônia · Marquis-Boire, Morgan · Lee, Micah -
Advanced HTTP Activity Analysis [pdf]
This NSA training document from 2009 teaches intelligence analysts how to analyze HTTP internet traffic and exploit it using the XKEYSCORE surveillance system to monitor and identify internet users globally.
“XKEYSCORE: NSA’s Google for the World’s Private Communications” - The Intercept ↗ (2015-07-01) · “No Place to Hide” - Metropolitan Books ↗ (2014-05-13) · Marquis-Boire, Morgan · Greenwald, Glenn · Lee, Micah -
Analyzing Mobile/Cellular DNI in XKEYSCORE, May 2009 [pdf]
The XKEYSCORE document explains how the agency analyzes mobile/cellular device internet usage by collecting and processing both converged DNI/DNR data within cellular networks and traditional internet traffic, using various tools and indicators to identify mobile users even when their device type isn't immediately evident. The document demonstrates how the system can track mobile internet activity across applications like Yahoo Mail and identifies devices through various technical "hints" including MSISDN codes, browser fingerprints, and user agent strings.
-
Atomic SIGINT Data Format (ASDF) Configuration Read Me [pdf]
This document is a technical configuration guide for setting up XKEYSCORE surveillance systems to generate and forward metadata in the Atomic SIGINT Data Format (ASDF) to various data repositories including MARINA, which tracks users' browser activity and online behavior.
Marking: SECRET -
CADENCE Read Me [pdf]
This NSA engineering manual details how to configure XKEYSCORE—the agency's sprawling surveillance system—to integrate with CADENCE, an automated keyword scanning tool that filters intercepted communications and routes intelligence based on search terms submitted by analysts.
-
How to configure category throttling in XKEYSCORE [pdf]
An NSA technical guide explains how to configure "category throttling" in XKEYSCORE—a feature that automatically stops forwarding intercepted communications once analyst-defined collection targets hit predetermined data volume or session count limits, allowing the surveillance system to continue processing but prevent overflow from high-volume targets.
-
CNE Analysis in XKEYSCORE [pdf]
This document is an NSA training presentation from October 2009 demonstrating how XKEYSCORE integrates with Computer Network Exploitation (CNE) operations, allowing analysts to search and analyze data extracted from compromised computers including keystrokes, browser history, registry files, documents, emails, and directory structures collected through implants like UNITEDRAKE and OLYMPUS.
Programs: XKEYSCORE · OLYMPUS · UNITEDRAKE · PINWALE · TUNINGFORK · TURBOCHASER · MARINA · JOLLYROGER · TREASUREMAP · TURBINE · PRESSUREWAVE · MAD · TUCKER · EXPANDINGPULLY · UNIX · SEARCHLIGHT · KEYLOGGER · PILBEAM · FOXACID · WAYTIDE · SHADOWQUEST · SEAGULLFARO · DARKHELMET · LUTEUSICARUS · FOGGYBOTTOM · GREENCHAOS · RADIUS · NKB · VALIDATOR -
Communications System - How To Guide [pdf]
A TOP SECRET and sensitive compartmented information (SCI) how-to document for the NSA’s XKEYSCORE “Comms” backbone, laying out ports, peer/allow rules, bandwidth pools, and proxy classifications that stitch clusters to central control—a scalable architecture for moving surveillance data with security markings baked in. It also walks through firewall exceptions, HTTPS port-sharing, and a MAILORDER file pipeline, exposing the nuts‑and‑bolts of how collection flows across sites, proxies, and viewers.
-
DEEPDIVE Readme [pdf]
This document is a confidential technical guide that provides step-by-step procedures for configuring an XKEYSCORE server to function as a DEEPDIVE server. DEEPDIVE enhances XKEYSCORE by adding a front-end filter and a rolling buffer of all unfiltered data, creating a federated query system that can scan all collection sites with a single query.
-
DNI101 [pdf]
This TOP SECRET training guide, "DNI 101," provides a workflow for Digital Network Intelligence analysts on how to investigate a target's email address, detailing the specific tools, databases, and methodology used, and stressing the critical difference between initial open-source/classified research and the active "targeting" of raw SIGINT repositories like XKEYSCORE, PINWALE, and MARINA.
Programs: NKB · FOXTRAIL · FOREMAN · PINWALE · LIONHEART · XKEYSCORE · DISHFIRE · MARINA · MECWEB · MAINWAY · TRAFFICTHIEF · MAUI · ANCHORY · M3 · INTELLINK · HOMEBASE · PROTON · UTT · CADENCE · SPEEDBUMP · TUNINGFORK · MASTERSHAKE · BLACKPEARL · JACKSPARROW · ROADBED · STORYDUTY · LIONROAR · DOPHONE · UIS
Related Documents
-
CADENCE Read Me [pdf]
This NSA engineering manual details how to configure XKEYSCORE—the agency's sprawling surveillance system—to integrate with CADENCE, an automated keyword scanning tool that filters intercepted communications and routes intelligence based on search terms submitted by analysts.
-
VoIP Configuration and Forwarding Read Me [pdf]
A SECRET XKeyscore configuration manual describing how to set up the system to intercept internet phone calls, faxes and video calls and forward them to NUCLEON — NSA's voice intercept repository which at time of writing supported over 8,000 users worldwide across 75 servers receiving 700,000 voice, fax, video and tag files per day.
-
Statistics Configuration Read Me [pdf]
XKEYSCORE administrator readme describing the configuration of statistics collection across the XKEYSCORE distributed infrastructure, including how collection nodes report operational metrics back to a central statistics server via MAILORDER, and defining the hierarchical terminology of the XKEYSCORE deployment architecture.
-
Atomic SIGINT Data Format (ASDF) Configuration Read Me [pdf]
This document is a technical configuration guide for setting up XKEYSCORE surveillance systems to generate and forward metadata in the Atomic SIGINT Data Format (ASDF) to various data repositories including MARINA, which tracks users' browser activity and online behavior.
Marking: SECRET -
How to configure category throttling in XKEYSCORE [pdf]
An NSA technical guide explains how to configure "category throttling" in XKEYSCORE—a feature that automatically stops forwarding intercepted communications once analyst-defined collection targets hit predetermined data volume or session count limits, allowing the surveillance system to continue processing but prevent overflow from high-volume targets.