Atomic SIGINT Data Format (ASDF) Configuration Read Me
Open PDFDocument date: -
Pages: 2
Genre: manual
Banner: SECRET
Classification: SECRET
Dissemination: -
This document is a technical configuration guide for setting up XKEYSCORE surveillance systems to generate and forward metadata in the Atomic SIGINT Data Format (ASDF) to various data repositories including MARINA, which tracks users' browser activity and online behavior.
My notes
SECRET
(U//FOUO) Atomic SIGINT Data Format (ASDF) Configuration Read Me
(U) Overview
(U//FOUO) The ASDF Configuration Read Me is designed to provide procedures for other organizations to configure their own XKEYSCORE servers tailored to their requirements and mission. More specific, it provides procedures for generating and forwarding ASDF data to FALLOUT, which then is forwarded to the MARINA repository.
(S) FALLOUT is a DNI metadata exploitation and integrity service which converts, validates, normalizes, classifies, and distributes DNI metadata according to algorithms and rules produced by Protocol Exploitation. FALLOUT feeds corporate metadata repositories, such as MARINA, METAWAVE, FASCIA, and MAINWAY, with full-take and selected metadata derived from collection and processing systems such as TURMOIL, WEALTHYCLUSTER 2.0, XKEYSCORE, BLACKPEARL, SCISSORS, and others.
(S) The MARINA metadata application tracks a user's browser experience, gathers contact information/content, and develops summaries of target activity. This tool offers retrieval, data aggregation, viewing, and some manipulation of specific data types collected worldwide.
(U//FOUO) Configuring XKEYSCORE for ASDF can be accomplished in three phases. First, you will address two pre-configuration requirements. Next you will configure xks.config for ASDF forwarding. Last, you will execute a group of set-up processes.
(U) Pre-Configuration Requirements
(U//FOUO) The Atomic SIGINT Data Format (ASDF) requires the following:
- The most recent StarProc update on each XKEYSCORE server that is processing data. Please see the Royale with Cheese Updater How To document to configure your XKEYSCORE to pull the latest StarProc updates.
- MAILORDER configured to pick up KLG MAILORDER files from the Master server in the $XSCORE_DATA_DIR/outputs/mailorder directory.
1 Page
SECRET
SECRET
(U//FOUO) Configuring xks.config to Generate ASDF Files
(U) Configuration for generating and forwarding ASDF can be accomplished in just five steps:
- (U) Logon as the user oper.
- (U//FOUO) At the command line from within any directory, type viconfig and then press Enter. The xks.config file will open.
- (U//FOUO) In the #outputs section of xks.config, set the following configurations:
a.asdf_output = yes: This enables the digester plugin.
b.asdf_trigraph = KLG: This sets the MAILORDER trigraph to correctly write out the asdf MAILORDER files.
c.asdf_priority = 2: This sets the priority in the MAILORDER file name. - (U//FOUO) In the #[processes] section of xks.config, set
mailorder = yesto enable MAILORDER. - (U) Type:
:wq!and then press Enter to save and exit xks.config.
(U) Additional Processes
(U) In addition to editing configurations in xks.config, it is important to run several set-up commands.
- (U//FOUO) At the command prompt, type
xks setup processesand press Enter. Because MAILORDER is enabled in step 4 above (i.e.,mailorder = yes), executing this command will create themailorder_procprocess on the Master server. - (U//FOUO) At the command prompt, type
xks proc startand press Enter. This will ensure all of running processes pick up any configuration changes. - (U//FOUO) At the command prompt, type
xks setup pluginsand press Enter. Because The digester plugin is enabled in step 3a above (i.e.,asdf_ouput = yes), executing this command will setup thedigestplugin on the Master server. - (U//FOUO) At the command prompt, type
xks rsync push_configand press Enter. This pushes the newplugin_configto the slaves, after running the step above. - (U//FOUO) On the Master server, from the command prompt, type
xks proc saferestartto restart theprocess_data_parent's.
2 Page
SECRET
Published in the Same Report
(10 of 46 documents)
-
XKEYSCORE intro [pdf]
A TOP SECRET NSA presentation from 2008 introducing XKeyscore — one of the earliest overview documents in the archive — describing the system as a globally distributed Linux cluster across approximately 150 sites and 700+ servers that stores a ~3-day rolling buffer of all unfiltered internet traffic, and demonstrating its core capabilities for finding previously unknown targets by mining metadata, tracking documents by embedded image hashes, detecting encryption and VPN use, and language tracking.
Programs: XKEYSCORE · TURMOIL · TURBULENCE · TRAFFICTHIEF · WEALTHYCLUSTER · MOONSHINE · SEARCHLIGHT · MARINA · PSC · SSO · TIMBERLINEPeople: bin Laden, Osama“'Brasil é um grande alvo', diz jornalista sobre vigilância dos EUA” - O Globo ↗ (2013-07-07) · “XKeyscore: NSA tool collects 'nearly everything a user does on the internet'” - The Guardian ↗ (2013-07-31) · “No Place to Hide” - Metropolitan Books ↗ (2014-05-13) · “XKEYSCORE: NSA’s Google for the World’s Private Communications” - The Intercept ↗ (2015-07-01) · Greenwald, Glenn · Bridi, Sônia · Marquis-Boire, Morgan · Lee, Micah -
Advanced HTTP Activity Analysis [pdf]
This NSA training document from 2009 teaches intelligence analysts how to analyze HTTP internet traffic and exploit it using the XKEYSCORE surveillance system to monitor and identify internet users globally.
“XKEYSCORE: NSA’s Google for the World’s Private Communications” - The Intercept ↗ (2015-07-01) · “No Place to Hide” - Metropolitan Books ↗ (2014-05-13) · Marquis-Boire, Morgan · Greenwald, Glenn · Lee, Micah -
Analyzing Mobile/Cellular DNI in XKEYSCORE, May 2009 [pdf]
The XKEYSCORE document explains how the agency analyzes mobile/cellular device internet usage by collecting and processing both converged DNI/DNR data within cellular networks and traditional internet traffic, using various tools and indicators to identify mobile users even when their device type isn't immediately evident. The document demonstrates how the system can track mobile internet activity across applications like Yahoo Mail and identifies devices through various technical "hints" including MSISDN codes, browser fingerprints, and user agent strings.
-
CADENCE Read Me [pdf]
This NSA engineering manual details how to configure XKEYSCORE—the agency's sprawling surveillance system—to integrate with CADENCE, an automated keyword scanning tool that filters intercepted communications and routes intelligence based on search terms submitted by analysts.
-
How to configure category throttling in XKEYSCORE [pdf]
An NSA technical guide explains how to configure "category throttling" in XKEYSCORE—a feature that automatically stops forwarding intercepted communications once analyst-defined collection targets hit predetermined data volume or session count limits, allowing the surveillance system to continue processing but prevent overflow from high-volume targets.
-
CNE Analysis in XKEYSCORE [pdf]
This document is an NSA training presentation from October 2009 demonstrating how XKEYSCORE integrates with Computer Network Exploitation (CNE) operations, allowing analysts to search and analyze data extracted from compromised computers including keystrokes, browser history, registry files, documents, emails, and directory structures collected through implants like UNITEDRAKE and OLYMPUS.
Programs: XKEYSCORE · OLYMPUS · UNITEDRAKE · PINWALE · TUNINGFORK · TURBOCHASER · MARINA · JOLLYROGER · TREASUREMAP · TURBINE · PRESSUREWAVE · MAD · TUCKER · EXPANDINGPULLY · UNIX · SEARCHLIGHT · KEYLOGGER · PILBEAM · FOXACID · WAYTIDE · SHADOWQUEST · SEAGULLFARO · DARKHELMET · LUTEUSICARUS · FOGGYBOTTOM · GREENCHAOS · RADIUS · NKB · VALIDATOR -
Communications System - How To Guide [pdf]
A TOP SECRET and sensitive compartmented information (SCI) how-to document for the NSA’s XKEYSCORE “Comms” backbone, laying out ports, peer/allow rules, bandwidth pools, and proxy classifications that stitch clusters to central control—a scalable architecture for moving surveillance data with security markings baked in. It also walks through firewall exceptions, HTTPS port-sharing, and a MAILORDER file pipeline, exposing the nuts‑and‑bolts of how collection flows across sites, proxies, and viewers.
-
DEEPDIVE Readme [pdf]
This document is a confidential technical guide that provides step-by-step procedures for configuring an XKEYSCORE server to function as a DEEPDIVE server. DEEPDIVE enhances XKEYSCORE by adding a front-end filter and a rolling buffer of all unfiltered data, creating a federated query system that can scan all collection sites with a single query.
-
DNI101 [pdf]
This TOP SECRET training guide, "DNI 101," provides a workflow for Digital Network Intelligence analysts on how to investigate a target's email address, detailing the specific tools, databases, and methodology used, and stressing the critical difference between initial open-source/classified research and the active "targeting" of raw SIGINT repositories like XKEYSCORE, PINWALE, and MARINA.
Programs: NKB · FOXTRAIL · FOREMAN · PINWALE · LIONHEART · XKEYSCORE · DISHFIRE · MARINA · MECWEB · MAINWAY · TRAFFICTHIEF · MAUI · ANCHORY · M3 · INTELLINK · HOMEBASE · PROTON · UTT · CADENCE · SPEEDBUMP · TUNINGFORK · MASTERSHAKE · BLACKPEARL · JACKSPARROW · ROADBED · STORYDUTY · LIONROAR · DOPHONE · UIS -
Email Address vs User Activity [pdf]
This 2009 NSA training presentation explains how to use the XKEYSCORE system to search raw SIGINT, detailing the differences between the "Email Address" query, which finds email addresses in full content, and the more flexible "User Activity" query, which can search for selectors like cookies, usernames, and phone numbers.
“XKEYSCORE: NSA’s Google for the World’s Private Communications” - The Intercept ↗ (2015-07-01) · “No Place to Hide” - Metropolitan Books ↗ (2014-05-13) · Marquis-Boire, Morgan · Greenwald, Glenn · Lee, Micah
Related Documents
-
FAIRVIEW dataflow charts April 2012 [pdf]
This document outlines data flow diagrams that show how various types of sensitive communication, including voice, fax, and modem traffic, are handled through systems operated by the NSA and its partners. These partners include corporations, telecommunications companies, ISPs, and other entities involved in processing and managing the data through secure and classified channels. The document covers processing flows for various data types, metadata, and communications systems under NSA's control, detailing how data is routed, stored, and analyzed across different platforms and sites.
Programs: FAIRVIEW · SAGUARO · TURNSTILE · MESA · TITANPOINT · RIMROCK · MAILORDER · PINECONE · BLARNEY · STONEGATE · TINSEL · CONVEYANCE · FISHWAY · NUCLEON · PINWALE · SCISSORS · COURIERSKILL · DRIFTWOOD · ASDF · FALLOUT · MARINA · LOPERS · WEALTHYCLUSTER · PUPR · CLEARSIGHT · MIDFIELD · CADENCE · BUGCATCHER · WAYLAND · DESOTO · GATEKEEP · HIGHDECIBEL · XKEYSCORE · YANKEE · TOPROCK · FASCIA · DISHFIRE · SEAGULL · BIGDIPPER · CORPORATE · LILDIPPER · HAVASU · WEALTHYCLUSTER2 · RAGTIME · TUMULT · TATTOO · DARKTHUNDER · ICYTWINS · TML-STATUS · WATERFRONT · VULCANDEATHGRIP · STEELFLAUTA · UNITY · KEYCARD · UTT · BTUBE · ISLANDTRANSPORT · SSO · MAINWAY · SSDM · TUBE · TITANPOINTE · TURMOIL · VPCS“NSA Spying Relies on AT&T’s ‘Extreme Willingness to Help’” - ProPublica ↗ (2015-08-15) · “AT&T Helped U.S. Spy on Internet on a Vast Scale” - The New York Times ↗ (2015-08-15) · “Titanpointe: The NSA’s Spy Hub in New York, Hidden in Plain Sight” - The Intercept ↗ (2016-11-16) · Angwin, Julia · Larson, Jeff · Savage, Charlie · Moltke, Henrik · Poitras, Laura · Risen, James · Gallagher, Ryan -
SSO dictionary relevant entries [pdf]
This document provides a detailed list of relevant entries related to NSA's Special Source Operations (SSO), highlighting various NSA surveillance and data collection systems used to monitor global communications. It includes descriptions of programs such as BIGBIRD and DISHFIRE, which focus on cable and SMS interception, as well as tools like LOPERS and COURIERSKILL for processing telephony metadata and refining target selection. The entries emphasize the extensive and automated nature of the NSA's intelligence-gathering operations, particularly in relation to undersea cables and telecommunication networks, and underscore the secretive and expansive scope of SSO's global surveillance activities.
Programs: ARTIFICE · BIGBIRD · FAIRVIEW · SSO · POORWILL · PINECONE · BLACKBELT · CADENCE · BLACKNIGHT · SHIPMASTER · COURIERSKILL · THEORYMASTER · WEALTHYCLUSTER · CLIFFSIDE · DISHFIRE · MARINA · MAINWAY · NUCLEON · PINWALE · FASCIA · BANYAN · LAMPSHADE · ASSOCIATION · HOMEBASE · SEDB · FRIAR · KEYCARD · TURMOIL · TURBULENCE · UTT · IRISHBEAUTY · OCTAVE · KOZYKOVE · LITHIUM · BLARNEY · LOPERS · MAILORDER · LUMBERYARD · MERLIN · NODDY · SAGUARO · NUTHATCH · CONTRAOCTAVE · XKEYSCORE · PLANK · RODEOSTAR · SAGURA · SERENADE · STARGATE · SILVERCOLLAM · SEALION · SEAGULL · SLIVER · SORA-2 · TITANPOINTE · TUBE · VPCS · STONEGATE“NSA Spying Relies on AT&T’s ‘Extreme Willingness to Help’” - ProPublica ↗ (2015-08-15) · “AT&T Helped U.S. Spy on Internet on a Vast Scale” - The New York Times ↗ (2015-08-15) · “The Wiretap Rooms: The NSA’s Hidden Spy Hubs in Eight U.S. Cities” - The Intercept ↗ (2018-06-25) · Angwin, Julia · Larson, Jeff · Savage, Charlie · Moltke, Henrik · Poitras, Laura · Risen, James · Gallagher, Ryan -
XKEYSCORE System Administration [pdf]
A December 2012 TOP SECRET XKeyscore System Administration training manual — the detailed technical guide for the engineers who ran XKeyscore field-site clusters — covering the system's Red Hat Linux architecture, master/slave cluster design, database and processing pipeline, the GUI (including built-in Tor, botnet and CNE search menus), the DeepDive full-take packet processor, the xks command-line tool, and monitoring and troubleshooting, with live screenshots from the Timberline site (USF-790) and Menwith Hill Station.
Programs: XKEYSCORE · WEALTHYCLUSTER · MAILORDER · CADENCE · ROGUESTITCH · QUANTUMBOT · COTTONCHAINSAW · MPTRII · SCARFACE · ZAMENSIS · MAPREDUCE · GENESIS · REDUCER · DEEPDIVE · ASDF · CYBER · BULKCRYPT · METTLESOME · PACKETSPLAYYER · PROMOTER · DEFRAG · XFIP · TRAFFICTHIEF · ENDACE · TIMBERLINE · MUSCULAR · FIST · STARPROC -
UTT Configuration READ ME [pdf]
A SECRET XKeyscore configuration manual describing how to set up the Unified Targeting Tool — NSA's centralised system for pushing surveillance selectors from analysts to field site collection systems worldwide — revealing the complete data flow from analyst targeting request through MAILORDER transport to XKeyscore ingestion, including a forwarding domain table showing how intercepted data is routed to all Five Eyes partners (NSA, GCHQ, CSE, DSD, GCSB) and third parties.
Programs: UTT · TURBULENCE · XKEYSCORE · MAILORDER · SSDM · FISHWAY · SCISSORS · PINWALE · TRAFFICTHIEF -
VALIANTSURF [pdf]
An NSA WikiInfo page for VALIANTSURF — the coverterm for NSA's IPSec VPN decryption capability within the TURMOIL collection platform — describing the four data flow types for intercepting and breaking encrypted VPN traffic, the full deployment roster across collection sites including Menwith Hill, SMOKYSINK, MUSCULAR, CROSSCUT and others, and the architectural taxonomy of the VALIANTSURF programme including its MALIBU and WAIMEA processing architectures and their relationship to LONGHAUL, POISONNUT and GALLANTWAVE.
Programs: VALIANTSURF · TURMOIL · TURBULENCE · HOOKED · HOOKEDHAND · CINDER · CINDERASH · CINDERTRACK · SCARLETFEVER · LONGHAUL · FALLOUT · TOYGRIPPE · PICARESQUE · XKEYSCORE · PRESSUREWAVE · ASDF · MAILORDER · TUBE · KEYCARD · POISONNUT · ISLANDHIDEAWAY · ISLANDTRANSPORT · EXOPUMP · CORALREEF · PINWALE · TUSCARORA · ONEROOF · CONVEYANCE · TURTLEPOWER · WDM · CADENCE · UTT · SCISSORS · FISHWAY · BTUBE · DNCA · ROMULANALE · GALLANTWAVE · ROMULANGAWK · ROMULANGOWL · MALIBU · WAIMEA · WAIMEAVISAGE · ROMULANVISAGE · WAIMEAVISE · ROMULANVISE · APEX · METROTUBE · MADFORGE · JSORCERER · CROSSCUT · CIET · LATTICE · STARQUAKE · SARATOGA · MUSCULAR · SMOKYSINK · SLACKKEY · TUMMS · PURGE · SURGE · NUCLEON · SPAGIC · PPF · SOAP · TUMULT · FIP · PIQBLADE · DEEPDIVE · IBRIDGE