Tag: metadata repositories 20 documents
Search documents matching “metadata repositories” →Free-text subjects the curators added. Unlike the other axes these are not necessarily from the documents' own vocabulary, so they are the loosest thing to browse by (and the most forgiving).
Graphs
By Banner
By Classification
Refine (combine with another filter, via search)
Sort: Title
Documents
-
Analytic Challenges from Active-Passive Integration [pdf]
This NSA presentation examines metadata attribution challenges arising when active TAO implants exfiltrate traffic through passive TURMOIL collection infrastructure, creating conflicts where intercepted data carries dual classification markings, case notations, and legal authorities from both collection systems. The briefing highlights how existing databases and analytic tools were not designed to handle traffic with two simultaneous sets of metadata parameters, requiring enterprise-wide solutions to consistently attribute and track actively-shaped intelligence collection.
Programs: BRAVENICKEL · HAMMERMILL · APEX · MARINA · TURMOIL · PRESSUREWAVE · METROTUBE · TOYGRIPPE · HAMMERSTEIN · HAMMERSTONE · METAWAVE · TURBULENCE · SSO -
Analytic Challenges from Active-Passive Integration [pdf]
This NSA briefing addresses the technical and administrative friction caused by "shaping" operations. Shaping occurs when an active malware implant, typically situated on network infrastructure like a router, copies specific data streams and redirects them past a passive surveillance sensor such as TURMOIL. While this ensures the capture of data that might otherwise be missed, it creates a "double metadata" conflict where a single piece of intelligence carries two competing sets of identifiers: one from the active Tailored Access Operations (TAO) implant and another from the passive collection site. The document tasks analysts with finding a way to reconcile these source notations to ensure the data remains searchable and legally compliant within legacy databases.
Programs: BRAVENICKEL · HAMMERMILL · APEX · TURMOIL · HAMMERSTONE · PRESSUREWAVE · METROTUBE · TOYGRIPPE · TURBULENCE · METAWAVE · MARINA · SSO · CONVEYANCE · NUCLEON · FASCIA · HAMMERCHANT -
AURORAGOLD – Target Technology Trends Center support to WPMO [pdf]
This briefing details the technical architecture and data pipeline used to process global cellular network information. It specifically highlights the synergy between data gathering on GSM/UMTS networks and the implementation of "Auto-Minimization" to ensure the resulting SIGINT reporting complies with NSA legal and policy requirements.
-
CSEC Cyber Threat Capabilities [pdf]
This document details CSEC’s full-spectrum cyber operations architecture, integrating SIGINT and IT Security to conduct both defensive and offensive activities under the guise of “information superiority.” While early sections emphasize detection, anomaly analysis, and collaboration with Five Eyes partners, later portions reveal active measures including network infiltration (implant deployment), command-and-control tracking, traffic manipulation and redirection, DNS interference, and potential disruption or deception operations—illustrating CSEC’s capacity not only to defend Canadian networks but also to surveil, exploit, and influence foreign and domestic digital infrastructure.
Programs: EONBLUE · CYBERDMZ · SEEDSPHERE · PHOTONICPRISM · SPECIALSOURCE · SUPERDRAKE · DARKSPACE · QUANTUM · MANDATE · STREAMING SENTRY · SSO · TKB · COVENANT -
CSEC Cyber Threat Detection [pdf]
This document details the Communications Security Establishment’s (CSEC) cyber threat detection and intelligence infrastructure, including systems like EONBLUE, CROSSBOW, CRUCIBLE, and LODESTONE, which enable passive monitoring, anomaly detection, and botnet identification through metadata and deep packet inspection. It outlines how CSEC integrates SIGINT, cryptanalysis, and network forensics to detect and mitigate cyber intrusions using both SIGINT and domestic sensors, while collaborating with Five Eyes partners to enhance Canada’s cyber defense and situational awareness. The presentation is duplicated from page 7 onwards - one of the duplicated slide has internal CSE comments.
Programs: EONBLUE · SPECIALSOURCE · WARRANTEDACCESS · COVENANT · SEEDSPHERE · DIESELRATTLE · SIENNABLUE · DOWNGRADE · CROSSBOW · CRUCIBLE · LODESTONE -
Cyber Defence Operations Legal Policy [pdf]
GCHQ's internal wiki laying out the legal plumbing for its Cyber Defence Operations (formerly NDIST)—detailing warrant types (RIPA, ISA, CNE), data retention schedules, targeting rules for UK/Five Eyes infrastructure, XKEYSCORE query protocols, and the bureaucratic hoops analysts must jump through to search attacker IPs, fingerprint UK networks, and share malware intelligence without tripping over location/nationality safeguards or commercial sensitivities.
Programs: JTRIG · SPAY · GORDIANKNOT · CHORDAL · DISCOVER · EVENTS · CASPORT · XKEYSCORE · CONTENT CLOUD · SQUEAL · MIRAGE · MUTANTBROTH · ANXIOUS · HALTERHITCH · BROADOAK · DONKEYKONG · ZOOL · PALANTIR · OVERLINK · RT · GATEKEEPER · IRONHAND · HARUSPEX · B3M · LOOKINGGLASS · GOLDENEYE · BLAZINGSADDLES · MOONRAKER · TRITON -
Data Stored in BLACK HOLE [pdf]
This document describes BLACK HOLE, a TOP SECRET data repository that stores approximately 200GB per day of internet activity "events" such as webmail, chat, and browsing. It details the system's various datamarts, including MUTANT BROTH and KARMA POLICE, which collectively hold terabytes of data for intelligence analysis.
Programs: BLACKHOLE · B13B · MUTANTBROTH · KARMAPOLICE · MEMORYHOLE · AUTOASSOC · SOCIALANIMAL · HRMAP · INFINITEMONKEYS -
ESOC: Report on the experiences of one NSA worker [pdf]
A NSA intelligence analysis intern details their 2007 temporary assignment to the European Security Operations Center (ESOC), where they successfully established a new signals intelligence (SIGINT) mission targeting Angola's energy sector. The author describes the end-to-end process from initial research and target development to successful collection and the first intelligence product.
-
Events Analysis [pdf]
This GCHQ document explains authorization policies for querying metadata databases (SALAMANCA, HAUSTORIUM, THUGGEE, IMMINGLE), stating that metadata is "less intrusive than communications content" and that authorization is "not needed for individuals in the UK," though searches are logged and audited with necessity and proportionality considerations still applying.
-
HTTP Activity vs User Activity [pdf]
This NSA training document from June 2009 explains the distinction between HTTP Activity (general web browsing data) and User Activity (communications metadata from specific protocols like webmail and chat) within the XKEYSCORE surveillance system, demonstrating through Iranian examples how analysts combine both search types to develop comprehensive intelligence on targets.
Programs: XKEYSCORE · MARINA · FOXTRAIL · DECODEORDAIN · YACHTSHOP · BRUTUS · WINDCHASER · NKB · UIS · STARPROC -
ICREACH Wholesale Sharing (2007) [pdf]
This document serves as an official submission by the SID Technical Director detailing the leadership and collaborative efforts required to transition ICREACH from a conceptual framework into the Intelligence Community's first wholesale metadata-sharing platform. This document emphasizes the role of the SID team in securing critical legal and policy approvals through the NSA's Office of the General Counsel and coordinating with Five Eyes partners to ensure international participation in the system.
Programs: ICREACH -
ICREACH: Sharing SIGINT Comms Metadata with the Intel Community SIDtoday [pdf]
This document serves as an internal announcement marking the 2007 operational launch of ICREACH, a massive signals intelligence (SIGINT) metadata-sharing platform designed to provide real-time access to raw communication records for authorized analysts across the U.S. Intelligence Community. By consolidating telephony, cellular, and satellite data into a "one-stop shop" similar to the NSA’s internal GLOBALREACH tool, the program aimed to fulfill post-9/11 mandates for increased inter-agency collaboration and enhanced target development.
-
Legalities - GCHQ Databases eg Pilbeam, Salamanca, UDAQ etc [pdf]
This October 2007 GCHQ document outlines the legal authorization requirements for querying surveillance databases, revealing that GCHQ analysts can search their own databases (like PILBEAM, SALAMANCA, UDAQ) for metadata on anyone regardless of nationality without additional authorization, while NSA databases (like MARINA, MAINWAY, DISHFIRE) prohibit searches for Americans and other Five Eyes citizens but permit searches for UK persons.
-
Metadata Policy Conference (2008) [pdf]
This document summarizes the high-level discussions from a 2008 conference focused on the legal, policy, and technical challenges of sharing communications metadata between the NSA, its domestic partners, and the Second Party (Five Eyes) SIGINT agencies. It highlights a strategic shift toward bulk data usage and "pattern of life" analysis, while navigating the complex legal distinctions between metadata and "content" under Executive Order 12333 and FISA.
-
NSA Technology Directorate Analysis of Converged Data [pdf]
This document outlines the strategic integration of passive and active signals intelligence (SIGINT) to create a more efficient surveillance loop. The document centers on TURBINE, an intelligent command-and-control system designed to manage a massive network of malware implants (Computer Network Exploitation or CNE). The primary goal is to automate the extraction of data from targeted machines, effectively "converging" traditional passive collection with active hacking to refine intelligence tasking.
-
PRESTON Study [pdf]
This GCHQ document details the PRESTON surveillance program, revealing that over a six-month period, 97% of intercepted communications—including calls, texts, and internet data—were not viewed by analysts despite being fully warranted, highlighting systemic challenges in processing and exploiting the vast volume of collected intelligence. It underscores the reliance on traditional PRESTON sources for events like the London G20, while noting gaps in data feeds from other repositories, reflecting both operational scale and limitations in effective oversight.
-
Report on an XKeyscore training session at the ECC / Dagger Complex [pdf]
This 2012 report details a highly successful XKEYSCORE (XKS) "Circuit Training" event at the European Cryptologic Center (ECC) in Germany, which dramatically increased analyst usage of the powerful SIGINT tool. The training, which used a "speed dating" format and gamification, helped analysts overcome their intimidation and led to significant intelligence discoveries.
-
Sensitive Metadata Analytic Collaboration (SMAC) - Concept of Operations, October 2006 [pdf]
This document outlines the Sensitive Metadata Analytic Collaboration (SMAC) initiative, a system developed by the Five Eyes intelligence agencies to facilitate the sharing and analysis of sensitive metadata to support SIGINT (signals intelligence) missions. While the concept aims to improve collaboration and data sharing, its success is hindered by ongoing policy, legal, and technical barriers, as well as concerns about trust and participation among the agencies involved. The document suggests a complex, somewhat opaque process of metadata sharing that risks undermining privacy protections.
Programs: SMAC -
TEMPORA -- "The World's Largest XKEYSCORE" -- Is Now Available to Qualified NSA Users [pdf]
This document announces expanded NSA analyst access to TEMPORA, GCHQ’s massive XKEYSCORE‑based internet buffer, positioning it as the largest single XKEYSCORE data source in existence. It promotes TEMPORA as a high‑value discovery and development platform for large‑scale internet surveillance.
-
TINT External July 2009 [pdf]
A GCHQ TINT external document from July 2009 describing the BLACKHOLE bulk metadata repository — designed to export unselected metadata in SLR format to enable higher-order analytics — noting anticipated very high volumes expected to exceed MAILORDER transport capacity and proposing a direct connection instead.