Tag: big data 10 documents
Search documents matching “big data” →Free-text subjects the curators added. Unlike the other axes these are not necessarily from the documents' own vocabulary, so they are the loosest thing to browse by (and the most forgiving).
Graphs
By Banner
By Classification
Refine (combine with another filter, via search)
Sort: Title
Documents
-
BOUNDLESSINFORMANT - Describing Mission Capabilities from Metadata Records [pdf]
This presentation on the Boundless Informant program (BOUNDLESSINFORMANT), which uses big data tools to analyze and visualize global SIGINT metadata collection in near real time. It explains how the system replaced older manual methods, enabling the NSA to track collection volumes, coverage, and trends for specific countries or sites through cloud-based infrastructure.
People: Patel, Vimesh“NSA-files repeatedly show collection of data «against countries» - not «from»” - Dagbladet ↗ (2013-11-22) · “Boundless Informant: the NSA's secret tool to track global surveillance data” - The Guardian ↗ (2013-06-08) · “Claves y giros del espionaje masivo en España” - El Mundo ↗ (2013-11-04) · Greenwald, Glenn · MacAskill, Ewen · Aranda, Germán -
FY 2013 Congressional Budget Justification, Volume I, National Intelligence Program Summary, February 2012 [pdf]
This document outlines the $52.6 billion “black budget,” detailing the structure, priorities, successes, and shortcomings of the 16 U.S. intelligence agencies. It reveals unprecedented insight into funding allocations, operations, and intelligence gaps—including the CIA’s dominant role, expanding cyber operations, and persistent challenges in areas such as North Korea and counterterrorism—while emphasizing the tension between secrecy and public accountability.
-
GCHQ Analytic Cloud Challenges [pdf]
The 2012 GCHQ Analytic Cloud document outlines a sophisticated data processing infrastructure handling approximately 50 billion daily events across multiple sites, utilizing QFD databases and HADOOP clusters for various analytic capabilities targeting intelligence gathering and analysis. It describes architectural components, challenges in scaling and enabling analysts to cope with data complexity, and proposes future technology options including consolidation approaches and new cloud architectures to enhance capabilities.
Programs: EVOLVEDMUTANTBROTH · MUTANTBROTH · HARDASSOC · HRMAP · KARMAPOLICE · SOCIALANTHROPOID · AUTOASSOC · GEOFUSION · MVR · TERRAIN · SALAMANCA · BLACKHOLE · SAMUELPEPYS · BROKER · VAIL · GUIDINGLIGHT · FIRSTCONTACT · CLOUDYCOBRA · RUMOURMILL · EVERYASSOC · EVERYPOLICE · EVERYCREATURE · EVERYEAD · EVERYCIPHER · GREYFOX · SILVERFOX · GOLDENAXE · TRIBALCARNEM · PUBLICANEMONE · EPICFAIL · STERLINGMOTH · FOGHORN · LOOKINGGLASS · FIREENGINE · NGGEO · ALPHACENTAURI · CLOUDBASE · ACCUMULO · ASEM · SEM · HAKIM · STREAMING · HBASE · LAUREL · SKINNYCLOUD · HARDY · GCHQCORE -
HIMR Data Mining Research Problem Book [pdf]
This document is a research roadmap created by GCHQ for the Heilbronn Institute for Mathematical Research (HIMR). Its purpose is to guide long-term mathematical and data-mining research that could improve signals intelligence (SIGINT) capabilities. It outlines key research problems, datasets, and collaboration plans between intelligence agencies and academic researchers. The main goal is to develop new algorithms and mathematical tools to analyze extremely large communications datasets produced by modern surveillance systems.
Programs: CLOUD · DISTILLERY · DISCOVER · PRESTON · LECKWITH · TERRAIN · BROADOAK · NGE · FIVEALIVE · CHARTBREAKER · FIRSTCONTACT · GEOFUSION · RADONSHARPEN-B · PRIMETIME · HIDDENOTTER · BAKERSDOZEN · CLASP · SALTYOTTER · BIRCH · SOLIDINK · FLUIDINK · MAMBA · GRINNINGROACH · PIRATECAREBEAR · AUTOASSOC · CROUCHINGSQUIRREL · HRMAP · KACHINA · SQUEAL · QUESTA · METEORSHOWER · SUNSTORM · GOLDMINE · HAGERAWEL · WOODY · BUZZ · SILVERLIBRARY · SILVERLINING · VALHALLA · MOUNTMCKINLEY · SEPANG · BLACKHOLE · SALAMANCA · HAGELAWEL · LUCKYSTRIKE · SPIKYROCK · MYOFIBRIL · INJUNCTION · PSYCHICSALMON · RAGINGBULLFROG · ROBOTICFISH · TIMIDTOAD · AURA · CARBONCOPY · CASK · CHORDAL · CRAN · INSTINCT · MADFORGE · SWAMP · VOLSUNGA · WHITERAVEN · MVR · PPF · JTRIG · MAPREDUCE · MAILORDERPeople: Tukey, John -
ICTR Cloud Efforts [pdf]
A GCHQ presentation details how the agency developed cloud-based big data analytics to achieve "population scale" surveillance—processing trillions of internet events to build geographic and behavioral profiles on every online identifier worldwide, enabling discovery of unknown targets through pattern analysis rather than traditional keyword searches.
Programs: EVERYASSOC · PROBABILITYCLOUD · SILVERLINING · CULTWEAVE · KARMAPOLICE · EVERYPOLICE · AWKWARDTURTLE · GORDIANKNOT · SANDIA · BLOODHOUND · FIVEALIVE · NGE · MAPREDUCE -
LEVITATION and the FFU Hypothesis [pdf]
This document describes LEVITATION, a top-secret Canadian Communications Security Establishment (CSE) project that monitors millions of daily file uploads and downloads from free file-sharing sites to identify potential extremist activity. It details the program’s data collection methods, analytical workflow, use of partner databases, and examples of successful discoveries related to terrorist materials and hostage videos.
-
Quadrennial Intelligence Review Final Report, 2009 [pdf]
This 2009 strategic review from the Director of National Intelligence outlines potential future operating concepts for the US Intelligence Community, including "safe bets" like mass data analysis (Sentient Enterprise) and controversial "strategic hedges" such as using clandestine cyber operations to acquire proprietary technology from foreign corporations to benefit US industry (Technology Acquisition by All Means).
People: Shedd, David R. -
Report on Architectural Risk 2012 - Summary [pdf]
This 2012 GCHQ report identifies that a historical under-investment in IT architecture has created significant risk, especially with new challenges like massive data growth, cyber defense, and increased partnering. The document recommends immediate, prioritized investment in big data strategies, integrated analytics frameworks, and security services to address the strain on the system.
Programs: BLACKHOLE -
SSO GHOSTMACHINE Analytics [pdf]
This document describes the NSA’s cloud analytics platform GHOSTMACHINE, used by Special Source Operations (SSO) to process and analyze massive volumes of data (e.g., from FASCIA) using Hadoop-based infrastructure. It’s essentially the backbone system for querying, correlating, and enriching surveillance data at scale — not just a single analytic, but the analytic/cloud environment powering many programs.
Programs: GHOSTMACHINE · FASCIA · CERF · POPTOP · DRT · SIGDASYS · JUGGERNAUT · SEADIVER · AST128 · TUSKATTIRE · ASDF · BROKENRECORD · REDHAWK · SSO -
The collection challenge [pdf]
A slide stating the collection problem in one line: “Collection is outpacing our ability to ingest, process and store to the ‘norms’ to which we have become accustomed.”