"Signal v Noise" Column: Do We Need a Bigger SIGINT Truck?
Open PDFDocument date: 2012-01-23
Type: SIDtoday
Pages: 2
Genre: newsletter
Classification: SECRET SI
Dissemination: REL
A 2012 NSA SIDtoday opinion column arguing that SID lacks adequate internal metrics to evaluate the value and cost of its SIGINT collection capabilities, using the analogy of melon sellers buying a bigger truck to illustrate how expanding capacity without measuring effectiveness is a flawed response to resource constraints.
-
People mentioned:
Hubbard, Douglas, Bethune, Gordon
Countries:
-
Tags:
metrics, SIGINT collection, telephony metadata, collection management, workforce
Identifier:
do-we-need-a-bigger-sigint-truck
PDF SHA-256:
ac96a818f7c5ee08c294fb99dc503dc8086727f0470113a419fc44c0af781ae9
My notes
(U) "Signal v Noise" Column: Do We Need a Bigger SIGINT Truck?
FROM: (U//FOUO) [REDACTED]
Run Date: 01/23/2012
(U) Once upon a time, two people formed a new business. "They built a small shed beside a busy road, rented a truck and drove it to a farmer's field, where they purchased a truckload of melons for a dollar per melon. They drove the loaded truck to their shed, where they sold them for a dollar per melon... As they drove back toward the farmer's field to get another load, one partner said to the other, "We're not making much money on this business, are we?" The partner replied, "Do you think we need a bigger truck?"* Sometimes more is just more, if you're not paying attention to what the metrics are telling you.
(U) This example is a pretty obvious demonstration of the need for business intelligence. The point of business intelligence is that "what gets measured, gets managed." If we don't have the right measurements available, we simply guess. In this example, the melon duo felt like they should just sell more melons to increase profits and to do so meant getting a bigger truck (don't confuse them with the facts).
(S//SI//REL) Throughout SID we often discuss what "a pound of SIGINT" costs or is worth. It's difficult, but not impossible, to figure out the value or cost of a specific pound of SIGINT. "Measuring lets us determine exactly what we can and can't control. Then what we can control, we fix; what we can't control, we either change or isolate. By measuring, we know where to spend our time and energy."** Figuring out what to count across the Enterprise and how to obtain those count results from our SIGINT world can take hours, multiple tools, a variety of frightening pivot tables, multiple queries, a small army of "phone-a-friends," and possibly a registry action. (I am not kidding.) Ironically, for an intelligence organization, operational business intelligence can be hard to come by.
(S//SI//REL) Recently I tried to answer what seemed like a relatively straightforward question about which telephony metadata collection capabilities are the most important in case we need to shut something off when the metadata coffers get full. By the end of the day, I felt like capitulating with the white flag of, "We need COLOSSAL data storage so we don't have to worry about it" (aka we need a bigger SIGINT truck), because getting the metrics for empirical evidence to review was so very difficult and, frankly, I'm still a little scarred by the experience.
(S//SI//REL) Without metrics, how do we know that we have improved something or made it worse? There's a running joke in the S3 community that we'll only know if collection is important by shutting it off and seeing if someone screams. That's one metric anyway, and sadly, the easiest one (and arguably the most effective) to obtain.
(S//SI//REL) So, in this year of decreasing budgets and ever-constrained resources, it's naturally more important to measure what we can to optimize our time and energy. Wherever you sit and whatever grade you clock in at, there are things you can measure to determine how effectively you are doing what you are doing. Start measuring today and set some new 2012 goals to improve your contributions. For a head start, I suggest reading "How to Measure Anything: Finding the Value of Intangibles" by Douglas Hubbard. As an added benefit, this skill will not only help improve our SIGINT business but also dramatically improve your EPA, resume, and ACE bullets! After all, it's one thing to say you need a bigger SIGINT truck, new tool, promotion, etc. and another thing altogether to prove it.
(U//FOUO) Have thoughts on this topic? Post them on the related Tapioca Pebble.
(U) Notes:
* (U) "Do you think we need a bigger truck?" (Source: http://www.hort.wisc.edu/cran/pubs_archive/newsletters/2002/news _2002_09_05.pdf).
** (U) Bethune, G. (1990). "From Worst to First: Behind the Scenes of Continental's Remarkable Comeback."
(U//FOUO) Standard disclaimer: All SIDtoday columns express the views of the columnist alone and do not represent the official views of any NSA organization.
Published in the Same Report
(10 documents)
-
Dealing With a 'Tsunami' of Intercept SIDtoday [pdf]
As part of NSAs efforts to implement Human Language Technology, prototype processors located at collection sites will soon be able to query content at "line speed" based on the language, it is written in.
People: Kulman, Anita H.The Intercept ↗ (2019-05-29) · “Inside NSA, Officials Privately Criticize "Collect It All" Surveillance” - The Intercept ↗ (2015-05-28) · Maass, Peter -
Is There a Sustainable Ops Tempo in S2? How Can Analysts Deal with the Flood of Collection? SIDtoday [pdf]
The NSA's Deputy Director for Analysis and Production discusses the unsustainable operational tempo caused by overwhelming data collection volumes, advocating for "Analytic Modernization" that would shift from treating SIGINT as "radioactive" NSA-only material to sharing raw intelligence with other agencies for collaborative exploitation.
Programs: A-SPACE -
The SIGINT Philosopher: Cognitive Overflow? SIDtoday [pdf]
A 2011 NSA SIDtoday opinion column reflecting on whether the SIGINT Directorate's collective analytical capacity has a finite "channel capacity" — borrowing from cognitive psychology to ask whether SID's ever-expanding mission, tools, and information load risks exceeding the combined cognitive limits of its workforce.
Marking: UNCLASSIFIED -
'Data Is Not Intelligence' SIDtoday [pdf]
A 2007 NSA SIDtoday summary of a keynote address by Deputy DNI for Analysis Thomas Fingar at the Analytic Transformation Symposium, outlining the Intelligence Community's agenda for improving analytical quality, cross-agency collaboration, and integrated operations through training, shared standards, and new tools like A-Space and the Library of National Intelligence.
Programs: A-SPACE -
Op-ed: Leave Bright Pebbles, Not Breadcrumbs, for Those Coming After You SIDtoday [pdf]
A 2010 NSA SIDtoday op-ed by a Chinese language analyst urging analysts to actively document and share their target knowledge through available tools rather than hoarding expertise in personal files or spreadsheets, arguing that the agency's increasing reliance on short tours and rotating assignments makes institutional knowledge-sharing an operational necessity.
-
The SIGINT Philosopher: The Fallacies Behind the Scenes SIDtoday [pdf]
A 2011 NSA SIDtoday opinion column in the "SIGINT Philosopher" series examining four cognitive fallacies — the Texas Sharpshooter, Gambler's, Sunk-Cost, and Appeal to Probability fallacies — and their implications for how analysts unconsciously distort intelligence assessments and decision-making.
-
SIGINT Mission Thread 3 SIDtoday [pdf]
A 2011 NSA SIDtoday article outlining SIGINT Mission Thread 3's strategic initiative to shift the SIGINT Directorate's analytical culture away from data production toward discovery and understanding, introducing tools like the Analytic Body of Knowledge, Analytics Taxonomy, and GAMBIT portal to systematize tradecraft sharing across the Five Eyes enterprise.
Programs: GAMBITPeople: Gladwell, Malcolm -
ELINT Analysts - Overcome by Overload? Help is Here with IM&S SIDtoday [pdf]
A 2011 NSA SIDtoday article announcing the Initial Operational Capability of the ELINT Modernization Information Management and Storage system, designed to give analysts worldwide unified access to five years of ELINT reports and raw data from an expanding range of traditional and non-traditional collection platforms.
-
"Signal v. Noise" Column: Summit Fever SIDtoday [pdf]
A 2012 NSA opinion column in the "Signal v. Noise" series using the mountaineering concept of "summit fever" — the dangerous fixation on reaching a goal despite worsening conditions — as a metaphor for the institutional tendency within SIGINT to continue pursuing failing projects, collection sites, or targets long past the point where objective data would justify stopping.
-
The SIGINT Philosopher: Too Many Choices SIDtoday [pdf]
The inaugural 2011 NSA SIDtoday "SIGINT Philosopher" column by a Russian language analyst, using the everyday paralysis of grocery shopping to introduce the concept of "analysis paralysis" and argue that the SIGINT enterprise's greatest challenge is not a lack of access or information but the cognitive difficulty of prioritizing among an overwhelming abundance of choices.
People: Simon, Herbert
Related Documents
-
"Signal v. Noise" Column: Summit Fever SIDtoday [pdf]
A 2012 NSA opinion column in the "Signal v. Noise" series using the mountaineering concept of "summit fever" — the dangerous fixation on reaching a goal despite worsening conditions — as a metaphor for the institutional tendency within SIGINT to continue pursuing failing projects, collection sites, or targets long past the point where objective data would justify stopping.
-
SIGINT Architecture: Why Do We Need One? SIDtoday [pdf]
Information architectures such as the NSA’s must meet a number of legal, budgetary, and compatibility requirements. Also, a new set of minimum design principles has been agreed upon.
People: McNamee, MichaelThe Intercept ↗ (2018-08-15) -
The SIGINT Philosopher: Unlike All My Terrible Teammates, I Am a SIDtoday [pdf]
A 2012 SIDtoday SIGINT Philosopher column using the annual ACE performance review process as a springboard to examine the structural tension between individual competition and collaborative teamwork at NSA, arguing that the agency's promotion and awards system inadvertently incentivizes analysts to hoard work rather than mentor colleagues or enable others.
Programs: SCQAWK -
The SIGINT Philosopher: Cognitive Overflow? SIDtoday [pdf]
A 2011 NSA SIDtoday opinion column reflecting on whether the SIGINT Directorate's collective analytical capacity has a finite "channel capacity" — borrowing from cognitive psychology to ask whether SID's ever-expanding mission, tools, and information load risks exceeding the combined cognitive limits of its workforce.
Marking: UNCLASSIFIED -
MG Quirk as the SIGINT Director: We're on the Right Path SIDtoday [pdf]
New Director Quirk greets the staff.
The Intercept ↗ (2016-05-16)