"Signal v Noise" Column: Do We Need a Bigger SIGINT Truck?

Open PDF
Creating agency: NSA National Security Agency (US)
Document date: 2012-01-23
Type: SIDtoday
Pages: 2
Genre: newsletter
Classification: SECRET SI
Dissemination: REL

A 2012 NSA SIDtoday opinion column arguing that SID lacks adequate internal metrics to evaluate the value and cost of its SIGINT collection capabilities, using the analogy of melon sellers buying a bigger truck to illustrate how expanding capacity without measuring effectiveness is a flawed response to resource constraints.

Long Description:
This SIDtoday column, published January 23, 2012 under the "Signal v Noise" series, makes an organizational argument that the SIGINT Directorate suffers from a significant deficit of internal business intelligence — the metrics and measurements needed to evaluate whether collection activities are actually producing value. The author opens with a parable about melon sellers who mistake volume for profit, using it to illustrate a broader institutional tendency within SID to respond to capacity problems by seeking more resources rather than measuring whether existing resources are being used effectively. The column's central critique is that determining something as seemingly basic as which telephony metadata collection capabilities are most important — for example, when storage limits require prioritization decisions — requires hours of work across multiple tools, queries, and informal consultations, and still may not yield reliable empirical answers. The author notes a running joke within the S3 collection community that the only reliable way to determine whether a collection capability is valuable is to shut it off and wait for complaints — a metric that is simultaneously the easiest to obtain and the most organizationally damaging to apply. In the context of 2012 budget constraints and declining resources, the column frames this measurement gap as increasingly consequential: without the ability to quantify the value of specific collection activities, SID cannot make rational prioritization decisions and risks defaulting to the "bigger truck" fallacy — seeking more storage, more tools, or more budget rather than optimizing what already exists. The author closes with a call for individual analysts at all levels to begin measuring their own contributions, framing it as both a mission imperative and a career development tool.
Programs:
-

People mentioned:
Hubbard, Douglas, Bethune, Gordon

Countries:
-

Tags:
metrics, SIGINT collection, telephony metadata, collection management, workforce

Identifier:
do-we-need-a-bigger-sigint-truck

PDF SHA-256:
ac96a818f7c5ee08c294fb99dc503dc8086727f0470113a419fc44c0af781ae9

(U) "Signal v Noise" Column: Do We Need a Bigger SIGINT Truck?

FROM: (U//FOUO) [REDACTED]
Run Date: 01/23/2012

(U) Once upon a time, two people formed a new business. "They built a small shed beside a busy road, rented a truck and drove it to a farmer's field, where they purchased a truckload of melons for a dollar per melon. They drove the loaded truck to their shed, where they sold them for a dollar per melon... As they drove back toward the farmer's field to get another load, one partner said to the other, "We're not making much money on this business, are we?" The partner replied, "Do you think we need a bigger truck?"* Sometimes more is just more, if you're not paying attention to what the metrics are telling you.

(U) This example is a pretty obvious demonstration of the need for business intelligence. The point of business intelligence is that "what gets measured, gets managed." If we don't have the right measurements available, we simply guess. In this example, the melon duo felt like they should just sell more melons to increase profits and to do so meant getting a bigger truck (don't confuse them with the facts).

(S//SI//REL) Throughout SID we often discuss what "a pound of SIGINT" costs or is worth. It's difficult, but not impossible, to figure out the value or cost of a specific pound of SIGINT. "Measuring lets us determine exactly what we can and can't control. Then what we can control, we fix; what we can't control, we either change or isolate. By measuring, we know where to spend our time and energy."** Figuring out what to count across the Enterprise and how to obtain those count results from our SIGINT world can take hours, multiple tools, a variety of frightening pivot tables, multiple queries, a small army of "phone-a-friends," and possibly a registry action. (I am not kidding.) Ironically, for an intelligence organization, operational business intelligence can be hard to come by.

(S//SI//REL) Recently I tried to answer what seemed like a relatively straightforward question about which telephony metadata collection capabilities are the most important in case we need to shut something off when the metadata coffers get full. By the end of the day, I felt like capitulating with the white flag of, "We need COLOSSAL data storage so we don't have to worry about it" (aka we need a bigger SIGINT truck), because getting the metrics for empirical evidence to review was so very difficult and, frankly, I'm still a little scarred by the experience.

(S//SI//REL) Without metrics, how do we know that we have improved something or made it worse? There's a running joke in the S3 community that we'll only know if collection is important by shutting it off and seeing if someone screams. That's one metric anyway, and sadly, the easiest one (and arguably the most effective) to obtain.

(S//SI//REL) So, in this year of decreasing budgets and ever-constrained resources, it's naturally more important to measure what we can to optimize our time and energy. Wherever you sit and whatever grade you clock in at, there are things you can measure to determine how effectively you are doing what you are doing. Start measuring today and set some new 2012 goals to improve your contributions. For a head start, I suggest reading "How to Measure Anything: Finding the Value of Intangibles" by Douglas Hubbard. As an added benefit, this skill will not only help improve our SIGINT business but also dramatically improve your EPA, resume, and ACE bullets! After all, it's one thing to say you need a bigger SIGINT truck, new tool, promotion, etc. and another thing altogether to prove it.

(U//FOUO) Have thoughts on this topic? Post them on the related Tapioca Pebble.

(U) Notes:
* (U) "Do you think we need a bigger truck?" (Source: http://www.hort.wisc.edu/cran/pubs_archive/newsletters/2002/news _2002_09_05.pdf).
** (U) Bethune, G. (1990). "From Worst to First: Behind the Scenes of Continental's Remarkable Comeback."

(U//FOUO) Standard disclaimer: All SIDtoday columns express the views of the columnist alone and do not represent the official views of any NSA organization.