Transcription

All statements of fact, opinion, or analysisexpressed in this study are those of the author.They do not necessarily reflect official positionsor views of the Central Intelligence Agency or anyother US Government entity, past or present.Nothing in the contents should be construed asasserting or implying US Government endorsement of the study’s factual statements and interpretations.

Curing Analytic Pathologies

The Center for the Study of Intelligence (CSI) was founded in 1974 in response to Director ofCentral Intelligence James Schlesinger’s desire to create within CIA an organization that could“think through the functions of intelligence and bring the best intellects available to bear onintelligence problems.” The Center, comprising professional historians and experiencedpractitioners, attempts to document lessons learned from past activities, explore the needs andexpectations of intelligence consumers, and stimulate serious debate on current and futureintelligence challenges.In carrying out its mission, CSI publishes Studies in Intelligence, as well as books and monographsaddressing historical, operational, doctrinal, and theoretical aspects of the intelligence profession.It also administers the CIA Museum and maintains the Agency’s Historical Intelligence Collection.Other works recently published by CSI include:Analytic Culture in the U.S. Intelligence Community, by Dr. Rob Johnston (2005)Directors of Central Intelligence as Leaders of the U.S. Intelligence Community, 1946–2005(2005)U.S. Intelligence Community Reform Studies Since 1947, by Michael Warner and J. KennethMcDonald (April 2005)Intelligence and Policy: The Evolving Relationship (June 2004)Intelligence for a New Era in American Foreign Policy (January 2004)Comments and questions may be addressed to:Center for the Study of IntelligenceCentral Intelligence AgencyWashington, DC 20505Copies of CSI-published works are available to non-US government requesters from:Government Printing Office (GPO)Superindent of DocumentsPO Box 391954Pittsburgh, PA 15250-7954Phone: (202) 512-1800E-mail: [email protected]

Curing Analytic Pathologies:Pathways to ImprovedIntelligence AnalysisJEFFREY R. COOPERDecember 2005

AcknowledgmentsThe journey that produced this study owes much to many people. Some of them can benamed, others cannot; but all of them have my deepest appreciation and deserve to beacknowledged for their support in this effort. And to my wife Lisa, who can be named—special thanks for putting up with me throughout.I began to appreciate the depth of the Intelligence Community’s analytic problems during aseries of research and analysis efforts for various of its components starting in the mid1990s, and I would like to thank the sponsors of those many efforts even though they cannotbe named. Frank Jenkins, General Manager of Science Applications InternationalCorporation (SAIC) Strategies Group, deserves special praise for allowing Fritz Ermarth andme to follow our instincts and start to investigate these issues before the failures became sovery public. Particular thanks are owed to Henry Abarbanel for a discussion of the contrastsbetween the practice of science and that of intelligence analysis, a conversation thatprompted me to focus on the deep cultural and process factors that affect analytic effortsrather than on the superficial symptoms and manifestations of the failures.I owe a debt to Mike May, Dean Wilkening, and Scott Sagan of Stanford University’s Centerfor International Security and Cooperation (CISAC) for inviting me to give a seminar onintelligence issues that forced me to organize and sharpen my concerns into the originalbriefing on “Analytic Pathologies,” and I am truly grateful to both Aris Pappas and Stan Federfor reviewing that lengthy presentation slide by slide. Paul Johnson, Director of the Centerfor the Study of Intelligence, has my appreciation for an invitation to CSI’s 2003 conferenceon “Intelligence for a New Era in American Foreign Policy”; as do the many intelligenceprofessionals at that conference who helped by bringing their concerns into the open. I wantto thank the members of the Marriott Group on the Revolution in Intelligence Affairs, as wellas David Kay, Mike Swetnam, Gordon Oehler, and Dennis McBride of the Potomac Institutefor providing forums for discussion and for helping me think through these issues with theirinsider perspectives. Thanks are also owed to several former senior intelligence officials whothen pushed me to go beyond diagnosis and address the harder questions of fixing theproblems.I want to thank the Commissioners and my fellow staff members of the President’sCommission on the Intelligence Capabilities of the United States Regarding Weapons ofMass Destruction (the Silberman-Robb Commission) for the lengthy opportunity to delve intothese issues, examine them in great depth, and analyze them within a truly professionalsearch for understanding. I am also grateful to both the former and the current ProgramManagers, Lucy Nowell and Rita Bush, of Advanced Research and Development Activity’s(ARDA’s) Novel Intelligence from Massive Data (NIMD) Program, as well as my teampartners on that effort—Stuart Card, Peter Pirolli, and Mark Stefik from the Palo AltoResearch Center (PARC) and John Bodnar from SAIC—for discussions and research thatled to significant insights on current practices of analysis. I must again thank Paul Johnsonand CSI for providing the opportunity to publish this study and reach a far broader audience;without that spur, I would not have completed it. And to the CSI editors, Mike Schneider andAndres Vaart, my appreciation for their great help in getting me through this entire processand in substantially improving this monograph.v

Beyond those I have already mentioned, I am also truly obligated to a large number of busypeople who took the time and made the serious effort to read and review the earlier briefingand draft study, as well as to share their perspectives and thoughts. Their comments andsuggestions were crucial to producing what I hope is a coherent structure and argument: ArtKleiner, Dan Cohen, Jeffrey Pfeffer, Charles Sabel, Dick Kerr, Stephen Marrin, Bill Nolte,Harry Rowen, Mike Mears, Bruce Berkowitz, Mike Warner, Deborah Barger, Joe Hayes, BillStudeman, Russ Swenson, Ed Waltz, Frank Hughes, Carol Dumaine, David Moore, RobJohnston, Mark Lowenthal, Kevin O’Connell, Carmen Medina, Jim Bruce, Joe Keogh, GregGiles, Winsor Whiton, Bob Cronin, Gilman Louie, and John Seely Brown. In addition, manythanks are due Emily Levasseur, my former research assistant, for her invaluablecontribution in helping me to conduct the research, find important sources and citations,review thousands of pages of source materials, organize and edit, and revise numerousdrafts—all in good humor.Finally, I give my sincerest apologies if I have forgotten anyone who contributed time andeffort to this project. For any errors of omission or commission, I have only myself to holdresponsible.vi

ForewordDear reader, my task in this foreword is to shackle your attention to the challenge of gettingthrough Jeffrey Cooper’s monograph that follows.Your attention is deserved because the subject—what we label with deceptive simplicity“intelligence analysis”—is so important and so interesting. The scope of this monograph, likethat of the analytic profession, is broad and deep, from support to military operations todivining the inherently unknowable future of mysterious phenomena, like the politicalprospects of important countries. Jeff Cooper's study, as befits the work of one who has longbeen an acute observer of the Intelligence Community and its work, is packed with critiques,observations, and judgments. It would be even more satisfying if the study could be furtherilluminated by clinical case studies of failures and successes. In principle, this lack could beremedied if the hurdle of classification could be cleared. In practice, it cannot currently befixed because an adequate body of clinical, diagnostic case studies of both successes andfailures and lessons learned, particularly from the most relevant, post-Cold War intelligenceexperience, simply does not now exist. Not surprisingly, Mr. Cooper, along with many othercritics and reformers, such as the Silberman-Robb Commission (of which he was a staffmember), recommends the institutionalization of a lessons-learned process in our nationalintelligence establishment. This is but one of a rich menu of admonitions to be found in thisstudy.Mr. Cooper has provided a good, thematic summary of the main points of his monograph. Ishall not attempt to summarize them further in this foreword. But some overview commentsare in order.This study is fundamentally about what I would call the intellectual professionalization ofintelligence analysis. It is about standards and practices and habits of mind. It is aboutinductive (evidence-based) analytical reasoning balanced against deductive (hypothesisbased and evidence tested) reasoning. It extols the value of truly scientific modes of thinking,including respect for the role of imagination and intuition, while warning against the pitfalls of“scientism,” a false pretense to scientific standards or a scientific pose without a scientificperformance. It talks about peer review and challenging assumptions and the need to buildthese therapeutic virtues into the analytical process.Mr. Cooper makes reference to the standards and practices of other professions with a highorder of cerebral content, such as law and medicine. Other recognized authors, such asStephen Marrin and Rob Johnston, have written persuasively on this theme. I am struck byhow frequently Mr. Cooper—and others—refers to the example of medicine, especiallyinternal medicine, which has much to offer our discipline. But I am not surprised. When I wasvery young in this business, I was fretting about its difficulties in the company of my uncle,an old and seasoned physician. He walked to his vast library and pulled out for me a volume,Clinical Judgment, by Alvan Feinstein, a work now often cited by intelligence reformers. Ilater asked my mother, my uncle's younger sister, what made Uncle Walt such a greatdoctor. Her answer: He always asks his patients at the beginning, “how do you feel?” and henever makes it home for dinner on time. The model of internal medicine is a great one forcritical emulation by intelligence analysis: science, training, internship, expertise,experience, and then seasoned judgment, intuition, unstinting diligence, and valued secondopinions.vii

Most of what Mr. Cooper writes about concerns the intellectual internals of good intelligenceanalysis, i.e., standards, methods, the tool box of techniques, and the vital element of attitudetoward understanding and knowledge building. With somewhat less emphasis but to goodeffect, he also addresses what might be called the environmental internals of the same:training, mentoring, incentives, management, and leadership. It is in this dimension that wemust overcome the plague recognized by all informed critics, the tyranny of currentintelligence, and restore the value of and resources for deep analysis.This leads to a consideration of the “externals” of good intelligence analysis. To wit:The full scope of analysis: This has to be appreciated for things to come out right. Analysisis not just what a hard-pressed analyst does at his desk. It is the whole process of cerebrationabout the mission and its product. This applies to not only the best answer to a currentintelligence question on the table, but to establishing priorities, guiding collection, and,especially, to judging whether the best effort on the question of the day is good enough tosupport the weight of the situation and the policy decisions that have to be made.Money and people: There is no gainsaying that a lot of our failings after the Cold War are thefault of resource and personnel cuts while old and new and more equally competing prioritieswere proliferating. We've got to fortify the bench strength of intelligence analysis. Thepresident has called for that. Without improved practices, however, new resources will bewasted. We press for improved practices; but they need more resources to be implementedeffectively.External knowledge environments: Half a century ago, when the United States came toappreciate that it faced an enigmatic and dangerous challenge from the Soviet Union, itinvested seriously in the building of knowledge environments on that subject, in thegovernment, in think tanks, in academia, and in other venues. These external sources ofexpertise, corrective judgment, and early warning proved vital in keeping us on track withrespect to the Soviet problem. We have yet to get serious about building such knowledgeenvironments for the challenges of proliferation and, especially, concerning the great strugglewithin the world of Islam, from which the main threat of terrorism emerges. Related to this,Mr. Cooper's study properly places great importance on our improving exploitation of opensources.Information security regimes: We are talking here about a complicated domain fromclassification to recruitment and clearance systems. What we have is hostile to the task ofdeveloping a comprehensive, communitywide knowledge base and operational efficacy inthe age of information and globalization. We need to be more open on a lot of things,especially where the original reason for secrecy perishes quickly and the value of opennessis great (as during the Cold War in regard to Soviet strategic forces), and to tighten up onsecrecy where it is vital, for example, in protecting true and valuable cover.One final—and perhaps most important—point: Mr. Cooper's study of intelligence analysis isshot through with a judgment that is shared by almost every serious professional I've heardfrom in recent years. And it applies to collection and other aspects of national intelligence aswell. We cannot just rely on the new Director of National Intelligence (DNI) superstructure toput things right with our national intelligence effort. The problems and pathologies that inhibitviii

our performance and the opportunities for radically improving that performance are to befound down in the bowels and plumbing of this largely dutiful ship we call the IntelligenceCommunity, and that is where we must studiously, and with determination, concentrate ourefforts and our money.—Fritz Ermarth 11Fritz Ermarth is a former chairman of the National Intelligence Council; he is now a security policyconsultant.ix

ContentsIntroduction . 1Summary . 3Chapter One:Making Sense of the US Intelligence Community . 9Chapter Two:Assessing Critical Analytical Shortfalls . 23Chapter Three:An Inventory of Analytic Pathologies . 29Chapter Four:A Program for Transforming Analysis . 41Appendix:The Analytic Pathologies Methodology . 59Bibliography . 63The Author . 69xi

IntroductionAs a result of a number of analytic projects for different intelligence agencies, amajor focus of my work during the past several years has involved examining thepractice of analysis within the US Intelligence Community. 1 This study was promptedby a growing conviction—shared by others, to be sure—that improving the analyticproducts delivered by Intelligence Community components had to begin with acritical and thorough appraisal of the way those products are created. Aconversation with a physicist friend in 2002 had triggered thoughts on several basicdifferences between the practice of science and intelligence analysis. Shortlythereafter, an invitation to give a seminar on intelligence analysis at StanfordUniversity led me to prepare a briefing entitled “Intelligence and Warning: AnalyticPathologies,” which focused on a diagnosis of the problems highlighted by recentintelligence failures. 2 As Donald Stokes noted in his seminal book on science andtechnological innovation, Pasteur’s Quadrant, “Pathologies have proved to be botha continuing source of insight into the system’s normal functioning and a motive forextending basic knowledge.” 3The Analytic Pathologies framework yields four insights that are crucial both toaccurate diagnosis and to developing effective remedies. First, the frameworkenables analysts to identify individual analytic impediments and determine theirsources. Second, it prompts analysts to detect the systemic pathologies that resultfrom closely-coupled networks and to find the linkages among the individualimpediments. Third, it demonstrates that each of these networks, and thus eachsystemic pathology, usually spans multiple levels within the hierarchy of theIntelligence Community. Fourth, the framework highlights the need to treat both thesystemic pathologies and the individual impediments by focusing effective remedialmeasures on the right target and at the appropriate level.In response to presentations to community audiences, a number of seniorintelligence officials subsequently recommended that I use the diagnostic frameworkof the briefing to develop corrective measures for the dysfunctional analysispractices identified there. I circulated the resulting draft for comment and wasdelighted to receive many useful suggestions, most of which have been incorporatedin this version.1 Although this paper will use the common terminology of “Intelligence Community” (IC), it is worthnoting that the agencies of which it is composed seldom exhibit the social cohesion or sense of purposethat a real community should. A more appropriate term might be “intelligence enterprise,” which isdefined in Webster’s Third International edition as “a unit of economic or business organization oractivity.”2The briefing was first presented in early November 2003 to a seminar at Stanford University’s Centerfor International Security and Cooperation (CISAC) and was revised for a Potomac Institute seminaron the “Revolution in Intelligence Affairs” on 17 May 2004. It will be cited hereafter as “AnalyticPathologies Briefing.”3 Donald E. Stokes, Pasteur’s Quadrant: Basic Science and Technological Innovation.1

Several knowledgeable readers of the draft also raised the issue of the intendedaudience, strongly suggesting that this should be the senior decisionmakers, in boththe Executive Branch and Congress, who could take action to implement the ideasit presented. They also pointedly recommended that the study be substantiallycondensed, as it was too long and “too rich” for that readership. That audience is,after all, composed of very busy people.From the beginning, however, I have intended this study to serve as a vehicle for anin-depth discussion of what I believe to be the real sources of the analyticpathologies identified in the briefing—the ingrained habits and practices of theIntelligence Community’s analytic corps—and not the organizational structures anddirective authorities that are the focus of most legislative and executive branchreformers. Thus, my intended audience has been the cadre of professionalintelligence officers who are the makers and keepers of the analytic culture. Withouttheir agreement on causes and corrective measures, I believe real transformation ofintelligence analysis will not occur.Moreover, during the writing of this study, I wa