MITRE uses risk assessment to improve project outcomes:Best Practice Award

KMWorld Best Practice Award 2001

Nearly a decade ago, the MITRE Corporation (mitre.org) began developing a system for internal use to capture lessons learned from various projects it carried out for its clients. The not-for-profit company operates federally funded research and development centers for the Department of Defense, the Federal Aviation Administration, and the Internal Revenue Service. Those centers provide systems engineering, research and development, and information technology support. The company applies leading-edge technologies to help its clients address such issues as interoperability, security and communications. Thus the ability for employees to share current information is critical.

The initial vision for the system came from Paul Garvey, chief scientist, who saw the value of analyzing and recording risk-related events so that staff across the corporation could benefit from lessons learned. A prototype was developed in SuperCard, a Macintosh software product that was among the first user-friendly cataloging and hyperlinking tools. Several years later, the Risk Assessment and Management Program (RAMP) was formally launched over MITRE’s intranet using an Access database.

Risk assessment is a generic term used across many different fields, including computer security, software development, insurance, finance, health and environment. The risk assessment process uses previous experience and current information to minimize the probability of a negative outcome. It includes activities such as identifying, analyzing, prioritizing and regularly monitoring risks.

Most organizations could benefit from analyzing risks associated with previous projects, but all too often, no formal system is in place to capture and disseminate information about past work. Those organizations that do take on the challenge are reaping significant benefits. Particularly when risks can be quantified, as they are in the insurance industry through statistical techniques, risk assessment can be a strong decision support tool. Risk assessment in professional services fields such as systems development is more difficult, because, among other things, the end products are highly complex and the technologies employed change rapidly. Despite those challenges, risk assessment has become a necessity in program planning and is vital to reducing unsuccessful outcomes.

Defining and analyzing the risks

Over the past six years, the RAMP system at MITRE has been used to capture information about hundreds of projects. The primary data collection method is the personal interview, during which RAMP staff meet with key project personnel who have detailed knowledge of risks encountered and addressed. A template is also available for users who wish to enter data directly into the system. RAMP contains an extensive list of risks, project descriptions, contact information and listings of relevant technologies. The system also incorporates project documents, briefings and related white papers.A taxonomy was developed that classifies different types of risks. Programmatic, operational and technical risks are some of the basic types included.

“We define risk broadly, as anything that might impact cost, schedule or mission,” says Audrey Taub, group leader in the Economic and Decision Analysis Center and RAMP project leader. Programmatic risks include estimating costs or schedules. Technical risks include such activities as migrating a system from one platform to another. The taxonomy provides a risk checklist that both enables the users to navigate through the body of information, and offers them a basis for developing their own risk plan.

“Many of our projects include an integration of commercial off the shelf software (COTS) with non-COTS software and hardware,” says Taub. “We have good documentation on the difficulties that can arise during this process, and can offer practical risk mitigation examples.” One such example is the need to incorporate a vendor viability assessment along with a product assessment when acquiring COTS products. The quality of the product can become irrelevant if the vendor is no longer in business to support it.

Taub stresses that the role of RAMP is not just to prevent negative outcomes, but also to highlight successes.

“RAMP includes examples of solutions as well as problems,” says Taub. “RAMP might provide an example in which the development of a simulation model for a complex system allowed the engineers to identify potential problems prior to making a costly development commitment.” That approach could then be used by engineers in other similar efforts.

In the early years, the strategy for RAMP focused on collecting information on individual projects to establish a critical mass of data. Most recently, RAMP has focused on developing a risk assessment and management toolkit that embeds the lessons learned and best practices of existing projects. That new material can be used to jump-start and sustain a life cycle risk management program. Its purpose is to provide one-stop access to risk analysis and management guidance, including sample risk management plans and templates, and tutorials on risk-related topics. One new tool provides a visualization of risk based on the probability of occurrence, priority and mitigation status. While managers often can identify risk, this tool helps them analyze risk and provides graphical representation that serves as a mechanism for focusing on the risks that are likely to be the most important to manage.

Nurturing the culture

RAMP was one of the earliest initiatives developed at MITRE for knowledge sharing. Since then, the company has established a collaborative environment that supports such sharing.

“Our knowledge management strategy includes encouraging a knowledge-sharing culture, enabling staff with knowledge resources and knowledge management processes, and providing the needed systems infrastructure,” says Jean Tatalias, director of Information Management. The organization has been aligning awards and recognition with goals for knowledge sharing.

“We have also worked to embed knowledge management as part of the way we do business,” adds Tatalias, “and to provide an infrastructure to assist with this.” The MITRE Information Infrastructure (MII) includes an intranet, extranet and public server that provide gateways to corporate applications, to project documentation, corporate and technical news, and other information.

Built on the collaborative environment at MITRE, RAMP also supports it. Employees at various site offices have commented that the system allows them to keep informed about and connected to projects throughout the corporation. In addition to being used by employees to gather information for projects, it is used to identify experts within MITRE, who can then be consulted for further details on an issue. One organization used RAMP as part of a systems engineering training course. As a class exercise, students provided lessons learned from their current projects for inclusion in RAMP. Regular surveys are conducted to get user feedback on the system, which provides valuable information on usage and also suggestions for enhancements.

RAMP is accessed most intensely at the beginning of the fiscal year when project planning is underway. It’s part of a corporate memory that helps to mitigate the impact of employee turnover and to eliminate redundant research. It has also helped time-constrained employees move quickly up the learning curve on a particular topic. Available 24/7, it has been an effective way to capture and disseminate critical information that otherwise might have been lost, and contribute in a significant way to MITRE’s overall mission.

Judith Lamont is a research analyst with Zentek Corp., e-mail jlamont@sprintmail.com