Published in The ISSA Journal, Vol. 1, No. 1, Dec. 1993. Pp. 15ff

by the

Information Systems Security Association

Chicago, Illinois

by

Daniel J. Ryan

Director

Information Systems Security

Office of the

Deputy Assistant Secretary of Defense

(Counterintelligence & Security Countermeasures)

Pentagon • Washington, D.C. 20301

The processes that we are using for evaluation, certification and accreditation of our computers, systems and networks are not working well. When vendors come forward with products, they may be turned away based on our lack of resources to conduct an evaluation. Those that have products accepted for evaluation can expect years to pass before their products are evaluated and placed on the Evaluated Products List. Frequently, their products are technologically obsolete before these processes reach completion. And when their products are finally evaluated, they too often find that the increase in demand that results does not justify the cost of attaining the goal. Certifiers and accreditors may (or may not) take into account that some of the components of the systems and networks they must decide whether to use are evaluated. Even so, each change to a system or network, its operating security mode or its prescribed safeguards requires additional resources for recertification and reaccreditation.

Certainly the existing processes can be improved, but even significant improvements may not prove sufficient given the declining Defense budget expected and the continuous increase in enhanced and new secure products and systems to be evaluated, certified and accredited. Wholly new processes need to be considered if we are to effectively and efficiently achieve the secure computers, systems and networks that both Government and industry require.

The terms "evaluation," "certification" and "accreditation" are used loosely in our community to describe what should be considered separate and distinct processes. In the following discussion, I will use these terms broadly:

• "evaluation" is the process of determining the extent to which a product or system matches an objective a priori standard for the degree of trust which can be placed in the product or system, and is independent of the specific operational environment,

• "certification" is the process of establishing the extent to which a particular design and implementation meets a set of specified security requirements developed expressly for the product or system with knowledge of the intended operational environment, and

• "accreditation" is the management decision that a system or network is approved to operate in a particular operational environment using a prescribed configuration.

The processes to which these terms refer are intended to rely on the results of the earlier processes: certification on evaluation, and accreditation on certification. This does not mean that evaluation has to be completed before certification begiins, nor certification beforeaccreditation begins.

These words are heavily freighted with historical connotations, and since the intent of this discussion is to explore new and different processes that might replace those which are in use today, it will be important to keep in mind the more general definitions given here rather than limiting these words to their applications in today's processes.

Deciding to use a systems requires us to weigh the configuration and vulnerabilities of the system, the environment in which it will be operated and how it will be operated, the value of the system or the information it contains and the threat to each. Knowing the vulnerabilities, we can determine the residual risk inherent in using the system. Knowledge of the operational environment allows us to evaluate how much of the residual risk is abated by other security functions complementary to those of information systems security, such as physical security, personnel security, operational security, and so forth, and by the use of information systems security countermeasures. Understanding the threat permits us to assess the dangers of compromise, destruction or loss of integrity, and loss of availability that remain after all of the security features of the system and the complementary ones in its environment are applied. The remaining danger to the system and the data it creates, stores, processes, and communicates is rarely completely eliminated. Instead, the Designated Approving Authority (DAA) must decide if the level of risk remaining is acceptable in the circumstances. If so, the system is accredited for operation in the given situation.

[p.16] The circumstances in which we operate our computers, systems and networks are, however, highly dynamic. The threat, both from inside our organizations and from without, changes, or our understanding of it does. The operational environment is not stable. Our capabilities in complementary security areas evolve. And, of course, our systems themselves are constantly being improved, either to eliminate residual bugs or to enhance their capabilities. These changes force the DAA to reassess the original decision to accredit. Significant changes -- a system upgrade for example -- may require a de novo certification of the system's security capabilities and limitations. More often, changes are small, subtle and accrete slowly, requiring some care on the part of the DAA in recognizing when the accumulation of changes warrants a recertification. In short, the decision to continue operations is one which must be made frequently - perhaps daily - if security is to be maintained. The notion that complex systems operated in real-world environments can be certified and then used for long periods before reaccreditation is simply not realistic. Risk management must take place continuously over the life-cycle of the system.

The complexity of the systems we use today also makes it difficult to decide how and by whom the accreditation decision is to be made. Consider, as examples, message processing computers or guards or gateways connecting different networks. Is the DAA the security manager of the facility in which the computer is to be operated? Or is the DAA the owner(s)/operator(s) of the network(s) to which the computer is attached and over which the messages it processes are sent and received? Or is the DAA the owner of the messages being processed? Each of these parties has a claim to the role of DAA, since each may be damaged by compromise, destruction or degradation of data caused by the computer. In fact, the DAA may be none of them, as when a system belonging to the intelligence community must be accredited and either DIA or NSA acts as the DAA. These choices allow the accreditation process to take advantage of centralized expertise that is simply too expensive to replicate for every site and every system or network. Similarly, an embedded computer system may be centrally accredited at a depot and strict configuration management used to assure its accreditation is maintained. Although no system should be placed into operation until all of the concerns of the relevant accreditation authorities are resolved, it is nevertheless important to identify a final accreditation authority rather than leave the decision to a committee.

Accreditation is best viewed as a continuous process based on a partnership between:

• a centralized team of experts who are on top of the technologies of systems and networks, security implementation, and security validation, and

• a site-resident security officer in close touch with the day-to-day changes in the operational situation, the availability of complementary security disciplines and their technologies, and the immediate threat, as well as the exact configuration of the system.

Such a partnership is the logical choice for exercising a total quality management approach to continuously accrediting all of the systems at a site. Both auditing by the site-resident staff and red-teaming by a centralized team are important parts of the accreditation process. The centralized team is also ideally suited to prepare standards for accreditation, to perform auditing and other quality control functions, and to train both its own staff and the site security officers.

The authorities who have responsibility for information passing through or stored on the system, or who are responsible for networks to which the system is connected, must rely on this partnership to protect their interests as the situation evolves. That is not to say that the reliance is all one way. The partnership also depends on those who own the networks and information upon which our systems act to provide timely information concerning changes which may effect the security of on-site operations.

In making the complex decision to accredit a computer, system or network for operations, the DAA is assisted by knowledge of the design, including the overall architecture of the system and the security features that have been incorporated into the system and its components. It is also important to be cognizant of the rigor of the programming practices, documentation, testing, and configuration management employed by the developer. Knowing that the implementation was carefully managed, that the architecture is sound and that the security features perform in accordance with the security policy for the system, the DAA can concentrate on the other factors influencing the accreditation decision. Fortunately, the system's architecture and design can be evaluated throughout the development and implementation cycle to ensure that it can operate securely. Management practices can be monitored. Code can be analyzed, line-by-line if necessary, to make sure that little possibility for compromise exists. And testing can be accomplished at the unit level, during integration, and under operational conditions to ascertain the level of assurance provided by the system in maintaining secure operations and communications. Given a standard against which assurance can be measured, a carefully evaluated system can be assigned a security rating that reflects our confidence that the vulnerabilities of the system have been [p.17] discovered and that we know the level of residual risk inherent in operating the system independent of the operating environment and the threat.

Analytically, the process of evaluation is different from and antecedent to the process of certification. It is possible to undertake the evaluation process starting as late as when the computer, system or network is available for delivery. At this point in time all of the design changes have been incorporated into the system and the documentation describing it is complete. The security evaluation may be considered, in such a case, to be an extension of the usual testing process that insures that the system satisfies the requirements that led to its implementation. Indeed, that part of evaluation that comprises testing of the validity of the security features may be considered simply to be part of the overall testing of the various design features and capabilities of the system. Testing of security features may in fact be accomplished as part of the final acceptance tests, and penetration testing may be accomplished as part of operational readiness testing, in order to avoid duplication and undue delay in delivery.

Other parts of the evaluation process, including security analysis of the architecture and auditing of the code are not normally part of the routine testing of a new system, and may add greatly to the delivery schedule if they are not begun early in the development cycle. Evaluation of even relatively simple products may take several months, and the average length of time required is on the order of years. Recent actual experience at NSA places the average time for evaluation at thirty-six months. The time required is, of course, highly dependent on the nature and complexity of the product being evaluated. Average evaluation times for complex networks are much higher than those for simple stand-alone devices.

Since the pre-obsolescence lifetime of high-technology products is sometimes measured in months, an evaluation process that averages years is simply not acceptable. The figure of eighteen months is often heard for high-technology products. However, this figure is also somewhat correlated to the complexity of the system or network. Even complex supercomputers may be quickly overtaken by competing brands, but wide-area communications networks may have life spans measured in years. Any significant delay in product availability under highly volatile market conditions negatively impacts the product's life-cycle value.

Ideally, evaluation and certification should begin at the inception of product or system development and should be completed very soon after completion of functional and operational testing. The efforts of security analysts to determine the soundness of the architecture and design can serve as valuable input to the system developers, avoiding costly and time consuming corrections downstream in the implementation cycle. Such efforts can be completed as the design is completed, so that no time is added to the delivery schedule by these activities. It may sometimes also be possible to review some portions of the code earlier in the implementation. Not all of the code keeps changing right up to the start of acceptance testing, certainly in the case of well-designed, highly modular systems. As already noted, testing of security features can be incorporated into routine unit, integration and operational testing in order to minimize its impact on schedule.

By overlaying the schedule for accomplishing the activities that comprise evaluation, it should be possible to complete the evaluation process nearly concurrently with completion of the other testing of the system, without adding materially to the delivery schedule. Nor does additional risk have to be accepted. While, for some lower assurance products and applications, it might be reasonable to accept some additional risk in trade for accelerated evaluation and certification, the overlapping of schedules described here does not contemplate such tradeoffs.

Certification, while analytically distinct from evaluation, can be accomplished quickly based upon knowledge gained during the evaluation process and need not further extend the delivery schedule. Today, of course, certification is not accomplished quickly. This is, in part, due to the separation between the organizations which evaluate and those which certify, impeding the free exchange of information between the authorities responsible for each of these processes.

Concurrency of schedules for the evaluation and certification processes with system design and implementation has, of course, implications for efficient allocation of scarce qualified, competent evaluator and certifier resources. Only for the largest systems and networks would we expect evaluation to occupy a cadre of evaluators and certifiers full-time. In most cases, evaluators and certifiers would be available to work more than one effort at a time, moving from system to system when their efforts are needed as the implementation progresses. This is, in fact, the way that evaluations are managed today.

Nevertheless, evaluation and certification are time-consuming processes. Today, systems and networks that handle sensitive but unclassified information or other unclassified information are not formally evaluated. We require that evaluations of computers, systems and networks that will create, process, store and communicate classified information be evaluated by Government personnel. But the Government does not have the internal resources needed to stay ahead of the growing [p.18] backlog. NSA's Project OUTREACH proposes helping military service personnel and perhaps other Government civilians -- appropriately trained and qualified, of course -- perform evaluations. For example, certification of the Global Decision Support System is being performed by the Air Force Cryptologic Support Center at Kelly Air Force Base, San Antonio, Texas.

Unfortunately, the service and other agency budgets are also under downward pressure, so relying on them to provide evaluation and certification resources will not solve the problem of long delays. It is interesting to note that at the current time, applications for evaluation may actually be decreasing, and some products that have been submitted are being withdrawn. This appears to be the result of vendor's concluding that there will be insufficient return on the required investment for getting evaluated and certified and their belief that demand is not increased thereby. In other words, the current processes themselves are self-limiting. Since Government and industry need secure products and systems, this is not desirable and would probably not be the case were the processes performing adequately.

NSA personnel do evaluations exceedingly well, with great competence and dedication. But there are not enough of them. Vendors are sometimes turned away, either because there are not enough resources to evaluate their products, or because a preliminary survey indicates that "the products may not be ready for evaluation." Of course, since evaluation is best accomplished beginning at the inception of product development, sending the vendor home on this latter basis is only expedient, not ideal. At the current time, applications for evaluation may actually be decreasing, and some products that have been submitted are being withdrawn. This appears to be the result of vendor's concluding that there will be insufficient return on the required investment for getting evaluated and certified and their belief that demand is not increased thereby. In other words, the processes themselves are self-limiting. This would probably not be the case were the processes performing adequately.

Based on the perceived and stated need for secure products and systems by both Government and industry, the number of products to be evaluated can be expected to grow geometrically, if not exponentially, during the coming years. Forecasts of Government budgets indicate that resources to accomplish evaluations and certifications using Government civilians or service personnel will face increasing backlogs. The solution lies, at least in part, with private industry. The role of Government must shift from that of evaluator/certifier to that of "certifier of certifiers." Such an approach is already in use for TEMPEST evaluations.

Analytically, it would be possible to have the Government retain the role of Certifier even if the technical acts of evaluation and gathering of evidence in support of the certification decision are performed by private industry. The argument in favor of such retention views certification as a management decision based on the findings of the evaluator. This is not in accordance with the formal definition of "certification" contained in the NSTISSC Glossary or the Rainbow Series published by NSA. Practically, if the evaluation and certification analyses are done by private industry, for the Government to retain the role of certifier would either require the Government to maintain a cadre of analysts to "evaluate the evaluation" -- and unnecessary expense -- or, conversely, place the Government in the role of rubber stamper of industry's findings. The latter is undesirable on philosophic grounds, and might place legal liability on the Government that should lie with the private industry evaluating and certifying teams.

Harnessing the power of private industry for the evaluation and certification process offers a variety of advantages. Chief among them is that by adopting the role of certifier of certifiers, the Government achieves a force-multiplier effect for its scarce resources. Said another way, this will allow us to leverage our resources to meet the growing demand. Government experts in evaluation and certification would train private industry to perform these functions in accordance with Government-developed standards. A company that has earned the right to perform evaluations and certifications by having trained and certified personnel would offer its services to developers who would pay to have their products reviewed. While this cost would eventually appear in the price the developer charges for its products, the cost would be spread across the vendor's entire business-base, not borne solely by the Government, thus reducing the overall tax burden of evaluation and certification.

Since information systems security features and assurance are desirable characteristics of computers, systems and networks -- always provided that they do not significantly increase cost or impose undue operational burdens on users -- being able to market a product with an approved certification should be a discriminant in the marketplace. That is, if two or more similar products are available and only one is evaluated and certified, users will prefer to buy the one which is certified. This will insure the demand for evaluator/certifier services.

In some cases, it may even be acceptable to dispense with an independent evaluation and certification. Vendors of computers, systems and networks might be allowed to "self-certify" their own products. Such a self-certification would, of course, be useless without some form of protection for the buyer who elects to buy the product. This protection might take the form of a warranty in which the vendor agrees to correct at no cost to the buyer any problems that arise during some warranty period. From the vendor viewpoint, such a course would be attractive since it would lock the buyer to the vendor for the warranty period -- the buyer would seldom seek another vendor to correct problems which he could have fixed for free by the original vendor. Since actual damages would be extremely difficult to [p.19] predict and in some Government applications might be impossible to calculate, it is unlikely that vendors would be able to accept such a form of liability. However, it might be possible to include a liquidated damages clause in the purchase contract. Finally, it might be possible to delay final acceptance until some "burn-in" period of successful operations had elapsed.

The Government's role in evaluation and certification, while it must change in response to the evolving situation, will remain extensive and important. While for physically protected systems operating in dedicated mode and using components that have not been rated, evaluation, certification and accreditation are not usually difficult, multilevel secure systems involve inherently greater risk. Thus, for some of our most critical computers, systems and networks (nuclear command and control, for example) and for some especially important functions (e.g. cryptology), we may always choose to use our best Government evaluators and certifiers.

Government personnel will also need to stay abreast of the state of the art in systems development and implementation in order to continue to perform and specify effective and efficient evaluations and certifications. Where those functions can be delegated, the Government will have to provide the standards for their application, and for the training and certification of those who will be allowed to perform them, whether in private industry or in Government agencies or military services. Training courses will have to be developed by, and may be taught by, Government personnel experienced in these disciplines.

Quality control will be an important issue. Re-certification at periodic intervals will be necessary for individuals, organizations and companies who wish to be able to provide these services. Legal standards and protective clauses will be needed where self-certification is permitted. Spot inspections will be appropriate to provide intermediate checks and balances. And data bases of security failures will have to be maintained in order to permit analysis of failure rates attributable to individuals and organizations performing evaluations and certifications.

Today, when we evaluate, certify and accredit computers and networks for operations involving sensitive or classified information, we do a thorough and effective job. However, these processes take much too long and the level of resources we can apply to these important processes is dwindling, while the number of systems that should be evaluated and certified can be expected to grow at an increasing rate. To avoid the inevitable overwhelming backlog that lies down our current path, we must seek more efficient and effective processes to accomplish our security assurance functions.

A spectrum of possible processes is available for evaluation and certification. At one end, self-certification may be acceptable if backed up by appropriate warranties or liquidated damages clauses. At the other end, Government personnel will continue to evaluate and certify our most critical systems. In between, private industry, acting in accordance with Government standards, may be able to take up the slack, which will also spread the cost of the functions across the entire business base for secure systems instead of paying for these services entirely with tax dollars. The Government will have to develop training courses and establish procedures and data bases for quality control of individuals and organizations certified to perform these functions.

Accreditation is a continuing process based on knowledge of the vulnerabilities of a system, its operating environment, the value of the system and the information it contains, and the threat. While more than one person and organization may have a stake in the security effectiveness of a system, the process must be based upon a partnership between a centralized team of experts that support and train site security officers and the on-site security authority who is best positioned to recognize security-relevant changes in the system, the operating environment or the threat. Total quality management by this partnership of accreditation of all of the computers, systems and networks operating at the site will ensure that security interests are continuously protected.