Congress requires the independent assessment of the operational effectiveness, suitability, survivability and — where applicable — lethality of Defense Department weapon and business systems by testing production representative systems, used by regular service members who are trained on the systems before a decision on full-rate production is reached.
This is where the Office of the Director, Operational Test and Evaluation comes in.
Established by Congress in 1983, the DOT&E serves as the principal official and adviser to the secretary of defense, the undersecretary of defense for acquisition and sustainment, the undersecretary of defense for research and engineering, and the secretaries of the military departments on all matters related to both the operational test and evaluation and live-fire test and evaluation of the services and systems acquired by DOD.
Currently, the DOT&E is independently assessing about 250 systems throughout DOD, with a system being "anything from a business system all the way to space satellites and everything in between," said Raymond O'Toole Jr., principal deputy director of operational test and evaluation during a recent interview.
The standard routine for evaluating a system begins with a team from DOT&E partnering with the program office responsible for the acquisition of the system. The team then works with the program office, as well as members of the Office of the Undersecretary of Defense for Research and Engineering, which is responsible for developmental testing, to develop a test and evaluation master plan or a test and evaluation strategy if it's following a pathway other than the major capability acquisition pathway.
During system testing, DOT&E works to get a credible evaluation of each system's suitability, survivability, operational effectiveness and — when appropriate — lethality with a goal of enabling the delivery and fielding of proven capabilities to warfighters.
One key aspect of DOT&E's testing process is that weapons systems are tested in realistic combat conditions.
"We're not restricted or bound by a requirements document with regard to our assessment of the operational effectiveness and suitability of the system," said Garry Bishop, deputy director of operational test and evaluation for land and expeditionary warfare.
"We look at it from a realistic combat environment [with] realistic combat conditions," Bishop continued. "A system may not be required to have certain capabilities against certain threats … but we assess that in that operational environment."
As an example of such testing in the operational environment rather than just testing basic system requirements, O'Toole referenced the Army's Bradley Fighting Vehicle.
"The Bradley doesn't go out by itself," he said. "So, while we do look at the individual Bradley with respect to its survivability against live-fire threats, when we do operational testing, we're also assessing how it's used in operations … as a unit of [fighting vehicles] and not just an individual Bradley."
In addition to the independent assessments that DOT&E conducts, the office also assembles an annual report each fiscal year for Congress and the secretary of defense.
"While we don't report on all 250 programs, we do report on the programs that have had significant operational and live-fire testing in that fiscal year," said O'Toole, who holds a doctorate in engineering. He added that the National Defense Authorization Act requires DOD's service secretaries to document their positions with respect to DOT&E's report.
Along with testing in realistic combat conditions, another aspect of what makes DOT&E somewhat unique among government organizations is the office's independence.
"Under Title 10 [of the U.S. Code], that's what we owe to Congress: our independent assessment, not filtered by the secretary of defense and not filtered by the services, but [rather] it's our assessment. The director's assessment is based upon the data that we provide and the operational testing and live-fire testing that was conducted," Bishop explained.
"So, ... an unbiased, unfiltered assessment is what you get from DOT&E," he said.
Though Congress and the secretary of defense are DOT&E's primary audience, O'Toole sees the organization as being a service to the front-line men and women in uniform.
"I really believe our customer is the warfighter," said O'Toole. "Because we are telling the warfighter what the truth is about, what they are getting out in the field."
To that end, O'Toole said DOT&E has been working for years to see how the organization can speed up the process of gathering data generated in other fields, in hopes of being able to use that data to satisfy some elements of a system's operational test and evaluation master plan, thus potentially expediting that system's delivery.
"We are constantly looking at the ability to increase the speed of our assessments as a faster way to support the needs of the warfighter," O'Toole said.
One way DOT&E is going about that is by working to get away from the linear test model — in which a system has to go through contractor testing and developmental testing before it can get to operational testing — to a more integrated test model.
"We're trying to get more of our operational insights earlier in the [system's] development phase where you can actually affect changes to the design, if needed, and get some insight that determines the scale and scoping of operational testing as you progress through the life cycle of the program," Bishop said.
As one example of that, Bishop referenced a recently completed test of the Army's terrestrial layer system, which is designed to deliver integrated cyber and electronic warfare capabilities to soldiers on the battlefield.
Bishop said that DOT&E was involved from the very early stages of that system's development, which allowed DOT&E to explain to the system developer what the team was looking for in operational testing. As a result, the developer was able to adjust some of the system's testing mechanisms to account for how the soldiers were actually using the system out in the field.
"Testing doesn't cause delays; the results of testing cause delays if deficiencies are found and the developer chooses to correct them," said Bishop. "So, the job of testers — both developmental and operational — is to inform the developer. The earlier they can find those deficiencies and vulnerabilities, the earlier the opportunity to fix the system, so that when it's filtered to our warfighters, they get a credible system."
In addition to test and evaluation within the Defense Department, DOT&E also is responsible for the International Test and Evaluation Program. According to DOT&E literature on the program, the ITEP "permits establishment of bilateral and multilateral agreements between the United States and international partners."
"We are unique," O'Toole said. "There is no other organization like DOT&E, in the world that provides independent assessment."
Being responsible for international test and evaluation sometimes affords DOT&E leadership the opportunity to interact with members of international defense communities. One example is when O'Toole recently held a briefing at the International Armoured Vehicles Conference in London in January.
"During the briefing, the room was very focused, and it's not because I was the guy standing up there; it's because of the message I was delivering," O'Toole said. "And that message was [that] we provide credible assessment, and we're not afraid to go and say where things are wrong, how to go fix it, and then retest it. We also say what is right or working as intended."
Moving forward, O'Toole said that DOT&E will be focusing not just on testing individual systems, but families of interconnected systems.
"And we're looking at that not just from a testing standpoint, but from a training standpoint," he added.
As an example of that, O'Toole mentioned the Joint Simulation Environment that is used to train pilots of the F-35 Lightning II — the DOD's premiere, multirole combat aircraft.
"People are saying to me … that they've gotten more out of sitting in a JSE cockpit than they ever got when they were on the range and out in the open air because of the threats and capabilities that they were able to actually utilize," he said, referring to the fact that safety and environmental restrictions on ranges sometimes preclude the testing of certain capabilities and threats.
O'Toole also said DOT&E is getting more involved with the world of artificial intelligence.
"We're very involved with the industry, and the best and the brightest [individuals] on how to utilize and test artificial intelligence," O'Toole said, adding that DOT&E is fully engaged with DOD's Chief Digital and Artificial Intelligence Office.
"We're thinking ahead, and that's where we're going; that's where we want to be going; and that's where we need to go for the future."