The grand challenge of the UltraLog project was to extend the Cougaar cognitive agent architecture using a layered, integrated approach with technologies in robustness, security, stability and scalability. UltraLog's goal was to operate with up to 45% information infrastructure loss in a very chaotic environment with not more than 20% capabilities degradation and not more than 30% performance degradation for a period representing 180 days of sustained military operations in a major regional contingency.
Rising to the Grand Challenge
One of the most significant challenges of the DARPA UltraLog program was to create an infrastructure to allow testing of the entire agent society as a system. There was significant concern that such an effort was formidable at best, and perhaps not even possible due to the emergent behavior of multi-agent systems. In the first year of the UltraLog program a test harness was attempted that was written in Java and utilized a commercial publish and subscribe server at significant cost to the program to achieve communications between the testing elements. Basic functionality was all that was achieved in that first year, and then the test harness only allowed primitive control of the environment that the agent society executed in, not control of the society itself, which remained a manual operation.
The ACME Solution
In spring of 2002 the UltraLog team decided to redouble the efforts to build a distributed testing framework. The team chose to implement its test harness using the Ruby programming language. Ruby is a unique, purely object-oriented scripting language, which allows for sophisticated dynamic behaviors and yet is easily integrated with native environments through its C extension API. Both of these traits were vital because of the complexity of the system under test and the need to tightly control the operating system and network that the system operated in. The outcome was the Automated Configuration Management Environment (ACME) which in its first year of operation allowed the program to perform over 175 assessable executions of the UltraLog system, a significant improvement over the dozen tests of the first year. Over the last two years of the UltraLog program, the UltraLog team continued to increase the power and effectiveness of the ACME framework, arriving finally in 2004 with a unique and powerful artifact in its own right. What started in 2001 as controlling basic elements of 30 Pentium-class PCs (prior to ACME) had evolved into a framework that configures, starts, controls, measures and reports on agent society executions across 350 dual-xeon rack-mounted servers, all with free and open-source components.
A Measured Value
ACME allowed DARPA UltraLog researchers complete control of the environment that their application executed in including simulating real world stresses. ACME also allowed measuring the effectiveness of the logistics system which was the main operational element of the UltraLog program. Although not directly involved in the core survivability research, the ACME testing framework allowed UltraLog researchers to reach higher and farther than they would ever have been able to achieve using manual techniques.