Identification of Emergent Collaborative Behaviors in Multi-Agent Systems

Identification of Emergent Collaborative Behaviors in Multi-Agent Systems

Bryson Howell

Multi-Agent Reinforcement Learning (MARL) has been used to allow groups of autonomous agents to perform complex cooperative tasks. When MARL methods such as the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm [1] are used to train teams of agents in cooperative tasks, it has been observed that the actions of individual agents are significantly influenced by the actions of their teammates [2]. Additionally, prior work has shown that teams of agents trained independently of one another under identical conditions display a variety of behaviors [3]. Since these teams have been proven to be coordinated, the MADDPG algorithm is implied to be capable of producing emergent collaborative strategies. If agents can identify these strategies, they can become more adaptive to new teammates by adjusting their behavior to match a successful strategy. In order to work towards this objective, we have designed a method to describe the strategy employed by a team of agents performing a predator-prey pursuit game. By collecting behavioral data for multiple metrics, we demonstrate that certain features are particularly useful for differentiating between team strategies. We verify that our method is capable of meaningfully describing team strategies by testing it on teams of agents using known strategies defined by simple controllers. We then experiment with teams composed of both MARL-trained agents and known strategy agents to test the efficacy of our method when used on teams whose strategy is not well-defined. We hope that this work will inform future attempts to classify groups of agents by team strategy.

Citations

[1] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch, "Multi-agent actor-critic for mixed cooperative-competitive environments," in Advances in neural information processing systems, 2017, pp. 6379-6390.

[2] R. Fernandez, E. Zaroukian, J. D. Humann, B. Perelman, M. R. Dorothy, S. S. Rodriguez, and D. E. Asher, "Emergent heterogeneous strategies from homogenous capabilities in multi-agent systems," Internal work-in-progress, 2020.

[3] D. Asher, M. Garber-Barron, S. Rodriguez, E. Zaroukian and N. Waytowich, "Multi-Agent Coordination Profiles through State Space Perturbations," 2019 International Conference on Computational Science and Computational Intelligence (CSCI), Las Vegas, NV, USA, 2019, pp. 249-252.