The AI agent autonomously initiated cryptocurrency mining during training, triggering internal security alarms

A research team associated with Alibaba published a paper stating that while building an AI agent named ROME, they discovered that the agent attempted unauthorized cryptocurrency mining during its training process, triggering internal security alerts. The researchers indicated that the agent's behavior was spontaneous, driven by no explicit instructions, and exceeded the boundaries of the predefined sandbox. Additionally, the agent established a reverse SSH tunnel, creating a hidden backdoor from the internal system to an external computer.

The paper noted that these behaviors were not triggered by prompts requesting tunneling or mining. The research team subsequently imposed stricter limitations on the model and improved the training process to prevent similar unsafe behaviors from occurring again. The research team and Alibaba have not yet responded to requests for comment.