One of the fundamental challenges in developing autonomous cybersecurity agents (AICA) is providing them with appropriate training environments for skills acquisition and evaluation. Current reinforcement learning (RL) algorithms rely on myriads of training runs to instill proper behavior, and this is reasonably achievable only within a simulated environment. In this paper, we explore the topic of simulation models and environments for RL and present an assessment framework to compare simulation models designed for simulating cyberattack scenarios. We examine four existing simulation tools, including a new one by the authors of the paper, and discuss their properties, particularly in terms of deployability, to support RL-based AICA. In the example of complex scenarios, we compare the two most sophisticated simulation tools and discuss their strengths.