About Tennessee Tech University

• Tennessee Tech University (TNTECH) is a public comprehensive university located in Cookeville, Tennessee, with approximately 10,700 students. The university offers more than 225 fields of study with a strong focus on STEM disciplines
• The university’s cybersecurity students are training to fight some of the most dangerous malware on earth, using CEROC’s Cyber Range
• The Cyber Range provides a specialized environment for studying live, contained malware, simulating offensive cyberattacks, and conducting dynamic analysis on critical infrastructure like industrial control systems and banking networks

Building the cybersecurity infrastructure of tomorrow

The Cybersecurity Education, Research and Outreach Center (CEROC), part of the College of Engineering at Tennessee Tech University, is driven by a clear mission: to equip students with the knowledge and skills needed to confront the world's most dangerous and destructive malware and cyber threats – building a safer digital infrastructure and a more secure future for society.

To get them ready for this vital work, CEROC provides defensive and offensive training, where students take on complex role-playing situations. For example, students might step into the shoes of SysAdmins in Security Operations Centers defending critical infrastructure against attackers, or practice penetration testing in virtual banks or cruise ships.

CEROC's Cyber Range is the home base for all of this training. The Cyber Range is a highly specialized, secure sandbox environment that allows students to perform dynamic analysis on live computer viruses, worms, and ransomware, or practice high-risk cyberattack situations where highly sensitive data or critical infrastructure is on the line. In this “laboratory” environment, it’s crucial that working with real threats doesn’t cause real harm to systems outside of the range. In the sandbox, the consequences of mistakes can simply be deleted along with the VM where they were made.

However, this critical infrastructure was threatened when CEROC’s legacy virtualization provider suddenly issued notice of unmanageable cost increases, and an end to their educational licensing tier.

For CEROC Cyber Range Engineer Travis Lee, this wasn't just a budget line item; it threatened the viability of a high-velocity, bespoke system that manages over 6,500 ephemeral virtual machines every semester. Tasked with supporting critical cybersecurity research involving live, contained malware and specialized NVIDIA A100 GPU workloads, the team had to find a way to migrate 1,700 concurrent VMs to a new platform before the new semester.

This is the story of how CEROC migrated the entire Cyber Range in under 4 weeks, using the open source, high-performance combination of Ubuntu, MicroCloud, and Ubuntu Pro + Support.

Challenge

The Tennessee Tech Cyber Range isn’t any ordinary virtualized environment. Unlike standard enterprise clouds, this 10-node system must handle a constant churn of thousands of concurrent VMs spun up and destroyed over the course of a single semester.

Before implementing MicroCloud, the university relied on a legacy virtualization stack that had been in place since 2017. That all came to an end when CEROC faced an unsustainable price hike. These astronomical costs meant that a migration to other providers or systems was necessary. An open source, more cost-effective option was greatly needed.

However, the search for a replacement was a complex undertaking. The Cyber Range’s highly rigid technical requirements meant that many conventional options simply weren’t on the table. For example, the Cyber Range utilizes a NVIDIA A100 GPU node for live malware analysis and AI research, meaning the new hypervisor had to be compatible with the NVIDIA AI Enterprise support matrix – immediately narrowing down the shortlist of supported platforms. Of the few remaining commercial options that could meet hardware compatibility requirements, several were dismissed as unsuitable for the Cyber Range's specific high-churn, high-velocity workflow. Additionally, the chosen solution needed to be securely designed and maintained against the latest vulnerabilities, given the highly sensitive critical infrastructure and risk factor of the work involved.

What’s more, it wasn't just budget and hardware demands that were blockers – it was finding the right fit for their mission of openness and community upliftment.

CEROC regularly gave back to its community, sharing cybersecurity tools, training, and awareness with K-12 schools and community colleges that lack the budget for enterprise software. In order to keep up this important work, the university needed a platform that was open source at its heart: one that didn’t just meet system requirements and performance specifications, but one which could be shared at low or no costs with schools, community centers, and the general public, for the greater cybersecurity good of CEROC’s wider community.

Ultimately, Ubuntu and MicroCloud were the open source path that CEROC chose to migrate and rebuild its vital Cyber Range.

Solution

To replace their now unaffordable legacy infrastructure, Tennessee Tech implemented a self-deployed MicroCloud stack built on Ubuntu 22.04 LTS and running on mostly Dell servers and hardware.

MicroCloud is a lightweight, featureful, and open source cloud for on-demand computing at the edge. MicroCloud reuses proven cloud primitives with unattended, autonomous, and clustering features that resolve typical edge computing challenges, and which allow you to scale from a small 1-node to a larger 50-node cluster. MicroCloud’s open source nature, simplicity, and easy scalability up to larger clusters made it a suitable choice.

The new architecture was designed for extreme high-density performance, consisting of a 10-node cluster featuring roughly 13TB of total RAM and 270TB of raw storage. A specialized node equipped with four NVIDIA A100 GPUs was integrated into the cluster to facilitate AI-driven malware research.

The MicroCloud installation process made for a rapid turnaround, thanks to its extensive documentation, and the ability to self-test the entire 3-node clustering logic within virtual machines before committing to the physical hardware.

Of course, the complexity and size of CEROC’s setup led to challenges in the migration where the technologies were pushed to their design limits. To deal with these, CEROC made use of Ubuntu Pro + Support, Canonical’s comprehensive subscription for security, support, and compliance, giving them access to fast answers and assistance with issues they couldn’t resolve themselves. Getting Ubuntu Pro + Support was simple and fast, as an add-on purchased directly through Dell, an established hardware provider to CEROC.

The expert and timely assistance of Canonical’s support engineers played a pivotal role in smoothing out the transition, providing rapid, direct assistance whenever issues came up. Rather than dealing with a traditional help desk, the CEROC team were able to work alongside engineers who issued hotfixes and database tweaks specifically tailored to the Cyber Range’s high-churn workload.

Results

CEROC’s migration to MicroCloud has delivered a reliable, long-term supported, and high-performance operational state for Tennessee Tech University. CEROC was able to successfully migrate 10 nodes and rebuild 1,700 production VMs in just less than four weeks.

Technically, the implementation has proven to be a powerhouse of reliability and performance at scale. The integration of MicroCeph (a lightweight, snap-deployed version of Ceph designed for simple and rapid deployment of distributed storage clusters on Linux) has resulted in an extremely fast storage environment where large-scale projects build and deploy significantly faster than under the previous legacy setup. The system now manages a concurrent load of 2,100 VMs with total stability, while the native containerization support in LXD has allowed the university to nearly triple its workload density – scaling toward a capacity of 4,700 instances on the same physical hardware.

The new setup has also benefitted from performance improvements: testing of a 100-box environment shows that MicroCloud takes only 4 minutes to build – a 20% faster build time than CEROC’s previous solution. This ensures that the new solution can be scaled with greater ease.

The new build also brings enhanced security, thanks to the streamlined security maintenance commitment delivered by Ubuntu Pro. This provides the long-term security, compliance, and operational predictability that they need as a major educational institution.

Beyond stability and economics, the migration gave CEROC the opportunity to rebuild, refresh, and optimize decades-old systems, while allowing for new and cutting-edge technology. With this large-scale migration, CEROC successfully rebuilt their entire automation pipeline and created new automations to manage IP address allocation. MicroCloud, running on Ubuntu, was naturally compatible with the NVIDIA AI Enterprise stack, allowing the university to pursue machine learning and quantum computing research within the same virtualized environment used for basic labs.

This new MicroCloud setup also respects the university’s outreach mission: with it, the university can share their research and tools with the K-12 schools, community colleges, and community colleges that need them most – without the traditional enterprise subscription costs.