ClusterFactoryTM, the Open-Source Kubernetes-based hybrid multi-cluster Infrastructure Orchestrator is now available to everyone. This software stack is designed and developed to automate the deployment, management, and monitoring of compute-optimised clusters. ClusterFactory combines best-in-class solutions from High-performance Computing, Cloud and DevOps to manage a cluster in combination with the GitOps practice, enabling admins to manage their infrastructure in a scalable and reproducible way.
This state-of-the-art turnkey solution, provides a comprehensive High Performance Computing (HPC) stack, including MPI, Slurm, and DFS, allowing organisations to deploy and scale their infrastructure layer in a cloud-agnostic and unified way. Everything is multi-cluster, and it will be the base of the software stack needed to join the DeepSquare Grid.
“We hope it will get a lot of traction from HPC centers and other types of organisations using big compute workloads because this is a modern compute-optimised infrastructure stack”, said Christophe Lillo, one of the lead brains behind this solution, and Lead HPC Architect at DeepSquare.
This is the most modern way to manage HPC clusters with all the benefits from the GitOps and Kubernetes. A new way of managing infrastructure in a declarative way but also using all “hardcore old school” features that the HPC industry brings for very optimised big compute workloads. ClusterFactory is an easy-to-use and quick-to-configure management solution.
Asked about the time needed for cluster deployment, Lillo said: “It is a matter of hours, depending on whether the provider already has an HPC Cluster in production and already understands the services ClusterFactory provides through similar experiences and configurations. If you are already using Kubernetes, that is not an issue as the user doesn’t have to comply with the grid software stack. If you are agnostic, you can deploy everything from scratch using ClusterFactory“.
ClusterFactory can be compared to Bright Cluster Manager or ClusterVision, but CusterFactory is unique and different. While the software is also open-source, it merges various industries such as HPC and Cloud. It uses Kubernetes to manage those, and none of the HPC cluster managers available right now is following this trend. In addition to this, ClusterFactory is GitOps compliant, meaning that the Git repository is the source of truth for the infrastructure. Every engineer can modify and track their infrastructure directly from their favourite Git hosting service like Github, without having to interact with Kubernetes, and everything gets deployed automatically through pull-based continuous deployment. As everything is configured declaratively on Git, everything can be deployed in seconds, with the same expected state. ClusterFactory brings HPC to Kubernetes and it can be leveraged in HPC centres, regional clouds and hosting providers to offer HPC – while also covering other workloads.
ClusterFactory allows multiple compute servers to be glued together, pulling the most of their computing power without using heavy abstraction layers (e.g. virtualisation) and using more than one server at the same time to solve problems in a very efficient way. The DeepSquare Grid, a decentralised network of high-performance computing clusters, is managed by ClusterFactory, enabling users to join the grid and capitalise on their spare computing resources. We believe in community-driven projects, modularity and decentralisation. This is why ClusterFactory is open-source, highly configurable and GitOps ready to make the deployment of production-ready compute clusters fun and easy.
For more information on the technical aspects of Cluster Factory check Code and Documentation.
This is an important milestone and the Team is already focused on adding more to the core project. A big part of it is related to Data Management – what we call the DeepSquare User Portal which includes permissions, user organisation, and the data management part. This is just the beginning of our development, there is much more to come!
***
We are continuously working on expanding our community and ecosystem. If you want to learn more about the Project or connect with the team and the community, follow us on Twitter, LinkedIn, Instagram, Telegram and Discord.
