README
¶
What is Colonies?
ColonyOS is an open-source framework designed to facilitate seamless execution of computational workloads across platforms, e.g. cloud, edge, devices, or High-Performance Computing (HPC), creating so-called Compute Continuums. Colonies is key component of ColonyOS and provides a Meta-Orchestrator, delivering functionalities like a traditional operating system, but functions as an overlay on top of existing platforms.
At its core, Colonies provides a distributed runtime environment, called a Colony, consisting of a network of loosely connected Executors. Executors are distributed microservices that can easily integrate with any third-party application or system. They can be implemented in any language and run anywhere on the Internet, e.g. on supercomputers or in a web browser on a smart phone.
Colonies has been developed with a strong emphasis on security and robustness to ensure scalability and reliability.
How does it work?
- Users submit function specifications to a Colonies server specifying computations they wish to run. These functions will then be assigned and executed by the Executors.
- Developers can focus on implementing Executors as independent microservices. The system can then easily scale just by deploying more Executors, enabling parallel execution. Failed processes are automatically re-assigned to other Executors.
- Developers can define and submit workflows describing a sequence of computations carried out by several Executors.
- Colonies makes it possible to develop a loosely-decoupled architecture that spans multiple platforms and infrastructures, with all coordination managed by Colonies.
- Colonies also functions as a ledger, containing full execution history. This enables developers to keep track of the system and more easily debug it.
Design
- Colonies serves as an intermediary layer to indirectly control and integrate with other computer systems. This enables Executors (or users) to publish instructions and subsequently assign these instructions to other Executors.
- By chaining instructions together, it becomes possible to execute workloads that can operate seamlessly across different platforms
- A build-in zero-trust protocol makes it possible to organize remote Exectors as a single unit (a Colony). This feature empowers users to maintain control, even when distributed are deployed on multiple platforms.
Example
Start a Colonier server
source devenv
colonies dev
Submit a meta-process
{
"conditions": {
"executortype": "cli"
},
"funcname": "echo sayhello"
}
colonies function submit --spec sayhello.json
Start a Unix executor (executes functions as Unix commands)
See Unix executor repo how to install an Unix executor.
./bin/unix_executor start
INFO[0000] Lauching process Args="[]" Func="echo sayhello"
sayhello
See this guide how to implement executors in Python, Julia, Go, and JavaScript.
Dashboard screenshots
Below are some screenshots from the Colonies Dashboard:
More information
Installation
Presentations
Guides
- Introduction
- Getting started
- How to implement a Colonies executor
- How to implement a Fibonacci executor in Go
- How to create workflows DAGs
- How to use generators
- How to use crons
- How to use the Colonies CLI
- Logging
Design
Executors
SDKs
- Golang Colonies SDK
- Rust Colonies SDK
- Julia Colonies SDK
- JavaScript Colonies SDK
- Python Colonies SDK
- Haskell Colonies SDK
Deployment
More information can also be found here.
Current users
- ColonyOS is currently being used by RockSigma AB to build a compute engine for automatic seismic processing in underground mines.
Running the tests
Follow the instructions at Installation Guide then type:
make test
Directories