Creating CI / CD infrastructure for open-source projects
My name is Denis Soldatov and I work as a DevOps at Parity Technologies. Our team works on clients for the Ethereum, Bitcoin and Polkadot platforms. The main programming language is RUST. We adhere to the same release policy as the RUST programming language team - once every 6 weeks and covering three branches: stable, beta and nightly.
Within RUST, cross-platform support comes as standard. We needed the high speed of assembly and testing, support for distributed teams, openness of process, and the convenience of control during all stages. Nearly two years ago, when we began to work on our project, Travis CI was sufficient for us.
While developing the code we came across our first problems – the tests were not carried out because of time constraints and the assembly of the code under MacOS was interrupted on timeout. At the same time there was a need for the assembly of a client for Windows using Visual Studio.
There was an attempt to use Travis Enterprise, but have you seen their prices? Here we had to pay not only for the hardware, but also for the license for each user, and these already cost at least several thousand dollars per year. You also had to purchase different hardware and software for Windows and MacOS.
We have made the project open-source and placed it under a GPLv3 license. We faced one more problem — when we were assembling the binary files for Windows, we could only use a licensed OS and Visual Studio, and for MacOS, Apple hardware as well. Yes, that’s right, Licenses and once again Licenses!
We began to look for other CI opportunities for RUST, but found nothing specific – there was no support for RUST, or there was a lot of code to write from scratch or, again, it was really slow and resulted in an assembly timeout. Well and of course — it was expensive. And so we chose Gitlab CI.
After careful analysis, Gitlab turned out to be very multi-functional, from the shell to the Docker and the Kubernates. We needed RUST and JavaScript (yes, how could we not use it, it is our frontend) and to not have assembly and time limits.
The result was a configured and selfhosted Gitlab. Our project is hosted on github. We had to write the scripts and to configure the synchronization with github and PR processing.
Now came the time to configure the runners for preliminary testing, assembly of binary files, Push Release and deploy of ready packets in a repository, S3 or just to download archive. It did not take us a lot of time.
The Syntax and logic is in many respects similar to travis.yaml. We watch assembly steps in one window, we manage runners in another, we do changes in the third. Everything is evident and easily configured. Of course there are webhooks and notifications.
In the case of cross-compilation under Linux-based systems, no special problems arose, we prepared the docker, we selected the OS, we set the dependencies, we set the necessary variables – the compiler and the paths to the headings and libraries.
From the standard solutions, we select the ones we need: triplets under architecture and OS. We have lots of choices from simple i386 to android and ios.
We place the images on docker hub (if you wish), or into its own registry. But on Windows and MacOS machines we have to install native RUST and nodejs assemblies and we start runner in shell mode.
Afterwards, we specify to the runners what images and what tags we use during each assembly step and we start our pipeline, which visually displays the current queue, and it is easy for us to manage, start, stop, cancel it or do any other action we want.
The processing of each task is presented in detail and we can observe what went wrong, when it went wrong, why it went wrong and who assumed that something went wrong!
The assembly process has started, and it uses a lot of CPU kernels and memory for a quicker assembly. For these purposes we rent several dedicated servers with ssd, which results in some small hardware expenses.
For Windows and Mac we found servers at a reasonable price (less than $100 a month), we had to install the necessary tools and packets ourselves. The runners there work in shell mode.
The runners are configured in such a way that they can be divided between different projects, to divide according to the tags. Here everything depends on what stages are necessary for you in the current project and afterwards, everything will depend only on your imagination and opportunities.
So we have achieved the object set before us, our assembly infrastructure expenses are about $500 a month, we have forgot about timeouts and slow assembly. Our client is available to all the main platforms, the team is hard at work and we aren't infringing on anyone's license!