Comparing the top 6 .NET continuous integration build servers

Written by Kristoffer Strube

,

The bigger a project gets the longer time it will take to manually build, run unit tests, integrations tests and deploy to your staging/release environment. A solution for this is to get started using a build management system. They vary a lot, some have a lot of integrations and some are highly customizable.

Jenkins

Jenkins is a very versatile build management system. It has integration with Azure for hosting the service on their servers or you can host it yourself on almost any operative system/Docker. It is very extensible with a huge variety of plugins that can extend it's capabilities all accessible through the client. An important one if you are developing in .NET will be MSBuild plugin that automates a lot of the processes of building .NET projects. Another nice feature is that it can easily be connected with GitHub and set up credentials for this integration.

Installation

When adding a new "Item" in the Jenkins dashboard you get a few options for how you want to configure your build automation.

The easiest to use when working with .NET is either the Freestyle project or the Pipeline. The Freestyle project makes it easy to choose how you will manage your Code Source while the Pipeline is really good for visualizing and structuring your tasks especially if have a more advanced build process. This could be an example of how you can configure a pipeline through Jenkins's pipeline script.

pipeline {
    agent any 
    stages {
        stage('Clone') { 
            steps {
                bat 'git clone https://github.com/KristofferStrube/BuildTest.git'
            }
        }
        stage('Restore') { 
            steps {
                bat 'dotnet restore BuildTest/WebApplication1'
            }
        }
        stage('Build') { 
            steps {
                bat 'dotnet build BuildTest/WebApplication1'
            }
        }
        stage('Publish') { 
            steps {
                bat 'dotnet publish BuildTest/WebApplication1'
            }
        }
        stage('Clean up') { 
            steps {
                bat 'rmdir /Q /S BuildTest'
            }
        }
    }
}

and it will result in an overview that will look something like this when you push a new commit to the repository.

Running multiple tasks in a pipeline simultaneously is not possible with the default pipeline plugin, but there is, as stated previously, a great market of plugins where you can find many of the features that are not present in the default set up.

Conclusion

No direct integration with .NET Core.
No simultaneous tasks in pipeline.
No direct integration with .NET.
A great plugin market.
Markup language for pipeline.
Self-host of client and server.

GitLab

GitLab takes care of almost every part of Git repositories, delegating tasks and automatic CI/CD (Continues Integration / Continues Delivery). You can host the GitLab client yourself or just use GitLab.com. You then need to start a new repository or fetch one from another platform. The last part you need is a runner. The runner is the machine that executes the tasks you set up in your pipeline and you either use Google's Kubernetes clusters or host one yourself on a server or locally. It's both an inconvenience that the runner is not part of the client, but at the same time pretty neat, since you can host it wherever you want if you have special requirements for the environment. This enables you to use the newest builds of .Net Core or customize the process in other ways with a lot of freedom.
You can also have multiple runners at the same time with delegated tasks using tags. This makes it possible to do a lot of tasks simultaneously using multiple runners, that doesn't necessarily have the same capabilities.

Installation

To decide which scripts should be run in the pipeline you add a YAML (.yml) file in your repository and reference it in under Settings>CI/CD>General pipelines in the field Custom CI config path. The following is an example of building, testing and deploying a simple web project using a docker image with .NET.

# We fist choose the docker image to use
image : microsoft/dotnet:latest
# We then define the different stages that there will be in te pipeline.
stages:
  - build
  - test
  - deploy
# This script will be ran before every task
before_script:
  - 'dotnet restore WebApplication1'
# This is a task in the pipeline
build:
 # It will be running in the build stage, which means that the test stage can't befin before this is done
 stage: build
 # It will be executed by a runner that has the tag build
 tags:
  - build
 # It will execute this script
 script:
  - 'dotnet build WebApplication1'
 # It will only do so for the master branch
 only:
   - master
# This the test task that will be ran in the test stage possible while other test tasks run
test:
 stage: test
 ...
# Lastly we deploy
deploy_production:
 stage: deploy
 ...

Sometimes when building or publishing you need system variables. An example could be to pass a password when publishing a project using .NET. This can be handled by adding the entry CI_REGISTRY_PASSWORD to Settings>CI/CD>Variables with your password as value.

This pipeline was configured to automatically start when a new commit was pushed using the principles of CI/CD. This is enabled by default and can be changed in Settings>CI/CD>Auto DevOps to options like timed rollout or manual deployment.

Conclusion

You have to host Runners yourself.
You have to use GitLab as your Git system as well.
Markup language for pipeline.
Tag system for runners to handle multiple tasks with different requirements.
Option to use their hosted client instead of hosting yourself.

Would your users appreciate fewer errors?

➡️ Reduce errors by 90% with elmah.io error logging and uptime monitoring ⬅️

CruiseControl.Net

Cruise Control is a pretty old system, and the last release was in 2010. Even though it's old it still makes it possible to integrate with Git and still does CI/CD pretty good. It does this by having a build loop which checks your source files every so often for changes and builds if any are present. Instead of having it check source files locally it's possible to set up a SourceControl as they call it before the build state and check for new releases. Another thing it does really great is separation by having the build loop and the dashboard be different applications that talk together. This makes it easy to have the build loop on a distant server and the dashboard local for lightweight hosting on demand. There are though still obvious shortcomings like the fact that the GUI application for configuring the service is a Java Swing application, which is not easy to access remotely unlike a website.

Conclusion

It's doesn't have integration with newer Git platforms.
The Dashboard is a Java Swing application.
No direct integration with MSBuild nor .NET Core.
Dashboard and build loop separated.

TeamCity

TeamCity is JetBrains' take on a build management system and they did well. It's probably one of the easiest systems to get started with when you take into count how customizable it is. It didn't take more than 20 minutes to download and set up a simple configured build (pipeline) from an existing repository. It's two parts with an agent (server) for build management and a web client for configuring and monitoring the pipeline.

Installation

It is intelligent when configuring new projects and detects what kind of project it is and what stages there should be, like seeing a test project and making a stage for that automatically or restoring before doing the build.

Afterward, it's easy to change the order of the tasks or add other triggers where CI/CD is the default setting.

And that's pretty much it, not much extra like also insisting on being your scrum system or hosting your Git-service.

Conclusion

No hosting of Git.
Self-host of build server.
Assisted set up of the build system.
Configuration for .NET Core.

AppVeyor

AppVeyor is an all cloud solution and integrates with, GitHub, BitBucket, VSTS, GitLab or Kiln with great ease. It automatically detects the project type when importing, but does have some problems doing this at times. An example is that it detects a .NET Core application as a .NET Framework, which you will have to change manually afterward. It supports close to the newest version of .NET Core (2.1.4) without any further setup. You get a 2 weeks trial, but after that, you will have to pay 29$/month for a single project and only serialized tasks.

Installation

The pipeline is structured very rigid, with a specific order of stages. This means that you have to place a task like dotnet restore in the Build category, which makes it more ambiguous where the error belongs when looking at the console log.

These are the stages that you have as options which can either be handled automatically if you have a project that can be built with MSBuild or scripted manually if you use .NET Core.

It is none the less really easy to get started with and can be customized somewhat by using direct cmd/PowerShell commands instead of the build in options.

Conclusion

Rigid stages.
No direct integration with .NET Core.
No option to host build server yourself.
Integration with MSBuild.
All cloud hosted.

Azure DevOps

It wouldn't be a complete list of build management systems without at least having Azure DevOps (previously VSTS) included. They are a big contestant on the market and takes care of every part of a build management system. There is a couple of things you have to configure before getting started like setting up your organization, which you might not be interested in, but it gives a lot back like a test domain for your staging environment.

Installation

It's also really easy to get started with a pipeline after you have created or imported a repository.

You have about 20 preconfigured pipelines that match the structure of most .Net repositories and if you are still not happy with that you also have an option to configure the pipeline manually using a YAML file for defining stages. It's also really convenient to use the agents that Microsoft has already made available, but you can actually host the agents yourself for more configurability/freedom.

An overview of all the tasks and errors/warnings is available with great ease.

Conclusion

Long setup for the first project.
Markup language for pipeline.
Test domain.
Build for both .NET Core and MSBuild.

These are the build management systems we know of. If we missed any feel free to reach out so that we can add them to the list and get a better insight in the different systems strengths/weaknesses.

The conclusions in this post are my opinion. Some of the products are made especially for building code, while others offer a range of additional features. This post focuses on the build-part only. What product you will need for you and your team depends on your situation and requirements. Most products are pretty easy to get started with, so make sure to test them out before you pick one.