Summary of the GitOps Workshop
On April 27th we had our Kubernetes GitOps Workshop at CERN, with 130 people attending both in-person and virtually. Slides as well as the full recording of the event are available.
Starting from the end here’s a list of the expected follow up actions:
- Agree on a recommendation for managing secrets and other sensitive data. Options presented included SealedSecrets, Mozilla SOPS with different plugins (Barbican, Age, …) and others. In general the audience agreed a central Vault offering would be the key to improve integration with other tools
- Group together the different teams relying on Flux and ArgoCD. The goal is not to recommend one over the other but to document differences and come up with documentation and best practices for each of the solutions
Some details below from the different sessions in the workshop.
Kicking off the day we had Benjamin Bergia from RCS-SIS presenting their integration with GitHub Actions triggering cluster updates with ArgoCD and Kustomize, along with their repository structure and workflows.
He highlighted the speed of getting things done, and the challenge of choosing the right directory structure for your own needs and some release compatibility issues with upstream Kustomize.
Next we heard from Arthur Outhenin-Chalandre from the CEPH team describing his home setup with k3s and a variety of useful CNCF projects providing networking, logging and monitoring. Plus helm libraries and charts for fun and profit…
Luca Bello from the MONIT team went through their setup with a very well established stack relying on Flux2, Helm3, GitLab CI and SOPS handling sensitive data in git.
He also provided details on how they split clusters and manage different environments in a single cluster, which add-ons they install everywhere and the workflows and release process. The SOPS usage in particular triggered several questions from the audience.
After a short break we heard from Radu Carpa from the ATLAS Rucio team describing their migration from Puppet to Kubernetes, and some interesting details for cluster bootstrapping using terraform - and how that is one of the current pain points. Radu live demoed their production repositories, and presented a now familiar stack with Flux2, SOPS and Kustomize. Discussion went around terraform usage, private key handling with SOPS and automation of AlertManager configurations.
Jack and Konstantinos from the OpenShift team were up next presenting the usage of GitOps in the OpenShift deployments, and more specifically for the Drupal deployments at CERN.
They explained how they achieve continuous delivery with a custom resource definition (CRD) and a corresponding operator, all managed via GitOps. They also highlighted the wish for a central Vault deployment to ease the management of sensitive data.
Next up we had Antonio and Ioannis from the JEEDY team describing how GitOps replaced their previous setup with Rundeck managing remote deployments, improving traceability, versioning and the ability to rollback changes. They demoed how changes in a Git repo are quickly propagated to the different clusters, and as for next steps from their point of view these would include: improved secret management, cluster management also using ArgoCD, notifications and extending usage to other applications.
The final session of the day covered the multiple options to manage secrets, and it worked as a good kickstart for the follow up discussion. Highlights include (but do check the recording):
- What’s the best way to handle secrets and other sensitive data? Lots of different options were presented, with SOPS being the most popular. But even then SOPS is the library for managing secrets, not the secret store itself. Focusing on getting a central Vault offering available from CERN IT, and some best practices on how to integrate it with Kubernetes should be a priority
- Is Terraform enough of a pain point to invest in something like Crossplane? Some work expected in the next couple months to add an OpenStack provider to Crossplane
- Should people use Flux or ArgoCD? The presentations showed that users of both tools are happy with their choice, and hinted that the way forward might be to strengthen the user communities behind them to ensure proper documentation and best practices. Future will tell if one ends up taking over the majority of deployments
- What is the best communication channel to discuss these and similar issues? The consensus was that the Cloud Containers channel on mattermost is currently the best place
And that was it. A morning packed with content and several actions to be followed up.
See you next time!