VCS GitOps

Advanced GitOps Installation

This page provides a detailed overview of how to set up the GitOps functionality for Seldon Deploy.

GitOps functionality for Seldon Deploy is current

Pre-requisites

This document assumes that you have Seldon Deploy installed in your cluster (without GitOps) and that all pre-requisites of the initial install are satisfied.

High level overview of required steps

If GitOps with Bitbucket is to be configured manually, this is the list of required actions:

  1. Create GitOps repository and initialize its initial directories
  2. Create App Password/Token to be used by Seldon Deploy and ArgoCD
  3. Annotate and label GitOps-enabled Kubernetes namespaces
  4. Install and configure ArgoCD in Kubernetes cluster
  5. Configure Bitbucket or Github webhooks for ArgoCD and Seldon Deploy endpoints

Make sure you follow the relevant steps based on whether you are making this configuration for Github or Bitbucket.

If all of these steps are aligned with your required configuration you can just run the installer from the file instead.

Detailed instructions

Below you will find detailed instructions on how to manually enable GitOps with Github or Bitbucket. As part of initial setup we source dockercreds.txt file.

DOCKERCREDS_FILE=${HOME}/.config/seldon/seldon-deploy/dockercreds.txt
source ${DOCKERCREDS_FILE}

Note that if you modify dockercreds.txt file you need to source it again as well as rerun

cd seldon-deploy-install
./sd-setup/sd-install

installation command for the new config to be available on the cluster.

We will also need to be inseldon-setup-install directory and prepare tempresources directory

cd seldon-deploy-install
TEMPRESOURCES=./sd-setup/tempresources
mkdir -p ${TEMPRESOURCES}

1. Creation of GitOps repository

We start with creating a GitOps repository on bitbucket or github. In this example this it is named seldon-gitops:

REPONAME=seldon-gitops

Once we create the repository we need to add folders that will store SeldonDeployment(s) definitions. For example, to enable GitOps for default and kubeflow namespace in the Kubernetes cluster we need a following directories structure:

├── default
│   └── README.md
├── kubeflow
│   └── README.md
└── README.md

2. Create App Password/ Personal Access Token

Once we have seldon-gitops repository we need to create an App password to access it programmatically.

Once it’s created, we add newly created token to dockercreds.txt as GIT_TOKEN and we make sure that GIT_USER is set accordingly. Note that if you setup these variables after installing Seldon Deploy you need to run sd-install again as explained above.

Github Personal Access Token

For Github we should have a personal access token which prodides the following permissions:

  • Repo, Repo:status, repo_deployment, public_repo and repo:invite
Bitbucket App Password

The App password will allow Seldon Deploy and ArgoCD access it over HTTPS. To create App password we go to Bitbucket profile page and then select Settings -> App passwords -> Create app password. We will need following permissions

scopes

3. Annotate and label Kubernetes namepsaces

In order to have GitOps enabled, Seldon Deploy require annotation on Kubernetes namespaces.

First, let’s create environmental variable for repository’s url (we will also need it later).

The GIT_REPO_URL will have to be either github or bitbucket, as below.

If you are running an on-prem setup, you will also be able to provide your on-prem hosted URL (and need to make sure it’s accessible)

BITBUCKET
GIT_REPO_URL=bitbucket.org/${GIT_USER}/${REPONAME}
GITHUB
GIT_REPO_URL=github.com/${GIT_USER}/${REPONAME}

Now, for each $namespace you want to enable gitops we add proper labels and annotations

namespace=default
kubectl create ns $namespace --dry-run -o json | kubectl apply -f -

kubectl label namespace $namespace seldon.gitops=enabled --overwrite=true
kubectl annotate namespace $namespace git-repo="https://${GIT_REPO_URL}" --overwrite=true

4. Install ArgoCD

ArgoCD is a key element responsible for GitOps operations. We will need argocd command line tool (CLI) that can be obtained following official documentation.

To install ArgoCD in the Kubernetes cluster we execute

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v1.4.2/manifests/install.yaml
kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'

We can verify completion of rollout with

kubectl rollout status deployment/argocd-application-controller -n argocd
kubectl rollout status deployment/argocd-repo-server -n argocd
kubectl rollout status deployment/argocd-server -n argocd
kubectl rollout status deployment/argocd-redis -n argocd
kubectl rollout status deployment/argocd-dex-server -n argocd

Setup ArgoCD password

ARGOCDINITIALPASS=12341234
kubectl patch secret -n argocd argocd-secret -p '{"stringData": { "admin.password": "'$(htpasswd -bnBC 10 "" ${ARGOCDINITIALPASS} | tr -d ':\n')'"}}'

Login to ArgoCD

ARGOCDURL=$(./sd-setup/show-argocd-url | sed 's|https://||g' | sed 's|/||g')
argocd login ${ARGOCDURL} --username admin --password ${ARGOCDINITIALPASS} --insecure

Add repo to argocd

argocd repo add "https://${GIT_REPO_URL}" \
    --username ${GIT_USER} \
    --password ${GIT_TOKEN} \
    --upsert

Configure seldon project and required permissions

argocd proj create seldon -s "https://${GIT_REPO_URL}" 2> /dev/null || true

argocd proj allow-cluster-resource seldon '*' '*'
argocd proj allow-namespace-resource seldon '*' '*'

argocd proj role create seldon seldon-admin 2> /dev/null || true
argocd proj role add-policy seldon seldon-admin --action get --permission allow --object '*' 2> /dev/null || true
argocd proj role add-policy seldon seldon-admin --action create --permission allow --object '*' 2> /dev/null || true
argocd proj role add-policy seldon seldon-admin --action update --permission allow --object '*' 2> /dev/null || true
argocd proj role add-policy seldon seldon-admin --action delete --permission allow --object '*' 2> /dev/null || true
argocd proj role add-policy seldon seldon-admin --action sync --permission allow --object '*' 2> /dev/null || true
JWT=$(argocd proj role create-token seldon seldon-admin)
echo -n $JWT > ./jwt-argocd.txt
kubectl create secret generic jwt-argocd --from-file=./jwt-argocd.txt --dry-run -o yaml | kubectl apply -f -

Finally, for each $namespace you want to enable GitOps we need to create ArgoCD app:

namespace=default
appname="${REPONAME}"-"${namespace}"
argocd proj add-destination seldon https://kubernetes.default.svc $namespace 2> /dev/null || true
argocd app delete $appname 2> /dev/null || true
argocd app create $appname \
    --repo  "https://${GIT_REPO_URL}" \
    --path $namespace \
    --dest-namespace $namespace \
    --dest-server https://kubernetes.default.svc \
    --sync-policy automated \
    --project seldon \
    --directory-recurse \
    --upsert
echo "Argocd app created for $REPONAME-$namespace"

5. Configure Webhooks

Here we show how it’s possible to programmatically set up the webhooks programmatically, however you can also set them up manually from the repo.

Note: In case webhooks are not arriving at its destination it is helpful to enable webhook’s history.

It may be necessary to also select Skip certificate verification option.

Github

First we need to configure the webhook that will be sending updates to the Seldon Deploy endpoint:

whIp=$(kubectl get service seldon-deploy-webhook -o jsonpath='{.status.loadBalancer.ingress[*].ip}')
curl -u ${GIT_USER}:${GIT_TOKEN} -v -H "Content-Type: application/json" -X POST -d "{\"name\": \"web\", \"active\": true, \"events\": [\"*\"], \"config\": {\"url\": \"http://${whIp}/api/git-webhook\", \"content_type\": \"json\" }}" https://${GIT_API}/repos/${GIT_USER}/${REPONAME}/hooks

Then we set up the webhook that will be sending the updates to ArgoCD.

curl -u ${GIT_USER}:${GIT_TOKEN} -v -H "Content-Type: application/json" -X POST -d "{\"name\": \"web\", \"active\": true, \"events\": [\"*\"], \"config\": {\"url\": \"https://${ARGOCDURL}/api/webhook\", \"content_type\": \"json\", \"secret\": \"${ARGOCDINITIALPASS}\", \"insecure_ssl\": \"1\" }}" https://${GIT_API}/repos/${GIT_USER}/${REPONAME}/hooks
Bitbucket

First we need to configure the webhook that will be sending updates to the Seldon Deploy endpoint:

whIp=$(kubectl get service seldon-deploy-webhook -o jsonpath='{.status.loadBalancer.ingress[*].ip}')
curl -v -u ${GIT_USER}:${GIT_TOKEN} -v -H "Content-Type: application/json" -d '
{
    "description": "web",
    "events": ["repo:push"],
    "url": "'"http://${whIp}/api/git-webhook"'",
    "active": true
}' https://${GIT_API}/2.0/repositories/${GIT_USER}/${REPONAME}/hooks

Then we set up the webhook that will be sending the updates to ArgoCD.

curl -v -u ${GIT_USER}:${GIT_TOKEN} -v -H "Content-Type: application/json" -d '
{
    "description": "web",
    "events": ["repo:push"],
    "url": "'"https://${ARGOCDURL}/api/webhook"'",
    "active": true
}' https://${GIT_API}/2.0/repositories/${GIT_USER}/${REPONAME}/hooks
Last modified March 9, 2020: Removed references to request logging from gitops docs (45629df)