Request Logging

Advanced Request Logging Configuration

The advanced request logging configuration allows to learn more about how the custom logging is configured, and provide intuition on how it could be extended.

Make sure that before you go ahead with this installation, you have all the pre-requisites listed in the Installation Overview Page.

Overview

This page will show you how to set up request logging, which is used to log the audit of every input and ouput from the models, which then can be used for reproducibility, debugging or requesting explanations.

This page contains the instructions of how to set up your own ELK stack from scratch, as well as the instructions for how to set up the request logger with or without KNative (the KNative one is the default).

The Request Logging is based in the open source Seldon Core Request logging.

Further Pre-requisites

The default payload logging depends on KNative (which also has a dependency in Istio). You can configure these by running the install_knative.sh script in seldon-core/centralised-logging/request-logging/.

KNative and istio are not hard requirements. See Running without KNative at the bottom of this document.

Running from base repository

In order to run all relevant scripts you should make sure you are in the base folder in seldon-core/centralised-logging.

Install ELK Stack

Elasticsearch

First we install elasticsearch from the elastic repo. We use version 7.5.2 but we also support version 7.6.0:

kubectl create namespace logs || echo "namespace logs exists"
helm upgrade --install elasticsearch elasticsearch --version 7.5.2 --namespace=logs --set service.type=ClusterIP --set antiAffinity="soft" --repo https://helm.elastic.co
kubectl rollout status statefulset/elasticsearch-master -n logs

Fluentd & Kibana

For log collection from each of the Seldon deployment models, we use Fluentd to collect the logs and Kibana to visualise them.

helm upgrade --install fluentd fluentd-elasticsearch --namespace=logs -f fluentd-values.yaml --repo https://kiwigrid.github.io
helm upgrade --install kibana kibana --version 7.5.2 --namespace=logs --set service.type=ClusterIP -f ./kubeflow/kibana-values.yaml --repo https://helm.elastic.co

Istio Routes

In order to make sure these are reachable we need to apply the istio virtual service through the routes.

kubectl apply -f ./kubeflow/virtualservice-kibana.yaml
kubectl apply -f ./kubeflow/virtualservice-elasticsearch.yaml

kubectl rollout status deployment/kibana-kibana -n logs

Running with KNative

These are the instructions to set up the request logger with KNative which is the default configuration. For the instructions to install KNative with default configurations you can set it up below.

Install Request Logger

Now we set up the request logger which will run on top of Knative eventing.

This is the component that will turn on only when a request is sent to a model to collect the contents and forward the processed log into elasticsearch.

# Delete logger if existing as otherwise get 'expected exactly one, got both' err if existing resource is v1alpha1
kubectl delete -f ./request-logging/seldon-request-logger.yaml || true
kubectl apply -f ./request-logging/seldon-request-logger.yaml
# remove and recreate broker if already have one to activate eventing
kubectl delete broker -n default default || true
kubectl label namespace default knative-eventing-injection- --overwrite=true
kubectl label namespace default knative-eventing-injection=enabled --overwrite=true

Setup Trigger

Once the knative request logger is set up, now we just need to configure the trigger:

kubectl -n default get broker default

kubectl apply -f ./request-logging/trigger.yaml

Check Setup

In order to check that it has been successfully set up, the only thing you need is to send a request through Seldon Deploy and check if a request logger pod has been created.

Custom Request Logger

It’s possible for you to add your own custom request logger, with any custom loggic you’d like to add. In order to do this, you need to make sure each Seldon Deployment points to the endpoint of your custom request logger.

For this, you will need to make sure you enable the Seldon Operator Environment variable "EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT_PREFIX", which can be enabled through the helm chart.

Below we show an example of how you would do this for our non-knative default request logger

Running without KNative

It’s also possible to set up request logging without KNative dependency. For this, you will have to run a non-knative request logger, which you can trigger by running the configuration below.

Make sure that you edit the elasticsearch host variable below to point to the correct elasticsearch service address.

Important: you need to make sure that you are running a request logger in every namespace where you plan to deploy seldon core models

apiVersion: apps/v1
kind: Deployment
metadata:
  name: seldon-request-logger
  namespace: default
  labels:
    app: seldon-request-logger
spec:
  replicas: 2
  selector:
    matchLabels:
      app: seldon-request-logger
  template:
    metadata:
      labels:
        app: seldon-request-logger
    spec:    
      containers:
        - name: user-container
          image: docker.io/seldonio/seldon-request-logger:0.2.0
          imagePullPolicy: Always
          env:
            - name: ELASTICSEARCH_HOST
              value: "elasticsearch-master.logs.svc.cluster.local"
            - name: ELASTICSEARCH_PORT
              value: "9200"
---
apiVersion: v1
kind: Service
metadata:
  name: seldon-request-logger
  namespace: default
spec:
  selector:
    app: seldon-request-logger
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080 

Configure Seldon Core to use that endpoint

In order to make sure the Seldon Deployments are sending the requests to that endpoint, you will need to make sure you provide the request logger prefix. In this case you will need the following extra attributes in the Seldon Core values.yaml:

        - name: EXECUTOR_REQUEST_LOGGER_DEFAULT_ENDPOINT
          value: "http://seldon-request-logger."

It’s important that you make sure it’s on that format, which is http://<LOGGER_SERVICE>.. The Seldon Service Orchestrator will then add the namespace where the Seldon DEployment is running as a suffix.

Overriding Request Logger Endpoint for specific Seldon Deployment

Once you have created the request logger, now you have to make sure your deployments are pointing to the correct custom request logger. You can set up the custom request logger address by adding the following configuration to every Seldon Core SeldonDeployment file:

      logger:
        mode: all
        url: http://seldon-request-logger.default

The mode configuration can be set to request, response or all.

The url is where this should be logged to. There’s a similar example for KFServing.