Configure the CIB seven EE Helm Chart

This page describes the configuration of the Helm chart cibseven-ee (version 0.0.2) for both distributions. Settings that only concern the WildFly distribution are described in Configure the WildFly Distribution.

How the Values Work

Keep your settings in one or more values files and pass them with -f. Later files override earlier ones, so you can layer an environment-specific file on top of a common one:

helm install cibseven cibseven-ee/cibseven-ee --version 0.0.2 -n cibseven \
  -f values-common.yaml -f values-production.yaml

Pass the same files again with every helm upgrade. Keep in mind how Helm merges values: maps are merged key by key, lists such as ingress.hosts or imagePullSecrets are replaced as a whole.

The chart evaluates templates in values. You can refer to other values or to the release, for example '{{ .Values.global.host }}' or '{{ .Release.Name }}'. A literal {{ in a value, for example in a configuration file, must be written as {\{.

To check the result before you install, render the chart locally. The checks of the chart run here as well, so most configuration errors show up without a cluster:

helm template cibseven cibseven-ee/cibseven-ee --version 0.0.2 -n cibseven -f my-values.yaml

Database

The chart does not install a database. Create the database and the user for CIB seven beforehand and make sure the pods can reach it. The chart defaults to PostgreSQL (jdbc:postgresql://postgres:5432/cibseven).

Run4

Set the database type of the engine and the Spring datasource together:

application:
  camunda:
    bpm:
      database:
        type: postgres
  spring:
    datasource:
      url: jdbc:postgresql://<db-host>:5432/<database>
      username: <db-user>
      password: "<database password>"
      driver-class-name: org.postgresql.Driver
      # connection pool, defaults of the chart
      hikari:
        maximumPoolSize: 20
        minimumIdle: 10

Other database types (mysql, oracle, mssql, db2) follow the same pattern: set application.camunda.bpm.database.type and point application.spring.datasource.* at the matching JDBC driver and connection string. See Connect to a Database for the requirements.

JDBC drivers: the run4 and the WildFly images contain the drivers for PostgreSQL, MySQL and H2. Drivers for Oracle, Microsoft SQL Server and DB2 are not included. For run4, put the driver JAR into /camunda/configuration/userlib, for example with your own image based on the CIB seven image (COPY ojdbc11.jar /camunda/configuration/userlib/).

For an evaluation, H2 can be used as a file-based database. The chart does not mount a persistent volume, so the data does not survive a new pod. Do not use H2 in production.

application:
  camunda:
    bpm:
      database:
        type: h2
        schema-update: "true"
  spring:
    datasource:
      url: jdbc:h2:file:/camunda/data/cibseven
      driver-class-name: org.h2.Driver
      username: ""
      password: ""

Database schema: the chart sets application.camunda.bpm.database.schema-update: "false". The engine then expects the tables to exist and does not create or change them. Create the schema beforehand as described in Install the Database Schema, which is the recommended way for production. Alternatively set schema-update: "true", then the engine creates the tables on the first start. The value must be quoted. Updates of the schema to a new CIB seven version are always done as described in the Update and Migration Guide.

WildFly

WildFly reads only application.spring.datasource.* from the application.* block. With wildfly.datasource.fromApplication: true (default) the chart passes url, username, password and driver-class-name to the image as the environment variables DB_URL, DB_USERNAME, DB_PASSWORD and DB_DRIVER and configures the ProcessEngine datasource with them.

application:
  spring:
    datasource:
      url: jdbc:postgresql://<db-host>:5432/<database>
      username: <db-user>
      driver-class-name: org.postgresql.Driver
configmaps:
  env:
    # wait for the database before WildFly starts (host:port)
    WAIT_FOR: <db-host>:5432
# password from an existing Secret instead of application.spring.datasource.password
externalEnv:
  DB_PASSWORD:
    secretKeyRef:
      name: cibseven-db
      key: password

• Supported are PostgreSQL, MySQL and H2 (org.postgresql.Driver, com.mysql.cj.jdbc.Driver, org.h2.Driver). If driver-class-name is empty, the driver is derived from the URL. Any other driver stops the installation with an error.
• Oracle, Microsoft SQL Server and DB2 need your own standalone.xml and a JDBC driver module, which is not included in the image.
• Pool settings: set DB_CONN_MAXACTIVE, DB_CONN_MINIDLE, DB_VALIDATE_ON_BORROW and DB_VALIDATION_QUERY in configmaps.env.
• For a quick demo without a database, set wildfly.datasource.fromApplication: false. The image then uses its built-in H2 database. The data does not survive a new pod.

Database schema: the engine property isAutoSchemaUpdate is true in the image, so the engine creates the tables if they do not exist. If you create the schema yourself, set wildfly.engine.properties.isAutoSchemaUpdate: "false". Updates of the schema to a new CIB seven version are done as described in the Update and Migration Guide.

Initial Administrator

Run4: define the initial administrator under application.camunda.bpm.admin-user. Otherwise the startup fails on a fresh database.

application:
  camunda:
    bpm:
      admin-user:
        id: cibsevenadmin
        password: "<admin password>"
        firstName: Admin

• id: letters and digits only, for example cibsevenadmin, not cibseven-admin.
• password: at least 10 characters, with a lowercase letter, an uppercase letter, a digit and a special character. It must not contain the user ID, the first name or the last name, see Password Policy.

Example application (run4): the run4 image deploys the example process application invoice and creates its demo users demo, john, mary and peter unless you switch it off. Do this for every installation other than a first evaluation:

application:
  camunda:
    bpm:
      run:
        example:
          enabled: false

Users and deployments that already exist in the database are not removed by this setting; delete them in the Admin application.

To take the password from an existing Secret, see Secrets. To use LDAP, configure the LDAP Identity Service of CIB seven Run under application.camunda.bpm.run.ldap.

WildFly: the image has no users. Create the first administrator with the setup page or configure LDAP and the Administrator Authorization Plugin, see Example Application and First Administrator.

License Key

The chart does not install the license key. Enter it once in the webclient as a user with system authorizations, see CIB seven License Key. Starting with CIB seven 2.2, the key is stored in the engine database and is then valid for all replicas that use this database, see Storing the License Key in the Database.

Open the webclient over HTTPS. Licensed features need a service worker in the browser, which only runs over HTTPS or on localhost, see HTTPS and Service Workers.

Webclient JWT Secret

The webclient signs its login tokens with global.authentication.jwtSecret. With composite authentication, the REST API checks these tokens with the same secret.

• It must be a base64 string of at least 155 characters. The webclient does not start with a shorter one, and the notes of the release show a warning. Generate one with openssl rand -base64 120 | tr -d '\n'.
• All replicas must use the same secret, otherwise a token issued by one pod is rejected by another. For WildFly the chart writes it to cibseven-webclient.properties (wildfly.webclientProperties, default true). Without a secret, the WildFly image generates a random one per pod. If you provide the secret through the environment variable CIBSEVEN_WEBCLIENT_AUTHENTICATION_JWTSECRET instead (from an existing Secret, below), the chart writes it into that file at every start, so all replicas use the same secret. With more than one replica or with autoscaling, the chart stops with an error if no JWT secret is configured at all.
• The secret is a credential: whoever knows it can create tokens for any user. Changing it makes all existing login tokens invalid.

For production, keep it in an existing Secret and leave global.authentication.jwtSecret empty:

kubectl create secret generic cibseven-jwt -n cibseven \
  --from-literal=jwtSecret="$(openssl rand -base64 120 | tr -d '\n')"

global:
  authentication:
    jwtSecret: ""
externalEnv:
  CIBSEVEN_WEBCLIENT_AUTHENTICATION_JWTSECRET:
    secretKeyRef:
      name: cibseven-jwt
      key: jwtSecret

Secrets

Values for Demos, Existing Secrets for Production

Passwords and keys can be given directly in the values, for example global.authentication.jwtSecret or application.spring.datasource.password. The chart then writes them into its own Secrets (<release>-cibseven-ee-env-secret and <release>-cibseven-ee-files-secret). Helm also stores all values in the release history in the namespace. This is fine for demos and tests.

For production, create the Secrets beforehand and only reference them in the values. Then no password is part of a values file or of the Helm release:

externalEnv adds environment variables that are read from existing Secrets or ConfigMaps (secretKeyRef, configMapKeyRef). For run4, Spring Boot maps environment variables to properties, so SPRING_DATASOURCE_PASSWORD sets spring.datasource.password and CAMUNDA_BPM_ADMINUSER_PASSWORD sets camunda.bpm.admin-user.password. Environment variables take precedence over the values in application.*, which you leave empty:

application:
  camunda:
    bpm:
      admin-user:
        id: cibsevenadmin
        password: ""
        firstName: Admin
  spring:
    datasource:
      url: jdbc:postgresql://<db-host>:5432/<database>
      username: <db-user>
      password: ""
externalEnv:
  SPRING_DATASOURCE_PASSWORD:
    secretKeyRef:
      name: cibseven-db
      key: password
  CAMUNDA_BPM_ADMINUSER_PASSWORD:
    secretKeyRef:
      name: cibseven-admin
      key: password
  CIBSEVEN_WEBCLIENT_AUTHENTICATION_JWTSECRET:
    secretKeyRef:
      name: cibseven-jwt
      key: jwtSecret

To mount a whole file from an existing Secret, use volumes. The following example mounts a WildFly configuration layer that contains passwords and applies it with wildfly.configLayerFiles:

volumes:
  ldap-layer:
    secret:
      secretName: cibseven-ldap-layer   # key 60-identity.yml
    mounts:
      - mountPath: /opt/custom/layers
        readOnly: true
wildfly:
  configLayerFiles:
    - /opt/custom/layers/60-identity.yml

The password of the keystore that the chart generates for TLS in the pod (certificates.keystore.password) is an exception: an init container reads it from the chart’s own Secret, so set it in the values. It only protects the keystore file inside the pod.

Fill the Secrets with Your Tools

The chart only references Kubernetes Secrets by name and key. It does not depend on a particular tool for secret management. Fill the Secrets with the tool you already use, for example:

• External Secrets Operator with AWS Secrets Manager, Azure Key Vault, Google Cloud Secret Manager or HashiCorp Vault as the backend,
• Sealed Secrets, encrypted in Git and decrypted in the cluster,
• SOPS or the helm-secrets plugin, for encrypted manifests or values files. Values decrypted at installation time end up in the chart’s Secrets and in the Helm release like plain values, so prefer encrypted Secret manifests that you apply yourself,
• HashiCorp Vault with the Vault Secrets Operator, or the Secrets Store CSI Driver with the synchronization to Kubernetes Secrets.

Use the variant of the tool that creates a Kubernetes Secret, because the settings above reference Secrets. As an example, an ExternalSecret of the External Secrets Operator that provides the Secret cibseven-db with the key password (adapt apiVersion and the store to your installation):

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: cibseven-db
  namespace: cibseven
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: <your secret store>
  target:
    name: cibseven-db
  data:
    - secretKey: password
      remoteRef:
        key: <name of the secret in the backend>
        property: password

Apply such objects before the installation, or let Helm create them together with the release through extraDeploy.

Changes of Secrets

Changes of values that the chart renders into its own ConfigMaps and Secrets restart the pods automatically with the next helm upgrade. Environment variables from Secrets that you manage yourself are read when the container starts. After you change such a Secret, restart the pods:

kubectl -n cibseven rollout restart deployment/cibseven-cibseven-ee

Exposure

Ingress

The chart can create an Ingress for the host global.host:

global:
  host: cibseven.example.org
  ingress:
    enabled: true
    className: nginx
    # annotations for all charts that share these global values
    annotations: {}
    tls:
      # kubernetes.io/tls Secret, for example issued by cert-manager
      secretName: cibseven-tls
ingress:
  # annotations of this chart, merged on top of global.ingress.annotations
  customAnnotations: {}

The paths of the Ingress depend on the distribution:

• run4: /. The REST API under /engine-rest requires authentication (application.camunda.bpm.run.auth.enabled: true, authentication: composite).
• WildFly: only /webapp. Keep it that way: browser users only need the webclient. Expose /engine-rest and the applications under /camunda only to internal clients.

To change the paths or the target port, override ingress.hosts. It is a list, so repeat the host:

ingress:
  hosts:
    - host: '{{ .Values.global.host }}'
      paths:
        - path: /webapp
          pathType: Prefix
          port: http

OpenShift Route

On OpenShift, use a Route instead of the Ingress. Set global.ingress.enabled: false and enable openshiftRoute:

global:
  ingress:
    enabled: false
openshiftRoute:
  enabled: true
  host: cibseven.apps.example.org
  path: /webapp
  tls:
    enabled: true
    termination: edge
    insecureEdgeTerminationPolicy: Redirect

This terminates TLS at the router (edge) and forwards plain HTTP to the Service port http. For TLS up to the pod, use reencrypt, see TLS. Do not use passthrough unless the whole host is internal: a passthrough Route cannot be limited to a path and exposes /engine-rest and /camunda too.

TLS

Options

Behind a TLS-terminating Ingress or Route, WildFly keeps its redirects on https:// (wildfly.proxyAddressForwarding, default true), and the production profile sends the HSTS header for requests that arrived over HTTPS, see Configure the WildFly Distribution.

TLS in the Pod

To terminate TLS in WildFly itself, provide your certificate as a kubernetes.io/tls Secret, for example issued by cert-manager or created with:

kubectl create secret tls cibseven-pod-tls -n cibseven --cert=tls.crt --key=tls.key

• tls.crt: the certificate chain.
• tls.key: a PKCS#8 key (BEGIN PRIVATE KEY, RSA or EC) or a PKCS#1 RSA key (BEGIN RSA PRIVATE KEY). Other formats, for example BEGIN EC PRIVATE KEY, are rejected. With cert-manager and ECDSA keys, set privateKey.encoding: PKCS8 in the Certificate.
• The certificate must be issued for the host name that the clients use, for example <release>-cibseven-ee.<namespace>.svc.

certificates:
  keystore:
    enabled: true
    secretName: cibseven-pod-tls
    # protects only the keystore file generated inside the pod; set your own value
    password: "<keystore password>"

An init container converts the Secret into /camunda/tls/keystore.p12 (certificates.keystore.mountPath). The chart configures the HTTPS listener of WildFly to use it, offers TLS 1.3 and 1.2 (wildfly.tls.tls13; with false only TLS 1.2) and adds the Service port https (8443). The probes stay on HTTP port 8080. With wildfly.tls.configureListener: false the chart only generates the keystore and leaves the listener unchanged.

Check the certificate:

kubectl -n cibseven port-forward svc/cibseven-cibseven-ee 8443:8443
openssl s_client -connect localhost:8443 </dev/null | openssl x509 -noout -subject

Other pods reach the application encrypted at https://<release>-cibseven-ee.<namespace>.svc:8443. They need the issuing CA in their truststore.

Let the ingress controller or router re-encrypt to port 8443:

• ingress-nginx: annotation nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" and port: https:

  ingress:
    customAnnotations:
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    hosts:
      - host: '{{ .Values.global.host }}'
        paths:
          - path: /webapp
            pathType: Prefix
            port: https
  

• AWS Load Balancer Controller: annotations alb.ingress.kubernetes.io/backend-protocol: HTTPS and alb.ingress.kubernetes.io/healthcheck-protocol: HTTPS, and port: https. For the health check, use the path of the probes of the chart: alb.ingress.kubernetes.io/healthcheck-path: /webapp/.

• OpenShift: a Route with termination: reencrypt and the CA that issued the pod certificate:

  openshiftRoute:
    enabled: true
    host: cibseven.apps.example.org
    path: /webapp
    service:
      targetPort: https
    tls:
      enabled: true
      termination: reencrypt
      insecureEdgeTerminationPolicy: Redirect
      destinationCACertificate: |
        -----BEGIN CERTIFICATE-----
        <CA that issued the certificate in certificates.keystore.secretName>
        -----END CERTIFICATE-----
  

  On OpenShift, the service CA can issue the pod certificate. The certificate covers <service>.<namespace>.svc:

  service:
    annotations:
      service.beta.openshift.io/serving-cert-secret-name: cibseven-pod-tls
  certificates:
    keystore:
      enabled: true
      secretName: cibseven-pod-tls
      password: "<keystore password>"
  

The chart configures TLS in the pod for the WildFly distribution. For run4, terminate TLS at the Ingress or Route.

Additional CA Certificates

To trust additional CAs in the application, for example the CA of an LDAPS server or of another HTTPS service, provide them in PEM format in a Secret. The file may contain several certificates.

kubectl create secret generic ldaps-ca -n cibseven --from-file=ca.pem=./ca.pem

certificates:
  truststore:
    enabled: true
    secretName: ldaps-ca
    # key in the Secret, default certificate.pem
    secretKey: ca.pem

An init container copies the Java truststore of the image, imports the certificates with the alias prefix certificates.truststore.alias (default custom-ca) and mounts the copy over the original, so all default CAs stay trusted. This works for both distributions.

Network Policy

Restrict the access to the pods with a NetworkPolicy. This is important for WildFly: it trusts the X-Forwarded-* headers of every client that reaches the pod, and the REST API /engine-rest should only be reachable for internal clients. Allow ports 8080 and 8443 only for the ingress controller or router, and for your own applications that call CIB seven inside the cluster. The WildFly management port 9990 is not part of the Service.

A NetworkPolicy only has an effect if the network plugin of the cluster enforces it. The chart does not create one by default. Add it with extraDeploy:

extraDeploy:
  - apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: '{{ .Release.Name }}-cibseven-ee-ingress'
    spec:
      podSelector:
        matchLabels:
          app.kubernetes.io/name: cibseven-ee
          app.kubernetes.io/instance: '{{ .Release.Name }}'
      policyTypes:
        - Ingress
      ingress:
        - from:
            # Kubernetes: namespace of the ingress controller
            - namespaceSelector:
                matchLabels:
                  kubernetes.io/metadata.name: ingress-nginx
            # OpenShift: namespaces of the router
            # - namespaceSelector:
            #     matchLabels:
            #       policy-group.network.openshift.io/ingress: ""
            # your applications that use the REST API
            # - namespaceSelector:
            #     matchLabels:
            #       kubernetes.io/metadata.name: <application namespace>
            #   podSelector:
            #     matchLabels:
            #       app.kubernetes.io/name: <application>
          ports:
            - port: 8080
              protocol: TCP
            - port: 8443
              protocol: TCP

Cloud load balancers in IP mode: some load balancers send the traffic directly to the IP addresses of the pods, for example the AWS Load Balancer Controller with alb.ingress.kubernetes.io/target-type: ip. This traffic does not come from a pod, so a namespaceSelector does not match it. Allow the subnets of the load balancer with ipBlock instead:

      ingress:
        - from:
            - ipBlock:
                cidr: 10.0.0.0/24   # subnet of the load balancer, zone a
            - ipBlock:
                cidr: 10.0.1.0/24   # subnet of the load balancer, zone b
          ports:
            - port: 8080
              protocol: TCP
            - port: 8443
              protocol: TCP

If you collect metrics, also allow your monitoring system on port 9464.

Additional Manifests

extraDeploy is a list of Kubernetes manifests that Helm installs, upgrades and deletes together with the release, for example a NetworkPolicy, an ExternalSecret or a cert-manager Certificate. Templates are evaluated, so you can use '{{ .Release.Name }}' or '{{ .Values.global.host }}':

extraDeploy:
  - apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: '{{ .Release.Name }}-pod-tls'
    spec:
      secretName: cibseven-pod-tls
      dnsNames:
        - '{{ .Release.Name }}-cibseven-ee.{{ .Release.Namespace }}.svc'
      privateKey:
        algorithm: RSA
        encoding: PKCS8
      issuerRef:
        kind: ClusterIssuer
        name: <your issuer>

Resources and Probes

Requests and limits are only applied with resources.enabled: true (or global.resources.enabled: true):

resources:
  enabled: true
  limits:
    cpu: 2
    memory: 2Gi
    ephemeral-storage: 2Gi
  requests:
    cpu: 500m
    memory: 1Gi
    ephemeral-storage: 1Gi

• values-example.yaml contains a starting point for run4 (requests 500m CPU, 1Gi memory, limits 2 CPU, 2Gi memory). Observe your installation and adjust the values.
• WildFly starts with -Xms512m -Xmx1024m plus up to 1024m Metaspace. Give the pod a memory limit of at least 2Gi, or set JAVA_OPTS in configmaps.env, and 1 to 2Gi ephemeral storage for the exploded deployments.
• The startup, readiness and liveness probes check /webapp/ on port http. The startup probe allows 30 attempts every 10 seconds. If the startup takes longer, for example with small CPU requests, increase probes.startupProbe.failureThreshold.

Scaling to Several Replicas

Set replicaCount, or enable a HorizontalPodAutoscaler with autoscaling.enabled, autoscaling.minReplicas, autoscaling.maxReplicas and autoscaling.targetCPUUtilizationPercentage (requires CPU requests).

• All replicas use the same database and form a cluster, see Clustering Model.
• All replicas need the same JWT secret, so that the webclient token of a user is accepted by every pod.
• The license key is stored in the database and is valid for all replicas.
• The chart creates a PodDisruptionBudget with maxUnavailable: 1. Change it with podDisruptionBudget.maxUnavailable or podDisruptionBudget.minAvailable (not both), or switch it off with podDisruptionBudget.enabled: false.
• By default the replicas are spread across nodes on a best-effort basis (global.affinity.preset: "podAntiAffinity:preferred"). Use "podAntiAffinity:required" for a hard rule. To spread them across zones, set global.topologySpreadConstraints.enabled: true (presets DoNotSchedule and ScheduleAnyway).
• nodeSelector and tolerations place the pods on particular nodes.

Further Settings

OpenShift

• The image runs as UID 1000 (global.securityContext.runAsUser). Use a Security Context Constraint that allows this UID, for example nonroot-v2 or anyuid. Grant it to a dedicated service account rather than to default:

  oc create serviceaccount cibseven -n cibseven
  oc adm policy add-scc-to-user nonroot-v2 -z cibseven -n cibseven
  

  serviceAccount:
    create: false
    name: cibseven
  

• Expose the webclient with an OpenShift Route instead of an Ingress.

• For TLS in the pod, the service CA can issue the certificate, see TLS in the Pod.

• In a NetworkPolicy, the router namespaces carry the label policy-group.network.openshift.io/ingress: "".