Upgrade and Troubleshoot the CIB seven EE Helm Chart

Upgrade

Upgrade the Chart

Update the repository index, check what changes, and upgrade the release with the same values files as before:

helm repo update
helm search repo cibseven-ee/cibseven-ee --versions
# compare the rendered objects of the new version with the installed ones
helm template cibseven cibseven-ee/cibseven-ee --version <new version> -n cibseven -f my-values.yaml > new.yaml
helm -n cibseven get manifest cibseven > current.yaml
diff current.yaml new.yaml
helm upgrade cibseven cibseven-ee/cibseven-ee --version <new version> -n cibseven -f my-values.yaml

helm -n cibseven get values cibseven shows the values of the installed release. Read the chart versions below before you upgrade.

CIB seven Version

By default the image tag is <distribution>-<CIB seven version of the chart>, so a new chart version can also bring a new CIB seven version. To decide about the CIB seven version yourself, set the tag explicitly. It must match image.distribution:

image:
  distribution: wildfly
  tag: wildfly-2.2.4-ee

Before you switch to a new CIB seven version:

Update the database schema as described in the Update and Migration Guide and in Install the Database Schema. The engine does not update an existing schema by itself.
With several replicas, old and new pods run side by side during a rolling update of the Deployment. Check the Rolling Update guide whether this is possible for your versions. Otherwise stop all pods before you start the new version, for example with deploymentStrategy: {type: Recreate}.
With your own standalone.xml (WildFly), compare it with the standalone.xml of the new image and adapt it, see Your Own Complete standalone.xml.

Configuration Changes

Changes of values that the chart renders into its own ConfigMaps and Secrets restart the pods with the next helm upgrade. Secrets that you manage yourself are read when the container starts; after a change, restart the pods with kubectl -n cibseven rollout restart deployment/cibseven-cibseven-ee.

Uninstall

helm -n cibseven uninstall cibseven

This removes the objects of the release. Secrets that you created yourself (image pull secret, database password, JWT secret, certificates), the database and its data remain.

Chart Versions

Chart version	CIB seven version (default image)	Changes
`0.0.2`	`2.2.4-ee`	WildFly distribution (`image.distribution: wildfly`): datasource from `application.spring.datasource.*`; redirects stay on `https://` behind a TLS-terminating ingress controller or router; TLS in the pod with `certificates.keystore`; additional CA certificates with `certificates.truststore`; WildFly YAML configuration layers (`wildfly.configLayers`, `wildfly.configLayerFiles`) and engine properties (`wildfly.engine.properties`); production profile `wildfly.production`; REST API authentication `wildfly.restAuthentication`; `values-production-wildfly.yaml`; no example application by default; the JWT secret is shared by all replicas; the Ingress routes only `/webapp`. All distributions: the startup probe allows 30 attempts (was 12).
`0.0.1`	`2.2.4-ee`	First version, run4 distribution.

From 0.0.1 to 0.0.2, a run4 installation needs no changes of its values.

Troubleshooting

Installation Fails with an Error

The chart checks the values while it renders the objects. helm install, helm upgrade and helm template then stop with one of these messages:

Message (shortened)	Cause and solution
`WildFly: application.spring.datasource.driver-class-name "…" is not supported by the image's DB_* mechanism`	WildFly supports PostgreSQL, MySQL and H2 from the values. Use your own `standalone.xml` for other databases, or set `wildfly.datasource.fromApplication: false` for the built-in H2 database.
`certificates.keystore.enabled=true requires certificates.keystore.secretName`	Set the name of an existing `kubernetes.io/tls` Secret.
`wildfly.configLayers.<name> sets process-engines.default.properties`	Remove `properties` from the layer and set single keys in `wildfly.engine.properties`, see Engine Properties.
`wildfly.configLayers: invalid name`	Layer names consist of letters, digits, `.`, `_` and `-`.
`wildfly.restAuthentication must be composite or pseudo`	Use one of these values, or leave it empty.

The notes printed after the installation (helm -n cibseven status cibseven) warn if global.authentication.jwtSecret is shorter than 155 characters or not base64.

Pod Does Not Start

Look at the events and the logs of the pod, including the init containers generate-keystore and import-truststore-certs:

kubectl -n cibseven get pods
kubectl -n cibseven describe pod <pod>
kubectl -n cibseven logs <pod> --all-containers
kubectl -n cibseven logs <pod> --previous

For WildFly, the chart writes its own messages to the startup log. They all start with cibseven-ee chart::

kubectl -n cibseven logs deployment/cibseven-cibseven-ee | grep 'cibseven-ee chart:'

Symptom or log line	Cause and solution
`ImagePullBackOff`, `ErrImagePull`	Set `image.repository: artifacts.cibseven.org/docker-ee`, create the image pull secret in the namespace of the release and reference it in `imagePullSecrets`. Check the credentials and the tag (`<distribution>-<version>-ee`).
`CreateContainerConfigError`	A Secret or a key referenced in `externalEnv` does not exist (yet).
Pod rejected on OpenShift (security context constraints)	The image runs as UID 1000. Grant an SCC such as `nonroot-v2` or `anyuid` to the service account, see OpenShift.
`cibseven-ee chart: WildFly configuration layers …`	Not an error: lists the active configuration layers.
`cibseven-ee chart: configuration layer … not found`	A file in `wildfly.configLayerFiles` is not mounted at this path.
Error in a configuration layer, the container stops	An invalid layer or an unknown resource stops WildFly. Check the YAML and the resource names of the layer in the log.
`cibseven-ee chart: certificates.keystore is enabled but /camunda/tls/keystore.p12 is missing or empty`	The init container `generate-keystore` could not write the keystore. Check its log, for example `Unsupported private key format, use PKCS#8 or PKCS#1 RSA`: convert the key with `openssl pkcs8 -topk8 -nocrypt -in tls.key -out tls-pkcs8.key`, or set `privateKey.encoding: PKCS8` in the cert-manager `Certificate`.
`cibseven-ee chart: REST authentication: …`	The chart could not prepare the engine-rest WAR for `composite`. Check that the image is a CIB seven WildFly image. As a workaround, `wildfly.restAuthentication: pseudo` (keep `/engine-rest` internal).
`cibseven-ee chart: REST API authentication composite (…)`	Not an error: the REST API requires credentials.
Startup probe fails, pod restarts during startup	The startup takes longer than 30 × 10 seconds, for example with small CPU requests. Increase the CPU request or `probes.startupProbe.failureThreshold`.
`OOMKilled`	WildFly needs a memory limit of at least 2Gi, or smaller heap settings in `JAVA_OPTS` (`configmaps.env`).
run4: startup fails on a fresh database	The initial administrator in `application.camunda.bpm.admin-user` is missing or invalid (ID with letters and digits only, password policy), or the tables do not exist while `schema-update` is `"false"`, see Database.
Webclient does not start	`global.authentication.jwtSecret` is shorter than 155 characters.

Requests to the REST API Return 401

With composite authentication (run4 default; WildFly with the production profile), /engine-rest requires credentials:

Send HTTP Basic credentials of an engine user (curl -u <user>:<password> …). The Context-User-ID header alone is not accepted.
Wrong passwords lock the user for a few seconds (login throttling); wait before you try again.
Grant the user the authorizations it needs; authorization is enabled.
Requests without an Authorization header log the warning No authHeader provided!.

See REST API Authentication.

Further Problems

Symptom	Cause and solution
Users are logged out or get errors when a request reaches another replica	The replicas use different JWT secrets. Set the same secret for all replicas, see Webclient JWT Secret.
Licensed features show “License Required”	The webclient is not opened over HTTPS, or no license key is entered, see License Key.
WildFly redirects `/webapp` to `http://`	Keep `wildfly.proxyAddressForwarding: true` and let the ingress controller or router send `X-Forwarded-Proto`.
No access through the Ingress after adding a NetworkPolicy	The policy does not match the source of the traffic: check the namespace label of the ingress controller, the router label on OpenShift, or allow the subnets of a load balancer in IP mode with `ipBlock`, see Network Policy.
`/engine-rest` or `/camunda` cannot be reached through the Ingress (WildFly)	Intended: by default only `/webapp` is exposed. Call the REST API from inside the cluster.