Spring Cloud Data Flow packaged by Bitnami¶

Spring Cloud Data Flow is a microservices-based toolkit for building streaming and batch data processing pipelines in Cloud Foundry and Kubernetes.

Overview of Spring Cloud Data Flow

TL;DR¶

使用加速地址添加仓库:

helm repo add bitnami "https://helm-charts.itboon.top/bitnami" --force-update
helm update bitnami

helm install my-release bitnami/spring-cloud-dataflow

Introduction¶

This chart bootstraps a Spring Cloud Data Flow deployment on a Kubernetes cluster using the Helm package manager.

Bitnami charts can be used with Kubeapps for deployment and management of Helm Charts in clusters.

Looking to use Spring Cloud Data Flow in production? Try VMware Application Catalog, the enterprise edition of Bitnami Application Catalog.

Prerequisites¶

• Kubernetes 1.19+
• Helm 3.2.0+
• PV provisioner support in the underlying infrastructure

Installing the Chart¶

To install the chart with the release name my-release:

helm install my-release bitnami/spring-cloud-dataflow

These commands deploy Spring Cloud Data Flow on the Kubernetes cluster with the default configuration. The parameters section lists the parameters that can be configured during installation.

Tip: List all releases using helm list

Uninstalling the Chart¶

To uninstall/delete the my-release chart:

helm uninstall my-release

Parameters¶

Global parameters¶

Common parameters¶

Dataflow Server parameters¶

Dataflow Skipper parameters¶

Deployer parameters¶

RBAC parameters¶

Metrics parameters¶

Init Container parameters¶

Database parameters¶

RabbitMQ chart parameters¶

Kafka chart parameters¶

Specify each parameter using the --set key=value[,key=value] argument to helm install. For example,

helm install my-release --set server.replicaCount=2 bitnami/spring-cloud-dataflow

The above command installs Spring Cloud Data Flow chart with 2 Dataflow server replicas.

Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,

helm install my-release -f values.yaml bitnami/spring-cloud-dataflow

Tip: You can use the default values.yaml

Configuration and installation details¶

Rolling VS Immutable tags¶

It is strongly recommended to use immutable tags in a production environment. This ensures your deployment does not change automatically if the same tag is updated with a different image.

Bitnami will release a new chart updating its containers if a new version of the main container, significant changes, or critical vulnerabilities exist.

Features¶

If you only need to deploy tasks and schedules, streaming and Skipper can be disabled:

server.configuration.batchEnabled=true
server.configuration.streamingEnabled=false
skipper.enabled=false
rabbitmq.enabled=false

If you only need to deploy streams, tasks and schedules can be disabled:

server.configuration.batchEnabled=false
server.configuration.streamingEnabled=true
skipper.enabled=true
rabbitmq.enabled=true

NOTE: Both server.configuration.batchEnabled and server.configuration.streamingEnabled should not be set to false at the same time.

Messaging solutions¶

There are two supported messaging solutions in this chart:

• RabbitMQ (default)
• Kafka

To change the messaging layer to Kafka, use the following parameters:

rabbitmq.enabled=false
kafka.enabled=true

Only one messaging layer can be used at a given time.

Using an external database¶

Sometimes you may want to have Spring Cloud components connect to an external database rather than installing one inside your cluster, e.g. to use a managed database service, or use a single database server for all your applications. To do this, the chart allows you to specify credentials for an external database under the externalDatabase parameter. You should also disable the MariaDB installation with the mariadb.enabled option. For example with the following parameters:

mariadb.enabled=false
externalDatabase.scheme=mariadb
externalDatabase.host=myexternalhost
externalDatabase.port=3306
externalDatabase.password=mypassword
externalDatabase.dataflow.user=mydataflowuser
externalDatabase.dataflow.database=mydataflowdatabase
externalDatabase.skipper.user=myskipperuser
externalDatabase.skipper.database=myskipperdatabase

NOTE: When using the individual properties (scheme, host, port, database, an optional jdbcParameters) this chart will format the JDBC URL as jdbc:{scheme}://{host}:{port}/{database}{jdbcParameters}. The URL format follows that of the MariaDB database drive but may not work for other database vendors.

To use an alternate database vendor (other than MariaDB) you can use the externalDatabase.dataflow.url and externalDatabase.skipper.url properties to provide the JDBC URLs for the dataflow server and skipper respectively. If these properties are defined, they will take precedence over the individual attributes. As an example of configuring an external MS SQL Server database:

mariadb.enabled=false
externalDatabase.password=mypassword
externalDatabase.dataflow.url=jdbc:sqlserver://mssql-server:1433
externalDatabase.dataflow.user=mydataflowuser
externalDatabase.skipper.url=jdbc:sqlserver://mssql-server:1433
externalDatabase.skipper.user=myskipperuser
externalDatabase.hibernateDialect=org.hibernate.dialect.SQLServer2012Dialect

NOTE: If you disable MariaDB per above you MUST supply values for the externalDatabase connection.

Adding extra flags¶

In case you want to add extra environment variables to any Spring Cloud component, you can use XXX.extraEnvs parameter(s), where XXX is placeholder you need to replace with the actual component(s). For instance, to add extra flags to Spring Cloud Data Flow, use:

server:
  extraEnvs:
    - name: FOO
      value: BAR

Using custom Dataflow configuration¶

This helm chart supports using custom configuration for Dataflow server.

You can specify the configuration for Dataflow server by setting the server.existingConfigmap parameter to an external ConfigMap with the configuration file.

Using custom Skipper configuration¶

This helm chart supports using custom configuration for Skipper server.

You can specify the configuration for Skipper server by setting the skipper.existingConfigmap parameter to an external ConfigMap with the configuration file.

Sidecars and Init Containers¶

If you need additional containers to run within the same pod as Dataflow or Skipper components (e.g. an additional metrics or logging exporter), you can do so via the XXX.sidecars parameter(s), where XXX is the placeholder you need to replace with the actual component(s). Simply define your container according to the Kubernetes container spec.

server:
  sidecars:
    - name: your-image-name
      image: your-image
      imagePullPolicy: Always
      ports:
        - name: portname
          containerPort: 1234

Similarly, you can add extra init containers using the XXX.initContainers parameter(s).

server:
  initContainers:
    - name: your-image-name
      image: your-image
      imagePullPolicy: Always
      ports:
        - name: portname
          containerPort: 1234

Ingress¶

This chart provides support for ingress resources. If you have an ingress controller installed on your cluster, such as nginx-ingress or traefik you can utilize the ingress controller to serve your Spring Cloud Data Flow server.

To enable ingress integration, please set server.ingress.enabled to true

Hosts¶

Most likely you will only want to have one hostname that maps to this Spring Cloud Data Flow installation. If that's your case, the property server.ingress.hostname will set it. However, it is possible to have more than one host. To facilitate this, the server.ingress.extraHosts object is can be specified as an array. You can also use server.ingress.extraTLS to add the TLS configuration for extra hosts.

For each host indicated at server.ingress.extraHosts, please indicate a name, path, and any annotations that you may want the ingress controller to know about.

For annotations, please see this document. Not all annotations are supported by all ingress controllers, but this document does a good job of indicating which annotation is supported by many popular ingress controllers.

TLS¶

This chart will facilitate the creation of TLS secrets for use with the ingress controller, however, this is not required. There are four common use cases:

• Helm generates/manages certificate secrets based on the parameters.
• User generates/manages certificates separately.
• Helm creates self-signed certificates and generates/manages certificate secrets.
• An additional tool (like cert-manager) manages the secrets for the application. In the first two cases, it's needed a certificate and a key. We would expect them to look like this:
• certificate files should look like (and there can be more than one certificate if there is a certificate chain)

-----BEGIN CERTIFICATE-----
MIID6TCCAtGgAwIBAgIJAIaCwivkeB5EMA0GCSqGSIb3DQEBCwUAMFYxCzAJBgNV
...
jScrvkiBO65F46KioCL9h5tDvomdU1aqpI/CBzhvZn1c0ZTf87tGQR8NK7v7
-----END CERTIFICATE-----

• keys should look like:

-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEAvLYcyu8f3skuRyUgeeNpeDvYBCDcgq+LsWap6zbX5f8oLqp4
...
wrj2wDbCDCFmfqnSJ+dKI3vFLlEz44sAV8jX/kd4Y6ZTQhlLbYc=
-----END RSA PRIVATE KEY-----

• If you are going to use Helm to manage the certificates based on the parameters, please copy these values into the certificate and key values for a given server.ingress.secrets entry.
• In case you are going to manage TLS secrets separately, please know that you must create a TLS secret with name INGRESS_HOSTNAME-tls (where INGRESS_HOSTNAME is a placeholder to be replaced with the hostname you set using the server.ingress.hostname parameter).
• To use self-signed certificates created by Helm, set server.ingress.tls to true and server.ingress.certManager to false.
• If your cluster has a cert-manager add-on to automate the management and issuance of TLS certificates, set server.ingress.certManager boolean to true to enable the corresponding annotations for cert-manager.

Setting Pod's affinity¶

This chart allows you to set your custom affinity using the XXX.affinity parameter(s). Find more information about Pod's affinity in the kubernetes documentation.

As an alternative, you can use the preset configurations for pod affinity, pod anti-affinity, and node affinity available at the bitnami/common chart. To do so, set the XXX.podAffinityPreset, XXX.podAntiAffinityPreset, or XXX.nodeAffinityPreset parameters.

Troubleshooting¶

Find more information about how to deal with common errors related to Bitnami Helm charts in this troubleshooting guide.

Upgrading¶

If you enabled RabbitMQ chart to be used as the messaging solution for Skipper to manage streaming content, then it's necessary to set the rabbitmq.auth.password and rabbitmq.auth.erlangCookie parameters when upgrading for readiness/liveness probes to work properly. Inspect the RabbitMQ secret to obtain the password and the Erlang cookie, then you can upgrade your chart using the command below:

To 18.0.0¶

This major updates the Kafka subchart to its newest major, 22.0.0. This new version of Kafka uses Kraft by default. For more information on this subchart's major, please refer to Kafka upgrade notes.

To 17.0.0¶

This major release bumps the MariaDB version to 10.11. Follow the upstream instructions for upgrading from MariaDB 10.6 to 10.11. No major issues are expected during the upgrade.

To 16.0.0¶

This major updates the Kafka subchart to its newest major, 21.0.0. Here you can find more information about the changes introduced in that version.

To 15.0.0¶

This major updates the Kafka subchart to its newest major, 20.0.0. Here you can find more information about the changes introduced in that version.

To 14.0.0¶

This major updates the RabbitMQ subchart to its newest major, 11.0.0. Here you can find more information about the changes introduced in that version.

To 13.0.0¶

This major updates the Kafka subchart to it newest major, 19.0.0. For more information on this subchart's major, please refer to Kafka upgrade notes.

To 12.0.0¶

This major updates the Kafka subchart to its newest major, 18.0.0. No major issues are expected during the upgrade.

To 11.0.0¶

This chart bumps the RabbitMQ version to 3.10.x. No issues are expected during the upgrade.

To 10.0.0¶

This major updates the Kafka subchart to its newest major, 17.0.0. No major issues are expected during the upgrade.

To 9.0.0¶

This major updates the RabbitMQ subchart to its newest major, 9.0.0. Here you can find more information about the changes introduced in that version.

To 8.0.0¶

This major release bumps the MariaDB version to 10.6. Follow the upstream instructions for upgrading from MariaDB 10.5 to 10.6. No major issues are expected during the upgrade.

To 7.0.0¶

This major updates the Kafka subchart to its newest major, 16.0.0. Here you can find more information about the changes introduced in that version.

To 6.0.0¶

This major release updates the Kafka subchart to its newest major 15.x.x, which contains several changes in the supported values and bumps Kafka major version to 3.x series (check the upgrade notes to obtain more information).

To upgrade to 6.0.0 from 5.x using Kafka as messaging solution, it should be done by maintaining the Kafka 2.x series. To do so, follow the instructions below (the following example assumes that the release name is scdf and the release namespace default):

1. Obtain the credentials on your current release:

   export MARIADB_ROOT_PASSWORD=$(kubectl get secret --namespace default scdf-mariadb -o jsonpath="{.data.mariadb-root-password}" | base64 -d)
   export MARIADB_PASSWORD=$(kubectl get secret --namespace default scdf-mariadb -o jsonpath="{.data.mariadb-password}" | base64 -d)
   

2. Upgrade your release using the same Kafka version:

   $ export CURRENT_KAFKA_VERSION=$(kubectl exec scdf-kafka-0 -- bash -c 'echo $BITNAMI_IMAGE_VERSION')
   helm upgrade scdf bitnami/spring-cloud-dataflow \
     --set rabbitmq.enabled=false \
     --set kafka.enabled=true \
     --set kafka.image.tag=$CURRENT_KAFKA_VERSION \
     --set mariadb.auth.password=$MARIADB_PASSWORD \
     --set mariadb.auth.rootPassword=$MARIADB_ROOT_PASSWORD
   

To 5.0.0¶

This major release renames several values in this chart and adds missing features, in order to be inline with the rest of assets in the Bitnami charts repository.

Affected values:

• serviceMonitor.extraLabels renamed as serviceMonitor.labels.
• server.strategyType and skikker.strategyType have been replaced by server.updateStrategy and skikker.updateStrategy respectively. While strategyType was interpreted as a String, while updateStrategy is interpreted as an object.
• The service account token is now set to automount false by default. To change this, set the value serviceAccount.automountServiceAccountToken to true.

Additionally updates the MariaDB subchart to its newest major, 10.0.0, which contains similar changes. Here you can find more information about the changes introduced in this version.

To 4.0.0¶

This major updates the Kafka subchart its newest major, 14.0.0. Here you can find more information about the changes introduced in this version.

To 3.0.0¶

This major updates the Kafka subchart to its newest major 13.0.0. For more information on this subchart's major, please refer to kafka upgrade notes.

To 2.0.0¶

On November 13, 2020, Helm v2 support was formally finished, this major version is the result of the required changes applied to the Helm Chart to be able to incorporate the different features added in Helm v3 and to be consistent with the Helm project itself regarding the Helm v2 EOL.

What changes were introduced in this major version?¶

• Previous versions of this Helm Chart use apiVersion: v1 (installable by both Helm 2 and 3), this Helm Chart was updated to apiVersion: v2 (installable by Helm 3 only). Here you can find more information about the apiVersion field.
• Move dependency information from the requirements.yaml to the Chart.yaml
• After running helm dependency update, a Chart.lock file is generated containing the same structure used in the previous requirements.lock
• The different fields present in the Chart.yaml file has been ordered alphabetically in a homogeneous way for all the Bitnami Helm Charts

Considerations when upgrading to this version¶

• If you want to upgrade to this version from a previous one installed with Helm v3, you shouldn't face any issues
• If you want to upgrade to this version using Helm v2, this scenario is not supported as this version doesn't support Helm v2 anymore
• If you installed the previous version with Helm v2 and wants to upgrade to this version with Helm v3, please refer to the official Helm documentation about migrating from Helm v2 to v3

Useful links¶

• https://docs.bitnami.com/tutorials/resolve-helm2-helm3-post-migration-issues/
• https://helm.sh/docs/topics/v2_v3_migration/
• https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/

v0.x.x¶

helm upgrade my-release bitnami/spring-cloud-dataflow --set mariadb.rootUser.password=[MARIADB_ROOT_PASSWORD] --set rabbitmq.auth.password=[RABBITMQ_PASSWORD] --set rabbitmq.auth.erlangCookie=[RABBITMQ_ERLANG_COOKIE]

v1.x.x¶

helm upgrade my-release bitnami/spring-cloud-dataflow --set mariadb.auth.rootPassword=[MARIADB_ROOT_PASSWORD] --set rabbitmq.auth.password=[RABBITMQ_PASSWORD] --set rabbitmq.auth.erlangCookie=[RABBITMQ_ERLANG_COOKIE]

To 1.0.0¶

MariaDB dependency version was bumped to a new major version that introduces several incompatibilities. Therefore, backwards compatibility is not guaranteed unless an external database is used. Check MariaDB Upgrading Notes for more information.

To upgrade to 1.0.0, you will need to reuse the PVC used to hold the MariaDB data on your previous release. To do so, follow the instructions below (the following example assumes that the release name is dataflow):

NOTE: Please, create a backup of your database before running any of those actions.

Obtain the credentials and the name of the PVC used to hold the MariaDB data on your current release:

export MARIADB_ROOT_PASSWORD=$(kubectl get secret --namespace default dataflow-mariadb -o jsonpath="{.data.mariadb-root-password}" | base64 -d)
export MARIADB_PASSWORD=$(kubectl get secret --namespace default dataflow-mariadb -o jsonpath="{.data.mariadb-password}" | base64 -d)
export MARIADB_PVC=$(kubectl get pvc -l app=mariadb,component=master,release=dataflow -o jsonpath="{.items[0].metadata.name}")
export RABBITMQ_PASSWORD=$(kubectl get secret --namespace default dataflow-rabbitmq -o jsonpath="{.data.rabbitmq-password}" | base64 -d)
export RABBITMQ_ERLANG_COOKIE=$(kubectl get secret --namespace default dataflow-rabbitmq -o jsonpath="{.data.rabbitmq-erlang-cookie}" | base64 -d)

Upgrade your release (maintaining the version) disabling MariaDB and scaling Data Flow replicas to 0:

helm upgrade dataflow bitnami/spring-cloud-dataflow --version 0.7.4 \
  --set server.replicaCount=0 \
  --set skipper.replicaCount=0 \
  --set mariadb.enabled=false \
  --set rabbitmq.auth.password=$RABBITMQ_PASSWORD \
  --set rabbitmq.auth.erlangCookie=$RABBITMQ_ERLANG_COOKIE

Finally, upgrade you release to 1.0.0 reusing the existing PVC, and enabling back MariaDB:

helm upgrade dataflow bitnami/spring-cloud-dataflow \
  --set mariadb.primary.persistence.existingClaim=$MARIADB_PVC \
  --set mariadb.auth.rootPassword=$MARIADB_ROOT_PASSWORD \
  --set mariadb.auth.password=$MARIADB_PASSWORD \
  --set rabbitmq.auth.password=$RABBITMQ_PASSWORD \
  --set rabbitmq.auth.erlangCookie=$RABBITMQ_ERLANG_COOKIE

You should see the lines below in MariaDB container logs:

$ kubectl logs $(kubectl get pods -l app.kubernetes.io/instance=dataflow,app.kubernetes.io/name=mariadb,app.kubernetes.io/component=primary -o jsonpath="{.items[0].metadata.name}")
...
mariadb 12:13:24.98 INFO  ==> Using persisted data
mariadb 12:13:25.01 INFO  ==> Running mysql_upgrade
...

License¶

Copyright © 2023 VMware, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.