Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@ All notable changes to this project will be documented in this file.
- Support objectOverrides using `.spec.objectOverrides`.
See [objectOverrides concepts page](https://docs.stackable.tech/home/nightly/concepts/overrides/#object-overrides) for details ([#885]).
- Enable the [restart-controller](https://docs.stackable.tech/home/nightly/commons-operator/restarter/), so that the Pods are automatically restarted on config changes ([#888]).
- Added support for `2.7.2` ([#893]).

### Removed

- Removed support for `1.27.0` and `2.4.0` ([#893]).

### Changed

Expand All @@ -25,6 +30,7 @@ All notable changes to this project will be documented in this file.
[#884]: https://github.com/stackabletech/nifi-operator/pull/884
[#885]: https://github.com/stackabletech/nifi-operator/pull/885
[#888]: https://github.com/stackabletech/nifi-operator/pull/888
[#893]: https://github.com/stackabletech/nifi-operator/pull/893

## [25.11.0] - 2025-11-07

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ metadata:
name: simple-nifi
spec:
image:
productVersion: 2.6.0
productVersion: 2.7.2
clusterConfig:
authentication:
- authenticationClass: simple-nifi-users
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ metadata:
name: simple-nifi
spec:
image:
productVersion: 2.6.0
productVersion: 2.7.2
clusterConfig:
authentication:
- authenticationClass: simple-nifi-users
Expand Down
20 changes: 9 additions & 11 deletions docs/modules/nifi/pages/usage_guide/index.adoc
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
= Usage guide
:description: Guide to configuring the Stackable Operator for Apache NiFi with YAML manifests, covering ZooKeeper, authentication, extra volumes, and resource settings.
:description: Guide to configuring the Stackable Operator for Apache NiFi with YAML manifests, covering authentication, extra volumes, and resource settings.

This section helps you to use various aspects of the Stackable operator for Apache NiFi.
For a general introduction into the operator follow the xref:getting_started/index.adoc[] guide.
Expand All @@ -17,12 +17,11 @@ metadata:
name: simple-nifi
spec:
image:
productVersion: 1.27.0
productVersion: 2.7.2
clusterConfig:
zookeeperConfigMapName: simple-nifi-znode # <1>
authentication: # <2>
authentication: # <1>
- authenticationClass: simple-nifi-admin-user
extraVolumes: # <3>
extraVolumes: # <2>
- name: nifi-client-certs
secret:
secretName: nifi-client-certs
Expand All @@ -35,18 +34,17 @@ spec:
roleGroups:
default:
config:
resources: # <4>
resources: # <3>
cpu:
min: "500m"
max: "4"
memory:
limit: '2Gi'
limit: 2Gi
replicas: 3
----

<1> The xref:usage_guide/clustering.adoc#backend-zookeeper[ZooKeeper instance] to use.
<2> How users should xref:usage_guide/security.adoc[authenticate] themselves.
<3> xref:usage_guide/extra-volumes.adoc[Extra volumes] with files that can be referenced in custom workflows.
<4> xref:usage_guide/resource-configuration.adoc[CPU and memory configuration] can be set per role group.
<1> How users should xref:usage_guide/security.adoc[authenticate] themselves.
<2> xref:usage_guide/extra-volumes.adoc[Extra volumes] with files that can be referenced in custom workflows.
<3> xref:usage_guide/resource-configuration.adoc[CPU and memory configuration] can be set per role group.

Not shown are the common settings for xref:usage_guide/operations/cluster-operations.adoc[starting and stopping the cluster] and xref:usage_guide/operations/pod-placement.adoc[distributing Pods]. Additionally, you can set any NiFi setting using xref:usage_guide/overrides.adoc[overrides]. You can also configure xref:usage_guide/log-aggregation.adoc[log aggregation].
6 changes: 3 additions & 3 deletions docs/modules/nifi/pages/usage_guide/updating.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
:description: Easily update or downgrade Apache NiFi on Kubernetes by changing the CRD version.

Updating (or downgrading for that matter) the deployed version of NiFi is as simple as changing the version stated in the CRD.
Continuing the example above, to change the deployed version from `1.27.0` to `2.0.0` you'd simply deploy the following CRD.
Continuing the example above, the deployed version will be updated by deploying the following custom resource.

[source,yaml]
----
Expand All @@ -12,7 +12,7 @@ metadata:
name: simple-nifi
spec:
image:
productVersion: 2.0.0 # <1>
productVersion: 2.7.2 # <1>
----

<1> Change the NiFi version here
Expand All @@ -24,7 +24,7 @@ NiFi clusters cannot be upgraded or downgraded in a rolling fashion due to a lim
When upgrading between NiFi 1 versions or from NiFi 1 to NiFi 2, any change to the NiFi version in the CRD triggers a full cluster restart with brief downtime.
However, the Stackable image version can be updated in a rolling manner, provided the NiFi version remains unchanged.

For upgrades between NiFi 2 versions, e.g. from `2.0.0` to `2.4.0`, rolling upgrades are supported.
Since NiFi version 2, rolling upgrades are supported.
====

== NiFi 2.0.0
Expand Down
18 changes: 16 additions & 2 deletions docs/modules/nifi/pages/usage_guide/writing-to-iceberg-tables.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,23 @@
Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Presto, Hive and Impala to safely work with the same tables, at the same time.

NiFi supports a `PutIceberg` processor to add rows to an existing Iceberg table https://issues.apache.org/jira/browse/NIFI-10442[starting from version 1.19.0].
As of NiFi version `2.4.0` only `PutIceberg` is supported, you need to create and compact your tables with other tools such as Trino or Spark (both included in the Stackable Data Platform).
As of NiFi version `2.7.2` only `PutIceberg` is supported, you need to create and compact your tables with other tools such as Trino or Spark (both included in the Stackable Data Platform).

== NiFi 2
== NiFi 2.7 and above

In NiFi `2.7.0` Iceberg support was https://issues.apache.org/jira/browse/NIFI-15062[re-added] after the removal in `2.0.0`.

The `2.7.x` version has the following changes over the `2.0.x` and `2.6.x` version, you need to adopt your setup accordingly:

* HDFS and Kerberos support was dropped
* Hive metastore support was dropped
* Iceberg REST catalog support was added
* It now uses the Iceberg S3 IO instead of the Hadoop S3 client libraries
* It uses much less dependencies and therefore reduces the amount of CVEs

There have been efforts from Stackable to re-add at least Hive metastore support, but we ran into NiFi classpath loader issues, which we haven't been able to solve so far.

== NiFi 2.0 - 2.6

In NiFi `2.0.0` Iceberg support https://issues.apache.org/jira/browse/NIFI-13938[has been removed] from upstream NiFi.

Expand Down
5 changes: 2 additions & 3 deletions docs/modules/nifi/partials/supported-versions.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,8 @@
// This is a separate file, since it is used by both the direct NiFi-Operator documentation, and the overarching
// Stackable Platform documentation.

* 2.6.0 (LTS, Please note that you need to upgrade to at least 1.27.x before upgrading to 2.x.x!)
* 2.4.0 (Deprecated, Please note that you need to upgrade to at least 1.27.x before upgrading to 2.x.x!)
* 2.7.2
* 2.6.0 (LTS)
* 1.28.1 (Deprecated)
* 1.27.0 (Deprecated)
For details on how to upgrade your NiFi version, refer to xref:nifi:usage_guide/updating.adoc[].
5 changes: 2 additions & 3 deletions examples/entra-static-nifi-policies/entra_nifi.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -173,10 +173,9 @@ metadata:
name: test-nifi
spec:
image:
productVersion: 2.6.0
productVersion: 2.7.2
pullPolicy: IfNotPresent
clusterConfig:
zookeeperConfigMapName: null
authentication:
- authenticationClass: simple-nifi-users
hostHeaderCheck:
Expand Down Expand Up @@ -226,7 +225,7 @@ spec:
name: oidc-secret
key: discovery.url
- name: prep-custom-authorizers
image: oci.stackable.tech/sdp/nifi:2.6.0-stackable0.0.0-dev
image: oci.stackable.tech/sdp/nifi:2.7.2-stackable0.0.0-dev
env:
- name: OIDC_AUTHORITY_ENDPOINT
valueFrom:
Expand Down
25 changes: 1 addition & 24 deletions examples/simple-cluster/simple-nifi-cluster.yaml
Original file line number Diff line number Diff line change
@@ -1,25 +1,3 @@
# Deploy a NiFi cluster, a user secret and a ZooKeeper cluster and a respective
# ZNode `simple-nifi-znode` which will be referenced
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperCluster
metadata:
name: simple-zk
spec:
image:
productVersion: 3.9.4
servers:
roleGroups:
default:
replicas: 1
---
apiVersion: zookeeper.stackable.tech/v1alpha1
kind: ZookeeperZnode
metadata:
name: simple-nifi-znode
spec:
clusterRef:
name: simple-zk
---
apiVersion: authentication.stackable.tech/v1alpha1
kind: AuthenticationClass
Expand Down Expand Up @@ -47,14 +25,13 @@ metadata:
name: simple-nifi
spec:
image:
productVersion: 2.6.0
productVersion: 2.7.2
clusterConfig:
authentication:
- authenticationClass: simple-nifi-admin-user
sensitiveProperties:
keySecret: nifi-sensitive-property-key
autoGenerate: true
zookeeperConfigMapName: simple-nifi-znode
nodes:
roleConfig:
listenerClass: external-unstable
Expand Down
6 changes: 2 additions & 4 deletions rust/operator-binary/src/config/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -818,14 +818,13 @@ mod tests {
name: simple-nifi
spec:
image:
productVersion: 1.27.0
productVersion: 2.7.2
clusterConfig:
authentication:
- authenticationClass: nifi-admin-credentials-simple
sensitiveProperties:
keySecret: simple-nifi-sensitive-property-key
autoGenerate: true
zookeeperConfigMapName: simple-nifi-znode
nodes:
roleGroups:
default:
Expand Down Expand Up @@ -867,14 +866,13 @@ mod tests {
name: simple-nifi
spec:
image:
productVersion: 1.27.0
productVersion: 2.7.2
clusterConfig:
authentication:
- authenticationClass: nifi-admin-credentials-simple
sensitiveProperties:
keySecret: simple-nifi-sensitive-property-key
autoGenerate: true
zookeeperConfigMapName: simple-nifi-znode
nodes:
config:
resources:
Expand Down
3 changes: 1 addition & 2 deletions rust/operator-binary/src/crd/affinity.rs
Original file line number Diff line number Diff line change
Expand Up @@ -43,14 +43,13 @@ mod tests {
name: simple-nifi
spec:
image:
productVersion: 1.27.0
productVersion: 2.7.2
clusterConfig:
authentication:
- authenticationClass: nifi-admin-credentials-simple
sensitiveProperties:
keySecret: simple-nifi-sensitive-property-key
autoGenerate: true
zookeeperConfigMapName: simple-nifi-znode
nodes:
roleGroups:
default:
Expand Down
28 changes: 0 additions & 28 deletions tests/templates/kuttl/cluster_operation/10-install-zk.yaml.j2

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,6 @@ spec:
{% if lookup('env', 'VECTOR_AGGREGATOR') %}
vectorAggregatorConfigMapName: vector-aggregator-discovery
{% endif %}
zookeeperConfigMapName: test-nifi-znode
nodes:
config:
gracefulShutdownTimeout: 1m
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ spec:
{% if lookup('env', 'VECTOR_AGGREGATOR') %}
vectorAggregatorConfigMapName: vector-aggregator-discovery
{% endif %}
zookeeperConfigMapName: test-nifi-znode
clusterOperation:
stopped: true
reconciliationPaused: false
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ spec:
{% if lookup('env', 'VECTOR_AGGREGATOR') %}
vectorAggregatorConfigMapName: vector-aggregator-discovery
{% endif %}
zookeeperConfigMapName: test-nifi-znode
clusterOperation:
stopped: false
reconciliationPaused: true
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@ spec:
{% if lookup('env', 'VECTOR_AGGREGATOR') %}
vectorAggregatorConfigMapName: vector-aggregator-discovery
{% endif %}
zookeeperConfigMapName: test-nifi-znode
clusterOperation:
stopped: false
reconciliationPaused: false
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ metadata:
name: nifi
spec:
image:
{% if test_scenario['values']['nifi-iceberg'].find(",") > 0 %}
custom: "{{ test_scenario['values']['nifi-iceberg'].split(',')[1] }}"
productVersion: "{{ test_scenario['values']['nifi-iceberg'].split(',')[0] }}"
{% if test_scenario['values']['nifi-iceberg-hive'].find(",") > 0 %}
custom: "{{ test_scenario['values']['nifi-iceberg-hive'].split(',')[1] }}"
productVersion: "{{ test_scenario['values']['nifi-iceberg-hive'].split(',')[0] }}"
{% else %}
custom: null
productVersion: "{{ test_scenario['values']['nifi-iceberg'] }}"
productVersion: "{{ test_scenario['values']['nifi-iceberg-hive'] }}"
{% endif %}
pullPolicy: IfNotPresent
clusterConfig:
Expand Down
9 changes: 9 additions & 0 deletions tests/templates/kuttl/iceberg-rest/00-patch-ns.yaml.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{% if test_scenario['values']['openshift'] == 'true' %}
# see https://github.com/stackabletech/issues/issues/566
---
apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: kubectl patch namespace $NAMESPACE -p '{"metadata":{"labels":{"pod-security.kubernetes.io/enforce":"privileged"}}}'
timeout: 120
{% endif %}
29 changes: 29 additions & 0 deletions tests/templates/kuttl/iceberg-rest/00-rbac.yaml.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-role
rules:
{% if test_scenario['values']['openshift'] == "true" %}
- apiGroups: ["security.openshift.io"]
resources: ["securitycontextconstraints"]
resourceNames: ["privileged"]
verbs: ["use"]
{% endif %}
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: test-sa
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: test-rb
subjects:
- kind: ServiceAccount
name: test-sa
roleRef:
kind: Role
name: test-role
apiGroup: rbac.authorization.k8s.io
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
apiVersion: kuttl.dev/v1beta1
kind: TestStep
commands:
- script: envsubst '$NAMESPACE' < 01_s3-connection.yaml | kubectl apply -n $NAMESPACE -f -
Loading