Skip to content

Commit d8d4381

Browse files
csun-cpointecsun-cpointe
authored
Merge pull request #416 from boozallen/403-pipeline-invocation-java17-upgrade
#403 pipeline-invocation-service java17 upgrade
2 parents f8378e3 + 450f91c commit d8d4381

File tree

29 files changed

+199
-4516
lines changed

29 files changed

+199
-4516
lines changed

‎DRAFT_RELEASE_NOTES.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,8 +71,24 @@ To start your aiSSEMBLE upgrade, update your project's pom.xml to use the 1.10.0
7171
</parent>
7272
```
7373

74+
### Split Data Records for the Spark Pipeline
75+
If your spark pipeline is using `aissemble-data-records-separate-module` profile for your data records, you must add the `<version>` tag for
76+
the `jackson-mapper-asl` dependency artifact in the root pom.xml file to enable the build.
77+
```xml
78+
<dependency>
79+
<groupId>org.codehaus.jackson</groupId>
80+
<artifactId>jackson-mapper-asl</artifactId>
81+
+ <version>${version.jackson.mapper.asl}</version>
82+
</dependency>
83+
```
84+
85+
7486
## Conditional Steps
7587

88+
### For projects that have customized the Spark Operator Service Account permissions
89+
The service account for the pipeline invocation service is now separated from spark operator and configured solely for the service.
90+
If you added any custom configurations to the `sparkoperator` service account pertaining to the pipeline invocation service, you will need to migrate the related changes to the new `pipeline-invocation-service-sa`. Refer to Pipeline Invocation Helm Chart [README](https://github.com/boozallen/aissemble/blob/dev/extensions/extensions-helm/extensions-helm-pipeline-invocation/aissemble-pipeline-invocation-app-chart/README.md) for detail.
91+
7692
## Final Steps - Required for All Projects
7793
### Finalizing the Upgrade
7894
1. Run `./mvnw org.technologybrewery.baton:baton-maven-plugin:baton-migrate` to apply the automatic migrations

‎build-parent/pom.xml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,7 @@
6767
<version.awaitility>4.0.3</version.awaitility>
6868
<version.plexus.util>3.5.1</version.plexus.util>
6969
<version.jackson.mapper.asl>1.9.3</version.jackson.mapper.asl>
70+
<version.exec.maven.plugin>3.4.1</version.exec.maven.plugin>
7071

7172
<!-- Java EE Dependencies -->
7273
<version.jakarta.cdi>4.0.1</version.jakarta.cdi>

‎extensions/extensions-docker/aissemble-pipeline-invocation/src/main/resources/docker/Dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
FROM registry.access.redhat.com/ubi9/openjdk-11-runtime:1.20 AS builder
1+
FROM registry.access.redhat.com/ubi9/openjdk-17-runtime:1.20 AS builder
22
USER root
33
RUN microdnf install -y openssl gzip && \
44
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

‎extensions/extensions-helm/aissemble-spark-application-chart/crds/sparkoperator.k8s.io_sparkapplications.yaml‎

Lines changed: 0 additions & 4427 deletions
This file was deleted.

‎extensions/extensions-helm/aissemble-spark-operator-chart/README.md‎

Lines changed: 0 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -45,25 +45,10 @@ aissemble-spark-operator-chart:
4545
| volumes | Volumes for the pod | No | `spark-logging=/tmp/spark-logging`, `ivy-cache=/home/spark/.ivy2` |
4646
| volumeMounts | Volume Mounts for the pod | No | `spark-logging=/tmp/spark-logging`, `ivy-cache=/home/spark/.ivy2` |
4747
| fullnameOverride | String to override release name | No | spark-operator |
48-
| rbac.createClusterRole | See `Migrated Properties` | No | false |
4948
| serviceAccounts.spark.name | Name for the spark service account | No | spark |
5049
| serviceAccounts.sparkoperator.name | Name for the spark service account | No | sparkoperator |
5150
| podSecurityContext | Pod security context | No | runAsUser: 185<br/>runAsGroup: 1000<br/>fsGroup: 1000<br/>fsGroupChangePolicy: "OnRootMismatch" |
5251

53-
## Migrated Properties
54-
The following properties have been migrated from the `spark-operator` subchart to the `aissemble-spark-operator-chart` chart.
55-
Any required overrides should be cognisant of the alternate path. For example:
56-
57-
```yaml
58-
aissemble-spark-operator-chart:
59-
rbac:
60-
createClusterRole: false
61-
```
62-
63-
| Property | Description | Default |
64-
|------------------------|-------------------------------------------------------------------------------|---------|
65-
| rbac.createClusterRole | Create and use RBAC `ClusterRole` resources. Migrated to use modified rules. | true |
66-
6752
# Shared Ivy Cache
6853

6954
Spark uses [Ivy](https://ant.apache.org/ivy/) to resolve and download dependencies for Spark applications. By default,

‎extensions/extensions-helm/aissemble-spark-operator-chart/values.template.yaml‎

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -29,12 +29,6 @@ spark-operator:
2929
runAsGroup: 1000
3030
fsGroup: 1000
3131
fsGroupChangePolicy: "OnRootMismatch"
32-
33-
rbac:
34-
# -- Create and use RBAC `ClusterRole` resources
35-
# -- Set to false in order to enable overriding with our own RBAC template
36-
createClusterRole: false
37-
3832
# volumes - Operator volumes
3933
volumes:
4034
- name: spark-logging
@@ -61,8 +55,4 @@ spark-operator:
6155

6256
sparkoperator:
6357
# -- Optional name for the operator service account
64-
name: "sparkoperator"
65-
66-
rbac:
67-
# -- Set to True in order to enable overriding with our own RBAC template
68-
createClusterRole: True
58+
name: "sparkoperator"

‎extensions/extensions-helm/extensions-helm-pipeline-invocation/aissemble-pipeline-invocation-app-chart/README.md‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -10,16 +10,17 @@ helm install pipeline-invocation-service oci://ghcr.io/boozallen/aissemble-pipel
1010
**Note**: *the version should match the aiSSEMBLE project version.*
1111

1212
# Properties
13-
| Property | Description | Required Override | Default |
14-
|------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|-------------------|-----------------------------------------------------|
15-
| ingress.apiVersion | k8s API version to use | No | networking.k8s.io/v1 |
16-
| ingress.enabled | k8s Whether to enable ingress | No | false |
17-
| ingress.kind | Type of kubernetes entity | No | Ingress |
18-
| ingress.metadata.name | Name of the ingress | No | pipeline-invocation-service-web |
19-
| ingress.metadata.annotations.kubernetes.io/ingress.class | Ingress class name | No | nginx |
20-
| ingress.metadata.annotations.ingress.metadata.annotations.nginx.ingress.kubernetes.io/server-snippet | Custom configurations for the nginx ingress class | No | gunzip on; gzip on; gzip_proxied any; gzip_types *; |
21-
| ingress.spec.rules.hosts | A list of hosts for ingress to support, each with their own path definition | No | |
22-
| ingress.status | Load balancer IP if required | No | None |
13+
| Property | Description | Required Override | Default |
14+
|-------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------|-------------------|-----------------------------------------------------|
15+
| ingress.apiVersion | k8s API version to use | No | networking.k8s.io/v1 |
16+
| ingress.enabled | k8s Whether to enable ingress | No | false |
17+
| ingress.kind | Type of kubernetes entity | No | Ingress |
18+
| ingress.metadata.name | Name of the ingress | No | pipeline-invocation-service-web |
19+
| ingress.metadata.annotations.kubernetes.io/ingress.class | Ingress class name | No | nginx |
20+
| ingress.metadata.annotations.nginx.ingress.kubernetes.io/server-snippet | Custom configurations for the nginx ingress class | No | gunzip on; gzip on; gzip_proxied any; gzip_types *; |
21+
| ingress.spec.rules.hosts | A list of hosts for ingress to support, each with their own path definition | No | |
22+
| ingress.status | Load balancer IP if required | No | None |
23+
| rbac.createClusterRole | Create and use RBAC `ClusterRole` resources. | No | true |
2324

2425
# Quarkus Configuration
2526

@@ -30,3 +31,5 @@ The following configuration of the service is provided. Additional configuratio
3031
|---------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------|
3132
| kafka.bootstrap.servers | Specifies the kafka bootstrap server when using kafka for messaging | Any valid URI |
3233
| mp.messaging.incoming.pipeline-invocation.* | Specifies and configures the smallrye connector to use. Supported connectors are `smallrye-amqp`, `smallrye-kafka`, `smallrye-mqtt`, and `smallrye-rabbitmq` | See xref:messaging-details.adoc[the Messaging documentation] for more details |
34+
35+

‎extensions/extensions-helm/extensions-helm-pipeline-invocation/aissemble-pipeline-invocation-app-chart/templates/.gitkeep‎

Whitespace-only changes.

‎extensions/extensions-helm/aissemble-spark-operator-chart/templates/rbac.yaml‎ renamed to ‎extensions/extensions-helm/extensions-helm-pipeline-invocation/aissemble-pipeline-invocation-app-chart/templates/rbac.yaml‎

Lines changed: 9 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,22 @@
11
{{- /*
22
aiSSEMBLE Custom rbac.yaml
33

4-
Required custom rbac.yaml file that grants the sparkoperator service account
4+
Required custom rbac.yaml file that grants the pipeline-invocation-service service account
55
create, delete, and update access to the apigroup apiextensions.k8s.io.
66

77
This is necessary for the pipeline-invocation-service to create instances of the
88
SparkApplication CRD to submit pipelines to the Spark Operator for execution.
99
*/}}
10-
11-
{{- if or .Values.rbac.create .Values.rbac.createClusterRole }}
10+
{{ if .Values.rbac.createClusterRole }}
11+
{{- $serviceAccountName := (index .Values "aissemble-quarkus-chart" "deployment" "serviceAccountName") -}}
1212
apiVersion: rbac.authorization.k8s.io/v1
1313
kind: ClusterRole
1414
metadata:
15-
name: {{ include "spark-operator.fullname" (index .Subcharts "spark-operator") }}
15+
name: {{ $serviceAccountName | default "pipeline-invocation-service" }}-clusterrole
1616
annotations:
1717
"helm.sh/hook": pre-install, pre-upgrade
1818
"helm.sh/hook-delete-policy": hook-failed, before-hook-creation
1919
"helm.sh/hook-weight": "-10"
20-
labels:
21-
{{- include "spark-operator.labels" (index .Subcharts "spark-operator") | nindent 4 }}
2220
rules:
2321
- apiGroups:
2422
- ""
@@ -34,6 +32,7 @@ rules:
3432
- configmaps
3533
- secrets
3634
verbs:
35+
- list
3736
- create
3837
- get
3938
- delete
@@ -75,6 +74,7 @@ rules:
7574
resources:
7675
- customresourcedefinitions
7776
verbs:
77+
- create
7878
- get
7979
- apiGroups:
8080
- admissionregistration.k8s.io
@@ -97,7 +97,6 @@ rules:
9797
- scheduledsparkapplications/finalizers
9898
verbs:
9999
- "*"
100-
{{- if .Values.batchScheduler.enable }}
101100
# required for the `volcano` batch scheduler
102101
- apiGroups:
103102
- scheduling.incubator.k8s.io
@@ -107,34 +106,29 @@ rules:
107106
- podgroups
108107
verbs:
109108
- "*"
110-
{{- end }}
111-
{{ if .Values.webhook.enable }}
112109
- apiGroups:
113110
- batch
114111
resources:
115112
- jobs
116113
verbs:
117114
- delete
118-
{{- end }}
119115

120116
---
121117

122118
apiVersion: rbac.authorization.k8s.io/v1
123119
kind: ClusterRoleBinding
124120
metadata:
125-
name: {{ include "spark-operator.fullname" (index .Subcharts "spark-operator") }}
121+
name: {{ $serviceAccountName | default "pipeline-invocation-service" }}-clusterrole-binding
126122
annotations:
127123
"helm.sh/hook": pre-install, pre-upgrade
128124
"helm.sh/hook-delete-policy": hook-failed, before-hook-creation
129125
"helm.sh/hook-weight": "-10"
130-
labels:
131-
{{- include "spark-operator.labels" (index .Subcharts "spark-operator") | nindent 4 }}
132126
subjects:
133127
- kind: ServiceAccount
134-
name: {{ include "spark-operator.serviceAccountName" (index .Subcharts "spark-operator") }}
128+
name: {{ $serviceAccountName | default "pipeline-invocation-service" }}
135129
namespace: {{ .Release.Namespace }}
136130
roleRef:
137131
kind: ClusterRole
138-
name: {{ include "spark-operator.fullname" (index .Subcharts "spark-operator") }}
132+
name: {{ $serviceAccountName | default "pipeline-invocation-service" }}-clusterrole
139133
apiGroup: rbac.authorization.k8s.io
140134
{{- end }}

‎extensions/extensions-helm/extensions-helm-pipeline-invocation/aissemble-pipeline-invocation-app-chart/templates/serviceaccount.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
{{- $serviceAccountName := (index .Values "aissemble-quarkus-chart" "deployment" "serviceAccountName") -}}
2+
apiVersion: v1
3+
kind: ServiceAccount
4+
metadata:
5+
name: {{ $serviceAccountName | default "pipeline-invocation-service" }}

0 commit comments

Comments
 (0)