You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kie.apache.org by "jordigilh (via GitHub)" <gi...@apache.org> on 2023/12/12 18:57:43 UTC

[PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

jordigilh opened a new pull request, #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322

   * Adds the Persistence field in the SonataFlow CRD
   * Adds the implementation to handle PostgreSQL persistence type 
   * If `Persistence` field is empty or nil, no persistence is defined.
   
   @tchughesiv @wmedvede @ricardozanini PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#discussion_r1426826113


##########
api/v1alpha08/sonataflow_types.go:
##########
@@ -657,6 +657,9 @@ type SonataFlowSpec struct {
 	// PodTemplate describes the deployment details of this SonataFlow instance.
 	//+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="podTemplate"
 	PodTemplate PodTemplateSpec `json:"podTemplate,omitempty"`
+	// Persists service to a datasource of choice. Ephemeral by default.
+	// +optional
+	Persistence *PersistenceOptions `json:"persistence,omitempty"`

Review Comment:
   The idea is that this field can be used to override the platform's default persistence, or define one in case it does not exist. The platform's persistence field does not exist yet, it will be implemented in [KOGITO-9980](https://issues.redhat.com/browse/KOGITO-9980)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#issuecomment-1910432629

   Let's wait for @wmedvede approval. I'll also take a look on it tomorrow. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "wmedvede (via GitHub)" <gi...@apache.org>.
wmedvede commented on code in PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#discussion_r1462044751


##########
test/testdata/persistence/workflow/by_service/02-sonataflow_platform.yaml:
##########
@@ -0,0 +1,27 @@
+# Copyright 2024 Apache Software Foundation (ASF)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlowPlatform
+metadata:
+  name: sonataflow-platform
+spec:
+  build:
+    template:
+      buildArgs:
+      - name: QUARKUS_EXTENSIONS
+        value: org.kie.kogito:kogito-addons-quarkus-jobs-knative-eventing:999-SNAPSHOT,org.kie.kogito:kogito-addons-quarkus-persistence-jdbc:999-SNAPSHOT,io.quarkus:quarkus-jdbc-postgresql:3.2.9.Final,io.quarkus:quarkus-agroal:3.2.9.Final

Review Comment:
   I think we need to assess here, because the persistence is a sort of all or nothing. I mean, if we add the org.kie.kogito:kogito-addons-quarkus-persistence-jdbc:999-SNAPSHOT to the swf-builder, then, every w the workflow must use persistence, otherwise we'll have strart-up failures.
   
   O think we have the following scenarios:
   
   1) The images is produced externally:
   The workflow must come already built with the proper extensions incorporated in the produced image, and users might decide in a per workflow basis to include the persistence related addons. Finally the operator just relies on the workflow CRD con connect with the DB.
   
   2) The Image is produced by the operator managed Kaniko or OpenShift based builds, here
   
   But we are dealing with the persistence as an optional thing, so, it could happen the a workflow CRD with no configured persistence might come.
   
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini merged PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#discussion_r1462013163


##########
test/testdata/persistence/workflow/by_service/03-sonataflow_callbackstatetimeouts.sw.yaml:
##########
@@ -0,0 +1,101 @@
+# Copyright 2024 Apache Software Foundation (ASF)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlow
+metadata:
+  name: callbackstatetimeouts
+  annotations:
+    sonataflow.org/description: Callback State Timeouts Example k8s
+    sonataflow.org/version: 0.0.1
+spec:
+  persistence:
+    postgresql:
+      secretRef:
+        name: postgres-secrets
+        userKey: POSTGRES_USER
+        passwordKey: POSTGRES_PASSWORD
+      serviceRef:
+        name: postgres
+        port: 5432
+        databaseName: sonataflow
+        databaseSchema: callbackstatetimeouts
+  podTemplate:
+    container:
+      env:
+      - name: QUARKUS_FLYWAY_MIGRATE_AT_START
+        value: "true"
+    initContainers:
+      - name: init-postgres
+        image: registry.access.redhat.com/ubi9/ubi-minimal:latest

Review Comment:
   Changed to use `ubi-micro:latest` in this manifest



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#issuecomment-1900225101

   > Please note that I had to remove the `DEBUG=true` env value in the job because it caused the `kubectl` commands to dump extra log entries and that caused errors when parsing the output as it did not match the expected output.
   
   Thanks! I've removed this env var too in an opened PR! 😄 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#issuecomment-1910361508

   @ricardozanini PR is green. Can we merge it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "masayag (via GitHub)" <gi...@apache.org>.
masayag commented on code in PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#discussion_r1426687208


##########
api/v1alpha08/sonataflow_types.go:
##########
@@ -657,6 +657,9 @@ type SonataFlowSpec struct {
 	// PodTemplate describes the deployment details of this SonataFlow instance.
 	//+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="podTemplate"
 	PodTemplate PodTemplateSpec `json:"podTemplate,omitempty"`
+	// Persists service to a datasource of choice. Ephemeral by default.
+	// +optional
+	Persistence *PersistenceOptions `json:"persistence,omitempty"`

Review Comment:
   @ricardozanini shouldn't the decision about persistency of the runtime be determined by the Platform?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "domhanak (via GitHub)" <gi...@apache.org>.
domhanak commented on PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#issuecomment-1893638453

   Looks like the added test case is not working here on CI:
   Needs to be stable to merge,  I am trying to reproduce the test failure locally.
   ```
   Validate the persistence  when deploying a SonataFlow CR with PostgreSQL persistence [It] defined in the workflow from an existing kubernetes service as a reference
   /home/runner/work/incubator-kie-kogito-serverless-operator/incubator-kie-kogito-serverless-operator/test/e2e/persistence_test.go:123
   
     [FAILED] Timed out after 600.001s.
     Expected success, but got an error:
         <*errors.errorString | 0xc00007a2a0>: 
         kubectl wait pod -n test-546 -l sonataflow.org/workflow-app=callbackstatetimeouts --for condition=Ready --timeout=5s failed with error: (exit status 1) error: timed out waiting for the condition on pods/callbackstatetimeouts-7bf57f9968-5nfsl
         
         {
             s: "kubectl wait pod -n test-546 -l sonataflow.org/workflow-app=callbackstatetimeouts --for condition=Ready --timeout=5s failed with error: (exit status 1) error: timed out waiting for the condition on pods/callbackstatetimeouts-7bf57f9968-5nfsl\n",
         }
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#issuecomment-1899486156

   @jordigilh many thanks! we Will wait for QE verification first with @domhanak 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on code in PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#discussion_r1463334010


##########
test/testdata/persistence/workflow/by_service/03-sonataflow_callbackstatetimeouts.sw.yaml:
##########
@@ -0,0 +1,101 @@
+# Copyright 2024 Apache Software Foundation (ASF)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlow
+metadata:
+  name: callbackstatetimeouts
+  annotations:
+    sonataflow.org/description: Callback State Timeouts Example k8s
+    sonataflow.org/version: 0.0.1
+spec:
+  persistence:
+    postgresql:
+      secretRef:
+        name: postgres-secrets
+        userKey: POSTGRES_USER
+        passwordKey: POSTGRES_PASSWORD
+      serviceRef:
+        name: postgres
+        port: 5432
+        databaseName: sonataflow
+        databaseSchema: callbackstatetimeouts
+  podTemplate:
+    container:
+      env:
+      - name: QUARKUS_FLYWAY_MIGRATE_AT_START
+        value: "true"
+    initContainers:
+      - name: init-postgres
+        image: registry.access.redhat.com/ubi9/ubi-minimal:latest
+        imagePullPolicy: IfNotPresent
+        command: [ 'sh', '-c', 'until (echo 1 > /dev/tcp/postgres.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local/5432) >/dev/null 2>&1; do echo "Waiting for postgres server"; sleep 3; done;' ]

Review Comment:
   [we are going to add the concept of traits](https://issues.redhat.com/browse/KOGITO-8255), that will handle use cases like this. So even if produced externally, the operator will know if the workflow depends on persistence or not and act accordingly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#issuecomment-1900262064

   @tchughesiv can you please review it too?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "wmedvede (via GitHub)" <gi...@apache.org>.
wmedvede commented on PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#issuecomment-1905517786

   Hi @jordigilh , it looks like we have some merge conflicts now, would you mind take a look please.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "wmedvede (via GitHub)" <gi...@apache.org>.
wmedvede commented on code in PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#discussion_r1462229891


##########
test/testdata/persistence/workflow/by_service/03-sonataflow_callbackstatetimeouts.sw.yaml:
##########
@@ -0,0 +1,101 @@
+# Copyright 2024 Apache Software Foundation (ASF)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlow
+metadata:
+  name: callbackstatetimeouts
+  annotations:
+    sonataflow.org/description: Callback State Timeouts Example k8s
+    sonataflow.org/version: 0.0.1
+spec:
+  persistence:
+    postgresql:
+      secretRef:
+        name: postgres-secrets
+        userKey: POSTGRES_USER
+        passwordKey: POSTGRES_PASSWORD
+      serviceRef:
+        name: postgres
+        port: 5432
+        databaseName: sonataflow
+        databaseSchema: callbackstatetimeouts
+  podTemplate:
+    container:
+      env:
+      - name: QUARKUS_FLYWAY_MIGRATE_AT_START
+        value: "true"
+    initContainers:
+      - name: init-postgres
+        image: registry.access.redhat.com/ubi9/ubi-minimal:latest
+        imagePullPolicy: IfNotPresent
+        command: [ 'sh', '-c', 'until (echo 1 > /dev/tcp/postgres.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local/5432) >/dev/null 2>&1; do echo "Waiting for postgres server"; sleep 3; done;' ]

Review Comment:
   General comment here guys  @jordigilh @ricardozanini that doesn't block this PR.
   
   I think we need to assess here, because the persistence is a sort of all or nothing.
   
   I mean, if we add the **org.kie.kogito:kogito-addons-quarkus-persistence-jdbc:999-SNAPSHOT** to the **kogito-swf-builder**, then, every workflow image we produce must use persistence, otherwise we'll have runtime start-up failures.
   Additionally, we have the following scenarios:
   
   **1) The image is produced externally:**
   The workflow must come already built with the proper extensions incorporated in the produced image, and users might decide in a per-workflow basis to include the persistence related addons.
   The operator just relies on the workflow CRD con connect with the DB, and just produce the ENV vars as Jordi is doing. 
   If the image is not ok, it's not the operator problem. Which make sense in my opinion.
   
   (right now, AFAIK Parodos team is using the kogito-swf-builder series to produce their images, and I think they might eventually produce the images by adding the persistence related extensions in a per-workflow basis,
   or always add them if they always want persistence)
   
   **2) The Image is produced by the operator, and thus, managed by Kaniko or OpenShift based builds, etc.**
   In these cases, we could eventually manage to add the persistence extensions somehow dynamically, since in the end, for example with the Kaniko build we are producing a sonataflow-my-workflow-builder CM and here we can maybe manage to add these extensions.
   
   But, since Parodos is working with 1), I think they should add those extensions in their images build system.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "wmedvede (via GitHub)" <gi...@apache.org>.
wmedvede commented on code in PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#discussion_r1462236449


##########
test/testdata/persistence/workflow/by_service/02-sonataflow_platform.yaml:
##########
@@ -0,0 +1,27 @@
+# Copyright 2024 Apache Software Foundation (ASF)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlowPlatform
+metadata:
+  name: sonataflow-platform
+spec:
+  build:
+    template:
+      buildArgs:
+      - name: QUARKUS_EXTENSIONS
+        value: org.kie.kogito:kogito-addons-quarkus-jobs-knative-eventing:999-SNAPSHOT,org.kie.kogito:kogito-addons-quarkus-persistence-jdbc:999-SNAPSHOT,io.quarkus:quarkus-jdbc-postgresql:3.2.9.Final,io.quarkus:quarkus-agroal:3.2.9.Final

Review Comment:
   @jordigilh @ricardozanini  @tchughesiv  @masayag 
   I think we need to assess here, because the persistence is a sort of all or nothing.
   
   I mean, if we add the org.kie.kogito:kogito-addons-quarkus-persistence-jdbc:999-SNAPSHOT to the kogito-swf-builder, then, every workflow image we produce must use persistence, otherwise we'll have runtime start-up failures.
   Additionally, we have the following scenarios:
   
   **1) The images is produced externally:**
   The workflow must come already built with the proper extensions incorporated in the produced image, and users might decide in a per-workflow basis to include the persistence related addons.
   The operator just relies on the workflow CRD con connect with the DB, and just produce the ENV vars as Jordi is doing.
   If the image is not ok, it's not the operator problem. Which make sense in my opinion.
   
   (right now, AFAIK Parodos team is using the kogito-swf-builder series, however they might eventually produce the images by adding the persistence related extensions in a per-workflow basis,
   or always add them if they always want persistence)
   
   **2) The Image is produced by the operator, and thus, managed by Kaniko or OpenShift based builds, etc.**
   In these cases, we could eventually manage to add the persistence extensions somehow dynamically, since in the end, for example with the Kaniko build we are producing a sonataflow-my-workflow-builder CM and here we can maybe manage to add these extensions.
   
   But, since Parodos is working with 1), I think they should add those extensions in their images build system.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on code in PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#discussion_r1425651578


##########
controllers/profiles/common/persistence.go:
##########
@@ -0,0 +1,109 @@
+// Copyright 2023 Apache Software Foundation (ASF)
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package common
+
+import (
+	"strconv"
+
+	corev1 "k8s.io/api/core/v1"
+
+	operatorapi "github.com/apache/incubator-kie-kogito-serverless-operator/api/v1alpha08"
+)
+
+const (
+	defaultSchemaName = "default"
+)
+
+func ConfigurePersistence(serviceContainer *corev1.Container, options *operatorapi.PersistenceOptions, defaultSchema, namespace string) *corev1.Container {
+	c := serviceContainer.DeepCopy()
+	if options != nil {
+		if options.PostgreSql != nil {
+			c.Image = DataIndexImageBase + PersistenceTypePostgressql
+			c.Env = append(
+				c.Env,
+				configurePostgreSqlEnv(options.PostgreSql, defaultSchema, namespace)...,
+			)
+		}
+	}
+	return c
+}
+
+func configurePostgreSqlEnv(postgresql *operatorapi.PersistencePostgreSql, databaseSchema, databaseNamespace string) []corev1.EnvVar {

Review Comment:
   Can you move these hardcoded strings and integers to a private const in this file?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#issuecomment-1898656145

   > Looks like the added test case is not working here on CI: Needs to be stable to merge, I am trying to reproduce the test failure locally.
   > 
   > ```
   > Validate the persistence  when deploying a SonataFlow CR with PostgreSQL persistence [It] defined in the workflow from an existing kubernetes service as a reference
   > /home/runner/work/incubator-kie-kogito-serverless-operator/incubator-kie-kogito-serverless-operator/test/e2e/persistence_test.go:123
   > 
   >   [FAILED] Timed out after 600.001s.
   >   Expected success, but got an error:
   >       <*errors.errorString | 0xc00007a2a0>: 
   >       kubectl wait pod -n test-546 -l sonataflow.org/workflow-app=callbackstatetimeouts --for condition=Ready --timeout=5s failed with error: (exit status 1) error: timed out waiting for the condition on pods/callbackstatetimeouts-7bf57f9968-5nfsl
   >       
   >       {
   >           s: "kubectl wait pod -n test-546 -l sonataflow.org/workflow-app=callbackstatetimeouts --for condition=Ready --timeout=5s failed with error: (exit status 1) error: timed out waiting for the condition on pods/callbackstatetimeouts-7bf57f9968-5nfsl\n",
   >       }
   > ```
   
   Fixed 😄 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on code in PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#discussion_r1458838423


##########
test/e2e/persistence_test.go:
##########
@@ -0,0 +1,146 @@
+// Copyright 2024 Apache Software Foundation (ASF)
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package e2e
+
+import (
+	"bytes"
+	"fmt"
+	"math/rand"
+	"os/exec"
+	"strings"
+	"time"
+
+	"github.com/apache/incubator-kie-kogito-serverless-operator/test"
+	"github.com/apache/incubator-kie-kogito-serverless-operator/test/utils"
+
+	//nolint:golint
+	//nolint:revive
+	. "github.com/onsi/ginkgo/v2"
+
+	//nolint:golint
+	//nolint:revive
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("Validate the persistence ", Ordered, func() {
+

Review Comment:
   Nice one!



##########
controllers/platform/services/services.go:
##########
@@ -134,7 +134,9 @@ func (d DataIndex) ConfigurePersistence(containerSpec *corev1.Container) *corev1
 	if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil {
 		c := containerSpec.DeepCopy()
 		c.Image = d.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, d.configurePostgreSqlEnv(d.platform.Spec.Services.DataIndex.Persistence.PostgreSql, d.GetServiceName(), d.platform.Namespace)...)
+		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(d.platform.Spec.Services.DataIndex.Persistence.PostgreSql, d.GetServiceName(), d.platform.Namespace)...)
+		// specific to DataIndex
+		c.Env = append(c.Env, corev1.EnvVar{Name: "QUARKUS_HIBERNATE_ORM_DATABASE_GENERATION", Value: "update"}, corev1.EnvVar{Name: "QUARKUS_FLYWAY_MIGRATE_AT_START", Value: "true"})

Review Comment:
   Can you move these to private consts? 



##########
test/testdata/persistence/workflow/by_service/02-sonataflow_platform.yaml:
##########
@@ -0,0 +1,27 @@
+# Copyright 2024 Apache Software Foundation (ASF)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlowPlatform
+metadata:
+  name: sonataflow-platform
+spec:
+  build:
+    template:
+      buildArgs:
+      - name: QUARKUS_EXTENSIONS
+        value: org.kie.kogito:kogito-addons-quarkus-jobs-knative-eventing:999-SNAPSHOT,org.kie.kogito:kogito-addons-quarkus-persistence-jdbc:999-SNAPSHOT,io.quarkus:quarkus-jdbc-postgresql:3.2.9.Final,io.quarkus:quarkus-agroal:3.2.9.Final

Review Comment:
   @wmedvede I think it's worth adding those to the images, so we keep them in the cache to facilitate persistence use cases. wdyt?



##########
test/testdata/persistence/workflow/by_service/03-sonataflow_callbackstatetimeouts.sw.yaml:
##########
@@ -0,0 +1,101 @@
+# Copyright 2024 Apache Software Foundation (ASF)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlow
+metadata:
+  name: callbackstatetimeouts
+  annotations:
+    sonataflow.org/description: Callback State Timeouts Example k8s
+    sonataflow.org/version: 0.0.1
+spec:
+  persistence:
+    postgresql:
+      secretRef:
+        name: postgres-secrets
+        userKey: POSTGRES_USER
+        passwordKey: POSTGRES_PASSWORD
+      serviceRef:
+        name: postgres
+        port: 5432
+        databaseName: sonataflow
+        databaseSchema: callbackstatetimeouts
+  podTemplate:
+    container:
+      env:
+      - name: QUARKUS_FLYWAY_MIGRATE_AT_START
+        value: "true"
+    initContainers:
+      - name: init-postgres
+        image: registry.access.redhat.com/ubi9/ubi-minimal:latest

Review Comment:
   I guess we can use https://catalog.redhat.com/software/containers/ubi9/ubi-micro/615bdf943f6014fa45ae1b58 instead. IIRC `coreutils` includes cat.
   
   It's 7 vs 36 MB. Quite a difference in storage.



##########
controllers/profiles/common/object_creators_test.go:
##########
@@ -169,3 +169,171 @@ func TestMergePodSpec_OverrideContainers(t *testing.T) {
 	assert.Equal(t, int32(8080), flowContainer.Ports[0].ContainerPort)
 	assert.Empty(t, flowContainer.Env)
 }
+
+func TestMergePodSpec_WithPostgreSQL_and_JDBC_URL_field(t *testing.T) {
+	workflow := test.GetBaseSonataFlow(t.Name())
+	workflow.Spec = v1alpha08.SonataFlowSpec{
+		PodTemplate: v1alpha08.PodTemplateSpec{
+			Container: v1alpha08.ContainerSpec{
+				// this one we can override
+				Image: "quay.io/example/my-workflow:1.0.0",
+				Ports: []corev1.ContainerPort{
+					// let's override a immutable attribute
+					{Name: utils.HttpScheme, ContainerPort: 9090},
+				},
+				Env: []corev1.EnvVar{
+					// We should be able to override this too
+					{Name: "ENV1", Value: "VALUE_CUSTOM"},
+				},
+				VolumeMounts: []corev1.VolumeMount{
+					{Name: "myvolume", ReadOnly: true, MountPath: "/tmp/any/path"},
+				},
+			},
+			PodSpec: v1alpha08.PodSpec{
+				ServiceAccountName: "superuser",
+				Containers: []corev1.Container{
+					{
+						Name: "sidecar",
+					},
+				},
+				Volumes: []corev1.Volume{
+					{
+						Name: "myvolume",
+						VolumeSource: corev1.VolumeSource{
+							ConfigMap: &corev1.ConfigMapVolumeSource{
+								LocalObjectReference: corev1.LocalObjectReference{Name: "customproperties"},
+							},
+						},
+					},
+				},
+			},
+		},
+		Persistence: &v1alpha08.PersistenceOptions{
+			PostgreSql: &v1alpha08.PersistencePostgreSql{
+				SecretRef: v1alpha08.PostgreSqlSecretOptions{Name: "test"},
+				JdbcUrl:   "jdbc:postgresql://host:port/database?currentSchema=workflow",
+			},
+		},
+	}
+
+	object, err := DeploymentCreator(workflow)
+	assert.NoError(t, err)
+
+	deployment := object.(*appsv1.Deployment)
+	expectedEnvVars := []corev1.EnvVar{
+		{
+			Name:  "ENV1",
+			Value: "VALUE_CUSTOM",
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_USERNAME",
+			Value: "",
+			ValueFrom: &corev1.EnvVarSource{
+				SecretKeyRef: &corev1.SecretKeySelector{
+					LocalObjectReference: corev1.LocalObjectReference{Name: "test"}, Key: "POSTGRESQL_USER",
+				},
+			},
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_PASSWORD",
+			Value: "",
+			ValueFrom: &corev1.EnvVarSource{
+				SecretKeyRef: &corev1.SecretKeySelector{
+					LocalObjectReference: corev1.LocalObjectReference{Name: "test"}, Key: "POSTGRESQL_PASSWORD",
+				},
+			},
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_DB_KIND",
+			Value: "postgresql",
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_JDBC_URL",
+			Value: "jdbc:postgresql://host:port/database?currentSchema=workflow",
+		},
+	}
+	assert.Len(t, deployment.Spec.Template.Spec.Containers, 2)
+	assert.Equal(t, "superuser", deployment.Spec.Template.Spec.ServiceAccountName)
+	assert.Len(t, deployment.Spec.Template.Spec.Volumes, 1)
+	flowContainer, _ := kubeutil.GetContainerByName(v1alpha08.DefaultContainerName, &deployment.Spec.Template.Spec)
+	assert.Equal(t, "quay.io/example/my-workflow:1.0.0", flowContainer.Image)
+	assert.Equal(t, int32(8080), flowContainer.Ports[0].ContainerPort)
+	assert.Equal(t, expectedEnvVars, flowContainer.Env)
+	assert.Len(t, flowContainer.VolumeMounts, 1)
+}
+
+var (
+	postgreSQLPort = 5432

Review Comment:
   nit; are you using this var outside the `TestMergePodSpec_OverrideContainers_WithPostgreSQL_and_ServiceRef` test scope?



##########
test/e2e/workflow_test.go:
##########
@@ -41,8 +42,8 @@ import (
 	. "github.com/onsi/gomega"
 )
 
-// namespace store the ns where the Operator and Operand will be executed
-const namespace = "sonataflow-operator-system"
+// sonataflow_operator_namespace store the ns where the Operator and Operand will be executed
+const sonataflow_operator_namespace = "sonataflow-operator-system"

Review Comment:
   👍 



##########
controllers/profiles/common/persistence/postgreSQL.go:
##########
@@ -0,0 +1,89 @@
+// Copyright 2023 Apache Software Foundation (ASF)
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package persistence
+
+import (
+	"strconv"
+
+	corev1 "k8s.io/api/core/v1"
+
+	operatorapi "github.com/apache/incubator-kie-kogito-serverless-operator/api/v1alpha08"
+	"github.com/apache/incubator-kie-kogito-serverless-operator/controllers/profiles/common/constants"
+)
+
+const (
+	defaultSchemaName   = "default"
+	defaultDatabaseName = "sonataflow"
+)
+
+func ConfigurePostgreSqlEnv(postgresql *operatorapi.PersistencePostgreSql, databaseSchema, databaseNamespace string) []corev1.EnvVar {
+	dataSourcePort := constants.DefaultPostgreSQLPort
+	databaseName := defaultDatabaseName
+	dataSourceURL := postgresql.JdbcUrl
+	if postgresql.ServiceRef != nil {
+		if len(postgresql.ServiceRef.DatabaseSchema) > 0 {
+			databaseSchema = postgresql.ServiceRef.DatabaseSchema
+		}
+		if len(postgresql.ServiceRef.Namespace) > 0 {
+			databaseNamespace = postgresql.ServiceRef.Namespace
+		}
+		if postgresql.ServiceRef.Port != nil {
+			dataSourcePort = *postgresql.ServiceRef.Port
+		}
+		if len(postgresql.ServiceRef.DatabaseName) > 0 {
+			databaseName = postgresql.ServiceRef.DatabaseName
+		}
+		dataSourceURL = "jdbc:" + constants.PersistenceTypePostgreSQL + "://" + postgresql.ServiceRef.Name + "." + databaseNamespace + ":" + strconv.Itoa(dataSourcePort) + "/" + databaseName + "?currentSchema=" + databaseSchema
+	}
+	secretRef := corev1.LocalObjectReference{
+		Name: postgresql.SecretRef.Name,
+	}
+	quarkusDatasourceUsername := "POSTGRESQL_USER"
+	if len(postgresql.SecretRef.UserKey) > 0 {
+		quarkusDatasourceUsername = postgresql.SecretRef.UserKey
+	}
+	quarkusDatasourcePassword := "POSTGRESQL_PASSWORD"
+	if len(postgresql.SecretRef.PasswordKey) > 0 {
+		quarkusDatasourcePassword = postgresql.SecretRef.PasswordKey
+	}
+	return []corev1.EnvVar{
+		{
+			Name: "QUARKUS_DATASOURCE_USERNAME",
+			ValueFrom: &corev1.EnvVarSource{
+				SecretKeyRef: &corev1.SecretKeySelector{
+					Key:                  quarkusDatasourceUsername,
+					LocalObjectReference: secretRef,
+				},
+			},
+		},
+		{
+			Name: "QUARKUS_DATASOURCE_PASSWORD",
+			ValueFrom: &corev1.EnvVarSource{
+				SecretKeyRef: &corev1.SecretKeySelector{
+					Key:                  quarkusDatasourcePassword,
+					LocalObjectReference: secretRef,
+				},
+			},
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_DB_KIND",
+			Value: constants.PersistenceTypePostgreSQL,
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_JDBC_URL",
+			Value: dataSourceURL,
+		},

Review Comment:
   I'd rather keep all these hardcoded strings in private const.



##########
test/testdata/persistence/workflow/by_service/03-sonataflow_callbackstatetimeouts.sw.yaml:
##########
@@ -0,0 +1,101 @@
+# Copyright 2024 Apache Software Foundation (ASF)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlow
+metadata:
+  name: callbackstatetimeouts
+  annotations:
+    sonataflow.org/description: Callback State Timeouts Example k8s
+    sonataflow.org/version: 0.0.1
+spec:
+  persistence:
+    postgresql:
+      secretRef:
+        name: postgres-secrets
+        userKey: POSTGRES_USER
+        passwordKey: POSTGRES_PASSWORD
+      serviceRef:
+        name: postgres
+        port: 5432
+        databaseName: sonataflow
+        databaseSchema: callbackstatetimeouts
+  podTemplate:
+    container:
+      env:
+      - name: QUARKUS_FLYWAY_MIGRATE_AT_START
+        value: "true"
+    initContainers:
+      - name: init-postgres
+        image: registry.access.redhat.com/ubi9/ubi-minimal:latest
+        imagePullPolicy: IfNotPresent
+        command: [ 'sh', '-c', 'until (echo 1 > /dev/tcp/postgres.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local/5432) >/dev/null 2>&1; do echo "Waiting for postgres server"; sleep 3; done;' ]

Review Comment:
   Just as a quick note; we have a JIRA to implement this in case the workflow requires persistence: https://issues.redhat.com/browse/KOGITO-9840
   
   Can you please add this link with `TODO:` in this file? 🙏 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#discussion_r1461982828


##########
controllers/profiles/common/object_creators_test.go:
##########
@@ -169,3 +169,171 @@ func TestMergePodSpec_OverrideContainers(t *testing.T) {
 	assert.Equal(t, int32(8080), flowContainer.Ports[0].ContainerPort)
 	assert.Empty(t, flowContainer.Env)
 }
+
+func TestMergePodSpec_WithPostgreSQL_and_JDBC_URL_field(t *testing.T) {
+	workflow := test.GetBaseSonataFlow(t.Name())
+	workflow.Spec = v1alpha08.SonataFlowSpec{
+		PodTemplate: v1alpha08.PodTemplateSpec{
+			Container: v1alpha08.ContainerSpec{
+				// this one we can override
+				Image: "quay.io/example/my-workflow:1.0.0",
+				Ports: []corev1.ContainerPort{
+					// let's override a immutable attribute
+					{Name: utils.HttpScheme, ContainerPort: 9090},
+				},
+				Env: []corev1.EnvVar{
+					// We should be able to override this too
+					{Name: "ENV1", Value: "VALUE_CUSTOM"},
+				},
+				VolumeMounts: []corev1.VolumeMount{
+					{Name: "myvolume", ReadOnly: true, MountPath: "/tmp/any/path"},
+				},
+			},
+			PodSpec: v1alpha08.PodSpec{
+				ServiceAccountName: "superuser",
+				Containers: []corev1.Container{
+					{
+						Name: "sidecar",
+					},
+				},
+				Volumes: []corev1.Volume{
+					{
+						Name: "myvolume",
+						VolumeSource: corev1.VolumeSource{
+							ConfigMap: &corev1.ConfigMapVolumeSource{
+								LocalObjectReference: corev1.LocalObjectReference{Name: "customproperties"},
+							},
+						},
+					},
+				},
+			},
+		},
+		Persistence: &v1alpha08.PersistenceOptions{
+			PostgreSql: &v1alpha08.PersistencePostgreSql{
+				SecretRef: v1alpha08.PostgreSqlSecretOptions{Name: "test"},
+				JdbcUrl:   "jdbc:postgresql://host:port/database?currentSchema=workflow",
+			},
+		},
+	}
+
+	object, err := DeploymentCreator(workflow)
+	assert.NoError(t, err)
+
+	deployment := object.(*appsv1.Deployment)
+	expectedEnvVars := []corev1.EnvVar{
+		{
+			Name:  "ENV1",
+			Value: "VALUE_CUSTOM",
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_USERNAME",
+			Value: "",
+			ValueFrom: &corev1.EnvVarSource{
+				SecretKeyRef: &corev1.SecretKeySelector{
+					LocalObjectReference: corev1.LocalObjectReference{Name: "test"}, Key: "POSTGRESQL_USER",
+				},
+			},
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_PASSWORD",
+			Value: "",
+			ValueFrom: &corev1.EnvVarSource{
+				SecretKeyRef: &corev1.SecretKeySelector{
+					LocalObjectReference: corev1.LocalObjectReference{Name: "test"}, Key: "POSTGRESQL_PASSWORD",
+				},
+			},
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_DB_KIND",
+			Value: "postgresql",
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_JDBC_URL",
+			Value: "jdbc:postgresql://host:port/database?currentSchema=workflow",
+		},
+	}
+	assert.Len(t, deployment.Spec.Template.Spec.Containers, 2)
+	assert.Equal(t, "superuser", deployment.Spec.Template.Spec.ServiceAccountName)
+	assert.Len(t, deployment.Spec.Template.Spec.Volumes, 1)
+	flowContainer, _ := kubeutil.GetContainerByName(v1alpha08.DefaultContainerName, &deployment.Spec.Template.Spec)
+	assert.Equal(t, "quay.io/example/my-workflow:1.0.0", flowContainer.Image)
+	assert.Equal(t, int32(8080), flowContainer.Ports[0].ContainerPort)
+	assert.Equal(t, expectedEnvVars, flowContainer.Env)
+	assert.Len(t, flowContainer.VolumeMounts, 1)
+}
+
+var (
+	postgreSQLPort = 5432

Review Comment:
   No. Why?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#discussion_r1461988050


##########
controllers/profiles/common/persistence/postgreSQL.go:
##########
@@ -0,0 +1,89 @@
+// Copyright 2023 Apache Software Foundation (ASF)
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package persistence
+
+import (
+	"strconv"
+
+	corev1 "k8s.io/api/core/v1"
+
+	operatorapi "github.com/apache/incubator-kie-kogito-serverless-operator/api/v1alpha08"
+	"github.com/apache/incubator-kie-kogito-serverless-operator/controllers/profiles/common/constants"
+)
+
+const (
+	defaultSchemaName   = "default"
+	defaultDatabaseName = "sonataflow"
+)
+
+func ConfigurePostgreSqlEnv(postgresql *operatorapi.PersistencePostgreSql, databaseSchema, databaseNamespace string) []corev1.EnvVar {
+	dataSourcePort := constants.DefaultPostgreSQLPort
+	databaseName := defaultDatabaseName
+	dataSourceURL := postgresql.JdbcUrl
+	if postgresql.ServiceRef != nil {
+		if len(postgresql.ServiceRef.DatabaseSchema) > 0 {
+			databaseSchema = postgresql.ServiceRef.DatabaseSchema
+		}
+		if len(postgresql.ServiceRef.Namespace) > 0 {
+			databaseNamespace = postgresql.ServiceRef.Namespace
+		}
+		if postgresql.ServiceRef.Port != nil {
+			dataSourcePort = *postgresql.ServiceRef.Port
+		}
+		if len(postgresql.ServiceRef.DatabaseName) > 0 {
+			databaseName = postgresql.ServiceRef.DatabaseName
+		}
+		dataSourceURL = "jdbc:" + constants.PersistenceTypePostgreSQL + "://" + postgresql.ServiceRef.Name + "." + databaseNamespace + ":" + strconv.Itoa(dataSourcePort) + "/" + databaseName + "?currentSchema=" + databaseSchema
+	}
+	secretRef := corev1.LocalObjectReference{
+		Name: postgresql.SecretRef.Name,
+	}
+	quarkusDatasourceUsername := "POSTGRESQL_USER"
+	if len(postgresql.SecretRef.UserKey) > 0 {
+		quarkusDatasourceUsername = postgresql.SecretRef.UserKey
+	}
+	quarkusDatasourcePassword := "POSTGRESQL_PASSWORD"
+	if len(postgresql.SecretRef.PasswordKey) > 0 {
+		quarkusDatasourcePassword = postgresql.SecretRef.PasswordKey
+	}
+	return []corev1.EnvVar{
+		{
+			Name: "QUARKUS_DATASOURCE_USERNAME",
+			ValueFrom: &corev1.EnvVarSource{
+				SecretKeyRef: &corev1.SecretKeySelector{
+					Key:                  quarkusDatasourceUsername,
+					LocalObjectReference: secretRef,
+				},
+			},
+		},
+		{
+			Name: "QUARKUS_DATASOURCE_PASSWORD",
+			ValueFrom: &corev1.EnvVarSource{
+				SecretKeyRef: &corev1.SecretKeySelector{
+					Key:                  quarkusDatasourcePassword,
+					LocalObjectReference: secretRef,
+				},
+			},
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_DB_KIND",
+			Value: constants.PersistenceTypePostgreSQL,
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_JDBC_URL",
+			Value: dataSourceURL,
+		},

Review Comment:
   Moved to private constants



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "wmedvede (via GitHub)" <gi...@apache.org>.
wmedvede commented on PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#issuecomment-1910589348

   > Let's wait for @wmedvede approval. I'll also take a look on it tomorrow.
   
   @jordigilh  @ricardozanini 
   This PR was already approved from my side guys.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#issuecomment-1912097123

   @jordigilh can you please open a task to update the docs too? Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on code in PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#discussion_r1461991458


##########
controllers/profiles/common/object_creators_test.go:
##########
@@ -169,3 +169,171 @@ func TestMergePodSpec_OverrideContainers(t *testing.T) {
 	assert.Equal(t, int32(8080), flowContainer.Ports[0].ContainerPort)
 	assert.Empty(t, flowContainer.Env)
 }
+
+func TestMergePodSpec_WithPostgreSQL_and_JDBC_URL_field(t *testing.T) {
+	workflow := test.GetBaseSonataFlow(t.Name())
+	workflow.Spec = v1alpha08.SonataFlowSpec{
+		PodTemplate: v1alpha08.PodTemplateSpec{
+			Container: v1alpha08.ContainerSpec{
+				// this one we can override
+				Image: "quay.io/example/my-workflow:1.0.0",
+				Ports: []corev1.ContainerPort{
+					// let's override a immutable attribute
+					{Name: utils.HttpScheme, ContainerPort: 9090},
+				},
+				Env: []corev1.EnvVar{
+					// We should be able to override this too
+					{Name: "ENV1", Value: "VALUE_CUSTOM"},
+				},
+				VolumeMounts: []corev1.VolumeMount{
+					{Name: "myvolume", ReadOnly: true, MountPath: "/tmp/any/path"},
+				},
+			},
+			PodSpec: v1alpha08.PodSpec{
+				ServiceAccountName: "superuser",
+				Containers: []corev1.Container{
+					{
+						Name: "sidecar",
+					},
+				},
+				Volumes: []corev1.Volume{
+					{
+						Name: "myvolume",
+						VolumeSource: corev1.VolumeSource{
+							ConfigMap: &corev1.ConfigMapVolumeSource{
+								LocalObjectReference: corev1.LocalObjectReference{Name: "customproperties"},
+							},
+						},
+					},
+				},
+			},
+		},
+		Persistence: &v1alpha08.PersistenceOptions{
+			PostgreSql: &v1alpha08.PersistencePostgreSql{
+				SecretRef: v1alpha08.PostgreSqlSecretOptions{Name: "test"},
+				JdbcUrl:   "jdbc:postgresql://host:port/database?currentSchema=workflow",
+			},
+		},
+	}
+
+	object, err := DeploymentCreator(workflow)
+	assert.NoError(t, err)
+
+	deployment := object.(*appsv1.Deployment)
+	expectedEnvVars := []corev1.EnvVar{
+		{
+			Name:  "ENV1",
+			Value: "VALUE_CUSTOM",
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_USERNAME",
+			Value: "",
+			ValueFrom: &corev1.EnvVarSource{
+				SecretKeyRef: &corev1.SecretKeySelector{
+					LocalObjectReference: corev1.LocalObjectReference{Name: "test"}, Key: "POSTGRESQL_USER",
+				},
+			},
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_PASSWORD",
+			Value: "",
+			ValueFrom: &corev1.EnvVarSource{
+				SecretKeyRef: &corev1.SecretKeySelector{
+					LocalObjectReference: corev1.LocalObjectReference{Name: "test"}, Key: "POSTGRESQL_PASSWORD",
+				},
+			},
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_DB_KIND",
+			Value: "postgresql",
+		},
+		{
+			Name:  "QUARKUS_DATASOURCE_JDBC_URL",
+			Value: "jdbc:postgresql://host:port/database?currentSchema=workflow",
+		},
+	}
+	assert.Len(t, deployment.Spec.Template.Spec.Containers, 2)
+	assert.Equal(t, "superuser", deployment.Spec.Template.Spec.ServiceAccountName)
+	assert.Len(t, deployment.Spec.Template.Spec.Volumes, 1)
+	flowContainer, _ := kubeutil.GetContainerByName(v1alpha08.DefaultContainerName, &deployment.Spec.Template.Spec)
+	assert.Equal(t, "quay.io/example/my-workflow:1.0.0", flowContainer.Image)
+	assert.Equal(t, int32(8080), flowContainer.Ports[0].ContainerPort)
+	assert.Equal(t, expectedEnvVars, flowContainer.Env)
+	assert.Len(t, flowContainer.VolumeMounts, 1)
+}
+
+var (
+	postgreSQLPort = 5432

Review Comment:
   I'd keep it within the test scope then.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "wmedvede (via GitHub)" <gi...@apache.org>.
wmedvede commented on code in PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#discussion_r1462044751


##########
test/testdata/persistence/workflow/by_service/02-sonataflow_platform.yaml:
##########
@@ -0,0 +1,27 @@
+# Copyright 2024 Apache Software Foundation (ASF)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlowPlatform
+metadata:
+  name: sonataflow-platform
+spec:
+  build:
+    template:
+      buildArgs:
+      - name: QUARKUS_EXTENSIONS
+        value: org.kie.kogito:kogito-addons-quarkus-jobs-knative-eventing:999-SNAPSHOT,org.kie.kogito:kogito-addons-quarkus-persistence-jdbc:999-SNAPSHOT,io.quarkus:quarkus-jdbc-postgresql:3.2.9.Final,io.quarkus:quarkus-agroal:3.2.9.Final

Review Comment:
   I think we need to assess here, because the persistence is a sort of all or nothing. I mean, if we add the org.kie.kogito:kogito-addons-quarkus-persistence-jdbc:999-SNAPSHOT to the swf-builder, then, every w the workflow must use persistence, otherwise we'll have strart-up failures.
   
   O think we have the following scenarios:
   
   1) The images is produced externally:
   The workflow must come already built with the proper extensions incorporated in the produced image, and users might decide in a per workflow basis to include the persistence related addons. Finally the operator just relies on the workflow CRD con connect with the DB.
   
   2) The Image is produced by the operator managed Kaniko or OpenShift based builds, here
   
   But we are dealing with the persistence as an optional thing, so, it could happen the a workflow CRD with no configured persistence might come.
   
   
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "masayag (via GitHub)" <gi...@apache.org>.
masayag commented on code in PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#discussion_r1426897410


##########
api/v1alpha08/sonataflow_types.go:
##########
@@ -657,6 +657,9 @@ type SonataFlowSpec struct {
 	// PodTemplate describes the deployment details of this SonataFlow instance.
 	//+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="podTemplate"
 	PodTemplate PodTemplateSpec `json:"podTemplate,omitempty"`
+	// Persists service to a datasource of choice. Ephemeral by default.
+	// +optional
+	Persistence *PersistenceOptions `json:"persistence,omitempty"`

Review Comment:
   Ok, that makes sense then.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "domhanak (via GitHub)" <gi...@apache.org>.
domhanak commented on PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#issuecomment-1891514988

   @jordigilh another rebase should solve the test issues. Other than that it looks good.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9982] Add Persistence field in SonataFlow CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on PR #322:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/322#issuecomment-1899464478

   @ricardozanini ready to be merged. Please note that I had to remove the `DEBUG=true` env value in the job because it caused the `kubectl` commands to dump extra log entries and that caused errors when parsing the output as it did not match the expected output.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org