You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kie.apache.org by "jordigilh (via GitHub)" <gi...@apache.org> on 2024/01/26 16:24:27 UTC

[PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

jordigilh opened a new pull request, #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372

   Adds a new field in the `Spec` struct to allow defining a shareable postgreSQL persistence configuration for workflows and to platform services:
   https://github.com/jordigilh/incubator-kie-kogito-serverless-operator/blob/7f96442d27ae773d23cab74e84423b069dfe2f90/api/v1alpha08/sonataflowplatform_types.go#L49-L53
   
   This configuration is inherited by workflows that require persistence when the persistence field in the workflow spec is initialized as empty.
   https://github.com/jordigilh/incubator-kie-kogito-serverless-operator/blob/ee06ddd88a03658dace236574108c2f446454186/controllers/profiles/common/object_creators_test.go#L383-L385
   
   When both workflow CR and platform CR contain persistence, the workflow CR specification [will take precedence](https://github.com/jordigilh/incubator-kie-kogito-serverless-operator/blob/ee06ddd88a03658dace236574108c2f446454186/controllers/profiles/common/object_creators_test.go#L437).
   
   To keep backwards compatibility, if the `Persistence` field in the workflow CR is nil, no persistence will be provided to it.
   
   Extends e2e tests with one where the workflow CR retrieves the persistence from the platform.
   
   @ricardozanini @wmedvede @tchughesiv @domhanak PTAL.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1947352455

   @tchughesiv we should remove it if it won't be used. Having an attribute that can be set and then set an error saying that must not be set smells.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478009903


##########
controllers/profiles/common/persistence/postgresql.go:
##########
@@ -0,0 +1,203 @@
+// Copyright 2023 Apache Software Foundation (ASF)
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package persistence
+
+import (
+	"fmt"
+	"sync"
+
+	corev1 "k8s.io/api/core/v1"
+
+	operatorapi "github.com/apache/incubator-kie-kogito-serverless-operator/api/v1alpha08"
+	"github.com/apache/incubator-kie-kogito-serverless-operator/controllers/profiles/common/constants"
+)
+
+const (
+	defaultSchemaName   = "default"
+	defaultDatabaseName = "sonataflow"
+
+	timeoutSeconds       = 3
+	failureThreshold     = 5
+	initialPeriodSeconds = 15
+	initialDelaySeconds  = 10
+	successThreshold     = 1
+
+	postgreSQLCPULimit      = "500m"
+	postgreSQLMemoryLimit   = "256Mi"
+	postgreSQLMemoryRequest = "256Mi"
+	postgreSQLCPURequest    = "100m"
+
+	defaultPostgreSQLUsername  = "sonataflow"
+	defaultPostgresSQLPassword = "sonataflow"
+)
+
+var (
+	WorkflowConfig *syncConfig

Review Comment:
   Reimplemented this by retrieving the platform CR on request, rather than catching it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478010452


##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -46,6 +46,22 @@ type SonataFlowPlatformSpec struct {
 	// `sonataflow.org/profile: prod`
 	// +optional
 	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PlatformPersistenceSpec `json:"persistence,omitempty"`

Review Comment:
   Sure, I'll rename it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478519233


##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -46,6 +46,22 @@ type SonataFlowPlatformSpec struct {
 	// `sonataflow.org/profile: prod`
 	// +optional
 	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PlatformPersistenceSpec `json:"persistence,omitempty"`

Review Comment:
   Updated to `PersistenceOptionsSpec`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474472019


##########
controllers/platform/services/services.go:
##########
@@ -138,11 +138,23 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||

Review Comment:
   Yes, golang's coding convention [states](https://google.github.io/styleguide/go/decisions.html#initialisms) that initialisms should be capitalized in this case, not like in Java that is camel case all the way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484399060


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   > In this case here, labels are well suited since we are using an enum to validate and it clearly expresses the operator's behavior. Also, @wmedvede @tchughesiv any thoughts?
   
   In this case, the enum is validated in the operator. The approach I'm suggesting is validated by the API server and it rejects the CR if it fails the validation, which I think it's better for the user and simplifies the logic in the operator.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1946209500

   > > > In this other use case below, we are using a SFP with the global persistence configured to use the JdbcUrl to connect with the database service.
   > > > https://github.com/flows-examples/techpreview2/tree/004b37b96ad5f75c804d8e8ba3d821ab67c86457/platforms/openshift-dataindex_and_jobservice_as_platform_service_and_global_persistence_jdbcurl_postgresql
   > > > ```
   > > > 
   > > >   persistence:
   > > > 
   > > >     postgresql:
   > > > 
   > > >       secretRef:
   > > > 
   > > >         name: postgres-secrets
   > > > 
   > > >         userKey: POSTGRESQL_USER
   > > > 
   > > >         passwordKey: POSTGRESQL_PASSWORD
   > > > 
   > > >       jdbcUrl: jdbc:postgresql://postgres.usecase3-platform-persistence-jdbcurl-oc:5432/sonataflow2
   > > > ```
   > > > 
   > > > 
   > > >     
   > > >       
   > > >     
   > > > 
   > > >       
   > > >     
   > > > 
   > > >     
   > > >   
   > > > But, the corresponding generated specific jdbc url for each service looks like the ones below.
   > > > I think we need a small adjustment to make each service use its specific schema, otherwise we have flyway issues when the services start. Not too much, a simple adjustment.
   > > > **DataIndex:**
   > > > ```
   > > > QUARKUS_DATASOURCE_JDBC_URL = jdbc:postgresql://postgres.usecase3-platform-persistence-jdbcurl-oc:5432/sonataflow2
   > > > ```
   > > > 
   > > > 
   > > >     
   > > >       
   > > >     
   > > > 
   > > >       
   > > >     
   > > > 
   > > >     
   > > >   
   > > > **JobsService:**
   > > > ```
   > > > QUARKUS_DATASOURCE_JDBC_URL = jdbc:postgresql://postgres.usecase3-platform-persistence-jdbcurl-oc:5432/sonataflow2
   > > > ```
   > > 
   > > 
   > > That's because when I posted about the schema being included in the JbdcUrl in slack, it was suggested we should not do anything to remove the schema from it and instead warn the user in the documentation that it can cause problems when defined in the platform.
   > 
   > ok, looks like I missed these comments. Maybe, if JdbcUrl is something that people can't use at platform level, we can throw an error in that case 🤔
   
   is it a 100% certainty that `JdbcUrl` can't be used in any capacity at the `platform.spec` level? if so, then yes... we either need to remove it from the `PlatformPersistenceOptionsSpec` (this is preferable... like we already did for databaseSchema) OR document it i guess.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484451946


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   > * After this PR is merged, then `persistence=nil` means it might have persistence depending on the platform, whether the workflow requested it or not.
   
   but it won't, because existing platform CRs won't have the `spec.persistence` feild set. it's a new field. hence, existing workflows won't inherit persistence unless the user decides to enable it either within the workflow or the existing platform. we should be good to go, no backwards compat issues.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484285291


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   imo @masayag's interpretation is how i've generally approached these things and seen them implemented in k8s.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "wmedvede (via GitHub)" <gi...@apache.org>.
wmedvede commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1938244531

   > > In this other use case below, we are using a SFP with the global persistence configured to use the JdbcUrl to connect with the database service.
   > > https://github.com/flows-examples/techpreview2/tree/004b37b96ad5f75c804d8e8ba3d821ab67c86457/platforms/openshift-dataindex_and_jobservice_as_platform_service_and_global_persistence_jdbcurl_postgresql
   > > ```
   > > 
   > >   persistence:
   > > 
   > >     postgresql:
   > > 
   > >       secretRef:
   > > 
   > >         name: postgres-secrets
   > > 
   > >         userKey: POSTGRESQL_USER
   > > 
   > >         passwordKey: POSTGRESQL_PASSWORD
   > > 
   > >       jdbcUrl: jdbc:postgresql://postgres.usecase3-platform-persistence-jdbcurl-oc:5432/sonataflow2
   > > ```
   > > 
   > > 
   > >     
   > >       
   > >     
   > > 
   > >       
   > >     
   > > 
   > >     
   > >   
   > > But, the corresponding generated specific jdbc url for each service looks like the ones below.
   > > I think we need a small adjustment to make each service use its specific schema, otherwise we have flyway issues when the services start. Not too much, a simple adjustment.
   > > **DataIndex:**
   > > ```
   > > QUARKUS_DATASOURCE_JDBC_URL = jdbc:postgresql://postgres.usecase3-platform-persistence-jdbcurl-oc:5432/sonataflow2
   > > ```
   > > 
   > > 
   > >     
   > >       
   > >     
   > > 
   > >       
   > >     
   > > 
   > >     
   > >   
   > > **JobsService:**
   > > ```
   > > QUARKUS_DATASOURCE_JDBC_URL = jdbc:postgresql://postgres.usecase3-platform-persistence-jdbcurl-oc:5432/sonataflow2
   > > ```
   > 
   > That's because when I posted about the schema being included in the JbdcUrl in slack, it was suggested we should not do anything to remove the schema from it and instead warn the user in the documentation that it can cause problems when defined in the platform.
   
   ok, looks like I missed these comments. Maybe, if JdbcUrl is something that people can't use at platform level, we can throw an error in that case :thinking:  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1968831922

   @jordigilh can you check the code gen?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1479805683


##########
controllers/platform/services/services.go:
##########
@@ -358,14 +371,26 @@ func (j JobServiceHandler) MergeContainerSpec(containerSpec *corev1.Container) (
 	return c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Job service specification is not nil
+func (j JobServiceHandler) hasPostgreSQLConfigured() bool {
+	return (j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil) ||
+		(j.platform.Spec.Persistence != nil && j.platform.Spec.Persistence.PostgreSQL != nil)
+}
+
 func (j JobServiceHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
 
-	if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSql != nil {
+	if j.hasPostgreSQLConfigured() {
+		var p *operatorapi.PersistencePostgreSQL
 		c := containerSpec.DeepCopy()
 		c.Image = j.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(j.platform.Spec.Services.JobService.Persistence.PostgreSql, j.GetServiceName(), j.platform.Namespace)...)
+		if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil {

Review Comment:
   assuming you add the proper checks to `hasPostgreSQLConfigured` then yes, agreed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478463453


##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -45,7 +45,23 @@ type SonataFlowPlatformSpec struct {
 	// Only workflows without the `sonataflow.org/profile: dev` annotation will be configured to use these service(s).
 	// Setting this will override the use of any cluster-scoped services that might be defined via `SonataFlowClusterPlatform`.
 	// +optional
-	Services *ServicesPlatformSpec `json:"services,omitempty"`
+	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PersistencePlatformSpec `json:"persistence,omitempty"`
+}
+
+// PersistencePlatformSpec configures the DataBase support for both platform services and workflows. For services, it allows
+// configuring a generic database connectivity if the service does not come with its own configured. In case of workflows,
+// the operator will add the necessary JDBC properties to in the workflow's application.properties so that it can communicate
+// with the persistence service based on the spec provided here.
+// +optional
+type PersistencePlatformSpec struct {

Review Comment:
   Will rename



##########
controllers/platform/services/properties_services_test.go:
##########
@@ -173,9 +173,6 @@ func generatePlatform(opts ...plfmOptionFn) *operatorapi.SonataFlowPlatform {
 
 func setJobServiceEnabledValue(v *bool) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   ack



##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -45,7 +45,23 @@ type SonataFlowPlatformSpec struct {
 	// Only workflows without the `sonataflow.org/profile: dev` annotation will be configured to use these service(s).
 	// Setting this will override the use of any cluster-scoped services that might be defined via `SonataFlowClusterPlatform`.
 	// +optional
-	Services *ServicesPlatformSpec `json:"services,omitempty"`

Review Comment:
   Not sure how this came here, definitely something I did by mistake but not a problem to revert it.



##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   There are 3 scenarios managed here:
   * Persistence field in the workflow spec is nil: If the persistence field is nil, then the workflow will have no persistence at all. 
   * Persistence is empty (`{}`): Persistence will be pull from the platform if it exists.
   * Persistence is populated: Use the one in workflow.
   
   Let me know if that makes sense in this case.



##########
controllers/platform/platformutils.go:
##########
@@ -111,15 +111,13 @@ func setPlatformDefaults(p *operatorapi.SonataFlowPlatform, verbose bool) error
 	}
 
 	// When dataIndex object set, default to enabled if bool not set
-	if p.Spec.Services != nil {
-		var enable = true
-		if p.Spec.Services.DataIndex != nil && p.Spec.Services.DataIndex.Enabled == nil {
-			p.Spec.Services.DataIndex.Enabled = &enable
-		}
-		// When the JobService field has a value, default to enabled if the `Enabled` field's value is nil
-		if p.Spec.Services.JobService != nil && p.Spec.Services.JobService.Enabled == nil {
-			p.Spec.Services.JobService.Enabled = &enable
-		}

Review Comment:
   ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474472019


##########
controllers/platform/services/services.go:
##########
@@ -138,11 +138,23 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||

Review Comment:
   Yes, golang's coding convention [states](https://google.github.io/styleguide/go/decisions.html#initialisms) that initialisms should be capitalized, not like in Java that is camel case all the way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474597044


##########
controllers/profiles/common/object_creators.go:
##########
@@ -218,10 +226,20 @@ func WorkflowPropsConfigMapCreator(workflow *operatorapi.SonataFlow) (client.Obj
 	return workflowproj.CreateNewAppPropsConfigMap(workflow, props), nil
 }
 
-func ConfigurePersistence(serviceContainer *corev1.Container, options *operatorapi.PersistenceOptions, defaultSchema, namespace string) *corev1.Container {
+func ConfigurePersistence(serviceContainer *corev1.Container, config *operatorapi.PersistenceOptions, defaultSchema, namespace string) (*corev1.Container, error) {
+	if config == nil {
+		return serviceContainer, nil
+	}
 	c := serviceContainer.DeepCopy()
-	if options != nil && options.PostgreSql != nil {
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(options.PostgreSql, defaultSchema, namespace)...)
+
+	if config.PostgreSql != nil {
+		c.Env = append(c.Env, persistence.ConfigurePostgreSQLEnv(config.PostgreSql, defaultSchema, namespace)...)
+		return c, nil
+	}
+	p := persistence.WorkflowConfig.GetPostgreSQLConfiguration()

Review Comment:
   This is merely means the platform object has not been marked as "Duplicated" in its status. Additional platforms added to a namespace will be marked as such and do not progress through the full reconciliation process. 
   
   if you need to retrieve the active platform, there should be a GetActivePlatform function you can use. Or, hopefully you can just use the service handlers, which will alrewady have the active platform attached to the interface... e.g. `d.platform`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474742580


##########
controllers/platform/services/services.go:
##########
@@ -138,11 +138,23 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||

Review Comment:
   @ricardozanini i've created an issue to address these concerns. https://github.com/apache/incubator-kie-kogito-serverless-operator/issues/380



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1480381361


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   Actually there is the `enabled` field in the service that is set to true if the service is defined (not nil) by the operator. It's something we're doing already in that sense: define an empty service for DI and the operator will populate it with the default configuration, in this case `enabled=true`.
   
   ~I'm fine using labels as well~. Consistency is important at the end, so if labels/annotations is the path forward for these cases, then this should be the approach accross whenever facing these situations.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484285291


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   imo @masayag's interpretation is how i've generally approached these things and seen them implemented in k8s. a user can "override" the platform at a workflow level by setting persistence in the sonataflow CR. if persistence is nil in the workflow, settings are pulled from platform CR. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "wmedvede (via GitHub)" <gi...@apache.org>.
wmedvede commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1936407277

   @jordigilh  @ricardozanini  Let me add some positive feedback:
   
   In the usecase below, we are using a SFP with a global persistence configuration that we can see here:
   
   []([](https://github.com/flows-examples/techpreview2/tree/main/platforms/openshift-dataindex_and_jobservice_as_platform_service_and_global_persistence_postgresql))
   
   Readme here:  https://github.com/flows-examples/techpreview2/blob/main/platforms/openshift-dataindex_and_jobservice_as_platform_service_and_global_persistence_postgresql/Readme.txt
   
   **Summary:**
   
   1. **SPF looks like this, and we expect DI and JS to use that configuration:**
   
   ```
     persistence:
       postgresql:
         secretRef:
           name: postgres-secrets
           userKey: POSTGRESQL_USER
           passwordKey: POSTGRESQL_PASSWORD
         serviceRef:
           name: postgres
           databaseName: sonataflow2
   ```
   
   2. **DI and JS started ok, and both services use their own schema as expected. Good man! @jordigilh** 
   
   3. **callbackstatetimeouts WF has started and worked ok, with a persistence configuration like this Good man!**
   
   ```
   apiVersion: sonataflow.org/v1alpha08
   kind: SonataFlow
   metadata:
     name: callbackstatetimeouts
     annotations:
       sonataflow.org/description: Callback State Timeouts Example k8s
       sonataflow.org/version: 0.0.1
   spec:
     persistence: {}
     flow:
       start: PrintStartMessage
   ```
   
   4. **workflowtimeouts WF, with the configuration below has failed to deploy:**
   Here we would expect that the WF picks the platform persistence configuration and use it's own configured schema wf_timeouts.
   
   But the following error was produced at deployment. (Please @jordigilh correct me if the configuration is wrong.
   
   `The SonataFlow "workflowtimeouts" is invalid: spec.persistence.postgresql: Invalid value: 1: spec.persistence.postgresql in body should have at least 2 properties`
   
   
   ```
   apiVersion: sonataflow.org/v1alpha08
   kind: SonataFlow
   metadata:
     name: workflowtimeouts
     annotations:
       sonataflow.org/description: Workflow Timeouts Example k8s
       sonataflow.org/version: 0.0.1
   spec:
     persistence:
       postgresql:
         serviceRef:
           databaseSchema: wf_timeouts
   ```
   
   
   
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1936473807

   > 4. **workflowtimeouts WF, with the configuration below has failed to deploy:**
   >    Here we would expect that the WF picks the platform persistence configuration and use it's own configured schema wf_timeouts.
   > 
   > But the following error was produced at deployment. (Please @jordigilh correct me if the configuration is wrong.
   > 
   > `The SonataFlow "workflowtimeouts" is invalid: spec.persistence.postgresql: Invalid value: 1: spec.persistence.postgresql in body should have at least 2 properties`
   > 
   > ```
   > apiVersion: sonataflow.org/v1alpha08
   > kind: SonataFlow
   > metadata:
   >   name: workflowtimeouts
   >   annotations:
   >     sonataflow.org/description: Workflow Timeouts Example k8s
   >     sonataflow.org/version: 0.0.1
   > spec:
   >   persistence:
   >     postgresql:
   >       serviceRef:
   >         databaseSchema: wf_timeouts
   > ```
   
   @wmedvede this is because `secretRef` is required here, as well as either the `serviceRef` or `jdbcUrl` field. Just add `secretRef` and the error should go away.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478444169


##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -45,7 +45,23 @@ type SonataFlowPlatformSpec struct {
 	// Only workflows without the `sonataflow.org/profile: dev` annotation will be configured to use these service(s).
 	// Setting this will override the use of any cluster-scoped services that might be defined via `SonataFlowClusterPlatform`.
 	// +optional
-	Services *ServicesPlatformSpec `json:"services,omitempty"`

Review Comment:
   this should not be changed. please revert this back to a pointer object. it is critical that this be a pointer object in order for the cluster-wide services logic to work as expected.
   
   why are we making changes to this? it has nothing to do w/ the PR.



##########
config/manager/kustomization.yaml:
##########
@@ -16,7 +16,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 images:
 - name: controller
-  newName: quay.io/kiegroup/kogito-serverless-operator-nightly

Review Comment:
   please revert this change



##########
controllers/platform/services/properties_services_test.go:
##########
@@ -208,9 +199,6 @@ func emptyDataIndexServiceSpec() plfmOptionFn {
 
 func emptyJobServiceSpec() plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   same



##########
controllers/platform/services/properties_services_test.go:
##########
@@ -185,9 +182,6 @@ func setJobServiceEnabledValue(v *bool) plfmOptionFn {
 
 func setDataIndexEnabledValue(v *bool) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   same



##########
controllers/platform/platformutils.go:
##########
@@ -111,15 +111,13 @@ func setPlatformDefaults(p *operatorapi.SonataFlowPlatform, verbose bool) error
 	}
 
 	// When dataIndex object set, default to enabled if bool not set
-	if p.Spec.Services != nil {
-		var enable = true
-		if p.Spec.Services.DataIndex != nil && p.Spec.Services.DataIndex.Enabled == nil {
-			p.Spec.Services.DataIndex.Enabled = &enable
-		}
-		// When the JobService field has a value, default to enabled if the `Enabled` field's value is nil
-		if p.Spec.Services.JobService != nil && p.Spec.Services.JobService.Enabled == nil {
-			p.Spec.Services.JobService.Enabled = &enable
-		}

Review Comment:
   please revert these changes



##########
controllers/platform/services/properties_services_test.go:
##########
@@ -231,36 +219,30 @@ func setPlatformName(name string) plfmOptionFn {
 
 func setJobServiceJDBC(jdbc string) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   same



##########
controllers/platform/services/properties_services_test.go:
##########
@@ -173,9 +173,6 @@ func generatePlatform(opts ...plfmOptionFn) *operatorapi.SonataFlowPlatform {
 
 func setJobServiceEnabledValue(v *bool) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   revert



##########
controllers/platform/services/properties_services_test.go:
##########
@@ -197,9 +191,6 @@ func setDataIndexEnabledValue(v *bool) plfmOptionFn {
 
 func emptyDataIndexServiceSpec() plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   same



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1505086121


##########
test/e2e/workflow_test.go:
##########
@@ -198,16 +198,27 @@ var _ = Describe("Validate the persistence ", Ordered, func() {
 				}
 				Expect(h.Status).To(Equal(upStatus), "Pod health is not UP")
 				for _, c := range h.Checks {
-					if c.Name == "Database connections health check" {
-						Expect(c.Status).To(Equal(upStatus), "Pod's database connection is not UP")
-						return true
+					if withPersistence {
+						if c.Name == "Database connections health check" {
+							Expect(c.Status).To(Equal(upStatus), "Pod's database connection is not UP")
+							return true
+						}
+					} else {
+						Expect(c.Name).NotTo(Equal("Database connections health check"), "persistence configuration found in workflow")
 					}
 				}
+				if !withPersistence {
+					return true
+				}
 			}
 			return false
-		}, 12*time.Minute).Should(BeTrue())
+		}, 1).Should(BeTrue())

Review Comment:
   Note that 1 seems to be `1 second`. 
   https://github.com/onsi/gomega/blob/32ef35ea6e233acc84502d9ec90c106aeec1615d/gomega_dsl.go#L245
   I will add a `*time.Minute` back.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1944403429

   @wmedvede @ricardozanini @tchughesiv changes made to logic so that `persistence: nil` in the workflow CR will instruct the operator to derive the persistence from the platform, if it exists. If `persistence: {}`, the operator will not include any persistence in the workflow. I've also added a new e2e scenario to cover for this last scenario.
   
   If you have no more concerns, I suggest to merge this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484431905


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   @ricardozanini here's one example https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/cpu-default-namespace/



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "masayag (via GitHub)" <gi...@apache.org>.
masayag commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1481012139


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   I'd look at this as:
   * No persistence configured: `.spec.persistence: {}`
     * Sets explicitly the empty value for  persistence.
   * Use workflow's persistence configuration: `.spec.persistence: <data>`
   * Use the system/platform default: `.spec.persistence: nil`
     * By omitting this, the decision is propagated to platform/cluster
     
   +1 for consistency (always).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1480322234


##########
test/testdata/workflow/persistence/by_service/02-sonataflow_platform.yaml:
##########
@@ -21,7 +21,7 @@ spec:
     template:
       buildArgs:
       - name: QUARKUS_EXTENSIONS
-        value: org.kie.kogito:kogito-addons-quarkus-jobs-knative-eventing:999-SNAPSHOT,org.kie.kogito:kogito-addons-quarkus-persistence-jdbc:999-SNAPSHOT,io.quarkus:quarkus-jdbc-postgresql:3.2.9.Final,io.quarkus:quarkus-agroal:3.2.9.Final
+        value: org.kie.kogito:kogito-addons-quarkus-persistence-jdbc:999-SNAPSHOT,io.quarkus:quarkus-jdbc-postgresql:3.2.9.Final,io.quarkus:quarkus-agroal:3.2.9.Final

Review Comment:
   I guess we can add these values by default when persistence is on. @jordigilh can you please open a follow-up issue? I'll do an assessment.



##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   It makes sense. But what about adding a label to address this problem `sonataflow.org/persistence=platform|workflow|none`, defaults to none.
   
   The problem with using this approach, @jordigilh @tchughesiv is that the API is not communicating what will be done. It's an error prune having `.spec.persistence: {}` meaning that we will pull from the platform.
   
   On the other hand, having `.spec.persistence: <data>` clears communicate that we are overriding. But `.spec.persistence: {}` or not having this attribute defined is dubious IMO. One can interpret that it will pull from platform if the attribute is not there.



##########
controllers/profiles/dev/states_dev.go:
##########
@@ -70,15 +73,14 @@ func (e *ensureRunningWorkflowState) Do(ctx context.Context, workflow *operatora
 
 	devBaseContainerImage := workflowdef.GetDefaultWorkflowDevModeImageTag()
 	// check if the Platform available
-	pl, err := platform.GetActivePlatform(ctx, e.C, workflow.Namespace)
-	if err == nil && len(pl.Spec.DevMode.BaseImage) > 0 {
-		devBaseContainerImage = pl.Spec.DevMode.BaseImage
+	if plf != nil && len(plf.Spec.DevMode.BaseImage) > 0 {
+		devBaseContainerImage = plf.Spec.DevMode.BaseImage
 	}
 	userPropsCM, _, err := e.ensurers.userPropsConfigMap.Ensure(ctx, workflow)
 	if err != nil {
 		return ctrl.Result{Requeue: false}, objs, err
 	}
-	managedPropsCM, _, err := e.ensurers.managedPropsConfigMap.Ensure(ctx, workflow, pl, common.ManagedPropertiesMutateVisitor(ctx, e.StateSupport.Catalog, workflow, pl, userPropsCM.(*corev1.ConfigMap)))
+	managedPropsCM, _, err := e.ensurers.managedPropsConfigMap.Ensure(ctx, workflow, plf, common.ManagedPropertiesMutateVisitor(ctx, e.StateSupport.Catalog, workflow, plf, userPropsCM.(*corev1.ConfigMap)))

Review Comment:
   Why have you changed to `pl` in some cases and here you use `plf`? 😄 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474565878


##########
controllers/profiles/common/object_creators.go:
##########
@@ -218,10 +226,20 @@ func WorkflowPropsConfigMapCreator(workflow *operatorapi.SonataFlow) (client.Obj
 	return workflowproj.CreateNewAppPropsConfigMap(workflow, props), nil
 }
 
-func ConfigurePersistence(serviceContainer *corev1.Container, options *operatorapi.PersistenceOptions, defaultSchema, namespace string) *corev1.Container {
+func ConfigurePersistence(serviceContainer *corev1.Container, config *operatorapi.PersistenceOptions, defaultSchema, namespace string) (*corev1.Container, error) {
+	if config == nil {
+		return serviceContainer, nil
+	}
 	c := serviceContainer.DeepCopy()
-	if options != nil && options.PostgreSql != nil {
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(options.PostgreSql, defaultSchema, namespace)...)
+
+	if config.PostgreSql != nil {
+		c.Env = append(c.Env, persistence.ConfigurePostgreSQLEnv(config.PostgreSql, defaultSchema, namespace)...)
+		return c, nil
+	}
+	p := persistence.WorkflowConfig.GetPostgreSQLConfiguration()

Review Comment:
   I see. Is there a label or annotation for "Active" added to the CR? I don't recall seeing that logic in the code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474561810


##########
controllers/platform/initialize.go:
##########
@@ -106,6 +107,13 @@ func (action *initializeAction) Handle(ctx context.Context, platform *operatorap
 	}
 	platform.Status.Version = metadata.SpecVersion
 
+	// store workflow persistence configuration
+	if platform.Spec.Persistence != nil {
+		persistence.WorkflowConfig.SetConfig(platform.Spec.Persistence)

Review Comment:
   we should only ever use the "Active" ns when configuring services & workflows.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1479358745


##########
controllers/platform/services/services.go:
##########
@@ -198,11 +198,24 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil) ||
+		(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil)
+}

Review Comment:
   Alright I'll add it just in case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1479365288


##########
controllers/platform/services/services.go:
##########
@@ -358,14 +371,26 @@ func (j JobServiceHandler) MergeContainerSpec(containerSpec *corev1.Container) (
 	return c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Job service specification is not nil
+func (j JobServiceHandler) hasPostgreSQLConfigured() bool {
+	return (j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil) ||
+		(j.platform.Spec.Persistence != nil && j.platform.Spec.Persistence.PostgreSQL != nil)
+}
+
 func (j JobServiceHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
 
-	if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSql != nil {
+	if j.hasPostgreSQLConfigured() {
+		var p *operatorapi.PersistencePostgreSQL
 		c := containerSpec.DeepCopy()
 		c.Image = j.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(j.platform.Spec.Services.JobService.Persistence.PostgreSql, j.GetServiceName(), j.platform.Namespace)...)
+		if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil {

Review Comment:
   Not sure because adding it to `hasPostgreSQLConfigured()` already validates it in line 382:
   https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372/files/95b390b7469f45c3cfad661dd5d1f845757fcdc1#diff-9168d999a0cd028523626009a2534c70de62a6e4322770f8bc21d8fc01fc2b83R382



##########
controllers/platform/services/services.go:
##########
@@ -358,14 +371,26 @@ func (j JobServiceHandler) MergeContainerSpec(containerSpec *corev1.Container) (
 	return c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Job service specification is not nil
+func (j JobServiceHandler) hasPostgreSQLConfigured() bool {
+	return (j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil) ||
+		(j.platform.Spec.Persistence != nil && j.platform.Spec.Persistence.PostgreSQL != nil)
+}

Review Comment:
   Will add



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1491647922


##########
controllers/platform/services/services.go:
##########
@@ -358,14 +366,35 @@ func (j JobServiceHandler) MergeContainerSpec(containerSpec *corev1.Container) (
 	return c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Job service specification is not nil
+func (j JobServiceHandler) hasPostgreSQLConfigured() bool {
+	return j.IsServiceSetInSpec() &&
+		((j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil) ||
+			(j.platform.Spec.Persistence != nil && j.platform.Spec.Persistence.PostgreSQL != nil))
+}
+
 func (j JobServiceHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
 
-	if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSql != nil {
+	if j.hasPostgreSQLConfigured() {
+		var p *operatorapi.PersistencePostgreSQL
 		c := containerSpec.DeepCopy()
 		c.Image = j.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(j.platform.Spec.Services.JobService.Persistence.PostgreSql, j.GetServiceName(), j.platform.Namespace)...)
+		if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil {
+			p = j.platform.Spec.Services.JobService.Persistence.PostgreSQL
+		} else {
+			p = &operatorapi.PersistencePostgreSQL{
+				SecretRef: j.platform.Spec.Persistence.PostgreSQL.SecretRef,
+				JdbcUrl:   j.platform.Spec.Persistence.PostgreSQL.JdbcUrl,
+			}
+			if j.platform.Spec.Persistence.PostgreSQL.ServiceRef != nil {
+				p.ServiceRef = &operatorapi.PostgreSQLServiceOptions{
+					SQLServiceOptions: j.platform.Spec.Persistence.PostgreSQL.ServiceRef,
+					DatabaseSchema:    j.GetServiceName()}
+			}
+		}

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1480016001


##########
controllers/platform/services/services.go:
##########
@@ -358,14 +371,26 @@ func (j JobServiceHandler) MergeContainerSpec(containerSpec *corev1.Container) (
 	return c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Job service specification is not nil
+func (j JobServiceHandler) hasPostgreSQLConfigured() bool {
+	return (j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil) ||
+		(j.platform.Spec.Persistence != nil && j.platform.Spec.Persistence.PostgreSQL != nil)
+}
+
 func (j JobServiceHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
 
-	if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSql != nil {
+	if j.hasPostgreSQLConfigured() {
+		var p *operatorapi.PersistencePostgreSQL
 		c := containerSpec.DeepCopy()
 		c.Image = j.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(j.platform.Spec.Services.JobService.Persistence.PostgreSql, j.GetServiceName(), j.platform.Namespace)...)
+		if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil {

Review Comment:
   Yes, it's hopefully on the last commit: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372/commits/90af05ec683c1c6ac00dd4715314dbd3e265e4fd



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478454966


##########
controllers/platform/services/properties_services_test.go:
##########
@@ -231,36 +219,30 @@ func setPlatformName(name string) plfmOptionFn {
 
 func setJobServiceJDBC(jdbc string) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}
 		if p.Spec.Services.JobService == nil {
 			p.Spec.Services.JobService = &operatorapi.ServiceSpec{}
 		}
 		if p.Spec.Services.JobService.Persistence == nil {
-			p.Spec.Services.JobService.Persistence = &operatorapi.PersistenceOptions{}
+			p.Spec.Services.JobService.Persistence = &operatorapi.PersistencePlatformSpec{}
 		}
-		if p.Spec.Services.JobService.Persistence.PostgreSql == nil {
-			p.Spec.Services.JobService.Persistence.PostgreSql = &operatorapi.PersistencePostgreSql{}
+		if p.Spec.Services.JobService.Persistence.PostgreSQL == nil {
+			p.Spec.Services.JobService.Persistence.PostgreSQL = &operatorapi.PersistencePostgreSQL{}
 		}
-		p.Spec.Services.JobService.Persistence.PostgreSql.JdbcUrl = jdbc
+		p.Spec.Services.JobService.Persistence.PostgreSQL.JdbcUrl = jdbc
 	}
 }
 
 func setDataIndexJDBC(jdbc string) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   same



##########
controllers/profiles/common/properties/application_test.go:
##########
@@ -658,18 +652,15 @@ func setPlatformName(name string) plfmOptionFn {
 
 func setJobServiceJDBC(jdbc string) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   same



##########
controllers/profiles/common/properties/application_test.go:
##########
@@ -622,9 +622,6 @@ func generatePlatform(opts ...plfmOptionFn) *operatorapi.SonataFlowPlatform {
 
 func setJobServiceEnabledValue(v *bool) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   revert



##########
controllers/profiles/common/properties/application_test.go:
##########
@@ -634,9 +631,6 @@ func setJobServiceEnabledValue(v *bool) plfmOptionFn {
 
 func setDataIndexEnabledValue(v *bool) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   same



##########
controllers/platform/services/services.go:
##########
@@ -427,7 +459,7 @@ func isJobServiceSet(platform *operatorapi.SonataFlowPlatform) bool {
 }
 
 func isServicesSet(platform *operatorapi.SonataFlowPlatform) bool {
-	return platform != nil && platform.Spec.Services != nil

Review Comment:
   add back `platform.Spec.Services != nil` 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478853349


##########
controllers/platform/services/services.go:
##########
@@ -198,11 +198,24 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil) ||
+		(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil)
+}

Review Comment:
   I'm not sure because this function is to determine if there is a postgreSQL configuration for the data index service (whether it's derived from the platform's own persistence spec or from the service's). Correct me if I'm wrong, but the function `IsServiceSetInSpec()` just returns true when the DI service is defined in the platform's spec. And at this point of the execution the DI service has been already established that it exists, otherwise the parent function would not be called in the first place.
   
   I can add the check if you think it's worth to have it anyway. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1471303649


##########
test/testdata/platform/persistence/generic_from_platform_cr/02-sonataflow_platform.yaml:
##########
@@ -0,0 +1,51 @@
+# Copyright 2024 Apache Software Foundation (ASF)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlowPlatform
+metadata:
+  name: sonataflow-platform
+spec:
+  build:
+    template:
+      buildArgs:
+      - name: QUARKUS_EXTENSIONS
+        value: org.kie.kogito:kogito-addons-quarkus-jobs-knative-eventing:2.0.0-SNAPSHOT

Review Comment:
   Should I remove this from the manifest then?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474656365


##########
controllers/platform/services/services.go:
##########
@@ -138,11 +138,23 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||

Review Comment:
   @tchughesiv then we have to review the API in a follow-up PR. The API **must communicate that**.
   
   For example, everything under ` .spec.local.builds` and `.spec.local.persistence`. Hence, no global config for these attributes. Can you file an issue so we can discuss further after your PR is merged?



##########
controllers/platform/services/services.go:
##########
@@ -138,11 +138,23 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||

Review Comment:
   @tchughesiv then we have to review the API in a follow-up PR. The API **must communicate that**.
   
   For example, everything under ` .spec.local.builds` and `.spec.local.persistence`. Hence, no global config for these attributes. Can you file an issue so we can discuss it further after your PR is merged?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484771915


##########
controllers/platform/services/services.go:
##########
@@ -198,11 +198,33 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return d.IsServiceSetInSpec() &&
+		((d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil) ||
+			(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil))
+}
+
 func (d DataIndexHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
-	if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil {
+
+	if d.hasPostgreSQLConfigured() {
+		var p *operatorapi.PersistencePostgreSQL
 		c := containerSpec.DeepCopy()
 		c.Image = d.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(d.platform.Spec.Services.DataIndex.Persistence.PostgreSql, d.GetServiceName(), d.platform.Namespace)...)
+		if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil {
+			p = d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL
+		} else {
+			p = &operatorapi.PersistencePostgreSQL{
+				SecretRef: d.platform.Spec.Persistence.PostgreSQL.SecretRef,
+				JdbcUrl:   d.platform.Spec.Persistence.PostgreSQL.JdbcUrl,
+			}
+			if d.platform.Spec.Persistence.PostgreSQL.ServiceRef != nil {
+				p.ServiceRef = &operatorapi.PostgreSQLServiceOptions{
+					SQLServiceOptions: d.platform.Spec.Persistence.PostgreSQL.ServiceRef,
+					DatabaseSchema:    d.GetServiceName()}
+			}
+		}

Review Comment:
   this entire `} else {` block of code seems like something that could maybe split our and shared between the DI and JS handlers' `ConfigurePersistence` functions. thoughts?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484285291


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   imo @masayag's interpretation is how i've generally approached these things and seen them implemented in k8s. a user can "override" the platform at a workflow level by setting persistence in the sonataflow CR. if persistence is `nil` in the workflow, settings are used from platform CR. if persistence is `empty` in the workflow, no persistence is used in the workflow.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484433968


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   > > if persistence is `nil` in the workflow, settings are used from platform CR.
   > 
   > Doesn't that break backwards compatibility? I mean, currently a workflow that doesn't have the persistence field populated is not expecting to have persistence if the platform contains one.
   
   what do you mean by `currently `? isn't this PR introducing this capability and we get to define how it functions? how would this break backwards compatibility?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1966630580

   @jordigilh can you please rebase, so I can merge it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478917687


##########
controllers/platform/services/services.go:
##########
@@ -198,11 +198,24 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil) ||
+		(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil)
+}

Review Comment:
   > at this point of the execution the DI service has been already established that it exists, otherwise the parent function would not be called in the first place.
   
   I'm not sure that's accurate though. Maybe at the point you're currently using `hasPostgreSQLConfigured` ... but we can't assume that will always be the case. This function should work regardless of where/when it's called imo



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1469769683


##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -46,6 +46,57 @@ type SonataFlowPlatformSpec struct {
 	// `sonataflow.org/profile: prod`
 	// +optional
 	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PlatformPersistenceSpec `json:"persistence,omitempty"`
+}
+
+// PlatformPersistenceSpec configures the DataBase support for both platform services and workflows. For services, it allows
+// configuring a generic database connectivity if the service does not come with its own configured. In case of the workflow
+// the configuration is defined by the image name of the supported database. When the workflow is configured to require

Review Comment:
   I think you inverted this description. The service has a DB type connectivity set to it. The workflow supports a generic database config.



##########
workflowproj/workflowproj_test.go:
##########
@@ -112,10 +113,21 @@ func Test_Handler_WorkflowMinimalAndPropsAndSpecAndGeneric(t *testing.T) {
 	assert.Equal(t, "02-hello-resources", proj.Resources[1].Name)
 	assert.Equal(t, proj.Workflow.Spec.Resources.ConfigMaps[0].ConfigMap.Name, proj.Resources[0].Name)
 	assert.Equal(t, proj.Workflow.Spec.Resources.ConfigMaps[1].ConfigMap.Name, proj.Resources[1].Name)
-	assert.NotEmpty(t, proj.Resources[0].Data, fmt.Sprintf("Data in proj.Resources[0] is empty %+v", proj.Resources[0]))
-	assert.NotEmpty(t, proj.Resources[1].Data, fmt.Sprintf("Data in proj.Resources[1] is empty %+v", proj.Resources[1]))
-	assert.NotEmpty(t, proj.Resources[0].Data["myopenapi.json"])
-	assert.NotEmpty(t, proj.Resources[1].Data["input.json"])
+	data, err := getResourceDataWithFileName(proj.Resources, "myopenapi.json")

Review Comment:
   We shouldn't need this since the array is sorted. But I guess we can force it here to avoid inconclusive tests all the time. Weirdly, I wasn't able to reproduce this error locally.



##########
test/testdata/platform/persistence/generic_from_platform_cr/02-sonataflow_platform.yaml:
##########
@@ -0,0 +1,51 @@
+# Copyright 2024 Apache Software Foundation (ASF)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlowPlatform
+metadata:
+  name: sonataflow-platform
+spec:
+  build:
+    template:
+      buildArgs:
+      - name: QUARKUS_EXTENSIONS
+        value: org.kie.kogito:kogito-addons-quarkus-jobs-knative-eventing:2.0.0-SNAPSHOT

Review Comment:
   @wmedvede I guess you have fixed this already on the images side?



##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -46,6 +46,57 @@ type SonataFlowPlatformSpec struct {
 	// `sonataflow.org/profile: prod`
 	// +optional
 	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PlatformPersistenceSpec `json:"persistence,omitempty"`
+}
+
+// PlatformPersistenceSpec configures the DataBase support for both platform services and workflows. For services, it allows
+// configuring a generic database connectivity if the service does not come with its own configured. In case of the workflow
+// the configuration is defined by the image name of the supported database. When the workflow is configured to require
+// a database backend type supported by the platform (that is, the platform CR contains an image name for that database type)
+// , the workflow pod is configured to contain a sidecar container running the database image defined here, and the workflow
+// JDBC connectivity is setup to point to the DB container.
+// +optional
+type PlatformPersistenceSpec struct {
+	// Connect configured services to a postgresql database.
+	// +optional
+	PostgreSQL *PostgreSQLPlatformSpec `json:"postgresql,omitempty"`
+}
+
+// PostgreSQLPlatformSpec provides the generic configuration details to configure the JDBC URL and establish a connection for each managed services when they don't provide their own configuration.
+type PostgreSQLPlatformSpec struct {
+	// SecretRef contains the database user credentials
+	SecretRef SecretReference `json:"secretRef"`
+	// ServiceRef contains the K8s service name and namespace location of the PostgreSQL service.
+	ServiceRef ServiceReference `json:"serviceRef,omitempty"`
+	// Name of postgresql database to be used. Defaults to "sonataflow"
+	// +kubebuilder:default:=sonataflow
+	DatabaseName string `json:"databaseName,omitempty"`
+}
+
+type ServiceReference struct {
+	// Name contains the name of the kubernetes service. This field is mandatory.
+	// +required
+	Name string `json:"name"`
+	// Namespace contains the name of the namespace where the kubernetes service resides. This field is optional.
+	// +optional
+	Namespace string `json:"namespace"`
+	// Port contains the port number associated to the kubernetes service. This field is mandatory.
+	// +required
+	Port int `json:"port,omitempty"`
+}
+
+// SecretReference use of a secret to store the credentials to authenticate in the JDBC connection.
+type SecretReference struct {
+	// Name of the postgresql credentials secret. This field is mandatory.
+	Name string `json:"name"`
+	// +optional
+	UserKey string `json:"userKey,omitempty"`
+	// +optional
+	PasswordKey string `json:"passwordKey,omitempty"`
 }

Review Comment:
   Can't we reuse these types? IIRC we already have them when defining the services.



##########
controllers/platform/services/services.go:
##########
@@ -138,13 +138,25 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||
+		(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil)
+}
+
 func (d DataIndexHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
-	if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil {
+
+	if d.hasPostgreSQLConfigured() {
+		p := d.platform.Spec.Services.DataIndex.Persistence
 		c := containerSpec.DeepCopy()
 		c.Image = d.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(d.platform.Spec.Services.DataIndex.Persistence.PostgreSql, d.GetServiceName(), d.platform.Namespace)...)
+		if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil {
+			c.Env = append(c.Env, persistence.ConfigurePostgreSQLEnv(p.PostgreSql, d.GetServiceName(), d.platform.Namespace)...)
+		} else {
+			c.Env = append(c.Env, persistence.ConfigurePostgreSQLEnvFromPlatformSpec(d.platform.Spec.Persistence.PostgreSQL, d.GetServiceName())...)
+		}
 		// specific to DataIndex
-		c.Env = append(c.Env, corev1.EnvVar{Name: quarkusHibernateORMDatabaseGeneration, Value: "update"}, corev1.EnvVar{Name: quarkusFlywayMigrateAtStart, Value: "true"})
+		c.Env = append(c.Env, corev1.EnvVar{Name: "QUARKUS_HIBERNATE_ORM_DATABASE_GENERATION", Value: "update"}, corev1.EnvVar{Name: "QUARKUS_FLYWAY_MIGRATE_AT_START", Value: "true"})

Review Comment:
   Can you move these values to const?



##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -46,6 +46,57 @@ type SonataFlowPlatformSpec struct {
 	// `sonataflow.org/profile: prod`
 	// +optional
 	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PlatformPersistenceSpec `json:"persistence,omitempty"`
+}
+
+// PlatformPersistenceSpec configures the DataBase support for both platform services and workflows. For services, it allows
+// configuring a generic database connectivity if the service does not come with its own configured. In case of the workflow
+// the configuration is defined by the image name of the supported database. When the workflow is configured to require
+// a database backend type supported by the platform (that is, the platform CR contains an image name for that database type)
+// , the workflow pod is configured to contain a sidecar container running the database image defined here, and the workflow
+// JDBC connectivity is setup to point to the DB container.

Review Comment:
   The sidecar should be in the services, right? JS/DI.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1469797790


##########
workflowproj/workflowproj_test.go:
##########
@@ -112,10 +113,21 @@ func Test_Handler_WorkflowMinimalAndPropsAndSpecAndGeneric(t *testing.T) {
 	assert.Equal(t, "02-hello-resources", proj.Resources[1].Name)
 	assert.Equal(t, proj.Workflow.Spec.Resources.ConfigMaps[0].ConfigMap.Name, proj.Resources[0].Name)
 	assert.Equal(t, proj.Workflow.Spec.Resources.ConfigMaps[1].ConfigMap.Name, proj.Resources[1].Name)
-	assert.NotEmpty(t, proj.Resources[0].Data, fmt.Sprintf("Data in proj.Resources[0] is empty %+v", proj.Resources[0]))
-	assert.NotEmpty(t, proj.Resources[1].Data, fmt.Sprintf("Data in proj.Resources[1] is empty %+v", proj.Resources[1]))
-	assert.NotEmpty(t, proj.Resources[0].Data["myopenapi.json"])
-	assert.NotEmpty(t, proj.Resources[1].Data["input.json"])
+	data, err := getResourceDataWithFileName(proj.Resources, "myopenapi.json")

Review Comment:
   I was able to reproduce the failure locally and it was just as you said in slack: the order of the resources, being a slice, is random and nondeterministic. In practice the order doesn't matter when executing the workflow, but it impacts the unit test, so adding this function to retrieve the contents should suffice without impacting the functional code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1470417623


##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -46,6 +46,45 @@ type SonataFlowPlatformSpec struct {
 	// `sonataflow.org/profile: prod`
 	// +optional
 	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PlatformPersistenceSpec `json:"persistence,omitempty"`

Review Comment:
   shouldn't we strive to use the existing `PersistenceOptions` struct? to keep things consistent?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "wmedvede (via GitHub)" <gi...@apache.org>.
wmedvede commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474095567


##########
controllers/platform/initialize.go:
##########
@@ -106,6 +107,13 @@ func (action *initializeAction) Handle(ctx context.Context, platform *operatorap
 	}
 	platform.Status.Version = metadata.SpecVersion
 
+	// store workflow persistence configuration
+	if platform.Spec.Persistence != nil {
+		persistence.WorkflowConfig.SetConfig(platform.Spec.Persistence)

Review Comment:
   I don't understand this. Are we storing globally the Persistence configuration present in current platform?
   And then, we use it in defaultContainer(workflow *operatorapi.SonataFlow)  creation when we create a workflow container?
   
   But what happens if we have two different platforms with different persistence settings?
   
   
   



##########
controllers/platform/services/services.go:
##########
@@ -138,11 +138,23 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||

Review Comment:
   silly comment here, that maybe we won't have any more if we move to use the PersistenceOptions in all places.
   But here we have. d.platform.Spec.Services.DataIndex.Persistence.**PostgreSql** and d.platform.Spec.Persistence.**PostgreSQL**
   
   I think that we should try to unify the capitalization, at least for the fields that are part of the API



##########
test/testdata/workflow/persistence/from_platform_overwritten_by_service/02-sonataflow_platform.yaml:
##########
@@ -0,0 +1,37 @@
+# Copyright 2024 Apache Software Foundation (ASF)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlowPlatform
+metadata:
+  name: sonataflow-platform
+spec:
+  persistence:
+    postgresql:
+      secretRef:
+        name: my-secret
+        userKey: MY_USER
+        passwordKey: MY_PASSWORD
+      serviceRef:
+        name: db_not_defined
+        port: 3456
+      databaseName: db_name
+  build:
+    template:
+      buildArgs:
+      - name: QUARKUS_EXTENSIONS
+        value: org.kie.kogito:kogito-addons-quarkus-jobs-knative-eventing:999-SNAPSHOT,org.kie.kogito:kogito-addons-quarkus-persistence-jdbc:999-SNAPSHOT,io.quarkus:quarkus-jdbc-postgresql:3.2.9.Final,io.quarkus:quarkus-agroal:3.2.9.Final

Review Comment:
   same here, org.kie.kogito:kogito-addons-quarkus-jobs-knative-eventing:999-SNAPSHOT we don't need.



##########
controllers/profiles/common/persistence/postgresql.go:
##########
@@ -0,0 +1,203 @@
+// Copyright 2023 Apache Software Foundation (ASF)
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package persistence
+
+import (
+	"fmt"
+	"sync"
+
+	corev1 "k8s.io/api/core/v1"
+
+	operatorapi "github.com/apache/incubator-kie-kogito-serverless-operator/api/v1alpha08"
+	"github.com/apache/incubator-kie-kogito-serverless-operator/controllers/profiles/common/constants"
+)
+
+const (
+	defaultSchemaName   = "default"

Review Comment:
   Looks like most of this constants are not being used



##########
test/testdata/platform/persistence/generic_from_platform_cr/02-sonataflow_platform.yaml:
##########
@@ -0,0 +1,51 @@
+# Copyright 2024 Apache Software Foundation (ASF)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlowPlatform
+metadata:
+  name: sonataflow-platform
+spec:
+  build:
+    template:
+      buildArgs:
+      - name: QUARKUS_EXTENSIONS
+        value: org.kie.kogito:kogito-addons-quarkus-jobs-knative-eventing:2.0.0-SNAPSHOT

Review Comment:
   yes, this extension is now included in swf-builder by default, so we can remove it here.



##########
controllers/platform/services/services.go:
##########
@@ -138,11 +138,23 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||
+		(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil)
+}
+
 func (d DataIndexHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
-	if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil {
+
+	if d.hasPostgreSQLConfigured() {
+		p := d.platform.Spec.Services.DataIndex.Persistence
 		c := containerSpec.DeepCopy()
 		c.Image = d.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(d.platform.Spec.Services.DataIndex.Persistence.PostgreSql, d.GetServiceName(), d.platform.Namespace)...)
+		if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil {

Review Comment:
   if we reached this point, maybe we don't need this  d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil condition.
   Maybe this changes a bit if we move to the PersistenceOptions, but yeah, just in case.
   



##########
controllers/profiles/common/object_creators.go:
##########
@@ -218,10 +226,20 @@ func WorkflowPropsConfigMapCreator(workflow *operatorapi.SonataFlow) (client.Obj
 	return workflowproj.CreateNewAppPropsConfigMap(workflow, props), nil
 }
 
-func ConfigurePersistence(serviceContainer *corev1.Container, options *operatorapi.PersistenceOptions, defaultSchema, namespace string) *corev1.Container {
+func ConfigurePersistence(serviceContainer *corev1.Container, config *operatorapi.PersistenceOptions, defaultSchema, namespace string) (*corev1.Container, error) {
+	if config == nil {
+		return serviceContainer, nil
+	}
 	c := serviceContainer.DeepCopy()
-	if options != nil && options.PostgreSql != nil {
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(options.PostgreSql, defaultSchema, namespace)...)
+
+	if config.PostgreSql != nil {
+		c.Env = append(c.Env, persistence.ConfigurePostgreSQLEnv(config.PostgreSql, defaultSchema, namespace)...)
+		return c, nil
+	}
+	p := persistence.WorkflowConfig.GetPostgreSQLConfiguration()

Review Comment:
   related to my previous comment, when we do persistence.WorkflowConfig are we getting a global value?
   Maybe I'm missing a go lang trick that IDK..., but looks like this value is globally set.



##########
controllers/profiles/common/persistence/postgresql.go:
##########
@@ -0,0 +1,203 @@
+// Copyright 2023 Apache Software Foundation (ASF)
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package persistence
+
+import (
+	"fmt"
+	"sync"
+
+	corev1 "k8s.io/api/core/v1"
+
+	operatorapi "github.com/apache/incubator-kie-kogito-serverless-operator/api/v1alpha08"
+	"github.com/apache/incubator-kie-kogito-serverless-operator/controllers/profiles/common/constants"
+)
+
+const (
+	defaultSchemaName   = "default"
+	defaultDatabaseName = "sonataflow"
+
+	timeoutSeconds       = 3
+	failureThreshold     = 5
+	initialPeriodSeconds = 15
+	initialDelaySeconds  = 10
+	successThreshold     = 1
+
+	postgreSQLCPULimit      = "500m"
+	postgreSQLMemoryLimit   = "256Mi"
+	postgreSQLMemoryRequest = "256Mi"
+	postgreSQLCPURequest    = "100m"
+
+	defaultPostgreSQLUsername  = "sonataflow"
+	defaultPostgresSQLPassword = "sonataflow"
+)
+
+var (
+	WorkflowConfig *syncConfig

Review Comment:
   Global var where we store the persistence information for a specific platform, I have asked already about how this works. Happy to learn if there's a go trick I'm missing.



##########
controllers/platform/services/services.go:
##########
@@ -252,14 +264,24 @@ func (j JobServiceHandler) MergeContainerSpec(containerSpec *corev1.Container) (
 	return c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Job service specification is not nil
+func (j JobServiceHandler) hasPostgreSQLConfigured() bool {
+	return (j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSql != nil) ||
+		(j.platform.Spec.Persistence != nil && j.platform.Spec.Persistence.PostgreSQL != nil)
+}
+
 func (j JobServiceHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
 
-	if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSql != nil {
+	if j.hasPostgreSQLConfigured() {
 		c := containerSpec.DeepCopy()
 		c.Image = j.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(j.platform.Spec.Services.JobService.Persistence.PostgreSql, j.GetServiceName(), j.platform.Namespace)...)
+		if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSql != nil {

Review Comment:
   same here, j.platform.Spec.Services.JobService.Persistence.PostgreSql don't needed.



##########
controllers/platform/services/services.go:
##########
@@ -276,14 +298,25 @@ func (j JobServiceHandler) GenerateServiceProperties() (*properties.Properties,
 	props.Set(constants.KogitoServiceURLProperty, generateServiceURL(constants.KogitoServiceURLProtocol, j.platform.Namespace, j.GetServiceName()))
 	props.Set(constants.JobServiceKafkaSmallRyeHealthProperty, "false")
 	// add data source reactive URL
-	jspec := j.platform.Spec.Services.JobService
-	if jspec != nil && jspec.Persistence != nil && jspec.Persistence.PostgreSql != nil {
-		dataSourceReactiveURL, err := generateReactiveURL(jspec.Persistence.PostgreSql, j.GetServiceName(), j.platform.Namespace, constants.DefaultDatabaseName, constants.DefaultPostgreSQLPort)
-		if err != nil {
-			return nil, err
+	if j.hasPostgreSQLConfigured() {
+		var dataSourceReactiveURL string
+		var err error
+		if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSql != nil {

Review Comment:
   same here j.platform.Spec.Services.JobService.Persistence.PostgreSql 



##########
controllers/platform/services/services.go:
##########
@@ -138,11 +138,23 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||

Review Comment:
   Here goes an open question guys @ricardozanini @jordigilh @tchughesiv, that of course can't be included here because the SCP is not merged yet, but in a flow-up PR.
   
   If we have a configured SCP with persistence, for example Postgresql, should it be considered here too?



##########
api/v1alpha08/sonataflow_types.go:
##########
@@ -657,8 +657,7 @@ type SonataFlowSpec struct {
 	// PodTemplate describes the deployment details of this SonataFlow instance.
 	//+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="podTemplate"
 	PodTemplate PodTemplateSpec `json:"podTemplate,omitempty"`
-	// Persists service to a datasource of choice. Ephemeral by default.
-	// +optional
+	// Persistence defines the database persistence configuration for the workflow
 	Persistence *PersistenceOptions `json:"persistence,omitempty"`

Review Comment:
   Now that we are going to share the PersistenceOptions type between services and workflows, and the other Postgresql types too, maybe it could make sense to have these structs in a separate file, maybe sonataflow_persitence_types.go
   @ricardozanini @jordigilh  wdyt?



##########
test/testdata/platform/persistence/overwritten_by_services/02-sonataflow_platform.yaml:
##########
@@ -0,0 +1,69 @@
+# Copyright 2024 Apache Software Foundation (ASF)
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: sonataflow.org/v1alpha08
+kind: SonataFlowPlatform
+metadata:
+  name: sonataflow-platform
+spec:
+  build:
+    template:
+      buildArgs:
+      - name: QUARKUS_EXTENSION
+        value: org.kie.kogito:kogito-addons-quarkus-jobs-knative-eventing:2.0.0-SNAPSHOT

Review Comment:
   same here, we can remove.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1921782317

   @jordigilh can you please rebase since we've merged the global platform PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478459779


##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -46,6 +46,22 @@ type SonataFlowPlatformSpec struct {
 	// `sonataflow.org/profile: prod`
 	// +optional
 	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PlatformPersistenceSpec `json:"persistence,omitempty"`

Review Comment:
   I think `PersistenceOptionsSpec` makes more sense because this is used in both the platform and workflow objects.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484720547


##########
controllers/profiles/common/object_creators.go:
##########
@@ -161,13 +149,34 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {
+		c := workflow.Spec.Persistence
+		if c.PostgreSQL == nil && plf.Spec.Persistence != nil {
+			c = &operatorapi.PersistenceOptionsSpec{}
+			if plf.Spec.Persistence.PostgreSQL != nil {
+				c.PostgreSQL = &operatorapi.PersistencePostgreSQL{
+					SecretRef: plf.Spec.Persistence.PostgreSQL.SecretRef,
+					JdbcUrl:   plf.Spec.Persistence.PostgreSQL.JdbcUrl,

Review Comment:
   maybe we should add an `if len(plf.Spec.Persistence.PostgreSQL.JdbcUrl) > 0` check here first? to ensure we're not breaking the MaxItems CRD validation here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1489922500


##########
controllers/profiles/common/object_creators.go:
##########
@@ -161,13 +149,34 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {
+		c := workflow.Spec.Persistence
+		if c.PostgreSQL == nil && plf.Spec.Persistence != nil {
+			c = &operatorapi.PersistenceOptionsSpec{}
+			if plf.Spec.Persistence.PostgreSQL != nil {
+				c.PostgreSQL = &operatorapi.PersistencePostgreSQL{
+					SecretRef: plf.Spec.Persistence.PostgreSQL.SecretRef,
+					JdbcUrl:   plf.Spec.Persistence.PostgreSQL.JdbcUrl,

Review Comment:
   Updated to check the `serviceRef` for nil to fallback to the `jdbcUrl` value. Othewrise the `serviceRef` will be used and the `jdbcUrl` is not initialized.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1479365288


##########
controllers/platform/services/services.go:
##########
@@ -358,14 +371,26 @@ func (j JobServiceHandler) MergeContainerSpec(containerSpec *corev1.Container) (
 	return c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Job service specification is not nil
+func (j JobServiceHandler) hasPostgreSQLConfigured() bool {
+	return (j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil) ||
+		(j.platform.Spec.Persistence != nil && j.platform.Spec.Persistence.PostgreSQL != nil)
+}
+
 func (j JobServiceHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
 
-	if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSql != nil {
+	if j.hasPostgreSQLConfigured() {
+		var p *operatorapi.PersistencePostgreSQL
 		c := containerSpec.DeepCopy()
 		c.Image = j.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(j.platform.Spec.Services.JobService.Persistence.PostgreSql, j.GetServiceName(), j.platform.Namespace)...)
+		if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil {

Review Comment:
   Not sure because adding it to `hasPostgreSQLConfigured()` already validates it in line 382:
   https://github.com/apache/incubator-kie-kogito-serverless-operator/blob/95b390b7469f45c3cfad661dd5d1f845757fcdc1/controllers/platform/services/services.go#L382



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478535572


##########
controllers/platform/services/services.go:
##########
@@ -358,14 +371,26 @@ func (j JobServiceHandler) MergeContainerSpec(containerSpec *corev1.Container) (
 	return c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Job service specification is not nil
+func (j JobServiceHandler) hasPostgreSQLConfigured() bool {
+	return (j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil) ||
+		(j.platform.Spec.Persistence != nil && j.platform.Spec.Persistence.PostgreSQL != nil)
+}
+
 func (j JobServiceHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
 
-	if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSql != nil {
+	if j.hasPostgreSQLConfigured() {
+		var p *operatorapi.PersistencePostgreSQL
 		c := containerSpec.DeepCopy()
 		c.Image = j.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(j.platform.Spec.Services.JobService.Persistence.PostgreSql, j.GetServiceName(), j.platform.Namespace)...)
+		if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil {

Review Comment:
   same



##########
controllers/platform/services/services.go:
##########
@@ -198,11 +198,24 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil) ||
+		(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil)
+}
+
 func (d DataIndexHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
-	if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil {
+
+	if d.hasPostgreSQLConfigured() {
+		var p *operatorapi.PersistencePostgreSQL
 		c := containerSpec.DeepCopy()
 		c.Image = d.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(d.platform.Spec.Services.DataIndex.Persistence.PostgreSql, d.GetServiceName(), d.platform.Namespace)...)
+		if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil {

Review Comment:
   same



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1470432852


##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -46,6 +46,45 @@ type SonataFlowPlatformSpec struct {
 	// `sonataflow.org/profile: prod`
 	// +optional
 	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PlatformPersistenceSpec `json:"persistence,omitempty"`

Review Comment:
   That was the initial idea, but I found 2 issues with that:
   
   * `PersistencePostgreSql` contains a `JdbcUrl` field, which opens the door for using a database outside the cluster, potentially a security risk. I also don't think it's a good idea to allow the user to enter free text to configure the database here as it's more prone to typos. Let me know if if's there are no such concerns and I will change the code to use the `JdbcUrl` field.
   * `PostgreSqlServiceOptions` contains the `databaseSchema`. Each workflow should use its own schema, rather than share it, so to avoid problems that might occur when two workflows interact with the database using the same schema. Instead, the schema used is the name of the workflow itself, thus avoiding any collision that might happen if they all share the same schema.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484279710


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   @masayag @jordigilh using the attributes nil/empty/data to define if we configure a certain feature is error-prone, leading to misconfiguration and user misinterpretation. @masayag comment's to prove my point, he interpreted differently from what you're proposing initially.
   
   The `.spec.dataIndex.enabled` is different since we are forcing an `enabled` attribute to clearly define that we opt-in/out to the feature.
   
   In this case here, labels are well suited since we are using an enum to validate and it clearly expresses the operator's behavior. Also, @wmedvede @tchughesiv any thoughts?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484435429


##########
api/v1alpha08/sonataflow_persistence_types.go:
##########
@@ -14,6 +14,32 @@
 
 package v1alpha08
 
+// PlatformPersistenceOptionsSpec configures the DataBase in the platform spec. This specification can
+// be used by workflows and platform services when they don't provide one of their own.
+// +optional
+// +kubebuilder:validation:MaxProperties=1

Review Comment:
   i guess you're right... and the name of the struct ensures it's only used for platform going forward.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474468924


##########
controllers/profiles/common/object_creators.go:
##########
@@ -218,10 +226,20 @@ func WorkflowPropsConfigMapCreator(workflow *operatorapi.SonataFlow) (client.Obj
 	return workflowproj.CreateNewAppPropsConfigMap(workflow, props), nil
 }
 
-func ConfigurePersistence(serviceContainer *corev1.Container, options *operatorapi.PersistenceOptions, defaultSchema, namespace string) *corev1.Container {
+func ConfigurePersistence(serviceContainer *corev1.Container, config *operatorapi.PersistenceOptions, defaultSchema, namespace string) (*corev1.Container, error) {
+	if config == nil {
+		return serviceContainer, nil
+	}
 	c := serviceContainer.DeepCopy()
-	if options != nil && options.PostgreSql != nil {
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(options.PostgreSql, defaultSchema, namespace)...)
+
+	if config.PostgreSql != nil {
+		c.Env = append(c.Env, persistence.ConfigurePostgreSQLEnv(config.PostgreSql, defaultSchema, namespace)...)
+		return c, nil
+	}
+	p := persistence.WorkflowConfig.GetPostgreSQLConfiguration()

Review Comment:
   It's a global variable for the operator But yes, I should add another level of abstraction to add the namespace.
   Question: is it possible to have multiple platform CRs in the same namespace? I don't think so, but asking to be sure.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474561065


##########
controllers/profiles/common/object_creators.go:
##########
@@ -218,10 +226,20 @@ func WorkflowPropsConfigMapCreator(workflow *operatorapi.SonataFlow) (client.Obj
 	return workflowproj.CreateNewAppPropsConfigMap(workflow, props), nil
 }
 
-func ConfigurePersistence(serviceContainer *corev1.Container, options *operatorapi.PersistenceOptions, defaultSchema, namespace string) *corev1.Container {
+func ConfigurePersistence(serviceContainer *corev1.Container, config *operatorapi.PersistenceOptions, defaultSchema, namespace string) (*corev1.Container, error) {
+	if config == nil {
+		return serviceContainer, nil
+	}
 	c := serviceContainer.DeepCopy()
-	if options != nil && options.PostgreSql != nil {
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(options.PostgreSql, defaultSchema, namespace)...)
+
+	if config.PostgreSql != nil {
+		c.Env = append(c.Env, persistence.ConfigurePostgreSQLEnv(config.PostgreSql, defaultSchema, namespace)...)
+		return c, nil
+	}
+	p := persistence.WorkflowConfig.GetPostgreSQLConfiguration()

Review Comment:
   yes, it is possible to have multiple platforms in a namespace, but only 1 platform will ever be considered "Active" in that namespace. Thus, we should only ever use the "Active" ns when configuring services & workflows.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474649125


##########
controllers/platform/services/services.go:
##########
@@ -138,11 +138,23 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||

Review Comment:
   @jordigilh I think he's referring to the `SQL` vs `Sql`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474651595


##########
controllers/platform/services/services.go:
##########
@@ -138,11 +138,23 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||

Review Comment:
   So I'd say that we should unify according to the [project's correct name: `PostgreSQL`](https://www.postgresql.org/)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474473796


##########
api/v1alpha08/sonataflow_types.go:
##########
@@ -657,8 +657,7 @@ type SonataFlowSpec struct {
 	// PodTemplate describes the deployment details of this SonataFlow instance.
 	//+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="podTemplate"
 	PodTemplate PodTemplateSpec `json:"podTemplate,omitempty"`
-	// Persists service to a datasource of choice. Ephemeral by default.
-	// +optional
+	// Persistence defines the database persistence configuration for the workflow
 	Persistence *PersistenceOptions `json:"persistence,omitempty"`

Review Comment:
   Sounds good to me.s



##########
api/v1alpha08/sonataflow_types.go:
##########
@@ -657,8 +657,7 @@ type SonataFlowSpec struct {
 	// PodTemplate describes the deployment details of this SonataFlow instance.
 	//+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="podTemplate"
 	PodTemplate PodTemplateSpec `json:"podTemplate,omitempty"`
-	// Persists service to a datasource of choice. Ephemeral by default.
-	// +optional
+	// Persistence defines the database persistence configuration for the workflow
 	Persistence *PersistenceOptions `json:"persistence,omitempty"`

Review Comment:
   Sounds good to me.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474467690


##########
controllers/platform/initialize.go:
##########
@@ -106,6 +107,13 @@ func (action *initializeAction) Handle(ctx context.Context, platform *operatorap
 	}
 	platform.Status.Version = metadata.SpecVersion
 
+	// store workflow persistence configuration
+	if platform.Spec.Persistence != nil {
+		persistence.WorkflowConfig.SetConfig(platform.Spec.Persistence)

Review Comment:
   Ah I see your point, I should add a level of abstraction to catch configuration based on the namespace, not a single one. I'll update that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474561810


##########
controllers/platform/initialize.go:
##########
@@ -106,6 +107,13 @@ func (action *initializeAction) Handle(ctx context.Context, platform *operatorap
 	}
 	platform.Status.Version = metadata.SpecVersion
 
+	// store workflow persistence configuration
+	if platform.Spec.Persistence != nil {
+		persistence.WorkflowConfig.SetConfig(platform.Spec.Persistence)

Review Comment:
   we should only ever use the "Active" platform when configuring services & workflows.



##########
controllers/profiles/common/object_creators.go:
##########
@@ -218,10 +226,20 @@ func WorkflowPropsConfigMapCreator(workflow *operatorapi.SonataFlow) (client.Obj
 	return workflowproj.CreateNewAppPropsConfigMap(workflow, props), nil
 }
 
-func ConfigurePersistence(serviceContainer *corev1.Container, options *operatorapi.PersistenceOptions, defaultSchema, namespace string) *corev1.Container {
+func ConfigurePersistence(serviceContainer *corev1.Container, config *operatorapi.PersistenceOptions, defaultSchema, namespace string) (*corev1.Container, error) {
+	if config == nil {
+		return serviceContainer, nil
+	}
 	c := serviceContainer.DeepCopy()
-	if options != nil && options.PostgreSql != nil {
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(options.PostgreSql, defaultSchema, namespace)...)
+
+	if config.PostgreSql != nil {
+		c.Env = append(c.Env, persistence.ConfigurePostgreSQLEnv(config.PostgreSql, defaultSchema, namespace)...)
+		return c, nil
+	}
+	p := persistence.WorkflowConfig.GetPostgreSQLConfiguration()

Review Comment:
   yes, it is possible to have multiple platforms in a namespace, but only 1 platform will ever be considered "Active" in that namespace. Thus, we should only ever use the "Active" platform when configuring services & workflows.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474679904


##########
controllers/platform/services/services.go:
##########
@@ -276,14 +301,25 @@ func (j JobServiceHandler) GenerateServiceProperties() (*properties.Properties,
 	props.Set(constants.KogitoServiceURLProperty, generateServiceURL(constants.KogitoServiceURLProtocol, j.platform.Namespace, j.GetServiceName()))
 	props.Set(constants.JobServiceKafkaSmallRyeHealthProperty, "false")
 	// add data source reactive URL
-	jspec := j.platform.Spec.Services.JobService
-	if jspec != nil && jspec.Persistence != nil && jspec.Persistence.PostgreSql != nil {
-		dataSourceReactiveURL, err := generateReactiveURL(jspec.Persistence.PostgreSql, j.GetServiceName(), j.platform.Namespace, constants.DefaultDatabaseName, constants.DefaultPostgreSQLPort)
-		if err != nil {
-			return nil, err
+	if j.hasPostgreSQLConfigured() {
+		var dataSourceReactiveURL string
+		var err error
+		if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil {
+			dataSourceReactiveURL, err = generateReactiveURL(j.platform.Spec.Services.JobService.Persistence.PostgreSQL, j.GetServiceName(), j.platform.Namespace, constants.DefaultDatabaseName, constants.DefaultPostgreSQLPort)
+			if err != nil {
+				return nil, err
+			}
+		} else {
+			p := j.platform.Spec.Persistence.PostgreSQL
+			var namespace string
+			if len(p.ServiceRef.Namespace) > 0 {
+				namespace = fmt.Sprintf(".%s", p.ServiceRef.Namespace)
+			}
+			dataSourceReactiveURL = fmt.Sprintf("%s://%s%s:%d/%s?search_path=%s", constants.PersistenceTypePostgreSQL, p.ServiceRef.Name, namespace, *p.ServiceRef.Port, p.ServiceRef.DatabaseName, j.GetServiceName())

Review Comment:
   Maybe here we should also have a function to better communication like you did with `generateReactiveURL`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "wmedvede (via GitHub)" <gi...@apache.org>.
wmedvede commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1923305557

   @jordigilh look like we need some love to fix merge conflicts  :crying_cat_face: 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1469931714


##########
workflowproj/workflowproj_test.go:
##########
@@ -112,10 +113,21 @@ func Test_Handler_WorkflowMinimalAndPropsAndSpecAndGeneric(t *testing.T) {
 	assert.Equal(t, "02-hello-resources", proj.Resources[1].Name)
 	assert.Equal(t, proj.Workflow.Spec.Resources.ConfigMaps[0].ConfigMap.Name, proj.Resources[0].Name)
 	assert.Equal(t, proj.Workflow.Spec.Resources.ConfigMaps[1].ConfigMap.Name, proj.Resources[1].Name)
-	assert.NotEmpty(t, proj.Resources[0].Data, fmt.Sprintf("Data in proj.Resources[0] is empty %+v", proj.Resources[0]))
-	assert.NotEmpty(t, proj.Resources[1].Data, fmt.Sprintf("Data in proj.Resources[1] is empty %+v", proj.Resources[1]))
-	assert.NotEmpty(t, proj.Resources[0].Data["myopenapi.json"])
-	assert.NotEmpty(t, proj.Resources[1].Data["input.json"])
+	data, err := getResourceDataWithFileName(proj.Resources, "myopenapi.json")

Review Comment:
   So I guess the slice order we are doing before is not working. :/



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini merged PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478485517


##########
api/v1alpha08/sonataflow_types.go:
##########
@@ -657,8 +657,7 @@ type SonataFlowSpec struct {
 	// PodTemplate describes the deployment details of this SonataFlow instance.
 	//+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="podTemplate"
 	PodTemplate PodTemplateSpec `json:"podTemplate,omitempty"`
-	// Persists service to a datasource of choice. Ephemeral by default.
-	// +optional
+	// Persistence defines the database persistence configuration for the workflow
 	Persistence *PersistenceOptions `json:"persistence,omitempty"`

Review Comment:
   Done.



##########
controllers/platform/services/services.go:
##########
@@ -276,14 +301,25 @@ func (j JobServiceHandler) GenerateServiceProperties() (*properties.Properties,
 	props.Set(constants.KogitoServiceURLProperty, generateServiceURL(constants.KogitoServiceURLProtocol, j.platform.Namespace, j.GetServiceName()))
 	props.Set(constants.JobServiceKafkaSmallRyeHealthProperty, "false")
 	// add data source reactive URL
-	jspec := j.platform.Spec.Services.JobService
-	if jspec != nil && jspec.Persistence != nil && jspec.Persistence.PostgreSql != nil {
-		dataSourceReactiveURL, err := generateReactiveURL(jspec.Persistence.PostgreSql, j.GetServiceName(), j.platform.Namespace, constants.DefaultDatabaseName, constants.DefaultPostgreSQLPort)
-		if err != nil {
-			return nil, err
+	if j.hasPostgreSQLConfigured() {
+		var dataSourceReactiveURL string
+		var err error
+		if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil {
+			dataSourceReactiveURL, err = generateReactiveURL(j.platform.Spec.Services.JobService.Persistence.PostgreSQL, j.GetServiceName(), j.platform.Namespace, constants.DefaultDatabaseName, constants.DefaultPostgreSQLPort)
+			if err != nil {
+				return nil, err
+			}
+		} else {
+			p := j.platform.Spec.Persistence.PostgreSQL
+			var namespace string
+			if len(p.ServiceRef.Namespace) > 0 {
+				namespace = fmt.Sprintf(".%s", p.ServiceRef.Namespace)
+			}
+			dataSourceReactiveURL = fmt.Sprintf("%s://%s%s:%d/%s?search_path=%s", constants.PersistenceTypePostgreSQL, p.ServiceRef.Name, namespace, *p.ServiceRef.Port, p.ServiceRef.DatabaseName, j.GetServiceName())

Review Comment:
   Done.



##########
api/v1alpha08/sonataflowplatform_services_types.go:
##########
@@ -32,61 +32,8 @@ type ServiceSpec struct {
 	Enabled *bool `json:"enabled,omitempty"`
 	// Persists service to a datasource of choice. Ephemeral by default.
 	// +optional
-	Persistence *PersistenceOptions `json:"persistence,omitempty"`
+	Persistence *PersistencePlatformSpec `json:"persistence,omitempty"`
 	// PodTemplate describes the deployment details of this platform service instance.
 	//+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="podTemplate"
 	PodTemplate PodTemplateSpec `json:"podTemplate,omitempty"`
 }
-
-// PersistenceOptions configure the services to persist to a datasource of choice
-// +kubebuilder:validation:MaxProperties=1

Review Comment:
   Added back.



##########
controllers/platform/services/properties_services_test.go:
##########
@@ -185,9 +182,6 @@ func setJobServiceEnabledValue(v *bool) plfmOptionFn {
 
 func setDataIndexEnabledValue(v *bool) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   Done.



##########
controllers/profiles/common/properties/application_test.go:
##########
@@ -658,18 +652,15 @@ func setPlatformName(name string) plfmOptionFn {
 
 func setJobServiceJDBC(jdbc string) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   Done.



##########
controllers/profiles/common/properties/application_test.go:
##########
@@ -634,9 +631,6 @@ func setJobServiceEnabledValue(v *bool) plfmOptionFn {
 
 func setDataIndexEnabledValue(v *bool) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   Done.



##########
controllers/platform/platformutils.go:
##########
@@ -111,15 +111,13 @@ func setPlatformDefaults(p *operatorapi.SonataFlowPlatform, verbose bool) error
 	}
 
 	// When dataIndex object set, default to enabled if bool not set
-	if p.Spec.Services != nil {
-		var enable = true
-		if p.Spec.Services.DataIndex != nil && p.Spec.Services.DataIndex.Enabled == nil {
-			p.Spec.Services.DataIndex.Enabled = &enable
-		}
-		// When the JobService field has a value, default to enabled if the `Enabled` field's value is nil
-		if p.Spec.Services.JobService != nil && p.Spec.Services.JobService.Enabled == nil {
-			p.Spec.Services.JobService.Enabled = &enable
-		}

Review Comment:
   Done.



##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -45,7 +45,23 @@ type SonataFlowPlatformSpec struct {
 	// Only workflows without the `sonataflow.org/profile: dev` annotation will be configured to use these service(s).
 	// Setting this will override the use of any cluster-scoped services that might be defined via `SonataFlowClusterPlatform`.
 	// +optional
-	Services *ServicesPlatformSpec `json:"services,omitempty"`

Review Comment:
   Done.



##########
config/manager/kustomization.yaml:
##########
@@ -16,7 +16,7 @@ apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 images:
 - name: controller
-  newName: quay.io/kiegroup/kogito-serverless-operator-nightly

Review Comment:
   Done



##########
controllers/platform/services/properties_services_test.go:
##########
@@ -231,36 +219,30 @@ func setPlatformName(name string) plfmOptionFn {
 
 func setJobServiceJDBC(jdbc string) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   Done.



##########
controllers/platform/services/properties_services_test.go:
##########
@@ -197,9 +191,6 @@ func setDataIndexEnabledValue(v *bool) plfmOptionFn {
 
 func emptyDataIndexServiceSpec() plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   Done.



##########
controllers/platform/services/properties_services_test.go:
##########
@@ -173,9 +173,6 @@ func generatePlatform(opts ...plfmOptionFn) *operatorapi.SonataFlowPlatform {
 
 func setJobServiceEnabledValue(v *bool) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   Done.



##########
controllers/profiles/common/properties/application_test.go:
##########
@@ -622,9 +622,6 @@ func generatePlatform(opts ...plfmOptionFn) *operatorapi.SonataFlowPlatform {
 
 func setJobServiceEnabledValue(v *bool) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   Done.



##########
controllers/platform/services/properties_services_test.go:
##########
@@ -208,9 +199,6 @@ func emptyDataIndexServiceSpec() plfmOptionFn {
 
 func emptyJobServiceSpec() plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   Done.



##########
controllers/platform/services/properties_services_test.go:
##########
@@ -231,36 +219,30 @@ func setPlatformName(name string) plfmOptionFn {
 
 func setJobServiceJDBC(jdbc string) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}
 		if p.Spec.Services.JobService == nil {
 			p.Spec.Services.JobService = &operatorapi.ServiceSpec{}
 		}
 		if p.Spec.Services.JobService.Persistence == nil {
-			p.Spec.Services.JobService.Persistence = &operatorapi.PersistenceOptions{}
+			p.Spec.Services.JobService.Persistence = &operatorapi.PersistencePlatformSpec{}
 		}
-		if p.Spec.Services.JobService.Persistence.PostgreSql == nil {
-			p.Spec.Services.JobService.Persistence.PostgreSql = &operatorapi.PersistencePostgreSql{}
+		if p.Spec.Services.JobService.Persistence.PostgreSQL == nil {
+			p.Spec.Services.JobService.Persistence.PostgreSQL = &operatorapi.PersistencePostgreSQL{}
 		}
-		p.Spec.Services.JobService.Persistence.PostgreSql.JdbcUrl = jdbc
+		p.Spec.Services.JobService.Persistence.PostgreSQL.JdbcUrl = jdbc
 	}
 }
 
 func setDataIndexJDBC(jdbc string) plfmOptionFn {
 	return func(p *operatorapi.SonataFlowPlatform) {
-		if p.Spec.Services == nil {
-			p.Spec.Services = &operatorapi.ServicesPlatformSpec{}
-		}

Review Comment:
   Done.



##########
controllers/platform/services/services.go:
##########
@@ -427,7 +459,7 @@ func isJobServiceSet(platform *operatorapi.SonataFlowPlatform) bool {
 }
 
 func isServicesSet(platform *operatorapi.SonataFlowPlatform) bool {
-	return platform != nil && platform.Spec.Services != nil

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1480381361


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   Actually there is the `enabled` field in the service that is set to true if the service is defined (not nil) by the operator. It's something we're doing already in that sense: define an empty service for DI and the operator will populate it with the default configuration, in this case `enabled=true`.
   
   I'm fine using labels as well. Consistency is important at the end, so if labels/annotations is the path forward for these cases, then this should be the approach accross whenever facing these situations.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1489920925


##########
controllers/platform/services/services.go:
##########
@@ -198,11 +198,33 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return d.IsServiceSetInSpec() &&
+		((d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil) ||
+			(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil))
+}
+
 func (d DataIndexHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
-	if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil {
+
+	if d.hasPostgreSQLConfigured() {
+		var p *operatorapi.PersistencePostgreSQL
 		c := containerSpec.DeepCopy()
 		c.Image = d.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(d.platform.Spec.Services.DataIndex.Persistence.PostgreSql, d.GetServiceName(), d.platform.Namespace)...)
+		if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil {
+			p = d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL
+		} else {
+			p = &operatorapi.PersistencePostgreSQL{
+				SecretRef: d.platform.Spec.Persistence.PostgreSQL.SecretRef,
+				JdbcUrl:   d.platform.Spec.Persistence.PostgreSQL.JdbcUrl,
+			}
+			if d.platform.Spec.Persistence.PostgreSQL.ServiceRef != nil {
+				p.ServiceRef = &operatorapi.PostgreSQLServiceOptions{
+					SQLServiceOptions: d.platform.Spec.Persistence.PostgreSQL.ServiceRef,
+					DatabaseSchema:    d.GetServiceName()}
+			}
+		}

Review Comment:
   Updated.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484401343


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   > if persistence is `nil` in the workflow, settings are used from platform CR. 
   
   Doesn't that break backwards compatibility? I mean, currently a workflow that doesn't have the persistence field populated is not expecting to have persistence if the platform contains one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1948478800

   @domhanak can you take a look at this PR so it can be finally merged? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474608207


##########
controllers/profiles/common/object_creators.go:
##########
@@ -218,10 +226,20 @@ func WorkflowPropsConfigMapCreator(workflow *operatorapi.SonataFlow) (client.Obj
 	return workflowproj.CreateNewAppPropsConfigMap(workflow, props), nil
 }
 
-func ConfigurePersistence(serviceContainer *corev1.Container, options *operatorapi.PersistenceOptions, defaultSchema, namespace string) *corev1.Container {
+func ConfigurePersistence(serviceContainer *corev1.Container, config *operatorapi.PersistenceOptions, defaultSchema, namespace string) (*corev1.Container, error) {
+	if config == nil {
+		return serviceContainer, nil
+	}
 	c := serviceContainer.DeepCopy()
-	if options != nil && options.PostgreSql != nil {
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(options.PostgreSql, defaultSchema, namespace)...)
+
+	if config.PostgreSql != nil {
+		c.Env = append(c.Env, persistence.ConfigurePostgreSQLEnv(config.PostgreSql, defaultSchema, namespace)...)
+		return c, nil
+	}
+	p := persistence.WorkflowConfig.GetPostgreSQLConfiguration()

Review Comment:
   So, for this PR, can I assume that there's only 1 platform in the namespace? Or is it the general opinion that now it's the right time to add the logic to filter now in the `deploymentReconciler`? if I know where to look for (annotations, labels...) I can do it and use this new function. Alternativelly I can use the kube client to retrieve the first platform retrieved from calling `List()`. Thoughts?
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474626385


##########
controllers/profiles/common/object_creators.go:
##########
@@ -218,10 +226,20 @@ func WorkflowPropsConfigMapCreator(workflow *operatorapi.SonataFlow) (client.Obj
 	return workflowproj.CreateNewAppPropsConfigMap(workflow, props), nil
 }
 
-func ConfigurePersistence(serviceContainer *corev1.Container, options *operatorapi.PersistenceOptions, defaultSchema, namespace string) *corev1.Container {
+func ConfigurePersistence(serviceContainer *corev1.Container, config *operatorapi.PersistenceOptions, defaultSchema, namespace string) (*corev1.Container, error) {
+	if config == nil {
+		return serviceContainer, nil
+	}
 	c := serviceContainer.DeepCopy()
-	if options != nil && options.PostgreSql != nil {
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(options.PostgreSql, defaultSchema, namespace)...)
+
+	if config.PostgreSql != nil {
+		c.Env = append(c.Env, persistence.ConfigurePostgreSQLEnv(config.PostgreSql, defaultSchema, namespace)...)
+		return c, nil
+	}
+	p := persistence.WorkflowConfig.GetPostgreSQLConfiguration()

Review Comment:
   If you think you need to retrieve the active platform, this should be done w/ the existing function `GetActivePlatform`... 
   https://github.com/apache/incubator-kie-kogito-serverless-operator/blob/27c0542985639c5f49a269e6b79b5c3cd5c4b6d6/controllers/platform/platform.go#L121-L156



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1471302526


##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -46,6 +46,57 @@ type SonataFlowPlatformSpec struct {
 	// `sonataflow.org/profile: prod`
 	// +optional
 	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PlatformPersistenceSpec `json:"persistence,omitempty"`
+}
+
+// PlatformPersistenceSpec configures the DataBase support for both platform services and workflows. For services, it allows
+// configuring a generic database connectivity if the service does not come with its own configured. In case of the workflow
+// the configuration is defined by the image name of the supported database. When the workflow is configured to require
+// a database backend type supported by the platform (that is, the platform CR contains an image name for that database type)
+// , the workflow pod is configured to contain a sidecar container running the database image defined here, and the workflow
+// JDBC connectivity is setup to point to the DB container.

Review Comment:
   Done



##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -46,6 +46,57 @@ type SonataFlowPlatformSpec struct {
 	// `sonataflow.org/profile: prod`
 	// +optional
 	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PlatformPersistenceSpec `json:"persistence,omitempty"`
+}
+
+// PlatformPersistenceSpec configures the DataBase support for both platform services and workflows. For services, it allows
+// configuring a generic database connectivity if the service does not come with its own configured. In case of the workflow
+// the configuration is defined by the image name of the supported database. When the workflow is configured to require
+// a database backend type supported by the platform (that is, the platform CR contains an image name for that database type)
+// , the workflow pod is configured to contain a sidecar container running the database image defined here, and the workflow
+// JDBC connectivity is setup to point to the DB container.
+// +optional
+type PlatformPersistenceSpec struct {
+	// Connect configured services to a postgresql database.
+	// +optional
+	PostgreSQL *PostgreSQLPlatformSpec `json:"postgresql,omitempty"`
+}
+
+// PostgreSQLPlatformSpec provides the generic configuration details to configure the JDBC URL and establish a connection for each managed services when they don't provide their own configuration.
+type PostgreSQLPlatformSpec struct {
+	// SecretRef contains the database user credentials
+	SecretRef SecretReference `json:"secretRef"`
+	// ServiceRef contains the K8s service name and namespace location of the PostgreSQL service.
+	ServiceRef ServiceReference `json:"serviceRef,omitempty"`
+	// Name of postgresql database to be used. Defaults to "sonataflow"
+	// +kubebuilder:default:=sonataflow
+	DatabaseName string `json:"databaseName,omitempty"`
+}
+
+type ServiceReference struct {
+	// Name contains the name of the kubernetes service. This field is mandatory.
+	// +required
+	Name string `json:"name"`
+	// Namespace contains the name of the namespace where the kubernetes service resides. This field is optional.
+	// +optional
+	Namespace string `json:"namespace"`
+	// Port contains the port number associated to the kubernetes service. This field is mandatory.
+	// +required
+	Port int `json:"port,omitempty"`
+}
+
+// SecretReference use of a secret to store the credentials to authenticate in the JDBC connection.
+type SecretReference struct {
+	// Name of the postgresql credentials secret. This field is mandatory.
+	Name string `json:"name"`
+	// +optional
+	UserKey string `json:"userKey,omitempty"`
+	// +optional
+	PasswordKey string `json:"passwordKey,omitempty"`
 }

Review Comment:
   Done



##########
controllers/platform/services/services.go:
##########
@@ -138,13 +138,25 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||
+		(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil)
+}
+
 func (d DataIndexHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
-	if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil {
+
+	if d.hasPostgreSQLConfigured() {
+		p := d.platform.Spec.Services.DataIndex.Persistence
 		c := containerSpec.DeepCopy()
 		c.Image = d.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(d.platform.Spec.Services.DataIndex.Persistence.PostgreSql, d.GetServiceName(), d.platform.Namespace)...)
+		if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil {
+			c.Env = append(c.Env, persistence.ConfigurePostgreSQLEnv(p.PostgreSql, d.GetServiceName(), d.platform.Namespace)...)
+		} else {
+			c.Env = append(c.Env, persistence.ConfigurePostgreSQLEnvFromPlatformSpec(d.platform.Spec.Persistence.PostgreSQL, d.GetServiceName())...)
+		}
 		// specific to DataIndex
-		c.Env = append(c.Env, corev1.EnvVar{Name: quarkusHibernateORMDatabaseGeneration, Value: "update"}, corev1.EnvVar{Name: quarkusFlywayMigrateAtStart, Value: "true"})
+		c.Env = append(c.Env, corev1.EnvVar{Name: "QUARKUS_HIBERNATE_ORM_DATABASE_GENERATION", Value: "update"}, corev1.EnvVar{Name: "QUARKUS_FLYWAY_MIGRATE_AT_START", Value: "true"})

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1948483228

   @tchughesiv I see that your review status still is request for changes, is there anything else left from your side?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jakubschwan (via GitHub)" <gi...@apache.org>.
jakubschwan commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1495455943


##########
test/e2e/workflow_test.go:
##########
@@ -198,16 +198,27 @@ var _ = Describe("Validate the persistence ", Ordered, func() {
 				}
 				Expect(h.Status).To(Equal(upStatus), "Pod health is not UP")
 				for _, c := range h.Checks {
-					if c.Name == "Database connections health check" {
-						Expect(c.Status).To(Equal(upStatus), "Pod's database connection is not UP")
-						return true
+					if withPersistence {
+						if c.Name == "Database connections health check" {
+							Expect(c.Status).To(Equal(upStatus), "Pod's database connection is not UP")
+							return true
+						}
+					} else {
+						Expect(c.Name).NotTo(Equal("Database connections health check"), "persistence configuration found in workflow")
 					}
 				}
+				if !withPersistence {
+					return true
+				}
 			}
 			return false
-		}, 12*time.Minute).Should(BeTrue())
+		}, 1).Should(BeTrue())

Review Comment:
   Looks like some time limit is lowered. Doesn't this change causing test time-outs?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474466054


##########
controllers/platform/initialize.go:
##########
@@ -106,6 +107,13 @@ func (action *initializeAction) Handle(ctx context.Context, platform *operatorap
 	}
 	platform.Status.Version = metadata.SpecVersion
 
+	// store workflow persistence configuration
+	if platform.Spec.Persistence != nil {
+		persistence.WorkflowConfig.SetConfig(platform.Spec.Persistence)

Review Comment:
   It's a cache for the platform persistence configuration that is retrieved by the workflow's logic when configuring its persistence (either workflow or fallback to the one in the platform in the same namespace).
   
   > But what happens if we have two different platforms with different persistence settings?
    
   That's a good question!. Can we have 2 platform CRs in the same namespace? If so, what should the logic do when that happens when their configuration conflicts in terms of DI, JS and persistence configuration? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474622026


##########
controllers/platform/services/services.go:
##########
@@ -138,11 +138,23 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||

Review Comment:
   No. The purpose of the SonataFlowClusterPlatform is to set cluster-wide configs for **workflow** consumption, NOT for service consumption. This means service configs, e.g. persistence, should only ever be affected by the active platform CR in its own namespace.
   
   To take things a step further, it's important to be *very* careful about what other platform configs are enabled cluster-wide for workflow consumption in the future. For example, In my opinion these new global persistence configs should NOT be enabled cluster-wide for workflows. It would result in a major security risk for users if our operator is providing creds and connectivity information haphazardly to every namespace in the cluster. Persistence should only be set at the namespace level which ensures that only those who should have the creds are using them.
   
   That said, something like `platform.spec.build` configs might be a good candidate at some point.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1481464329


##########
controllers/profiles/dev/states_dev.go:
##########
@@ -70,15 +73,14 @@ func (e *ensureRunningWorkflowState) Do(ctx context.Context, workflow *operatora
 
 	devBaseContainerImage := workflowdef.GetDefaultWorkflowDevModeImageTag()
 	// check if the Platform available
-	pl, err := platform.GetActivePlatform(ctx, e.C, workflow.Namespace)
-	if err == nil && len(pl.Spec.DevMode.BaseImage) > 0 {
-		devBaseContainerImage = pl.Spec.DevMode.BaseImage
+	if plf != nil && len(plf.Spec.DevMode.BaseImage) > 0 {
+		devBaseContainerImage = plf.Spec.DevMode.BaseImage
 	}
 	userPropsCM, _, err := e.ensurers.userPropsConfigMap.Ensure(ctx, workflow)
 	if err != nil {
 		return ctrl.Result{Requeue: false}, objs, err
 	}
-	managedPropsCM, _, err := e.ensurers.managedPropsConfigMap.Ensure(ctx, workflow, pl, common.ManagedPropertiesMutateVisitor(ctx, e.StateSupport.Catalog, workflow, pl, userPropsCM.(*corev1.ConfigMap)))
+	managedPropsCM, _, err := e.ensurers.managedPropsConfigMap.Ensure(ctx, workflow, plf, common.ManagedPropertiesMutateVisitor(ctx, e.StateSupport.Catalog, workflow, plf, userPropsCM.(*corev1.ConfigMap)))

Review Comment:
   No, don't need to bother unless you need to make another change in this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1479365008


##########
controllers/platform/services/services.go:
##########
@@ -198,11 +198,24 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil) ||
+		(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil)
+}
+
 func (d DataIndexHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
-	if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil {
+
+	if d.hasPostgreSQLConfigured() {
+		var p *operatorapi.PersistencePostgreSQL
 		c := containerSpec.DeepCopy()
 		c.Image = d.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(d.platform.Spec.Services.DataIndex.Persistence.PostgreSql, d.GetServiceName(), d.platform.Namespace)...)
+		if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil {

Review Comment:
   Not sure because adding it to `hasPostgreSQLConfigured()` already validates it in line 209:
   https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372/files/95b390b7469f45c3cfad661dd5d1f845757fcdc1#diff-9168d999a0cd028523626009a2534c70de62a6e4322770f8bc21d8fc01fc2b83R209
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478459116


##########
api/v1alpha08/sonataflow_types.go:
##########
@@ -657,8 +657,7 @@ type SonataFlowSpec struct {
 	// PodTemplate describes the deployment details of this SonataFlow instance.
 	//+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="podTemplate"
 	PodTemplate PodTemplateSpec `json:"podTemplate,omitempty"`
-	// Persists service to a datasource of choice. Ephemeral by default.
-	// +optional
+	// Persistence defines the database persistence configuration for the workflow
 	Persistence *PersistenceOptions `json:"persistence,omitempty"`

Review Comment:
   I think `PersistenceOptionsSpec` makes more sense because this is used in both the platform and workflow objects.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484261301


##########
api/v1alpha08/sonataflow_persistence_types.go:
##########
@@ -14,6 +14,32 @@
 
 package v1alpha08
 
+// PlatformPersistenceOptionsSpec configures the DataBase in the platform spec. This specification can
+// be used by workflows and platform services when they don't provide one of their own.
+// +optional
+// +kubebuilder:validation:MaxProperties=1

Review Comment:
   nit - maybe add notes to these persistence struct descriptions regarding the addition (or subtraction) of the `databaseSchema` field?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "ricardozanini (via GitHub)" <gi...@apache.org>.
ricardozanini commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484373297


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   @tchughesiv if k8s is using this approach I understand that's a well-known pattern. In this case, +1 to @masayag approach. Can you point me to any other places using this approach?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484440685


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   Be
   
   > > > if persistence is `nil` in the workflow, settings are used from platform CR.
   > > 
   > > 
   > > Doesn't that break backwards compatibility? I mean, currently a workflow that doesn't have the persistence field populated is not expecting to have persistence if the platform contains one.
   > 
   > what do you mean by `currently `? isn't this PR introducing this capability and we get to define how it functions? how would this break backwards compatibility?
   
   Because if we define that when persistence is nil, which is the default value, then it should derive the persistence from the platform if it´s defined, then workflows that are currently running without persistence (the ones before merging this PR), will suddently now inherit persistence from the platform even though they don´t require it.
   
   That´s what I mean by backwards compatibility:
   * Currently `persistence=nil` means no persistence, it matches the behavior that the workflows expect before this PR is merged
   * After this PR is merged, then `persistence=nil` means it might have persistence depending on the platform, whether the workflow requested it or not.
   
   I think that if we have a way to discern if the workflow requires persistence or not, regardless of the value of the persistence struct, we will save ourselves all this trouble and just apply what needs to be applied.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484451946


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   > * After this PR is merged, then `persistence=nil` means it might have persistence depending on the platform, whether the workflow requested it or not.
   
   but it won't, because existing platform CRs won't have the `spec.persistence` field set. this PR is introducing it as a new field. hence, existing workflows won't inherit persistence unless the user decides to enable it either within the workflow or the existing platform. we should be good to go, no backwards compat issues.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1936945512

   > In this other use case below, we are using a SFP with the global persistence configured to use the JdbcUrl to connect with the database service.
   > 
   > 
   > 
   > https://github.com/flows-examples/techpreview2/tree/004b37b96ad5f75c804d8e8ba3d821ab67c86457/platforms/openshift-dataindex_and_jobservice_as_platform_service_and_global_persistence_jdbcurl_postgresql
   > 
   > 
   > 
   > 
   > 
   > ```
   > 
   >   persistence:
   > 
   >     postgresql:
   > 
   >       secretRef:
   > 
   >         name: postgres-secrets
   > 
   >         userKey: POSTGRESQL_USER
   > 
   >         passwordKey: POSTGRESQL_PASSWORD
   > 
   >       jdbcUrl: jdbc:postgresql://postgres.usecase3-platform-persistence-jdbcurl-oc:5432/sonataflow2
   > 
   > ```
   > 
   > 
   > 
   > But, the corresponding generated specific jdbc url for each service looks like the ones below.
   > 
   > I think we need a small adjustment to make each service use its specific schema, otherwise we have flyway issues when the services start. Not too much, a simple adjustment. 
   > 
   > 
   > 
   > **DataIndex:**
   > 
   > 
   > 
   >     QUARKUS_DATASOURCE_JDBC_URL = jdbc:postgresql://postgres.usecase3-platform-persistence-jdbcurl-oc:5432/sonataflow2
   > 
   > 
   > 
   > **JobsService:**
   > 
   > 
   > 
   >     QUARKUS_DATASOURCE_JDBC_URL = jdbc:postgresql://postgres.usecase3-platform-persistence-jdbcurl-oc:5432/sonataflow2
   > 
   > 
   > 
   > 
   > 
   > 
   > 
   > 
   
   That's because when I posted about the schema being included in the JbdcUrl in slack, it was suggested we should not do anything to remove the schema from it and instead warn the user in the documentation that it can cause problems when defined in the platform.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1469807206


##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -46,6 +46,57 @@ type SonataFlowPlatformSpec struct {
 	// `sonataflow.org/profile: prod`
 	// +optional
 	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PlatformPersistenceSpec `json:"persistence,omitempty"`
+}
+
+// PlatformPersistenceSpec configures the DataBase support for both platform services and workflows. For services, it allows
+// configuring a generic database connectivity if the service does not come with its own configured. In case of the workflow
+// the configuration is defined by the image name of the supported database. When the workflow is configured to require
+// a database backend type supported by the platform (that is, the platform CR contains an image name for that database type)
+// , the workflow pod is configured to contain a sidecar container running the database image defined here, and the workflow
+// JDBC connectivity is setup to point to the DB container.

Review Comment:
   Good catch!. This was in my initial proposal to add the postgreSQL container to the workflow for local persistence, but after discussing with @wmedvede I removed this since it made no sense.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1469814641


##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -46,6 +46,57 @@ type SonataFlowPlatformSpec struct {
 	// `sonataflow.org/profile: prod`
 	// +optional
 	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PlatformPersistenceSpec `json:"persistence,omitempty"`
+}
+
+// PlatformPersistenceSpec configures the DataBase support for both platform services and workflows. For services, it allows
+// configuring a generic database connectivity if the service does not come with its own configured. In case of the workflow
+// the configuration is defined by the image name of the supported database. When the workflow is configured to require

Review Comment:
   Correct. This description needs rewriting as the implementation no longer matches it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1479365008


##########
controllers/platform/services/services.go:
##########
@@ -198,11 +198,24 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil) ||
+		(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil)
+}
+
 func (d DataIndexHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
-	if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil {
+
+	if d.hasPostgreSQLConfigured() {
+		var p *operatorapi.PersistencePostgreSQL
 		c := containerSpec.DeepCopy()
 		c.Image = d.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(d.platform.Spec.Services.DataIndex.Persistence.PostgreSql, d.GetServiceName(), d.platform.Namespace)...)
+		if d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil {

Review Comment:
   Not sure because adding it to `hasPostgreSQLConfigured()` already validates it in line 209:
   https://github.com/apache/incubator-kie-kogito-serverless-operator/blob/95b390b7469f45c3cfad661dd5d1f845757fcdc1/controllers/platform/services/services.go#L209



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474622026


##########
controllers/platform/services/services.go:
##########
@@ -138,11 +138,23 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||

Review Comment:
   No. The purpose of the SonataFlowClusterPlatform is to set cluster-wide configs for **workflow** consumption, NOT for service consumption. This means service configs, e.g. persistence, should only ever be affected by the active platform CR in its own namespace.
   
   To take things a step further, it's important to be *very* careful about what other platform configs are enabled cluster-wide for workflow consumption in the future. For example, In my opinion these new global persistence configs should NOT be enabled cluster-wide for workflows. It would result in a major security risk for users if our operator is providing creds and connectivity information haphazardly to every namespace in the cluster. Persistence should only be set at the namespace level which ensures that only those who should have the creds are using them.
   
   That said, something like `platform.spec.builds` configs might be a good candidate at some point.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1474655537


##########
controllers/platform/services/services.go:
##########
@@ -138,11 +138,23 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSql != nil) ||

Review Comment:
   Yes, that was my understanding in my reply: `postgreSql` vs `postgreSQL`. But in this case you're right: the given name should take precedence regardless of the convention. Luckily for me it's `[p|P]ostgreSQL` (local, global variable) 😄 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "wmedvede (via GitHub)" <gi...@apache.org>.
wmedvede commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1475809671


##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -46,6 +46,22 @@ type SonataFlowPlatformSpec struct {
 	// `sonataflow.org/profile: prod`
 	// +optional
 	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PlatformPersistenceSpec `json:"persistence,omitempty"`

Review Comment:
   Sorry for being picky, I haven't note this before, not  a blocker...
   But, basically we have.
   BuildPlatformSpec, DevModePlatformSpec, SerivicesPlatormSpec and now we introduce PlatformPersistenceSpec, shouldn't this be PersistencePlatformSpec instead?
   



##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   Please correct me if I'm wrong, but believe this code is not considering the case where the workflow don't have a configured persistence, but the platform has. In that case, we should try to get it from the platform.
   
   I am correct?
   



##########
controllers/platform/initialize.go:
##########
@@ -106,6 +107,13 @@ func (action *initializeAction) Handle(ctx context.Context, platform *operatorap
 	}
 	platform.Status.Version = metadata.SpecVersion
 
+	// store workflow persistence configuration
+	if platform.Spec.Persistence != nil {
+		persistence.WorkflowConfig.SetConfig(platform.Spec.Persistence)

Review Comment:
   > I should add a level of abstraction to catch configuration based on the namespace, not a single one. I'll update that.
   
   Yes, this was my question. The platform is for the namespace, all good, and yeah, we can have only one platform persistence configuration per platform/namespace.
   But my question was if this caching you are doing is "global" or per "namespace", that's the part I missing.
   



##########
controllers/profiles/common/object_creators.go:
##########
@@ -218,10 +226,21 @@ func WorkflowPropsConfigMapCreator(workflow *operatorapi.SonataFlow) (client.Obj
 	return workflowproj.CreateNewAppPropsConfigMap(workflow, props), nil
 }
 
-func ConfigurePersistence(serviceContainer *corev1.Container, options *operatorapi.PersistenceOptions, defaultSchema, namespace string) *corev1.Container {
+func ConfigurePersistence(serviceContainer *corev1.Container, config *operatorapi.PersistenceOptions, defaultSchema, namespace string) (*corev1.Container, error) {
+	if config == nil {
+		return serviceContainer, nil
+	}
 	c := serviceContainer.DeepCopy()
-	if options != nil && options.PostgreSql != nil {
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(options.PostgreSql, defaultSchema, namespace)...)
+
+	var p *operatorapi.PersistencePostgreSQL
+	if config.PostgreSQL == nil && persistence.WorkflowConfig.GetPostgreSQLConfiguration() == nil {
+		return nil, fmt.Errorf("no persistence specification available in the workflow or in the platform")
+	}
+	if config.PostgreSQL != nil {
+		p = config.PostgreSQL

Review Comment:
   If the workflow persistence was set, we need to give the chance to see if a schema was provided, and if it's provided, below we must use that schema instead of the defaultSchema that we receive as parameter. 
   In this way we give users the chance to fine tune the schema name for a particular workflow.



##########
controllers/profiles/common/object_creators.go:
##########
@@ -218,10 +226,21 @@ func WorkflowPropsConfigMapCreator(workflow *operatorapi.SonataFlow) (client.Obj
 	return workflowproj.CreateNewAppPropsConfigMap(workflow, props), nil
 }
 
-func ConfigurePersistence(serviceContainer *corev1.Container, options *operatorapi.PersistenceOptions, defaultSchema, namespace string) *corev1.Container {
+func ConfigurePersistence(serviceContainer *corev1.Container, config *operatorapi.PersistenceOptions, defaultSchema, namespace string) (*corev1.Container, error) {
+	if config == nil {
+		return serviceContainer, nil
+	}
 	c := serviceContainer.DeepCopy()
-	if options != nil && options.PostgreSql != nil {
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(options.PostgreSql, defaultSchema, namespace)...)
+
+	var p *operatorapi.PersistencePostgreSQL
+	if config.PostgreSQL == nil && persistence.WorkflowConfig.GetPostgreSQLConfiguration() == nil {
+		return nil, fmt.Errorf("no persistence specification available in the workflow or in the platform")
+	}
+	if config.PostgreSQL != nil {
+		p = config.PostgreSQL
+	} else if persistence.WorkflowConfig.GetPostgreSQLConfiguration() != nil {
+		p = persistence.WorkflowConfig.GetPostgreSQLConfiguration()

Review Comment:
   if we pick the platform configuration, yes, we must use the defaultSchema as it comes, basically the workflow name.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478433266


##########
api/v1alpha08/sonataflowplatform_types.go:
##########
@@ -45,7 +45,23 @@ type SonataFlowPlatformSpec struct {
 	// Only workflows without the `sonataflow.org/profile: dev` annotation will be configured to use these service(s).
 	// Setting this will override the use of any cluster-scoped services that might be defined via `SonataFlowClusterPlatform`.
 	// +optional
-	Services *ServicesPlatformSpec `json:"services,omitempty"`
+	Services ServicesPlatformSpec `json:"services,omitempty"`
+	// Persistence defines the platform persistence configuration. When this field is set,
+	// the configuration is used as the persistence for platform services and sonataflow instances
+	// that don't provide one of their own.
+	// +optional
+	Persistence *PersistencePlatformSpec `json:"persistence,omitempty"`
+}
+
+// PersistencePlatformSpec configures the DataBase support for both platform services and workflows. For services, it allows
+// configuring a generic database connectivity if the service does not come with its own configured. In case of workflows,
+// the operator will add the necessary JDBC properties to in the workflow's application.properties so that it can communicate
+// with the persistence service based on the spec provided here.
+// +optional
+type PersistencePlatformSpec struct {

Review Comment:
   I think `PersistenceOptionsSpec` makes more sense because this is used in both the platform and workflow objects.



##########
api/v1alpha08/sonataflowplatform_services_types.go:
##########
@@ -32,61 +32,8 @@ type ServiceSpec struct {
 	Enabled *bool `json:"enabled,omitempty"`
 	// Persists service to a datasource of choice. Ephemeral by default.
 	// +optional
-	Persistence *PersistenceOptions `json:"persistence,omitempty"`
+	Persistence *PersistencePlatformSpec `json:"persistence,omitempty"`
 	// PodTemplate describes the deployment details of this platform service instance.
 	//+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="podTemplate"
 	PodTemplate PodTemplateSpec `json:"podTemplate,omitempty"`
 }
-
-// PersistenceOptions configure the services to persist to a datasource of choice
-// +kubebuilder:validation:MaxProperties=1

Review Comment:
   it's ok to rename this but this CRD schema validation config should be brought over to the new struct. this is critical going forward as additional persistence types are added. a user should only be allowed to select one at a given time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478459116


##########
api/v1alpha08/sonataflow_types.go:
##########
@@ -657,8 +657,7 @@ type SonataFlowSpec struct {
 	// PodTemplate describes the deployment details of this SonataFlow instance.
 	//+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="podTemplate"
 	PodTemplate PodTemplateSpec `json:"podTemplate,omitempty"`
-	// Persists service to a datasource of choice. Ephemeral by default.
-	// +optional
+	// Persistence defines the database persistence configuration for the workflow
 	Persistence *PersistenceOptions `json:"persistence,omitempty"`

Review Comment:
   I think `PersistenceOptionsSpec` makes more sense because this is used in both the platform and workflow objects.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478532326


##########
controllers/platform/services/services.go:
##########
@@ -358,14 +371,26 @@ func (j JobServiceHandler) MergeContainerSpec(containerSpec *corev1.Container) (
 	return c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Job service specification is not nil
+func (j JobServiceHandler) hasPostgreSQLConfigured() bool {
+	return (j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil) ||
+		(j.platform.Spec.Persistence != nil && j.platform.Spec.Persistence.PostgreSQL != nil)
+}

Review Comment:
   i wonder if this should be enhanced by adding `j.IsServiceSetInSpec` to the check as well. like this?
   ```golang
   	return (j.IsServiceSetInSpec && j.platform.Spec.Services.DataIndex.Persistence != nil && j.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil) ||
   		(j.platform.Spec.Persistence != nil && j.platform.Spec.Persistence.PostgreSQL != nil)
   ```



##########
controllers/platform/services/services.go:
##########
@@ -198,11 +198,24 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil) ||
+		(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil)
+}

Review Comment:
   i wonder if this should be enhanced by adding `d.IsServiceSetInSpec` to the check as well. like this?
   ```golang
   	return (d.IsServiceSetInSpec && d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil) ||
   		(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1478887261


##########
controllers/platform/services/services.go:
##########
@@ -198,11 +198,24 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil) ||
+		(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil)
+}

Review Comment:
   my suggestion is more about avoiding a null pointer exception. how can you check if `d.platform.Spec.Services.DataIndex.Persistence != nil` when `d.platform.Spec.Services.DataIndex == nil`?
   
   for the same reason you're already checking `d.platform.Spec.Services.DataIndex.Persistence` before you check `d.platform.Spec.Services.DataIndex.Persistence. PostgreSQL`



##########
controllers/platform/services/services.go:
##########
@@ -198,11 +198,24 @@ func (d DataIndexHandler) MergePodSpec(podSpec corev1.PodSpec) (corev1.PodSpec,
 	return *c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Data Index service specification is not nil
+func (d DataIndexHandler) hasPostgreSQLConfigured() bool {
+	return (d.platform.Spec.Services.DataIndex.Persistence != nil && d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL != nil) ||
+		(d.platform.Spec.Persistence != nil && d.platform.Spec.Persistence.PostgreSQL != nil)
+}

Review Comment:
   my suggestion is more about avoiding a null pointer exception. how can you check if `d.platform.Spec.Services.DataIndex.Persistence != nil` when `d.platform.Spec.Services.DataIndex == nil`?
   
   for the same reason you're already checking `d.platform.Spec.Services.DataIndex.Persistence` before you check `d.platform.Spec.Services.DataIndex.Persistence.PostgreSQL`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1947643844

   > @tchughesiv we should remove it if it won't be used. Having an attribute that can be set and then set an error saying that must not be set smells.
   
   Right now it doesn't show any error. The documentation contains a note mentioning that the JDBC string should not contain a schema.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1480376534


##########
controllers/profiles/dev/states_dev.go:
##########
@@ -70,15 +73,14 @@ func (e *ensureRunningWorkflowState) Do(ctx context.Context, workflow *operatora
 
 	devBaseContainerImage := workflowdef.GetDefaultWorkflowDevModeImageTag()
 	// check if the Platform available
-	pl, err := platform.GetActivePlatform(ctx, e.C, workflow.Namespace)
-	if err == nil && len(pl.Spec.DevMode.BaseImage) > 0 {
-		devBaseContainerImage = pl.Spec.DevMode.BaseImage
+	if plf != nil && len(plf.Spec.DevMode.BaseImage) > 0 {
+		devBaseContainerImage = plf.Spec.DevMode.BaseImage
 	}
 	userPropsCM, _, err := e.ensurers.userPropsConfigMap.Ensure(ctx, workflow)
 	if err != nil {
 		return ctrl.Result{Requeue: false}, objs, err
 	}
-	managedPropsCM, _, err := e.ensurers.managedPropsConfigMap.Ensure(ctx, workflow, pl, common.ManagedPropertiesMutateVisitor(ctx, e.StateSupport.Catalog, workflow, pl, userPropsCM.(*corev1.ConfigMap)))
+	managedPropsCM, _, err := e.ensurers.managedPropsConfigMap.Ensure(ctx, workflow, plf, common.ManagedPropertiesMutateVisitor(ctx, e.StateSupport.Catalog, workflow, plf, userPropsCM.(*corev1.ConfigMap)))

Review Comment:
   Honestly, I don't know, perhaps it was during one of my re-re-re-factorings... I intended to change all variables named `platform` to avoid collision with the one package with that name, maybe I went too far here because there was no reason for the change 😄. Do you want me to revert it?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1480376906


##########
test/testdata/workflow/persistence/by_service/02-sonataflow_platform.yaml:
##########
@@ -21,7 +21,7 @@ spec:
     template:
       buildArgs:
       - name: QUARKUS_EXTENSIONS
-        value: org.kie.kogito:kogito-addons-quarkus-jobs-knative-eventing:999-SNAPSHOT,org.kie.kogito:kogito-addons-quarkus-persistence-jdbc:999-SNAPSHOT,io.quarkus:quarkus-jdbc-postgresql:3.2.9.Final,io.quarkus:quarkus-agroal:3.2.9.Final
+        value: org.kie.kogito:kogito-addons-quarkus-persistence-jdbc:999-SNAPSHOT,io.quarkus:quarkus-jdbc-postgresql:3.2.9.Final,io.quarkus:quarkus-agroal:3.2.9.Final

Review Comment:
   Will do



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1505079016


##########
test/e2e/workflow_test.go:
##########
@@ -198,16 +198,27 @@ var _ = Describe("Validate the persistence ", Ordered, func() {
 				}
 				Expect(h.Status).To(Equal(upStatus), "Pod health is not UP")
 				for _, c := range h.Checks {
-					if c.Name == "Database connections health check" {
-						Expect(c.Status).To(Equal(upStatus), "Pod's database connection is not UP")
-						return true
+					if withPersistence {
+						if c.Name == "Database connections health check" {
+							Expect(c.Status).To(Equal(upStatus), "Pod's database connection is not UP")
+							return true
+						}
+					} else {
+						Expect(c.Name).NotTo(Equal("Database connections health check"), "persistence configuration found in workflow")
 					}
 				}
+				if !withPersistence {
+					return true
+				}
 			}
 			return false
-		}, 12*time.Minute).Should(BeTrue())
+		}, 1).Should(BeTrue())

Review Comment:
   It should not take more than 1 minute to stabilize the pod's health status. The previous value, 12 minutes, is too long for something that should take seconds. Worst case it would take 12 minutes for the test to fail and that I think it's too long.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "tchughesiv (via GitHub)" <gi...@apache.org>.
tchughesiv commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1491091670


##########
controllers/platform/services/services.go:
##########
@@ -358,14 +366,35 @@ func (j JobServiceHandler) MergeContainerSpec(containerSpec *corev1.Container) (
 	return c, err
 }
 
+// hasPostgreSQLConfigured returns true when either the SonataFlow Platform PostgreSQL CR's structure or the one in the Job service specification is not nil
+func (j JobServiceHandler) hasPostgreSQLConfigured() bool {
+	return j.IsServiceSetInSpec() &&
+		((j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil) ||
+			(j.platform.Spec.Persistence != nil && j.platform.Spec.Persistence.PostgreSQL != nil))
+}
+
 func (j JobServiceHandler) ConfigurePersistence(containerSpec *corev1.Container) *corev1.Container {
 
-	if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSql != nil {
+	if j.hasPostgreSQLConfigured() {
+		var p *operatorapi.PersistencePostgreSQL
 		c := containerSpec.DeepCopy()
 		c.Image = j.GetServiceImageName(constants.PersistenceTypePostgreSQL)
-		c.Env = append(c.Env, persistence.ConfigurePostgreSqlEnv(j.platform.Spec.Services.JobService.Persistence.PostgreSql, j.GetServiceName(), j.platform.Namespace)...)
+		if j.platform.Spec.Services.JobService.Persistence != nil && j.platform.Spec.Services.JobService.Persistence.PostgreSQL != nil {
+			p = j.platform.Spec.Services.JobService.Persistence.PostgreSQL
+		} else {
+			p = &operatorapi.PersistencePostgreSQL{
+				SecretRef: j.platform.Spec.Persistence.PostgreSQL.SecretRef,
+				JdbcUrl:   j.platform.Spec.Persistence.PostgreSQL.JdbcUrl,
+			}
+			if j.platform.Spec.Persistence.PostgreSQL.ServiceRef != nil {
+				p.ServiceRef = &operatorapi.PostgreSQLServiceOptions{
+					SQLServiceOptions: j.platform.Spec.Persistence.PostgreSQL.ServiceRef,
+					DatabaseSchema:    j.GetServiceName()}
+			}
+		}

Review Comment:
   Any reason `persistence.RetrieveConfiguration` is not also being used here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1484373523


##########
api/v1alpha08/sonataflow_persistence_types.go:
##########
@@ -14,6 +14,32 @@
 
 package v1alpha08
 
+// PlatformPersistenceOptionsSpec configures the DataBase in the platform spec. This specification can
+// be used by workflows and platform services when they don't provide one of their own.
+// +optional
+// +kubebuilder:validation:MaxProperties=1

Review Comment:
   Isn´t it a new struct? The original one still remains unchanged from the API point of view. So mentioning that it doesn´t have the `databaseSchema` when it never had one sounds a bit strange to me. Maybe I'm missing something here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "jordigilh (via GitHub)" <gi...@apache.org>.
jordigilh commented on code in PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#discussion_r1481178920


##########
controllers/profiles/common/object_creators.go:
##########
@@ -157,7 +159,13 @@ func defaultContainer(workflow *operatorapi.SonataFlow) (*corev1.Container, erro
 	if err := mergo.Merge(defaultFlowContainer, workflow.Spec.PodTemplate.Container.ToContainer(), mergo.WithOverride); err != nil {
 		return nil, err
 	}
-	defaultFlowContainer = ConfigurePersistence(defaultFlowContainer, workflow.Spec.Persistence, defaultSchemaName, workflow.Namespace)
+	if workflow.Spec.Persistence != nil {

Review Comment:
   The problem is that by default it will always be nil, so it will try to retrieve from the platform. For backwards compatibility `nil` is the best choice as it will not change the behavior in existing workflows and doesn't require the CR to declare the field if no persistence is required.
   
   What @ricardozanini is proposing, in my opinion is not ideal because label/annotations are unstructured data (map of strings), whereas with a field we can enforce validation by defining accepted values, which we own and control. Example is the `imagePullPolicy` in the pod schema. I see  labels/annotations useful when the values can vary depending on the use case, such as the multus network configuration.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "wmedvede (via GitHub)" <gi...@apache.org>.
wmedvede commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1938230555

   > > 4. **workflowtimeouts WF, with the configuration below has failed to deploy:**
   > >    Here we would expect that the WF picks the platform persistence configuration and use it's own configured schema wf_timeouts.
   > 
   > this is not the expected behavior as i understand it. in the below scenario, since that you're setting `workflow.spec.persistence`, the platform persistence settings are NOT used. whatever is set in the workflow will be used instead. if you wanted to use the platform persistence settings, then remove `persistence` entirely from the workflow `spec` below.
   > 
   > > But the following error was produced at deployment. (Please @jordigilh correct me if the configuration is wrong.
   > > `The SonataFlow "workflowtimeouts" is invalid: spec.persistence.postgresql: Invalid value: 1: spec.persistence.postgresql in body should have at least 2 properties`
   > > ```
   > > apiVersion: sonataflow.org/v1alpha08
   > > kind: SonataFlow
   > > metadata:
   > >   name: workflowtimeouts
   > >   annotations:
   > >     sonataflow.org/description: Workflow Timeouts Example k8s
   > >     sonataflow.org/version: 0.0.1
   > > spec:
   > >   persistence:
   > >     postgresql:
   > >       serviceRef:
   > >         databaseSchema: wf_timeouts
   > > ```
   > 
   > @wmedvede this is because `secretRef` is required here, as well as either the `serviceRef` or `jdbcUrl` field. Just add `secretRef` and the error should go away.
   
   I see, so in this case we are not inheriting but defining all the persistence for this particular workflow instead.
   My initial understanding was that is should be possible inherit persistence and only declare the WF schema to use if desired.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org


Re: [PR] [KOGITO-9980] Add persistence to SonataflowPlatform CRD [incubator-kie-kogito-serverless-operator]

Posted by "wmedvede (via GitHub)" <gi...@apache.org>.
wmedvede commented on PR #372:
URL: https://github.com/apache/incubator-kie-kogito-serverless-operator/pull/372#issuecomment-1936585325

   In this other use case below, we are using a SFP with the global persistence configured to use the JdbcUrl to connect with the database service.
   
   https://github.com/flows-examples/techpreview2/tree/004b37b96ad5f75c804d8e8ba3d821ab67c86457/platforms/openshift-dataindex_and_jobservice_as_platform_service_and_global_persistence_jdbcurl_postgresql
   
   
   ```
     persistence:
       postgresql:
         secretRef:
           name: postgres-secrets
           userKey: POSTGRESQL_USER
           passwordKey: POSTGRESQL_PASSWORD
         jdbcUrl: jdbc:postgresql://postgres.usecase3-platform-persistence-jdbcurl-oc:5432/sonataflow2
   ```
   
   But, the corresponding generated specific jdbc url for each service looks like the ones below.
   I think we need a small adjustment to make each service use its specific schema, otherwise we have flyway issues when the services start. Not too much, a simple adjustment. 
   
   **DataIndex:**
   
       QUARKUS_DATASOURCE_JDBC_URL = jdbc:postgresql://postgres.usecase3-platform-persistence-jdbcurl-oc:5432/sonataflow2
   
   **JobsService:**
   
       QUARKUS_DATASOURCE_JDBC_URL = jdbc:postgresql://postgres.usecase3-platform-persistence-jdbcurl-oc:5432/sonataflow2
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@kie.apache.org
For additional commands, e-mail: commits-help@kie.apache.org