You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@streampipes.apache.org by "dominikriemer (via GitHub)" <gi...@apache.org> on 2023/08/03 21:01:05 UTC

[D] Integrate service discovery into core?

GitHub user dominikriemer created a discussion: Integrate service discovery into core?

Hi,
currently, StreamPipes has a dependency on Consul for two reasons:
* Extensions services register at Consul and the core makes use of Consul's service discovery to find endpoints that support pipeline elements and adapters
* The core uses Consul's key/value store to store configs such as email, authentication settings, broker settings and more. In addition, extensions can define configurations which are also stored in Consul.

I think it should be pretty to integrate the service discovery logic directly into the core and store the configurations in CouchDB, which would make our Consul dependency obsolete, reducing system complexity.

## Approach

The approach would be:
* Extensions services register at the core upon startup, using a new endpoint which stores the service configuration in CouchDB. The core's hostname would be `backend` by default in the docker-compose installation, link to the backend service in k8s and could be overridden with an environment variable similar to `SP_CONSUL_LOCATION` for development purposes.
* The core then uses its own service discovery mechanism based on stored service configurations when pipelines and adapters are started.
* All configurations of the core which are currently stored in Consul would move to a new CouchDB database `core-configuration`.
* Extensions services could store their configs in another CouchDB database which can be accessed over the client from the core.

## Migration
* We don't need a migration for the service discovery change, assuming that extensions services and the core are running the same StreamPipes version
* For the key/value store configurations, we need a migration. We could add a migration in release 0.93.0 which transfers all Consul configs to the CouchDB database, and remove Consul in the release thereafter.

What do ou think of this idea?

GitHub link: https://github.com/apache/streampipes/discussions/1811

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Integrate service discovery into core?

Posted by "bossenti (via GitHub)" <gi...@apache.org>.

GitHub user bossenti added a comment to the discussion: Integrate service discovery into core?

> it's not overly complex since we can reuse most existing APIs, so that the change actually deletes more code than it adds (I guess). So I think it reduces complexity by a lot.

I will mark your words here 😉 

Joke aside, then I'm also +1 for this idea


> Concerning the version, I think we've never supported or tested compatibility between core and extensions services that do not have the same version, so I don't think it's an overly restrictive assumption.

Indee

GitHub link: https://github.com/apache/streampipes/discussions/1811#discussioncomment-6642380

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Integrate service discovery into core?

Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.

GitHub user dominikriemer added a comment to the discussion: Integrate service discovery into core?

@bossenti it's not overly complex since we can reuse most existing APIs, so that the change actually deletes more code than it adds (I guess). So I think it reduces complexity by a lot.

Concerning the version, I think we've never supported or tested compatibility between core and extensions services that do not have the same version, so I don't think it's an overly restrictive assumption.

GitHub link: https://github.com/apache/streampipes/discussions/1811#discussioncomment-6642332

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Integrate service discovery into core?

Posted by "tenthe (via GitHub)" <gi...@apache.org>.

GitHub user tenthe added a comment to the discussion: Integrate service discovery into core?

Hi @dominikriemer,
I think this idea is great and will reduce the size of StreamPipes installation.
+1 from my side.

I didn't quite understand why the migration is split into two versions. Wouldn't that mean we have to maintain the data in two places in release 0.93.0?

Cheers,
Philipp

GitHub link: https://github.com/apache/streampipes/discussions/1811#discussioncomment-6639546

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Integrate service discovery into core?

Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.

GitHub user dominikriemer added a comment to the discussion: Integrate service discovery into core?

Thanks for asking @tenthe!
In v0.93.0, we would add an migration to copy all existing configs from the key value store to CouchDB. All variables will then already be read from and written to CouchDB. In the next release, we can then remove Consul.
We need Consul in another release to make sure that customized configs are properly copied to CouchDB.

GitHub link: https://github.com/apache/streampipes/discussions/1811#discussioncomment-6639765

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Integrate service discovery into core?

Posted by "tenthe (via GitHub)" <gi...@apache.org>.

GitHub user tenthe added a comment to the discussion: Integrate service discovery into core?

Ah ok, thanks for the explanation. 
That makes perfect sense. So if a user wants to get rid of Consul manually, it could be removed after the migration script was executed, right? 

GitHub link: https://github.com/apache/streampipes/discussions/1811#discussioncomment-6640502

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Integrate service discovery into core?

Posted by "bossenti (via GitHub)" <gi...@apache.org>.

GitHub user bossenti added a comment to the discussion: Integrate service discovery into core?

Thanks for bringing this up for discussion and for the well-written suggestion @dominikriemer 
In general, I like the idea of moving towards a slimmer setup of StreamPipes.
However, I think when making such a decision, we should always weigh the benefit of getting rid of an external dependency against the complexity we add to our codebase by implementing the functionality ourselves.
Can you estimate how much and how complex custom code we would add to StreamPipes by replacing Consul? 

> We don't need a migration for the service discovery change, assuming that extensions services and the core are running the same StreamPipes version

Can we really hold on to this assumption? Isn't it to restrictive?

Other than that, your migration plan sounds got to me 👍🏼 

GitHub link: https://github.com/apache/streampipes/discussions/1811#discussioncomment-6642145

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Integrate service discovery into core?

Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.

GitHub user dominikriemer edited a discussion: Integrate service discovery into core?

I think it should be pretty easy to integrate the service discovery logic directly into the core and store the configurations in CouchDB, which would make our Consul dependency obsolete, reducing system complexity.

## Approach

What do ou think of this idea?

GitHub link: https://github.com/apache/streampipes/discussions/1811

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Integrate service discovery into core?

Posted by "bossenti (via GitHub)" <gi...@apache.org>.

GitHub user bossenti edited a comment on the discussion: Integrate service discovery into core?

> it's not overly complex since we can reuse most existing APIs, so that the change actually deletes more code than it adds (I guess). So I think it reduces complexity by a lot.

I will mark your words here 😉 

Joke aside, then I'm also +1 for this idea


> Concerning the version, I think we've never supported or tested compatibility between core and extensions services that do not have the same version, so I don't think it's an overly restrictive assumption.

Indeed

GitHub link: https://github.com/apache/streampipes/discussions/1811#discussioncomment-6642380

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org

Re: [D] Integrate service discovery into core?

Posted by "dominikriemer (via GitHub)" <gi...@apache.org>.

GitHub user dominikriemer added a comment to the discussion: Integrate service discovery into core?

@tenthe right now I don't think this can be done since we probably can remove the `BackendConfig.INSTANCE.isConfigured()` call which is needed in `StreamPipesCoreApplication` only after the migration has taken place to ensure that the system can properly identify an unconfigured system and start the initial installation procedure.

GitHub link: https://github.com/apache/streampipes/discussions/1811#discussioncomment-6642300

----
This is an automatically sent email for dev@streampipes.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@streampipes.apache.org