You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2020/10/23 07:29:41 UTC
[GitHub] [camel-k] lburgazzoli opened a new issue #1779: KameletBiding : support for partitions for sources
lburgazzoli opened a new issue #1779:
URL: https://github.com/apache/camel-k/issues/1779
To improve performances, we should be able to partition a source.
We should have support for static partitioning and optionally a dynamic one.
**Static partitions**
we could add the relevant information in the form of properties to a KameletBinding, something like:
```yaml
apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
name: jdbc-source
spec:
source:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1alpha1
name: jdbc-source
properties:
partition-keys: <1>
- table1
- table2
- table3
partition-property: <2>
- tables
sink:
ref:
kind: KafkaTopic
apiVersion: kafka.strimzi.io/v1beta1
name: my-topic
```
1. list of values used to logically partition the source
2. the property used by the underlying kamelet to get a reference to the partition keys it has to use
The operator then detect the presence of the special properties `partition-keys` and `partition-property` and could create as example a `StatefulSet` of integrations where each integration receives only a subset of the `partition-keys`
**Dynamic partitions**
Instead of hardcoding the values of the partitions, the operator could use a meta service provided by the kamelet to determine how a source can be paritioned
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] heiko-braun commented on issue #1779: KameletBiding : support for partitions for sources
Posted by GitBox <gi...@apache.org>.
heiko-braun commented on issue #1779:
URL: https://github.com/apache/camel-k/issues/1779#issuecomment-715085290
Using a specific `partition-*` property would allow us to distinguish between partionable and non-partionable sources.
This information might be useful to determine a rollover/redployments strategy for instance.
If the properties remain domain/kamelet specific, rather being generalised, identifying partionable sources might not easily be possible, would it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] github-actions[bot] commented on issue #1779: KameletBiding : support for partitions for sources
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #1779:
URL: https://github.com/apache/camel-k/issues/1779#issuecomment-994157745
This issue has been automatically marked as stale due to 90 days of inactivity.
It will be closed if no further activity occurs within 15 days.
If you think that’s incorrect or the issue should never stale, please simply write any comment.
Thanks for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] lburgazzoli commented on issue #1779: KameletBiding : support for partitions for sources
Posted by GitBox <gi...@apache.org>.
lburgazzoli commented on issue #1779:
URL: https://github.com/apache/camel-k/issues/1779#issuecomment-715133305
@nicolaferraro I think we should have support `StatefulSet` and the `master` one as the `StatefulSet` is simple to reason about whereas the dynamic one may be a little bit tricky from an operation point of view.
@heiko-braun I think @nicolaferraro proposla to to add metadata to the kamelet is right as it would let any tool to act according (i.e. by leveraging meta services to get a possible list of partitions) so we could add some additional info as labels which would make it easy to search them using standard k8s tool and practice (i.e. kubectl get kameletbindings -l type=partitionable)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-k] nicolaferraro commented on issue #1779: KameletBiding : support for partitions for sources
Posted by GitBox <gi...@apache.org>.
nicolaferraro commented on issue #1779:
URL: https://github.com/apache/camel-k/issues/1779#issuecomment-715074973
I think we can keep using the same model that we have now, like:
```yaml
apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
name: jdbc-source
spec:
source:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1alpha1
name: jdbc-source
properties:
tables:
- table1
- table2
- table3
sink:
ref:
kind: KafkaTopic
apiVersion: kafka.strimzi.io/v1beta1
name: my-topic
```
The information that tells the operator that "tables" is a partition key is present on the kamelet, e.g. with a `x-camel-partition-key` flag in the descriptor.
The assignment of partition and rebalancing can be also done with a modified version of the master component instead of StatefulSets, to make it more dynamic.
Wdyt?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org