You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2021/02/19 19:14:27 UTC
[GitHub] [camel-kafka-connector] ffang opened a new issue #1044: add processor configuration for source connectors
ffang opened a new issue #1044:
URL: https://github.com/apache/camel-kafka-connector/issues/1044
Source connectors basically will start a consumer endpoint "from uri", but currently no way to add a processor class after from.
The processor is really flexible and can do pretty much everything on the camel exchange before send into kafka, and I believe it's good if we can specify processor for source connectors
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1044: Should processor configuration for source connectors be added?
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1044:
URL: https://github.com/apache/camel-kafka-connector/issues/1044#issuecomment-783370019
I think we should close the related PR and close this discussion then.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] ffang commented on issue #1044: add processor configuration for source connectors
Posted by GitBox <gi...@apache.org>.
ffang commented on issue #1044:
URL: https://github.com/apache/camel-kafka-connector/issues/1044#issuecomment-782285288
Open a PR here
https://github.com/apache/camel-kafka-connector/pull/1045
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd edited a comment on issue #1044: add processor configuration for source connectors
Posted by GitBox <gi...@apache.org>.
oscerd edited a comment on issue #1044:
URL: https://github.com/apache/camel-kafka-connector/issues/1044#issuecomment-782292439
Basically the idea of ckc is avoid exposing Camel details and creating a tiny abstraction layers between kafka connect and camel. To use a processor people should know at least a bit of camel and I fear that starting to add too much camel concepts in the project, will make it just a mega camel wrapper.
Btw let's discuss this with the community.
@valdar @orpiske etc. Please give your feedback :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] ffang commented on issue #1044: add processor configuration for source connectors
Posted by GitBox <gi...@apache.org>.
ffang commented on issue #1044:
URL: https://github.com/apache/camel-kafka-connector/issues/1044#issuecomment-782289661
Hi @oscerd ,
I may miss something, but could you please elaborate how to use camel directly?
Because if I read the CKC code correctly, when initialising a source connector, the consumer endpoint is created(from) automatically, in this case how can I use processor if I do want to add something for the consumer endpoint?
Thanks!
Freeman
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1044: add processor configuration for source connectors
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1044:
URL: https://github.com/apache/camel-kafka-connector/issues/1044#issuecomment-782292439
Basically the idea of ckc is avoid exposing Camel details and creating a tiny abstraction layers between kafka connect and camel. To use a processor people should know at least a bit of camel and I fear that starting to add too much camel concepts in the project, while make it just a mega camel wrapper.
Btw let's discuss this with the community.
@valdar @orpiske etc. Please give your feedback :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] tadayosi commented on issue #1044: add processor configuration for source connectors
Posted by GitBox <gi...@apache.org>.
tadayosi commented on issue #1044:
URL: https://github.com/apache/camel-kafka-connector/issues/1044#issuecomment-782804218
I think it's a great discussion. Personally what @oscerd and @orpiske explained really makes sense: CKC is the complehensive collection of generic Kafka connectors abstracted away from the implementation by Camel. It's a great, focused strategy for the project. Here are some constructive suggestions from me in this perspective:
1. At the same time, what @ffang expected with CKC shouldn't be uncommon among users, especially for existing Camel users. We should write clearly somewhere this basic philosophy of keeping itself as a thin abstraction layer. The existing basic concepts document doesn't seem to address it well:
https://camel.apache.org/camel-kafka-connector/latest/basic-concepts.html
2. Also we should address common questions from users, such as when CKC cannot be the best solution, when we should get away from CKC and directly use Camel Core, what my Camel application (architecture) would be like when we move away from CKC, etc.
3. Talking about branding, perhaps **Camel** Kafka Connector might not be the best name for the project, as it might give users a wrong expectation that they could use the full power of Camel framework with it. Probably we should stick to the abbreviation, CKC, as the primary name for the project, implying it rather as the _Common_ Kafka Connectors, or the _Collection_ of Kafka Connectors (like what [OKD](https://www.okd.io/) is trying to do). You could still use _Powered by Apache Camel_ as the second punch-line or subtitle.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1044: add processor configuration for source connectors
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1044:
URL: https://github.com/apache/camel-kafka-connector/issues/1044#issuecomment-782285478
This is out of the roadmap. In camel-kafka-connector there is no need for processors. If a user need a processor then he should use camel directly. Camel is the engine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] valdar commented on issue #1044: add processor configuration for source connectors
Posted by GitBox <gi...@apache.org>.
valdar commented on issue #1044:
URL: https://github.com/apache/camel-kafka-connector/issues/1044#issuecomment-783306526
Hello, so my opinion on this is aligned with what @orpiske , @oscerd and partially @tadayosi explained so well.
The idea is to abstract away camel details as much as possible. Moreover I am introducing in a PoC the use of Kamelets in ckc and additions to the ckc core makes this transition harder. Ideally once and if we move to use kamelets those processors can be added in the kamelet itself just for the connectors that would need it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] orpiske commented on issue #1044: add processor configuration for source connectors
Posted by GitBox <gi...@apache.org>.
orpiske commented on issue #1044:
URL: https://github.com/apache/camel-kafka-connector/issues/1044#issuecomment-782299193
My understanding is in line w/ @oscerd's explanation.
Personally, I'd prefer the project to abstract away some Camel features and patterns in order to offer a *simple* "Kafka connect way" for the users to plug other systems into their connectors. Although we may need to leave certain features behind - such as processors - my belief is that solutions that are complex enough to require some these features may be better done with Camel Core in the first place.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] ffang commented on issue #1044: Should processor configuration for source connectors be added?
Posted by GitBox <gi...@apache.org>.
ffang commented on issue #1044:
URL: https://github.com/apache/camel-kafka-connector/issues/1044#issuecomment-783584493
Thanks for all the great feedback!
I'm digesting the input now, I may ask more questions later on.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1044: add processor configuration for source connectors
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1044:
URL: https://github.com/apache/camel-kafka-connector/issues/1044#issuecomment-782672831
If you need to do something like manipulate the message while moving from source to kafka, you should use an SMT or a converter (kafka concepts), introducing processors is not really in the Kafka connect perspective.
If SMT and converter are not enough, probably CKC is not the best solution for the user. At that point he should use plain Camel by combining a consumer and camel-kafka producer.
We have aggregation and idempotent repository, because these are concepts used also in the Kafka connect world.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel-kafka-connector] oscerd commented on issue #1044: add processor configuration for source connectors
Posted by GitBox <gi...@apache.org>.
oscerd commented on issue #1044:
URL: https://github.com/apache/camel-kafka-connector/issues/1044#issuecomment-782874796
I agree on 1. and 2., I disagree a bit on .3, because the connectors work exactly as the camel component behaves and they are generated directly from the components. So Camel should be in the name, because basically we're still using the components, but in the kafka connect way.
There are some documentation issue about this and about use cases.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org