You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by Pavel Tupitsyn <pt...@apache.org> on 2023/05/23 10:48:37 UTC
[DISCUSSION] IEP-102: Data Streamer for Ignite 3.x
Igniters,
Please review Ignite 3.x Data Streamer proposal
https://cwiki.apache.org/confluence/display/IGNITE/IEP-102%3A+Data+Streamer
Re: [DISCUSSION] IEP-102: Data Streamer for Ignite 3.x
Posted by Pavel Tupitsyn <pt...@apache.org>.
> 1. How will partition awareness be achieved? Is the plan to consume the
stream and then partition and transmit the data to where it needs to go?
Yes, the client performs partitioning. I'm not aware of custom affinity
plans for Ignite 3.x
> 2. Is the plan to use consistent key/thread assignment to provide the
sequence guarantee?
Can you please clarify? Is this about a multithreaded publisher or
subscriber?
On Mon, Jun 5, 2023 at 1:48 AM Colin Cassidy
<co...@yahoo.co.uk.invalid> wrote:
> This looks great. A couple of questions:
> 1. How will partition awareness be achieved? Is the plan to consume the
> stream and then partition and transmit the data to where it needs to go? Or
> will there be some way to work with 3rd party event streams to ensure that
> events are delivered to the right place? For example, it is possible in
> Ignite 2 to override the affinityfunction to align to Kafka partitions &
> consumer assignment - thereby avoiding a second hop. 2. Is the plan to use
> consistent key/thread assignment to provide the sequence guarantee? That
> would avoid the need to use transactions or other locks in many cases -
> especially if the same thread assignment could be used for accessing the
> data.
> Regards,Colin.
>
> On Tuesday, 23 May 2023 at 11:48:55 BST, Pavel Tupitsyn <
> ptupitsyn@apache.org> wrote:
>
> Igniters,
>
> Please review Ignite 3.x Data Streamer proposal
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-102%3A+Data+Streamer
>
Re: [DISCUSSION] IEP-102: Data Streamer for Ignite 3.x
Posted by Colin Cassidy <co...@yahoo.co.uk.INVALID>.
This looks great. A couple of questions:
1. How will partition awareness be achieved? Is the plan to consume the stream and then partition and transmit the data to where it needs to go? Or will there be some way to work with 3rd party event streams to ensure that events are delivered to the right place? For example, it is possible in Ignite 2 to override the affinityfunction to align to Kafka partitions & consumer assignment - thereby avoiding a second hop. 2. Is the plan to use consistent key/thread assignment to provide the sequence guarantee? That would avoid the need to use transactions or other locks in many cases - especially if the same thread assignment could be used for accessing the data.
Regards,Colin.
On Tuesday, 23 May 2023 at 11:48:55 BST, Pavel Tupitsyn <pt...@apache.org> wrote:
Igniters,
Please review Ignite 3.x Data Streamer proposal
https://cwiki.apache.org/confluence/display/IGNITE/IEP-102%3A+Data+Streamer
Re: [DISCUSSION] IEP-102: Data Streamer for Ignite 3.x
Posted by Pavel Tupitsyn <pt...@apache.org>.
Basic streamer in Java client is ready for review:
https://issues.apache.org/jira/browse/IGNITE-19540
https://github.com/apache/ignite-3/pull/2128
On Tue, May 23, 2023 at 1:48 PM Pavel Tupitsyn <pt...@apache.org> wrote:
> Igniters,
>
> Please review Ignite 3.x Data Streamer proposal
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-102%3A+Data+Streamer
>
>