You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@streampipes.apache.org by zhangjun <zh...@126.com> on 2021/08/04 09:36:12 UTC
Some questions about kafka adapter
Hi,all:
I found that when I create a pipeline with a Kafka adapter,
It will first consume data from Kafka, write a new randomly generated topic, and then consume data from this new topic. I think this will cause data redundancy. I want to know why to do this?
Is there any relevant design document about the streampipes project？
Thanks.
Re: Some questions about kafka adapter
Posted by Dominik Riemer <ri...@apache.org>.
Hi Zhangjun,
sorry for replying so late - I really like your idea, so my plan is to create a wiki page within the next days so that we can collect requirements and further discuss a technical roadmap!
From my point of view, it would be a good start to integrate processors and sinks into a Flink Job while keeping adapters separately - the reason is that we have quite a few deployments where users run the adapter part closer to the edge, where the Flink job might not be able to access the data source.
Concerning the formats, we use the event class, but there is also an underlying Map representation which we could use, that's how we currently also do with the Flink wrapper. I'm not yet familiar with the RowData format - does it work with a schema which is unknown at runtime?
Dominik
On 2021/08/23 08:58:26, zhangjun <zh...@126.com> wrote:
> Hi , Dominik：
>
> Sorry for the late reply。
>
> 1.
> I read the wiki about the ‘Improve Service Discovery’, I understand the implementation of the new architecture.
>
> In fact, my idea is to integrate all pipeline elements into a flink job(adapter, processor, sink), and each pipeline element no longer writes intermediate data to Kafka. The reason is that we can make full use of the advantages of flink (fault tolerance mechanism , Large throughput), processor and sink are relatively easy to implement. If adapter is also implemented with flink,I think some refactoring may be needed. I don't know whether the architecture I want can be realized.
>
> 2.Develop pipeline elements for both flink and JVM sound good, we can save a lot of development work. How do you plan to transfer data between elements? In streampipes, we use Event to transfer data, and generally use RowData in flink. How do we efficiently perform data conversion?
>
>
>
>
> 在 11.8.21 下午2:54，“Dominik Riemer”<dev-return-1990-zhangjunemail=126.com@streampipes.apache.org 代表 riemer@apache.org> 写入:
>
> Hi Zhangjun,
>
> sounds great - I also think that supporting JVM and Flink in distributed mode would be sufficient and better focused.
> I like your idea and putting the elements into a single Flink Job should work. We just recently did some improvements related to service discovery which should make it easier to integrate all pipeline elements into a single jar, which would be a precondition for this approach (see [1])
>
> Also, I just roughly thought over the idea of improving the Flink wrapper so that all pipeline elements can be run in Flink, e.g., by wrapping the new "compact" definition of pipeline elements [2] into a ProcessFunction so that we wouldn't need to develop pipeline elements for either the flink or the JVM wrapper but support both at the same time.
>
> What do you think?
>
> Dominik
>
>
> [1] https://github.com/apache/incubator-streampipes-extensions/blob/STREAMPIPES-319/streampipes-pipeline-elements-all-jvm/src/main/java/org/apache/streampipes/pe/jvm/AllPipelineElementsInit.java
>
> [2] https://github.com/apache/incubator-streampipes-extensions/blob/STREAMPIPES-319/streampipes-sinks-brokers-jvm/src/main/java/org/apache/streampipes/sinks/brokers/jvm/websocket/WebsocketServerSink.java
>
>
> On 2021/08/10 02:13:13, zhangjun <zh...@126.com> wrote:
> >
> > Hi , Dominik :
> > By reading the source code, I guessed that using Kafka for transform may be to adapt to different engines in the same pipeline, such as flink, spark, and jvm. For this situation, I haven't thought of a better method for the time being.
> >
> > For flink job, my idea is to do a check, if all the elements in the pipeline are flink tasks, we can merge them into one flink job instead of the current N-1 flink jobs (N is the number of elements in the pipeline).In this way, flink can also be used to ensure the fault tolerance of job and the accuracy of data.
> >
> >
> >
> > 在 10.8.21 上午4:53，“Dominik Riemer”<dev-return-1977-zhangjunemail=126.com@streampipes.apache.org 代表 riemer@apache.org> 写入:
> >
> > Hi Zhangjun,
> >
> > welcome to the list! Sorry for the late reply, I somehow didn't get your email and just found it in the web interface.
> >
> > The reason for that is that all adapters are integrated into the same message broker, which is mainly Kafka (in theory, it should also be possible to change the messaging layer, but this is not fully automated right now). So while this is clearly suboptimal for Kafka, we didn't want to introduce any special treatment for the Kafka adapter. Another reason is that users can apply pre-processing functions at ingestion time (e.g., converting timestamps, filtering fields and so on) so that it is not guaranteed that the original Kafka topic can be reused. A workaround could be to create a custom source using the SDK which points to the original Kafka topic, in this case the topic wouldn't be replicated.
> >
> > Concerning Flink, we haven't focused so much on the Flink wrapper recently due to other priorities, so there is defintely room for improvement. Do you have any idea how to approach this?
> >
> > You can find some developer-oriented information in the wiki [1], although not a complete design document exists. So please feel free to ask any questions you might have.
> >
> > I hope I could answer some of your questions!
> >
> > Dominik
> >
> >
> > [1] https://cwiki.apache.org/confluence/display/STREAMPIPES/Apache+StreamPipes+%28incubating%29+Developer+Wiki
> >
> >
> >
> >
> > On 2021/08/04 09:46:13, zhangjun <zh...@126.com> wrote:
> > >
> > >
> > > I think this architecture design has the following problems：
> > >
> > >
> > >
> > > 1. Use Kafka java api to consume Kafka data. If the data is large, it will cause a data backlog.
> > >
> > > 2. If the downstream is a flink job, using kafka java api to consume kafka data is not fault-tolerant. Once a problem occurs, data will be lost, and Exactly-once semantics cannot be guaranteed.
> > >
> > > 3. We consume data from the original Kafka and write to a new topic, which will cause data redundancy and duplication
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > 发件人: <us...@streampipes.apache.org> 代表 zhangjun <zh...@126.com>
> > > 答复: <us...@streampipes.apache.org>
> > > 日期: 2021年8月4日 星期三 下午5:36
> > > 收件人: <de...@streampipes.apache.org>, <us...@streampipes.apache.org>
> > > 主题: Some questions about kafka adapter
> > >
> > >
> > >
> > >
> > >
> > > Hi,all:
> > >
> > > I found that when I create a pipeline with a Kafka adapter,
> > >
> > >
> > >
> > > It will first consume data from Kafka, write a new randomly generated topic, and then consume data from this new topic. I think this will cause data redundancy. I want to know why to do this?
> > >
> > >
> > >
> > > Is there any relevant design document about the streampipes project？
> > >
> > >
> > >
> > > Thanks.
> > >
> > >
> >
> >
> >
>
>
>
Re: Some questions about kafka adapter
Posted by zhangjun <zh...@126.com>.
Hi , Dominik：
Sorry for the late reply。
1.
I read the wiki about the ‘Improve Service Discovery’, I understand the implementation of the new architecture.
In fact, my idea is to integrate all pipeline elements into a flink job(adapter, processor, sink), and each pipeline element no longer writes intermediate data to Kafka. The reason is that we can make full use of the advantages of flink (fault tolerance mechanism , Large throughput), processor and sink are relatively easy to implement. If adapter is also implemented with flink,I think some refactoring may be needed. I don't know whether the architecture I want can be realized.
2.Develop pipeline elements for both flink and JVM sound good, we can save a lot of development work. How do you plan to transfer data between elements? In streampipes, we use Event to transfer data, and generally use RowData in flink. How do we efficiently perform data conversion?
在 11.8.21 下午2:54，“Dominik Riemer”<dev-return-1990-zhangjunemail=126.com@streampipes.apache.org 代表 riemer@apache.org> 写入:
Hi Zhangjun,
sounds great - I also think that supporting JVM and Flink in distributed mode would be sufficient and better focused.
I like your idea and putting the elements into a single Flink Job should work. We just recently did some improvements related to service discovery which should make it easier to integrate all pipeline elements into a single jar, which would be a precondition for this approach (see [1])
Also, I just roughly thought over the idea of improving the Flink wrapper so that all pipeline elements can be run in Flink, e.g., by wrapping the new "compact" definition of pipeline elements [2] into a ProcessFunction so that we wouldn't need to develop pipeline elements for either the flink or the JVM wrapper but support both at the same time.
What do you think?
Dominik
[1] https://github.com/apache/incubator-streampipes-extensions/blob/STREAMPIPES-319/streampipes-pipeline-elements-all-jvm/src/main/java/org/apache/streampipes/pe/jvm/AllPipelineElementsInit.java
[2] https://github.com/apache/incubator-streampipes-extensions/blob/STREAMPIPES-319/streampipes-sinks-brokers-jvm/src/main/java/org/apache/streampipes/sinks/brokers/jvm/websocket/WebsocketServerSink.java
On 2021/08/10 02:13:13, zhangjun <zh...@126.com> wrote:
>
> Hi , Dominik :
> By reading the source code, I guessed that using Kafka for transform may be to adapt to different engines in the same pipeline, such as flink, spark, and jvm. For this situation, I haven't thought of a better method for the time being.
>
> For flink job, my idea is to do a check, if all the elements in the pipeline are flink tasks, we can merge them into one flink job instead of the current N-1 flink jobs (N is the number of elements in the pipeline).In this way, flink can also be used to ensure the fault tolerance of job and the accuracy of data.
>
>
>
> 在 10.8.21 上午4:53，“Dominik Riemer”<dev-return-1977-zhangjunemail=126.com@streampipes.apache.org 代表 riemer@apache.org> 写入:
>
> Hi Zhangjun,
>
> welcome to the list! Sorry for the late reply, I somehow didn't get your email and just found it in the web interface.
>
> The reason for that is that all adapters are integrated into the same message broker, which is mainly Kafka (in theory, it should also be possible to change the messaging layer, but this is not fully automated right now). So while this is clearly suboptimal for Kafka, we didn't want to introduce any special treatment for the Kafka adapter. Another reason is that users can apply pre-processing functions at ingestion time (e.g., converting timestamps, filtering fields and so on) so that it is not guaranteed that the original Kafka topic can be reused. A workaround could be to create a custom source using the SDK which points to the original Kafka topic, in this case the topic wouldn't be replicated.
>
> Concerning Flink, we haven't focused so much on the Flink wrapper recently due to other priorities, so there is defintely room for improvement. Do you have any idea how to approach this?
>
> You can find some developer-oriented information in the wiki [1], although not a complete design document exists. So please feel free to ask any questions you might have.
>
> I hope I could answer some of your questions!
>
> Dominik
>
>
> [1] https://cwiki.apache.org/confluence/display/STREAMPIPES/Apache+StreamPipes+%28incubating%29+Developer+Wiki
>
>
>
>
> On 2021/08/04 09:46:13, zhangjun <zh...@126.com> wrote:
> >
> >
> > I think this architecture design has the following problems：
> >
> >
> >
> > 1. Use Kafka java api to consume Kafka data. If the data is large, it will cause a data backlog.
> >
> > 2. If the downstream is a flink job, using kafka java api to consume kafka data is not fault-tolerant. Once a problem occurs, data will be lost, and Exactly-once semantics cannot be guaranteed.
> >
> > 3. We consume data from the original Kafka and write to a new topic, which will cause data redundancy and duplication
> >
> >
> >
> >
> >
> >
> >
> > 发件人: <us...@streampipes.apache.org> 代表 zhangjun <zh...@126.com>
> > 答复: <us...@streampipes.apache.org>
> > 日期: 2021年8月4日 星期三 下午5:36
> > 收件人: <de...@streampipes.apache.org>, <us...@streampipes.apache.org>
> > 主题: Some questions about kafka adapter
> >
> >
> >
> >
> >
> > Hi,all:
> >
> > I found that when I create a pipeline with a Kafka adapter,
> >
> >
> >
> > It will first consume data from Kafka, write a new randomly generated topic, and then consume data from this new topic. I think this will cause data redundancy. I want to know why to do this?
> >
> >
> >
> > Is there any relevant design document about the streampipes project？
> >
> >
> >
> > Thanks.
> >
> >
>
>
>
Re: Some questions about kafka adapter
Posted by Dominik Riemer <ri...@apache.org>.
Hi Zhangjun,
sounds great - I also think that supporting JVM and Flink in distributed mode would be sufficient and better focused.
I like your idea and putting the elements into a single Flink Job should work. We just recently did some improvements related to service discovery which should make it easier to integrate all pipeline elements into a single jar, which would be a precondition for this approach (see [1])
Also, I just roughly thought over the idea of improving the Flink wrapper so that all pipeline elements can be run in Flink, e.g., by wrapping the new "compact" definition of pipeline elements [2] into a ProcessFunction so that we wouldn't need to develop pipeline elements for either the flink or the JVM wrapper but support both at the same time.
What do you think?
Dominik
[1] https://github.com/apache/incubator-streampipes-extensions/blob/STREAMPIPES-319/streampipes-pipeline-elements-all-jvm/src/main/java/org/apache/streampipes/pe/jvm/AllPipelineElementsInit.java
[2] https://github.com/apache/incubator-streampipes-extensions/blob/STREAMPIPES-319/streampipes-sinks-brokers-jvm/src/main/java/org/apache/streampipes/sinks/brokers/jvm/websocket/WebsocketServerSink.java
On 2021/08/10 02:13:13, zhangjun <zh...@126.com> wrote:
>
> Hi , Dominik :
> By reading the source code, I guessed that using Kafka for transform may be to adapt to different engines in the same pipeline, such as flink, spark, and jvm. For this situation, I haven't thought of a better method for the time being.
>
> For flink job, my idea is to do a check, if all the elements in the pipeline are flink tasks, we can merge them into one flink job instead of the current N-1 flink jobs (N is the number of elements in the pipeline).In this way, flink can also be used to ensure the fault tolerance of job and the accuracy of data.
>
>
>
> 在 10.8.21 上午4:53，“Dominik Riemer”<dev-return-1977-zhangjunemail=126.com@streampipes.apache.org 代表 riemer@apache.org> 写入:
>
> Hi Zhangjun,
>
> welcome to the list! Sorry for the late reply, I somehow didn't get your email and just found it in the web interface.
>
> The reason for that is that all adapters are integrated into the same message broker, which is mainly Kafka (in theory, it should also be possible to change the messaging layer, but this is not fully automated right now). So while this is clearly suboptimal for Kafka, we didn't want to introduce any special treatment for the Kafka adapter. Another reason is that users can apply pre-processing functions at ingestion time (e.g., converting timestamps, filtering fields and so on) so that it is not guaranteed that the original Kafka topic can be reused. A workaround could be to create a custom source using the SDK which points to the original Kafka topic, in this case the topic wouldn't be replicated.
>
> Concerning Flink, we haven't focused so much on the Flink wrapper recently due to other priorities, so there is defintely room for improvement. Do you have any idea how to approach this?
>
> You can find some developer-oriented information in the wiki [1], although not a complete design document exists. So please feel free to ask any questions you might have.
>
> I hope I could answer some of your questions!
>
> Dominik
>
>
> [1] https://cwiki.apache.org/confluence/display/STREAMPIPES/Apache+StreamPipes+%28incubating%29+Developer+Wiki
>
>
>
>
> On 2021/08/04 09:46:13, zhangjun <zh...@126.com> wrote:
> >
> >
> > I think this architecture design has the following problems：
> >
> >
> >
> > 1. Use Kafka java api to consume Kafka data. If the data is large, it will cause a data backlog.
> >
> > 2. If the downstream is a flink job, using kafka java api to consume kafka data is not fault-tolerant. Once a problem occurs, data will be lost, and Exactly-once semantics cannot be guaranteed.
> >
> > 3. We consume data from the original Kafka and write to a new topic, which will cause data redundancy and duplication
> >
> >
> >
> >
> >
> >
> >
> > 发件人: <us...@streampipes.apache.org> 代表 zhangjun <zh...@126.com>
> > 答复: <us...@streampipes.apache.org>
> > 日期: 2021年8月4日 星期三 下午5:36
> > 收件人: <de...@streampipes.apache.org>, <us...@streampipes.apache.org>
> > 主题: Some questions about kafka adapter
> >
> >
> >
> >
> >
> > Hi,all:
> >
> > I found that when I create a pipeline with a Kafka adapter,
> >
> >
> >
> > It will first consume data from Kafka, write a new randomly generated topic, and then consume data from this new topic. I think this will cause data redundancy. I want to know why to do this?
> >
> >
> >
> > Is there any relevant design document about the streampipes project？
> >
> >
> >
> > Thanks.
> >
> >
>
>
>
Re: Some questions about kafka adapter
Posted by zhangjun <zh...@126.com>.
Hi , Dominik :
By reading the source code, I guessed that using Kafka for transform may be to adapt to different engines in the same pipeline, such as flink, spark, and jvm. For this situation, I haven't thought of a better method for the time being.
For flink job, my idea is to do a check, if all the elements in the pipeline are flink tasks, we can merge them into one flink job instead of the current N-1 flink jobs (N is the number of elements in the pipeline).In this way, flink can also be used to ensure the fault tolerance of job and the accuracy of data.
在 10.8.21 上午4:53，“Dominik Riemer”<dev-return-1977-zhangjunemail=126.com@streampipes.apache.org 代表 riemer@apache.org> 写入:
Hi Zhangjun,
welcome to the list! Sorry for the late reply, I somehow didn't get your email and just found it in the web interface.
The reason for that is that all adapters are integrated into the same message broker, which is mainly Kafka (in theory, it should also be possible to change the messaging layer, but this is not fully automated right now). So while this is clearly suboptimal for Kafka, we didn't want to introduce any special treatment for the Kafka adapter. Another reason is that users can apply pre-processing functions at ingestion time (e.g., converting timestamps, filtering fields and so on) so that it is not guaranteed that the original Kafka topic can be reused. A workaround could be to create a custom source using the SDK which points to the original Kafka topic, in this case the topic wouldn't be replicated.
Concerning Flink, we haven't focused so much on the Flink wrapper recently due to other priorities, so there is defintely room for improvement. Do you have any idea how to approach this?
You can find some developer-oriented information in the wiki [1], although not a complete design document exists. So please feel free to ask any questions you might have.
I hope I could answer some of your questions!
Dominik
[1] https://cwiki.apache.org/confluence/display/STREAMPIPES/Apache+StreamPipes+%28incubating%29+Developer+Wiki
On 2021/08/04 09:46:13, zhangjun <zh...@126.com> wrote:
>
>
> I think this architecture design has the following problems：
>
>
>
> 1. Use Kafka java api to consume Kafka data. If the data is large, it will cause a data backlog.
>
> 2. If the downstream is a flink job, using kafka java api to consume kafka data is not fault-tolerant. Once a problem occurs, data will be lost, and Exactly-once semantics cannot be guaranteed.
>
> 3. We consume data from the original Kafka and write to a new topic, which will cause data redundancy and duplication
>
>
>
>
>
>
>
> 发件人: <us...@streampipes.apache.org> 代表 zhangjun <zh...@126.com>
> 答复: <us...@streampipes.apache.org>
> 日期: 2021年8月4日 星期三 下午5:36
> 收件人: <de...@streampipes.apache.org>, <us...@streampipes.apache.org>
> 主题: Some questions about kafka adapter
>
>
>
>
>
> Hi,all:
>
> I found that when I create a pipeline with a Kafka adapter,
>
>
>
> It will first consume data from Kafka, write a new randomly generated topic, and then consume data from this new topic. I think this will cause data redundancy. I want to know why to do this?
>
>
>
> Is there any relevant design document about the streampipes project？
>
>
>
> Thanks.
>
>
Re: Some questions about kafka adapter
Posted by Dominik Riemer <ri...@apache.org>.
Hi Zhangjun,
welcome to the list! Sorry for the late reply, I somehow didn't get your email and just found it in the web interface.
The reason for that is that all adapters are integrated into the same message broker, which is mainly Kafka (in theory, it should also be possible to change the messaging layer, but this is not fully automated right now). So while this is clearly suboptimal for Kafka, we didn't want to introduce any special treatment for the Kafka adapter. Another reason is that users can apply pre-processing functions at ingestion time (e.g., converting timestamps, filtering fields and so on) so that it is not guaranteed that the original Kafka topic can be reused. A workaround could be to create a custom source using the SDK which points to the original Kafka topic, in this case the topic wouldn't be replicated.
Concerning Flink, we haven't focused so much on the Flink wrapper recently due to other priorities, so there is defintely room for improvement. Do you have any idea how to approach this?
You can find some developer-oriented information in the wiki [1], although not a complete design document exists. So please feel free to ask any questions you might have.
I hope I could answer some of your questions!
Dominik
[1] https://cwiki.apache.org/confluence/display/STREAMPIPES/Apache+StreamPipes+%28incubating%29+Developer+Wiki
On 2021/08/04 09:46:13, zhangjun <zh...@126.com> wrote:
>
>
> I think this architecture design has the following problems：
>
>
>
> 1. Use Kafka java api to consume Kafka data. If the data is large, it will cause a data backlog.
>
> 2. If the downstream is a flink job, using kafka java api to consume kafka data is not fault-tolerant. Once a problem occurs, data will be lost, and Exactly-once semantics cannot be guaranteed.
>
> 3. We consume data from the original Kafka and write to a new topic, which will cause data redundancy and duplication
>
>
>
>
>
>
>
> 发件人: <us...@streampipes.apache.org> 代表 zhangjun <zh...@126.com>
> 答复: <us...@streampipes.apache.org>
> 日期: 2021年8月4日 星期三 下午5:36
> 收件人: <de...@streampipes.apache.org>, <us...@streampipes.apache.org>
> 主题: Some questions about kafka adapter
>
>
>
>
>
> Hi,all:
>
> I found that when I create a pipeline with a Kafka adapter,
>
>
>
> It will first consume data from Kafka, write a new randomly generated topic, and then consume data from this new topic. I think this will cause data redundancy. I want to know why to do this?
>
>
>
> Is there any relevant design document about the streampipes project？
>
>
>
> Thanks.
>
>
Re: Some questions about kafka adapter
Posted by zhangjun <zh...@126.com>.
I think this architecture design has the following problems：
1. Use Kafka java api to consume Kafka data. If the data is large, it will cause a data backlog.
2. If the downstream is a flink job, using kafka java api to consume kafka data is not fault-tolerant. Once a problem occurs, data will be lost, and Exactly-once semantics cannot be guaranteed.
3. We consume data from the original Kafka and write to a new topic, which will cause data redundancy and duplication
发件人: <us...@streampipes.apache.org> 代表 zhangjun <zh...@126.com>
答复: <us...@streampipes.apache.org>
日期: 2021年8月4日 星期三 下午5:36
收件人: <de...@streampipes.apache.org>, <us...@streampipes.apache.org>
主题: Some questions about kafka adapter
Hi,all:
I found that when I create a pipeline with a Kafka adapter,
It will first consume data from Kafka, write a new randomly generated topic, and then consume data from this new topic. I think this will cause data redundancy. I want to know why to do this?
Is there any relevant design document about the streampipes project？
Thanks.
Re: Some questions about kafka adapter
Posted by zhangjun <zh...@126.com>.
I think this architecture design has the following problems：
1. Use Kafka java api to consume Kafka data. If the data is large, it will cause a data backlog.
2. If the downstream is a flink job, using kafka java api to consume kafka data is not fault-tolerant. Once a problem occurs, data will be lost, and Exactly-once semantics cannot be guaranteed.
3. We consume data from the original Kafka and write to a new topic, which will cause data redundancy and duplication
发件人: <us...@streampipes.apache.org> 代表 zhangjun <zh...@126.com>
答复: <us...@streampipes.apache.org>
日期: 2021年8月4日 星期三 下午5:36
收件人: <de...@streampipes.apache.org>, <us...@streampipes.apache.org>
主题: Some questions about kafka adapter
Hi,all:
I found that when I create a pipeline with a Kafka adapter,
It will first consume data from Kafka, write a new randomly generated topic, and then consume data from this new topic. I think this will cause data redundancy. I want to know why to do this?
Is there any relevant design document about the streampipes project？
Thanks.