You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by sri hari kali charan Tummala <ka...@gmail.com> on 2022/06/07 18:18:36 UTC
Flink config driven tool ?
Hi Flink Community,
can someone point me to a good config-driven flink data movement tool
Github repos? Imagine I build my ETL dag connecting source -->
transformations --> target just using a config file.
below are a few spark examples:-
https://github.com/mvrpl/big-shipper
https://github.com/BitwiseInc/Hydrograph
Thanks & Regards
Sri Tummala
Re: Flink config driven tool ?
Posted by sri hari kali charan Tummala <ka...@gmail.com>.
this looks interesting Nasa Akka project https://github.com/NASARace/race
On Mon, Jun 20, 2022 at 7:06 PM sri hari kali charan Tummala <
kali.tummala@gmail.com> wrote:
> found one more flink tool
> https://www.splunk.com/en_us/products/stream-processing.html
>
> On Wed, Jun 15, 2022 at 1:18 PM Jing Ge <ji...@ververica.com> wrote:
>
>> Hi,
>>
>> Just like Shengkai mentioned. I would strongly suggest trying SQL for ETL
>> dag. If you find anything that SQL does not work for you, please share your
>> requirements with us. We might check if it makes sense to build new
>> features in Flink to support them.
>>
>> Best regards,
>> Jing
>>
>>
>> On Wed, Jun 15, 2022 at 11:22 AM Rakshit Ramesh <
>> rakshit.ramesh@datakaveri.org> wrote:
>>
>>> I'm working on such a thing.
>>> It's in early stages and needs a lot more work.
>>> I'm open to collaborating.
>>> https://github.com/datakaveri/iudx-adaptor-framework
>>>
>>> On Tue, 7 Jun 2022 at 23:49, sri hari kali charan Tummala <
>>> kali.tummala@gmail.com> wrote:
>>>
>>>> Hi Flink Community,
>>>>
>>>> can someone point me to a good config-driven flink data movement tool
>>>> Github repos? Imagine I build my ETL dag connecting source -->
>>>> transformations --> target just using a config file.
>>>>
>>>> below are a few spark examples:-
>>>> https://github.com/mvrpl/big-shipper
>>>> https://github.com/BitwiseInc/Hydrograph
>>>>
>>>> Thanks & Regards
>>>> Sri Tummala
>>>>
>>>>
>
> --
> Thanks & Regards
> Sri Tummala
>
>
--
Thanks & Regards
Sri Tummala
Re: Flink config driven tool ?
Posted by sri hari kali charan Tummala <ka...@gmail.com>.
found one more flink tool
https://www.splunk.com/en_us/products/stream-processing.html
On Wed, Jun 15, 2022 at 1:18 PM Jing Ge <ji...@ververica.com> wrote:
> Hi,
>
> Just like Shengkai mentioned. I would strongly suggest trying SQL for ETL
> dag. If you find anything that SQL does not work for you, please share your
> requirements with us. We might check if it makes sense to build new
> features in Flink to support them.
>
> Best regards,
> Jing
>
>
> On Wed, Jun 15, 2022 at 11:22 AM Rakshit Ramesh <
> rakshit.ramesh@datakaveri.org> wrote:
>
>> I'm working on such a thing.
>> It's in early stages and needs a lot more work.
>> I'm open to collaborating.
>> https://github.com/datakaveri/iudx-adaptor-framework
>>
>> On Tue, 7 Jun 2022 at 23:49, sri hari kali charan Tummala <
>> kali.tummala@gmail.com> wrote:
>>
>>> Hi Flink Community,
>>>
>>> can someone point me to a good config-driven flink data movement tool
>>> Github repos? Imagine I build my ETL dag connecting source -->
>>> transformations --> target just using a config file.
>>>
>>> below are a few spark examples:-
>>> https://github.com/mvrpl/big-shipper
>>> https://github.com/BitwiseInc/Hydrograph
>>>
>>> Thanks & Regards
>>> Sri Tummala
>>>
>>>
--
Thanks & Regards
Sri Tummala
Re: Flink config driven tool ?
Posted by Jing Ge <ji...@ververica.com>.
Hi,
Just like Shengkai mentioned. I would strongly suggest trying SQL for ETL
dag. If you find anything that SQL does not work for you, please share your
requirements with us. We might check if it makes sense to build new
features in Flink to support them.
Best regards,
Jing
On Wed, Jun 15, 2022 at 11:22 AM Rakshit Ramesh <
rakshit.ramesh@datakaveri.org> wrote:
> I'm working on such a thing.
> It's in early stages and needs a lot more work.
> I'm open to collaborating.
> https://github.com/datakaveri/iudx-adaptor-framework
>
> On Tue, 7 Jun 2022 at 23:49, sri hari kali charan Tummala <
> kali.tummala@gmail.com> wrote:
>
>> Hi Flink Community,
>>
>> can someone point me to a good config-driven flink data movement tool
>> Github repos? Imagine I build my ETL dag connecting source -->
>> transformations --> target just using a config file.
>>
>> below are a few spark examples:-
>> https://github.com/mvrpl/big-shipper
>> https://github.com/BitwiseInc/Hydrograph
>>
>> Thanks & Regards
>> Sri Tummala
>>
>>
Re: Flink config driven tool ?
Posted by Sucheth S <su...@gmail.com>.
You are just spamming the inbox by sending these emails. You can just
ignore sending those emails if it isn't adding any value.
Regards,
Sucheth Shivakumar
website : https://sucheths.com
mobile : +1(650)-576-8050
San Mateo, United States
On Wed, Jun 15, 2022 at 1:41 PM sri hari kali charan Tummala <
kali.tummala@gmail.com> wrote:
> I would have helped you if it was written in Scala instead of Java.
>
> On Wed, Jun 15, 2022 at 2:22 AM Rakshit Ramesh <
> rakshit.ramesh@datakaveri.org> wrote:
>
>> I'm working on such a thing.
>> It's in early stages and needs a lot more work.
>> I'm open to collaborating.
>> https://github.com/datakaveri/iudx-adaptor-framework
>>
>> On Tue, 7 Jun 2022 at 23:49, sri hari kali charan Tummala <
>> kali.tummala@gmail.com> wrote:
>>
>>> Hi Flink Community,
>>>
>>> can someone point me to a good config-driven flink data movement tool
>>> Github repos? Imagine I build my ETL dag connecting source -->
>>> transformations --> target just using a config file.
>>>
>>> below are a few spark examples:-
>>> https://github.com/mvrpl/big-shipper
>>> https://github.com/BitwiseInc/Hydrograph
>>>
>>> Thanks & Regards
>>> Sri Tummala
>>>
>>>
>
> --
> Thanks & Regards
> Sri Tummala
>
>
Re: Flink config driven tool ?
Posted by sri hari kali charan Tummala <ka...@gmail.com>.
I would have helped you if it was written in Scala instead of Java.
On Wed, Jun 15, 2022 at 2:22 AM Rakshit Ramesh <
rakshit.ramesh@datakaveri.org> wrote:
> I'm working on such a thing.
> It's in early stages and needs a lot more work.
> I'm open to collaborating.
> https://github.com/datakaveri/iudx-adaptor-framework
>
> On Tue, 7 Jun 2022 at 23:49, sri hari kali charan Tummala <
> kali.tummala@gmail.com> wrote:
>
>> Hi Flink Community,
>>
>> can someone point me to a good config-driven flink data movement tool
>> Github repos? Imagine I build my ETL dag connecting source -->
>> transformations --> target just using a config file.
>>
>> below are a few spark examples:-
>> https://github.com/mvrpl/big-shipper
>> https://github.com/BitwiseInc/Hydrograph
>>
>> Thanks & Regards
>> Sri Tummala
>>
>>
--
Thanks & Regards
Sri Tummala
Re: Flink config driven tool ?
Posted by Rakshit Ramesh <ra...@datakaveri.org>.
I'm working on such a thing.
It's in early stages and needs a lot more work.
I'm open to collaborating.
https://github.com/datakaveri/iudx-adaptor-framework
On Tue, 7 Jun 2022 at 23:49, sri hari kali charan Tummala <
kali.tummala@gmail.com> wrote:
> Hi Flink Community,
>
> can someone point me to a good config-driven flink data movement tool
> Github repos? Imagine I build my ETL dag connecting source -->
> transformations --> target just using a config file.
>
> below are a few spark examples:-
> https://github.com/mvrpl/big-shipper
> https://github.com/BitwiseInc/Hydrograph
>
> Thanks & Regards
> Sri Tummala
>
>
Re: Flink config driven tool ?
Posted by Shengkai Fang <fs...@gmail.com>.
Hi.
I am not sure whether Flink SQL satisfies your requirement or not. You can
just write the SQL in the file and use the SQL Client to submit it to your
cluster. We have a quick start in the Flink CDC and you can make a try[1].
Best,
Shengkai
[1]
https://ververica.github.io/flink-cdc-connectors/master/content/quickstart/mysql-postgres-tutorial.html
sri hari kali charan Tummala <ka...@gmail.com> 于2022年6月8日周三 04:49写道：
> Thanks, I'll check it out.
>
> On Tue, Jun 7, 2022, 1:31 PM Austin Cawley-Edwards <
> austin.cawley@gmail.com> wrote:
>
>> They support Flink as well. Looks like they even support the new Flink
>> k8s operator.[1]
>>
>> Austin
>>
>> [1]:
>> https://seatunnel.apache.org/docs/2.1.1/start/kubernetes#deploying-the-operator
>>
>> On Tue, Jun 7, 2022 at 3:11 PM sri hari kali charan Tummala <
>> kali.tummala@gmail.com> wrote:
>>
>>> thanks but looks like a spark tool is there something similar in flink?
>>>
>>> Thanks
>>> Sri
>>>
>>> On Tue, Jun 7, 2022 at 12:07 PM Austin Cawley-Edwards <
>>> austin.cawley@gmail.com> wrote:
>>>
>>>> Hey there,
>>>>
>>>> No idea if it's any good, but just saw Apache SeaTunnel[1] today which
>>>> seems to fit your requirements.
>>>>
>>>> Best,
>>>> Austin
>>>>
>>>> [1]: https://seatunnel.apache.org/
>>>>
>>>> On Tue, Jun 7, 2022 at 2:19 PM sri hari kali charan Tummala <
>>>> kali.tummala@gmail.com> wrote:
>>>>
>>>>> Hi Flink Community,
>>>>>
>>>>> can someone point me to a good config-driven flink data movement tool
>>>>> Github repos? Imagine I build my ETL dag connecting source -->
>>>>> transformations --> target just using a config file.
>>>>>
>>>>> below are a few spark examples:-
>>>>> https://github.com/mvrpl/big-shipper
>>>>> https://github.com/BitwiseInc/Hydrograph
>>>>>
>>>>> Thanks & Regards
>>>>> Sri Tummala
>>>>>
>>>>>
>>>
>>> --
>>> Thanks & Regards
>>> Sri Tummala
>>>
>>>
Re: Flink config driven tool ?
Posted by sri hari kali charan Tummala <ka...@gmail.com>.
Thanks, I'll check it out.
On Tue, Jun 7, 2022, 1:31 PM Austin Cawley-Edwards <au...@gmail.com>
wrote:
> They support Flink as well. Looks like they even support the new Flink k8s
> operator.[1]
>
> Austin
>
> [1]:
> https://seatunnel.apache.org/docs/2.1.1/start/kubernetes#deploying-the-operator
>
> On Tue, Jun 7, 2022 at 3:11 PM sri hari kali charan Tummala <
> kali.tummala@gmail.com> wrote:
>
>> thanks but looks like a spark tool is there something similar in flink?
>>
>> Thanks
>> Sri
>>
>> On Tue, Jun 7, 2022 at 12:07 PM Austin Cawley-Edwards <
>> austin.cawley@gmail.com> wrote:
>>
>>> Hey there,
>>>
>>> No idea if it's any good, but just saw Apache SeaTunnel[1] today which
>>> seems to fit your requirements.
>>>
>>> Best,
>>> Austin
>>>
>>> [1]: https://seatunnel.apache.org/
>>>
>>> On Tue, Jun 7, 2022 at 2:19 PM sri hari kali charan Tummala <
>>> kali.tummala@gmail.com> wrote:
>>>
>>>> Hi Flink Community,
>>>>
>>>> can someone point me to a good config-driven flink data movement tool
>>>> Github repos? Imagine I build my ETL dag connecting source -->
>>>> transformations --> target just using a config file.
>>>>
>>>> below are a few spark examples:-
>>>> https://github.com/mvrpl/big-shipper
>>>> https://github.com/BitwiseInc/Hydrograph
>>>>
>>>> Thanks & Regards
>>>> Sri Tummala
>>>>
>>>>
>>
>> --
>> Thanks & Regards
>> Sri Tummala
>>
>>
Re: Flink config driven tool ?
Posted by Austin Cawley-Edwards <au...@gmail.com>.
They support Flink as well. Looks like they even support the new Flink k8s
operator.[1]
Austin
[1]:
https://seatunnel.apache.org/docs/2.1.1/start/kubernetes#deploying-the-operator
On Tue, Jun 7, 2022 at 3:11 PM sri hari kali charan Tummala <
kali.tummala@gmail.com> wrote:
> thanks but looks like a spark tool is there something similar in flink?
>
> Thanks
> Sri
>
> On Tue, Jun 7, 2022 at 12:07 PM Austin Cawley-Edwards <
> austin.cawley@gmail.com> wrote:
>
>> Hey there,
>>
>> No idea if it's any good, but just saw Apache SeaTunnel[1] today which
>> seems to fit your requirements.
>>
>> Best,
>> Austin
>>
>> [1]: https://seatunnel.apache.org/
>>
>> On Tue, Jun 7, 2022 at 2:19 PM sri hari kali charan Tummala <
>> kali.tummala@gmail.com> wrote:
>>
>>> Hi Flink Community,
>>>
>>> can someone point me to a good config-driven flink data movement tool
>>> Github repos? Imagine I build my ETL dag connecting source -->
>>> transformations --> target just using a config file.
>>>
>>> below are a few spark examples:-
>>> https://github.com/mvrpl/big-shipper
>>> https://github.com/BitwiseInc/Hydrograph
>>>
>>> Thanks & Regards
>>> Sri Tummala
>>>
>>>
>
> --
> Thanks & Regards
> Sri Tummala
>
>
Re: Flink config driven tool ?
Posted by sri hari kali charan Tummala <ka...@gmail.com>.
thanks but looks like a spark tool is there something similar in flink?
Thanks
Sri
On Tue, Jun 7, 2022 at 12:07 PM Austin Cawley-Edwards <
austin.cawley@gmail.com> wrote:
> Hey there,
>
> No idea if it's any good, but just saw Apache SeaTunnel[1] today which
> seems to fit your requirements.
>
> Best,
> Austin
>
> [1]: https://seatunnel.apache.org/
>
> On Tue, Jun 7, 2022 at 2:19 PM sri hari kali charan Tummala <
> kali.tummala@gmail.com> wrote:
>
>> Hi Flink Community,
>>
>> can someone point me to a good config-driven flink data movement tool
>> Github repos? Imagine I build my ETL dag connecting source -->
>> transformations --> target just using a config file.
>>
>> below are a few spark examples:-
>> https://github.com/mvrpl/big-shipper
>> https://github.com/BitwiseInc/Hydrograph
>>
>> Thanks & Regards
>> Sri Tummala
>>
>>
--
Thanks & Regards
Sri Tummala
Re: Flink config driven tool ?
Posted by Austin Cawley-Edwards <au...@gmail.com>.
Hey there,
No idea if it's any good, but just saw Apache SeaTunnel[1] today which
seems to fit your requirements.
Best,
Austin
[1]: https://seatunnel.apache.org/
On Tue, Jun 7, 2022 at 2:19 PM sri hari kali charan Tummala <
kali.tummala@gmail.com> wrote:
> Hi Flink Community,
>
> can someone point me to a good config-driven flink data movement tool
> Github repos? Imagine I build my ETL dag connecting source -->
> transformations --> target just using a config file.
>
> below are a few spark examples:-
> https://github.com/mvrpl/big-shipper
> https://github.com/BitwiseInc/Hydrograph
>
> Thanks & Regards
> Sri Tummala
>
>