You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Junyao Huang <Ju...@microsoft.com.INVALID> on 2022/12/20 05:11:08 UTC
[DISCUSS] Add Flink Web3 Connector
Web3 is very hot. But you could search GitHub open source blockchain explorer, the most stars project is blockscout, https://github.com/blockscout/blockscout<https://github.com/blockscout/blockscout,> which use Elixir as a parallel engine to sync block from blockchain node into a file(CSV format). I think Flink is the best solution of ingestion. Reason:
(1)blockchain needs to match different chain, including Ethereum, Bitcoin, Solana, etc. through JSON RPC.
(2)Like EtherScan, the blockchain needs to fetch the latest block into storage for the index to search.
(3)Also as a supplement to (2), we need a connector to fully sync all block from Blockchain Node. I think Flink Stream/Batch alignment feature is suit for this scenarios.
(4)According to FLIP-27, we could use block number as SourceSplit to read. It is very natural.
(5)Flink Community could use web3 topic to get PR effects on web3 cycle.
[1]https://issues.apache.org/jira/projects/FLINK/issues/FLINK-30445?filter=allissues
[2]https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
Re: [EXTERNAL] Re: [DISCUSS] Add Flink Web3 Connector
Posted by Junyao Huang <Ju...@microsoft.com.INVALID>.
Hi Martin,
Thanks for the replies.
But actually, Web3 represents all kinds of blockchain node.
Like https://github.com/ethereum/go-ethereum, which will release Ethereum 2.0 this year.
[cid:e1dcf0ae-2dde-49f2-9dbb-8d23f428a27a]
Will cover sharding & indexing like ElasticSearch & Kafka.
Bitcoin, Ethereum, Solana solutions are called Web3 in this era.
So I think it is still valuable.
Regards,
Junyao
________________________________
From: Martijn Visser <ma...@apache.org>
Sent: Tuesday, December 20, 2022 16:31
To: dev@flink.apache.org <de...@flink.apache.org>; Junyao Huang <Ju...@microsoft.com>
Subject: [EXTERNAL] Re: [DISCUSS] Add Flink Web3 Connector
[You don't often get email from martijnvisser@apache.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Hi Junyao,
I don't see value in this for the Apache Flink community. You're stating
this as a 'Web3 connector' but are only referring to one specific project.
If that project needs a Flink connector, I think it should be built and
maintained by that project, not by the Flink community.
Best regards,
Martijn
On Tue, Dec 20, 2022 at 6:11 AM Junyao Huang
<Ju...@microsoft.com.invalid> wrote:
> Web3 is very hot. But you could search GitHub open source blockchain
> explorer, the most stars project is blockscout,
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fblockscout%2Fblockscout&data=05%7C01%7CJunyao.Huang%40microsoft.com%7Cfe740f3d1eb04aa0a48d08dae264a1e6%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C638071219129406668%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ojY7UrNTY3nxxP7ZNcvXwZpsTMvUExGU%2BIPKm8uRU6w%3D&reserved=0<
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fblockscout%2Fblockscout&data=05%7C01%7CJunyao.Huang%40microsoft.com%7Cfe740f3d1eb04aa0a48d08dae264a1e6%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C638071219129406668%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ojY7UrNTY3nxxP7ZNcvXwZpsTMvUExGU%2BIPKm8uRU6w%3D&reserved=0,> which use Elixir as a parallel
> engine to sync block from blockchain node into a file(CSV format). I think
> Flink is the best solution of ingestion. Reason:
>
> (1)blockchain needs to match different chain, including Ethereum, Bitcoin,
> Solana, etc. through JSON RPC.
>
> (2)Like EtherScan, the blockchain needs to fetch the latest block into
> storage for the index to search.
>
> (3)Also as a supplement to (2), we need a connector to fully sync all
> block from Blockchain Node. I think Flink Stream/Batch alignment feature is
> suit for this scenarios.
>
> (4)According to FLIP-27, we could use block number as SourceSplit to read.
> It is very natural.
>
> (5)Flink Community could use web3 topic to get PR effects on web3 cycle.
>
>
> [1]
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fprojects%2FFLINK%2Fissues%2FFLINK-30445%3Ffilter%3Dallissues&data=05%7C01%7CJunyao.Huang%40microsoft.com%7Cfe740f3d1eb04aa0a48d08dae264a1e6%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C638071219129406668%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=LDTKqvQGlYfv%2B60OZT0mFM%2Bf2Oc1Fv8ZufVqcl4HLi4%3D&reserved=0
>
> [2]
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FFLINK%2FFLIP-27%253A%2BRefactor%2BSource%2BInterface&data=05%7C01%7CJunyao.Huang%40microsoft.com%7Cfe740f3d1eb04aa0a48d08dae264a1e6%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C638071219129406668%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=GX%2FfCClAq1lhaPtz0W4SxdS%2BH86%2F%2Bea7Xfg%2FSyJCIJM%3D&reserved=0
>
>
Re: [DISCUSS] Add Flink Web3 Connector
Posted by Martijn Visser <ma...@apache.org>.
Hi Junyao,
I don't see value in this for the Apache Flink community. You're stating
this as a 'Web3 connector' but are only referring to one specific project.
If that project needs a Flink connector, I think it should be built and
maintained by that project, not by the Flink community.
Best regards,
Martijn
On Tue, Dec 20, 2022 at 6:11 AM Junyao Huang
<Ju...@microsoft.com.invalid> wrote:
> Web3 is very hot. But you could search GitHub open source blockchain
> explorer, the most stars project is blockscout,
> https://github.com/blockscout/blockscout<
> https://github.com/blockscout/blockscout,> which use Elixir as a parallel
> engine to sync block from blockchain node into a file(CSV format). I think
> Flink is the best solution of ingestion. Reason:
>
> (1)blockchain needs to match different chain, including Ethereum, Bitcoin,
> Solana, etc. through JSON RPC.
>
> (2)Like EtherScan, the blockchain needs to fetch the latest block into
> storage for the index to search.
>
> (3)Also as a supplement to (2), we need a connector to fully sync all
> block from Blockchain Node. I think Flink Stream/Batch alignment feature is
> suit for this scenarios.
>
> (4)According to FLIP-27, we could use block number as SourceSplit to read.
> It is very natural.
>
> (5)Flink Community could use web3 topic to get PR effects on web3 cycle.
>
>
> [1]
> https://issues.apache.org/jira/projects/FLINK/issues/FLINK-30445?filter=allissues
>
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface
>
>