You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "melin (via GitHub)" <gi...@apache.org> on 2024/09/18 06:51:08 UTC
[I] [SUPPORT] flink cdc 3.0 pipeline hudi sink [hudi]
melin opened a new issue, #11953:
URL: https://github.com/apache/hudi/issues/11953
flink cdc 3.0 pipeline hudi sink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink cdc 3.x pipeline hudi sink [hudi]
Posted by "melin (via GitHub)" <gi...@apache.org>.
melin commented on issue #11953:
URL: https://github.com/apache/hudi/issues/11953#issuecomment-2359795776
> It looks like a bunch of flink cdc specific APIs. We will put this on roadmap if we have energy.
yes，cdc data writing to a data lake is a very important scenario, multi-table or database synchronization without a good open source tool. flink cdc pipeline can easily write cdc data to hudi and support schema evolution (hudi version 0.X flinkcatalog does not support schema evolution).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink cdc 3.x pipeline hudi sink [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #11953:
URL: https://github.com/apache/hudi/issues/11953#issuecomment-2567252552
@melin Thanks for the feedback, hope we can cooperate in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink cdc 3.x pipeline hudi sink [hudi]
Posted by "melin (via GitHub)" <gi...@apache.org>.
melin commented on issue #11953:
URL: https://github.com/apache/hudi/issues/11953#issuecomment-2359881439
> > hudi version 0.X flinkcatalog does not support schema evolution
>
> For schema evolution, do you mean the `alter table` cmd?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink cdc 3.x pipeline hudi sink [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #11953:
URL: https://github.com/apache/hudi/issues/11953#issuecomment-2572675789
yeah, we will try to support this in 1.1 release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink cdc 3.x pipeline hudi sink [hudi]
Posted by "lschampion (via GitHub)" <gi...@apache.org>.
lschampion commented on issue #11953:
URL: https://github.com/apache/hudi/issues/11953#issuecomment-2572731194
flink-cdc use event driving schema evlution solution, it integrates this feature in a datastream without restarting flink job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink cdc 3.x pipeline hudi sink [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #11953:
URL: https://github.com/apache/hudi/issues/11953#issuecomment-2357981960
Hi, does Flink cdc 3.0 requires the sink to support specific APIs for SQL or DataStream?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink cdc 3.x pipeline hudi sink [hudi]
Posted by "melin (via GitHub)" <gi...@apache.org>.
melin commented on issue #11953:
URL: https://github.com/apache/hudi/issues/11953#issuecomment-2565354431
国内有很多使用hudi，是因为实时入湖需求。现在piamon 支持cdc 多表整库入湖，flink cdc pipelne 也支持paimon 多表或者整库入湖。hudi 对这块能力比较弱，被迫切换到paimon。希望hudi 增强这块能力，iceberg 和 delta 天然不适合这样场景。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink cdc 3.x pipeline hudi sink [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #11953:
URL: https://github.com/apache/hudi/issues/11953#issuecomment-2359855312
> hudi version 0.X flinkcatalog does not support schema evolution
For schema evolution, do you mean the `alter table` cmd?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink cdc 3.x pipeline hudi sink [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #11953:
URL: https://github.com/apache/hudi/issues/11953#issuecomment-2359731341
It looks like a bunch of flink cdc specific APIs. We will put this on roadmap if we have energy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink cdc 3.x pipeline hudi sink [hudi]
Posted by "melin (via GitHub)" <gi...@apache.org>.
melin commented on issue #11953:
URL: https://github.com/apache/hudi/issues/11953#issuecomment-2358044287
> Hi, does Flink cdc 3.0 requires the sink to support specific APIs for SQL or DataStream?
api: https://nightlies.apache.org/flink/flink-cdc-docs-release-3.2/docs/developer-guide/understand-flink-cdc-api/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink cdc 3.x pipeline hudi sink [hudi]
Posted by "melin (via GitHub)" <gi...@apache.org>.
melin commented on issue #11953:
URL: https://github.com/apache/hudi/issues/11953#issuecomment-2511663010
@danny0405 Plans to develop this feature?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] flink cdc 3.x pipeline hudi sink [hudi]
Posted by "lschampion (via GitHub)" <gi...@apache.org>.
lschampion commented on issue #11953:
URL: https://github.com/apache/hudi/issues/11953#issuecomment-2572659317
yeah, i need this feathure too, and am afriad that hudi may not support flink cdc schema evolution, because hudi must specify table schema when writing in flink api. schema evo of flink cdc pipeline is a more flexiable plan in achiving.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org