You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "selvarajperiyasamy (via GitHub)" <gi...@apache.org> on 2023/03/14 19:45:10 UTC
[GitHub] [hudi] selvarajperiyasamy opened a new issue, #8186: [SUPPORT]
selvarajperiyasamy opened a new issue, #8186:
URL: https://github.com/apache/hudi/issues/8186
Our organization is migrating from Hadoop 2.6 to 3.1 and spark 2.3 to 3.1. Our existing data sets (100s of TBs of data) are written using Hudi 0.5.0. We would like to move to Hudi 0.13.0.
The latest version of Hudi has come way since 0.5.0, we are not sure about how to use 0.13.0 directly.
Could someone provide the steps for upgrading from 0.5.0 to 0.13.0?
Thanks,
Selva
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] selvarajperiyasamy commented on issue #8186: upgrade from 0.5.0 to 0.13.0
Posted by "selvarajperiyasamy (via GitHub)" <gi...@apache.org>.
selvarajperiyasamy commented on issue #8186:
URL: https://github.com/apache/hudi/issues/8186#issuecomment-1478588840
Folks, could someone shed some light here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] selvarajperiyasamy commented on issue #8186: upgrade from 0.5.0 to 0.13.0
Posted by "selvarajperiyasamy (via GitHub)" <gi...@apache.org>.
selvarajperiyasamy commented on issue #8186:
URL: https://github.com/apache/hudi/issues/8186#issuecomment-1493556611
@yihua could you please clarify ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on issue #8186: upgrade from 0.5.0 to 0.13.0
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on issue #8186:
URL: https://github.com/apache/hudi/issues/8186#issuecomment-1487789003
Hi @selvarajperiyasamy Thanks for your interest in 0.13.0 release. You may read the release notes, especially the migration guide, of each major release, particularly the releases that change the table version:
- 0.6.0: https://hudi.apache.org/releases/older-releases#migration-guide-for-this-release-3
- 0.9.0: https://hudi.apache.org/releases/older-releases#migration-guide-for-this-release
- 0.10.0: https://hudi.apache.org/releases/older-releases#migration-guide-3
- 0.11.0: https://hudi.apache.org/releases/older-releases#migration-guide-2
- 0.12.0: https://hudi.apache.org/releases/older-releases#migration-guide-1
- 0.13.0: https://hudi.apache.org/releases/release-0.13.0#migration-guide-overview
You should be aware that there are known regressions in 0.13.0 (https://hudi.apache.org/releases/release-0.13.0#known-regressions) so if you'd like to upgrade to 0.13, it would be good to wait for 0.13.1 with more reliability.
You may also consider upgrading to 0.10.1 release first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8186: upgrade from 0.5.0 to 0.13.0
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8186:
URL: https://github.com/apache/hudi/issues/8186#issuecomment-1569888633
@selvarajperiyasamy We should ideally use 0.12.3 or 0.13.1 which are the more stable versions compared to 0.11.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nfarah86 commented on issue #8186: upgrade from 0.5.0 to 0.13.0
Posted by "nfarah86 (via GitHub)" <gi...@apache.org>.
nfarah86 commented on issue #8186:
URL: https://github.com/apache/hudi/issues/8186#issuecomment-1534981775
following up from office hours via @codope :
It's better to update 0.5.0 to 0.10.1; this is when the metadata table is enabled by default.
Once you have the pipeline running for a few days, you can update to 0.12.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] selvarajperiyasamy commented on issue #8186: upgrade from 0.5.0 to 0.13.0
Posted by "selvarajperiyasamy (via GitHub)" <gi...@apache.org>.
selvarajperiyasamy commented on issue #8186:
URL: https://github.com/apache/hudi/issues/8186#issuecomment-1535040193
@codope We faced https://issues.apache.org/jira/browse/HUDI-2899 while reading using 0.10.1 .
So we went ahead with 0.11.1 . Is it better to use 0.11.1 ? I have disabled metadata table flag in our code .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] selvarajperiyasamy commented on issue #8186: upgrade from 0.5.0 to 0.13.0
Posted by "selvarajperiyasamy (via GitHub)" <gi...@apache.org>.
selvarajperiyasamy commented on issue #8186:
URL: https://github.com/apache/hudi/issues/8186#issuecomment-1499210646
@vinothchandar @nsivabalan @bvaradar could you guys help me to understand? Sorry. i tagged you guys because i know your names from my last year conversations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] selvarajperiyasamy commented on issue #8186: upgrade from 0.5.0 to 0.13.0
Posted by "selvarajperiyasamy (via GitHub)" <gi...@apache.org>.
selvarajperiyasamy commented on issue #8186:
URL: https://github.com/apache/hudi/issues/8186#issuecomment-1489078645
Thanks @yihua . I can look into previous versions, which are < 0.13.0. May i know what is the reason you are specifically asking to go with 0.10.1, instead of 0.12.2 or 0.11.1?
One more question is ,
We are actively using COW table using spark data source writer in 0.5.0 and not using delta streaming . Do i need to upgrade one table at a time like upgrade to 0.6.0 then 0.9.0 then 0.10.1?
OR i can directly change my reader to use 0.10.1 and then writer to start writing using 0.10.1 library? If i do this, do i need to take care of anything on the existing datasets?
Thanks,
Selva
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org