You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2023/01/12 16:58:01 UTC
[GitHub] [hudi] soma1712 opened a new issue, #7659: [SUPPORT] - Incremental Records are not applied to the Hudi Table
soma1712 opened a new issue, #7659:
URL: https://github.com/apache/hudi/issues/7659
Hi,
We have a situation to load some tables incrementally from Oracle to S3 and build HUDI tables. During this process, we are missing records when the spark-submits are executed to apply the deltas on the existing data. Below are the steps we followed in sequence to load the tables.
1. We extracted all the data from Oracle table using AWS DMS and created parquet files in S3 bucket as a onetime activity for initial load.
2. Executed Spark-Submits to convert the parquet files into HUDI table on S3 and synced the table to Glue Catalog.
3. Query the data in HUDI table and validate it against golden source Oracle to make sure it looks good.
4. Enabled AWS DMS CDC tasks on the same table to read data incrementally from redo logs and save them as parquet in S3 bucket.
5. Executed Spark-Submits to apply these delta records on to the initially created HUDI table.
6. Validated to check if the deltas are applied to the table.
During data validation, we figured out that the records are available in S3 CDC bucket but the HUDI table is not updated with the incremental data.
Here are the configs and spark submits we used
Hudi version - 0.7.0
spark version - 2.4.0
[SparkSubmits.txt](https://github.com/apache/hudi/files/10404025/SparkSubmits.txt)
[actvy_log_full.txt](https://github.com/apache/hudi/files/10404030/actvy_log_full.txt)
[actvy_log_cdc.txt](https://github.com/apache/hudi/files/10404031/actvy_log_cdc.txt)
Please let me know if more details are needed.
Regards,
Soma
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] soumilshah1995 commented on issue #7659: [SUPPORT] - Incremental Records are not applied to the Hudi Table
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #7659:
URL: https://github.com/apache/hudi/issues/7659#issuecomment-1399251368
looks like you need to update to latest version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #7659: [SUPPORT] - Incremental Records are not applied to the Hudi Table
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #7659: [SUPPORT] - Incremental Records are not applied to the Hudi Table
URL: https://github.com/apache/hudi/issues/7659
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #7659: [SUPPORT] - Incremental Records are not applied to the Hudi Table
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #7659:
URL: https://github.com/apache/hudi/issues/7659#issuecomment-1412133801
@soma1712 Can you please update the version and try again?
@soumilshah1995 also has a nice video showing incremental processing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #7659: [SUPPORT] - Incremental Records are not applied to the Hudi Table
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #7659:
URL: https://github.com/apache/hudi/issues/7659#issuecomment-1521953855
@soma1712 Were you able to test this again with latest version? Are we good to close this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] soumilshah1995 commented on issue #7659: [SUPPORT] - Incremental Records are not applied to the Hudi Table
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #7659:
URL: https://github.com/apache/hudi/issues/7659#issuecomment-1410375347
can we close this ticket i dont think there is issue with INC processing with latest versions
@nsivabalan or @danny0405
## Video
https://www.youtube.com/watch?v=4eevYkBClQ4&t=3s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] soumilshah1995 commented on issue #7659: [SUPPORT] - Incremental Records are not applied to the Hudi Table
Posted by "soumilshah1995 (via GitHub)" <gi...@apache.org>.
soumilshah1995 commented on issue #7659:
URL: https://github.com/apache/hudi/issues/7659#issuecomment-1414542316
> codope Thank you sir :D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #7659: [SUPPORT] - Incremental Records are not applied to the Hudi Table
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #7659:
URL: https://github.com/apache/hudi/issues/7659#issuecomment-1529479677
Closing the issue as latest version should fix. @soma1712 Please reopen in case you still faces the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #7659: [SUPPORT] - Incremental Records are not applied to the Hudi Table
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #7659:
URL: https://github.com/apache/hudi/issues/7659#issuecomment-1381571163
The hudi version is very old, we just released the 0.12.2 version, do you have any chance to try that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org