You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "NishantBaheti (via GitHub)" <gi...@apache.org> on 2024/03/12 07:33:14 UTC
[I] [SUPPORT] Incremental query not working on COW table [hudi]
NishantBaheti opened a new issue, #10850:
URL: https://github.com/apache/hudi/issues/10850
## Error
Error Category: QUERY_ERROR; AnalysisException: Found duplicate column(s) in the data schema: `_hoodie_commit_seqno`, `_hoodie_commit_time`, `_hoodie_file_name`, `_hoodie_partition_path`, `_hoodie_record_key`
##
hudi_options={
'hoodie.datasource.query.type': 'incremental',
'hoodie.datasource.read.begin.instanttime': start_time,
'hoodie.datasource.read.end.instanttime': end_time,
}
df=spark.read\
.format("org.apache.hudi")\
.options(**hudi_options)\
.load(tablePath)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental query not working on COW table [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10850:
URL: https://github.com/apache/hudi/issues/10850#issuecomment-2050061699
@NishantBaheti Thanks for the update. Surprisingly MOR worked but COW didn't work you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental query not working on COW table [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10850:
URL: https://github.com/apache/hudi/issues/10850#issuecomment-1991461575
@NishantBaheti I checked before, incremental query works fine with 0.14.1.
can you paste the full reproducible script or table/writer properties you used to populate. Which writer you used to populate this table?
I checked the below code to quickly to reproduce -
https://gist.github.com/ad1happy2go/e7a2f8c695fde4c3db060a7113610931
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental query not working on COW table [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #10850:
URL: https://github.com/apache/hudi/issues/10850#issuecomment-1991109123
Hi, @NishantBaheti , thanks for your feedback, could you also supplement the release version for Spark and Hudi respectively.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental query not working on COW table [hudi]
Posted by "NishantBaheti (via GitHub)" <gi...@apache.org>.
NishantBaheti commented on issue #10850:
URL: https://github.com/apache/hudi/issues/10850#issuecomment-1991494289
doesn't work. another issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental query not working on COW table [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10850:
URL: https://github.com/apache/hudi/issues/10850#issuecomment-2050117679
@NishantBaheti For incremental queries we can face FileNotFound Exception if the file for that query got deleted by the cleaner. We can set [hoodie.datasource.read.incr.fallback.fulltablescan.enable](https://hudi.apache.org/docs/configurations/#hoodiedatasourcereadincrfallbackfulltablescanenable) to true to get around this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental query not working on COW table [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10850:
URL: https://github.com/apache/hudi/issues/10850#issuecomment-2431798187
@NishantBaheti Closing this as it is the expected behaviour. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental query not working on COW table [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10850:
URL: https://github.com/apache/hudi/issues/10850#issuecomment-2029789548
@NishantBaheti Were you able to get it resolve? Can you let us know full stack trace. Looks like Unable to load class means some library conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental query not working on COW table [hudi]
Posted by "NishantBaheti (via GitHub)" <gi...@apache.org>.
NishantBaheti commented on issue #10850:
URL: https://github.com/apache/hudi/issues/10850#issuecomment-1991167509
Hello, I am using this jar
- hudi-spark3.3-bundle_2.12-0.14.1.jar
- spark 3.3
- hudi 0.14.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental query not working on COW table [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #10850:
URL: https://github.com/apache/hudi/issues/10850#issuecomment-1991460541
@NishantBaheti I checked before, incremental query works fine with 0.14.1.
can you paste the full reproducible script or table/writer properties you used to populate.
I checked the below code to quickly to reproduce -
https://gist.github.com/ad1happy2go/e7a2f8c695fde4c3db060a7113610931
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental query not working on COW table [hudi]
Posted by "NishantBaheti (via GitHub)" <gi...@apache.org>.
NishantBaheti commented on issue #10850:
URL: https://github.com/apache/hudi/issues/10850#issuecomment-2030086053
@ad1happy2go moved to the MOR table. COW configurations felt a little unstable. had to rush the project to production quickly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental query not working on COW table [hudi]
Posted by "NishantBaheti (via GitHub)" <gi...@apache.org>.
NishantBaheti commented on issue #10850:
URL: https://github.com/apache/hudi/issues/10850#issuecomment-2050108415
@ad1happy2go COW tables were failing a lot, like at the time of reading parquet file not found, no incremental query or getting the error mentioned above. Not saying that MOR is perfect but still had to put something in production with static configurations of MOR with quick compact cleaner so that athena `ro` tables behave like delta tables from delta framework and should be able to do point query using record index. I hope they figure out a stable version of hudi soon like how delta did.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Incremental query not working on COW table [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go closed issue #10850: [SUPPORT] Incremental query not working on COW table
URL: https://github.com/apache/hudi/issues/10850
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org