You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "ghost (via GitHub)" <gi...@apache.org> on 2023/05/24 13:54:41 UTC
[GitHub] [hudi] ghost opened a new issue, #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
ghost opened a new issue, #4978:
URL: https://github.com/apache/hudi/issues/4978
**Describe the problem you faced**
When using Hive to query xxx_rt table，if there is no parquet file but only log files, we get a wrong table path.
But when the parquet files are generated, the table path is correct and we can get the data.
Is this expected behavior?
```
ERROR : Job failed with java.io.FileNotFoundException: File does not exist: hdfs://da-hdfs/tmp/hive/hadoop/90b7d231-0e0a-42e5-a72a-6faad6a9ac89/.hoodie
org.apache.hudi.exception.TableNotFoundException: Hoodie table not found in path hdfs://da-hdfs/tmp/hive/hadoop/90b7d231-0e0a-42e5-a72a-6faad6a9ac89/.hoodie
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://da-hdfs/tmp/hive/hadoop/90b7d231-0e0a-42e5-a72a-6faad6a9ac89/.hoodie
```
**Environment Description**
* Hudi version : 0.10.1
* Hive version : 3.1.2
* Hadoop version : 3.3.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xiarixiaoyao commented on issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
Posted by GitBox <gi...@apache.org>.
xiarixiaoyao commented on issue #4978:
URL: https://github.com/apache/hudi/issues/4978#issuecomment-1128340650
@nsivabalan i donot think 0.11 can solve this problem.
@CrazyBeeline thanks for your help. could you pls raise a pr to solve this problem, thanks very much
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
URL: https://github.com/apache/hudi/issues/4978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4978:
URL: https://github.com/apache/hudi/issues/4978#issuecomment-1216664212
@CrazyBeeline : gentle ping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4978:
URL: https://github.com/apache/hudi/issues/4978#issuecomment-1210062619
@CrazyBeeline : can you put up a patch w/ the fix you have. Happy to review and get it to landing. btw, are you using hbase or some other set up. wondering how did you end up w/ a file group w/ log file but w/o a base file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4978:
URL: https://github.com/apache/hudi/issues/4978#issuecomment-1125082789
but might be good to get it fixed irrespective of that. just trying to guage the use-case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] CrazyBeeline commented on issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
Posted by GitBox <gi...@apache.org>.
CrazyBeeline commented on issue #4978:
URL: https://github.com/apache/hudi/issues/4978#issuecomment-1094599704
maybe it can help you @awpengfei @xiarixiaoyao
modify source code like this:
org.apache.hudi.hadoop.HoodieParquetInputFormat
org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4978:
URL: https://github.com/apache/hudi/issues/4978#issuecomment-1125084414
@xiarixiaoyao : do we have a tracking ticket for this. Alexey did a revamp of all query engine code paths recently w/ 0.11. Do we have this issue even now after 0.11? do you have any idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
URL: https://github.com/apache/hudi/issues/4978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
Posted by GitBox <gi...@apache.org>.
danny0405 commented on issue #4978:
URL: https://github.com/apache/hudi/issues/4978#issuecomment-1302929509
No, we have not fixed it, the Hive/Trino all can not access file group with pure logs, can we move it to higher priority for release 0.13.0 and solve it then ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4978:
URL: https://github.com/apache/hudi/issues/4978#issuecomment-1125081869
is this written using flink? using spark, we won't create log files directly. data files will be created first and then only log files will be created for file groups.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #4978:
URL: https://github.com/apache/hudi/issues/4978#issuecomment-4832969128
Hi @CrazyBeeline
The Hive/realtime read path that caused this has been reworked, and log-only file groups (file groups with only .log files and no base parquet) are now handled explicitly in 1.x:
- The new FileGroup reader (default in 1.x) reads log-only file slices natively.
- Even the legacy realtime path now has an explicit branch — for a log-only split it uses a HoodieEmptyRecordReader as the base reader instead of falling through to a wrong path:
```
// HoodieParquetRealtimeInputFormat#getRecordReader
if (shouldUseFilegroupReader(jobConf, split)) {
return super.getRecordReader(realtimeSplit, jobConf, reporter);
}
// for log only split, set the parquet reader as empty.
if (isLogFile(realtimeSplit.getPath())) {
return new HoodieRealtimeRecordReader(realtimeSplit, jobConf,
new HoodieEmptyRecordReader(realtimeSplit, jobConf));
}
```
This is exactly the case that was missing in 0.10.1 (querying xxx_rt before the first compaction, when a file group has only log files).
Would you be able to retry your scenario on a recent Hudi 1.x release and let us know if it's resolved? If so, we can close this out. If you still hit it on 1.x, please share the table type, writer (Flink/Spark/HBase index), and the stack trace and we'll dig in.
Thanks! cc @xiarixiaoyao @nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4978:
URL: https://github.com/apache/hudi/issues/4978#issuecomment-1302901252
@danny0405 @xiarixiaoyao : do we know if we have fixed this anytime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #4978:
URL: https://github.com/apache/hudi/issues/4978#issuecomment-1561028773
Inserts always creates parquet files first, so first commit will create parquet before creating any logs.
Thus, this should not be a issue anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #4978: [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on issue #4978:
URL: https://github.com/apache/hudi/issues/4978#issuecomment-1561206103
@ad1happy2go to reproduce
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Wrong table path when using Hive to query xxx_rt table before the first compaction [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #4978:
URL: https://github.com/apache/hudi/issues/4978#issuecomment-4841932506
Closing this as fixed in Hudi 1.x.
Root cause: Hive's input format filters out .-prefixed files, so a MOR file group with only log files and no base parquet (the state before the first compaction) had no readable files, and the realtime input format fell through to a wrong path → TableNotFoundException.
Fix: the engine-agnostic HoodieFileGroupReader for Hive, added in [HUDI-6787 / [#10422](https://github.com/apache/hudi/pull/10422)]. It reads file slices (base optional + log files) directly, so a log-only file slice is read natively rather than dropped. It's controlled by hoodie.file.group.reader.enabled, which defaults to true since 1.0.0, making it the default Hive read path in 1.x.
This is present in 1.0.0 and all later releases (1.0.x, 1.1.0, 1.1.1, 1.2.0). Reading xxx_rt before the first compaction (log-only file groups) works on these versions.
If anyone still reproduces this on a 1.x release, please reopen with the Hudi version and a minimal repro and we'll take another look. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org