You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "king5holiday (via GitHub)" <gi...@apache.org> on 2023/05/26 02:05:13 UTC
[GitHub] [hudi] king5holiday opened a new issue, #8810: when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]"
king5holiday opened a new issue, #8810:
URL: https://github.com/apache/hudi/issues/8810
so i want to know why there will be some parquet files which the file name start with dot(.) when I write data to hudi? And how to filter these files when I read hudi by spark? Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] king5holiday commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]"
Posted by "king5holiday (via GitHub)" <gi...@apache.org>.
king5holiday commented on issue #8810:
URL: https://github.com/apache/hudi/issues/8810#issuecomment-1565799976
> > > Flink
> >
> >
> > @danny0405 Thank you for your replay! yes, I wrote data to hudi by flink and the version of hudi is 0.11.1
>
> Can you use 0.12.3 or 0.13.1, it should be fixed.
@danny0405 thank you! Should hudi-spark-bundle version and hudi-flink-bundle version be changed to 0.12.3 or 0.13.1 at the same time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]" [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 commented on issue #8810:
URL: https://github.com/apache/hudi/issues/8810#issuecomment-4844991653
This is fixed in Hudi 1.2.0.
Root cause: Flink can leave intermediate/uncommitted data files prefixed with . (hidden). The file-system-view's partition listing loaded every file in the partition without checking it was a valid Hudi data/log file, so the Spark reader tried to read a stray .-prefixed (incomplete) parquet → is not a Parquet file. expected magic number.
Fix: [#18047](https://github.com/apache/hudi/pull/18047) ("Filter stray files when loading partitions in AbstractTableFileSystemView") adds filterValidDataFiles() (using FSUtils.isDataFile()) so partition listing skips hidden/temp/.crc/non-conformant files. As a result, spark.read.format("hudi") no longer attempts to read these stray files. This is in release-1.2.0. On the Flink side, intermediate-file handling/cleanup was also improved (HUDI-1902 [#2949](https://github.com/apache/hudi/pull/2949), HUDI-5223 partial failover).
As @danny0405 noted at the time, older releases had no automatic handling (the workaround was deleting the hidden files manually). On 1.2.0+ the reader filters them out automatically.
Closing as fixed in 1.2.0. If you still hit is not a Parquet file reading a Hudi table on 1.2.0+, please reopen with the offending file name and how the table was written. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] king5holiday commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]"
Posted by "king5holiday (via GitHub)" <gi...@apache.org>.
king5holiday commented on issue #8810:
URL: https://github.com/apache/hudi/issues/8810#issuecomment-1563768270
>
Thank you for your replay! yes, I wrote data to hudi by flink and the version of hudi is 0.11.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]"
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8810:
URL: https://github.com/apache/hudi/issues/8810#issuecomment-1569409786
Not sure, maybe you can delete the hidden files manually, there is no atomatic fix when upgrading to 0.13.1.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] king5holiday commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]"
Posted by "king5holiday (via GitHub)" <gi...@apache.org>.
king5holiday commented on issue #8810:
URL: https://github.com/apache/hudi/issues/8810#issuecomment-1566649864
> Yeah, we better upgrade the bundle jars altogether.
I changed hudi-spark-bundle version from 0.11.1 to 0.13.1 and try to read data by spark, but the error reported again. when i use flink to read, although old version, it went well, so does flink has a filtering mechanism or remove the parquet files that start with dot(.)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]" [hudi]
Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]"
URL: https://github.com/apache/hudi/issues/8810
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]"
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8810:
URL: https://github.com/apache/hudi/issues/8810#issuecomment-1568058125
Did you read the table by specifying hudi as the format? Or just read it as a raw parquet table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]"
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8810:
URL: https://github.com/apache/hudi/issues/8810#issuecomment-1564203407
> > Flink
>
> @danny0405 Thank you for your replay! yes, I wrote data to hudi by flink and the version of hudi is 0.11.1
Can you use 0.12.3 or 0.13.1, it should be fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] king5holiday commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]"
Posted by "king5holiday (via GitHub)" <gi...@apache.org>.
king5holiday commented on issue #8810:
URL: https://github.com/apache/hudi/issues/8810#issuecomment-1563768292
>
Thank you for your replay! yes, I wrote data to hudi by flink and the version of hudi is 0.11.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] king5holiday commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]"
Posted by "king5holiday (via GitHub)" <gi...@apache.org>.
king5holiday commented on issue #8810:
URL: https://github.com/apache/hudi/issues/8810#issuecomment-1563768961
> Flink
@danny0405 Thank you for your replay! yes, I wrote data to hudi by flink and the version of hudi is 0.11.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] king5holiday commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]"
Posted by "king5holiday (via GitHub)" <gi...@apache.org>.
king5holiday commented on issue #8810:
URL: https://github.com/apache/hudi/issues/8810#issuecomment-1569406447
> Did you read the table by specifying hudi as the format? Or just read it as a raw parquet table.
hi, there are some codes,
` spark.read
.format("org.apache.hudi")
.load(HUDIPATH)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]"
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8810:
URL: https://github.com/apache/hudi/issues/8810#issuecomment-1563762919
Did you write the table using Flink, flink could create some imtermediate data files which starts with `.`. And what verion of Hudi did you use then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]"
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8810:
URL: https://github.com/apache/hudi/issues/8810#issuecomment-1566514189
Yeah, we better upgrade the bundle jars altogether.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] king5holiday commented on issue #8810: [SUPPORT] when read parquet files which the file name starts with dot(.) by spark, there will create an error like "Caused by: java.lang.RuntimeException: hdfs://path/.0726018d-0e03-48e2-9b88-d7e228cf1aff_0-4-0-0_20230111085451189.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [39, 78, -93, 10]"
Posted by "king5holiday (via GitHub)" <gi...@apache.org>.
king5holiday commented on issue #8810:
URL: https://github.com/apache/hudi/issues/8810#issuecomment-1573275278
> Not sure, maybe you can delete the hidden files manually, there is no atomatic fix when upgrading to 0.13.1.
ok, maybe it's an effective way until now. Looking forward to fixing the bug in new version. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org