You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/09/21 09:13:20 UTC
[GitHub] [hudi] RajasekarSribalan opened a new issue #2098: [SUPPORT]
RajasekarSribalan opened a new issue #2098:
URL: https://github.com/apache/hudi/issues/2098
Hi Team,
We are getting Parquet not found error while reading a Hudi table from Spark.
What we do?
We read a Hudi table from Spark(Select * from table) and do an insert and overwrrite on another hive table.Basically we take the snapshot of Hudi table and insert it in another table. Last couple of days, we are not able to read the entire table from Spark. We are getting bellow error.
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://baikal-prod/user/tempuser/hudi/hudipath/tbale_name/343f7bec-e29d-4b1e-a429-463c8efb09fb-0_91-23390-2236222_20200921075346.parquet
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1266)
at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1258)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1258)
at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:386)
at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:372)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.getSplit(ParquetRecordReaderWrapper.java:252)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:95)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:81)
at org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:72)
at org.apache.hudi.hadoop.HoodieParquetInputFormat.getRecordReader(HoodieParquetInputFormat.java:297)
at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:246)
at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:245)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:203)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:94)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Note: Parallely we have Streaming job which populating the source hudi table.
How to fix these kind of issues? Pls assist. If there are any Hudi metadata problem? It was running perfectly for these many days but not sure why it throws file not found error. Whether cleaner didnot work prooperly? Pls help
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar closed issue #2098: [SUPPORT] File does not exisit(parquet) while reading Hudi Table from Spark
Posted by GitBox <gi...@apache.org>.
bvaradar closed issue #2098:
URL: https://github.com/apache/hudi/issues/2098
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ShortFinger commented on issue #2098: [SUPPORT] File does not exisit(parquet) while reading Hudi Table from Spark
Posted by GitBox <gi...@apache.org>.
ShortFinger commented on issue #2098:
URL: https://github.com/apache/hudi/issues/2098#issuecomment-697061454
I have the same situation too.
Maybe there is some API or config can prevent Hudi to merge log files to base parquet file.
COW table can not do this,I trying to use MOR table to control the version of the parquet file and update the base parquet file by my code and timing.
We still looking for API or config to avoid this problem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2098: [SUPPORT] File does not exisit(parquet) while reading Hudi Table from Spark
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2098:
URL: https://github.com/apache/hudi/issues/2098#issuecomment-696209149
@RajasekarSribalan A FileNotFound error indicates that you are reading a version of the parquet file that has been deleted or no longer exists. This can happen due to the following reason : Say your job in writing to the Hudi table every 15 mins and you have chosen to keep only the latest version of the parquet file. Now, your snapshot job runs every 1 hr and takes around 1 hr to finish. What can happen is that the snapshot job ends up reading an older version of the parquet file while the new version is being created by the ingestion job and the cleaner deletes the older version.
Since your job was running for many days, it seems like either a) The frequency of the ingestion job to Hudi or the snapshot job to Hive changed b) The snapshot job runs for longer period of time causing file not found c) It was just working by chance
To fix this issue, please make sure you keep enough number of file versions so that a long running job (like the snapshot job) can find the file it started to read in the first place.
Please take a look at your configurations for the cleaner policy and then tune them using this config -> https://hudi.apache.org/docs/configurations.html#withCleanerPolicy
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] bvaradar commented on issue #2098: [SUPPORT] File does not exisit(parquet) while reading Hudi Table from Spark
Posted by GitBox <gi...@apache.org>.
bvaradar commented on issue #2098:
URL: https://github.com/apache/hudi/issues/2098#issuecomment-744885017
@ShortFinger : Let us know if you need any clarifications.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ShortFinger commented on issue #2098: [SUPPORT] File does not exisit(parquet) while reading Hudi Table from Spark
Posted by GitBox <gi...@apache.org>.
ShortFinger commented on issue #2098:
URL: https://github.com/apache/hudi/issues/2098#issuecomment-697061454
I have the same situation too.
Maybe there is some API or config can prevent Hudi to merge log files to base parquet file.
COW table can not do this,I trying to use MOR table to control the version of the parquet file and update the base parquet file by my code and timing.
We still looking for API or config to avoid this problem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2098: [SUPPORT] File does not exisit(parquet) while reading Hudi Table from Spark
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2098:
URL: https://github.com/apache/hudi/issues/2098#issuecomment-696209149
@RajasekarSribalan A FileNotFound error indicates that you are reading a version of the parquet file that has been deleted or no longer exists. This can happen due to the following reason : Say your job in writing to the Hudi table every 15 mins and you have chosen to keep only the latest version of the parquet file. Now, your snapshot job runs every 1 hr and takes around 1 hr to finish. What can happen is that the snapshot job ends up reading an older version of the parquet file while the new version is being created by the ingestion job and the cleaner deletes the older version.
Since your job was running for many days, it seems like either a) The frequency of the ingestion job to Hudi or the snapshot job to Hive changed b) The snapshot job runs for longer period of time causing file not found c) It was just working by chance
To fix this issue, please make sure you keep enough number of file versions so that a long running job (like the snapshot job) can find the file it started to read in the first place.
Please take a look at your configurations for the cleaner policy and then tune them using this config -> https://hudi.apache.org/docs/configurations.html#withCleanerPolicy
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on issue #2098: [SUPPORT] File does not exisit(parquet) while reading Hudi Table from Spark
Posted by GitBox <gi...@apache.org>.
n3nash commented on issue #2098:
URL: https://github.com/apache/hudi/issues/2098#issuecomment-697705248
@ShortFinger For COW -> The number of versions to keep is a function of a) how frequently you run the ingestion job which may have updates b) how long running is the consumer of this table. So, if the consumer of this table runs a query lasting for 1 hr, this means you need to keep atleast the version of the file that was generated 1 hr ago since this query might end up reading it.
If your job frequency is lets say 15 mins, you need to set the COMMITS_RETAINED to 4 since worst case, you could get an update for the same record key in every 15 mins batch leading to multiple versions of the same parquet file
For MOR -> The number of versions to keep is a function of how frequently you run the compaction process as opposed to how frequently you run ingestion, apart from that, the same logic applies here as well.
Hope that provides some clarity.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org