You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/26 07:24:38 UTC

[GitHub] [hudi] wqwl611 opened a new issue #5136: [SUPPORT] Empty parquet "is not a Parquet file (length is too low: 0)"

wqwl611 opened a new issue #5136:
URL: https://github.com/apache/hudi/issues/5136


   sometimes i will find this bug, and i delete the empty parquet file, it will work but I guess I lost data.
   
   Steps to reproduce the behavior: i don't know how to reproduce 
   
   commit info show totalWriteBytes > 0 but the parquet is empty .:
   
   "_db=xxxx/_tbl=xxxx_014" : [ {
   "fileId" : "f2ecd769-40ba-42ac-a03d-0c527db6f1b2-0",
   "path" : "_db=xxxxx/_tbl=xxx_014/f2ecd769-40ba-42ac-a03d-0c527db6f1b2-0_69-13711-1702754_20220326121335059.parquet",
   "prevCommit" : "20220326115500480",
   "numWrites" : 671338,
   "numDeletes" : 0,
   "numUpdateWrites" : 0,
   "numInserts" : 58,
   "totalWriteBytes" : 43100370,
   
   but:
   
   f2ecd769-40ba-42ac-a03d-0c527db6f1b2-0_69-13711-1713783_20220326121335059.parquet is not a Parquet file (length is too low: 0)
   
   	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:539)
   	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:527)
   	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:521)
   	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:469)
   	at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:454)
   	at org.apache.hudi.common.util.ParquetUtils.readMetadata(ParquetUtils.java:89)
   	at org.apache.hudi.common.util.ParquetUtils.readFooter(ParquetUtils.java:211)
   	at org.apache.hudi.common.util.BaseFileUtils.readMinMaxRecordKeys(BaseFileUtils.java:109)
   	at org.apache.hudi.io.storage.HoodieParquetReader.readMinMaxRecordKeys(HoodieParquetReader.java:50)
   	at org.apache.hudi.io.HoodieRangeInfoHandle.getMinMaxKeys(HoodieRangeInfoHandle.java:41)
   	at org.apache.hudi.index.bloom.HoodieBloomIndex.lambda$loadColumnRangesFromFiles$4cbadf07$1(HoodieBloomIndex.java:156)
   	at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
   	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
   	at scala.collection.Iterator.foreach(Iterator.scala:943)
   	at scala.collection.Iterator.foreach$(Iterator.scala:943)
   	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
   	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
   	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
   	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:105)
   	at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:49)
   	at scala.collection.TraversableOnce.to(TraversableOnce.scala:366)
   	at scala.collection.TraversableOnce.to$(TraversableOnce.scala:364)
   	at scala.collection.AbstractIterator.to(Iterator.scala:1431)
   	at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:358)
   	at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:358)
   	at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1431)
   	at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:345)
   	at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:339)
   	at scala.collection.AbstractIterator.toArray(Iterator.scala:1431)
   	at org.apache.spark.rdd.RDD.$anonfun$collect$2(RDD.scala:1030)
   	at org.apache.spark.SparkContext.$anonfun$runJob$5(SparkContext.scala:2254)
   	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
   	at org.apache.spark.scheduler.Task.run(Task.scala:131)
   	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
   	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1489)
   	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   	at java.lang.Thread.run(Thread.java:745)
   
   Driver stacktrace:
   	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2414)
   	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2363)
   	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2362)
   	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
   	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
   	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
   	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2362)
   
   **Environment Description**
   
   * Hudi version : 0.11.snapshot
   
   * Spark version : 3.2.0
   
   * Hive version : 
   
   * Hadoop version : 2.7
   
   * Storage (HDFS/S3/GCS..) : hdfs 
   
   * Running on Docker? (yes/no) : no
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [hudi] wqwl611 commented on issue #5136: [SUPPORT] Empty parquet "is not a Parquet file (length is too low: 0)"

Posted by GitBox <gi...@apache.org>.
wqwl611 commented on issue #5136:
URL: https://github.com/apache/hudi/issues/5136#issuecomment-1079836598


   <img width="1239" alt="image" src="https://user-images.githubusercontent.com/67826098/160266592-be4dd02d-dcb6-4914-837b-987d42c35a88.png">
   
   I found maybe spark speculative execution cause this broken parquet.
   
   but now I am wonder why cleaner don't clean this broken file?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org