You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "RushabhK (via GitHub)" <gi...@apache.org> on 2025/05/29 14:07:45 UTC
[I] Native parquet writer writing a corrupt / invalid parquet file on spark task failure [incubator-gluten]
RushabhK opened a new issue, #9801:
URL: https://github.com/apache/incubator-gluten/issues/9801
### Backend
VL (Velox)
### Bug description
Hello team, I am using Gluten 1.3.0 compiled with JDK 17 on Ubuntu 20.04.
I ingested parquet files with a spark job with gluten which underwent 1 task failure and a successful retry, with the job succeeding.
Although, while reading the parquet folder, I can find one file which says the parquet file is corrupt with the following exception:
`OSError: Could not open parquet input source 'gluten-part-239f94b0-2c93-4b5d-a8a7-3be9a9e05dac.zstd.parquet': Invalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.`
This is a 300 MB file and I verified from the parquet-tools that it is not a valid parquet file.
Although, reading the path with spark.read.option("ignoreCorruptFiles", "true").parquet(path), I get exactly the data that is expected (I did the checksum validation the dataframe with the expectation, and it passed).
I am suspecting the failed task should not have copied the partial parquet file which it had written to the temp location to the final location as a part of the commit.
I am using the Manifest committer with version 2, please find the following configs:
`--conf spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 \
--conf spark.hadoop.mapreduce.fileoutputcommitter.cleanup.skipped=true \
--conf spark.hadoop.mapreduce.fileoutputcommitter.marksuccessfuljobs=false \
--conf spark.hadoop.mapreduce.outputcommitter.factory.scheme.gs=org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterFactory \
--conf spark.sql.parquet.output.committer.class=org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter`
Can anyone point out if I am missing some gluten configs while ingesting parquet which is not preventing this copy? Also, does Gluten honor the committer specified in the spark conf?
I am not facing any such issues with vanilla spark for the same runs.
### Gluten version
Gluten-1.3
### Spark version
Spark-3.5.x
### Spark configurations
--conf spark.hadoop.mapreduce.outputcommitter.factory.scheme.gs=org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterFactory \
--conf spark.sql.parquet.output.committer.class=org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter \
--conf spark.sql.sources.commitProtocolClass=org.apache.spark.internal.io.cloud.PathOutputCommitProtocol \
--conf spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version=2 \
--conf spark.hadoop.mapreduce.fileoutputcommitter.cleanup.skipped=true \
--conf spark.hadoop.mapreduce.fileoutputcommitter.marksuccessfuljobs=false \
--conf spark.executor.cores=8 \
--conf spark.executor.memory=10G \
--conf spark.executor.memoryOverhead=6G \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=35G \
--conf spark.driver.core=4 \
--conf spark.driver.memoryOverhead=1500M \
--conf spark.network.timeout=900s \
--conf spark.driver.maxResultSize=4G \
--conf spark.executor.instances=12 \
--conf spark.dynamicAllocation.enabled=false \
--conf spark.hadoop.hive.exec.dynamic.partition=true \
--conf spark.hadoop.hive.exec.dynamic.partition.mode=nonstrict \
--conf spark.plugins=org.apache.gluten.GlutenPlugin \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--conf spark.driver.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \
--conf spark.executor.extraJavaOptions="-Dio.netty.tryReflectionSetAccessible=true" \
--conf spark.gluten.sql.complexType.scan.fallback.enabled=false \
--conf spark.gluten.sql.columnar.maxBatchSize=4096 \
--conf spark.gluten.sql.columnar.tableCache=true \
--conf spark.gluten.memory.isolation=true \
--conf spark.cleaner.periodicGC.interval="3min" \
--conf spark.gluten.sql.columnar.shuffle.codec="zstd" \
--conf spark.executorEnv.LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libjemalloc.so.2" \
### System information
_No response_
### Relevant logs
```bash
Caused by: org.apache.spark.SparkException: [CANNOT_READ_FILE_FOOTER] Could not read footer for file: gs://some_path/gluten-part-239f94b0-2c93-4b5d-a8a7-3be9a9e05dac.zstd.parquet. Please ensure that the file is in either ORC or Parquet format. If not, please convert it to a valid format. If the file is in the valid format, please check if it is corrupt. If it is, you can choose to either ignore it or fix the corruption.
at org.apache.spark.sql.errors.QueryExecutionErrors$.cannotReadFooterForFileError(QueryExecutionErrors.scala:1057)
at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.$anonfun$readParquetFootersInParallel$1(ParquetFileFormat.scala:456)
at org.apache.spark.util.ThreadUtils$.$anonfun$parmap$2(ThreadUtils.scala:384)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: java.lang.RuntimeException: gs://some_path/gluten-part-239f94b0-2c93-4b5d-a8a7-3be9a9e05dac.zstd.parquet is not a Parquet file. Expected magic number at tail, but found [2, 0, 0, 0]
at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:565)
at org.apache.parquet.hadoop.ParquetFileReader.<init>(ParquetFileReader.java:799)
at org.apache.parquet.hadoop.ParquetFileReader.open(ParquetFileReader.java:666)
at org.apache.spark.sql.execution.datasources.parquet.ParquetFooterReader.readFooter(ParquetFooterReader.java:85)
at org.apache.spark.sql.execution.datasources.parquet.ParquetFooterReader.readFooter(ParquetFooterReader.java:76)
at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$.$anonfun$readParquetFootersInParallel$1(ParquetFileFormat.scala:450)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Native parquet writer writing a corrupt / invalid parquet file on spark task failure [incubator-gluten]
Posted by "RushabhK (via GitHub)" <gi...@apache.org>.
RushabhK commented on issue #9801:
URL: https://github.com/apache/incubator-gluten/issues/9801#issuecomment-2965552203
Hello @FelixYBW I had a question on the file output committers in Gluten.
I was using the Manifest committter in one of my jobs with Gluten and in the driver logs I could see that this Manifest committer, file count was 0 in all the `manifest.json` and data size 0 was not doing anything with Gluten.
So my question is who renames / moves these files in gluten with Manifest? Does Gluten have its own mechanism for handling commits?
These are the driver logs I could see when running with gluten:
```
25/06/11 15:37:21 [main] INFO PathOutputCommitterFactory: Using OutputCommitter factory class class org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitterFactory from key mapreduce.outputcommitter.factory.scheme.gs
25/06/11 15:37:21 [main] INFO ManifestCommitter: Created ManifestCommitter with JobID job_202506111537213028082964935343887_0000, Task Attempt attempt_202506111537213028082964935343887_0000_m_000000_0 and destination gs://<some_path>/.spark-staging-a0487498-59b2-4317-a70f-b72f303e3bfb
25/06/11 15:48:52 [main] INFO AbstractJobOrTaskStage: [Job-Attempt fa204b99-986e-4d65-8912-a107afc4105c/00]: Executing Stage job_stage_load_manifests
25/06/11 15:48:52 [main] INFO LoadManifestsStage: [Job-Attempt fa204b99-986e-4d65-8912-a107afc4105c/00]: Executing Manifest Job Commit with manifests in gs://<some_path>/.spark-staging-a0487498-59b2-4317-a70f-b72f303e3bfb/_temporary/fa204b99-986e-4d65-8912-a107afc4105c/00/manifests
25/06/11 15:48:52 [manifest-committer-fa204b99-986e-4d65-8912-a107afc4105c_0-31] INFO LoadManifestsStage: [Job-Attempt fa204b99-986e-4d65-8912-a107afc4105c/00]: Task Attempt attempt_202506111537213403690146387912535_0002_m_000031_1633 file gs://<some_path>/.spark-staging-a0487498-59b2-4317-a70f-b72f303e3bfb/_temporary/fa204b99-986e-4d65-8912-a107afc4105c/00/manifests/task_202506111537213403690146387912535_0002_m_000031-manifest.json: File count: 0; data size=0
25/06/11 15:48:52 [manifest-committer-fa204b99-986e-4d65-8912-a107afc4105c_0-24] INFO LoadManifestsStage: [Job-Attempt fa204b99-986e-4d65-8912-a107afc4105c/00]: Task Attempt attempt_202506111537213403690146387912535_0002_m_000024_1695 file gs://<some_path>/.spark-staging-a0487498-59b2-4317-a70f-b72f303e3bfb/_temporary/fa204b99-986e-4d65-8912-a107afc4105c/00/manifests/task_202506111537213403690146387912535_0002_m_000024-manifest.json: File count: 0; data size=0
25/06/11 15:48:52 [manifest-committer-fa204b99-986e-4d65-8912-a107afc4105c_0-16] INFO LoadManifestsStage: [Job-Attempt fa204b99-986e-4d65-8912-a107afc4105c/00]: Task Attempt attempt_202506111537213403690146387912535_0002_m_000016_1683 file gs://<some_path>/.spark-staging-a0487498-59b2-4317-a70f-b72f303e3bfb/_temporary/fa204b99-986e-4d65-8912-a107afc4105c/00/manifests/task_202506111537213403690146387912535_0002_m_000016-manifest.json: File count: 0; data size=0
25/06/11 15:48:52 [manifest-committer-fa204b99-986e-4d65-8912-a107afc4105c_0-23] INFO LoadManifestsStage: [Job-Attempt fa204b99-986e-4d65-8912-a107afc4105c/00]: Task Attempt attempt_202506111537213403690146387912535_0002_m_000023_1682 file gs://<some_path>/.spark-staging-a0487498-59b2-4317-a70f-b72f303e3bfb/_temporary/fa204b99-986e-4d65-8912-a107afc4105c/00/manifests/task_202506111537213403690146387912535_0002_m_000023-manifest.json: File count: 0; data size=0
25/06/11 15:48:52 [manifest-committer-fa204b99-986e-4d65-8912-a107afc4105c_0-25] INFO LoadManifestsStage: [Job-Attempt fa204b99-986e-4d65-8912-a107afc4105c/00]: Task Attempt attempt_202506111537213403690146387912535_0002_m_000025_1630 file gs://<some_path>/.spark-staging-a0487498-59b2-4317-a70f-b72f303e3bfb/_temporary/fa204b99-986e-4d65-8912-a107afc4105c/00/manifests/task_202506111537213403690146387912535_0002_m_000025-manifest.json: File count: 0; data size=0
25/06/11 15:48:52 [manifest-committer-fa204b99-986e-4d65-8912-a107afc4105c_0-26] INFO LoadManifestsStage: [Job-Attempt fa204b99-986e-4d65-8912-a107afc4105c/00]: Task Attempt attempt_202506111537213403690146387912535_0002_m_000026_1636 file gs://<some_path>/.spark-staging-a0487498-59b2-4317-a70f-b72f303e3bfb/_temporary/fa204b99-986e-4d65-8912-a107afc4105c/00/manifests/task_202506111537213403690146387912535_0002_m_000026-manifest.json: File count: 0; data size=0
25/06/11 15:48:52 [manifest-committer-fa204b99-986e-4d65-8912-a107afc4105c_0-7] INFO LoadManifestsStage: [Job-Attempt fa204b99-986e-4d65-8912-a107afc4105c/00]: Task Attempt attempt_202506111537213403690146387912535_0002_m_000007_1824 file gs://<some_path>/.spark-staging-a0487498-59b2-4317-a70f-b72f303e3bfb/_temporary/fa204b99-986e-4d65-8912-a107afc4105c/00/manifests/task_202506111537213403690146387912535_0002_m_000007-manifest.json: File count: 0; data size=0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Native parquet writer writing a corrupt / invalid parquet file on spark task failure [incubator-gluten]
Posted by "FelixYBW (via GitHub)" <gi...@apache.org>.
FelixYBW commented on issue #9801:
URL: https://github.com/apache/incubator-gluten/issues/9801#issuecomment-2920715133
@JkSelf we didn't considered the failure write.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Native parquet writer writing a corrupt / invalid parquet file on spark task failure [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf commented on issue #9801:
URL: https://github.com/apache/incubator-gluten/issues/9801#issuecomment-2921270206
@RushabhK Can you help to verify this [pr ](https://github.com/apache/incubator-gluten/pull/9808)in your env? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Native parquet writer writing a corrupt / invalid parquet file on spark task failure [incubator-gluten]
Posted by "RushabhK (via GitHub)" <gi...@apache.org>.
RushabhK commented on issue #9801:
URL: https://github.com/apache/incubator-gluten/issues/9801#issuecomment-2929038203
Hello @JkSelf, I did try running with this fix in my environment. Although now there are job failures with FileNotFoundException. Please find the error logs below.
`Caused by: java.io.FileNotFoundException: listStatus(hadoopPath: gs://<some_path>/.spark-staging-6815c37c-acfc-4167-95ba-cd9fefaf325a/_temporary/a6b6b0b2-bc38-4620-8ca2-18be8938fbc4/00/tasks/attempt_202505301505413719290422332255928_0002_m_000298_1958): 'gs://<some_path>/.spark-staging-6815c37c-acfc-4167-95ba-cd9fefaf325a/_temporary/a6b6b0b2-bc38-4620-8ca2-18be8938fbc4/00/tasks/attempt_202505301505413719290422332255928_0002_m_000298_1958' does not exist.
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.listStatus(GoogleHadoopFileSystemBase.java:937)
at org.apache.hadoop.fs.FileSystem.listStatusBatch(FileSystem.java:2066)
at org.apache.hadoop.fs.FileSystem$DirListingIterator.<init>(FileSystem.java:2317)
at org.apache.hadoop.fs.FileSystem.listStatusIterator(FileSystem.java:2361)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.impl.ManifestStoreOperationsThroughFileSystem.listStatusIterator(ManifestStoreOperationsThroughFileSystem.java:126)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage.lambda$listStatusIterator$5(AbstractJobOrTaskStage.java:488)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage.listStatusIterator(AbstractJobOrTaskStage.java:487)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.TaskAttemptScanDirectoryStage.scanDirectoryTree(TaskAttemptScanDirectoryStage.java:148)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.TaskAttemptScanDirectoryStage.executeStage(TaskAttemptScanDirectoryStage.java:77)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.TaskAttemptScanDirectoryStage.executeStage(TaskAttemptScanDirectoryStage.java:51)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage.apply(AbstractJobOrTaskStage.java:210)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.CommitTaskStage.executeStage(CommitTaskStage.java:66)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.CommitTaskStage.executeStage(CommitTaskStage.java:41)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.stages.AbstractJobOrTaskStage.apply(AbstractJobOrTaskStage.java:210)
at org.apache.hadoop.mapreduce.lib.output.committer.manifest.ManifestCommitter.commitTask(ManifestCommitter.java:310)
at org.apache.spark.mapred.SparkHadoopMapRedUtil$.$anonfun$commitTask$1(SparkHadoopMapRedUtil.scala:51)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:552)
at org.apache.spark.mapred.SparkHadoopMapRedUtil$.performCommit$1(SparkHadoopMapRedUtil.scala:51)
at org.apache.spark.mapred.SparkHadoopMapRedUtil$.commitTask(SparkHadoopMapRedUtil.scala:78)
at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitTask(HadoopMapReduceCommitProtocol.scala:401)
at org.apache.spark.sql.execution.SparkWriteFilesCommitProtocol.$anonfun$commitTask$1(SparkWriteFilesCommitProtocol.scala:91)
at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:552)
at org.apache.spark.sql.execution.SparkWriteFilesCommitProtocol.commitTask(SparkWriteFilesCommitProtocol.scala:91)
at org.apache.spark.sql.execution.VeloxColumnarWriteFilesRDD.$anonfun$compute$2(VeloxColumnarWriteFilesExec.scala:218)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1397)
at org.apache.spark.sql.execution.VeloxColumnarWriteFilesRDD.compute(VeloxColumnarWriteFilesExec.scala:201)
... 13 more
Caused by: java.io.FileNotFoundException: Item not found: gs://<some_path>/.spark-staging-6815c37c-acfc-4167-95ba-cd9fefaf325a/_temporary/a6b6b0b2-bc38-4620-8ca2-18be8938fbc4/00/tasks/attempt_202505301505413719290422332255928_0002_m_000298_1958
at com.google.cloud.hadoop.repackaged.gcs.com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.listFileInfo(GoogleCloudStorageFileSystem.java:1052)
at com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.listStatus(GoogleHadoopFileSystemBase.java:928)
... 43 more
Caused by: org.apache.spark.SparkException: Task failed while writing rows to staging path: gs://<some_path>/.spark-staging-6815c37c-acfc-4167-95ba-cd9fefaf325a, output path: gs://<some_path>/.spark-staging-6815c37c-acfc-4167-95ba-cd9fefaf325a`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Native parquet writer writing a corrupt / invalid parquet file on spark task failure [incubator-gluten]
Posted by "RushabhK (via GitHub)" <gi...@apache.org>.
RushabhK commented on issue #9801:
URL: https://github.com/apache/incubator-gluten/issues/9801#issuecomment-2919833366
I tried reproducing the same issue with turning the vanilla spark writer on with this config: `--conf spark.gluten.sql.native.writer.enabled=false`.
The issue did not surface with the spark writer, so this seems like some handling in Gluten.
Possible to point out the code in Gluten repository which takes care of the file output committer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Native parquet writer writing a corrupt / invalid parquet file on spark task failure [incubator-gluten]
Posted by "RushabhK (via GitHub)" <gi...@apache.org>.
RushabhK commented on issue #9801:
URL: https://github.com/apache/incubator-gluten/issues/9801#issuecomment-2921707134
> [@RushabhK](https://github.com/RushabhK) Can you help to verify this [pr ](https://github.com/apache/incubator-gluten/pull/9808)in your env? Thanks.
Sure @JkSelf , I will create build with this fix and will try this out. Will revert on the results, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Native parquet writer writing a corrupt / invalid parquet file on spark task failure [incubator-gluten]
Posted by "RushabhK (via GitHub)" <gi...@apache.org>.
RushabhK commented on issue #9801:
URL: https://github.com/apache/incubator-gluten/issues/9801#issuecomment-2979331168
@JkSelf @PHILO-HE @FelixYBW I tested this fix and it's working fine, can we please review it: https://github.com/apache/incubator-gluten/pull/9993
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Native parquet writer writing a corrupt / invalid parquet file on spark task failure [incubator-gluten]
Posted by "JkSelf (via GitHub)" <gi...@apache.org>.
JkSelf closed issue #9801: [VL] Native parquet writer writing a corrupt / invalid parquet file on spark task failure
URL: https://github.com/apache/incubator-gluten/issues/9801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org