[jira] [Created] (SPARK-46247) Invalid bucket file error when reading from bucketed table created with PathOutputCommitProtocol

["Никита Соколов (Jira)" <ji...@apache.org>]

Никита Соколов created SPARK-46247:
--------------------------------------

             Summary: Invalid bucket file error when reading from bucketed table created with PathOutputCommitProtocol
                 Key: SPARK-46247
                 URL: https://issues.apache.org/jira/browse/SPARK-46247
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 3.5.0
            Reporter: Никита Соколов


I am trying to create an external partioned bucketed table using this code:
{code:java}
spark.read.parquet("s3://faucct/input")
  .repartition(128, col("product_id"))
  .write.partitionBy("features_date").bucketBy(128, "product_id")
  .option("path", "s3://faucct/tmp/output")
  .option("compression", "uncompressed")
  .saveAsTable("tmp.output"){code}
At first it took more time than expected because it had to rename a lot of files in the end, which requires copying in S3. But I have used the configuration from the documentation – [https://spark.apache.org/docs/3.0.0-preview/cloud-integration.html#committing-work-into-cloud-storage-safely-and-fast]:
{code:java}
spark.hadoop.fs.s3a.committer.name directory
spark.sql.sources.commitProtocolClass org.apache.spark.internal.io.cloud.PathOutputCommitProtocol
spark.sql.parquet.output.committer.class org.apache.spark.internal.io.cloud.BindingParquetOutputCommitter {code}
It is properly partitioned: every partition_date has exactly 128 files named like [part-00117-43293810-d0e9-4eee-9be8-e9e50a3e10fd_00117-5eb66a54-2fbb-4775-8f3b-3040b2966a71.c000.parquet|https://s3.console.aws.amazon.com/s3/object/joom-analytics-recom?region=eu-central-1&prefix=recom/dataset/best/best-to-cart-rt/user-product-v4/to_cart-faucct/fnw/ipw/msv2/2023-09-15/14d/tmp_3/features_date%3D2023-09-01/part-00117-43293810-d0e9-4eee-9be8-e9e50a3e10fd_00117-5eb66a54-2fbb-4775-8f3b-3040b2966a71.c000.parquet].
Then I am trying to join this table with another one, for example like this:
{code:java}
spark.table("tmp.output").repartition(128, $"product_id")
  .join(spark.table("tmp.output").repartition(128, $"product_id"), Seq("product_id")).count(){code}
Because of the configuration I get the following errors:
{code:java}
org.apache.spark.SparkException: [INVALID_BUCKET_FILE] Invalid bucket file: s3://faucct/tmp/output/features_date=2023-09-01/part-00000-43293810-d0e9-4eee-9be8-e9e50a3e10fd_00000-5eb66a54-2fbb-4775-8f3b-3040b2966a71.c000.parquet.
  at org.apache.spark.sql.errors.QueryExecutionErrors$.invalidBucketFile(QueryExecutionErrors.scala:2731)
  at org.apache.spark.sql.execution.FileSourceScanExec.$anonfun$createBucketedReadRDD$5(DataSourceScanExec.scala:636) {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org