[jira] [Commented] (SPARK-40287) Load Data using Spark by a single partition moves entire dataset under same location in S3

["Steve Loughran (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-40287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17598835#comment-17598835 ] 

Steve Loughran commented on SPARK-40287:
----------------------------------------

does this happen when
# you switch to an ASF spark build with the s3a connector
# and use an s3a committer safe to use with spark

this is clearly EMR (s3:// URLs), so they have to be the people to talk to if you can't replicate it in the apache code

> Load Data using Spark by a single partition moves entire dataset under same location in S3
> ------------------------------------------------------------------------------------------
>
>                 Key: SPARK-40287
>                 URL: https://issues.apache.org/jira/browse/SPARK-40287
>             Project: Spark
>          Issue Type: Question
>          Components: Spark Core
>    Affects Versions: 3.2.1
>            Reporter: Drew
>            Priority: Major
>
> Hello,
> I'm experiencing an issue in PySpark when creating a hive table and loading in the data to the table. So I'm using an Amazon s3 bucket as a data location and I'm creating a table as parquet and trying to load data into that table by a single partition, and I'm seeing some weird behavior. When selecting the data location in s3 of a parquet file to load into my table. All of the data is moved into the specified location in my create table command including the partitions I didn't specify in the load data command. For example:
> {code:java}
> # create a data frame in pyspark with partitions
> df = spark.createDataFrame([("a", 1, "x"), ("b", 2, "y"), ("c", 3, "y")], ["c1", "c2", "p"])
> # save it to S3
> df.write.format("parquet").mode("overwrite").partitionBy("p").save("s3://bucket/data/")
> {code}
> In the current state S3 should have a new folder `data` with two folders which contain a parquet file in each partition. 
>   
>  - s3://bucket/data/p=x/
>     - part-00001.snappy.parquet
>  - s3://bucket/data/p=y/
>     - part-00002.snappy.parquet
>     - part-00003.snappy.parquet
>  
> {code:java}
> # create new table
> spark.sql("create table src (c1 string,c2 int) PARTITIONED BY (p string) STORED AS parquet LOCATION 's3://bucket/new/'")
> # load the saved table data from s3 specifying single partition value x
> spark.sql("LOAD DATA INPATH 's3://bucket/data/'INTO TABLE src PARTITION (p='x')")
> spark.sql("select * from src").show()
> # output: 
> # +---+---+---+
> # | c1| c2|  p|
> # +---+---+---+
> # +---+---+---+
> {code}
> After running the `load data` command, and looking at the table I'm left with no data loaded in. When checking S3 the data source we saved earlier is moved under `s3://bucket/new/` oddly enough it also brought over the other partitions along with it directory structure listed below. 
> - s3://bucket/new/
>     - p=x/
>         - p=x/
>             - part-00001.snappy.parquet
>         - p=y/
>             - part-00002.snappy.parquet
>             - part-00003.snappy.parquet
> Is this the intended behavior of loading the data in from a partitioned parquet file? Is the previous file supposed to be moved/deleted from source directory? 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org