[GitHub] [iceberg] krisvaz edited a comment on issue #4363: DataFrameWriterV2 doesn't create partitions when using partitionedBy

[GitBox <gi...@apache.org>]

krisvaz edited a comment on issue #4363:
URL: https://github.com/apache/iceberg/issues/4363#issuecomment-1075214036


   @hililiwei 
   
   Here's an example of using the DataFrameWriterV2 to try creating the partitioned table and it's result:
   ```
   >>> df = spark.createDataFrame([{'partition_a': 'A', 'partition_b': 'B', 'data': 'somestuff'}])
   >>> df.writeTo("iceberg_catalog.p2.test").using("iceberg").partitionedBy("partition_a", "partition_b").create()
   >>> spark.sql("describe iceberg_catalog.p2.test").show()
   +---------------+---------+-------+
   |       col_name|data_type|comment|
   +---------------+---------+-------+
   |           data|   string|       |
   |    partition_a|   string|       |
   |    partition_b|   string|       |
   |               |         |       |
   | # Partitioning|         |       |
   |Not partitioned|         |       |
   +---------------+---------+-------+
   ```
   
   Here's an example of using spark sql to create the partitioned table and it's result:
   ```
   >>> df = spark.createDataFrame([{'partition_a': 'A', 'partition_b': 'B', 'data': 'somestuff'}])
   >>> df.createOrReplaceTempView('test_view')
   >>> spark.sql("""
   ... create table iceberg_catalog.p2.test using iceberg partitioned by (partition_a, partition_b)
   ... as select * from test_view""")
   DataFrame[]
   >>> spark.sql("describe iceberg_catalog.p2.test").show()
   +--------------+-----------+-------+
   |      col_name|  data_type|comment|
   +--------------+-----------+-------+
   |          data|     string|       |
   |   partition_a|     string|       |
   |   partition_b|     string|       |
   |              |           |       |
   |# Partitioning|           |       |
   |        Part 0|partition_a|       |
   |        Part 1|partition_b|       |
   +--------------+-----------+-------+
   ```
   
   I should note that I am running this within an AWS EMR cluster that's configured as defined [here](https://iceberg.apache.org/docs/latest/aws/#spark). 
   Instead of starting a spark-sql shell, I'm starting a pyspark shell with those same configs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org