You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/03/18 20:54:18 UTC
[GitHub] [iceberg] krisvaz opened a new issue #4363: DataFrameWriterV2 doesn't create partitions when creating an Iceberg table
krisvaz opened a new issue #4363:
URL: https://github.com/apache/iceberg/issues/4363
When I try using the DataFrameWriterV2 to create an Iceberg table, the partitioning columns seem to get ignored (thereby not partitioned) even though they are specified.
Spark version: 3.1.2
Iceberg version: 0.13.0
Note: The application is written in python.
```
orders_df.writeTo("iceberg_catalog.p2.orders").using("iceberg").option(
"fanout-enabled", "true"
).partitionedBy("date(pickupts)", "appname").create()
```
Using Spark SQL to describe the table using this way shows this:
```
# Partitioning
Not partitioned
```
The workaround for creating the tables with the appropriate partitions is via Spark SQL.
i.e. this works as expected (see note below snippet):
```
orders_df.createOrReplaceTempView("p2_orders_view")
spark.sql(
"""
create table iceberg_catalog.p2.orders using iceberg
PARTITIONED BY (date(pickupts), appname)
as select * from p2_orders_view
"""
)
```
Here, when using Spark SQL to describe the table using this above method to create the table shows this:
```
# Partitioning
Part 0 days(pickupts)
Part 1 appname
```
My question is, am I using the DataFrameWriterV2 correctly when specifying the partitionedBy columns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on issue #4363: DataFrameWriterV2 doesn't create partitions when using partitionedBy
Posted by GitBox <gi...@apache.org>.
hililiwei commented on issue #4363:
URL: https://github.com/apache/iceberg/issues/4363#issuecomment-1072943877
@krisvaz I debugged it locally and it works as expected, can you share more information? Like your test data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] hililiwei commented on issue #4363: DataFrameWriterV2 doesn't create partitions when using partitionedBy
Posted by GitBox <gi...@apache.org>.
hililiwei commented on issue #4363:
URL: https://github.com/apache/iceberg/issues/4363#issuecomment-1079573700
@krisvaz
```
./pyspark --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.local.type=hadoop \
--conf spark.sql.catalog.local.warehouse=$PWD/warehouse
```
```
>>> spark.sql("create database local.p2").show()
>>> df = spark.createDataFrame([{'partition_a': 'A', 'partition_b': 'B', 'data': 'somestuff'}])
>>> df.writeTo("local.p2.test").using("iceberg").partitionedBy("partition_a", "partition_b").create()
>>> spark.sql("describe local.p2.test").show()
+--------------+-----------+-------+
| col_name| data_type|comment|
+--------------+-----------+-------+
| data| string| |
| partition_a| string| |
| partition_b| string| |
| | | |
|# Partitioning| | |
| Part 0|partition_a| |
| Part 1|partition_b| |
+--------------+-----------+-------+
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #4363: DataFrameWriterV2 doesn't create partitions when using partitionedBy
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #4363:
URL: https://github.com/apache/iceberg/issues/4363#issuecomment-1072838608
Looks right to me, and the code looks fine on the Spark Side
https://github.com/apache/spark/blob/v3.1.2/sql/core/src/main/scala/org/apache/spark/sql/DataFrameWriterV2.scala#L83-L126
Probably need to debug this later to find out exactly what is going wrong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] liuchunhua commented on issue #4363: DataFrameWriterV2 doesn't create partitions when using partitionedBy
Posted by GitBox <gi...@apache.org>.
liuchunhua commented on issue #4363:
URL: https://github.com/apache/iceberg/issues/4363#issuecomment-1086878041
@krisvaz @hililiwei
https://github.com/apache/spark/blob/v3.1.2/python/pyspark/sql/readwriter.py#L1496-L1525
missing "self._jwriter.partitionedBy(col, cols)"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] krisvaz commented on issue #4363: DataFrameWriterV2 doesn't create partitions when using partitionedBy
Posted by GitBox <gi...@apache.org>.
krisvaz commented on issue #4363:
URL: https://github.com/apache/iceberg/issues/4363#issuecomment-1075214036
@hililiwei
Here's an example of using the DataFrameWriterV2 to try creating the partitioned table and it's result:
```
>>> df = spark.createDataFrame([{'partition_a': 'A', 'partition_b': 'B', 'data': 'somestuff'}])
>>> df.writeTo("iceberg_catalog.p2.test").partitionedBy("partition_a", "partition_b").create()
>>> spark.sql("describe iceberg_catalog.p2.test").show()
+---------------+---------+-------+
| col_name|data_type|comment|
+---------------+---------+-------+
| data| string| |
| partition_a| string| |
| partition_b| string| |
| | | |
| # Partitioning| | |
|Not partitioned| | |
+---------------+---------+-------+
```
Here's an example of using spark sql to create the partitioned table and it's result:
```
>>> df = spark.createDataFrame([{'partition_a': 'A', 'partition_b': 'B', 'data': 'somestuff'}])
>>> df.createOrReplaceTempView('test_view')
>>> spark.sql("""
... create table iceberg_catalog.p2.test using iceberg partitioned by (partition_a, partition_b)
... as select * from test_view""")
DataFrame[]
>>> spark.sql("describe iceberg_catalog.p2.test").show()
+--------------+-----------+-------+
| col_name| data_type|comment|
+--------------+-----------+-------+
| data| string| |
| partition_a| string| |
| partition_b| string| |
| | | |
|# Partitioning| | |
| Part 0|partition_a| |
| Part 1|partition_b| |
+--------------+-----------+-------+
```
I should note that I am running this within an AWS EMR cluster that's configured as defined [here](https://iceberg.apache.org/docs/latest/aws/#spark).
Instead of starting a spark-sql shell, I'm starting a pyspark shell with those same configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] krisvaz edited a comment on issue #4363: DataFrameWriterV2 doesn't create partitions when using partitionedBy
Posted by GitBox <gi...@apache.org>.
krisvaz edited a comment on issue #4363:
URL: https://github.com/apache/iceberg/issues/4363#issuecomment-1075214036
@hililiwei
Here's an example of using the DataFrameWriterV2 to try creating the partitioned table and it's result:
```
>>> df = spark.createDataFrame([{'partition_a': 'A', 'partition_b': 'B', 'data': 'somestuff'}])
>>> df.writeTo("iceberg_catalog.p2.test").using("iceberg").partitionedBy("partition_a", "partition_b").create()
>>> spark.sql("describe iceberg_catalog.p2.test").show()
+---------------+---------+-------+
| col_name|data_type|comment|
+---------------+---------+-------+
| data| string| |
| partition_a| string| |
| partition_b| string| |
| | | |
| # Partitioning| | |
|Not partitioned| | |
+---------------+---------+-------+
```
Here's an example of using spark sql to create the partitioned table and it's result:
```
>>> df = spark.createDataFrame([{'partition_a': 'A', 'partition_b': 'B', 'data': 'somestuff'}])
>>> df.createOrReplaceTempView('test_view')
>>> spark.sql("""
... create table iceberg_catalog.p2.test using iceberg partitioned by (partition_a, partition_b)
... as select * from test_view""")
DataFrame[]
>>> spark.sql("describe iceberg_catalog.p2.test").show()
+--------------+-----------+-------+
| col_name| data_type|comment|
+--------------+-----------+-------+
| data| string| |
| partition_a| string| |
| partition_b| string| |
| | | |
|# Partitioning| | |
| Part 0|partition_a| |
| Part 1|partition_b| |
+--------------+-----------+-------+
```
I should note that I am running this within an AWS EMR cluster that's configured as defined [here](https://iceberg.apache.org/docs/latest/aws/#spark).
Instead of starting a spark-sql shell, I'm starting a pyspark shell with those same configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org