You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/10/13 17:59:06 UTC
[GitHub] [iceberg] RLashofRegas opened a new issue, #5977: How to write to a bucket-partitioned table using PySpark?
RLashofRegas opened a new issue, #5977:
URL: https://github.com/apache/iceberg/issues/5977
### Query engine
Spark
### Question
In the documentation for Spark writes, under the section for [Writing to partitioned tables](https://iceberg.apache.org/docs/latest/spark-writes/#writing-to-partitioned-tables), there are two Spark Jira issues linked that explain why (1) pre-write sort is required ([SPARK-23889](https://issues.apache.org/jira/browse/SPARK-23889)), and (2) manually registering the iceberg bucket function as a UDF is required for bucket-partitioned tables ([SPARK-27658](https://issues.apache.org/jira/browse/SPARK-27658)). Both of these issues are in the resolved state.
1. Is the Iceberg documentation still valid?
2. If not, what release versions of Spark/Iceberg support writing to bucket-partitioned tables without the steps above?
3. If so, how can I write to a bucket-partitioned table with PySpark given that (as far as I can tell) the bucket function is not available as a Python API so I cannot register it as a UDF?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] robinsinghstudios commented on issue #5977: How to write to a bucket-partitioned table using PySpark?
Posted by GitBox <gi...@apache.org>.
robinsinghstudios commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1336089178
@RussellSpitzer Any way to register the bucket UDF with Pyspark? Will the built-in bucket method not work for the same?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to write to a bucket-partitioned table using PySpark? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-2297732147
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] ignaski commented on issue #5977: How to write to a bucket-partitioned table using PySpark?
Posted by "ignaski (via GitHub)" <gi...@apache.org>.
ignaski commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1715274192
Apparently there is sql catalog functions `catalog.system.bucket()` and `catalog.system.truncate()` which allows to pre-order the data before writing to partitioned table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] robinsinghstudios commented on issue #5977: How to write to a bucket-partitioned table using PySpark?
Posted by "robinsinghstudios (via GitHub)" <gi...@apache.org>.
robinsinghstudios commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1407462492
@fireking77 Merge statement works with SparkSQL. Had the same issue with append pyspark method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] robinsinghstudios commented on issue #5977: How to write to a bucket-partitioned table using PySpark?
Posted by GitBox <gi...@apache.org>.
robinsinghstudios commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1334861057
@RussellSpitzer I'm sorting the data frame globally before inserting the data but still facing the same issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Neuw84 commented on issue #5977: How to write to a bucket-partitioned table using PySpark?
Posted by GitBox <gi...@apache.org>.
Neuw84 commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1331727558
Hi,
Any hints of this? Stuck on same problem!
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] mjf-89 commented on issue #5977: How to write to a bucket-partitioned table using PySpark?
Posted by "mjf-89 (via GitHub)" <gi...@apache.org>.
mjf-89 commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1413328965
Hi, I don't have a deep understandin of pyspark internals but I think that you can write to a partitioned iceberg table with the following approach:
```
# registering iceberg udf for partition transformation bucket(32,string_column)
spark.sparkContext._jvm.org.apache.iceberg.spark.IcebergSpark.registerBucketUDF(spark._jsparkSession,'iceberg_bucket_str_32',spark.sparkContext._jvm.org.apache.spark.sql.types.DataTypes.StringType,32)
# sorting using the registerd udf and write to the partitioned iceberg table
df.sortWithinPartitions(F.expr("iceberg_bucket_str_32(string_column)")) \
.writeTo("iceberg_table") \
.using("iceberg") \
.partitionedBy(F.bucket(32,"string_column"))
```
Another option as far as I understand is to avoid sorting and use the fanout writer instead:
```
df.writeTo("iceberg_table") \
.using('iceberg') \
.option("fanout-enabled", "true") \
.partitionedBy(F.bucket(32,"esum_id")) \
.createOrReplace()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] robinsinghstudios commented on issue #5977: How to write to a bucket-partitioned table using PySpark?
Posted by GitBox <gi...@apache.org>.
robinsinghstudios commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1333788692
Hi,
My data is being appended to the partitioned table without registering any UDFs but the data seems to be written as one row per file which is creating a huge performance impact. Any suggestions or fixes will be really appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] fireking77 commented on issue #5977: How to write to a bucket-partitioned table using PySpark?
Posted by "fireking77 (via GitHub)" <gi...@apache.org>.
fireking77 commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1407435198
Hi Guys!
I would also curious about this:
Any way to register the bucket UDF with Pyspark? Will the built-in bucket method not work for the same?
@robinsinghstudios have you figured out something? - are you able to write partitioned table with pyspark?
Thanks in advance,
Darvi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to write to a bucket-partitioned table using PySpark? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #5977: How to write to a bucket-partitioned table using PySpark?
URL: https://github.com/apache/iceberg/issues/5977
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] fireking77 commented on issue #5977: How to write to a bucket-partitioned table using PySpark?
Posted by "fireking77 (via GitHub)" <gi...@apache.org>.
fireking77 commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1407466795
@robinsinghstudios Thanks! :D you were right :D
I hope there will be someone fix insert too... ;)
Thanks again!
Darvi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Gatsby-Lee commented on issue #5977: How to write to a bucket-partitioned table using PySpark?
Posted by "Gatsby-Lee (via GitHub)" <gi...@apache.org>.
Gatsby-Lee commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1603678602
thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to write to a bucket-partitioned table using PySpark? [iceberg]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-2342374407
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] How to write to a bucket-partitioned table using PySpark? [iceberg]
Posted by "pnain (via GitHub)" <gi...@apache.org>.
pnain commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1911074967
df.writeTo("iceberg_table") \
.using('iceberg') \
.option("fanout-enabled", "true") \
.partitionedBy(F.bucket(32,"esum_id")) \
.createOrReplace()
-- This works well but create small files. Hash/Range options are not working despite using the sorting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] robinsinghstudios commented on issue #5977: How to write to a bucket-partitioned table using PySpark?
Posted by "robinsinghstudios (via GitHub)" <gi...@apache.org>.
robinsinghstudios commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1407438732
Hi @fireking77 , I was able to create bucketed tables by using SparkSQL query without any data. It just works for some reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #5977: How to write to a bucket-partitioned table using PySpark?
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1335440185
If you aren’t sorting on the partition transform the rows will be in random order with regards to that transform. So if you don’t have the UDF registered and you are just sorting on the column itself I would expect you to generate at least 1 file per bucket per Spark taskOn Dec 2, 2022, at 1:36 AM, robinsinghstudios ***@***.***> wrote:
@RussellSpitzer I'm sorting the data frame globally before inserting the data but still facing the same issue.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] fireking77 commented on issue #5977: How to write to a bucket-partitioned table using PySpark?
Posted by "fireking77 (via GitHub)" <gi...@apache.org>.
fireking77 commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1407454746
@robinsinghstudios that is OK, but were youable to write data into that table with pyspark?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on issue #5977: How to write to a bucket-partitioned table using PySpark?
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on issue #5977:
URL: https://github.com/apache/iceberg/issues/5977#issuecomment-1333976156
@robinsinghstudios data needs to be globally sorted on the bucketing function, at least that is my guess with those symptoms. Luckily this should all be less of an issue once the function catalog things in Spark 3.3 are all sorted out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org