You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "a-agmon (via GitHub)" <gi...@apache.org> on 2023/10/20 07:12:38 UTC
[I] Document that distribution modes do not work with Create statements [Documentation] [iceberg]
a-agmon opened a new issue, #8887:
URL: https://github.com/apache/iceberg/issues/8887
### Apache Iceberg version
1.4.0 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
Spark writes are currently using hash as default distribution modes, however distribution modes are not handeled in create statement.
There is a PR (https://github.com/apache/spark/pull/40734) that is supposed to handle this, but until then perhaps its a good idea to document this limitation and the requirement to manually deal with distribution in CTAS statements.
So this will not work as expected:
```scala
df.writeTo(t).create()
```
This might generate the following exception:
```
java.lang.IllegalStateException: Incoming records violate the writer assumption that records are clustered by spec and by partition within each spec.
Either cluster the incoming records or switch to fanout writers.
```
Instead, distribution mode needs to be explicitly stated in create
```scala
private def executeDDL(fullTablePath: String): DataFrame = {
spark.sql(
s"""
|CREATE TABLE $fullTablePath (
| insert_timestamp TIMESTAMP,
| timestamp TIMESTAMP,
| role STRING,
| type STRING,
| file_key STRING,
| primary_dir STRING,
| secondary_dir STRING)
|USING iceberg
|PARTITIONED BY (days(timestamp))
|TBLPROPERTIES (
| 'format-version' = '2',
| 'sort-order' = 'timestamp ASC NULLS FIRST',
| 'write.distribution-mode' = 'range')
|""".stripMargin
)
spark.sql(s"ALTER TABLE $fullTablePath WRITE ORDERED BY timestamp")
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Document that distribution modes do not work with DataFrameWriterV2 API Create statements [Documentation] [iceberg]
Posted by "manuzhang (via GitHub)" <gi...@apache.org>.
manuzhang commented on issue #8887:
URL: https://github.com/apache/iceberg/issues/8887#issuecomment-1880604102
Explicit distribution modes are required in DDL for CTAS/RTAS before Spark 3.5.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Document that distribution modes do not work with DataFrameWriterV2 API Create statements [Documentation] [iceberg]
Posted by "nastra (via GitHub)" <gi...@apache.org>.
nastra closed issue #8887: Document that distribution modes do not work with DataFrameWriterV2 API Create statements [Documentation]
URL: https://github.com/apache/iceberg/issues/8887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org