You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/07/29 03:49:04 UTC
[GitHub] [iceberg] Cqz666 opened a new issue, #5384: Exception occurred while writing to Iceberg tables by 'INSERT OVERWRITE'
Cqz666 opened a new issue, #5384:
URL: https://github.com/apache/iceberg/issues/5384
I used ’ INSERT OVERWRITE’ to writing to Iceberg tables, but Spark ran with the following exception:
`Caused by: java.lang.IllegalStateException: Incoming records violate the writer assumption that records are clustered by spec and by partition within each spec. Either cluster the incoming records or switch to fanout writers.
Encountered records that belong to already closed files:
partition 'datekey=20220727/event=search_show' in spec [
1000: datekey: identity(1)
1001: event: identity(2)
]
at org.apache.iceberg.io.ClusteredWriter.write(ClusteredWriter.java:95)
at org.apache.iceberg.io.ClusteredDataWriter.write(ClusteredDataWriter.java:34)
at org.apache.iceberg.spark.source.SparkWrite$PartitionedDataWriter.write(SparkWrite.java:641)
at org.apache.iceberg.spark.source.SparkWrite$PartitionedDataWriter.write(SparkWrite.java:616)
at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$7(WriteToDataSourceV2Exec.scala:441)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1411)
at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:477)
at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:385)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
`
The code looks something like this:
` spark.sql(
s"""
| INSERT OVERWRITE
| `$icebergCatalog`.`$icebergDatabase`.`$icebergTable`
| SELECT $selectSql
| FROM $hiveDatabase.$topic
| WHERE datekey>=$lowerDate AND datekey<=$upperDate AND event in ($eventStr)
|""".stripMargin
)`
The data source is a partition table of Hive.
How do I correctly write iceberg partition tables by `INSERT OVERWRITE` or something else?
Iceberg version: 0.13.2
spark version: 3.0.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Cqz666 closed issue #5384: Exception occurred while writing to Iceberg tables by 'INSERT OVERWRITE'
Posted by GitBox <gi...@apache.org>.
Cqz666 closed issue #5384: Exception occurred while writing to Iceberg tables by 'INSERT OVERWRITE'
URL: https://github.com/apache/iceberg/issues/5384
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Exception occurred while writing to Iceberg tables by 'INSERT OVERWRITE' [iceberg]
Posted by "sanromeo (via GitHub)" <gi...@apache.org>.
sanromeo commented on issue #5384:
URL: https://github.com/apache/iceberg/issues/5384#issuecomment-1777461016
Have same issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Exception occurred while writing to Iceberg tables by 'INSERT OVERWRITE' [iceberg]
Posted by "rokity (via GitHub)" <gi...@apache.org>.
rokity commented on issue #5384:
URL: https://github.com/apache/iceberg/issues/5384#issuecomment-1923639270
@sanromeo @MoveLiu Before to create the tempview in pyspark we must to sort the dataframe by partitions specified on sql query.
```python
df = df.repartition(*prt_cols).sortWithinPartitions(*prt_cols)
df.createOrReplaceTempView(tmp_name)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Cqz666 commented on issue #5384: Exception occurred while writing to Iceberg tables by 'INSERT OVERWRITE'
Posted by GitBox <gi...@apache.org>.
Cqz666 commented on issue #5384:
URL: https://github.com/apache/iceberg/issues/5384#issuecomment-1200564145
Thanks, It works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] MoveLiu commented on issue #5384: Exception occurred while writing to Iceberg tables by 'INSERT OVERWRITE'
Posted by "MoveLiu (via GitHub)" <gi...@apache.org>.
MoveLiu commented on issue #5384:
URL: https://github.com/apache/iceberg/issues/5384#issuecomment-1730635461
I have the same problem, how can I solve it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] amogh-jahagirdar commented on issue #5384: Exception occurred while writing to Iceberg tables by 'INSERT OVERWRITE'
Posted by GitBox <gi...@apache.org>.
amogh-jahagirdar commented on issue #5384:
URL: https://github.com/apache/iceberg/issues/5384#issuecomment-1198876892
An explicit sort is required based on the partition spec per task in Spark before writing to a partitioned table (or for streaming use cases, using the fanout writer is common, but that doesn't look to apply here). I think the example here https://iceberg.apache.org/docs/latest/spark-writes/#writing-to-partitioned-tables should help, but please update the issue if the docs aren't clear!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org