You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/08/11 09:39:05 UTC
[GitHub] [iceberg] zephaniah-wzf opened a new issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
zephaniah-wzf opened a new issue #2958:
URL: https://github.com/apache/iceberg/issues/2958
环境: spark3.0 iceberg0.11.0 spark.sql.catalog.catalog-name.type = hadoop
[报错信息.txt](https://github.com/apache/iceberg/files/6967307/default.txt)
使用api时创建多个分区字段的表后，向里插入数据也会报错。
个人觉得：是锁导致的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zephaniah-wzf removed a comment on issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
zephaniah-wzf removed a comment on issue #2958:
URL: https://github.com/apache/iceberg/issues/2958#issuecomment-898217688
> @zephaniah-wzf 可以发一下具体的堆栈吗
2021-08-13 11:51:25,636 WARN iceberg.BaseTransaction: Failed to load metadata for a committed snapshot, skipping clean-up
Time taken: 0.116 seconds
2021-08-13 11:51:25,645 INFO thriftserver.SparkSQLCLIDriver: Time taken: 0.116 seconds
就有这两条信息 是不是我哪里的配置配错了 尴尬
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zephaniah-wzf commented on issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
zephaniah-wzf commented on issue #2958:
URL: https://github.com/apache/iceberg/issues/2958#issuecomment-898217688
> @zephaniah-wzf 可以发一下具体的堆栈吗
2021-08-13 11:51:25,636 WARN iceberg.BaseTransaction: Failed to load metadata for a committed snapshot, skipping clean-up
Time taken: 0.116 seconds
2021-08-13 11:51:25,645 INFO thriftserver.SparkSQLCLIDriver: Time taken: 0.116 seconds
就有这两条信息 是不是我哪里的配置配错了 尴尬
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yanqinghe commented on issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
yanqinghe commented on issue #2958:
URL: https://github.com/apache/iceberg/issues/2958#issuecomment-898868062
> > 如果你使用的是Spark Structured Streaming的话，可以在写入的时候指定fanout-enabled配置：
> > ```scala
> > .writeStream
> > .format("iceberg")
> > .options(Map(
> > "fanout-enabled" -> "true"
> > ))
> > .outputMode(OutputMode.Append())
> > .trigger(Trigger.ProcessingTime(5, TimeUnit.MINUTES))
> > .start()
> > .awaitTermination()
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > > > @zephaniah-wzf 可以发一下具体的堆栈吗
> > >
> > >
> > > [报错信息.txt](https://github.com/apache/iceberg/files/6980457/default.txt)
> > > 还是会报那样的错误，有点不合理啊
>
> 我使用的是spark-sql客户端写的
如果使用Spark-sql客户端的话应该是批量任务，你可以尝试对你的分区列进行order by排序，这样能够避免在写入的过程中在同一个分区内出现多个文件的并发操作。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zephaniah-wzf commented on issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
zephaniah-wzf commented on issue #2958:
URL: https://github.com/apache/iceberg/issues/2958#issuecomment-898172671
> 我已经遇到了这个问题，这是我的堆栈：
>
> ```
> 2021-05-14,12:52:40,185 WARN org.apache.iceberg.io.PartitionedWriter: Duplicate key: [11] == [11]
> 2021-05-14,12:52:40,185 ERROR org.apache.spark.util.Utils: Aborting task
> java.lang.IllegalStateException: Already closed files for partition: id_bucket=11
> ```
>
> 我用set解决 `alter table xxx set TBLPROPERTIES ('write.spark.fanout.enabled'='true');`
我用这个方式设置会报failed to load metadata for a committed snapshot,skipping clean-up 这个问题。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zephaniah-wzf removed a comment on issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
zephaniah-wzf removed a comment on issue #2958:
URL: https://github.com/apache/iceberg/issues/2958#issuecomment-898172671
> 我已经遇到了这个问题，这是我的堆栈：
>
> ```
> 2021-05-14,12:52:40,185 WARN org.apache.iceberg.io.PartitionedWriter: Duplicate key: [11] == [11]
> 2021-05-14,12:52:40,185 ERROR org.apache.spark.util.Utils: Aborting task
> java.lang.IllegalStateException: Already closed files for partition: id_bucket=11
> ```
>
> 我用set解决 `alter table xxx set TBLPROPERTIES ('write.spark.fanout.enabled'='true');`
我用这个方式设置会报failed to load metadata for a committed snapshot,skipping clean-up 这个问题。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zephaniah-wzf closed issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
zephaniah-wzf closed issue #2958:
URL: https://github.com/apache/iceberg/issues/2958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zephaniah-wzf commented on issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
zephaniah-wzf commented on issue #2958:
URL: https://github.com/apache/iceberg/issues/2958#issuecomment-898234171
> @zephaniah-wzf 可以发一下具体的堆栈吗
[报错信息.txt](https://github.com/apache/iceberg/files/6980457/default.txt)
还是会报那样的错误，有点不合理啊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zephaniah-wzf commented on issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
zephaniah-wzf commented on issue #2958:
URL: https://github.com/apache/iceberg/issues/2958#issuecomment-898860358
> 如果你使用的是Spark Structured Streaming的话，可以在写入的时候指定fanout-enabled配置：
>
> ```scala
> .writeStream
> .format("iceberg")
> .options(Map(
> "fanout-enabled" -> "true"
> ))
> .outputMode(OutputMode.Append())
> .trigger(Trigger.ProcessingTime(5, TimeUnit.MINUTES))
> .start()
> .awaitTermination()
> ```
>
> > > @zephaniah-wzf 可以发一下具体的堆栈吗
> >
> >
> > [报错信息.txt](https://github.com/apache/iceberg/files/6980457/default.txt)
> > 还是会报那样的错误，有点不合理啊
我使用的是spark-sql客户端写的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on issue #2958:
URL: https://github.com/apache/iceberg/issues/2958#issuecomment-897972554
```
Caused by: java.lang.IllegalStateException: Already closed files for partition: id_bucket=3
at org.apache.iceberg.io.PartitionedWriter.write(PartitionedWriter.java:69)
at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$7(WriteToDataSourceV2Exec.scala:441)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1411)
at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:477)
at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:385)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:127)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
看你上传的报错，@coolderli 的方法应该是对的，需要使用fanout writer。因为在使用bucket函数的情况下通常你无法对数据进行提前预判排序，所以不使用fanout writer必然会出现这个问题。理论上你也可以提前计算出每一行是哪个bucket然后以此进行排序再写入，这样就不需要fanout，但是就比较麻烦。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] coolderli commented on issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
coolderli commented on issue #2958:
URL: https://github.com/apache/iceberg/issues/2958#issuecomment-897527833
I have already meet this problem, here is my stack:
```
2021-05-14,12:52:40,185 WARN org.apache.iceberg.io.PartitionedWriter: Duplicate key: [11] == [11]
2021-05-14,12:52:40,185 ERROR org.apache.spark.util.Utils: Aborting task
java.lang.IllegalStateException: Already closed files for partition: id_bucket=11
```
I solve it by set `alter table xxx set TBLPROPERTIES ('write.spark.fanout.enabled'='true');`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jackye1995 commented on issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
jackye1995 commented on issue #2958:
URL: https://github.com/apache/iceberg/issues/2958#issuecomment-898174231
@zephaniah-wzf 可以发一下具体的堆栈吗
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yanqinghe edited a comment on issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
yanqinghe edited a comment on issue #2958:
URL: https://github.com/apache/iceberg/issues/2958#issuecomment-898849650
如果你使用的是Spark Structured Streaming的话，可以在写入的时候指定fanout-enabled配置：
``` scala
.writeStream
.format("iceberg")
.options(Map(
"fanout-enabled" -> "true"
))
.outputMode(OutputMode.Append())
.trigger(Trigger.ProcessingTime(5, TimeUnit.MINUTES))
.start()
.awaitTermination()
```
> > @zephaniah-wzf 可以发一下具体的堆栈吗
>
> [报错信息.txt](https://github.com/apache/iceberg/files/6980457/default.txt)
> 还是会报那样的错误，有点不合理啊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zephaniah-wzf removed a comment on issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
zephaniah-wzf removed a comment on issue #2958:
URL: https://github.com/apache/iceberg/issues/2958#issuecomment-896700737
环境: spark3.0 iceberg0.11.0 spark.sql.catalog.catalog-name.type = hadoop
报错信息.txt
使用api时创建多个分区字段的表后，向里插入数据也会报错。
个人觉得：是锁导致的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] zephaniah-wzf commented on issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
zephaniah-wzf commented on issue #2958:
URL: https://github.com/apache/iceberg/issues/2958#issuecomment-896700737
环境: spark3.0 iceberg0.11.0 spark.sql.catalog.catalog-name.type = hadoop
报错信息.txt
使用api时创建多个分区字段的表后，向里插入数据也会报错。
个人觉得：是锁导致的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] yanqinghe commented on issue #2958: 使用bucket函数创建表后，向里面批量导入数据会报错
Posted by GitBox <gi...@apache.org>.
yanqinghe commented on issue #2958:
URL: https://github.com/apache/iceberg/issues/2958#issuecomment-898849650
如果你使用的是Spark Structured Streaming的话，可以在写入的时候指定配置：
``` scala
.writeStream
.format("iceberg")
.options(Map(
"fanout-enabled" -> "true"
))
.outputMode(OutputMode.Append())
.trigger(Trigger.ProcessingTime(5, TimeUnit.MINUTES))
.start()
.awaitTermination()
```
> > @zephaniah-wzf 可以发一下具体的堆栈吗
>
> [报错信息.txt](https://github.com/apache/iceberg/files/6980457/default.txt)
> 还是会报那样的错误，有点不合理啊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org