You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "hugw777 (via GitHub)" <gi...@apache.org> on 2025/03/24 02:19:19 UTC
[I] [Bug] When using insert overwrite, a StackOverflowError exception is thrown [paimon]
hugw777 opened a new issue, #5331:
URL: https://github.com/apache/paimon/issues/5331
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
paimon0.9.0
### Compute Engine
spark3.3.0
hive3.1.2
### Minimal reproduce step
for example
a double partition table
CREATE TABLE IF NOT EXISTS paimon.${dbname}.${tablename}(
column1 string COMMENT 'column1',
column2 string COMMENT 'column2',
column3 string COMMENT 'column3',
column4 string COMMENT 'column4',
p_col_1 string COMMENT 'p_col_1',
p_col_2 string COMMENT 'p_col_2'
) PARTITIONED BY (p_col_1,p_col_2)
COMMENT ''
TBLPROPERTIES (
'primary-key' = 'p_col_1,p_col_2,column1,column2,column3',
'bucket-key' = 'column1,column2,column3',
'bucket' = '1',
'file.format' = 'parquet',
'deletion-vectors.enabled' = 'true',
'metastore.partitioned-table' = 'true',
'scan.mode' = 'compacted-full',
'compaction.optimization-interval' = '3600000',
'target-file-size' = '256mb',
'sink.parallelism' = '20',
'num-sorted-run.stop-trigger' = '2147483647',
'sort-spill-threshold' = '10',
'merge-engine' = 'deduplicate'
);
insert overwrite table paimon.${dbname}.${tablename} partition(cdate,p_col_2)
select
t1.column1
,t1.column2
,t1.column3
,t2.column4
,t2.column5
,t1.cdate
,t1.p_col_2
from (
select cdate,p_col_2,column1,column2,column3
from hive_db.hive_table1
where cdate between '${start_day}' and '${end_day}'
)t1 join(
select cdate,p_col_2,column1,column2,column3,column4,column5
from hive_db.hive_table2
where cdate between '${start_day}' and '${end_day}'
)t2 on
t1.cdate = t2.cdate and t1.p_col_2 = t2.p_col_2 and t1.column1 = t2.column1 and t1.column2 = t2.column2
and t1.column3 = t2.column3;
hive_db.hive_table1:unique key is cdate,p_col_2,column1,column2,column3
hive_db.hive_table2:unique key is cdate,p_col_2,column1,column2,column3,column4
The estimated number of partitions written is around 4500.
### What doesn't meet your expectations?
When the last stage is written, an error is reported
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
at org.apache.paimon.spark.commands.PaimonSparkWriter.commit(PaimonSparkWriter.scala:285)
at org.apache.paimon.spark.commands.WriteIntoPaimonTable.run(WriteIntoPaimonTable.scala:64)
at org.apache.paimon.spark.commands.PaimonDynamicPartitionOverwriteCommand.run(PaimonDynamicPartitionOverwriteCommand.scala:69)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:220)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:384)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:504)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:498)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:498)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:207)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When using insert overwrite, a StackOverflowError exception is thrown [paimon]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on issue #5331:
URL: https://github.com/apache/paimon/issues/5331#issuecomment-2760334446
Can you try the latest 1.0.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When using insert overwrite, a StackOverflowError exception is thrown [paimon]
Posted by "hugw777 (via GitHub)" <gi...@apache.org>.
hugw777 commented on issue #5331:
URL: https://github.com/apache/paimon/issues/5331#issuecomment-2747476484
spark sql conf
```sql
set spark.sql.optimizer.dynamicPartitionPruning.enabled=false;
```
If I don't set the value of the spark.sql.optimizer.dynamicPartitionPruning.enabled parameter to false, I get Error in query: unresolved operator 'Filter dynamicpruning950'
If I set the value of the spark.sql.optimizer.dynamicPartitionPruning.enabled parameter to false, throw java.lang.StackOverflowError.
```
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
at org.apache.paimon.spark.commands.PaimonSparkWriter.commit(PaimonSparkWriter.scala:285)
at org.apache.paimon.spark.commands.WriteIntoPaimonTable.run(WriteIntoPaimonTable.scala:64)
at org.apache.paimon.spark.commands.PaimonDynamicPartitionOverwriteCommand.run(PaimonDynamicPartitionOverwriteCommand.scala:69)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:220)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:384)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:504)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:498)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:498)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:207)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
at org.apache.paimon.utils.ScanParallelExecutor$1.advanceIfNeeded(ScanParallelExecutor.java:85)
at org.apache.paimon.utils.ScanParallelExecutor$1.hasNext(ScanParallelExecutor.java:60)
at org.apache.paimon.manifest.FileEntry.mergeEntries(FileEntry.java:126)
at org.apache.paimon.manifest.FileEntry.mergeEntries(FileEntry.java:112)
at org.apache.paimon.operation.AbstractFileStoreScan.readAndMergeFileEntries(AbstractFileStoreScan.java:395)
at org.apache.paimon.operation.AbstractFileStoreScan.doPlan(AbstractFileStoreScan.java:299)
at org.apache.paimon.operation.AbstractFileStoreScan.plan(AbstractFileStoreScan.java:223)
at org.apache.paimon.operation.FileStoreCommitImpl.tryOverwrite(FileStoreCommitImpl.java:729)
at org.apache.paimon.operation.FileStoreCommitImpl.overwrite(FileStoreCommitImpl.java:451)
at org.apache.paimon.table.sink.TableCommitImpl.commitMultiple(TableCommitImpl.java:225)
at org.apache.paimon.table.sink.TableCommitImpl.commit(TableCommitImpl.java:200)
at org.apache.paimon.table.sink.TableCommitImpl.commit(TableCommitImpl.java:179)
at org.apache.paimon.table.sink.TableCommitImpl.commit(TableCommitImpl.java:163)
at org.apache.paimon.spark.commands.PaimonSparkWriter.commit(PaimonSparkWriter.scala:283)
... 59 more
Caused by: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.paimon.utils.ScanParallelExecutor$1.advanceIfNeeded(ScanParallelExecutor.java:83)
... 72 more
Caused by: java.lang.StackOverflowError
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.apache.paimon.operation.AbstractFileStoreScan.lambda$readAndMergeFileEntries$6(AbstractFileStoreScan.java:386)
at org.apache.paimon.utils.ScanParallelExecutor$1.lambda$advanceIfNeeded$0(ScanParallelExecutor.java:81)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
Caused by: java.lang.StackOverflowError
at org.apache.paimon.predicate.Or.test(Or.java:54)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When using insert overwrite, a StackOverflowError exception is thrown [paimon]
Posted by "hugw777 (via GitHub)" <gi...@apache.org>.
hugw777 commented on issue #5331:
URL: https://github.com/apache/paimon/issues/5331#issuecomment-2747464989
spark sql conf
```sql
set spark.sql.optimizer.dynamicPartitionPruning.enabled=false;
```
If I don't set the value of the spark.sql.optimizer.dynamicPartitionPruning.enabled parameter to false, I get Error in query: unresolved operator 'Filter dynamicpruning950'
If I set the value of the spark.sql.optimizer.dynamicPartitionPruning.enabled parameter to false, throw java.lang.StackOverflowError.
```
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
at org.apache.paimon.spark.commands.PaimonSparkWriter.commit(PaimonSparkWriter.scala:285)
at org.apache.paimon.spark.commands.WriteIntoPaimonTable.run(WriteIntoPaimonTable.scala:64)
at org.apache.paimon.spark.commands.PaimonDynamicPartitionOverwriteCommand.run(PaimonDynamicPartitionOverwriteCommand.scala:69)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:109)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:169)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:95)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:176)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:584)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:560)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)
at org.apache.spark.sql.Dataset.<init>(Dataset.scala:220)
at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:100)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:97)
at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:622)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:617)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:651)
at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:67)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:384)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1(SparkSQLCLIDriver.scala:504)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.$anonfun$processLine$1$adapted(SparkSQLCLIDriver.scala:498)
at scala.collection.Iterator.foreach(Iterator.scala:943)
at scala.collection.Iterator.foreach$(Iterator.scala:943)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processLine(SparkSQLCLIDriver.scala:498)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:207)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
at org.apache.paimon.utils.ScanParallelExecutor$1.advanceIfNeeded(ScanParallelExecutor.java:85)
at org.apache.paimon.utils.ScanParallelExecutor$1.hasNext(ScanParallelExecutor.java:60)
at org.apache.paimon.manifest.FileEntry.mergeEntries(FileEntry.java:126)
at org.apache.paimon.manifest.FileEntry.mergeEntries(FileEntry.java:112)
at org.apache.paimon.operation.AbstractFileStoreScan.readAndMergeFileEntries(AbstractFileStoreScan.java:395)
at org.apache.paimon.operation.AbstractFileStoreScan.doPlan(AbstractFileStoreScan.java:299)
at org.apache.paimon.operation.AbstractFileStoreScan.plan(AbstractFileStoreScan.java:223)
at org.apache.paimon.operation.FileStoreCommitImpl.tryOverwrite(FileStoreCommitImpl.java:729)
at org.apache.paimon.operation.FileStoreCommitImpl.overwrite(FileStoreCommitImpl.java:451)
at org.apache.paimon.table.sink.TableCommitImpl.commitMultiple(TableCommitImpl.java:225)
at org.apache.paimon.table.sink.TableCommitImpl.commit(TableCommitImpl.java:200)
at org.apache.paimon.table.sink.TableCommitImpl.commit(TableCommitImpl.java:179)
at org.apache.paimon.table.sink.TableCommitImpl.commit(TableCommitImpl.java:163)
at org.apache.paimon.spark.commands.PaimonSparkWriter.commit(PaimonSparkWriter.scala:283)
... 59 more
Caused by: java.util.concurrent.ExecutionException: java.lang.StackOverflowError
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at org.apache.paimon.utils.ScanParallelExecutor$1.advanceIfNeeded(ScanParallelExecutor.java:83)
... 72 more
Caused by: java.lang.StackOverflowError
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.apache.paimon.operation.AbstractFileStoreScan.lambda$readAndMergeFileEntries$6(AbstractFileStoreScan.java:386)
at org.apache.paimon.utils.ScanParallelExecutor$1.lambda$advanceIfNeeded$0(ScanParallelExecutor.java:81)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)
Caused by: java.lang.StackOverflowError
at org.apache.paimon.predicate.Or.test(Or.java:54)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
at org.apache.paimon.predicate.CompoundPredicate.test(CompoundPredicate.java:59)
at org.apache.paimon.predicate.Or.test(Or.java:55)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When using insert overwrite, a StackOverflowError exception is thrown [paimon]
Posted by "hugw777 (via GitHub)" <gi...@apache.org>.
hugw777 closed issue #5331: [Bug] When using insert overwrite, a StackOverflowError exception is thrown
URL: https://github.com/apache/paimon/issues/5331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When using insert overwrite, a StackOverflowError exception is thrown [paimon]
Posted by "hugw777 (via GitHub)" <gi...@apache.org>.
hugw777 closed issue #5331: [Bug] When using insert overwrite, a StackOverflowError exception is thrown
URL: https://github.com/apache/paimon/issues/5331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] When using insert overwrite, a StackOverflowError exception is thrown [paimon]
Posted by "hugw777 (via GitHub)" <gi...@apache.org>.
hugw777 commented on issue #5331:
URL: https://github.com/apache/paimon/issues/5331#issuecomment-2760357657
> Can you try the latest 1.0.1
I've used 1.0.1
This problem seems to be brought about by Spark 3
When I set the following three parameters, I didn't encounter any problems
set spark.sql.optimizer.dynamicPartitionPruning.enabled=false;
set spark.sql.optimizer.nestedSchemaPruning.enabled=false;
set spark.sql.hive.convertMetastoreParquet=false;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org