You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "CodyPin (via GitHub)" <gi...@apache.org> on 2024/07/09 06:46:02 UTC
[I] [Bug] Trino error with java.lang.IllegalStateException: undefined when joining table [paimon]
CodyPin opened a new issue, #3712:
URL: https://github.com/apache/paimon/issues/3712
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
0.8.1
### Compute Engine
Trino 433
### Minimal reproduce step
Not sure, more detail below
### What doesn't meet your expectations?
### Some backgroud
The database we are getting data from is done with Flink and Paimon CDC, the CDC process is ongoing but has no problem.
The original query that I wanted to run is:
```
select distinct
opo.order_number,
odz.delivery_zone_code,
opo.order_creation_date,
opo.customer_pk,
osw.expected_delivery_date,
ost.log_new_status,
ost.log_date,
ost.log_status_type
from "oix_status_track" AS ost
LEFT join "oix_shipment_waybill" AS osw on ost.shipment_waybill_id = osw.shipment_waybill_id
join "oix_delivery_zone" as odz on odz.delivery_zone_id = osw.delivery_zone_id
join "oix_order_item_line" as oil on oil.shipment_waybill_id = osw.shipment_waybill_id
join "oix_sub_order" as oso on oso.sub_order_id = osw.sub_order_id
join "oix_parent_order" as opo on opo.order_id = oso.order_id and opo.order_number like 'H%'
```
It returned this error (and any mention of errors from here on is referring to this error):
```
java.lang.IllegalStateException: undefined
at io.trino.hadoop.$internal.com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:118)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:162)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at jdk.proxy9/jdk.proxy9.$Proxy443.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1665)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1582)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1579)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1594)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:454)
at io.trino.filesystem.hdfs.HdfsFileSystem.lambda$directoryExists$6(HdfsFileSystem.java:271)
at io.trino.hdfs.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:25)
at io.trino.hdfs.HdfsEnvironment.doAs(HdfsEnvironment.java:125)
at io.trino.filesystem.hdfs.HdfsFileSystem.directoryExists(HdfsFileSystem.java:257)
at io.trino.filesystem.manager.SwitchingFileSystem.directoryExists(SwitchingFileSystem.java:117)
at io.trino.filesystem.tracing.TracingFileSystem.lambda$directoryExists$5(TracingFileSystem.java:119)
at io.trino.filesystem.tracing.Tracing.withTracing(Tracing.java:47)
at io.trino.filesystem.tracing.TracingFileSystem.directoryExists(TracingFileSystem.java:119)
at org.apache.paimon.trino.fileio.TrinoFileIO.exists(TrinoFileIO.java:122)
at org.apache.paimon.utils.FileUtils.listVersionedFileStatus(FileUtils.java:104)
at org.apache.paimon.utils.FileUtils.listOriginalVersionedFiles(FileUtils.java:91)
at org.apache.paimon.utils.FileUtils.listVersionedFiles(FileUtils.java:81)
at org.apache.paimon.schema.SchemaManager.latest(SchemaManager.java:105)
at org.apache.paimon.schema.SchemaManager.latest(SchemaManager.java:96)
at org.apache.paimon.catalog.FileSystemCatalog.getDataTableSchema(FileSystemCatalog.java:128)
at org.apache.paimon.catalog.AbstractCatalog.getDataTable(AbstractCatalog.java:343)
at org.apache.paimon.catalog.AbstractCatalog.getTable(AbstractCatalog.java:338)
at org.apache.paimon.trino.catalog.TrinoCatalog.getTable(TrinoCatalog.java:157)
at org.apache.paimon.trino.TrinoTableHandle.table(TrinoTableHandle.java:137)
at org.apache.paimon.trino.TrinoTableHandle.tableWithDynamicOptions(TrinoTableHandle.java:115)
at org.apache.paimon.trino.TrinoPageSourceProvider.createPageSource(TrinoPageSourceProvider.java:107)
at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:48)
at io.trino.split.PageSourceManager.createPageSource(PageSourceManager.java:61)
at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:263)
at io.trino.operator.Driver.processInternal(Driver.java:395)
at io.trino.operator.Driver.lambda$process$8(Driver.java:298)
at io.trino.operator.Driver.tryWithLock(Driver.java:694)
at io.trino.operator.Driver.process(Driver.java:290)
at io.trino.operator.Driver.processForDuration(Driver.java:261)
at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:887)
at io.trino.execution.executor.timesharing.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:187)
at io.trino.execution.executor.timesharing.TimeSharingTaskExecutor$TaskRunner.run(TimeSharingTaskExecutor.java:565)
at io.trino.$gen.Trino_433____20240702_104434_2.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
```
Sometimes it would be this error:
```
java.lang.RuntimeException: java.lang.IllegalStateException
at org.apache.paimon.trino.TrinoPageSourceProvider.createPageSource(TrinoPageSourceProvider.java:235)
at org.apache.paimon.trino.TrinoPageSourceProvider.lambda$createPageSource$0(TrinoPageSourceProvider.java:110)
at org.apache.paimon.trino.ClassLoaderUtils.runWithContextClassLoader(ClassLoaderUtils.java:30)
at org.apache.paimon.trino.TrinoPageSourceProvider.createPageSource(TrinoPageSourceProvider.java:108)
at io.trino.plugin.base.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:48)
at io.trino.split.PageSourceManager.createPageSource(PageSourceManager.java:61)
at io.trino.operator.TableScanOperator.getOutput(TableScanOperator.java:263)
at io.trino.operator.Driver.processInternal(Driver.java:395)
at io.trino.operator.Driver.lambda$process$8(Driver.java:298)
at io.trino.operator.Driver.tryWithLock(Driver.java:694)
at io.trino.operator.Driver.process(Driver.java:290)
at io.trino.operator.Driver.processForDuration(Driver.java:261)
at io.trino.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:887)
at io.trino.execution.executor.timesharing.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:187)
at io.trino.execution.executor.timesharing.TimeSharingTaskExecutor$TaskRunner.run(TimeSharingTaskExecutor.java:565)
at io.trino.$gen.Trino_433____20240702_104434_2.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalStateException: undefined
at io.trino.hadoop.$internal.com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.hadoop.ipc.Client.setCallIdAndRetryCount(Client.java:118)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:162)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at jdk.proxy9/jdk.proxy9.$Proxy443.getBlockLocations(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:863)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:852)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:841)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1009)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:325)
at org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:321)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:333)
at io.trino.hdfs.TrinoFileSystemCache$FileSystemWrapper.open(TrinoFileSystemCache.java:341)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:906)
at io.trino.filesystem.hdfs.HdfsInputFile.lambda$openFile$1(HdfsInputFile.java:119)
at io.trino.hdfs.authentication.NoHdfsAuthentication.doAs(NoHdfsAuthentication.java:25)
at io.trino.hdfs.HdfsEnvironment.doAs(HdfsEnvironment.java:125)
at io.trino.filesystem.hdfs.HdfsInputFile.openFile(HdfsInputFile.java:117)
at io.trino.filesystem.hdfs.HdfsInputFile.newStream(HdfsInputFile.java:72)
at io.trino.filesystem.tracing.Tracing.withTracing(Tracing.java:47)
at io.trino.filesystem.tracing.TracingInputFile.newStream(TracingInputFile.java:64)
at org.apache.paimon.trino.fileio.TrinoFileIO.newInputStream(TrinoFileIO.java:64)
at org.apache.paimon.fs.FileIO.readFileUtf8(FileIO.java:210)
at org.apache.paimon.schema.SchemaManager.schema(SchemaManager.java:486)
at org.apache.paimon.io.KeyValueFileReaderFactory.lambda$createRecordReader$1(KeyValueFileReaderFactory.java:122)
at org.apache.paimon.io.KeyValueFileReaderFactory.lambda$createRecordReader$2(KeyValueFileReaderFactory.java:128)
at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1228)
at org.apache.paimon.io.KeyValueFileReaderFactory.createRecordReader(KeyValueFileReaderFactory.java:126)
at org.apache.paimon.io.KeyValueFileReaderFactory.createRecordReader(KeyValueFileReaderFactory.java:104)
at org.apache.paimon.mergetree.MergeTreeReaders.lambda$readerForRun$2(MergeTreeReaders.java:88)
at org.apache.paimon.mergetree.compact.ConcatRecordReader.create(ConcatRecordReader.java:51)
at org.apache.paimon.mergetree.MergeTreeReaders.readerForRun(MergeTreeReaders.java:94)
at org.apache.paimon.mergetree.MergeTreeReaders.lambda$readerForSection$1(MergeTreeReaders.java:76)
at org.apache.paimon.mergetree.MergeSorter.mergeSort(MergeSorter.java:120)
at org.apache.paimon.mergetree.MergeTreeReaders.readerForSection(MergeTreeReaders.java:78)
at org.apache.paimon.operation.MergeFileSplitRead.lambda$mergeRead$5(MergeFileSplitRead.java:303)
at org.apache.paimon.mergetree.compact.ConcatRecordReader.readBatch(ConcatRecordReader.java:71)
at org.apache.paimon.mergetree.DropDeleteReader.readBatch(DropDeleteReader.java:44)
at org.apache.paimon.reader.RecordReader$1.readBatch(RecordReader.java:194)
at org.apache.paimon.table.source.KeyValueTableRead$1.readBatch(KeyValueTableRead.java:130)
at org.apache.paimon.reader.RecordReaderIterator.<init>(RecordReaderIterator.java:36)
at org.apache.paimon.reader.RecordReader.toCloseableIterator(RecordReader.java:231)
at org.apache.paimon.trino.TrinoPageSource.<init>(TrinoPageSource.java:104)
at org.apache.paimon.trino.TrinoPageSourceProvider.createPageSource(TrinoPageSourceProvider.java:232)
... 18 more
```
### Some simple testing
When we try to query, for example:
`select * from "oix_shipment_waybill"`
It will query and return the data without issue.
Then comes the weird part, when I try to left join another table with the query:
```
select * from "oix_shipment_waybill" AS osw
left join "oix_status_track" AS ost on osw.shipment_waybill_id = ost.shipment_waybill_id
```
It will return error.
------------------
If I switch the tables like this:
```
select * from "oix_status_track" AS ost
LEFT join "oix_shipment_waybill" AS osw on ost.shipment_waybill_id = osw.shipment_waybill_id
```
The query will run without issue.
------------------
I also tried if it is a full join, not just a left join:
```
select * from "oix_status_track" AS ost
join "oix_shipment_waybill" AS osw on ost.shipment_waybill_id = osw.shipment_waybill_id
```
It will return error
------------------
At this point I also tried other combination with other tables with different permutation, playing around with joins, the tables orders, but this error will happen most of the time when I tried to join any tables.
### Anything else?
For starters, I am not even sure if which issue this resides on, I don't know if it's Trino, Paimon or the connector that is the root cause, so please let me know if I should move this issue to other projects or maybe is a simple oversight of something.
Given the nature of this issue and the exception is a simple 'undefined', I'm not sure what to look for or what I am doing wrong. We have other MySQL sources that are also using the same CDC method and they can query with no problem, so I am out of ideas. Please let me know if you want more details that might be of help of this issue.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Trino error with java.lang.IllegalStateException: undefined when joining table [paimon]
Posted by "CodyPin (via GitHub)" <gi...@apache.org>.
CodyPin commented on issue #3712:
URL: https://github.com/apache/paimon/issues/3712#issuecomment-2224909736
On further investigation, I suspect there might be something to do with errors CDC might encounter
My other data source encountered this error while doing CDC in Flink (I censored a little detail there since I think it might be sensitive)
```
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[<IP_HERE>,DS-c22fbdd7-9533-41d2-919c-5826dd4ff0b9,DISK]], original=[DatanodeInfoWithStorage[<IP_HERE>,DS-c22fbdd7-9533-41d2-919c-5826dd4ff0b9,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.
at java.base/java.util.concurrent.FutureTask.report(Unknown Source)
at java.base/java.util.concurrent.FutureTask.get(Unknown Source)
at org.apache.paimon.compact.CompactFutureManager.obtainCompactResult(CompactFutureManager.java:67)
at org.apache.paimon.compact.CompactFutureManager.innerGetCompactionResult(CompactFutureManager.java:53)
at org.apache.paimon.mergetree.compact.MergeTreeCompactManager.getCompactionResult(MergeTreeCompactManager.java:205)
at org.apache.paimon.mergetree.MergeTreeWriter.trySyncLatestCompaction(MergeTreeWriter.java:319)
at org.apache.paimon.mergetree.MergeTreeWriter.flushWriteBuffer(MergeTreeWriter.java:242)
at org.apache.paimon.mergetree.MergeTreeWriter.prepareCommit(MergeTreeWriter.java:248)
at org.apache.paimon.operation.AbstractFileStoreWrite.prepareCommit(AbstractFileStoreWrite.java:198)
at org.apache.paimon.table.sink.TableWriteImpl.prepareCommit(TableWriteImpl.java:207)
at org.apache.paimon.flink.sink.StoreSinkWriteImpl.prepareCommit(StoreSinkWriteImpl.java:215)
... 31 more
```
This error notify us that there is a failed datanode and the writing is stopped for a moment. The job will restart and continue to do CDC, which looks like it recovered without error.
Now if we look at the table with
`SELECT * from addresses`
Seems normal, but if we do any operation on the table, no matter if its a count(*):
or if it's a simple where condition:
### How I came out with the above conclusion/suspicion
After I made this issue, I dropped the whole schema, wiping clean of all data, and start the CDC from the beginning. Then any query will run on that schema's tables, right until this error happened. I think more testing needs to be done with this in order to confirm, but so far this seems to be the most likely cause
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org