You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "xccui (via GitHub)" <gi...@apache.org> on 2023/02/28 05:45:08 UTC
[GitHub] [hudi] xccui opened a new issue, #8074: [SUPPORT] Incompatible commons.io version
xccui opened a new issue, #8074:
URL: https://github.com/apache/hudi/issues/8074
Got the following issue when running a Flink writer job.
```
java.lang.NoSuchMethodError: 'void [org.apache.hudi.org.apache.commons.io](http://org.apache.hudi.org.apache.commons.io/).IOUtils.closeQuietly([java.io](http://java.io/).Closeable, java.util.function.Consumer)'
at [org.apache.hudi.org.apache.hadoop.hbase.io](http://org.apache.hudi.org.apache.hadoop.hbase.io/).hfile.HFileInfo.initTrailerAndContext(HFileInfo.java:352)
at [org.apache.hudi.org.apache.hadoop.hbase.io](http://org.apache.hudi.org.apache.hadoop.hbase.io/).hfile.HFileInfo.<init>(HFileInfo.java:124)
at [org.apache.hudi.io](http://org.apache.hudi.io/).storage.HoodieHFileUtils.createHFileReader(HoodieHFileUtils.java:84)
at [org.apache.hudi.io](http://org.apache.hudi.io/).storage.HoodieHFileReader.<init>(HoodieHFileReader.java:104)
at org.apache.hudi.common.table.log.block.HoodieHFileDataBlock.deserializeRecords(HoodieHFileDataBlock.java:168)
at org.apache.hudi.common.table.log.block.HoodieDataBlock.readRecordsFromBlockPayload(HoodieDataBlock.java:189)
at org.apache.hudi.common.table.log.block.HoodieDataBlock.getRecordIterator(HoodieDataBlock.java:147)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.getRecordsIterator(AbstractHoodieLogRecordReader.java:492)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.processDataBlock(AbstractHoodieLogRecordReader.java:379)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.processQueuedBlocksForInstant(AbstractHoodieLogRecordReader.java:467)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scanInternal(AbstractHoodieLogRecordReader.java:343)
at org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader.scan(AbstractHoodieLogRecordReader.java:192)
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.performScan(HoodieMergedLogRecordScanner.java:109)
at org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner.<init>(HoodieMergedLogRecordScanner.java:102)
at org.apache.hudi.metadata.HoodieMetadataMergedLogRecordReader.<init>(HoodieMetadataMergedLogRecordReader.java:63)
at org.apache.hudi.metadata.HoodieMetadataMergedLogRecordReader.<init>(HoodieMetadataMergedLogRecordReader.java:51)
at org.apache.hudi.metadata.HoodieMetadataMergedLogRecordReader$Builder.build(HoodieMetadataMergedLogRecordReader.java:230)
at org.apache.hudi.metadata.HoodieBackedTableMetadata.getLogRecordScanner(HoodieBackedTableMetadata.java:515)
at org.apache.hudi.metadata.HoodieBackedTableMetadata.openReaders(HoodieBackedTableMetadata.java:428)
at org.apache.hudi.metadata.HoodieBackedTableMetadata.getOrCreateReaders(HoodieBackedTableMetadata.java:413)
at org.apache.hudi.metadata.HoodieBackedTableMetadata.lambda$getRecordsByKeys$1(HoodieBackedTableMetadata.java:219)
at java.base/java.util.HashMap.forEach(Unknown Source)
at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordsByKeys(HoodieBackedTableMetadata.java:217)
at org.apache.hudi.metadata.HoodieBackedTableMetadata.getRecordByKey(HoodieBackedTableMetadata.java:148)
at org.apache.hudi.metadata.BaseTableMetadata.fetchAllFilesInPartition(BaseTableMetadata.java:323)
at org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:141)
at org.apache.hudi.metadata.HoodieMetadataFileSystemView.listPartition(HoodieMetadataFileSystemView.java:65)
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$9(AbstractTableFileSystemView.java:307)
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(Unknown Source)
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:298)
at org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestMergedFileSlicesBeforeOrOn(AbstractTableFileSystemView.java:704)
at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:103)
at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.getLatestMergedFileSlicesBeforeOrOn(PriorityBasedFileSystemView.java:188)
at org.apache.hudi.sink.partitioner.profile.DeltaWriteProfile.smallFilesProfile(DeltaWriteProfile.java:62)
at org.apache.hudi.sink.partitioner.profile.WriteProfile.getSmallFiles(WriteProfile.java:190)
at org.apache.hudi.sink.partitioner.BucketAssigner.getSmallFileAssign(BucketAssigner.java:179)
at org.apache.hudi.sink.partitioner.BucketAssigner.addInsert(BucketAssigner.java:137)
at org.apache.hudi.sink.partitioner.BucketAssignFunction.getNewRecordLocation(BucketAssignFunction.java:208)
at org.apache.hudi.sink.partitioner.BucketAssignFunction.processRecord(BucketAssignFunction.java:197)
at org.apache.hudi.sink.partitioner.BucketAssignFunction.processElement(BucketAssignFunction.java:162)
at org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:83)
at org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:233)
at [org.apache.flink.streaming.runtime.io](http://org.apache.flink.streaming.runtime.io/).AbstractStreamTaskNetworkInput.processElement(AbstractStreamTaskNetworkInput.java:134)
at [org.apache.flink.streaming.runtime.io](http://org.apache.flink.streaming.runtime.io/).AbstractStreamTaskNetworkInput.emitNext(AbstractStreamTaskNetworkInput.java:105)
at [org.apache.flink.streaming.runtime.io](http://org.apache.flink.streaming.runtime.io/).StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:496)
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:203)
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:809)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:761)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:937)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.base/java.lang.Thread.run(Unknown Source)
```
`IOUtils.closeQuietly([java.io](http://java.io/).Closeable, java.util.function.Consumer)'` was added in `commons.io` v2.7. But we are actually using v2.4.
**Environment Description**
* Hudi version : 0.12.2
* Flink version : 1.14.4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8074: [SUPPORT] Incompatible commons.io version
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8074:
URL: https://github.com/apache/hudi/issues/8074#issuecomment-1447926278
Seems introduced by Hbase, guess we must shade the guava instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8074: [SUPPORT] Incompatible commons.io version
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8074:
URL: https://github.com/apache/hudi/issues/8074#issuecomment-1449270178
> java.lang.NoSuchMethodError: 'void org.apache.hudi.org.apache.commons.io.IOUtils.closeQuietly
Seems we already shade the guava in the bundle jar, I guess we just need to use the suitable guava version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #8074: [SUPPORT] Incompatible commons.io version
Posted by "xccui (via GitHub)" <gi...@apache.org>.
xccui commented on issue #8074:
URL: https://github.com/apache/hudi/issues/8074#issuecomment-1448147205
This is an internal dependency issue. The Hbase server lib relies on commons.io v2.7+, but we are using v2.4 (imported by other libs). I roughly checked the pom files but didn't find any recent changes related to this. Not sure why I hit it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #8074: [SUPPORT] Incompatible commons.io version
Posted by "xccui (via GitHub)" <gi...@apache.org>.
xccui commented on issue #8074:
URL: https://github.com/apache/hudi/issues/8074#issuecomment-1449286865
> > java.lang.NoSuchMethodError: 'void org.apache.hudi.org.apache.commons.io.IOUtils.closeQuietly
>
> Seems we already shade the guava in the bundle jar, I guess we just need to use the suitable guava version.
It's `commons-io:commons-io` (not guava). We need to force Hudi to use version 2.11.0 instead of 2.4.
The `hbase-server:2.4.9` we use relies on `commons.io` v2.7+, but the actual resolved version is 2.4. Try checking `HFileInfo.java` in the code base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #8074: [SUPPORT] Incompatible commons.io version
Posted by "xccui (via GitHub)" <gi...@apache.org>.
xccui commented on issue #8074:
URL: https://github.com/apache/hudi/issues/8074#issuecomment-1449291779
Hi @yihua, would also like to get your thoughts on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on issue #8074: [SUPPORT] Incompatible commons.io version
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #8074:
URL: https://github.com/apache/hudi/issues/8074#issuecomment-1449721707
Yeah, do you have interest on contributing for the commons-io upgrade ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xccui commented on issue #8074: [SUPPORT] Incompatible commons.io version
Posted by "xccui (via GitHub)" <gi...@apache.org>.
xccui commented on issue #8074:
URL: https://github.com/apache/hudi/issues/8074#issuecomment-1450122617
Sure. I can try to fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 closed issue #8074: [SUPPORT] Incompatible commons.io version
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 closed issue #8074: [SUPPORT] Incompatible commons.io version
URL: https://github.com/apache/hudi/issues/8074
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org