You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/06/22 17:55:39 UTC
[GitHub] [druid] paul-rogers opened a new issue, #12692: Flaky IT:
paul-rogers opened a new issue, #12692:
URL: https://github.com/apache/druid/issues/12692
The IT test [perfect rollup parallel batch index integration test](https://app.travis-ci.com/github/apache/druid/jobs/573989709) failed. The PR in question did not change the indexer (or indeed, any existing code; it adds new, as-yet-unused code.) So, the failure is likely flaky.
```text
2022-06-18T22:33:03,506 ERROR [main] org.apache.druid.cli.CliPeon - Error when starting up. Failing.
java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_332]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_332]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_332]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_332]
at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) ~[druid-core-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-core-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:152) ~[druid-core-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:165) [druid-services-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
at org.apache.druid.cli.CliPeon.run(CliPeon.java:306) [druid-services-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
at org.apache.druid.cli.Main.main(Main.java:114) [druid-services-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
Caused by: java.io.UncheckedIOException: /tmp/intermediary-segments/index_parallel_wikipedia_parallel_druid_input_source_index_test %Россия 한국 中国!?_ndiecicn_2022-06-18T22:26:38.659Z
at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2153) ~[commons-io-2.11.0.jar:2.11.0]
at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.lambda$discoverSupervisorTaskPartitions$3(LocalIntermediaryDataManager.java:200) ~[druid-indexing-service-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_332]
at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.discoverSupervisorTaskPartitions(LocalIntermediaryDataManager.java:197) ~[druid-indexing-service-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.start(LocalIntermediaryDataManager.java:134) ~[druid-indexing-service-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
... 10 more
Caused by: java.nio.file.NoSuchFileException: /tmp/intermediary-segments/index_parallel_wikipedia_parallel_druid_input_source_index_test %Россия 한국 中国!?_ndiecicn_2022-06-18T22:26:38.659Z
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:1.8.0_332]
...
2022-06-18T22:38:25,438 ERROR [main] org.apache.druid.testing.utils.LoggerListener - Failed [org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest.testIndexData] with parameters [SingleDimensionPartitionsSpec{targetRowsPerSegment=2, maxRowsPerSegment=null, partitionDimension='namespace', assumeGrouped=false, resolvedMaxRowPerSegment=3}]
org.apache.druid.java.util.common.ISE: Error while making request to indexer [404 Not Found No task reports were found for this task. The task may not exist, or it may not have completed yet.]
at org.apache.druid.testing.clients.OverlordResourceTestClient.makeRequest(OverlordResourceTestClient.java:727) ~[druid-integration-tests-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
at org.apache.druid.testing.clients.OverlordResourceTestClient.getTaskReport(OverlordResourceTestClient.java:263) ~[druid-integration-tests-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
at org.apache.druid.testing.clients.OverlordResourceTestClient.getTaskErrorMessage(OverlordResourceTestClient.java:246) ~[druid-integration-tests-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
at org.apache.druid.tests.indexer.AbstractIndexerTest.unloadAndKillData(AbstractIndexerTest.java:91) ~[test-classes/:?]
at org.apache.druid.tests.indexer.AbstractIndexerTest.lambda$unloader$0(AbstractIndexerTest.java:77) ~[test-classes/:?]
at org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest.testIndexData(ITPerfectRollupParallelIndexTest.java:127) ~[test-classes/:?]
...
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1,815.156 s <<< FAILURE! - in TestSuite
[ERROR] testIndexData(org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest) Time elapsed: 667.606 s
[ERROR] testIndexData(org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest) Time elapsed: 576.529 s
[ERROR] testIndexData(org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest) Time elapsed: 474.567 s <<< FAILURE!
org.apache.druid.java.util.common.ISE: Error while making request to indexer [404 Not Found No task reports were found for this task. The task may not exist, or it may not have completed yet.]
at org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest.testIndexData(ITPerfectRollupParallelIndexTest.java:127)
...
[ERROR] Failures:
[ERROR] org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest.testIndexData(org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest)
[INFO] Run 1: PASS
[INFO] Run 2: PASS
[ERROR] Run 3: ITPerfectRollupParallelIndexTest.testIndexData:127->AbstractIndexerTest.lambda$unloader$0:77->AbstractIndexerTest.unloadAndKillData:91 » ISE
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] kfaraz commented on issue #12692: Flaky IT: perfect rollup parallel batch index integration test
Posted by GitBox <gi...@apache.org>.
kfaraz commented on issue #12692:
URL: https://github.com/apache/druid/issues/12692#issuecomment-1170869825
Great find, @abhishekagarwal87 ! I agree with the fix for the expiration bug.
> We should definitely fix the peons to ignore the contents belonging to a different task group.
This would make sense but there is an issue. The peon needs to be aware of the contents to know if it can reserve a location or not. In other words, the instance of `StorageLocation` must be updated with all the files that exist so that it knows how much space it can allocate for new partition files.
---
I thought of breaking up the `IntermediaryDataManager` into two parts, one which is used by the peon, solely to `addSegment()` and the other used by indexer/middle manager to perform discovery and cleanup. But there are two issues with this:
- As before, `StorageLocation.reserve()` would not work correctly for peon
- `IntermediaryDataManager` is an extension point, so it would not be desirable to change this contract
Also, there are some race conditions possible even outside the ones you mentioned. In particular, between two invocations of the discovery method, a task would have reserved space and exhausted the storage location which might cause another task to fail. But since the default discovery period is small enough (60s), it has not been an issue, I guess.
---
**Proposal:**
Expose a `reserve` API on the ShuffleResource (used by middle manager/indexer) and peons would call that to reserve a location.
- eliminates race conditions as middle manager becomes the sole storage resource manager
- peon logic would become much simpler
- try to reserve using ShuffleResource API
- if succeeds, add segment
- if fails, try next location
- does not need to keep track of anything else
- does not perform cleanup or discovery
- we can continue to use the same `IntermediaryDataManager` contract
Cons:
- new API call between peon and middle manager whenever pushing segment (not very frequent)
- the new `reserve` API is relevant only to local storage, as deep storage need not perform a reserve functionality (could be a noop for deep storage)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] paul-rogers commented on issue #12692: Flaky IT: perfect rollup parallel batch index integration test
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on issue #12692:
URL: https://github.com/apache/druid/issues/12692#issuecomment-1171427707
@abhishekagarwal87, thanks much for diagnosing this issue!
Failed again in this build: [37272.52 (Compile=openjdk8, Run=openjdk8) perfect rollup parallel batch index integration test with Indexer](https://app.travis-ci.com/github/apache/druid/jobs/575188113)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 commented on issue #12692: Flaky IT: perfect rollup parallel batch index integration test
Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 commented on issue #12692:
URL: https://github.com/apache/druid/issues/12692#issuecomment-1166225796
As I was debugging this, I found two problems with `LocalIntermediaryDataManager`.
- Both middle-manager and peons start `LocalIntermediaryDataManager`. There is no guard to protect from race conditions as multiple services on the same machine try to fiddle with local files. E.g. here is a potential race condition
- Task Y starts iterating through all the supervisor task directories on local disk. There is no need to walk through all the supervisor task directories but `LocalIntermediaryDataManager` does not know whether it is running inside a peon or a middle manager. Task Y detects that there is a directory for Task X. It is about to iterate through the contents of X's dir.
- Middle manager detects that X already completed and is expired. So it will delete the contents of X's dir.
- Task Y gets an error during startup because X's dir cannot be found.
- Though looking further, it turns out the default expiration period for deleting intermediate data is 1 day. The test certainly did not take a day. so why did the MM delete the intermediate data at all? Weird. Another dead-end? Well, thankfully not. Poking around the code further, you find this line of code in `LocalIntermediaryDataManager#deleteExpiredSuprevisorTaskPartitionsIfNotRunning`
```
if (checkTime.isAfter(now)) {
expiredSupervisorTasks.add(supervisorTaskId);
}
```
`checkTime` is regularly updated and is usually set to `now + expiryTime`. If the above code snippet should really be as follows
```
if (now.isAfter(checkTime)) {
expiredSupervisorTasks.add(supervisorTaskId);
}
```
This bug has been around for ages now. It appears to be a simple fix but fixing it has implications. Because of this bug, stale intermediate data gets deleted in 5 minutes once the supervisor is completed. 5 minutes since that's the frequency at which expired data is checked. If we fix this bug, on all the druid installations out there, intermediate data will lie around for a day. That can be a problem. Especially on the clusters that have a lot of short-running batch tasks. We should definitely fix the peons to ignore the contents belonging to a different task group. For the `expiration` bug, we should fix it and reduce the default expiration time to 5 minutes in code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] abhishekagarwal87 closed issue #12692: Flaky IT: perfect rollup parallel batch index integration test
Posted by GitBox <gi...@apache.org>.
abhishekagarwal87 closed issue #12692: Flaky IT: perfect rollup parallel batch index integration test
URL: https://github.com/apache/druid/issues/12692
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Flaky IT: perfect rollup parallel batch index integration test (druid)
Posted by "layoaster (via GitHub)" <gi...@apache.org>.
layoaster commented on issue #12692:
URL: https://github.com/apache/druid/issues/12692#issuecomment-4267703327
This is still an issue in 2026, I'm on Druid v34.0.0 and my periodic reindexing task are also failing with upon initialization the MMs delete the intermediate data of compaction tasks.
[task_failed_logs.txt](https://github.com/user-attachments/files/26823116/task_failed_logs.txt)
[task_failed_logs2.txt](https://github.com/user-attachments/files/26823117/task_failed_logs2.txt)
I've attached the full log files for two affected tasks, but this is the error:
```
026-04-14T20:05:06,866 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Starting lifecycle [DiscoveryServiceLocator#start]
2026-04-14T20:05:06,867 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Starting lifecycle [LocalIntermediaryDataManager#start]
2026-04-14T20:05:06,869 ERROR [main] org.apache.druid.cli.CliPeon - Error when starting up. Failing.
java.lang.reflect.InvocationTargetException: null
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) ~[druid-processing-34.0.0.jar:34.0.0]
at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-processing-34.0.0.jar:34.0.0]
at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:152) ~[druid-processing-34.0.0.jar:34.0.0]
at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:137) [druid-services-34.0.0.jar:34.0.0]
at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:94) [druid-services-34.0.0.jar:34.0.0]
at org.apache.druid.cli.CliPeon.run(CliPeon.java:390) [druid-services-34.0.0.jar:34.0.0]
at org.apache.druid.cli.Main.main(Main.java:112) [druid-services-34.0.0.jar:34.0.0]
Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /opt/druid/var/tmp/intermediary-segments/coordinator-issued_compact_sdk-emea-realtime_eipjffkn_2026-04-14T19:54:42.252Z
at org.apache.commons.io.function.Uncheck.wrap(Uncheck.java:339) ~[commons-io-2.17.0.jar:2.17.0]
at org.apache.commons.io.function.Uncheck.get(Uncheck.java:199) ~[commons-io-2.17.0.jar:2.17.0]
at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.lambda$discoverSupervisorTaskPartitions$3(LocalIntermediaryDataManager.java:198) ~[druid-indexing-service-34.0.0.jar:34.0.0]
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.discoverSupervisorTaskPartitions(LocalIntermediaryDataManager.java:195) ~[druid-indexing-service-34.0.0.jar:34.0.0]
at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.start(LocalIntermediaryDataManager.java:135) ~[druid-indexing-service-34.0.0.jar:34.0.0]
... 11 more
Caused by: java.nio.file.NoSuchFileException: /opt/druid/var/tmp/intermediary-segments/coordinator-issued_compact_sdk-emea-realtime_eipjffkn_2026-04-14T19:54:42.252Z
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) ~[?:?]
at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148) ~[?:?]
at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
at java.base/java.nio.file.Files.readAttributes(Files.java:1851) ~[?:?]
at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:226) ~[?:?]
at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:277) ~[?:?]
at java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:323) ~[?:?]
at java.base/java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:71) ~[?:?]
at java.base/java.nio.file.Files.walk(Files.java:3918) ~[?:?]
at org.apache.commons.io.file.PathUtils.walk(PathUtils.java:1847) ~[commons-io-2.17.0.jar:2.17.0]
at org.apache.commons.io.FileUtils.streamFiles(FileUtils.java:2964) ~[commons-io-2.17.0.jar:2.17.0]
at org.apache.commons.io.FileUtils.lambda$listFiles$10(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
at org.apache.commons.io.function.Uncheck.get(Uncheck.java:197) ~[commons-io-2.17.0.jar:2.17.0]
at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.lambda$discoverSupervisorTaskPartitions$3(LocalIntermediaryDataManager.java:198) ~[druid-indexing-service-34.0.0.jar:34.0.0]
at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.discoverSupervisorTaskPartitions(LocalIntermediaryDataManager.java:195) ~[druid-indexing-service-34.0.0.jar:34.0.0]
at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.start(LocalIntermediaryDataManager.java:135) ~[druid-indexing-service-34.0.0.jar:34.0.0]
... 11 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org