You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "shuan1026 (via GitHub)" <gi...@apache.org> on 2026/08/22 15:51:00 UTC
[PR] HDDS-16253. Fix getPipelines() in Recon always returning an empty pipeline list [ozone]
shuan1026 opened a new pull request, #11090:
URL: https://github.com/apache/ozone/pull/11090
## What changes were proposed in this pull request?
`ReconContainerMetadataManagerImpl#getPipelines(ContainerKeyPrefix)` builds a nested `Stream` pipeline to collect the `Pipeline`s for a key's block locations, but both `.stream().map(...)` calls are intermediate operations with no terminal operation attached. Because Java streams are lazily evaluated, the `pipelines.add(...)` is never actually executed, so the method unconditionally returns the empty `ArrayList` created at the top of the method.
This method is the only implementation backing `ContainerMetadataIterator#next()`, which is used by `ReconContainerMetadataManager#getContainersIterator()`. That iterator feeds two REST endpoints in `ContainerEndpoint`: `GET /containers/mismatch` and `GET /containers/mismatch/deleted`. Both currently return `"pipelines": []` for every container, even when the container's key actually has valid pipeline data in OM.
This PR replaces the two dangling `.stream().map(...)` chains with `.forEach(...)`
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16253
## How was this patch tested?
- Ran the existing `TestReconContainerMetadataManagerImpl`: 15/15 tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16253. Fix getPipelines() in Recon always returning an empty pipeline list [ozone]
Posted by "devmadhuu (via GitHub)" <gi...@apache.org>.
devmadhuu commented on code in PR #11090:
URL: https://github.com/apache/ozone/pull/11090#discussion_r3868378684
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconContainerMetadataManagerImpl.java:
##########
@@ -514,10 +514,10 @@ private List<Pipeline> getPipelines(ContainerKeyPrefix containerKeyPrefix)
}
List<Pipeline> pipelines = new ArrayList<>();
if (null != omKeyInfo) {
- omKeyInfo.getKeyLocationVersions().stream().map(
+ omKeyInfo.getKeyLocationVersions().forEach(
omKeyLocationInfoGroup ->
- omKeyLocationInfoGroup.getLocationList()
- .stream().map(omKeyLocationInfo -> pipelines.add(
+ omKeyLocationInfoGroup.getLocationList().forEach(
Review Comment:
```suggestion
pipelines.addAll(
omKeyInfo.getKeyLocationVersions().stream()
.flatMap(g -> g.getLocationList().stream())
.map(OmKeyLocationInfo::getPipeline)
.collect(Collectors.toList()));
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconContainerMetadataManagerImpl.java:
##########
@@ -514,10 +514,10 @@ private List<Pipeline> getPipelines(ContainerKeyPrefix containerKeyPrefix)
}
List<Pipeline> pipelines = new ArrayList<>();
if (null != omKeyInfo) {
- omKeyInfo.getKeyLocationVersions().stream().map(
+ omKeyInfo.getKeyLocationVersions().forEach(
omKeyLocationInfoGroup ->
- omKeyLocationInfoGroup.getLocationList()
- .stream().map(omKeyLocationInfo -> pipelines.add(
+ omKeyLocationInfoGroup.getLocationList().forEach(
+ omKeyLocationInfo -> pipelines.add(
Review Comment:
Also add the following assertions in our existing test [here](https://github.com/apache/ozone/blob/5ff8b53aa6b73e3026a41c7e04741896932cb92d/hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/spi/impl/TestReconContainerMetadataManagerImpl.java#L380)
```
ContainerMetadata cm = containerMap.get(containerId);
assertNotNull(cm.getPipelines());
assertFalse(cm.getPipelines().isEmpty(),
"Pipelines list should be populated from OmKeyLocationInfo");
// If test data has known pipeline count, assert it exactly.
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconContainerMetadataManagerImpl.java:
##########
@@ -514,10 +514,10 @@ private List<Pipeline> getPipelines(ContainerKeyPrefix containerKeyPrefix)
}
List<Pipeline> pipelines = new ArrayList<>();
if (null != omKeyInfo) {
- omKeyInfo.getKeyLocationVersions().stream().map(
+ omKeyInfo.getKeyLocationVersions().forEach(
omKeyLocationInfoGroup ->
- omKeyLocationInfoGroup.getLocationList()
- .stream().map(omKeyLocationInfo -> pipelines.add(
+ omKeyLocationInfoGroup.getLocationList().forEach(
Review Comment:
Its better, if we should de-duplicate the pipeline list here itself rather at the caller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16253. Fix getPipelines() in Recon always returning an empty pipeline list [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11090:
URL: https://github.com/apache/ozone/pull/11090#issuecomment-6329395779
Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16253. Fix getPipelines() in Recon always returning an empty pipeline list [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #11090: HDDS-16253. Fix getPipelines() in Recon always returning an empty pipeline list
URL: https://github.com/apache/ozone/pull/11090
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16253. Fix getPipelines() in Recon always returning an empty pipeline list [ozone]
Posted by "ArafatKhan2198 (via GitHub)" <gi...@apache.org>.
ArafatKhan2198 commented on code in PR #11090:
URL: https://github.com/apache/ozone/pull/11090#discussion_r3895368233
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconContainerMetadataManagerImpl.java:
##########
@@ -514,10 +514,10 @@ private List<Pipeline> getPipelines(ContainerKeyPrefix containerKeyPrefix)
}
List<Pipeline> pipelines = new ArrayList<>();
if (null != omKeyInfo) {
- omKeyInfo.getKeyLocationVersions().stream().map(
+ omKeyInfo.getKeyLocationVersions().forEach(
Review Comment:
**Blocking:** `omKeyInfo` is read from the persisted OM key/file table, whose codec serializes keys with `ignorePipeline=true`. Therefore `OmKeyLocationInfo#getPipeline()` is normally null here. This change adds that null to the result, and the caller immediately dereferences it through `pipeline.getId()`, causing the reproduced HTTP 500. Filtering null values would prevent the crash, but it would still leave the pipeline list empty and would not solve this,
please do correct me if I am wrong on this
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/spi/impl/ReconContainerMetadataManagerImpl.java:
##########
@@ -514,10 +514,10 @@ private List<Pipeline> getPipelines(ContainerKeyPrefix containerKeyPrefix)
}
List<Pipeline> pipelines = new ArrayList<>();
if (null != omKeyInfo) {
- omKeyInfo.getKeyLocationVersions().stream().map(
+ omKeyInfo.getKeyLocationVersions().forEach(
omKeyLocationInfoGroup ->
- omKeyLocationInfoGroup.getLocationList()
- .stream().map(omKeyLocationInfo -> pipelines.add(
+ omKeyLocationInfoGroup.getLocationList().forEach(
Review Comment:
Please add an endpoint-level regression test that stores the key through the real OM codec and calls both `/containers/mismatch?missingIn=SCM` and `/containers/mismatch/deleted`. The test should verify that the response does not return 500 and that each container receives only its own pipeline information. The existing test only populates container-key mappings, so it does not exercise this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16253. Fix getPipelines() in Recon always returning an empty pipeline list [ozone]
Posted by "Russole (via GitHub)" <gi...@apache.org>.
Russole commented on PR #11090:
URL: https://github.com/apache/ozone/pull/11090#issuecomment-5747833199
Thanks @shuan1026 for working on this! Could you please resolve the merge conflict?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-16253. Fix getPipelines() in Recon always returning an empty pipeline list [ozone]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #11090:
URL: https://github.com/apache/ozone/pull/11090#issuecomment-6181253939
This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org