You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "duongkame (via GitHub)" <gi...@apache.org> on 2023/11/18 01:46:42 UTC
[PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
duongkame opened a new pull request, #5625:
URL: https://github.com/apache/ozone/pull/5625
## What changes were proposed in this pull request?
Today, container pipelines are cached in OM and the cache data consistency is eventually ensured by client behavior. This means, if a container is replicated to another set of datanodes, the client detects this change when using the outdated cached pipeline to read data from datanodes and requests OM to refresh the pipeline cache from SCM.
When the datanodes belonging to a container go offline, there are chances that an empty pipeline could be cached in OM. However, when client get an empty pipeline, it fails to ask OM to refresh the pipeline.
```
Caused by: java.lang.IllegalArgumentException: NO_REPLICA_FOUND
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145)
at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClient(XceiverClientManager.java:164)
at org.apache.hadoop.hdds.scm.XceiverClientManager.acquireClientForReadData(XceiverClientManager.java:157)
at org.apache.hadoop.hdds.scm.storage.BlockInputStream.acquireClient(BlockInputStream.java:285)
at org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:238)
at org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:146)
at org.apache.hadoop.hdds.scm.storage.BlockInputStream.readWithStrategy(BlockInputStream.java:308)
```
This makes empty pipelines cached for an extended duration.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9709
## How was this patch tested?
Normal CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "duongkame (via GitHub)" <gi...@apache.org>.
duongkame commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1820118148
I'm not sure why the CI build-branch fails. It works on my laptop and I can't eyeball any problem with the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1834265890
Based on discussion with @duongkame, the behavior to the client will not change, the caching model as is implemented right now should be the simplest to remove entries that do not have replicas to force a read from SCM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "duongkame (via GitHub)" <gi...@apache.org>.
duongkame commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1830420918
> I would suggest that if scm returns empty nodes; they are not added to the cache @kerneltime @adoroszlai @duongkame how do you think?
The container cache is a read-through cache, meaning it loads data and adds to the cache on its own. This saves us the effort of maintaining things like key-level synchronization (to avoid double reads). However, guava does not provide the flexibility to filter/decide what not to put in the cache. The invalidation right after reads is the closest to get to the behavior we want, and it does it job just right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "duongkame (via GitHub)" <gi...@apache.org>.
duongkame commented on code in PR #5625:
URL: https://github.com/apache/ozone/pull/5625#discussion_r1399847167
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ScmClient.java:
##########
@@ -111,7 +112,14 @@ public Map<Long, Pipeline> getContainerLocations(Iterable<Long> containerIds,
containerLocationCache.invalidateAll(containerIds);
}
try {
- return containerLocationCache.getAll(containerIds);
+ Map<Long, Pipeline> result = containerLocationCache.getAll(containerIds);
+ // Don't keep empty pipelines in the cache.
+ List<Long> emptyPipelines = result.entrySet().stream()
Review Comment:
Only the containers for this particular request. (Each request can bind to more than one container if a key has 2 or more blocks on 2 or more containers).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "swamirishi (via GitHub)" <gi...@apache.org>.
swamirishi commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1819485539
cc: @GeorgeJahad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime merged PR #5625:
URL: https://github.com/apache/ozone/pull/5625
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #5625:
URL: https://github.com/apache/ozone/pull/5625#discussion_r1406098750
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ScmClient.java:
##########
@@ -111,7 +112,14 @@ public Map<Long, Pipeline> getContainerLocations(Iterable<Long> containerIds,
containerLocationCache.invalidateAll(containerIds);
}
try {
- return containerLocationCache.getAll(containerIds);
+ Map<Long, Pipeline> result = containerLocationCache.getAll(containerIds);
+ // Don't keep empty pipelines in the cache.
+ List<Long> emptyPipelines = result.entrySet().stream()
Review Comment:
As fas as I know, this method is only called in the `KeyManagerImpl` and `Iterable<Long> containerIds` parameter contains only the containerIds for a particular `OmKeyInfo`.
Across how many containers can a key span? What's the worst case scenario?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "duongkame (via GitHub)" <gi...@apache.org>.
duongkame commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1821302697
> > I'm not sure why the CI build-branch fails. It works on my laptop and I can't eyeball any problem with the code.
>
> "This branch is [3 commits ahead](https://github.com/duongkame/ozone/compare/apache:ozone:master...HDDS-9709), [33 commits behind](https://github.com/duongkame/ozone/compare/HDDS-9709...apache:ozone:master) `apache:master`."
>
> Github doesn't show conflict, but that only means same areas of code were not changed in the two branches. E.g. if one branch adds a new abstract method, and other branch adds a new subclass to the same abstract parent class, there will be no code conflict, but compilation will fail.
>
> In this case I have replaced ([15c77c6](https://github.com/apache/ozone/commit/15c77c6e9bb22a0bbf5d0080f81c9310051229a9)) `static long testContainerId` with `static final AtomicLong CONTAINER_ID` to fix SpotBugs warnings previously suppressed. The PR is adding a new test method which uses `testContainerId`. This is fine locally, since the branch is behind, but it would fail if the PR was merged into current `master`.
>
> PR workflow [builds](https://github.com/apache/ozone/actions/runs/6938142340/job/18873425543?pr=5625#step:2:491) and tests a hypothetical merge commit to detect such problems (at the time of the latest commit in the PR):
>
> > HEAD is now at [0167f66](https://github.com/apache/ozone/commit/0167f6605e18f8e867457e37208bbd60b3ca256b) Merge [3bfa090](https://github.com/apache/ozone/commit/3bfa0906a3629fbeba59e278039855406772e850) into [945e8bf](https://github.com/apache/ozone/commit/945e8bfdcfe0be1fecbca3716d41305c4ae23d00)
>
> Solved by merging from `master` locally and fixing compile errors.
Thanks a lot, @adoroszlai. It's very tricky that the "merge from master" is done silently in the CI build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1819489468
I think i have seen this before. IIRC in my case the SCM was not reachable. Did that happen to you too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #5625:
URL: https://github.com/apache/ozone/pull/5625#discussion_r1409978405
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ScmClient.java:
##########
@@ -111,7 +112,14 @@ public Map<Long, Pipeline> getContainerLocations(Iterable<Long> containerIds,
containerLocationCache.invalidateAll(containerIds);
}
try {
- return containerLocationCache.getAll(containerIds);
+ Map<Long, Pipeline> result = containerLocationCache.getAll(containerIds);
+ // Don't keep empty pipelines in the cache.
+ List<Long> emptyPipelines = result.entrySet().stream()
Review Comment:
@xBis7 we can add a background refresh loop but I am not sure it will lead to significant improvement, would something interesting to test and implement if we see a difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1819542537
@jojochuang to be clear, the problem persisted long after the scm came back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1820329645
> I'm not sure why the CI build-branch fails. It works on my laptop and I can't eyeball any problem with the code.
"This branch is [3 commits ahead](https://github.com/duongkame/ozone/compare/apache:ozone:master...HDDS-9709), [33 commits behind](https://github.com/duongkame/ozone/compare/HDDS-9709...apache:ozone:master) `apache:master`."
Github doesn't show conflict, but that only means same areas of code were not changed in the two branches. E.g. if one branch adds a new abstract method, and other branch adds a new subclass to the same abstract parent class, there will be no code conflict, but compilation will fail.
In this case I have replaced (15c77c6e9bb22a0bbf5d0080f81c9310051229a9) `static long testContainerId` with `static final AtomicLong CONTAINER_ID` to fix SpotBugs warnings previously suppressed. The PR is adding a new test method which uses `testContainerId`. This is fine locally, since the branch is behind, but it would fail if the PR was merged into current `master`.
PR workflow [builds](https://github.com/apache/ozone/actions/runs/6938142340/job/18873425543?pr=5625#step:2:491) and tests a hypothetical merge commit to detect such problems (at the time of the latest commit in the PR):
> HEAD is now at 0167f66 Merge 3bfa0906a3629fbeba59e278039855406772e850 into 945e8bfdcfe0be1fecbca3716d41305c4ae23d00
Solved by merging from `master` locally and fixing compile errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "duongkame (via GitHub)" <gi...@apache.org>.
duongkame commented on code in PR #5625:
URL: https://github.com/apache/ozone/pull/5625#discussion_r1406603177
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ScmClient.java:
##########
@@ -111,7 +112,14 @@ public Map<Long, Pipeline> getContainerLocations(Iterable<Long> containerIds,
containerLocationCache.invalidateAll(containerIds);
}
try {
- return containerLocationCache.getAll(containerIds);
+ Map<Long, Pipeline> result = containerLocationCache.getAll(containerIds);
+ // Don't keep empty pipelines in the cache.
+ List<Long> emptyPipelines = result.entrySet().stream()
Review Comment:
It's simply linear to the number of blocks the key has. E.g. a key with 10 blocks can span 10 containers. Anyway, the cost is contained by the number of blocks, just like before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #5625:
URL: https://github.com/apache/ozone/pull/5625#discussion_r1409977836
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ScmClient.java:
##########
@@ -111,7 +112,14 @@ public Map<Long, Pipeline> getContainerLocations(Iterable<Long> containerIds,
containerLocationCache.invalidateAll(containerIds);
}
try {
- return containerLocationCache.getAll(containerIds);
+ Map<Long, Pipeline> result = containerLocationCache.getAll(containerIds);
+ // Don't keep empty pipelines in the cache.
+ List<Long> emptyPipelines = result.entrySet().stream()
Review Comment:
This should be fine, even a large object will have a few hundred blocks or low thousand containers in the worst case. The current implementation is based on cache on demand with ttl or force refresh. If at the time of client read the containers are not cached, it will be fetched again from SCM hopefully with replica information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on code in PR #5625:
URL: https://github.com/apache/ozone/pull/5625#discussion_r1399740209
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ScmClient.java:
##########
@@ -111,7 +112,14 @@ public Map<Long, Pipeline> getContainerLocations(Iterable<Long> containerIds,
containerLocationCache.invalidateAll(containerIds);
}
try {
- return containerLocationCache.getAll(containerIds);
+ Map<Long, Pipeline> result = containerLocationCache.getAll(containerIds);
+ // Don't keep empty pipelines in the cache.
+ List<Long> emptyPipelines = result.entrySet().stream()
Review Comment:
Will we land up scanning all containers for each get request?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "duongkame (via GitHub)" <gi...@apache.org>.
duongkame commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1820042106
Added test, PR ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1834767894
@guohao-rosicky please take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "xBis7 (via GitHub)" <gi...@apache.org>.
xBis7 commented on code in PR #5625:
URL: https://github.com/apache/ozone/pull/5625#discussion_r1406623287
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/ScmClient.java:
##########
@@ -111,7 +112,14 @@ public Map<Long, Pipeline> getContainerLocations(Iterable<Long> containerIds,
containerLocationCache.invalidateAll(containerIds);
}
try {
- return containerLocationCache.getAll(containerIds);
+ Map<Long, Pipeline> result = containerLocationCache.getAll(containerIds);
+ // Don't keep empty pipelines in the cache.
+ List<Long> emptyPipelines = result.entrySet().stream()
Review Comment:
What about a background thread that checks and removes empty pipelines from cache, every 10 or 20 mins? After looking at `KeyManagerImpl#refreshPipelineFromCache()`, it's getting the containerIds for a list of Keys. The number of container pipelines can be too much to iterate on every request.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "guohao-rosicky (via GitHub)" <gi...@apache.org>.
guohao-rosicky commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1829483936
I would suggest that if scm returns empty nodes; they are not added to the cache
@kerneltime @adoroszlai @duongkame how do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "kerneltime (via GitHub)" <gi...@apache.org>.
kerneltime commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1832852091
> I would suggest that if scm returns empty nodes; they are not added to the cache @kerneltime @adoroszlai @duongkame how do you think?
That is a good point, for a brief window the cache will have the empty containers in the cache before they are invalidated. @duongkame the the SCM client filter these out before adding to the cache?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "GeorgeJahad (via GitHub)" <gi...@apache.org>.
GeorgeJahad commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1819541614
@jojochuang yes the scm was briefly unreachable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "jojochuang (via GitHub)" <gi...@apache.org>.
jojochuang commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1819609025
Make sense @GeorgeJahad I had to restart SCM to resolve the issue in my case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-9709. Avoid empty pipelines in the container pipeline cache. [ozone]
Posted by "adoroszlai (via GitHub)" <gi...@apache.org>.
adoroszlai commented on PR #5625:
URL: https://github.com/apache/ozone/pull/5625#issuecomment-1821313735
> It's very tricky that the "merge from master" is done silently in the CI build
As far as I remember it's a Github feature, not our magic. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org