You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2023/01/06 18:27:26 UTC
[GitHub] [ozone] duongkame opened a new pull request, #4154: HDDS-7738. SCM terminates when adding container to a closed pipeline
duongkame opened a new pull request, #4154:
URL: https://github.com/apache/ozone/pull/4154
## What changes were proposed in this pull request?
It should be fine to add a container to a CLOSED pipeline as a pipeline state can be changed while the container creating transaction is waiting to be processed by SCM.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-7738
## How was this patch tested?
Unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] duongkame commented on pull request #4154: HDDS-7738. SCM terminates when adding container to a closed pipeline
Posted by GitBox <gi...@apache.org>.
duongkame commented on PR #4154:
URL: https://github.com/apache/ozone/pull/4154#issuecomment-1374394849
>
> * We should print a WARN message when it happens; see below.
> * Also, let's keep the `addContainerToPipelineSCMStart` method for now so that it is easier to back port this change. We may do the code refactoring later.
Thanks for the suggestions, @szetszwo. I've made the updates.
Once this is merged I'll submit another PR to clean up the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] duongkame commented on pull request #4154: HDDS-7738. SCM terminates when adding container to a closed pipeline
Posted by GitBox <gi...@apache.org>.
duongkame commented on PR #4154:
URL: https://github.com/apache/ozone/pull/4154#issuecomment-1374274423
Thanks for having a look @sodonnel .
> Does the SCM terminate on the active SCM, or is this on the follower SCMs?
The same transactions get replayed in all SCMs and result the same errors preventing SCM to start up.
> If we allow an open container on a closed pipeline, what will close the open container? The normal close flow is triggered when either the container fills up and the DN triggers a close, or the pipeline is closed and it triggers a close to all containers on the pipeline.
I think such containers will be closed by the pipeline scrubber, which periodically scans and [closes containers associated with closed pipelines](https://github.com/apache/ozone/blob/master/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManagerImpl.java#L522).
> I am also wondering, what happens to a container which is allocated on SCM, but never gets anything written to it. It will never get replicas on a DN, and hence will never have any replicas reported. Will it get cleaned up or will it hang around forever?
I'm not sure about this. Basically, I can't find any process that cleans up empty containers and it looks like a container can only be removed via admin CLI.
Alternatively, SCM can also just reject the transaction (throwing a non-terminus) and move on. Yet, I'm not confident about the consequences.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] duongkame commented on pull request #4154: HDDS-7738. SCM terminates when adding container to a closed pipeline
Posted by GitBox <gi...@apache.org>.
duongkame commented on PR #4154:
URL: https://github.com/apache/ozone/pull/4154#issuecomment-1374124770
@szetszwo @errose28 @aswinshakil please have a look. Some CI integration steps are failing but they're fine when running locally from my laptop, or by the [branch CI](https://github.com/duongkame/ozone/actions/runs/3857497185/jobs/6574962174), guess they're just flaky. Would be nice if one of you can help retry them individually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] szetszwo merged pull request #4154: HDDS-7738. SCM terminates when adding container to a closed pipeline
Posted by GitBox <gi...@apache.org>.
szetszwo merged PR #4154:
URL: https://github.com/apache/ozone/pull/4154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sodonnel commented on pull request #4154: HDDS-7738. SCM terminates when adding container to a closed pipeline
Posted by GitBox <gi...@apache.org>.
sodonnel commented on PR #4154:
URL: https://github.com/apache/ozone/pull/4154#issuecomment-1374230304
Does the SCM terminate on the active SCM, or is this on the follower SCMs?
If we allow an open container on a closed pipeline, what will close the open container? The normal close flow is triggered when either the container fills up and the DN triggers a close, or the pipeline is closed and it triggers a close to all containers on the pipeline.
I am also wondering, what happens to a container which is allocated on SCM, but never gets anything written to it. It will never get replicas on a DN, and hence will never have any replicas reported. Will it get cleaned up or will it hang around forever?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org