You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "Arpit Agarwal (Jira)" <ji...@apache.org> on 2020/03/02 21:07:00 UTC
[jira] [Updated] (HDDS-3067) Fix Bug in Scrub Pipeline causing
destory pipelines after SCM restart
[ https://issues.apache.org/jira/browse/HDDS-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arpit Agarwal updated HDDS-3067:
--------------------------------
Fix Version/s: (was: 0.6.0)
> Fix Bug in Scrub Pipeline causing destory pipelines after SCM restart
> ---------------------------------------------------------------------
>
> Key: HDDS-3067
> URL: https://issues.apache.org/jira/browse/HDDS-3067
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: SCM
> Reporter: Bharat Viswanadham
> Assignee: Bharat Viswanadham
> Priority: Blocker
> Labels: OMHATest, pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, the scrubber is run as part of create pipeline.
> When SCM is started, scrubber is coming up and cleaning up all the containers in SCM. Because when loading pipelines, the pipelineCreationTimeStamp is set from when the pipeline is created.
>
> Because of this, below condition is satisfied and destroying all the pipelines when SCM is restarted. This can be easily reproduced start SCM, wait for 10 minutes and restart SCM.
>
> {code:java}
> List<Pipeline> needToSrubPipelines = stateManager.getPipelines(type, factor,
> Pipeline.PipelineState.ALLOCATED).stream()
> .filter(p -> currentTime.toEpochMilli() - p.getCreationTimestamp()
> .toEpochMilli() >= pipelineScrubTimeoutInMills)
> .collect(Collectors.toList());
> for (Pipeline p : needToSrubPipelines) {
> LOG.info("srubbing pipeline: id: " + p.getId().toString() +
> " since it stays at ALLOCATED stage for " +
> Duration.between(currentTime, p.getCreationTimestamp()).toMinutes() +
> " mins.");
> finalizeAndDestroyPipeline(p, false);
> }{code}
>
> *Log showing scrubbing of pipeline*
>
> {code:java}
> 2020-02-20 12:42:18,946 [RatisPipelineUtilsThread] INFO org.apache.hadoop.hdds.scm.pipeline.SCMPipelineManager: srubbing pipeline: id: PipelineID=35dff62d-9bfa-449b-b6e8-6f00cc8c1b6e since it stays at ALLOCATED stage for -1003 mins.{code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org