You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (Created) (JIRA)" <ji...@apache.org> on 2012/04/03 00:23:30 UTC
[jira] [Created] (HBASE-5702)
MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a
MonitoredTask per call
MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
--------------------------------------------------------------------------------------------
Key: HBASE-5702
URL: https://issues.apache.org/jira/browse/HBASE-5702
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: Jean-Daniel Cryans
Fix For: 0.94.0, 0.96.0
This bug is so easy to reproduce I'm wondering why it hasn't been reported yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in the master interface one task per stopped region server saying the following:
|Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in progress. Skipping exclusion for server = sv4r28s44,62023,1333402175342 (since 5sec ago)|
It's gonna stay there until the master cleans it:
bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340: status=No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, completionTime=-1 appears to have been leaked
It's not clear to me why it's using a MonitoredTask in the first place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5702)
MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a
MonitoredTask per call
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245793#comment-13245793 ]
Jean-Daniel Cryans commented on HBASE-5702:
-------------------------------------------
Yeah when I opened this jira it had a narrow scope but I think it could be much bigger. Basically if you try to instant alter a table you'll see there's about two or three (sorry I can't be more precise at the moment, I'm testing something else right now) tasks that are leaked.
I can see some issues just looking at the code:
MasterSchemaChangeTracker
- processCompletedSchemaChanges: creates a task, sets the status, never closes it. It's a misuse of MonitoredTask I think, a task is normally something that's long running and you need to report progress.
- processAlterStatus: creates a task but stops it right away, also creates one then kills it.
- handleFailedOrExpiredSchemaChanges: creates a task, sets the status, never closes it. Also it has an extra white space.
- createSchemaChangeNode: creates a task then closes/aborts it right away
SchemaChangeTracker
- handleSchemaChange: creates a task, sets the status, never closes it.
- reportAndLogSchemaRefreshError: can create task and set a status but doesn't close it.
There really should only be 1 task throughout the alter process.
> MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-5702
> URL: https://issues.apache.org/jira/browse/HBASE-5702
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: Jean-Daniel Cryans
> Assignee: Subbu M Iyer
> Priority: Critical
> Fix For: 0.94.0, 0.96.0
>
>
> This bug is so easy to reproduce I'm wondering why it hasn't been reported yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in the master interface one task per stopped region server saying the following:
> |Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340 (since 5sec ago)|
> It's gonna stay there until the master cleans it:
> bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340: status=No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, completionTime=-1 appears to have been leaked
> It's not clear to me why it's using a MonitoredTask in the first place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5702)
MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a
MonitoredTask per call
Posted by "Jean-Daniel Cryans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-5702:
--------------------------------------
Description:
This bug is so easy to reproduce I'm wondering why it hasn't been reported yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in the master interface one task per stopped region server saying the following:
|Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340 (since 5sec ago)|
It's gonna stay there until the master cleans it:
bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340: status=No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, completionTime=-1 appears to have been leaked
It's not clear to me why it's using a MonitoredTask in the first place.
was:
This bug is so easy to reproduce I'm wondering why it hasn't been reported yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in the master interface one task per stopped region server saying the following:
|Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in progress. Skipping exclusion for server = sv4r28s44,62023,1333402175342 (since 5sec ago)|
It's gonna stay there until the master cleans it:
bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340: status=No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, completionTime=-1 appears to have been leaked
It's not clear to me why it's using a MonitoredTask in the first place.
> MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-5702
> URL: https://issues.apache.org/jira/browse/HBASE-5702
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
>
> This bug is so easy to reproduce I'm wondering why it hasn't been reported yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in the master interface one task per stopped region server saying the following:
> |Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340 (since 5sec ago)|
> It's gonna stay there until the master cleans it:
> bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340: status=No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, completionTime=-1 appears to have been leaked
> It's not clear to me why it's using a MonitoredTask in the first place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5702)
MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a
MonitoredTask per call
Posted by "Subbu M Iyer (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Subbu M Iyer reassigned HBASE-5702:
-----------------------------------
Assignee: Subbu M Iyer
I will take a look at it and address it as soon as possible.
> MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-5702
> URL: https://issues.apache.org/jira/browse/HBASE-5702
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: Jean-Daniel Cryans
> Assignee: Subbu M Iyer
> Priority: Critical
> Fix For: 0.94.0, 0.96.0
>
>
> This bug is so easy to reproduce I'm wondering why it hasn't been reported yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in the master interface one task per stopped region server saying the following:
> |Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340 (since 5sec ago)|
> It's gonna stay there until the master cleans it:
> bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340: status=No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, completionTime=-1 appears to have been leaked
> It's not clear to me why it's using a MonitoredTask in the first place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5702)
MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a
MonitoredTask per call
Posted by "Subbu M Iyer (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245650#comment-13245650 ]
Subbu M Iyer commented on HBASE-5702:
-------------------------------------
JD,
Can you share with me the "many more issues with MonitoredTask" that you have mentioned? I believe the problems you have seen are all related to the MonitoredTask reporting during instant schema change process and not with the actual schema change process itself? Please let me know so we can categorize and address the issues accordingly.
> MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-5702
> URL: https://issues.apache.org/jira/browse/HBASE-5702
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: Jean-Daniel Cryans
> Assignee: Subbu M Iyer
> Priority: Critical
> Fix For: 0.94.0, 0.96.0
>
>
> This bug is so easy to reproduce I'm wondering why it hasn't been reported yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in the master interface one task per stopped region server saying the following:
> |Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340 (since 5sec ago)|
> It's gonna stay there until the master cleans it:
> bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340: status=No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, completionTime=-1 appears to have been leaked
> It's not clear to me why it's using a MonitoredTask in the first place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5702)
MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a
MonitoredTask per call
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245496#comment-13245496 ]
Zhihong Yu commented on HBASE-5702:
-----------------------------------
The following method in ServerManager.java doesn't check the current value for hbase.instant.schema.alter.enabled config:
{code}
private void excludeRegionServerFromSchemaChanges(final ServerName serverName) {
this.services.getSchemaChangeTracker()
.excludeRegionServerForSchemaChanges(serverName.getServerName());
}
{code}
> MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-5702
> URL: https://issues.apache.org/jira/browse/HBASE-5702
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.94.0, 0.96.0
>
>
> This bug is so easy to reproduce I'm wondering why it hasn't been reported yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in the master interface one task per stopped region server saying the following:
> |Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340 (since 5sec ago)|
> It's gonna stay there until the master cleans it:
> bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340: status=No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, completionTime=-1 appears to have been leaked
> It's not clear to me why it's using a MonitoredTask in the first place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5702)
MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a
MonitoredTask per call
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5702:
---------------------------------
Priority: Critical (was: Major)
Seems bad, upping to critical.
> MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-5702
> URL: https://issues.apache.org/jira/browse/HBASE-5702
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: Jean-Daniel Cryans
> Priority: Critical
> Fix For: 0.94.0, 0.96.0
>
>
> This bug is so easy to reproduce I'm wondering why it hasn't been reported yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in the master interface one task per stopped region server saying the following:
> |Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340 (since 5sec ago)|
> It's gonna stay there until the master cleans it:
> bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340: status=No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, completionTime=-1 appears to have been leaked
> It's not clear to me why it's using a MonitoredTask in the first place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5702)
MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a
MonitoredTask per call
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244801#comment-13244801 ]
Jean-Daniel Cryans commented on HBASE-5702:
-------------------------------------------
Looking at the code it seems it's leaking in other places... and that message shouldn't even be there in the first place because hbase.instant.schema.alter.enabled isn't enabled on this cluster.
> MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-5702
> URL: https://issues.apache.org/jira/browse/HBASE-5702
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
>
> This bug is so easy to reproduce I'm wondering why it hasn't been reported yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in the master interface one task per stopped region server saying the following:
> |Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340 (since 5sec ago)|
> It's gonna stay there until the master cleans it:
> bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340: status=No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, completionTime=-1 appears to have been leaked
> It's not clear to me why it's using a MonitoredTask in the first place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5702)
MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a
MonitoredTask per call
Posted by "Lars Hofhansl (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl updated HBASE-5702:
---------------------------------
Fix Version/s: (was: 0.96.0)
(was: 0.94.0)
Unscheduling, since HBASE-4213 was reverted.
> MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-5702
> URL: https://issues.apache.org/jira/browse/HBASE-5702
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: Jean-Daniel Cryans
> Assignee: Subbu M Iyer
> Priority: Critical
>
> This bug is so easy to reproduce I'm wondering why it hasn't been reported yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in the master interface one task per stopped region server saying the following:
> |Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340 (since 5sec ago)|
> It's gonna stay there until the master cleans it:
> bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340: status=No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, completionTime=-1 appears to have been leaked
> It's not clear to me why it's using a MonitoredTask in the first place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5702)
MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a
MonitoredTask per call
Posted by "Jean-Daniel Cryans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244816#comment-13244816 ]
Jean-Daniel Cryans commented on HBASE-5702:
-------------------------------------------
Now that I've tested "instant" schema updates, there's many more issues with MonitoredTask than I originally thought :(
> MasterSchemaChangeTracker.excludeRegionServerForSchemaChanges leaks a MonitoredTask per call
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-5702
> URL: https://issues.apache.org/jira/browse/HBASE-5702
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.94.0
> Reporter: Jean-Daniel Cryans
> Fix For: 0.94.0, 0.96.0
>
>
> This bug is so easy to reproduce I'm wondering why it hasn't been reported yet. Stop any number of region servers on a 0.94/6 cluster and you'll see in the master interface one task per stopped region server saying the following:
> |Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340|RUNNING (since 5sec ago)|No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340 (since 5sec ago)|
> It's gonna stay there until the master cleans it:
> bq. WARN org.apache.hadoop.hbase.monitoring.TaskMonitor: Status Processing schema change exclusion for region server = sv4r27s44,62023,1333402175340: status=No schema change in progress. Skipping exclusion for server = sv4r27s44,62023,1333402175340, state=RUNNING, startTime=1333404636419, completionTime=-1 appears to have been leaked
> It's not clear to me why it's using a MonitoredTask in the first place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira