You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Michaela Buergle (JIRA)" <ji...@apache.org> on 2008/05/15 11:59:58 UTC
[jira] Created: (HBASE-627) Disable table doesn't work reliably
Disable table doesn't work reliably
-----------------------------------
Key: HBASE-627
URL: https://issues.apache.org/jira/browse/HBASE-627
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.2.0
Environment: Hadoop/HBase on two nodes
Reporter: Michaela Buergle
Priority: Critical
Fix For: 0.2.0
When creating a couple of tables like this:
1) create an empty table
2) disable table, add new column family, enable table
3) put 100 small documents into newly created column
around once in 10 tries the disable doesn't happen.
I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
- after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
...
2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
...
- the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-627) Disable table doesn't work reliably
Posted by "Michaela Buergle (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12607244#action_12607244 ]
Michaela Buergle commented on HBASE-627:
----------------------------------------
In my setting, the disabling-problem described above doesn't seem to happen now as there is hardly any reassignment of regions anymore. So this measure certainly decreases the probability that the error will arise. I'm not sure however that it really solves the issue, since the underlying problem was outdated information in .META., while HBASE-615 affects only the conditions under which the problem occurrs. There might be other line-ups leading to this situation.
Using HTable.isTableOnline/Offline in a loop is probably a good way to avoid the problem. A few remarks on that:
- The "// Wait until first region is disabled" - section in HBaseAdmin.disableTable() looks very similar. Wouldn't it make sense to use isTableOffline() here instead?
- In both disableTable() and isTableOn-/Offline(), regions of other tables than the requested one are checked for being on-/offline (see Description of HBASE-627). That doesn't seem right to me. If any one of those other regions is on-/offline by chance, there will be a wrong result.
- Shouldn't a table be either online or offline? In that case, why are there two different functions for determining the table's state instead of using e.g. !isTableOnline().
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-627) Disable table doesn't work reliably
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597999#action_12597999 ]
stack commented on HBASE-627:
-----------------------------
Thanks for digging in Michaela.
Yeah, its a fundamental issue somewhat recognized in HBASE-543. I think HBASE-543 needs to do more than make state changes atomic; it should also add logic that would prevent or intercept nonsensical transitions (e.g., from above, allowing an offlined table go online).
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-627) Disable table doesn't work reliably
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-627:
--------------------------------
Attachment: patch.txt
Added fix for reliable column operations and table deletion
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log, patch.txt, patch.txt
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-627) Disable table doesn't work reliably
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608931#action_12608931 ]
stack commented on HBASE-627:
-----------------------------
Thanks
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log, patch.txt, patch.txt
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-627) Disable table doesn't work reliably
Posted by "Michaela Buergle (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597984#action_12597984 ]
Michaela Buergle commented on HBASE-627:
----------------------------------------
Some debugging later I now have a clearer idea of what's happening after ServerManager, in his load balancing activities, receives MSG_REPORT_CLOSE for region xy from node1:
- RegionManager assigns region xy to node2 and sets it to "unassigned" (towards the end of assignRegionsToMultipleServers())
- At this point, HBaseAdmin.disableTable is called; in .META., region xy still has info:server node1, region xy is marked as beingServed by node1 in ProcessTableOperation.call() and thus added to the local kill list of node1
- Now we wait for region xy to go offline. But in the meantime, region xy opens on node2. .META. changes, region xy now has info:server node2
So there is a short period during which the information in .META. is not consistent with the actual state of regions. But disableTable() relies on the information found in .META. How could this best be solved? It looks like quite a fundamental problem to me.
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-627) Disable table doesn't work reliably
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-627:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed.
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log, patch.txt, patch.txt
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-627) Disable table doesn't work reliably
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-627:
--------------------------------
Status: Patch Available (was: Open)
patch.txt should address remaining issues. All tests pass locally. Please review.
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log, patch.txt
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-627) Disable table doesn't work reliably
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597201#action_12597201 ]
Jim Kellerman commented on HBASE-627:
-------------------------------------
Are you sure you have updated trunk past the patch for this issue? The same patch was applied to both trunk and branch, so it should work *provided* region assignment is not oscillating. If region assignment is oscillating, enable/disable do not work reliably, and probably won't until we fix that problem.
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-627) Disable table doesn't work reliably
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-627:
--------------------------------
Status: Open (was: Patch Available)
Make column operations and table deletion reliable
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log, patch.txt
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-627) Disable table doesn't work reliably
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608904#action_12608904 ]
Jean-Daniel Cryans commented on HBASE-627:
------------------------------------------
Passes test that it should. +1
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log, patch.txt, patch.txt
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-627) Disable table doesn't work reliably
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608866#action_12608866 ]
Jim Kellerman commented on HBASE-627:
-------------------------------------
Modifying TableOperation$ProcessTableOperation.call to insert the region into the unservedRegions list if it is not being served or is not enabled will fix this problem.
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log, patch.txt
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-627) Disable table doesn't work reliably
Posted by "Michaela Buergle (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12597911#action_12597911 ]
Michaela Buergle commented on HBASE-627:
----------------------------------------
Yes, I am sure I've updated. I can still observe the behaviour described above.
I did read in HBASE-615 that there might be some oscillation due to balancing but the fact that this oscillation affects enabling/disabling of tables has not been mentioned to my knowledge.
In my case, I wouldn't speak of oscillation though. Often it is just one region being assigned to the other one of my two nodes, which is certainly not unusual after having created a table. After all this is what the balancing is about. Still this reassignment and the more or less simultaneous disabling don't seem to go together well.
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-627) Disable table doesn't work reliably
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-627:
--------------------------------
Attachment: patch.txt
HConnection
- tableExists now throws MasterNotRunningException if instance is not on-line
- added method isTableEnabled
HConnectionManager
- tableExists, getRegionLocation, listTables, locateRegion, relocateRegion, getHRegionConnection, getRegionServerWithRetries now check to see if master is running
- added method isTableEnabled
- getHTableDescriptor now checks if table exists, uses ScannerCallable instead of HTable.getScanner
HBaseAdmin
- enableTable, disableTable are now synchronous and wait until entire table is enabled/disabled and use HConnection.isTableEnabled to make the check
- added methods isTableEnabled(Text), isTableEnabled(String), isTableEnabled(byte[]). They use HConnection.isTableEnabled
HTable
- renamed isTableOnline methods to isTableEnabled, now use HConnection.isTableEnabled
- removed isTableOffline methods since they are redundant
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log, patch.txt
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-627) Disable table doesn't work reliably
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HBASE-627:
--------------------------------
Status: Patch Available (was: Open)
New patch. Please review.
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log, patch.txt, patch.txt
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-627) Disable table doesn't work reliably
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608864#action_12608864 ]
Jim Kellerman commented on HBASE-627:
-------------------------------------
> stack - 27/Jun/08 08:18 AM
> HBASE-713 is about offliining/onlining not being reliiable on branch. I got around the
> issue by writing a script that went through the regions that wouldn't offline individually
> offlining them. I was then able to add my new column. But I just noticed that the add
> column didn't apply to all regions, just to most. Would suggest that this issue also
> cover this case; i.e. not just enabling/disabling but also guarantee that
> add/edit/delete of columns works too.
This is a bug in both column operations and table deletion.
If they think a region is being served that region is not added to the list of regions to process.
Unfortunately, determination of whether a region is being served is imprecise, because it
is based on info:server and info:startcode matching a server that the master knows about.
When a region is offlined, it should delete the server and startcode fields. Apparently sometimes, these fields are not deleted and those regions never get processed, even though the region info indicates the region is disabled.
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log, patch.txt
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-627) Disable table doesn't work reliably
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman reassigned HBASE-627:
-----------------------------------
Assignee: Jim Kellerman
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-627) Disable table doesn't work reliably
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608924#action_12608924 ]
stack commented on HBASE-627:
-----------------------------
J-D: Did you review Jim's patch also? If you have the time, do so. Give it +/- 1 so Jim can (or cannot) commit. Good stuff.
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log, patch.txt, patch.txt
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-627) Disable table doesn't work reliably
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12606845#action_12606845 ]
Jim Kellerman commented on HBASE-627:
-------------------------------------
Now that HBASE-615 has been fixed and the master stabilizes on region assignments, disable table and enable table appear to work reliably.
Remember that both enable and disable table run asychronously in the master so if you immediately follow either operation with with another, you may get table is not disabled or table is not enabled exceptions.
In the shell, if you do enable table and immediately follow that with a disable table, some regions may still be on-line because they were in transition during the disable operation. Issuing a second disable table should disable the remaining on-line regions.
>From the API you can use HTable.isTableOnline or HTable.isTableOffline to determine a table's state. You can then wait in a loop for the operation to finish.
We believe that these operations are now reliable. Can you confirm?
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-627) Disable table doesn't work reliably
Posted by "Michaela Buergle (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michaela Buergle updated HBASE-627:
-----------------------------------
Attachment: disableTable31.log
This is the log output for table31, one of the tables for which disable didn't work.
I've noticed that this table's region was added to the kill list of node2, but had previously been reassigned to node1 and was opened there shortly afterwards. This does look like a timing problem with load balancing.
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-627) Disable table doesn't work reliably
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608803#action_12608803 ]
stack commented on HBASE-627:
-----------------------------
HBASE-713 is about offliining/onlining not being reliiable on branch. I got around the issue by writing a script that went through the regions that wouldn't offline individually offlining them. I was then able to add my new column. But I just noticed that the add column didn't apply to all regions, just to most. Would suggest that this issue also cover this case; i.e. not just enabling/disabling but also guarantee that add/edit/delete of columns works too.
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log, patch.txt
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-627) Disable table doesn't work reliably
Posted by "Michaela Buergle (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michaela Buergle updated HBASE-627:
-----------------------------------
Attachment: disableTable5.log
I've attached another case. Here table5 is being reassigned to the same server (see disableTable5.log). In the log I find:
- RegionManager: Going to close region table5,,1210232134304
- ServerManager: Received MSG_REPORT_CLOSE : table5
- TableOperation: adding region table5,,1210232134304 to kill list
- ServerManager: Received MSG_REPORT_OPEN : table5
>From now on the RegionManager says: "Skipping region table5,,1210232134304 because it is already closing." when selecting regions for reassignment.
And in HBaseAdmin.disableTable, we are "Waiting for first region to be disabled from table5".
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-627) Disable table doesn't work reliably
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608926#action_12608926 ]
Jean-Daniel Cryans commented on HBASE-627:
------------------------------------------
Yes, patch was reviewed. +1
> Disable table doesn't work reliably
> -----------------------------------
>
> Key: HBASE-627
> URL: https://issues.apache.org/jira/browse/HBASE-627
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.2.0
> Environment: Hadoop/HBase on two nodes
> Reporter: Michaela Buergle
> Assignee: Jim Kellerman
> Priority: Critical
> Fix For: 0.2.0
>
> Attachments: disableTable31.log, disableTable5.log, patch.txt, patch.txt
>
>
> When creating a couple of tables like this:
> 1) create an empty table
> 2) disable table, add new column family, enable table
> 3) put 100 small documents into newly created column
> around once in 10 tries the disable doesn't happen.
> I have no clue as to why the table isn't disabled in the first place, but if this occurs, two things in HBaseAdmin.disableTable() strike me as odd:
> - after numRetries tries to wait for disabling we exit the loop; there is no exception or error message:
> ...
> 2008-05-14 16:19:47,903 INFO org.apache.hadoop.hbase.client.HBaseAdmin: Disabled table table31
> 2008-05-14 16:19:47,910 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 60000, call addColumn(table31, {name: document, max versions: 3, compression: NONE, in memory: false, block cache enabled: false, max length: 2147483647, time to live: FOREVER, bloom filter: none}) from XXX.XX.40.36:47116: error: org.apache.hadoop.hbase.TableNotDisabledException: table31
> ...
> - the scanner iterates over HRegionInfos of several tables. If any one of those is disabled, we also leave the loop as if the requested table had been disabled.
> I've had this disabling problem occur quite reliably over the last days - today I couldn't reproduce it, though HBase version hasn't changed. ???
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.