You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Vidhyashankar Venkataraman (JIRA)" <ji...@apache.org> on 2011/05/20 00:49:47 UTC
[jira] [Created] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
HConnection.isTableAvailable returns true even with not all regions available.
------------------------------------------------------------------------------
Key: HBASE-3904
URL: https://issues.apache.org/jira/browse/HBASE-3904
Project: HBase
Issue Type: Bug
Components: client
Reporter: Vidhyashankar Venkataraman
Priority: Minor
This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3904:
-------------------------
Comment: was deleted
(was: Ok, I tested your patch with the code attached below:
And I get the following output:
Caught Socket timeout.. Mostly caused by a slow region assignment by master!
11/05/20 23:26:00 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=b3110640.yst.yahoo.net:44481,b3110600.yst.yahoo.net:44481,b3110560.yst.yahoo.net:44481,b3110520.yst.yahoo.net:44481,b3110680.yst.yahoo.net:44481 sessionTimeout=180000 watcher=hconnection
11/05/20 23:26:00 INFO zookeeper.ClientCnxn: Opening socket connection to server b3110560.yst.yahoo.net/67.195.55.234:44481
11/05/20 23:26:00 INFO zookeeper.ClientCnxn: Socket connection established to b3110560.yst.yahoo.net/67.195.55.234:44481, initiating session
11/05/20 23:26:00 INFO zookeeper.ClientCnxn: Session establishment complete on server b3110560.yst.yahoo.net/67.195.55.234:44481, sessionid = 0x12ff6d3911179e8, negotiated timeout = 180000
Table test-v6 not yet available... Sleeping for 5 more minutes... Expected #regions = 17933
Table is probably available!! : test-v6 Available? true
Table test-v6 may not be available... Double checking: Sleeping for 5 minutes more...
Table test-v6: Expected # Regions = 17933 Actual number = 4744
Table test-v6 may not be available... Double checking: Sleeping for 5 minutes more...
And it is still trying to assign.
1. The good: Notice that tableAvailable got out of the loop because it was true and it also printed true in the following print message. This has never happened without the patch.
2. The doubtful part: isTableAvailable still doesn't return back when all regions are online as we see in the subsequent output.
Can you let me know what your patch intended to do?
Thank you
Vidhya
THE CODE:
try {
hbAdmin.createTableAsync(htd, keysArray.toArray(new byte[0][0]));
} catch (java.net.SocketTimeoutException e) {
System.err.println("Caught Socket timeout.. " +
"Mostly caused by a slow region assignment by master!");
}
HTable table = new HTable(tableName);
HConnection conn = table.getConnection();
do {
System.out.println("Table " + tableName + "not yet available... " +
"Sleeping for 5 more minutes... Expected #regions = " +
(keysArray.size()+1));
Thread.sleep(300000);
} while (!conn.isTableAvailable(table.getTableName()));
System.err.println("Table is probably available!! : " +
tableName +
" Available? " +
conn.isTableAvailable(table.getTableName()));
Map<HRegionInfo, HServerAddress> regionList = null;
do {
System.out.println("Table " + tableName + "may not be available... " +
"Double checking: Sleeping for 5 minutes more...");
Thread.sleep(300000);
regionList = table.getRegionsInfo();
System.out.println("Table " + tableName + ": Expected # Regions = " +
(keysArray.size()+1) +
" Actual number = " +
((regionList!=null)?regionList.size():0) );
} while ((regionList==null) ||
(regionList.size()!=(keysArray.size()+1)));
On 5/20/11 4:19 PM, "Ted Yu (JIRA)" <ji...@apache.org> wrote:
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037150#comment-13037150 ]
Ted Yu commented on HBASE-3904:
-------------------------------
I have run tests related to table creation and availability checking.
Namely this code in LoadIncrementalHFiles:
{code}
while (!conn.isTableAvailable(table.getTableName()) && (ctr<TABLE_CREATE_MAX_RETRIES)) {
{code}
TestHFileOutputFormat, TestLoadIncrementalHFiles and TestAdmin.
Please outline what more test(s) should be devised.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
)
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
> Attachments: 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ita Pai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ita Pai updated HBASE-3904:
---------------------------
Attachment: 3904_v6_trunk.patch
The file 3904_v6.patch is for 0.90 branch. Added the patch 3904_v6_trunk.patch for TRUNK.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt, 3904_v6.patch, 3904_v6_trunk.patch
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057384#comment-13057384 ]
Nicolas Spiegelberg commented on HBASE-3904:
--------------------------------------------
@Ted: Are all these versions different patches or revisions on the same patch? Maybe saying part1, part2 would clarify. We were working on the createTable part of this issue internally as well. We'll peer review.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3904:
--------------------------
Attachment: (was: 3904-v5.txt)
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037150#comment-13037150 ]
Ted Yu commented on HBASE-3904:
-------------------------------
I have run tests related to table creation and availability checking.
Namely this code in LoadIncrementalHFiles:
{code}
while (!conn.isTableAvailable(table.getTableName()) && (ctr<TABLE_CREATE_MAX_RETRIES)) {
{code}
TestHFileOutputFormat, TestLoadIncrementalHFiles and TestAdmin.
Please outline what more test(s) should be devised.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
> Attachments: 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057510#comment-13057510 ]
Jean-Daniel Cryans commented on HBASE-3904:
-------------------------------------------
On the patch:
- The visitor looks completely copy pasted HTable.getRegionsInfo, should be refactored.
- Inside the visitor it creates a Addressing.createInetSocketAddressFromHostAndPortStr, which IIRC does a DNS lookup when creating the InetSocketAddress. Now let's say you create a 10k regions table, and you need to loop a few times, I'm afraid this call will become very costly. I believe you don't even need to create the HSA, all you want is to check if something is there or not right?
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064478#comment-13064478 ]
Hudson commented on HBASE-3904:
-------------------------------
Integrated in HBase-TRUNK #2024 (See [https://builds.apache.org/job/HBase-TRUNK/2024/])
HBASE-3904 Reapply patch v6 for TRUNK with addendum
HBASE-3904 Revert patch v6 from TRUNK
HBASE-3904 info.getTableName() should be used to get table name
tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt, 3904_v6.patch, 3904_v6_trunk.patch
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064315#comment-13064315 ]
Ted Yu commented on HBASE-3904:
-------------------------------
Ran patch v6 through all 0.90 tests.
Integrated patch v6 to branch and TRUNK.
Thanks Ita for the patches.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt, 3904_v6.patch, 3904_v6_trunk.patch
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Vidhyashankar Venkataraman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13040395#comment-13040395 ]
Vidhyashankar Venkataraman commented on HBASE-3904:
---------------------------------------------------
I tested the patch while creating a table with 10000 regions using createTableAsync. Tested it thrice and isTableAvailable worked as expected. (the only weird thing was createTableAsync always threw a Socket timeout exception and I had to catch it and move on. Of course that should be raised somewhere else I guess).
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036947#comment-13036947 ]
stack commented on HBASE-3904:
------------------------------
bq. From what I read in the isTableAvailable function, the Metascanvisitor ensures that if there is at least one region not assigned, then the function will return false.
That and at least one region must be assigned (where 'assigned' is a non-null server column which is a far from definitive test of assignedness).
bq. This isn't enough since the createTable function in master assigns one region after another. (Refer to HMAster.createTable(final HRegionInfo [] newRegions, boolean sync))
Yes, it adds regions one at at time to .META. but then uses the bulk assign engine (this was a recent addition by Ted -- do you have this?)
bq. Hence there might be a case when all regions are indeed fully assigned in META but it is just that the master is yet to populate META with the rest of the regions.
Is this so? We add the regions to .META. before we assign. On add to .META. they will have an empty server field so isTableAssigned should be returning false.
I wonder if this check inside in HBaseAdmin#isTableAssigned is 'off':
{code}
if (value == null) {
available.set(false);
return false;
}
{code}
Maybe the value is 'empty', zero-length byte array. We should check for that? Perhaps this is why you got "...inconsistent responses from isTableAvailable."
bq. Therefor for isTableAvailable to work correctly with createTable(splitkeys), the master will have to populate all the regions in meta first before assigning them.
Unless I'm reading it wrong, this is what it *is* doing. Something else is up (maybe the above check?).
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036913#comment-13036913 ]
Ted Yu commented on HBASE-3904:
-------------------------------
In Master, we have:
{code}
public void createTable(HTableDescriptor desc, byte [][] splitKeys)
throws IOException {
createTable(desc, splitKeys, false);
}
{code}
The above is called by HBaseAdmin.createTableAsync() - meaning it is asynchronous.
In order to keep backward compatibility, I propose introducing new methods in HBaseAdmin:
{code}
public void createTableSync(HTableDescriptor desc);
public void createTableSync(HTableDescriptor desc, byte [][] splitKeys);
{code}
They would end up with Master calling:
{code}
public void createTableSync(HTableDescriptor desc, byte [][] splitKeys)
throws IOException {
createTable(desc, splitKeys, true);
}
{code}
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3904:
--------------------------
Attachment: 3904-v5.txt
Version 5 added check in HBA.createTable(final HTableDescriptor desc, byte [][] splitKeys) to wait for the number of regions online equal to initial region count.
TestAdmin passes.
There is only one createTable() method in HMasterInterface which is currently asynchronous and is called by HBA.createTableAsync(). Please advise whether createTableAsync() should be added to HMasterInterface.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ita Pai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ita Pai updated HBASE-3904:
---------------------------
Attachment: (was: 3904_v6_trunk.patch)
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt, 3904_v6.patch
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036948#comment-13036948 ]
stack commented on HBASE-3904:
------------------------------
@Ted What issue are you trying to fix? Thanks.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3904:
--------------------------
Attachment: 3904-v5.txt
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038380#comment-13038380 ]
stack commented on HBASE-3904:
------------------------------
@Ted Seems good. Lets get Vidhya to try it.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
> Attachments: 3904-v2.txt, 3904-v3.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036968#comment-13036968 ]
Ted Yu commented on HBASE-3904:
-------------------------------
Looking at MetaEditor.addRegionToMeta() which is called by HMaster.createTable():
{code}
public static void addRegionToMeta(CatalogTracker catalogTracker,
HRegionInfo regionInfo)
throws IOException {
Put put = new Put(regionInfo.getRegionName());
put.add(HConstants.CATALOG_FAMILY, HConstants.REGIONINFO_QUALIFIER,
Writables.getBytes(regionInfo));
{code}
Server info was initially omitted.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu reassigned HBASE-3904:
-----------------------------
Assignee: Ted Yu
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3904:
--------------------------
Attachment: 3904-v4.txt
Version 3 uses String form for the value of NUM_INITIAL_REGS so that the display on Web UI is meaningful.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057423#comment-13057423 ]
Ted Yu commented on HBASE-3904:
-------------------------------
Vidhyashankar verified that version 4 works. But J-D doesn't like populating values map in HTD.
So I came up with patch version 5 which actually fixes HBASE-4041.
Comment is welcome to move this forward.
Thanks for your attention Nicholas.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050086#comment-13050086 ]
Ted Yu commented on HBASE-3904:
-------------------------------
Using HTD is a way of passing region count for subsequent check.
(See Stack's comment @ 20/May/11 05:21)
We can also introduce createTableSync() which waits for initial region creation to complete before returning. (My comment @ 20/May/11 16:26)
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3904:
--------------------------
Attachment: (was: 3904-v2.txt)
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ita Pai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ita Pai updated HBASE-3904:
---------------------------
Attachment: 3904_v6_trunk.patch
Re-attached v6 patch for TRUNK with the correction.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt, 3904_v6.patch, 3904_v6_trunk.patch
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050155#comment-13050155 ]
Jean-Daniel Cryans commented on HBASE-3904:
-------------------------------------------
Creating createTableSync() wouldn't make much sense since the HBA.createTable() methods are per their javadoc already synchronized with the creation of the table. To me this means that they aren't respecting their contract so I wouldn't worry about keeping backward compatibility.
About HCM.isTableAvailable, I'd think it's fine if it returns true even tho not all regions are available in the case where the table creation wasn't completed because the process was async. Because it's async.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3904:
--------------------------
Attachment: (was: 3904-v5.txt)
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064694#comment-13064694 ]
stack commented on HBASE-3904:
------------------------------
bq. I didn't run test suite before committing patch v6 for TRUNK.
np. I've never broken the build (smile).
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt, 3904_v6.patch, 3904_v6_trunk.patch
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3904:
--------------------------
Attachment: 3904-v5.txt
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060708#comment-13060708 ]
jiraposter@reviews.apache.org commented on HBASE-3904:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1016/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
Check is added in HBA.createTable(final HTableDescriptor desc, byte [][] splitKeys) to wait for the number of regions online equal to initial region count.
This makes HBA.createTable() synchronous.
This addresses bug HBASE-3904.
https://issues.apache.org/jira/browse/HBASE-3904
Diffs
-----
/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 1141318
Diff: https://reviews.apache.org/r/1016/diff
Testing
-------
TestAdmin passes.
Thanks,
Ted
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038173#comment-13038173 ]
stack commented on HBASE-3904:
------------------------------
Adding the getter and setter getNumberOfInitialRegions to HTD is a little odd. You are using HTD to carry a message used by the create table where you pass lots of regions. Seems like an odd thing to expose as public new methods. If you need to pass this message, perhaps use the generic get/set attributes.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
> Attachments: 3904-v2.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ita Pai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ita Pai updated HBASE-3904:
---------------------------
Attachment: 3904_v6.patch
The calls in the try block don't reliably throw RegionException, so most of the time after several retries, it would silently returns.
Attached is another version that fixed this problem, along with some minor changes to the logic.
TestAdmin passed after applying this patch.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt, 3904_v6.patch
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037005#comment-13037005 ]
stack commented on HBASE-3904:
------------------------------
bq. What Vidhyashankar meant was that the existing entries for the table in .META. carried server information, but there were more regions to be assigned by Master which weren't in .META. yet.
I'm not sure how this is possible but I may not have done sufficient study.
bq. Server info was initially omitted.
Yes, thats how it works. Later, on assignment, the server gets filled in.
On your patch:
{code}
+ if (!(info.isOffline() || info.isSplit())) {
+ regionCount.incrementAndGet();
+ }
{code}
This is good Ted.
This is good too --> '+ MetaScanner.metaScan(conf, visitor, tableName);'
Vidhya, do you want to try Ted's patch? Its client-side only patch.
@Ted A unit test would be nice.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
> Attachments: 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3904:
--------------------------
Attachment: 3904-v5.txt
Updated patch version 5 according to J-D's comment.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Vidhyashankar Venkataraman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037163#comment-13037163 ]
Vidhyashankar Venkataraman commented on HBASE-3904:
---------------------------------------------------
Ok, I tested your patch with the code attached below:
And I get the following output:
Caught Socket timeout.. Mostly caused by a slow region assignment by master!
11/05/20 23:26:00 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=b3110640.yst.yahoo.net:44481,b3110600.yst.yahoo.net:44481,b3110560.yst.yahoo.net:44481,b3110520.yst.yahoo.net:44481,b3110680.yst.yahoo.net:44481 sessionTimeout=180000 watcher=hconnection
11/05/20 23:26:00 INFO zookeeper.ClientCnxn: Opening socket connection to server b3110560.yst.yahoo.net/67.195.55.234:44481
11/05/20 23:26:00 INFO zookeeper.ClientCnxn: Socket connection established to b3110560.yst.yahoo.net/67.195.55.234:44481, initiating session
11/05/20 23:26:00 INFO zookeeper.ClientCnxn: Session establishment complete on server b3110560.yst.yahoo.net/67.195.55.234:44481, sessionid = 0x12ff6d3911179e8, negotiated timeout = 180000
Table test-v6 not yet available... Sleeping for 5 more minutes... Expected #regions = 17933
Table is probably available!! : test-v6 Available? true
Table test-v6 may not be available... Double checking: Sleeping for 5 minutes more...
Table test-v6: Expected # Regions = 17933 Actual number = 4744
Table test-v6 may not be available... Double checking: Sleeping for 5 minutes more...
And it is still trying to assign.
1. The good: Notice that tableAvailable got out of the loop because it was true and it also printed true in the following print message. This has never happened without the patch.
2. The doubtful part: isTableAvailable still doesn't return back when all regions are online as we see in the subsequent output.
Can you let me know what your patch intended to do?
Thank you
Vidhya
THE CODE:
try {
hbAdmin.createTableAsync(htd, keysArray.toArray(new byte[0][0]));
} catch (java.net.SocketTimeoutException e) {
System.err.println("Caught Socket timeout.. " +
"Mostly caused by a slow region assignment by master!");
}
HTable table = new HTable(tableName);
HConnection conn = table.getConnection();
do {
System.out.println("Table " + tableName + "not yet available... " +
"Sleeping for 5 more minutes... Expected #regions = " +
(keysArray.size()+1));
Thread.sleep(300000);
} while (!conn.isTableAvailable(table.getTableName()));
System.err.println("Table is probably available!! : " +
tableName +
" Available? " +
conn.isTableAvailable(table.getTableName()));
Map<HRegionInfo, HServerAddress> regionList = null;
do {
System.out.println("Table " + tableName + "may not be available... " +
"Double checking: Sleeping for 5 minutes more...");
Thread.sleep(300000);
regionList = table.getRegionsInfo();
System.out.println("Table " + tableName + ": Expected # Regions = " +
(keysArray.size()+1) +
" Actual number = " +
((regionList!=null)?regionList.size():0) );
} while ((regionList==null) ||
(regionList.size()!=(keysArray.size()+1)));
On 5/20/11 4:19 PM, "Ted Yu (JIRA)" <ji...@apache.org> wrote:
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037150#comment-13037150 ]
Ted Yu commented on HBASE-3904:
-------------------------------
I have run tests related to table creation and availability checking.
Namely this code in LoadIncrementalHFiles:
{code}
while (!conn.isTableAvailable(table.getTableName()) && (ctr<TABLE_CREATE_MAX_RETRIES)) {
{code}
TestHFileOutputFormat, TestLoadIncrementalHFiles and TestAdmin.
Please outline what more test(s) should be devised.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
> Attachments: 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060802#comment-13060802 ]
Ted Yu commented on HBASE-3904:
-------------------------------
All tests passed for 0.90 branch with today's checkins.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Ita Pai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064113#comment-13064113 ]
Ita Pai commented on HBASE-3904:
--------------------------------
"Another version" refers to v6.
It also changes the behavior when user interrupts the sleep.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt, 3904_v6.patch
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060740#comment-13060740 ]
jiraposter@reviews.apache.org commented on HBASE-3904:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1016/#review974
-----------------------------------------------------------
Ship it!
- Andrew
On 2011-07-06 17:30:36, Ted Yu wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1016/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-07-06 17:30:36)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Check is added in HBA.createTable(final HTableDescriptor desc, byte [][] splitKeys) to wait for the number of regions online equal to initial region count.
bq. This makes HBA.createTable() synchronous.
bq.
bq.
bq. This addresses bug HBASE-3904.
bq. https://issues.apache.org/jira/browse/HBASE-3904
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. /src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 1141318
bq.
bq. Diff: https://reviews.apache.org/r/1016/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. TestAdmin passes.
bq.
bq.
bq. Thanks,
bq.
bq. Ted
bq.
bq.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060751#comment-13060751 ]
Ted Yu commented on HBASE-3904:
-------------------------------
Integrate to branch and TRUNK.
Thanks for the review J-D and Andrew.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3904:
--------------------------
Attachment: 3904.txt
This may not be the final fix.
We can use this patch to verify Vidhyashankar's observation.
I borrowed some code from HTable.getRegionsInfo()
Also added check for empty server info as suggested by Stack.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
> Attachments: 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049905#comment-13049905 ]
Jean-Daniel Cryans commented on HBASE-3904:
-------------------------------------------
Putting the number of initial regions in HTD is really odd, it means it's going to show up for every table (created after this patch) in the master web UI, will show when someone does a describe of a table, etc.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Vidhyashankar Venkataraman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036687#comment-13036687 ]
Vidhyashankar Venkataraman commented on HBASE-3904:
---------------------------------------------------
That's exactly what I did with my table existence check! And getRegionsInfo works accurately.
I think we have kind of misunderstood what is actually happening: Or I am wrong this time :)
From what I read in the isTableAvailable function, the Metascanvisitor ensures that if there is at least one region not assigned, then the function will return false. This isn't enough since the createTable function in master assigns one region after another. (Refer to HMAster.createTable(final HRegionInfo [] newRegions, boolean sync)) Hence there might be a case when all regions are indeed fully assigned in META but it is just that the master is yet to populate META with the rest of the regions. I believe that's the reason why I got those inconsistent responses from isTableAvailable.
Therefor for isTableAvailable to work correctly with createTable(splitkeys), the master will have to populate all the regions in meta first before assigning them. I think that will do the trick.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3904:
--------------------------
Attachment: 3904-v5.txt
Updated version 5 according to J-D's comments.
TestAdmin passes.
I want to keep the version number at 5 since I referenced it in HBASE-4041
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3904:
--------------------------
Attachment: (was: 3904-v5.txt)
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050700#comment-13050700 ]
stack commented on HBASE-3904:
------------------------------
Can we do something else? The HBaseAdmin createTable knows how many regions it has to create so in here, it could wait till all are on line? You might want to add an isTableAvailable that takes a count of regions which will only return true if it finds that many or more regions on line?
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064351#comment-13064351 ]
Hudson commented on HBASE-3904:
-------------------------------
Integrated in HBase-TRUNK #2022 (See [https://builds.apache.org/job/HBase-TRUNK/2022/])
HBASE-3904 Addendum that fixes number of retries (Ita Pai)
tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt, 3904_v6.patch, 3904_v6_trunk.patch
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3904:
--------------------------
Attachment: (was: 3904-v5.txt)
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3904:
--------------------------
Attachment: 3904-v2.txt
The second version of patch stores the number of initial regions in HTableDescriptor's values map.
HConnectionManager.isTableAvailable() tries to retrieve this metadata. If it is present, isTableAvailable() would wait for this many regions to be online before returning.
TestAdmin, TestLoadIncrementalHFiles and TestHFileOutputFormat passed.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
> Attachments: 3904-v2.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13037218#comment-13037218 ]
Ted Yu commented on HBASE-3904:
-------------------------------
>From Vidhyashankar's test:
{code}
Table test-v6 not yet available... Sleeping for 5 more minutes... Expected #regions = 17933
Table is probably available!! : test-v6 Available? true
Table test-v6may not be available... Double checking: Sleeping for 5 minutes more...
Table test-v6: Expected # Regions = 17933 Actual number = 4744
{code}
We can see that after conn.isTableAvailable() returned true, there were still at least 13189 regions that were not assigned - not reaching .META.
I think we should implement createTableSync() as I proposed earlier.
We can ask user to call table.getRegionsInfo() but that is not convenient, and getRegionsInfo() is marked deprecated.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
> Attachments: 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated HBASE-3904:
--------------------------
Attachment: 3904-v3.txt
I removed the getNumberOfInitialRegions and switched to the generic getValue/setValue methods.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
> Attachments: 3904-v2.txt, 3904-v3.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036678#comment-13036678 ]
stack commented on HBASE-3904:
------------------------------
So, how do you think this should work Vidhya? isTableAvailable won't ever know how many regions there are supposed to be in a table. Checking for > 0 is probably the best it could ever do. It sounds to me like we need something else or that after isTableAvailable returns true, that then you should get an HTable instance and call http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#getRegionsInfo() in a loop with a bit of wait until the number of regions comes up to close the number you passed to create table? (FYI: I haven't tried this to see if it will work).
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064375#comment-13064375 ]
stack commented on HBASE-3904:
------------------------------
Is this commit good for you Ted? I see NPEs like this:
{code}
testForCheckingIfEnableAndDisableWorksFineAfterSwitch(org.apache.hadoop.hbase.master.TestMasterRestartAfterDisablingTable) Time elapsed: 12.491 sec <<< ERROR!
java.lang.NullPointerException
at org.apache.hadoop.hbase.client.HBaseAdmin$1.processRow(HBaseAdmin.java:348)
at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:212)
{code}
v6 patch messed with processRow.
When I run full suite on trunk I see failures stating with org.apache.hadoop.hbase.client.TestMultipleTimestamps.
If I back out this patch, things work for me again.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt, 3904_v6.patch, 3904_v6_trunk.patch
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064431#comment-13064431 ]
Ted Yu commented on HBASE-3904:
-------------------------------
info.getTableName() should have been used to get table name.
I didn't run test suite before committing patch v6 for TRUNK.
Sorry.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt, 3904_v6.patch, 3904_v6_trunk.patch
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3904) HConnection.isTableAvailable returns
true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu resolved HBASE-3904.
---------------------------
Resolution: Fixed
Release Note:
HBA.createTable(final HTableDescriptor desc, byte [][] splitKeys)
is synchronous. Use this method instead of creating table asynchronously followed by call to HConnection.isTableAvailable().
Hadoop Flags: [Reviewed]
I am closing this JIRA as there lacks consensus how HConnection.isTableAvailable() should be implemented for table which was created asynchronously.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13036959#comment-13036959 ]
Ted Yu commented on HBASE-3904:
-------------------------------
My proposal is based on the observation that Vidhyashankar (and other users) used a loop to check for table availability.
This is equivalent to calling the newly introduced createTableSync() method where there is no need to write the loop above.
bq. Hence there might be a case when all regions are indeed fully assigned in META but it is just that the master is yet to populate META with the rest of the regions.
What Vidhyashankar meant was that the existing entries for the table in .META. carried server information, but there were more regions to be assigned by Master which weren't in .META. yet.
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Priority: Minor
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060808#comment-13060808 ]
Hudson commented on HBASE-3904:
-------------------------------
Integrated in HBase-TRUNK #2007 (See [https://builds.apache.org/job/HBase-TRUNK/2007/])
HBASE-3904 HBA.createTable(final HTableDescriptor desc, byte [][] splitKeys)
should be synchronous
tedyu :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
* /hbase/trunk/CHANGES.txt
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3904) HConnection.isTableAvailable
returns true even with not all regions available.
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13057531#comment-13057531 ]
Jean-Daniel Cryans commented on HBASE-3904:
-------------------------------------------
On the new v5 (should have been v6?):
- I don't think you need to keep that map of regions, all you really want is a count? Like it's creating the map with the hostAndPort possibly null and the rest of the code couldn't care less. I think you could keep a counter and reset it when you loop.
- About that hostAndPort, if it's null should you consider that region as unassigned and not counted as an available region?
> HConnection.isTableAvailable returns true even with not all regions available.
> ------------------------------------------------------------------------------
>
> Key: HBASE-3904
> URL: https://issues.apache.org/jira/browse/HBASE-3904
> Project: HBase
> Issue Type: Bug
> Components: client
> Reporter: Vidhyashankar Venkataraman
> Assignee: Ted Yu
> Priority: Minor
> Attachments: 3904-v3.txt, 3904-v4.txt, 3904-v5.txt, 3904.txt
>
>
> This function as per the java doc is supposed to return true iff "all the regions in the table are available". But if the table is still being created this function may return inconsistent results (For example, when a table with a large number of split keys is created).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira