You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/06/11 02:43:46 UTC

[jira] Created: (HBASE-679) Regionserver addresses are still not right in the new tables page

Regionserver addresses are still not right in the new tables page
-----------------------------------------------------------------

                 Key: HBASE-679
                 URL: https://issues.apache.org/jira/browse/HBASE-679
             Project: Hadoop HBase
          Issue Type: Bug
            Reporter: stack


They are mostly right.

I'm guessing its stale cache of regions in the client hosted by the UI.  If the webserver ran a scan, it'd probably fix it all up but thats a bit messy.  I tried using the address that is in the .META. table directly but that doesn't work.... we don't seem to deploy table properly and UI complains "No server address for row TestTable,,1213074650399".  I'll attach my patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-679) Regionserver addresses are still not right in the new tables page

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-679:
------------------------

    Attachment: ms.patch

Path for metascanner to use .META. server address rather than ask the client cache for what it thinks the address is

> Regionserver addresses are still not right in the new tables page
> -----------------------------------------------------------------
>
>                 Key: HBASE-679
>                 URL: https://issues.apache.org/jira/browse/HBASE-679
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>         Attachments: ms.patch
>
>
> They are mostly right.
> I'm guessing its stale cache of regions in the client hosted by the UI.  If the webserver ran a scan, it'd probably fix it all up but thats a bit messy.  I tried using the address that is in the .META. table directly but that doesn't work.... we don't seem to deploy table properly and UI complains "No server address for row TestTable,,1213074650399".  I'll attach my patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-679) Regionserver addresses are still not right in the new tables page

Posted by "Izaak Rubin (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614551#action_12614551 ] 

Izaak Rubin commented on HBASE-679:
-----------------------------------

I've been going through this issue for a little while now, and I think I have an idea of what ultimately needs to be done.

First, I was wrong in my last comment when I said that the problem had changed from the original issue description.  I've observed the problem in the HBase instance that Jim is running from the aa0-00... machines.  And, after a bit more fiddling with the UI and some data on my own computer, I think I'm seeing trace bits of the same problem as well.

In my opinion, this issue boils down to the exact same problem we saw in HBASE-727.  The problem is this: we are caching information about regionserver location that most of the time will be wrong.  In HBASE-727 we were able to work around this by being a bit hacky - if the regionserver location was wrong and the page was unable to load after several tries, we threw an IOException into the system that caused some gears to churn, and usually this would stir things up enough to get the right location out.  But, being the one who made patch for HBASE-727, I have to admit that this is a bit hacky.

Now, in this issue, the cached info coming back from HRegionInfo.getValue().getBindAddress() is also incorrect.  As with HBASE-727, the data isn't consistently incorrect either - sometimes the address that comes back is correct, and sometimes it isn't.  Sometimes after a few page refreshes it gets it right, and sometimes it continues to be wrong indefinitely.  

There are a number of things that can be done, both for this particular issue and for the larger problem in general.  

For HBASE-679:
 * I can modify table.jsp to fail a bit more gracefully when the wrong address comes back.  We could even put up a picture of the twitter fail whale: http://www.pestaola.gr/img1/twitter-whale.png (just kidding!)
 * We can leave everything as is and just let the user get an anonymous error code 500 in certain cases.  Maybe it's better to tell them nothing if it isn't working?
 * I can try to make some kind of a hacky fix.  I'm not sure this will work though, since throwing an exception from inside a .jsp page won't really do anything.  The fix would have to be in the java code, but then there would be an issue of how to determine if the address is right or not.
 * We can punt on the issue until the overall problem is fixed (see below).

For the greater problem of incorrect caching:
 * I would ask why we are caching this information in the first place.  Information like regionserver host:port is likely to change if the user is shutting down HBase and restarting.  There needs to be some way for every node to be informed of fresh location data when HBase is started.  Admittedly, I am still incredibly naive when it comes to the inner-workings and practicalities of HBase, and I would imagine that this is probably a lot easier said than done.  I'd imagine that this is something that would get pushed to 0.3.  Still, I maintain that this is something worth doing.  This problem continues to manifest itself in interesting ways all over the UI, and if we really want the UI to be a reliable reflection of the data in HBase, we need to make this fix.

> Regionserver addresses are still not right in the new tables page
> -----------------------------------------------------------------
>
>                 Key: HBASE-679
>                 URL: https://issues.apache.org/jira/browse/HBASE-679
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Izaak Rubin
>         Attachments: ms.patch, ms_revised.patch
>
>
> They are mostly right.
> I'm guessing its stale cache of regions in the client hosted by the UI.  If the webserver ran a scan, it'd probably fix it all up but thats a bit messy.  I tried using the address that is in the .META. table directly but that doesn't work.... we don't seem to deploy table properly and UI complains "No server address for row TestTable,,1213074650399".  I'll attach my patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HBASE-679) Regionserver addresses are still not right in the new tables page

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman reassigned HBASE-679:
-----------------------------------

    Assignee: Jim Kellerman  (was: Izaak Rubin)

> Regionserver addresses are still not right in the new tables page
> -----------------------------------------------------------------
>
>                 Key: HBASE-679
>                 URL: https://issues.apache.org/jira/browse/HBASE-679
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Jim Kellerman
>         Attachments: ms.patch, ms_revised.patch
>
>
> They are mostly right.
> I'm guessing its stale cache of regions in the client hosted by the UI.  If the webserver ran a scan, it'd probably fix it all up but thats a bit messy.  I tried using the address that is in the .META. table directly but that doesn't work.... we don't seem to deploy table properly and UI complains "No server address for row TestTable,,1213074650399".  I'll attach my patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (HBASE-679) Regionserver addresses are still not right in the new tables page

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman resolved HBASE-679.
---------------------------------

       Resolution: Fixed
    Fix Version/s: 0.2.0

Passed all regression tests and PerformanceEvaluation randomWrite with 4 region servers. table.jsp now displays the correct region server for each region. Committed.

> Regionserver addresses are still not right in the new tables page
> -----------------------------------------------------------------
>
>                 Key: HBASE-679
>                 URL: https://issues.apache.org/jira/browse/HBASE-679
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.2.0
>
>         Attachments: ms.patch, ms_revised.patch
>
>
> They are mostly right.
> I'm guessing its stale cache of regions in the client hosted by the UI.  If the webserver ran a scan, it'd probably fix it all up but thats a bit messy.  I tried using the address that is in the .META. table directly but that doesn't work.... we don't seem to deploy table properly and UI complains "No server address for row TestTable,,1213074650399".  I'll attach my patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HBASE-679) Regionserver addresses are still not right in the new tables page

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jim Kellerman reassigned HBASE-679:
-----------------------------------

    Assignee: Izaak Rubin

Please verify that this is still an issue.

> Regionserver addresses are still not right in the new tables page
> -----------------------------------------------------------------
>
>                 Key: HBASE-679
>                 URL: https://issues.apache.org/jira/browse/HBASE-679
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Izaak Rubin
>         Attachments: ms.patch
>
>
> They are mostly right.
> I'm guessing its stale cache of regions in the client hosted by the UI.  If the webserver ran a scan, it'd probably fix it all up but thats a bit messy.  I tried using the address that is in the .META. table directly but that doesn't work.... we don't seem to deploy table properly and UI complains "No server address for row TestTable,,1213074650399".  I'll attach my patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-679) Regionserver addresses are still not right in the new tables page

Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12614938#action_12614938 ] 

Jim Kellerman commented on HBASE-679:
-------------------------------------

I am very close to having a patch for this issue. Expect it this weekend.

> Regionserver addresses are still not right in the new tables page
> -----------------------------------------------------------------
>
>                 Key: HBASE-679
>                 URL: https://issues.apache.org/jira/browse/HBASE-679
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Izaak Rubin
>         Attachments: ms.patch, ms_revised.patch
>
>
> They are mostly right.
> I'm guessing its stale cache of regions in the client hosted by the UI.  If the webserver ran a scan, it'd probably fix it all up but thats a bit messy.  I tried using the address that is in the .META. table directly but that doesn't work.... we don't seem to deploy table properly and UI complains "No server address for row TestTable,,1213074650399".  I'll attach my patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-679) Regionserver addresses are still not right in the new tables page

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615053#action_12615053 ] 

stack commented on HBASE-679:
-----------------------------

Nice patch Jim.

> Regionserver addresses are still not right in the new tables page
> -----------------------------------------------------------------
>
>                 Key: HBASE-679
>                 URL: https://issues.apache.org/jira/browse/HBASE-679
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Jim Kellerman
>             Fix For: 0.2.0
>
>         Attachments: ms.patch, ms_revised.patch
>
>
> They are mostly right.
> I'm guessing its stale cache of regions in the client hosted by the UI.  If the webserver ran a scan, it'd probably fix it all up but thats a bit messy.  I tried using the address that is in the .META. table directly but that doesn't work.... we don't seem to deploy table properly and UI complains "No server address for row TestTable,,1213074650399".  I'll attach my patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-679) Regionserver addresses are still not right in the new tables page

Posted by "Izaak Rubin (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Izaak Rubin updated HBASE-679:
------------------------------

    Attachment: ms_revised.patch

These are my observations on the issue:

I can't replicate what the issue is describing.  When I start up HBase and add a large table, everything on table.jsp appears correct.  It's hard to say if there really isn't any problem with the regionserver addresses though, since when I run HBase I'm only using one regionserver.  

There are other problems though.  If I restart HBase, table.jsp won't load at all (error code 500).  The rest of the UI and the regionserver info it shows is fine.

Even worse, if I apply ms.patch, the UI stops working completely.  If I go to the master page (master.jsp), I get an error code 500.  

I don't think we want to use ms.patch, but just in case I've updated it (ms_revised.patch) to align with the current hbase-trunk (since the patch was made about 6 weeks ago).  

So my conclusions would be: something is clearly wrong with table.jsp, although not the same problem that prompted this issue.  If people would like, I'll go in and try to fix this new problem.

> Regionserver addresses are still not right in the new tables page
> -----------------------------------------------------------------
>
>                 Key: HBASE-679
>                 URL: https://issues.apache.org/jira/browse/HBASE-679
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: Izaak Rubin
>         Attachments: ms.patch, ms_revised.patch
>
>
> They are mostly right.
> I'm guessing its stale cache of regions in the client hosted by the UI.  If the webserver ran a scan, it'd probably fix it all up but thats a bit messy.  I tried using the address that is in the .META. table directly but that doesn't work.... we don't seem to deploy table properly and UI complains "No server address for row TestTable,,1213074650399".  I'll attach my patch.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.