You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Bradford Stephens (JIRA)" <ji...@apache.org> on 2009/08/25 18:27:59 UTC
[jira] Created: (HBASE-1791) Timeout in IndexRecordWriter
Timeout in IndexRecordWriter
----------------------------
Key: HBASE-1791
URL: https://issues.apache.org/jira/browse/HBASE-1791
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.20.0, 0.20.1, 0.21.0
Environment: 19 HBase nodes, 8 cores, 8 GB RAM, CentOS
Reporter: Bradford Stephens
Priority: Minor
A MapReduce job to generate Lucene Indexes from HBase will fail on sufficiently large tables. After the indexing finished, the close() method of IndexRecordWriter is called. The writer.optimize() call in this method can take many minutes, forcing most MapReduce tasks to timeout. There is a HeartBeatsThread, but it does not seem to send progress updates.
A suggested fix may be to add context.progress(); in the HeardbeatsThread run() method, after the context.setStatus call. Not sure why context.setStatus is not "good enough".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1791) Timeout in IndexRecordWriter
Posted by "Lars George (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747490#action_12747490 ]
Lars George commented on HBASE-1791:
------------------------------------
+1
I attribute this to the new Hadoop mapreduce package and the new context class layout. There is an old documentation that refers to setting the status to report progress but I am not surprised that the new classes rather use the proper progress() call. The unit test is too small to notice this change, so good that Bradford found it!
> Timeout in IndexRecordWriter
> ----------------------------
>
> Key: HBASE-1791
> URL: https://issues.apache.org/jira/browse/HBASE-1791
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: 19 HBase nodes, 8 cores, 8 GB RAM, CentOS
> Reporter: Bradford Stephens
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1791.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> A MapReduce job to generate Lucene Indexes from HBase will fail on sufficiently large tables. After the indexing finished, the close() method of IndexRecordWriter is called. The writer.optimize() call in this method can take many minutes, forcing most MapReduce tasks to timeout. There is a HeartBeatsThread, but it does not seem to send progress updates.
> A suggested fix may be to add context.progress(); in the HeardbeatsThread run() method, after the context.setStatus call. Not sure why context.setStatus is not "good enough".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1791) Timeout in IndexRecordWriter
Posted by "Bradford Stephens (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bradford Stephens updated HBASE-1791:
-------------------------------------
Attachment: hbase-1791.patch
> Timeout in IndexRecordWriter
> ----------------------------
>
> Key: HBASE-1791
> URL: https://issues.apache.org/jira/browse/HBASE-1791
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: 19 HBase nodes, 8 cores, 8 GB RAM, CentOS
> Reporter: Bradford Stephens
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1791.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> A MapReduce job to generate Lucene Indexes from HBase will fail on sufficiently large tables. After the indexing finished, the close() method of IndexRecordWriter is called. The writer.optimize() call in this method can take many minutes, forcing most MapReduce tasks to timeout. There is a HeartBeatsThread, but it does not seem to send progress updates.
> A suggested fix may be to add context.progress(); in the HeardbeatsThread run() method, after the context.setStatus call. Not sure why context.setStatus is not "good enough".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1791) Timeout in IndexRecordWriter
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1791:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Status: Resolved (was: Patch Available)
Committed to branch and trunk. Thanks for the patch Bradford!
Stack, please set up Bradford as a contributor so we can assign this issue to him.
> Timeout in IndexRecordWriter
> ----------------------------
>
> Key: HBASE-1791
> URL: https://issues.apache.org/jira/browse/HBASE-1791
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: 19 HBase nodes, 8 cores, 8 GB RAM, CentOS
> Reporter: Bradford Stephens
> Priority: Minor
> Fix For: 0.20.0, 0.21.0
>
> Attachments: hbase-1791.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> A MapReduce job to generate Lucene Indexes from HBase will fail on sufficiently large tables. After the indexing finished, the close() method of IndexRecordWriter is called. The writer.optimize() call in this method can take many minutes, forcing most MapReduce tasks to timeout. There is a HeartBeatsThread, but it does not seem to send progress updates.
> A suggested fix may be to add context.progress(); in the HeardbeatsThread run() method, after the context.setStatus call. Not sure why context.setStatus is not "good enough".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1791) Timeout in IndexRecordWriter
Posted by "Bradford Stephens (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bradford Stephens updated HBASE-1791:
-------------------------------------
Fix Version/s: 0.20.0
Status: Patch Available (was: Open)
Added context.progress() to HeartbeatsThread, moved context.setStatus to more useful place.
> Timeout in IndexRecordWriter
> ----------------------------
>
> Key: HBASE-1791
> URL: https://issues.apache.org/jira/browse/HBASE-1791
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: 19 HBase nodes, 8 cores, 8 GB RAM, CentOS
> Reporter: Bradford Stephens
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: hbase-1791.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> A MapReduce job to generate Lucene Indexes from HBase will fail on sufficiently large tables. After the indexing finished, the close() method of IndexRecordWriter is called. The writer.optimize() call in this method can take many minutes, forcing most MapReduce tasks to timeout. There is a HeartBeatsThread, but it does not seem to send progress updates.
> A suggested fix may be to add context.progress(); in the HeardbeatsThread run() method, after the context.setStatus call. Not sure why context.setStatus is not "good enough".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1791) Timeout in IndexRecordWriter
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1791:
----------------------------------
Fix Version/s: (was: 0.20.0)
0.20.1
Sorry, set fix version to 0.20.1.
> Timeout in IndexRecordWriter
> ----------------------------
>
> Key: HBASE-1791
> URL: https://issues.apache.org/jira/browse/HBASE-1791
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: 19 HBase nodes, 8 cores, 8 GB RAM, CentOS
> Reporter: Bradford Stephens
> Assignee: Bradford Stephens
> Priority: Minor
> Fix For: 0.20.1, 0.21.0
>
> Attachments: hbase-1791.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> A MapReduce job to generate Lucene Indexes from HBase will fail on sufficiently large tables. After the indexing finished, the close() method of IndexRecordWriter is called. The writer.optimize() call in this method can take many minutes, forcing most MapReduce tasks to timeout. There is a HeartBeatsThread, but it does not seem to send progress updates.
> A suggested fix may be to add context.progress(); in the HeardbeatsThread run() method, after the context.setStatus call. Not sure why context.setStatus is not "good enough".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1791) Timeout in IndexRecordWriter
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans reassigned HBASE-1791:
-----------------------------------------
Assignee: Bradford Stephens
Assigned. Every committer can change the roles.
> Timeout in IndexRecordWriter
> ----------------------------
>
> Key: HBASE-1791
> URL: https://issues.apache.org/jira/browse/HBASE-1791
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: 19 HBase nodes, 8 cores, 8 GB RAM, CentOS
> Reporter: Bradford Stephens
> Assignee: Bradford Stephens
> Priority: Minor
> Fix For: 0.20.0, 0.21.0
>
> Attachments: hbase-1791.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> A MapReduce job to generate Lucene Indexes from HBase will fail on sufficiently large tables. After the indexing finished, the close() method of IndexRecordWriter is called. The writer.optimize() call in this method can take many minutes, forcing most MapReduce tasks to timeout. There is a HeartBeatsThread, but it does not seem to send progress updates.
> A suggested fix may be to add context.progress(); in the HeardbeatsThread run() method, after the context.setStatus call. Not sure why context.setStatus is not "good enough".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1791) Timeout in IndexRecordWriter
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12747555#action_12747555 ]
stack commented on HBASE-1791:
------------------------------
OK that this ain't going to be in 0.20.0? If its needed, need to sink the current RC. Otherwise it'll be in 0.20.1. Good stuff.
> Timeout in IndexRecordWriter
> ----------------------------
>
> Key: HBASE-1791
> URL: https://issues.apache.org/jira/browse/HBASE-1791
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.20.0, 0.20.1, 0.21.0
> Environment: 19 HBase nodes, 8 cores, 8 GB RAM, CentOS
> Reporter: Bradford Stephens
> Assignee: Bradford Stephens
> Priority: Minor
> Fix For: 0.20.0, 0.21.0
>
> Attachments: hbase-1791.patch
>
> Original Estimate: 3h
> Remaining Estimate: 3h
>
> A MapReduce job to generate Lucene Indexes from HBase will fail on sufficiently large tables. After the indexing finished, the close() method of IndexRecordWriter is called. The writer.optimize() call in this method can take many minutes, forcing most MapReduce tasks to timeout. There is a HeartBeatsThread, but it does not seem to send progress updates.
> A suggested fix may be to add context.progress(); in the HeardbeatsThread run() method, after the context.setStatus call. Not sure why context.setStatus is not "good enough".
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.