You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2009/11/10 05:55:32 UTC
[jira] Created: (HBASE-1968) Give clients access to the write
buffer
Give clients access to the write buffer
---------------------------------------
Key: HBASE-1968
URL: https://issues.apache.org/jira/browse/HBASE-1968
Project: Hadoop HBase
Issue Type: Improvement
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Trivial
Fix For: 0.20.2
>From a Trend dev team:
{quote}
When insert rows into one table by calling the method public synchronized void put(final Put put), if the column family of one row does not exist, the insert operation will failed and throw NoSuchColumnFamilyException.. We observed that all the following insert operation will fails even though all of them have valid column family. That is one exception of insert operation can cause failure of all the following insert operation.
{quote}
Their further analysis explains in detail the scenario, which I will summarize here:
1) An invalid put is added to the writeBuffer by put(Put put). It will trigger a NoSuchColumnFamilyException once it goes to the region server.
2) At some point, the buffer is flushed.
3) When the invalid put is processed, an exception is thrown. The finally clause of flushCommits() removes all successful puts from the writebuffer list but the failed put remains at the top. This entry becomes an immovable blocker which prevents any subsequent entry from being processed.
4) Subsequent puts will add more entries to the write buffer until the buffer limit is reached, compounding the problem by allowing more edits to be queued which can never be processed.
A workaround could be for the client to call getWriteBuffer() -- on trunk -- and remove the entry at the head of the list manually, but without the patch on this issue, the client cannot get access to the list on branch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1968) Give clients access to the write
buffer
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775291#action_12775291 ]
Andrew Purtell commented on HBASE-1968:
---------------------------------------
getWriteBuffer is already available on trunk.
> Give clients access to the write buffer
> ---------------------------------------
>
> Key: HBASE-1968
> URL: https://issues.apache.org/jira/browse/HBASE-1968
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Trivial
> Fix For: 0.20.2
>
>
> From a Trend dev team:
> {quote}
> When insert rows into one table by calling the method public synchronized void put(final Put put), if the column family of one row does not exist, the insert operation will failed and throw NoSuchColumnFamilyException.. We observed that all the following insert operation will fails even though all of them have valid column family. That is one exception of insert operation can cause failure of all the following insert operation.
> {quote}
> Their further analysis explains in detail the scenario, which I will summarize here:
> 1) An invalid put is added to the writeBuffer by put(Put put). It will trigger a NoSuchColumnFamilyException once it goes to the region server.
> 2) At some point, the buffer is flushed.
> 3) When the invalid put is processed, an exception is thrown. The finally clause of flushCommits() removes all successful puts from the writebuffer list but the failed put remains at the top. This entry becomes an immovable blocker which prevents any subsequent entry from being processed.
> 4) Subsequent puts will add more entries to the write buffer until the buffer limit is reached, compounding the problem by allowing more edits to be queued which can never be processed.
> A workaround could be for the client to call getWriteBuffer() -- on trunk -- and remove the entry at the head of the list manually, but without the patch on this issue, the client cannot get access to the list on branch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1968) Give clients access to the write
buffer
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775284#action_12775284 ]
Jean-Daniel Cryans commented on HBASE-1968:
-------------------------------------------
Any reason not to apply it on trunk?
> Give clients access to the write buffer
> ---------------------------------------
>
> Key: HBASE-1968
> URL: https://issues.apache.org/jira/browse/HBASE-1968
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Trivial
> Fix For: 0.20.2
>
>
> From a Trend dev team:
> {quote}
> When insert rows into one table by calling the method public synchronized void put(final Put put), if the column family of one row does not exist, the insert operation will failed and throw NoSuchColumnFamilyException.. We observed that all the following insert operation will fails even though all of them have valid column family. That is one exception of insert operation can cause failure of all the following insert operation.
> {quote}
> Their further analysis explains in detail the scenario, which I will summarize here:
> 1) An invalid put is added to the writeBuffer by put(Put put). It will trigger a NoSuchColumnFamilyException once it goes to the region server.
> 2) At some point, the buffer is flushed.
> 3) When the invalid put is processed, an exception is thrown. The finally clause of flushCommits() removes all successful puts from the writebuffer list but the failed put remains at the top. This entry becomes an immovable blocker which prevents any subsequent entry from being processed.
> 4) Subsequent puts will add more entries to the write buffer until the buffer limit is reached, compounding the problem by allowing more edits to be queued which can never be processed.
> A workaround could be for the client to call getWriteBuffer() -- on trunk -- and remove the entry at the head of the list manually, but without the patch on this issue, the client cannot get access to the list on branch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1968) Give clients access to the write
buffer
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775293#action_12775293 ]
Jean-Daniel Cryans commented on HBASE-1968:
-------------------------------------------
Sorry brain lag ;)
> Give clients access to the write buffer
> ---------------------------------------
>
> Key: HBASE-1968
> URL: https://issues.apache.org/jira/browse/HBASE-1968
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Trivial
> Fix For: 0.20.2
>
>
> From a Trend dev team:
> {quote}
> When insert rows into one table by calling the method public synchronized void put(final Put put), if the column family of one row does not exist, the insert operation will failed and throw NoSuchColumnFamilyException.. We observed that all the following insert operation will fails even though all of them have valid column family. That is one exception of insert operation can cause failure of all the following insert operation.
> {quote}
> Their further analysis explains in detail the scenario, which I will summarize here:
> 1) An invalid put is added to the writeBuffer by put(Put put). It will trigger a NoSuchColumnFamilyException once it goes to the region server.
> 2) At some point, the buffer is flushed.
> 3) When the invalid put is processed, an exception is thrown. The finally clause of flushCommits() removes all successful puts from the writebuffer list but the failed put remains at the top. This entry becomes an immovable blocker which prevents any subsequent entry from being processed.
> 4) Subsequent puts will add more entries to the write buffer until the buffer limit is reached, compounding the problem by allowing more edits to be queued which can never be processed.
> A workaround could be for the client to call getWriteBuffer() -- on trunk -- and remove the entry at the head of the list manually, but without the patch on this issue, the client cannot get access to the list on branch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1968) Give clients access to the write
buffer
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell resolved HBASE-1968.
-----------------------------------
Resolution: Fixed
Committed trivial patch to 0.20 branch.
> Give clients access to the write buffer
> ---------------------------------------
>
> Key: HBASE-1968
> URL: https://issues.apache.org/jira/browse/HBASE-1968
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Trivial
> Fix For: 0.20.2
>
>
> From a Trend dev team:
> {quote}
> When insert rows into one table by calling the method public synchronized void put(final Put put), if the column family of one row does not exist, the insert operation will failed and throw NoSuchColumnFamilyException.. We observed that all the following insert operation will fails even though all of them have valid column family. That is one exception of insert operation can cause failure of all the following insert operation.
> {quote}
> Their further analysis explains in detail the scenario, which I will summarize here:
> 1) An invalid put is added to the writeBuffer by put(Put put). It will trigger a NoSuchColumnFamilyException once it goes to the region server.
> 2) At some point, the buffer is flushed.
> 3) When the invalid put is processed, an exception is thrown. The finally clause of flushCommits() removes all successful puts from the writebuffer list but the failed put remains at the top. This entry becomes an immovable blocker which prevents any subsequent entry from being processed.
> 4) Subsequent puts will add more entries to the write buffer until the buffer limit is reached, compounding the problem by allowing more edits to be queued which can never be processed.
> A workaround could be for the client to call getWriteBuffer() -- on trunk -- and remove the entry at the head of the list manually, but without the patch on this issue, the client cannot get access to the list on branch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1968) Give clients access to the write
buffer
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12775281#action_12775281 ]
Andrew Purtell commented on HBASE-1968:
---------------------------------------
{code}
Index: src/java/org/apache/hadoop/hbase/client/HTable.java
===================================================================
--- src/java/org/apache/hadoop/hbase/client/HTable.java (revision 834067)
+++ src/java/org/apache/hadoop/hbase/client/HTable.java (working copy)
@@ -709,6 +709,14 @@
}
}
+ /**
+ * Get the write buffer
+ * @return the current write buffer
+ */
+ public ArrayList<Put> getWriteBuffer() {
+ return writeBuffer;
+ }
+
// Old API. Pre-hbase-880, hbase-1304.
/**
{code}
> Give clients access to the write buffer
> ---------------------------------------
>
> Key: HBASE-1968
> URL: https://issues.apache.org/jira/browse/HBASE-1968
> Project: Hadoop HBase
> Issue Type: Improvement
> Reporter: Andrew Purtell
> Assignee: Andrew Purtell
> Priority: Trivial
> Fix For: 0.20.2
>
>
> From a Trend dev team:
> {quote}
> When insert rows into one table by calling the method public synchronized void put(final Put put), if the column family of one row does not exist, the insert operation will failed and throw NoSuchColumnFamilyException.. We observed that all the following insert operation will fails even though all of them have valid column family. That is one exception of insert operation can cause failure of all the following insert operation.
> {quote}
> Their further analysis explains in detail the scenario, which I will summarize here:
> 1) An invalid put is added to the writeBuffer by put(Put put). It will trigger a NoSuchColumnFamilyException once it goes to the region server.
> 2) At some point, the buffer is flushed.
> 3) When the invalid put is processed, an exception is thrown. The finally clause of flushCommits() removes all successful puts from the writebuffer list but the failed put remains at the top. This entry becomes an immovable blocker which prevents any subsequent entry from being processed.
> 4) Subsequent puts will add more entries to the write buffer until the buffer limit is reached, compounding the problem by allowing more edits to be queued which can never be processed.
> A workaround could be for the client to call getWriteBuffer() -- on trunk -- and remove the entry at the head of the list manually, but without the patch on this issue, the client cannot get access to the list on branch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.