You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Jim Kellerman (JIRA)" <ji...@apache.org> on 2007/06/06 19:14:25 UTC
[jira] Created: (HADOOP-1468) Add HBase batch update to reduce RPC
overhead
Add HBase batch update to reduce RPC overhead
---------------------------------------------
Key: HADOOP-1468
URL: https://issues.apache.org/jira/browse/HADOOP-1468
Project: Hadoop
Issue Type: New Feature
Components: contrib/hbase
Affects Versions: 0.14.0
Reporter: Jim Kellerman
Fix For: 0.14.0
On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
Hi,
>
> I'm noticing that since the HClient/HRegionServer interface only allows
> for a per-column put(), there is a lot of RPC and some lease management
> overhead when writing large amounts of data. For example:
>
> for (int i = 0; i < 10000; i++) {
> Text rowKey = new Text(i+"");
> long lock = client.startUpdate(rowKey);
> client.put(lock, COL1, rowKey.getBytes());
> client.put(lock, COL2, someValue.getBytes());
> client.commit(lock);
> }
>
> This code takes my machine (using a single HMaster/HRegionServer on
> local filesystem) approximately 13 seconds to execute. When i measure
> the execution time within HRegionServer.put() I get total time spent in
> put() < 2 seconds. So it looks like there's definately overhead in the
> RPC communication and serialization/deserialization between client and
> server.
>
> To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> 40000 RPC operations.
>
> What I'm thinking, and please tell me if i'm wrong or if this is already
> in the works, is that if I create a row-level put() method that submits
> a map of column values at once, I would reduce the 2 + (#cols) RPC
> operations to one single atomic row-write RPC as well as eliminate the
> small but noticeable overhead in lease creation, renewal, and cancellation.
>
> It's not clear exactly what the performance improvement would be. The
> same amount of serialization/deserilalization must occur, but YourKit
> profiling tells me that the serialization overhead is negligible.
>
> Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (HADOOP-1468) Add HBase batch update to reduce
RPC overhead
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HADOOP-1468 started by Jim Kellerman.
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.14.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.14.0
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1468) Add HBase batch update to reduce RPC
overhead
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HADOOP-1468:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed.
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.15.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.15.0
>
> Attachments: patch.txt, patch.txt
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1468) Add HBase batch update to reduce
RPC overhead
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12515884 ]
Hadoop QA commented on HADOOP-1468:
-----------------------------------
+1
http://issues.apache.org/jira/secure/attachment/12362641/patch.txt applied and successfully tested against trunk revision r559886.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/476/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/476/console
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.15.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.15.0
>
> Attachments: patch.txt, patch.txt
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1468) Add HBase batch update to reduce RPC
overhead
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HADOOP-1468:
----------------------------------
Status: Patch Available (was: In Progress)
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.14.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.14.0
>
> Attachments: patch.txt
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1468) Add HBase batch update to reduce RPC
overhead
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman reassigned HADOOP-1468:
-------------------------------------
Assignee: Jim Kellerman
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.14.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.14.0
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1468) Add HBase batch update to reduce RPC
overhead
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HADOOP-1468:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed.
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.14.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.14.0
>
> Attachments: patch.txt
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1468) Add HBase batch update to reduce RPC
overhead
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HADOOP-1468:
----------------------------------
Fix Version/s: (was: 0.14.0)
0.15.0
Affects Version/s: (was: 0.14.0)
0.15.0
Status: Patch Available (was: In Progress)
Works in my environment. Try to get the +1 from Hudson.
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.15.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.15.0
>
> Attachments: patch.txt, patch.txt
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-1468) Add HBase batch update to reduce RPC
overhead
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman reopened HADOOP-1468:
-----------------------------------
Michael Stack and I were discussing the current implementation and there are multiple problems with updating more than a single row at a time:
- a region might be reassigned between the start of the batch update and the commit
- a region split might occur
- the server might receive a shut down request before all the commits complete
- if a lot of data is being passed, the rpc could time out before all the data is transmitted.
We came up with this list in a short five minute discussion and I'm sure we could think of a lot more issues without trying too hard.
For these reasons we decided that batch updates should be limited to a single row and consequently this issue is being reopened.
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.14.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.14.0
>
> Attachments: patch.txt
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work stopped: (HADOOP-1468) Add HBase batch update to reduce
RPC overhead
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HADOOP-1468 stopped by Jim Kellerman.
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.14.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.14.0
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1468) Add HBase batch update to reduce
RPC overhead
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12513060 ]
Hadoop QA commented on HADOOP-1468:
-----------------------------------
+1
http://issues.apache.org/jira/secure/attachment/12361870/patch.txt applied and successfully tested against trunk revision r556707.
Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/419/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/419/console
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.14.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.14.0
>
> Attachments: patch.txt
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1468) Add HBase batch update to reduce RPC
overhead
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HADOOP-1468:
----------------------------------
Attachment: patch.txt
Works in my environment. Ensure Hudson agrees.
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.14.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.14.0
>
> Attachments: patch.txt
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (HADOOP-1468) Add HBase batch update to reduce
RPC overhead
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HADOOP-1468 started by Jim Kellerman.
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.14.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.14.0
>
> Attachments: patch.txt
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1468) Add HBase batch update to reduce RPC
overhead
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jim Kellerman updated HADOOP-1468:
----------------------------------
Attachment: patch.txt
Restricts batches to one row per batch
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.15.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.15.0
>
> Attachments: patch.txt, patch.txt
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1468) Add HBase batch update to reduce
RPC overhead
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12511500 ]
Jim Kellerman commented on HADOOP-1468:
---------------------------------------
> Another possibility is batch update of multiple rows wherein the client buffers up a number
> of row updates and flushes them out together.
This may still require multiple RPCs if the rows being updated are in different regions and are being served by different servers. However, the client could split the batch into per-server chunks, which would still greatly reduce the number of RPCs.
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.14.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.14.0
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1468) Add HBase batch update to reduce
RPC overhead
Posted by "James Kennedy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12502003 ]
James Kennedy commented on HADOOP-1468:
---------------------------------------
The above batch update description implies batch update of columns for a single row.
Another possibility is batch update of multiple rows wherein the client buffers up a number of row updates and flushes them out together.
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.14.0
> Reporter: Jim Kellerman
> Fix For: 0.14.0
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (HADOOP-1468) Add HBase batch update to reduce
RPC overhead
Posted by "Jim Kellerman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HADOOP-1468 started by Jim Kellerman.
> Add HBase batch update to reduce RPC overhead
> ---------------------------------------------
>
> Key: HADOOP-1468
> URL: https://issues.apache.org/jira/browse/HADOOP-1468
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/hbase
> Affects Versions: 0.14.0
> Reporter: Jim Kellerman
> Assignee: Jim Kellerman
> Fix For: 0.14.0
>
>
> On Wed, 2007-06-06 at 10:05 -0700, James Kennedy wrote:
> Hi,
> >
> > I'm noticing that since the HClient/HRegionServer interface only allows
> > for a per-column put(), there is a lot of RPC and some lease management
> > overhead when writing large amounts of data. For example:
> >
> > for (int i = 0; i < 10000; i++) {
> > Text rowKey = new Text(i+"");
> > long lock = client.startUpdate(rowKey);
> > client.put(lock, COL1, rowKey.getBytes());
> > client.put(lock, COL2, someValue.getBytes());
> > client.commit(lock);
> > }
> >
> > This code takes my machine (using a single HMaster/HRegionServer on
> > local filesystem) approximately 13 seconds to execute. When i measure
> > the execution time within HRegionServer.put() I get total time spent in
> > put() < 2 seconds. So it looks like there's definately overhead in the
> > RPC communication and serialization/deserialization between client and
> > server.
> >
> > To write 10000 rows, 10000 x (startUpdate=1 + #cols=2 + commit=1) =
> > 40000 RPC operations.
> >
> > What I'm thinking, and please tell me if i'm wrong or if this is already
> > in the works, is that if I create a row-level put() method that submits
> > a map of column values at once, I would reduce the 2 + (#cols) RPC
> > operations to one single atomic row-write RPC as well as eliminate the
> > small but noticeable overhead in lease creation, renewal, and cancellation.
> >
> > It's not clear exactly what the performance improvement would be. The
> > same amount of serialization/deserilalization must occur, but YourKit
> > profiling tells me that the serialization overhead is negligible.
> >
> > Any thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.