You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2012/08/30 06:31:07 UTC
[jira] [Created] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
stack created HBASE-6694:
----------------------------
Summary: Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
Key: HBASE-6694
URL: https://issues.apache.org/jira/browse/HBASE-6694
Project: HBase
Issue Type: Task
Reporter: stack
>From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-6694:
----------------------------------
Status: Patch Available (was: Open)
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450400#comment-13450400 ]
stack commented on HBASE-6694:
------------------------------
No worries. Thanks.
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Alexander Alten-Lorenz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461184#comment-13461184 ]
Alexander Alten-Lorenz commented on HBASE-6694:
-----------------------------------------------
Confirmed that the patch is working. Job.xml contains:
I created over a whole day CF's:
hbase shell> for r in 1 .. 10 do for c in 1 .. 100000000 do put 'test1', "row-#{r}", "cf1:c#{c}", "1" end end
===========
With -Dhbase.export.scanner.batch=100:
HBASE_CLASSPATH="/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.0.0-mr1-cdh4.0.1.jar" bin/hbase org.apache.hadoop.hbase.mapreduce.Export -Dhbase.export.scanner.batch=100 test1 /home/hdfs/test2.export
---
{code}
12/09/22 01:17:23 DEBUG mapreduce.TableInputFormatBase: getSplits: split -> 0 -> hadoop4.internal:,
12/09/22 01:17:24 WARN conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
12/09/22 01:17:24 WARN conf.Configuration: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum
12/09/22 01:17:25 INFO mapred.JobClient: Running job: job_201209212254_0010
12/09/22 01:17:26 INFO mapred.JobClient: map 0% reduce 0%
12/09/22 01:17:59 INFO mapred.JobClient: map 100% reduce 0%
12/09/22 01:18:02 INFO mapred.JobClient: Job complete: job_201209212254_0010
12/09/22 01:18:03 INFO mapred.JobClient: Counters: 24
12/09/22 01:18:03 INFO mapred.JobClient: File System Counters
12/09/22 01:18:03 INFO mapred.JobClient: FILE: Number of bytes read=0
12/09/22 01:18:03 INFO mapred.JobClient: FILE: Number of bytes written=84332
12/09/22 01:18:03 INFO mapred.JobClient: FILE: Number of read operations=0
12/09/22 01:18:03 INFO mapred.JobClient: FILE: Number of large read operations=0
12/09/22 01:18:03 INFO mapred.JobClient: FILE: Number of write operations=0
12/09/22 01:18:03 INFO mapred.JobClient: HDFS: Number of bytes read=70
12/09/22 01:18:03 INFO mapred.JobClient: HDFS: Number of bytes written=62070728
12/09/22 01:18:03 INFO mapred.JobClient: HDFS: Number of read operations=1
12/09/22 01:18:03 INFO mapred.JobClient: HDFS: Number of large read operations=0
12/09/22 01:18:03 INFO mapred.JobClient: HDFS: Number of write operations=1
12/09/22 01:18:03 INFO mapred.JobClient: Job Counters
12/09/22 01:18:03 INFO mapred.JobClient: Launched map tasks=1
12/09/22 01:18:03 INFO mapred.JobClient: Data-local map tasks=1
12/09/22 01:18:03 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=35760
12/09/22 01:18:03 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=0
12/09/22 01:18:03 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/09/22 01:18:03 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/09/22 01:18:03 INFO mapred.JobClient: Map-Reduce Framework
12/09/22 01:18:03 INFO mapred.JobClient: Map input records=15258
12/09/22 01:18:03 INFO mapred.JobClient: Map output records=15258
12/09/22 01:18:03 INFO mapred.JobClient: Input split bytes=70
12/09/22 01:18:03 INFO mapred.JobClient: Spilled Records=0
12/09/22 01:18:03 INFO mapred.JobClient: CPU time spent (ms)=5970
12/09/22 01:18:03 INFO mapred.JobClient: Physical memory (bytes) snapshot=106557440
12/09/22 01:18:03 INFO mapred.JobClient: Virtual memory (bytes) snapshot=570249216
12/09/22 01:18:03 INFO mapred.JobClient: Total committed heap usage (bytes)=42663936
{code}
Export readable in hdfs.
==========
Without -D switch:
RS timed out:
{code}
2012-09-22 01:27:27,937 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fhadoop4%3A8020%2Fhbase%2F.logs%2Fhadoop4.internal%2C60020%2C1348269190284-splitting%2Fhadoop4.internal%252C60020%252C1348269190284.1348269200784 ver = 0
2012-09-22 01:27:28,938 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned = 1
2012-09-22 01:27:28,938 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: resubmitting unassigned task(s) after timeout
2012-09-22 01:27:29,237 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: task not yet acquired /hbase/splitlog/hdfs%3A%2F%2Fhadoop4%3A8020%2Fhbase%2F.logs%2Fhadoop4.internal%2C60020%2C1348269190284-splitting%2Fhadoop4.internal%252C60020%252C1348269190284.1348269200784 ver = 0
2012-09-22 01:27:29,239 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/RESCAN0000000057 entered state done hadoop4.internal,60000,1348269184131
2012-09-22 01:27:29,239 INFO org.apache.hadoop.hbase.master.SplitLogManager: task /hbase/splitlog/RESCAN0000000058 entered state done hadoop4.internal,60000,1348269184131
2012-09-22 01:27:29,282 DEBUG org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted /hbase/splitlog/RESCAN0000000057
2012-09-22 01:27:29,282 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: deleted task without in memory state /hbase/splitlog/RESCAN0000000057
2012-09-22 01:27:29,283 DEBUG org.apache.hadoop.hbase.master.SplitLogManager$DeleteAsyncCallback: deleted /hbase/splitlog/RESCAN0000000058
2012-09-22 01:27:29,283 DEBUG org.apache.hadoop.hbase.master.SplitLogManager: deleted task without in memory state /hbase/splitlog/RESCAN0000000058
{code}
====
Test environment:
Virtual machine, 2GB RAM, 512MB exported Heap for Hbase, Hadoop cluster mode, HBase pseudo distributed. I would say, it worked.
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467078#comment-13467078 ]
Jonathan Hsieh commented on HBASE-6694:
---------------------------------------
Here's some stack traces --
{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 39.802 sec <<< FAILURE!
testSimpleCase(org.apache.hadoop.hbase.mapreduce.TestImportExport) Time elapsed: 14.012 sec <<< FAILURE!
java.lang.AssertionError: expected:<3> but was:<1>
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.hadoop.hbase.mapreduce.TestImportExport.testSimpleCase(TestImportExport.java:137)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
{code}
{code}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 40.498 sec <<< FAILURE!
testWithDeletes(org.apache.hadoop.hbase.mapreduce.TestImportExport) Time elapsed: 14.056 sec <<< ERROR!
java.lang.ArrayIndexOutOfBoundsException: 2
at org.apache.hadoop.hbase.mapreduce.TestImportExport.testWithDeletes(TestImportExport.java:237)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
{code}
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-6694:
----------------------------
Assignee: Alexander Alten-Lorenz
Assigning Alexander at his request (Added you as a contributor Alexander)
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Alexander Alten-Lorenz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455664#comment-13455664 ]
Alexander Alten-Lorenz commented on HBASE-6694:
-----------------------------------------------
YCSB won't work I use a ruby script now, nut that take some time to create a excessive number of columns.
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Alexander Alten-Lorenz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467070#comment-13467070 ]
Alexander Alten-Lorenz commented on HBASE-6694:
-----------------------------------------------
The patch I submitted here is the missed test patch, I'll check what's going wrong here. The original patch was already submitted by Michael (st.Ack) in HBASE-6372
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Alexander Alten-Lorenz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Alten-Lorenz updated HBASE-6694:
------------------------------------------
Attachment: (was: HBASE-6694.patch)
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Alexander Alten-Lorenz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Alten-Lorenz updated HBASE-6694:
------------------------------------------
Attachment: HBASE-6694.patch
Test added
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Alexander Alten-Lorenz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Alten-Lorenz updated HBASE-6694:
------------------------------------------
Attachment: HBASE-6694.patch
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466964#comment-13466964 ]
Jonathan Hsieh commented on HBASE-6694:
---------------------------------------
[~alo.alt] Thanks for the ping and the test results -- looks good to me. I'll commit and resolve.
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466982#comment-13466982 ]
Jonathan Hsieh commented on HBASE-6694:
---------------------------------------
Just tested patch and it fails. :(
{code}
Running org.apache.hadoop.hbase.mapreduce.TestImportExport
Tests run: 3, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 60.171 sec <<< FAILURE!
Results :
Failed tests: testSimpleCase(org.apache.hadoop.hbase.mapreduce.TestImportExport): expected:<3> but was:<
1>
Tests in error:
testWithDeletes(org.apache.hadoop.hbase.mapreduce.TestImportExport): 2
Tests run: 3, Failures: 1, Errors: 1, Skipped: 0
{code}
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Alexander Alten-Lorenz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494853#comment-13494853 ]
Alexander Alten-Lorenz commented on HBASE-6694:
-----------------------------------------------
Ping you guys for ;)
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450392#comment-13450392 ]
stack commented on HBASE-6694:
------------------------------
I don't see a test in here Alexander
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487970#comment-13487970 ]
Jonathan Hsieh commented on HBASE-6694:
---------------------------------------
[~alo.alt], I've tried locally a few times and seem to get the same errors I pointed out running this command line with the attached patch. Can you double check that the correct patch has been attached to the jira?
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Alexander Alten-Lorenz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450396#comment-13450396 ]
Alexander Alten-Lorenz commented on HBASE-6694:
-----------------------------------------------
Not yet, I have a YCSB running, numbers will follow. Lemme some days to finish.
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467039#comment-13467039 ]
Hadoop QA commented on HBASE-6694:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12543625/HBASE-6694.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 149 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportExport
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2976//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2976//console
This message is automatically generated.
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Alexander Alten-Lorenz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490554#comment-13490554 ]
Alexander Alten-Lorenz commented on HBASE-6694:
-----------------------------------------------
New patch attached, but I get a surefire error from testWithDeletes.
{code}
testSimpleCase(org.apache.hadoop.hbase.mapreduce.TestImportExport) Time elapsed: 0.029 sec <<< FAILURE!
java.lang.AssertionError: expected:<3> but was:<1>
at org.junit.Assert.fail(Assert.java:93)
....
testWithDeletes(org.apache.hadoop.hbase.mapreduce.TestImportExport) Time elapsed: 0.007 sec <<< ERROR!
java.lang.ArrayIndexOutOfBoundsException: 2
at org.apache.hadoop.hbase.mapreduce.TestImportExport.testWithDeletes(TestImportExport.java:277)
{code}
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Alexander Alten-Lorenz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444716#comment-13444716 ]
Alexander Alten-Lorenz commented on HBASE-6694:
-----------------------------------------------
Assign please to me, I'll do that in next days.
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Alexander Alten-Lorenz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466694#comment-13466694 ]
Alexander Alten-Lorenz commented on HBASE-6694:
-----------------------------------------------
Michael, Jon - can we mark as solved?
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490576#comment-13490576 ]
Hadoop QA commented on HBASE-6694:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12552084/HBASE-6694.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 85 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.mapreduce.TestImportExport
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/3228//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3228//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3228//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3228//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3228//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/3228//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3228//console
This message is automatically generated.
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6694) Test scanner batching in export job
feature HBASE-6372 AND report on improvement HBASE-6372 adds
Posted by "Alexander Alten-Lorenz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13485886#comment-13485886 ]
Alexander Alten-Lorenz commented on HBASE-6694:
-----------------------------------------------
MAVEN_OPTS="-Xmx512mb" mvn clean package test -P localTests -Dtest=org.apache.hadoop.hbase.mapreduce.TestImportExport
{code}
[INFO] HBase ............................................. SUCCESS [1.778s]
[INFO] HBase - Common .................................... SUCCESS [2.917s]
[INFO] HBase - Hadoop Compatibility ...................... SUCCESS [0.655s]
[INFO] HBase - Hadoop One Compatibility .................. SUCCESS [0.890s]
[INFO] HBase - Server .................................... SUCCESS [1:49.331s]
[INFO] HBase - Hadoop Two Compatibility .................. SUCCESS [2.472s]
[INFO] HBase - Integration Tests ......................... SUCCESS [0.984s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:59.453s
[INFO] Finished at: Mon Oct 22 09:30:58 CEST 2012
[INFO] Final Memory: 50M/256M
[INFO] ------------------------------------------------------------------------
> Test scanner batching in export job feature HBASE-6372 AND report on improvement HBASE-6372 adds
> ------------------------------------------------------------------------------------------------
>
> Key: HBASE-6694
> URL: https://issues.apache.org/jira/browse/HBASE-6694
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: Alexander Alten-Lorenz
> Attachments: HBASE-6694.patch
>
>
> From tail of HBASE-6372, Jon had raised issue that test added did not actually test the feature. This issue is about adding a test of HBASE-6372. We should also have numbers for the improvement that HBASE-6372 brings.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira