You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Todd Lipcon (Created) (JIRA)" <ji...@apache.org> on 2012/02/22 07:52:54 UTC
[jira] [Created] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Switch on-disk formats (reference files, HFile meta fields, etc) to PB
----------------------------------------------------------------------
Key: HBASE-5453
URL: https://issues.apache.org/jira/browse/HBASE-5453
Project: HBase
Issue Type: Sub-task
Reporter: Todd Lipcon
Assignee: stack
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Status: Open (was: Patch Available)
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Attachment: 5453v6.txt
Stashing v6 here for the moment. Does not address yet Gregory's comments. Working on that now in a v7.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "Elliott Clark (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271859#comment-13271859 ]
Elliott Clark commented on HBASE-5453:
--------------------------------------
Are you talking about KeyValue as well ?
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Status: Open (was: Patch Available)
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt, 5453v3.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-5453) Switch on-disk formats
(reference files, HFile meta fields, etc) to PB
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279646#comment-13279646 ]
Zhihong Yu edited comment on HBASE-5453 at 5/20/12 1:25 AM:
------------------------------------------------------------
Since integration of this JIRA, TestGetClosestAtOrBefore#testUsingMetaAndBinary has been failing since build #2902.
Replication-related tests have been failing for Hadoop QA since build #1938.
was (Author: zhihyu@ebaysf.com):
Since integration of this JIRA, TestGetClosestAtOrBefore#testUsingMetaAndBinary has been failing since build #2902.
Replication-related tests have been failing since build #1938.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272549#comment-13272549 ]
Hadoop QA commented on HBASE-5453:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12526378/5453v2.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestColumnSeeking
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1834//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1834//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1834//console
This message is automatically generated.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Attachment: 5453v3.txt
Trying against hadoopqa.
Moves over the five listed files to pb:
.tableinfo, .regioninfo, reference, hbase.id and hbase.version
Will post on pb w/ a better commit note now.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt, 5453v3.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279646#comment-13279646 ]
Zhihong Yu commented on HBASE-5453:
-----------------------------------
Since integration of this JIRA, TestGetClosestAtOrBefore#testUsingMetaAndBinary has been failing since build #2902.
Replication-related tests have been failing since build #1938.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276320#comment-13276320 ]
jiraposter@reviews.apache.org commented on HBASE-5453:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7916
-----------------------------------------------------------
Do we have a story for when we can remove the writable stuff?
src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
<https://reviews.apache.org/r/5130/#comment17217>
Looks like we are stepping on each other's toes a bit; I defined my own protos for this and HTableDescriptor (see the review request for HBASE-5445). Your definitions are pretty close though; I'll convert mine over.
src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
<https://reviews.apache.org/r/5130/#comment17218>
"not be what ou want" -> "not be what you want"
src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
<https://reviews.apache.org/r/5130/#comment17219>
Did you give any thought to storing the actual data in a TableSchema rather than converting? Perhaps it's not worth it because:
1) we still need to maintain the writable for now, so would require rewriting that part, which is a waste
2) might not perform well, because we have to call copyFrom a bunch of times.
src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
<https://reviews.apache.org/r/5130/#comment17220>
Nice. I wrote up similar things for HTableDescriptor/HColumnDescriptor; I can just get rid of them and use your convert functions.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<https://reviews.apache.org/r/5130/#comment17226>
Are we sure this assumption is valid (that if the file lengths are the same, the files are the same format)?
We can't check if there is a pb-prefix at the start of file or something?
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
<https://reviews.apache.org/r/5130/#comment17223>
Same here, is this assumption valid?
src/main/protobuf/FS.proto
<https://reviews.apache.org/r/5130/#comment17221>
extra whitespace
src/main/protobuf/hbase.proto
<https://reviews.apache.org/r/5130/#comment17222>
extra whitespace
- Gregory
On 2012-05-15 22:14:17, Michael Stack wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/5130/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-05-15 22:14:17)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq. New class to hold clusterid in.
bq. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq. Make it so can do pb serialization. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq. Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq. Make it so can do pb serialization. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq. ClusterId under ZK got renamed as ZKClusterId
bq. M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq. Hide the Reference#Range enums. Don't let them out of this class.
bq. Make it so can do pb serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq. Use new methods on Reference for getting top and bottom.
bq. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq. ClusterId under zk has been renamed ZKClusterId.
bq. Use new ClusterId class too.
bq. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq. Use new clusterid class.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq. Move the RegionInfo convertion up into HRegionInfo instead of here.
bq. Added generic toDelimitedByteArray helper.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq. Use HRegionInfo convertions instead.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq. Use HRegionInfo convertions instead.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq. Use new utility writing out .regioninfo files.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq. Formatting.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq. Range in Reference is no longer public.
bq. Range in Reference is no longer public.
bq. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq. M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq. ClusterId got renamed ZKClusterId
bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq. Use new serialization utlity in HTD.
bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq. Generic method for writing dot file content.
bq. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq. Reference#Range is not public any more
bq. M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
bq. Deprecated getHRegionInfo, etc.
bq. D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
bq. A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
bq. Rename
bq. A b/src/main/protobuf/ClusterId.proto
bq. Added file for ClusterId only since its written to fs and to zk.
bq. A b/src/main/protobuf/FS.proto
bq. Protos for fs files.
bq. M b/src/main/protobuf/ZooKeeper.proto
bq. Moved ClusterId out to own proto file
bq. M b/src/main/protobuf/hbase.proto
bq. Added TableSchema and ColumnFamilySchema
bq.
bq.
bq. This addresses bug hbase-5453.
bq. https://issues.apache.org/jira/browse/hbase-5453
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15
bq. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3
bq. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e
bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af
bq. src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059
bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 9e4ada9
bq. src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f
bq. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e
bq. src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 058c006
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 20c7738
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 9f16fee
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 9d3898c
bq. src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 6a9f2fe
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 5e1e16d
bq. src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 5050df0
bq. src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java 049ed8d
bq. src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java efb2b84
bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3d35d3e
bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2
bq. src/main/java/org/apache/hadoop/hbase/util/Writables.java 3d20723
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java f804810
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java PRE-CREATION
bq. src/main/protobuf/ClusterId.proto PRE-CREATION
bq. src/main/protobuf/FS.proto PRE-CREATION
bq. src/main/protobuf/ZooKeeper.proto b72cb28
bq. src/main/protobuf/hbase.proto 30a4c3f
bq. src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java e7fa8b2
bq. src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java f7c0cca
bq. src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 83d8408
bq. src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65
bq. src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6dfba41
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 988d0bf
bq. src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java 339a120
bq.
bq. Diff: https://reviews.apache.org/r/5130/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt, 5453v3.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Attachment: 5453v13.txt
Fix the hbck failure (was creating a .regioninfo using old serializations)
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Attachment: 5453v12.txt
Fixed nasty bug. Default value system and deserialization of HColumnDescriptor clashed. Resulting in cryptic odd failed scanner issue. Yuck.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279116#comment-13279116 ]
Hadoop QA commented on HBASE-5453:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528129/5453v13.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 33 new or modified tests.
+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 33 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestColumnSeeking
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1937//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1937//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1937//console
This message is automatically generated.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277593#comment-13277593 ]
jiraposter@reviews.apache.org commented on HBASE-5453:
------------------------------------------------------
bq. On 2012-05-17 05:41:53, Gregory Chanan wrote:
bq. > src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java, line 275
bq. > <https://reviews.apache.org/r/5130/diff/3/?file=109313#file109313line275>
bq. >
bq. > This is still here? Or just reviewboard showing it for some reason?
Flotsam. An abandoned direction that I failed to clean up. Thanks.
bq. On 2012-05-17 05:41:53, Gregory Chanan wrote:
bq. > src/main/java/org/apache/hadoop/hbase/util/FSUtils.java, line 91
bq. > <https://reviews.apache.org/r/5130/diff/3/?file=109334#file109334line91>
bq. >
bq. > This is what you are using to ensure the sizes cannot be the same? Very nice!
bq. >
bq. > I cannot find a call-site for this, though, maybe I missed it?
Ditto. I was going to use this to write .regioninfo and .tableinfo but then I ran into your review. Study this method which I'll purge in the next version. See how it puts serialized class at top, two '\n's, and then a toString on the class? Thats how .regioninfo files and .tableinfo files are currently written. My patch now changes it so these files only contained the serialized object... no '\n' and no toString. I do this so its very unlikely a Writable length will class w/ the length of a pb'd content (See HRegion for where we write .regioninfo and in FSTableDescriptors for where we write .tableinfo only we don't need the 'trick' here since we have actually read the file and can see if its pb'd).
Thanks for the review G.
- Michael
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7951
-----------------------------------------------------------
On 2012-05-16 23:56:18, Michael Stack wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/5130/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-05-16 23:56:18)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq. New class to hold clusterid in.
bq. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq. Make it so can do pb serialization. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq. Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq. Make it so can do pb serialization. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq. ClusterId under ZK got renamed as ZKClusterId
bq. M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq. Hide the Reference#Range enums. Don't let them out of this class.
bq. Make it so can do pb serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq. Use new methods on Reference for getting top and bottom.
bq. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq. ClusterId under zk has been renamed ZKClusterId.
bq. Use new ClusterId class too.
bq. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq. Use new clusterid class.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq. Move the RegionInfo convertion up into HRegionInfo instead of here.
bq. Added generic toDelimitedByteArray helper.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq. Use HRegionInfo convertions instead.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq. Use HRegionInfo convertions instead.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq. Use new utility writing out .regioninfo files.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq. Formatting.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq. Range in Reference is no longer public.
bq. Range in Reference is no longer public.
bq. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq. M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq. ClusterId got renamed ZKClusterId
bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq. Use new serialization utlity in HTD.
bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq. Generic method for writing dot file content.
bq. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq. Reference#Range is not public any more
bq. M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
bq. Deprecated getHRegionInfo, etc.
bq. D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
bq. A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
bq. Rename
bq. A b/src/main/protobuf/ClusterId.proto
bq. Added file for ClusterId only since its written to fs and to zk.
bq. A b/src/main/protobuf/FS.proto
bq. Protos for fs files.
bq. M b/src/main/protobuf/ZooKeeper.proto
bq. Moved ClusterId out to own proto file
bq. M b/src/main/protobuf/hbase.proto
bq. Added TableSchema and ColumnFamilySchema
bq.
bq.
bq. This addresses bug hbase-5453.
bq. https://issues.apache.org/jira/browse/hbase-5453
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15
bq. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3
bq. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e
bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af
bq. src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059
bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 9e4ada9
bq. src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f
bq. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e
bq. src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 058c006
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 20c7738
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 9f16fee
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6dc0517
bq. src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 6a9f2fe
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 5e1e16d
bq. src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 5050df0
bq. src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java 049ed8d
bq. src/main/java/org/apache/hadoop/hbase/util/Bytes.java 2ccda17
bq. src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java efb2b84
bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3d35d3e
bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2
bq. src/main/java/org/apache/hadoop/hbase/util/Writables.java 3d20723
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java f804810
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java PRE-CREATION
bq. src/main/protobuf/ClusterId.proto PRE-CREATION
bq. src/main/protobuf/FS.proto PRE-CREATION
bq. src/main/protobuf/ZooKeeper.proto b72cb28
bq. src/main/protobuf/hbase.proto 30a4c3f
bq. src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java e7fa8b2
bq. src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java f7c0cca
bq. src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 83d8408
bq. src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65
bq. src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6dfba41
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java 98c09eb
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 988d0bf
bq. src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java 339a120
bq.
bq. Diff: https://reviews.apache.org/r/5130/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v10.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276887#comment-13276887 ]
jiraposter@reviews.apache.org commented on HBASE-5453:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/
-----------------------------------------------------------
(Updated 2012-05-16 17:02:35.527708)
Review request for hbase.
Changes
-------
Gregory. I addressed your comments.
I've changed the format of .regioninfo and .tableinfo. Now instead of serialized Writable followed by toString of the serialized object, instead its just the serialized pb. This removes our having a human readable .regioninfo/.tablinfo file but my guess no one relied on this anyways.
Having just serialized content in the file means a check of file length should be enough figuring whether the file properly serialized. If ever a chance that a Writable + its toString + two '\n' characters was equal to a serialized pb, I'd think this a pathological state. If this state is not cleared up 'naturally' by splits or a schema change, then lets deal if it happens.
I only need this length-checking in one place on region open. I want to avoid reading the .regioninfo file on region open. The alternative means more load on NN and DNs at region open time which could be problematic at big-bang cluster start (Thinking 500 nodes w/ 80k regions, an actual known case -- this is the case I have in mind when I'm trying to avoid more load on NN/DNs).
Otherwise, Gregory's comments led to me to check and I was missing convertion of fs files to pb in all cases (I was just reading the clusterid and hbase.version files, not converting if still Writable). This should be fixed now.
There are some failing tests still but running by hadoopqa to see what it says anyways. Also putting up on rb to get feedback if problem w/ this approach.
Summary
-------
A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
New class to hold clusterid in.
M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
Make it so can do pb serialization. Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
Make it so can do pb serialization. Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
ClusterId under ZK got renamed as ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
Hide the Reference#Range enums. Don't let them out of this class.
Make it so can do pb serialization.
M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
Use new methods on Reference for getting top and bottom.
M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
ClusterId under zk has been renamed ZKClusterId.
Use new ClusterId class too.
M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
Use new clusterid class.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
Move the RegionInfo convertion up into HRegionInfo instead of here.
Added generic toDelimitedByteArray helper.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Use new utility writing out .regioninfo files.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Formatting.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
Range in Reference is no longer public.
Range in Reference is no longer public.
M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
ClusterId got renamed ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
Use new serialization utlity in HTD.
M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Generic method for writing dot file content.
M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
Reference#Range is not public any more
M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
Deprecated getHRegionInfo, etc.
D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
Rename
A b/src/main/protobuf/ClusterId.proto
Added file for ClusterId only since its written to fs and to zk.
A b/src/main/protobuf/FS.proto
Protos for fs files.
M b/src/main/protobuf/ZooKeeper.proto
Moved ClusterId out to own proto file
M b/src/main/protobuf/hbase.proto
Added TableSchema and ColumnFamilySchema
This addresses bug hbase-5453.
https://issues.apache.org/jira/browse/hbase-5453
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15
src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3
src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af
src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059
src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 9e4ada9
src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878
src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e
src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab
src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 058c006
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 20c7738
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 9f16fee
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6dc0517
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 6a9f2fe
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 5e1e16d
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 5050df0
src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java 049ed8d
src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java efb2b84
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3d35d3e
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2
src/main/java/org/apache/hadoop/hbase/util/Writables.java 3d20723
src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java f804810
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java PRE-CREATION
src/main/protobuf/ClusterId.proto PRE-CREATION
src/main/protobuf/FS.proto PRE-CREATION
src/main/protobuf/ZooKeeper.proto b72cb28
src/main/protobuf/hbase.proto 30a4c3f
src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java e7fa8b2
src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java f7c0cca
src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 83d8408
src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65
src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6dfba41
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 988d0bf
src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java 339a120
Diff: https://reviews.apache.org/r/5130/diff
Testing
-------
Thanks,
Michael
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272539#comment-13272539 ]
Jimmy Xiang commented on HBASE-5453:
------------------------------------
That would be great, I think. Thanks.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279134#comment-13279134 ]
stack commented on HBASE-5453:
------------------------------
The failed test passes locally. Will commit soon unless objection.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks for review Gregory.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276276#comment-13276276 ]
jiraposter@reviews.apache.org commented on HBASE-5453:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/
-----------------------------------------------------------
Review request for hbase.
Summary
-------
A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
New class to hold clusterid in.
M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
Make it so can do pb serialization. Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
Make it so can do pb serialization. Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
ClusterId under ZK got renamed as ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
Hide the Reference#Range enums. Don't let them out of this class.
Make it so can do pb serialization.
M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
Use new methods on Reference for getting top and bottom.
M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
ClusterId under zk has been renamed ZKClusterId.
Use new ClusterId class too.
M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
Use new clusterid class.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
Move the RegionInfo convertion up into HRegionInfo instead of here.
Added generic toDelimitedByteArray helper.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Use new utility writing out .regioninfo files.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Formatting.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
Range in Reference is no longer public.
Range in Reference is no longer public.
M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
ClusterId got renamed ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
Use new serialization utlity in HTD.
M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Generic method for writing dot file content.
M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
Reference#Range is not public any more
M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
Deprecated getHRegionInfo, etc.
D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
Rename
A b/src/main/protobuf/ClusterId.proto
Added file for ClusterId only since its written to fs and to zk.
A b/src/main/protobuf/FS.proto
Protos for fs files.
M b/src/main/protobuf/ZooKeeper.proto
Moved ClusterId out to own proto file
M b/src/main/protobuf/hbase.proto
Added TableSchema and ColumnFamilySchema
This addresses bug hbase-5453.
https://issues.apache.org/jira/browse/hbase-5453
Diffs
-----
src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15
src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3
src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af
src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059
src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 9e4ada9
src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878
src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e
src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab
src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 058c006
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 20c7738
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 9f16fee
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 9d3898c
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 6a9f2fe
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 5e1e16d
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 5050df0
src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java 049ed8d
src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java efb2b84
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3d35d3e
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2
src/main/java/org/apache/hadoop/hbase/util/Writables.java 3d20723
src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java f804810
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java PRE-CREATION
src/main/protobuf/ClusterId.proto PRE-CREATION
src/main/protobuf/FS.proto PRE-CREATION
src/main/protobuf/ZooKeeper.proto b72cb28
src/main/protobuf/hbase.proto 30a4c3f
src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java e7fa8b2
src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java f7c0cca
src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 83d8408
src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65
src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6dfba41
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 988d0bf
src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java 339a120
Diff: https://reviews.apache.org/r/5130/diff
Testing
-------
Thanks,
Michael
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt, 5453v3.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276330#comment-13276330 ]
jiraposter@reviews.apache.org commented on HBASE-5453:
------------------------------------------------------
bq. On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq. > Do we have a story for when we can remove the writable stuff?
No. Smile. Being optimistic. Have to start somewhere. No harm marking this stuff deprecated in meantime as heads-up.
I think we won't be able to migrate over these old classes. I'm thinking we'll have to do new ones and deprecate the old given they inherit from Writable. It'll be a bit of a pain though given some of these classes come out in our public API.
But we have to start somewhere.
bq. On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq. > src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java, line 1017
bq. > <https://reviews.apache.org/r/5130/diff/1/?file=109035#file109035line1017>
bq. >
bq. > Looks like we are stepping on each other's toes a bit; I defined my own protos for this and HTableDescriptor (see the review request for HBASE-5445). Your definitions are pretty close though; I'll convert mine over.
Well, Andrew beat us both to it over in his REST pb stuff. We need to reconcile his w/ ours too....
bq. On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq. > src/main/java/org/apache/hadoop/hbase/HRegionInfo.java, line 954
bq. > <https://reviews.apache.org/r/5130/diff/1/?file=109036#file109036line954>
bq. >
bq. > "not be what ou want" -> "not be what you want"
Fixed
bq. On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq. > src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java, line 1240
bq. > <https://reviews.apache.org/r/5130/diff/1/?file=109037#file109037line1240>
bq. >
bq. > Did you give any thought to storing the actual data in a TableSchema rather than converting? Perhaps it's not worth it because:
bq. > 1) we still need to maintain the writable for now, so would require rewriting that part, which is a waste
bq. > 2) might not perform well, because we have to call copyFrom a bunch of times.
Yeah. Thought about it but we want Writable still converting old serializations. Mostly I reasoned that these classes are rarely, relatively serialized, and performance is not important when we do want to pass these classes serialized. Yeah, it'd be a pain converting.
bq. On 2012-05-15 23:08:35, Gregory Chanan wrote:
bq. > src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 748
bq. > <https://reviews.apache.org/r/5130/diff/1/?file=109050#file109050line748>
bq. >
bq. > Are we sure this assumption is valid (that if the file lengths are the same, the files are the same format)?
bq. >
bq. > We can't check if there is a pb-prefix at the start of file or something?
This is my way of figuring if I need to convert the file. There is an off chance that file could be same length though serialized with Writables. Good one. Let me fix that.
- Michael
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7916
-----------------------------------------------------------
On 2012-05-15 22:14:17, Michael Stack wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/5130/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-05-15 22:14:17)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq. New class to hold clusterid in.
bq. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq. Make it so can do pb serialization. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq. Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq. Make it so can do pb serialization. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq. ClusterId under ZK got renamed as ZKClusterId
bq. M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq. Hide the Reference#Range enums. Don't let them out of this class.
bq. Make it so can do pb serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq. Use new methods on Reference for getting top and bottom.
bq. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq. ClusterId under zk has been renamed ZKClusterId.
bq. Use new ClusterId class too.
bq. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq. Use new clusterid class.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq. Move the RegionInfo convertion up into HRegionInfo instead of here.
bq. Added generic toDelimitedByteArray helper.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq. Use HRegionInfo convertions instead.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq. Use HRegionInfo convertions instead.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq. Use new utility writing out .regioninfo files.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq. Formatting.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq. Range in Reference is no longer public.
bq. Range in Reference is no longer public.
bq. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq. M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq. ClusterId got renamed ZKClusterId
bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq. Use new serialization utlity in HTD.
bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq. Generic method for writing dot file content.
bq. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq. Reference#Range is not public any more
bq. M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
bq. Deprecated getHRegionInfo, etc.
bq. D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
bq. A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
bq. Rename
bq. A b/src/main/protobuf/ClusterId.proto
bq. Added file for ClusterId only since its written to fs and to zk.
bq. A b/src/main/protobuf/FS.proto
bq. Protos for fs files.
bq. M b/src/main/protobuf/ZooKeeper.proto
bq. Moved ClusterId out to own proto file
bq. M b/src/main/protobuf/hbase.proto
bq. Added TableSchema and ColumnFamilySchema
bq.
bq.
bq. This addresses bug hbase-5453.
bq. https://issues.apache.org/jira/browse/hbase-5453
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15
bq. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3
bq. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e
bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af
bq. src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059
bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 9e4ada9
bq. src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f
bq. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e
bq. src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 058c006
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 20c7738
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 9f16fee
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 9d3898c
bq. src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 6a9f2fe
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 5e1e16d
bq. src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 5050df0
bq. src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java 049ed8d
bq. src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java efb2b84
bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3d35d3e
bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2
bq. src/main/java/org/apache/hadoop/hbase/util/Writables.java 3d20723
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java f804810
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java PRE-CREATION
bq. src/main/protobuf/ClusterId.proto PRE-CREATION
bq. src/main/protobuf/FS.proto PRE-CREATION
bq. src/main/protobuf/ZooKeeper.proto b72cb28
bq. src/main/protobuf/hbase.proto 30a4c3f
bq. src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java e7fa8b2
bq. src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java f7c0cca
bq. src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 83d8408
bq. src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65
bq. src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6dfba41
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 988d0bf
bq. src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java 339a120
bq.
bq. Diff: https://reviews.apache.org/r/5130/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279053#comment-13279053 ]
Hadoop QA commented on HBASE-5453:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12528107/5453v12.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 30 new or modified tests.
+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 32 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.coprocessor.TestClassLoading
org.apache.hadoop.hbase.replication.TestMasterReplication
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1934//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1934//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1934//console
This message is automatically generated.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13278263#comment-13278263 ]
jiraposter@reviews.apache.org commented on HBASE-5453:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7963
-----------------------------------------------------------
Ship it!
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
<https://reviews.apache.org/r/5130/#comment17303>
Your logic sounds correct and I agree we should avoid having to read the file each time.
It would be cool to just see an example of the old writable vs new pb size to ensure our intuition is correct.
Worst case (let's say they happen to be the same) we could write out the new pb stuff concatenated with the old way of writing and then they are guaranteed to be different sizes. Hopefully we don't have to do something like that, though.
- Gregory
On 2012-05-17 05:58:11, Michael Stack wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/5130/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-05-17 05:58:11)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq. New class to hold clusterid in.
bq. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq. Make it so can do pb serialization. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq. Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq. Make it so can do pb serialization. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq. ClusterId under ZK got renamed as ZKClusterId
bq. M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq. Hide the Reference#Range enums. Don't let them out of this class.
bq. Make it so can do pb serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq. Use new methods on Reference for getting top and bottom.
bq. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq. ClusterId under zk has been renamed ZKClusterId.
bq. Use new ClusterId class too.
bq. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq. Use new clusterid class.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq. Move the RegionInfo convertion up into HRegionInfo instead of here.
bq. Added generic toDelimitedByteArray helper.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq. Use HRegionInfo convertions instead.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq. Use HRegionInfo convertions instead.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq. Use new utility writing out .regioninfo files.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq. Formatting.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq. Range in Reference is no longer public.
bq. Range in Reference is no longer public.
bq. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq. M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq. ClusterId got renamed ZKClusterId
bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq. Use new serialization utlity in HTD.
bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq. Generic method for writing dot file content.
bq. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq. Reference#Range is not public any more
bq. M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
bq. Deprecated getHRegionInfo, etc.
bq. D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
bq. A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
bq. Rename
bq. A b/src/main/protobuf/ClusterId.proto
bq. Added file for ClusterId only since its written to fs and to zk.
bq. A b/src/main/protobuf/FS.proto
bq. Protos for fs files.
bq. M b/src/main/protobuf/ZooKeeper.proto
bq. Moved ClusterId out to own proto file
bq. M b/src/main/protobuf/hbase.proto
bq. Added TableSchema and ColumnFamilySchema
bq.
bq.
bq. This addresses bug hbase-5453.
bq. https://issues.apache.org/jira/browse/hbase-5453
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15
bq. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3
bq. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e
bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af
bq. src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059
bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 9e4ada9
bq. src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f
bq. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e
bq. src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 058c006
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 20c7738
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 9f16fee
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6dc0517
bq. src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 6a9f2fe
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 5e1e16d
bq. src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 5050df0
bq. src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java 049ed8d
bq. src/main/java/org/apache/hadoop/hbase/util/Bytes.java 2ccda17
bq. src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java efb2b84
bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3d35d3e
bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2
bq. src/main/java/org/apache/hadoop/hbase/util/Writables.java 3d20723
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java f804810
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java PRE-CREATION
bq. src/main/protobuf/ClusterId.proto PRE-CREATION
bq. src/main/protobuf/FS.proto PRE-CREATION
bq. src/main/protobuf/ZooKeeper.proto b72cb28
bq. src/main/protobuf/hbase.proto 30a4c3f
bq. src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java e7fa8b2
bq. src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java f7c0cca
bq. src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 83d8408
bq. src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65
bq. src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6dfba41
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java 98c09eb
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 988d0bf
bq. src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java 339a120
bq.
bq. Diff: https://reviews.apache.org/r/5130/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277589#comment-13277589 ]
jiraposter@reviews.apache.org commented on HBASE-5453:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7951
-----------------------------------------------------------
src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
<https://reviews.apache.org/r/5130/#comment17291>
This is still here? Or just reviewboard showing it for some reason?
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
<https://reviews.apache.org/r/5130/#comment17290>
This is what you are using to ensure the sizes cannot be the same? Very nice!
I cannot find a call-site for this, though, maybe I missed it?
- Gregory
On 2012-05-16 23:56:18, Michael Stack wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/5130/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-05-16 23:56:18)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq. New class to hold clusterid in.
bq. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq. Make it so can do pb serialization. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq. Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq. Make it so can do pb serialization. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq. ClusterId under ZK got renamed as ZKClusterId
bq. M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq. Hide the Reference#Range enums. Don't let them out of this class.
bq. Make it so can do pb serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq. Use new methods on Reference for getting top and bottom.
bq. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq. ClusterId under zk has been renamed ZKClusterId.
bq. Use new ClusterId class too.
bq. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq. Use new clusterid class.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq. Move the RegionInfo convertion up into HRegionInfo instead of here.
bq. Added generic toDelimitedByteArray helper.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq. Use HRegionInfo convertions instead.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq. Use HRegionInfo convertions instead.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq. Use new utility writing out .regioninfo files.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq. Formatting.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq. Range in Reference is no longer public.
bq. Range in Reference is no longer public.
bq. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq. M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq. ClusterId got renamed ZKClusterId
bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq. Use new serialization utlity in HTD.
bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq. Generic method for writing dot file content.
bq. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq. Reference#Range is not public any more
bq. M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
bq. Deprecated getHRegionInfo, etc.
bq. D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
bq. A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
bq. Rename
bq. A b/src/main/protobuf/ClusterId.proto
bq. Added file for ClusterId only since its written to fs and to zk.
bq. A b/src/main/protobuf/FS.proto
bq. Protos for fs files.
bq. M b/src/main/protobuf/ZooKeeper.proto
bq. Moved ClusterId out to own proto file
bq. M b/src/main/protobuf/hbase.proto
bq. Added TableSchema and ColumnFamilySchema
bq.
bq.
bq. This addresses bug hbase-5453.
bq. https://issues.apache.org/jira/browse/hbase-5453
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15
bq. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3
bq. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e
bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af
bq. src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059
bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 9e4ada9
bq. src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f
bq. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e
bq. src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 058c006
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 20c7738
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 9f16fee
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6dc0517
bq. src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 6a9f2fe
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 5e1e16d
bq. src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 5050df0
bq. src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java 049ed8d
bq. src/main/java/org/apache/hadoop/hbase/util/Bytes.java 2ccda17
bq. src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java efb2b84
bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3d35d3e
bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2
bq. src/main/java/org/apache/hadoop/hbase/util/Writables.java 3d20723
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java f804810
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java PRE-CREATION
bq. src/main/protobuf/ClusterId.proto PRE-CREATION
bq. src/main/protobuf/FS.proto PRE-CREATION
bq. src/main/protobuf/ZooKeeper.proto b72cb28
bq. src/main/protobuf/hbase.proto 30a4c3f
bq. src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java e7fa8b2
bq. src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java f7c0cca
bq. src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 83d8408
bq. src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65
bq. src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6dfba41
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java 98c09eb
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 988d0bf
bq. src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java 339a120
bq.
bq. Diff: https://reviews.apache.org/r/5130/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v10.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271864#comment-13271864 ]
stack commented on HBASE-5453:
------------------------------
no to kv
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277596#comment-13277596 ]
jiraposter@reviews.apache.org commented on HBASE-5453:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/
-----------------------------------------------------------
(Updated 2012-05-17 05:58:11.354772)
Review request for hbase.
Changes
-------
Address Gregory's two review comments.
Summary
-------
A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
New class to hold clusterid in.
M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
Make it so can do pb serialization. Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
Make it so can do pb serialization. Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
ClusterId under ZK got renamed as ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
Hide the Reference#Range enums. Don't let them out of this class.
Make it so can do pb serialization.
M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
Use new methods on Reference for getting top and bottom.
M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
ClusterId under zk has been renamed ZKClusterId.
Use new ClusterId class too.
M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
Use new clusterid class.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
Move the RegionInfo convertion up into HRegionInfo instead of here.
Added generic toDelimitedByteArray helper.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Use new utility writing out .regioninfo files.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Formatting.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
Range in Reference is no longer public.
Range in Reference is no longer public.
M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
ClusterId got renamed ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
Use new serialization utlity in HTD.
M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Generic method for writing dot file content.
M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
Reference#Range is not public any more
M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
Deprecated getHRegionInfo, etc.
D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
Rename
A b/src/main/protobuf/ClusterId.proto
Added file for ClusterId only since its written to fs and to zk.
A b/src/main/protobuf/FS.proto
Protos for fs files.
M b/src/main/protobuf/ZooKeeper.proto
Moved ClusterId out to own proto file
M b/src/main/protobuf/hbase.proto
Added TableSchema and ColumnFamilySchema
This addresses bug hbase-5453.
https://issues.apache.org/jira/browse/hbase-5453
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15
src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3
src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af
src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059
src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 9e4ada9
src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878
src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e
src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab
src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 058c006
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 20c7738
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 9f16fee
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6dc0517
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 6a9f2fe
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 5e1e16d
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 5050df0
src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java 049ed8d
src/main/java/org/apache/hadoop/hbase/util/Bytes.java 2ccda17
src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java efb2b84
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3d35d3e
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2
src/main/java/org/apache/hadoop/hbase/util/Writables.java 3d20723
src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java f804810
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java PRE-CREATION
src/main/protobuf/ClusterId.proto PRE-CREATION
src/main/protobuf/FS.proto PRE-CREATION
src/main/protobuf/ZooKeeper.proto b72cb28
src/main/protobuf/hbase.proto 30a4c3f
src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java e7fa8b2
src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java f7c0cca
src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 83d8408
src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65
src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6dfba41
src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java 98c09eb
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 988d0bf
src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java 339a120
Diff: https://reviews.apache.org/r/5130/diff
Testing
-------
Thanks,
Michael
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277272#comment-13277272 ]
jiraposter@reviews.apache.org commented on HBASE-5453:
------------------------------------------------------
bq. On 2012-05-16 22:12:10, Gregory Chanan wrote:
bq. > In a previous comment you said (about the HTableDescriptor/HColumnDesriptor pb stuff):
bq. > "Well, Andrew beat us both to it over in his REST pb stuff. We need to reconcile his w/ ours too...."
bq. >
bq. > Where is his stuff? I couldn't find it. Should we create a JIRA about reconciling? It would be nice to have something, however imperfect, up in trunk to work against, then we could fix up later.
Its under src/main/resources/org.... I just tripped over it myself yesterday. I made HBASE-6026 to do the reconcile (after HBASE-6000 goes in).
Are you down w/ the change in .regioninfo Gregory as means of 'ensuring' we don't have a Writable and pb serialization end up as same size?
Thanks for review.
bq. On 2012-05-16 22:12:10, Gregory Chanan wrote:
bq. > src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java, line 276
bq. > <https://reviews.apache.org/r/5130/diff/1-2/?file=109037#file109037line276>
bq. >
bq. > This isn't ever read?
This is gone now. I used another technique figuring if object serialized -- read file fully into byte array and test for the pb prefix -- rather than this hacky setting attribute on class.
- Michael
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7944
-----------------------------------------------------------
On 2012-05-16 17:02:35, Michael Stack wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/5130/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-05-16 17:02:35)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq. New class to hold clusterid in.
bq. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq. Make it so can do pb serialization. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq. Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq. Make it so can do pb serialization. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq. ClusterId under ZK got renamed as ZKClusterId
bq. M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq. Hide the Reference#Range enums. Don't let them out of this class.
bq. Make it so can do pb serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq. Use new methods on Reference for getting top and bottom.
bq. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq. ClusterId under zk has been renamed ZKClusterId.
bq. Use new ClusterId class too.
bq. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq. Use new clusterid class.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq. Move the RegionInfo convertion up into HRegionInfo instead of here.
bq. Added generic toDelimitedByteArray helper.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq. Use HRegionInfo convertions instead.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq. Use HRegionInfo convertions instead.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq. Use new utility writing out .regioninfo files.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq. Formatting.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq. Range in Reference is no longer public.
bq. Range in Reference is no longer public.
bq. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq. M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq. ClusterId got renamed ZKClusterId
bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq. Use new serialization utlity in HTD.
bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq. Generic method for writing dot file content.
bq. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq. Reference#Range is not public any more
bq. M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
bq. Deprecated getHRegionInfo, etc.
bq. D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
bq. A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
bq. Rename
bq. A b/src/main/protobuf/ClusterId.proto
bq. Added file for ClusterId only since its written to fs and to zk.
bq. A b/src/main/protobuf/FS.proto
bq. Protos for fs files.
bq. M b/src/main/protobuf/ZooKeeper.proto
bq. Moved ClusterId out to own proto file
bq. M b/src/main/protobuf/hbase.proto
bq. Added TableSchema and ColumnFamilySchema
bq.
bq.
bq. This addresses bug hbase-5453.
bq. https://issues.apache.org/jira/browse/hbase-5453
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15
bq. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3
bq. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e
bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af
bq. src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059
bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 9e4ada9
bq. src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f
bq. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e
bq. src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 058c006
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 20c7738
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 9f16fee
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6dc0517
bq. src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 6a9f2fe
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 5e1e16d
bq. src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 5050df0
bq. src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java 049ed8d
bq. src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java efb2b84
bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3d35d3e
bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2
bq. src/main/java/org/apache/hadoop/hbase/util/Writables.java 3d20723
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java f804810
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java PRE-CREATION
bq. src/main/protobuf/ClusterId.proto PRE-CREATION
bq. src/main/protobuf/FS.proto PRE-CREATION
bq. src/main/protobuf/ZooKeeper.proto b72cb28
bq. src/main/protobuf/hbase.proto 30a4c3f
bq. src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java e7fa8b2
bq. src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java f7c0cca
bq. src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 83d8408
bq. src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65
bq. src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6dfba41
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 988d0bf
bq. src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java 339a120
bq.
bq. Diff: https://reviews.apache.org/r/5130/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v10.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13271855#comment-13271855 ]
stack commented on HBASE-5453:
------------------------------
Files to move over:
{code}
/${HBASE_ROOTDIR}/hbase.version
/${HBASE_ROOTDIR}/hbase.id
/${HBASE_ROOTDIR}/TABLENAME/.tableinfo
/${HBASE_ROOTDIR}/TABLENAME/REGIONNAME/.regioninfo
/${HBASE_ROOTDIR}/TABLENAME/REGIONNAME/COLUMN_FAMILY/reference
{code}
Let me see too where else we write; e.g. into hfiles. Is it necessary bringing these over? They are versioned already w/ readers than can do v1, v2, etc.
Let me see if I can do it in a portable way.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Attachment: 5453.txt
Move over the hbase.version file.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Attachment: 5453v11.txt
5453v11.txt
v11 addressing Gregory's last review comments.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277620#comment-13277620 ]
Hadoop QA commented on HBASE-5453:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12527785/5453v11.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 27 new or modified tests.
+1 hadoop23. The patch compiles against the hadoop 0.23.x profile.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 32 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultipleTimestamps
org.apache.hadoop.hbase.client.TestMetaMigrationRemovingHTD
org.apache.hadoop.hbase.util.hbck.TestOfflineMetaRebuildOverlap
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.client.TestTimestampsFilter
org.apache.hadoop.hbase.replication.TestMultiSlaveReplication
org.apache.hadoop.hbase.client.TestFromClientSide
org.apache.hadoop.hbase.mapreduce.TestImportExport
org.apache.hadoop.hbase.replication.TestMasterReplication
Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1906//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1906//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1906//console
This message is automatically generated.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272503#comment-13272503 ]
Jimmy Xiang commented on HBASE-5453:
------------------------------------
Good stuff. Are the proto file and the generated java file in a different patch?
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13272718#comment-13272718 ]
Todd Lipcon commented on HBASE-5453:
------------------------------------
Agree it would be great if HFile used protobuf where possible. But I don't think it needs to block 0.96, since we already have decent compatibility paths here, and the writables used are already fairly generic (mostly maps if I remember correctly).
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Status: Open (was: Patch Available)
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277273#comment-13277273 ]
jiraposter@reviews.apache.org commented on HBASE-5453:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/
-----------------------------------------------------------
(Updated 2012-05-16 23:56:18.496225)
Review request for hbase.
Changes
-------
v10... the last thing I added to JIRA
Summary
-------
A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
New class to hold clusterid in.
M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
Make it so can do pb serialization. Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
Make it so can do pb serialization. Deprecated Writable serialization.
M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
ClusterId under ZK got renamed as ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
Hide the Reference#Range enums. Don't let them out of this class.
Make it so can do pb serialization.
M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
Use new methods on Reference for getting top and bottom.
M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
ClusterId under zk has been renamed ZKClusterId.
Use new ClusterId class too.
M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
Use new clusterid class.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
Move the RegionInfo convertion up into HRegionInfo instead of here.
Added generic toDelimitedByteArray helper.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
Use HRegionInfo convertions instead.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Use new utility writing out .regioninfo files.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Formatting.
M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
Range in Reference is no longer public.
Range in Reference is no longer public.
M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
ClusterId got renamed ZKClusterId
M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
Use new serialization utlity in HTD.
M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
Generic method for writing dot file content.
M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
Reference#Range is not public any more
M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
Deprecated getHRegionInfo, etc.
D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
Rename
A b/src/main/protobuf/ClusterId.proto
Added file for ClusterId only since its written to fs and to zk.
A b/src/main/protobuf/FS.proto
Protos for fs files.
M b/src/main/protobuf/ZooKeeper.proto
Moved ClusterId out to own proto file
M b/src/main/protobuf/hbase.proto
Added TableSchema and ColumnFamilySchema
This addresses bug hbase-5453.
https://issues.apache.org/jira/browse/hbase-5453
Diffs (updated)
-----
src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15
src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3
src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af
src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059
src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 9e4ada9
src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f
src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878
src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e
src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab
src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 058c006
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 20c7738
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 9f16fee
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6dc0517
src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 6a9f2fe
src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 5e1e16d
src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 5050df0
src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java 049ed8d
src/main/java/org/apache/hadoop/hbase/util/Bytes.java 2ccda17
src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java efb2b84
src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3d35d3e
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2
src/main/java/org/apache/hadoop/hbase/util/Writables.java 3d20723
src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java f804810
src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java PRE-CREATION
src/main/protobuf/ClusterId.proto PRE-CREATION
src/main/protobuf/FS.proto PRE-CREATION
src/main/protobuf/ZooKeeper.proto b72cb28
src/main/protobuf/hbase.proto 30a4c3f
src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java e7fa8b2
src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java f7c0cca
src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 83d8408
src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65
src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374
src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6dfba41
src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java 98c09eb
src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 988d0bf
src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java 339a120
Diff: https://reviews.apache.org/r/5130/diff
Testing
-------
Thanks,
Michael
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v10.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13276269#comment-13276269 ]
Hadoop QA commented on HBASE-5453:
----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12527528/5453v3.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 24 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1879//console
This message is automatically generated.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt, 5453v3.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13283833#comment-13283833 ]
Hudson commented on HBASE-5453:
-------------------------------
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #16 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/16/])
HBASE-6063 Replication related failures on trunk after HBASE-5453 (Gregory Channan) (Revision 1342095)
Result = FAILURE
larsh :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ClusterId.java
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Attachment: 5453v9.txt
Address Gregory's comments.
I've changed the format of .regioninfo and .tableinfo. Now instead of serialized Writable followed by toString of the serialized object, instead its just the serialized pb.
This removes our having a human readable .regioninfo/.tablinfo file but my guess no one relied on this anyways.
Having just serialized content in the file means a check of file length should be enough figuring whether the file properly serialized. If ever a chance that a Writable + its toString + two '\n' characters was equal to a serialized pb, I'd think this likely a pathological state. If this state is not cleared up 'naturally' by splits or a schema change, then lets deal if it happens.
I only need this length-checking in one place on region open. I want to avoid reading the .regioninfo file on region open. The alternative means more load on NN and DNs at region open time which could be problematic at big-bang cluster start (Thinking 500 nodes w/ 80k regions, an actual known case).
Otherwise, Gregory's comments led to me to check and I was missing convertion of fs files to pb in all cases. This should be fixed now.
There are some failing tests still but running by hadoopqa to see what it says anyways. Also putting up on rb to get feedback if problem w/ this approach.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "Jimmy Xiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274159#comment-13274159 ]
Jimmy Xiang commented on HBASE-5453:
------------------------------------
v2 looks good to me.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "Zhihong Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279137#comment-13279137 ]
Zhihong Yu commented on HBASE-5453:
-----------------------------------
{code}
+ * @return This instance serialized with pb with pb magic prefix
{code}
Remove redundant 'with pb '.
{code}
+ * @return Convert this instance to a the pb column family type
{code}
Remove 'a '
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Attachment: 5453v2.txt
Ooops. Thanks Jimmy. You think I should change over the hfile metadata to pb too? I suppose its MapWritable currently so should go over if we want to purge Writables...
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v2.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277200#comment-13277200 ]
jiraposter@reviews.apache.org commented on HBASE-5453:
------------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5130/#review7944
-----------------------------------------------------------
In a previous comment you said (about the HTableDescriptor/HColumnDesriptor pb stuff):
"Well, Andrew beat us both to it over in his REST pb stuff. We need to reconcile his w/ ours too...."
Where is his stuff? I couldn't find it. Should we create a JIRA about reconciling? It would be nice to have something, however imperfect, up in trunk to work against, then we could fix up later.
src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
<https://reviews.apache.org/r/5130/#comment17274>
This isn't ever read?
src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
<https://reviews.apache.org/r/5130/#comment17273>
This constructor sets pbMade to false, but it should be true in this case, right?
- Gregory
On 2012-05-16 17:02:35, Michael Stack wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/5130/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2012-05-16 17:02:35)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. A b/src/main/java/org/apache/hadoop/hbase/ClusterId.java
bq. New class to hold clusterid in.
bq. M b/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
bq. Make it so can do pb serialization. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
bq. Make it so methods in here follow the pattern in HCD an HTD pb 'ing.
bq. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java
bq. Make it so can do pb serialization. Deprecated Writable serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq. ClusterId under ZK got renamed as ZKClusterId
bq. M b/src/main/java/org/apache/hadoop/hbase/io/Reference.java
bq. Hide the Reference#Range enums. Don't let them out of this class.
bq. Make it so can do pb serialization.
bq. M b/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
bq. Use new methods on Reference for getting top and bottom.
bq. M b/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
bq. ClusterId under zk has been renamed ZKClusterId.
bq. Use new ClusterId class too.
bq. M b/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
bq. Use new clusterid class.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq. Move the RegionInfo convertion up into HRegionInfo instead of here.
bq. Added generic toDelimitedByteArray helper.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
bq. Use HRegionInfo convertions instead.
bq. M b/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java
bq. Use HRegionInfo convertions instead.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
bq. Use new utility writing out .regioninfo files.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq. Formatting.
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java
bq. M b/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
bq. Range in Reference is no longer public.
bq. Range in Reference is no longer public.
bq. M b/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
bq. M b/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java
bq. ClusterId got renamed ZKClusterId
bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
bq. Use new serialization utlity in HTD.
bq. M b/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java
bq. Generic method for writing dot file content.
bq. M b/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq. Reference#Range is not public any more
bq. M b/src/main/java/org/apache/hadoop/hbase/util/Writables.java
bq. Deprecated getHRegionInfo, etc.
bq. D b/src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
bq. A b/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java
bq. Rename
bq. A b/src/main/protobuf/ClusterId.proto
bq. Added file for ClusterId only since its written to fs and to zk.
bq. A b/src/main/protobuf/FS.proto
bq. Protos for fs files.
bq. M b/src/main/protobuf/ZooKeeper.proto
bq. Moved ClusterId out to own proto file
bq. M b/src/main/protobuf/hbase.proto
bq. Added TableSchema and ColumnFamilySchema
bq.
bq.
bq. This addresses bug hbase-5453.
bq. https://issues.apache.org/jira/browse/hbase-5453
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. src/main/java/org/apache/hadoop/hbase/ClusterId.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java 5862f15
bq. src/main/java/org/apache/hadoop/hbase/HRegionInfo.java 8d83ff3
bq. src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java af89e3e
bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 5cac9af
bq. src/main/java/org/apache/hadoop/hbase/io/Reference.java 6360059
bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 9e4ada9
bq. src/main/java/org/apache/hadoop/hbase/master/HMaster.java 947ec5f
bq. src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 5052878
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java ccc964e
bq. src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java dabfbab
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java 45cb6cf
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClusterIdProtos.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/FSProtos.java PRE-CREATION
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 058c006
bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 20c7738
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 9f16fee
bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 6dc0517
bq. src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java 6a9f2fe
bq. src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 5e1e16d
bq. src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 5050df0
bq. src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java 049ed8d
bq. src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java efb2b84
bq. src/main/java/org/apache/hadoop/hbase/util/FSUtils.java 3d35d3e
bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 7b4f4a2
bq. src/main/java/org/apache/hadoop/hbase/util/Writables.java 3d20723
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java f804810
bq. src/main/java/org/apache/hadoop/hbase/zookeeper/ZKClusterId.java PRE-CREATION
bq. src/main/protobuf/ClusterId.proto PRE-CREATION
bq. src/main/protobuf/FS.proto PRE-CREATION
bq. src/main/protobuf/ZooKeeper.proto b72cb28
bq. src/main/protobuf/hbase.proto 30a4c3f
bq. src/test/java/org/apache/hadoop/hbase/TestHColumnDescriptor.java e7fa8b2
bq. src/test/java/org/apache/hadoop/hbase/TestHTableDescriptor.java f7c0cca
bq. src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 83d8408
bq. src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 69ccc65
bq. src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 1020374
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java 6dfba41
bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreFile.java 988d0bf
bq. src/test/java/org/apache/hadoop/hbase/util/TestFSUtils.java 339a120
bq.
bq. Diff: https://reviews.apache.org/r/5130/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v10.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Status: Patch Available (was: Open)
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Attachment: 5453v10.txt
Fix bug in my Reference file serialize/deserialize
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Attachments: 5453.txt, 5453v10.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Status: Patch Available (was: Open)
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Status: Patch Available (was: Open)
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13282118#comment-13282118 ]
Hudson commented on HBASE-5453:
-------------------------------
Integrated in HBase-TRUNK #2919 (See [https://builds.apache.org/job/HBase-TRUNK/2919/])
HBASE-6063 Replication related failures on trunk after HBASE-5453 (Gregory Channan) (Revision 1342095)
Result = FAILURE
larsh :
Files :
* /hbase/trunk/src/main/java/org/apache/hadoop/hbase/ClusterId.java
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v12.txt, 5453v13.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5453) Switch on-disk formats (reference
files, HFile meta fields, etc) to PB
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-5453:
-------------------------
Fix Version/s: 0.96.0
Release Note: Convert Reference, .tableinfo, .regioninfo, hbase.version, and hbase.id files to have pb content.
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> Switch on-disk formats (reference files, HFile meta fields, etc) to PB
> ----------------------------------------------------------------------
>
> Key: HBASE-5453
> URL: https://issues.apache.org/jira/browse/HBASE-5453
> Project: HBase
> Issue Type: Sub-task
> Components: ipc, master, migration, regionserver
> Reporter: Todd Lipcon
> Assignee: stack
> Fix For: 0.96.0
>
> Attachments: 5453.txt, 5453v10.txt, 5453v11.txt, 5453v11.txt, 5453v2.txt, 5453v3.txt, 5453v6.txt, 5453v9.txt
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira