You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Roshan Naik <ro...@hortonworks.com> on 2013/04/30 04:41:53 UTC
Review Request: Draft patch for review. Based on phase 1 mentioned in design
doc.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10857/
-----------------------------------------------------------
Review request for hive.
Description
-------
Draft patch for review. based on phase 1 mentioned in design doc. Deviates slighlty from doc in the follow ways...
1) adds a couple of (temporary) rest calls to enable/disable streaming on a table. Later these will be replaced with support in DDL.
2) Also also HTTP methods are GET for easy testing with web browser
3) Authentication disabled on the new streaming HTTP methods
Usage Examples on db named 'sdb' & table named 'log' :
1) Setup db & table with single partition column 'date':
hcat -e "create database sdb; use sdb; create table log(msg string, region string) partitioned by (date string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE; "
2) To check streaming status:
http://localhost:50111/templeton/v1/streaming/status?database=sdb&table=log
3) Enable Streaming:
http://localhost:50111/templeton/v1/streaming/enable?database=sdb&table=log&col=date&value=1000
4) Get Chunk File to write to:
http://localhost:50111/templeton/v1/streaming/chunkget?database=sdb&table=log&schema=blah&format=blah&record_separator=blah&field_separator=blah
5) Commit Chunk File:
http://localhost:50111/templeton/v1/streaming/chunkcommit?database=sdb&table=log&chunkfile=/user/hive/streaming/tmp/sdb/log/2
6) Abort Chunk File:
http://localhost:50111/templeton/v1/streaming/chunkabort?database=sdb&table=log&chunkfile=/user/hive/streaming/tmp/sdb/log/3
7) Roll Partition:
http://localhost:50111/templeton/v1/streaming/partitionroll?database=sdb&table=log&partition_column=date&partition_value=3000
This addresses bug HIVE-4196.
https://issues.apache.org/jira/browse/HIVE-4196
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c61d95b
hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/HcatStreamingDelegator.java PRE-CREATION
hcatalog/webhcat/svr/src/main/java/org/apache/hcatalog/templeton/Server.java 29ac4b3
metastore/if/hive_metastore.thrift c2051f4
metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 7b31d28
metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 3d69472
metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 3b90b44
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java d8d6e71
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedValueList.java 030b54a
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 5929cda
metastore/src/gen/thrift/gen-php/metastore/ThriftHiveMetastore.php a69d214
metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 6fd2cce
metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 9b856e5
metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 25aa30c
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java dc14084
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java cef50f4
metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java a2d6b1b
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 2079337
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 233fb46
metastore/src/model/org/apache/hadoop/hive/metastore/model/MTable.java 2a78ce9
metastore/src/model/package.jdo a84d2bf
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 00eb0b4
Diff: https://reviews.apache.org/r/10857/diff/
Testing
-------
Manual testing only
Thanks,
Roshan Naik
Re: Review Request 10857: Draft patch for review. Based on phase 1 mentioned
in design doc.
Posted by Roshan Naik <ro...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10857/
-----------------------------------------------------------
(Updated Sept. 17, 2013, 11:58 p.m.)
Review request for hive.
Changes
-------
I have dropped the generated thrift files from this patch update for ease of review.
Bugs: HIVE-5138
https://issues.apache.org/jira/browse/HIVE-5138
Repository: hive-git
Description
-------
Draft patch for review. based on phase 1 mentioned in design doc. Deviates slighlty from doc in the follow ways...
1) adds a couple of (temporary) rest calls to enable/disable streaming on a table. Later these will be replaced with support in DDL.
2) Also also HTTP methods are GET for easy testing with web browser
3) Authentication disabled on the new streaming HTTP methods
Usage Examples on db named 'sdb' & table named 'log' :
1) Setup db & table with single partition column 'date':
hcat -e "create database sdb; use sdb; create table log(msg string, region string) partitioned by (date string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE; "
2) To check streaming status:
http://localhost:50111/templeton/v1/streaming/status?database=sdb&table=log
3) Enable Streaming:
http://localhost:50111/templeton/v1/streaming/enable?database=sdb&table=log&col=date&value=1000
4) Get Chunk File to write to:
http://localhost:50111/templeton/v1/streaming/chunkget?database=sdb&table=log&schema=blah&format=blah&record_separator=blah&field_separator=blah
5) Commit Chunk File:
http://localhost:50111/templeton/v1/streaming/chunkcommit?database=sdb&table=log&chunkfile=/user/hive/streaming/tmp/sdb/log/2
6) Abort Chunk File:
http://localhost:50111/templeton/v1/streaming/chunkabort?database=sdb&table=log&chunkfile=/user/hive/streaming/tmp/sdb/log/3
7) Roll Partition:
http://localhost:50111/templeton/v1/streaming/partitionroll?database=sdb&table=log&partition_column=date&partition_value=3000
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1af68a6
hcatalog/webhcat/svr/pom.xml 16a870e
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/HcatStreamingDelegator.java PRE-CREATION
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Main.java 0f37278
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Server.java d8b6ce2
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestStreaming.java PRE-CREATION
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/HiveJdbcClient.java PRE-CREATION
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/WebHCatClient.java PRE-CREATION
metastore/if/hive_metastore.thrift 3fea057
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 39dda92
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java e165b2d
metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java a2d6b1b
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java a27243d
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java e410c3a
metastore/src/model/org/apache/hadoop/hive/metastore/model/MTable.java 2a78ce9
metastore/src/model/package.jdo c42b5b0
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 8066784
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 0f9b16c
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 80fbbac
Diff: https://reviews.apache.org/r/10857/diff/
Testing
-------
Manual testing only
Thanks,
Roshan Naik
Re: Review Request 10857: Draft patch for review. Based on phase 1 mentioned
in design doc.
Posted by Roshan Naik <ro...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/10857/
-----------------------------------------------------------
(Updated Sept. 17, 2013, 11:56 p.m.)
Review request for hive.
Changes
-------
Updating patch. Addresses all comments captured so far form Brock, Allan and Eugene.
Also added unit tests.
Bugs: HIVE-4196
https://issues.apache.org/jira/browse/HIVE-4196
Repository: hive-git
Description
-------
Draft patch for review. based on phase 1 mentioned in design doc. Deviates slighlty from doc in the follow ways...
1) adds a couple of (temporary) rest calls to enable/disable streaming on a table. Later these will be replaced with support in DDL.
2) Also also HTTP methods are GET for easy testing with web browser
3) Authentication disabled on the new streaming HTTP methods
Usage Examples on db named 'sdb' & table named 'log' :
1) Setup db & table with single partition column 'date':
hcat -e "create database sdb; use sdb; create table log(msg string, region string) partitioned by (date string) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' STORED AS TEXTFILE; "
2) To check streaming status:
http://localhost:50111/templeton/v1/streaming/status?database=sdb&table=log
3) Enable Streaming:
http://localhost:50111/templeton/v1/streaming/enable?database=sdb&table=log&col=date&value=1000
4) Get Chunk File to write to:
http://localhost:50111/templeton/v1/streaming/chunkget?database=sdb&table=log&schema=blah&format=blah&record_separator=blah&field_separator=blah
5) Commit Chunk File:
http://localhost:50111/templeton/v1/streaming/chunkcommit?database=sdb&table=log&chunkfile=/user/hive/streaming/tmp/sdb/log/2
6) Abort Chunk File:
http://localhost:50111/templeton/v1/streaming/chunkabort?database=sdb&table=log&chunkfile=/user/hive/streaming/tmp/sdb/log/3
7) Roll Partition:
http://localhost:50111/templeton/v1/streaming/partitionroll?database=sdb&table=log&partition_column=date&partition_value=3000
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1af68a6
hcatalog/webhcat/svr/pom.xml 16a870e
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/HcatStreamingDelegator.java PRE-CREATION
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Main.java 0f37278
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Server.java d8b6ce2
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/TestStreaming.java PRE-CREATION
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/HiveJdbcClient.java PRE-CREATION
hcatalog/webhcat/svr/src/test/java/org/apache/hive/hcatalog/templeton/tool/WebHCatClient.java PRE-CREATION
metastore/if/hive_metastore.thrift 3fea057
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 39dda92
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java e165b2d
metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java a2d6b1b
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java a27243d
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java e410c3a
metastore/src/model/org/apache/hadoop/hive/metastore/model/MTable.java 2a78ce9
metastore/src/model/package.jdo c42b5b0
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 8066784
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 0f9b16c
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 80fbbac
Diff: https://reviews.apache.org/r/10857/diff/
Testing
-------
Manual testing only
Thanks,
Roshan Naik