You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Takanobu Asanuma <ta...@yahoo-corp.jp> on 2016/02/03 10:32:07 UTC
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
-----------------------------------------------------------
(Updated 2月 3, 2016, 9:32 a.m.)
Review request for hive.
Changes
-------
Sorry, I forgot to regenerate thrift files in the before patch. Please ignore that.
About this patch:
I changed typeName from required to optional in thrift API. If this is null, when new client sends the request to the old server, clients use old path without bypass.
I regenerated thrift files.
I merged latest master branch.
Repository: hive-git
Description
-------
This is a WIP patch for HIVE-11527
* I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
* I still have not considered security such as Kerberos and SSL at present.
* I have not implement Statement#setFetchSize for bypass yet.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6678de6
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209
jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812
ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336
service-rpc/if/TCLIService.thrift 0aa9d13
service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 5364293
service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp e62e8b7
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
service-rpc/src/gen/thrift/gen-php/Types.php 76805df
service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5
service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION
service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4
service/src/gen/thrift/gen-cpp/ThriftHive.h PRE-CREATION
service/src/gen/thrift/gen-cpp/ThriftHive.cpp PRE-CREATION
service/src/gen/thrift/gen-cpp/ThriftHive_server.skeleton.cpp PRE-CREATION
service/src/gen/thrift/gen-cpp/hive_service_constants.h PRE-CREATION
service/src/gen/thrift/gen-cpp/hive_service_constants.cpp PRE-CREATION
service/src/gen/thrift/gen-cpp/hive_service_types.h PRE-CREATION
service/src/gen/thrift/gen-cpp/hive_service_types.cpp PRE-CREATION
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveClusterStatus.java PRE-CREATION
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveServerException.java PRE-CREATION
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/JobTrackerState.java PRE-CREATION
service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java PRE-CREATION
service/src/gen/thrift/gen-php/ThriftHive.php PRE-CREATION
service/src/gen/thrift/gen-php/Types.php PRE-CREATION
service/src/gen/thrift/gen-py/__init__.py PRE-CREATION
service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote PRE-CREATION
service/src/gen/thrift/gen-py/hive_service/ThriftHive.py PRE-CREATION
service/src/gen/thrift/gen-py/hive_service/__init__.py PRE-CREATION
service/src/gen/thrift/gen-py/hive_service/constants.py PRE-CREATION
service/src/gen/thrift/gen-py/hive_service/ttypes.py PRE-CREATION
service/src/gen/thrift/gen-rb/hive_service_constants.rb PRE-CREATION
service/src/gen/thrift/gen-rb/hive_service_types.rb PRE-CREATION
service/src/gen/thrift/gen-rb/thrift_hive.rb PRE-CREATION
service/src/java/org/apache/hive/service/cli/CLIService.java cc6e138
service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06
service/src/java/org/apache/hive/service/cli/operation/Operation.java 0c263cf
service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f1ce6f6
service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 01b1d3d
service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d
service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 6aee80c
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java d9a9e3b
Diff: https://reviews.apache.org/r/40867/diff/
Testing
-------
I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
Thanks,
Takanobu Asanuma
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Takanobu Asanuma <ta...@yahoo-corp.jp>.
> On 3月 18, 2016, 5:57 p.m., Sergey Shelukhin wrote:
> >
Thanks for the reviewing.
> On 3月 18, 2016, 5:57 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 210
> > <https://reviews.apache.org/r/40867/diff/6-7/?file=1230821#file1230821line210>
> >
> > nit: whitespace
Sorry, I'll fix it.
> On 3月 18, 2016, 5:57 p.m., Sergey Shelukhin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java, line 1919
> > <https://reviews.apache.org/r/40867/diff/6-7/?file=1230823#file1230823line1919>
> >
> > why cannot the conf itself be passed in? why is the map necessary?
> > Or, if that doesn't work for some reason, what about creating a small struct class to hold the fields with logical names and types, instead of the map.
> why cannot the conf itself be passed in?
In this patch, HiveServer2 gives JDBC clients the configurations which is the necessary and sufficient condition for resolving namespace of HA.
In present implementation of Hive, the configurations on the JDBC clients side are different from ones on the cluster side. So if all the configurations are passed in here, it would destroy compatibility with existing JDBC applications. Even if it is better, I think it should be done in another jira.
> what about creating a small struct class to hold the fields with logical names and types, instead of the map.
We need a simple key-value structure in here. Map is a simple way to do this and we don't need to define the new structure in the thrift API. So I used the map. What the advantages of creating a new structure class?
- Takanobu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review124234
-----------------------------------------------------------
On 3月 16, 2016, 8:51 a.m., Takanobu Asanuma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> -----------------------------------------------------------
>
> (Updated 3月 16, 2016, 8:51 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This is a WIP patch for HIVE-11527
>
> * I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
>
> * I still have not considered security such as Kerberos and SSL at present.
>
> * I have not implement Statement#setFetchSize for bypass yet.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372
> jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209
> jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 7327a42
> ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336
> service-rpc/if/TCLIService.thrift aa28b6e
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java 1aa3f94
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
> service-rpc/src/gen/thrift/gen-php/Types.php b7df50a
> service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781
> service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION
> service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c
> service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2
> service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06
> service/src/java/org/apache/hive/service/cli/operation/Operation.java d9a273b
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 56a9c18
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 04d816a
> service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 8baecdf
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 62fcde5
>
> Diff: https://reviews.apache.org/r/40867/diff/
>
>
> Testing
> -------
>
> I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
>
>
> Thanks,
>
> Takanobu Asanuma
>
>
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review124234
-----------------------------------------------------------
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 210)
<https://reviews.apache.org/r/40867/#comment186691>
nit: whitespace
ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1917)
<https://reviews.apache.org/r/40867/#comment186693>
why cannot the conf itself be passed in? why is the map necessary?
Or, if that doesn't work for some reason, what about creating a small struct class to hold the fields with logical names and types, instead of the map.
- Sergey Shelukhin
On March 16, 2016, 8:51 a.m., Takanobu Asanuma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> -----------------------------------------------------------
>
> (Updated March 16, 2016, 8:51 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This is a WIP patch for HIVE-11527
>
> * I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
>
> * I still have not considered security such as Kerberos and SSL at present.
>
> * I have not implement Statement#setFetchSize for bypass yet.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372
> jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209
> jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 7327a42
> ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336
> service-rpc/if/TCLIService.thrift aa28b6e
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java 1aa3f94
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
> service-rpc/src/gen/thrift/gen-php/Types.php b7df50a
> service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781
> service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION
> service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c
> service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2
> service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06
> service/src/java/org/apache/hive/service/cli/operation/Operation.java d9a273b
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 56a9c18
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 04d816a
> service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 8baecdf
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 62fcde5
>
> Diff: https://reviews.apache.org/r/40867/diff/
>
>
> Testing
> -------
>
> I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
>
>
> Thanks,
>
> Takanobu Asanuma
>
>
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Takanobu Asanuma <ta...@yahoo-corp.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review138763
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1992)
<https://reviews.apache.org/r/40867/#comment203959>
I forgot to add a log when useBypass is true. I will add it in the next patch.
- Takanobu Asanuma
On 6\u6708 21, 2016, 2:10 a.m., Takanobu Asanuma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> -----------------------------------------------------------
>
> (Updated 6\u6708 21, 2016, 2:10 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This is a WIP patch for HIVE-11527
>
> * I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
>
> * I still have not considered security such as Kerberos and SSL at present.
>
> * I have not implement Statement#setFetchSize for bypass yet.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1e0ffa4
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 84644d1
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 0c313a2
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 637e51a
> jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 8ac040e
> jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca
> jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java a242501
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 2263192
> ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815
> service-rpc/if/TCLIService.thrift 5a9a785
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h d23b3cd
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 0f53cb2
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java 1aa3f94
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq.java c6279dc
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionResp.java 1eaeee6
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
> service-rpc/src/gen/thrift/gen-php/Types.php a6a257f
> service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fcd330f
> service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 71148a0
> service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a
> service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135
> service/src/java/org/apache/hive/service/cli/operation/Operation.java d48b92c
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 2f18231
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 3bf40eb
> service/src/java/org/apache/hive/service/cli/session/HiveSession.java 78ff388
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 7341635
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 8bc3d94
>
> Diff: https://reviews.apache.org/r/40867/diff/
>
>
> Testing
> -------
>
> I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
>
>
> Thanks,
>
> Takanobu Asanuma
>
>
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Takanobu Asanuma <ta...@yahoo-corp.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
-----------------------------------------------------------
(Updated 6\u6708 21, 2016, 2:10 a.m.)
Review request for hive.
Changes
-------
I updated the patch based on Thejas's advice.
Repository: hive-git
Description
-------
This is a WIP patch for HIVE-11527
* I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
* I still have not considered security such as Kerberos and SSL at present.
* I have not implement Statement#setFetchSize for bypass yet.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1e0ffa4
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 84644d1
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 0c313a2
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 637e51a
jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 8ac040e
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca
jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java a242501
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 2263192
ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815
service-rpc/if/TCLIService.thrift 5a9a785
service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h d23b3cd
service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 0f53cb2
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java 1aa3f94
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq.java c6279dc
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionResp.java 1eaeee6
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
service-rpc/src/gen/thrift/gen-php/Types.php a6a257f
service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fcd330f
service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 71148a0
service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a
service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135
service/src/java/org/apache/hive/service/cli/operation/Operation.java d48b92c
service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 2f18231
service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 3bf40eb
service/src/java/org/apache/hive/service/cli/session/HiveSession.java 78ff388
service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 7341635
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 8bc3d94
Diff: https://reviews.apache.org/r/40867/diff/
Testing
-------
I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
Thanks,
Takanobu Asanuma
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Takanobu Asanuma <ta...@yahoo-corp.jp>.
> On 6\u6708 18, 2016, 12:21 a.m., Thejas Nair wrote:
> >
I modified some codes in the latest patch.
> On 6\u6708 18, 2016, 12:21 a.m., Thejas Nair wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 57
> > <https://reviews.apache.org/r/40867/diff/10/?file=1419433#file1419433line57>
> >
> > Use ReflectionUtils that exists within Hive ?
> > The hadoop one is not a public class.
> > I know that we use the hadoop one in other parts, but that is something we should move away from.
I fixed it to use Hive ReflectionUtils.
> On 6\u6708 18, 2016, 12:21 a.m., Thejas Nair wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 536
> > <https://reviews.apache.org/r/40867/diff/10/?file=1419433#file1419433line536>
> >
> > Wouldn't it be more performant to use LazyBinarySimpleSerde ?
Excuse me, what is LazyBinarySimpleSerde?
> On 6\u6708 18, 2016, 12:21 a.m., Thejas Nair wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java, line 1972
> > <https://reviews.apache.org/r/40867/diff/10/?file=1419435#file1419435line1972>
> >
> > We should log if the bypass is kicking in or not , and the reason for disabling it.
> > There are so many cases where it gets disabled, it would be hard to debug in a production environment what the reason is.
I fixed it to log the reasons. Since I am not a native speaker of English, please let me know if my English expressions are mistaken.
> On 6\u6708 18, 2016, 12:21 a.m., Thejas Nair wrote:
> > service-rpc/if/TCLIService.thrift, line 277
> > <https://reviews.apache.org/r/40867/diff/10/?file=1419437#file1419437line277>
> >
> > There is already typeDesc, why do we need typeName
It is to handle complex column types. typeDesc does not have enough infomation for complex column types. For example, "struct<int, string>" is regarded as "struct". On the other hand, typename has detailed infomation as is.
- Takanobu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review138347
-----------------------------------------------------------
On 6\u6708 21, 2016, 2:10 a.m., Takanobu Asanuma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> -----------------------------------------------------------
>
> (Updated 6\u6708 21, 2016, 2:10 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This is a WIP patch for HIVE-11527
>
> * I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
>
> * I still have not considered security such as Kerberos and SSL at present.
>
> * I have not implement Statement#setFetchSize for bypass yet.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1e0ffa4
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 84644d1
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 0c313a2
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 637e51a
> jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java 8ac040e
> jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca
> jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java a242501
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 2263192
> ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815
> service-rpc/if/TCLIService.thrift 5a9a785
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h d23b3cd
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 0f53cb2
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java 1aa3f94
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionReq.java c6279dc
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TOpenSessionResp.java 1eaeee6
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
> service-rpc/src/gen/thrift/gen-php/Types.php a6a257f
> service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fcd330f
> service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 71148a0
> service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a
> service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135
> service/src/java/org/apache/hive/service/cli/operation/Operation.java d48b92c
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 2f18231
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 3bf40eb
> service/src/java/org/apache/hive/service/cli/session/HiveSession.java 78ff388
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 7341635
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 8bc3d94
>
> Diff: https://reviews.apache.org/r/40867/diff/
>
>
> Testing
> -------
>
> I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
>
>
> Thanks,
>
> Takanobu Asanuma
>
>
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review138347
-----------------------------------------------------------
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 57)
<https://reviews.apache.org/r/40867/#comment203540>
Use ReflectionUtils that exists within Hive ?
The hadoop one is not a public class.
I know that we use the hadoop one in other parts, but that is something we should move away from.
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 535)
<https://reviews.apache.org/r/40867/#comment203535>
Wouldn't it be more performant to use LazyBinarySimpleSerde ?
ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1972)
<https://reviews.apache.org/r/40867/#comment203541>
We should log if the bypass is kicking in or not , and the reason for disabling it.
There are so many cases where it gets disabled, it would be hard to debug in a production environment what the reason is.
service-rpc/if/TCLIService.thrift (line 277)
<https://reviews.apache.org/r/40867/#comment203519>
There is already typeDesc, why do we need typeName
- Thejas Nair
On June 15, 2016, 6:50 a.m., Takanobu Asanuma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> -----------------------------------------------------------
>
> (Updated June 15, 2016, 6:50 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This is a WIP patch for HIVE-11527
>
> * I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
>
> * I still have not considered security such as Kerberos and SSL at present.
>
> * I have not implement Statement#setFetchSize for bypass yet.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 761dbb2
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 84644d1
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 0c313a2
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 637e51a
> jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca
> jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java a242501
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 2263192
> ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815
> service-rpc/if/TCLIService.thrift 5a9a785
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h d23b3cd
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 0f53cb2
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java 1aa3f94
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
> service-rpc/src/gen/thrift/gen-php/Types.php a6a257f
> service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fcd330f
> service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 71148a0
> service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a
> service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135
> service/src/java/org/apache/hive/service/cli/operation/Operation.java d48b92c
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 2f18231
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 3bf40eb
> service/src/java/org/apache/hive/service/cli/session/HiveSession.java 78ff388
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 7341635
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 8bc3d94
>
> Diff: https://reviews.apache.org/r/40867/diff/
>
>
> Testing
> -------
>
> I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
>
>
> Thanks,
>
> Takanobu Asanuma
>
>
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Takanobu Asanuma <ta...@yahoo-corp.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
-----------------------------------------------------------
(Updated 6\u6708 15, 2016, 6:50 a.m.)
Review request for hive.
Changes
-------
I updated the patch based on Vaibhav's advice.
Repository: hive-git
Description
-------
This is a WIP patch for HIVE-11527
* I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
* I still have not considered security such as Kerberos and SSL at present.
* I have not implement Statement#setFetchSize for bypass yet.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 761dbb2
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 84644d1
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 0c313a2
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 637e51a
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca
jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java a242501
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 2263192
ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815
service-rpc/if/TCLIService.thrift 5a9a785
service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h d23b3cd
service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 0f53cb2
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java 1aa3f94
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
service-rpc/src/gen/thrift/gen-php/Types.php a6a257f
service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fcd330f
service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 71148a0
service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a
service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135
service/src/java/org/apache/hive/service/cli/operation/Operation.java d48b92c
service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 2f18231
service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 3bf40eb
service/src/java/org/apache/hive/service/cli/session/HiveSession.java 78ff388
service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 7341635
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 8bc3d94
Diff: https://reviews.apache.org/r/40867/diff/
Testing
-------
I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
Thanks,
Takanobu Asanuma
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Takanobu Asanuma <ta...@yahoo-corp.jp>.
> On 6\u6708 3, 2016, 5:50 p.m., Vaibhav Gumashta wrote:
> > itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java, line 157
> > <https://reviews.apache.org/r/40867/diff/9/?file=1386567#file1386567line157>
> >
> > Can you add a test with a join query as well? The join query should write the results in a new intermediate file on hdfs and it will be good to test that.
I added it in the latest patch.
> On 6\u6708 3, 2016, 5:50 p.m., Vaibhav Gumashta wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 552
> > <https://reviews.apache.org/r/40867/diff/9/?file=1386570#file1386570line552>
> >
> > Nit: Usually an iterator implements https://docs.oracle.com/javase/7/docs/api/java/util/Iterator.html.
This class wrappers SequenceFile.Reader. And I think implementing an iterator is not suitable in this class. (Implementing hasNext() is not easy.) I just changed the class name to avoid confusion.
> On 6\u6708 3, 2016, 5:50 p.m., Vaibhav Gumashta wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 573
> > <https://reviews.apache.org/r/40867/diff/9/?file=1386570#file1386570line573>
> >
> > Nit: getXXX methods are usually used to return something other than void.
I think the name is not valid. I changed the name in the latest patch.
> On 6\u6708 3, 2016, 5:50 p.m., Vaibhav Gumashta wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 590
> > <https://reviews.apache.org/r/40867/diff/9/?file=1386570#file1386570line590>
> >
> > This should be a private method.
I fixed it.
> On 6\u6708 3, 2016, 5:50 p.m., Vaibhav Gumashta wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java, line 1978
> > <https://reviews.apache.org/r/40867/diff/9/?file=1386572#file1386572line1978>
> >
> > Shouldn't the read happen independent of the file format? For example, the default format used to be TextFile until very recently and the user can as well choose to configure it that way.
I thought handling other formats will make code complex. If this should handle TextFile or other formats, I want to create a follow-up jira.
> On 6\u6708 3, 2016, 5:50 p.m., Vaibhav Gumashta wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 536
> > <https://reviews.apache.org/r/40867/diff/9/?file=1386570#file1386570line536>
> >
> > This hardcodes the serde to LazySimpleSerde. There is work in https://issues.apache.org/jira/browse/HIVE-12049 where we write using a different serde in the final tasks. However, I'll create a follow-up jira for making this more generic.
I got it. Thank you.
> On 6\u6708 3, 2016, 5:50 p.m., Vaibhav Gumashta wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Driver.java, line 1976
> > <https://reviews.apache.org/r/40867/diff/9/?file=1386572#file1386572line1976>
> >
> > We can create a follow-up jira to handle this.
Shall I create the jira?
- Takanobu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review136073
-----------------------------------------------------------
On 6\u6708 15, 2016, 6:50 a.m., Takanobu Asanuma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> -----------------------------------------------------------
>
> (Updated 6\u6708 15, 2016, 6:50 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This is a WIP patch for HIVE-11527
>
> * I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
>
> * I still have not considered security such as Kerberos and SSL at present.
>
> * I have not implement Statement#setFetchSize for bypass yet.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 761dbb2
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 84644d1
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 0c313a2
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 637e51a
> jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca
> jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java a242501
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 2263192
> ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815
> service-rpc/if/TCLIService.thrift 5a9a785
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h d23b3cd
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 0f53cb2
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java 1aa3f94
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
> service-rpc/src/gen/thrift/gen-php/Types.php a6a257f
> service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py fcd330f
> service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 71148a0
> service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a
> service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135
> service/src/java/org/apache/hive/service/cli/operation/Operation.java d48b92c
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 2f18231
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 3bf40eb
> service/src/java/org/apache/hive/service/cli/session/HiveSession.java 78ff388
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 7341635
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 8bc3d94
>
> Diff: https://reviews.apache.org/r/40867/diff/
>
>
> Testing
> -------
>
> I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
>
>
> Thanks,
>
> Takanobu Asanuma
>
>
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Vaibhav Gumashta <vg...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review136073
-----------------------------------------------------------
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java (line 157)
<https://reviews.apache.org/r/40867/#comment201047>
Can you add a test with a join query as well? The join query should write the results in a new intermediate file on hdfs and it will be good to test that.
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 535)
<https://reviews.apache.org/r/40867/#comment201054>
This hardcodes the serde to LazySimpleSerde. There is work in https://issues.apache.org/jira/browse/HIVE-12049 where we write using a different serde in the final tasks. However, I'll create a follow-up jira for making this more generic.
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 551)
<https://reviews.apache.org/r/40867/#comment201056>
Nit: Usually an iterator implements https://docs.oracle.com/javase/7/docs/api/java/util/Iterator.html.
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 572)
<https://reviews.apache.org/r/40867/#comment201055>
Nit: getXXX methods are usually used to return something other than void.
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 589)
<https://reviews.apache.org/r/40867/#comment201058>
This should be a private method.
ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1976)
<https://reviews.apache.org/r/40867/#comment201065>
We can create a follow-up jira to handle this.
ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1978)
<https://reviews.apache.org/r/40867/#comment201067>
Shouldn't the read happen independent of the file format? For example, the default format used to be TextFile until very recently and the user can as well choose to configure it that way.
- Vaibhav Gumashta
On May 18, 2016, 9:54 a.m., Takanobu Asanuma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> -----------------------------------------------------------
>
> (Updated May 18, 2016, 9:54 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This is a WIP patch for HIVE-11527
>
> * I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
>
> * I still have not considered security such as Kerberos and SSL at present.
>
> * I have not implement Statement#setFetchSize for bypass yet.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbb3a72
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 84644d1
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 815ccfa
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 637e51a
> jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca
> jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 38ccc78
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3fecc5c
> ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815
> service-rpc/if/TCLIService.thrift 9879b1b
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 838bf17
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 5229230
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java 1aa3f94
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
> service-rpc/src/gen/thrift/gen-php/Types.php 9ed7403
> service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py 44e5462
> service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb b39ec1e
> service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a
> service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 0932884
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 2f18231
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 67e0e52
> service/src/java/org/apache/hive/service/cli/session/HiveSession.java 78ff388
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java a0015eb
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 5464e58
>
> Diff: https://reviews.apache.org/r/40867/diff/
>
>
> Testing
> -------
>
> I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
>
>
> Thanks,
>
> Takanobu Asanuma
>
>
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Takanobu Asanuma <ta...@yahoo-corp.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
-----------------------------------------------------------
(Updated 5\u6708 18, 2016, 9:54 a.m.)
Review request for hive.
Changes
-------
I updated the patch based on Sergey's advice. Also I merged the latest master branch.
Repository: hive-git
Description
-------
This is a WIP patch for HIVE-11527
* I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
* I still have not considered security such as Kerberos and SSL at present.
* I have not implement Statement#setFetchSize for bypass yet.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java cbb3a72
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 84644d1
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 815ccfa
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 637e51a
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 92fdbca
jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 38ccc78
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 3fecc5c
ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java dff1815
service-rpc/if/TCLIService.thrift 9879b1b
service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 838bf17
service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 5229230
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java 1aa3f94
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
service-rpc/src/gen/thrift/gen-php/Types.php 9ed7403
service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py 44e5462
service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb b39ec1e
service/src/java/org/apache/hive/service/cli/CLIService.java ed52b4a
service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java bfd7135
service/src/java/org/apache/hive/service/cli/operation/Operation.java 0932884
service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 2f18231
service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 67e0e52
service/src/java/org/apache/hive/service/cli/session/HiveSession.java 78ff388
service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java a0015eb
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 5464e58
Diff: https://reviews.apache.org/r/40867/diff/
Testing
-------
I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
Thanks,
Takanobu Asanuma
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Takanobu Asanuma <ta...@yahoo-corp.jp>.
> On 4\u6708 22, 2016, 1:33 a.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 565
> > <https://reviews.apache.org/r/40867/diff/7-8/?file=1300872#file1300872line565>
> >
> > nit: is it enough to just use Arrays.asList?
That's right. I will use just List here.
> On 4\u6708 22, 2016, 1:33 a.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 588
> > <https://reviews.apache.org/r/40867/diff/7-8/?file=1300872#file1300872line588>
> >
> > why does this swallow the error? It should fail on error
This is my mistake. I think it should not use a try-catch statement.
- Takanobu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review130014
-----------------------------------------------------------
On 4\u6708 15, 2016, 11:38 a.m., Takanobu Asanuma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> -----------------------------------------------------------
>
> (Updated 4\u6708 15, 2016, 11:38 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This is a WIP patch for HIVE-11527
>
> * I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
>
> * I still have not considered security such as Kerberos and SSL at present.
>
> * I have not implement Statement#setFetchSize for bypass yet.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c7e5b33
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 84644d1
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 857805a
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 637e51a
> jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209
> jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 0bbd0e3
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 92c2c76
> ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336
> service-rpc/if/TCLIService.thrift aa28b6e
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java 1aa3f94
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
> service-rpc/src/gen/thrift/gen-php/Types.php b7df50a
> service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781
> service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c
> service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2
> service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06
> service/src/java/org/apache/hive/service/cli/operation/Operation.java b7d6549
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 56a9c18
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 9ce6055
> service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 0f36cd6
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java be9833d
>
> Diff: https://reviews.apache.org/r/40867/diff/
>
>
> Testing
> -------
>
> I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
>
>
> Thanks,
>
> Takanobu Asanuma
>
>
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review130014
-----------------------------------------------------------
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 557)
<https://reviews.apache.org/r/40867/#comment193626>
nit: is it enough to just use Arrays.asList?
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 580)
<https://reviews.apache.org/r/40867/#comment193630>
why does this swallow the error? It should fail on error
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 581)
<https://reviews.apache.org/r/40867/#comment193629>
nit: null check, there could be an error before stream is created
- Sergey Shelukhin
On April 15, 2016, 11:38 a.m., Takanobu Asanuma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> -----------------------------------------------------------
>
> (Updated April 15, 2016, 11:38 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This is a WIP patch for HIVE-11527
>
> * I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
>
> * I still have not considered security such as Kerberos and SSL at present.
>
> * I have not implement Statement#setFetchSize for bypass yet.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c7e5b33
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 84644d1
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 857805a
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 637e51a
> jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209
> jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 0bbd0e3
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 92c2c76
> ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336
> service-rpc/if/TCLIService.thrift aa28b6e
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java 1aa3f94
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
> service-rpc/src/gen/thrift/gen-php/Types.php b7df50a
> service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781
> service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c
> service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2
> service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06
> service/src/java/org/apache/hive/service/cli/operation/Operation.java b7d6549
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 56a9c18
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 9ce6055
> service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 0f36cd6
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java be9833d
>
> Diff: https://reviews.apache.org/r/40867/diff/
>
>
> Testing
> -------
>
> I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
>
>
> Thanks,
>
> Takanobu Asanuma
>
>
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Takanobu Asanuma <ta...@yahoo-corp.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
-----------------------------------------------------------
(Updated 4月 15, 2016, 11:38 a.m.)
Review request for hive.
Changes
-------
I uploaded a new patch. I implemented some features for handling SequenceFile, handling multiple result files, handling ResultSet#beforeFirst. And I added some unit tests.
I think I have almost finished implementing the codes. So I will left a comment which is the summary of all my implementation.
Repository: hive-git
Description
-------
This is a WIP patch for HIVE-11527
* I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
* I still have not considered security such as Kerberos and SSL at present.
* I have not implement Statement#setFetchSize for bypass yet.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c7e5b33
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHA.java 84644d1
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 857805a
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniMr.java 637e51a
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209
jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java 0bbd0e3
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 92c2c76
ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336
service-rpc/if/TCLIService.thrift aa28b6e
service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd
service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java 1aa3f94
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
service-rpc/src/gen/thrift/gen-php/Types.php b7df50a
service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781
service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c
service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2
service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06
service/src/java/org/apache/hive/service/cli/operation/Operation.java b7d6549
service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 56a9c18
service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 9ce6055
service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d
service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 0f36cd6
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java be9833d
Diff: https://reviews.apache.org/r/40867/diff/
Testing
-------
I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
Thanks,
Takanobu Asanuma
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Takanobu Asanuma <ta...@yahoo-corp.jp>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/
-----------------------------------------------------------
(Updated 3月 16, 2016, 8:51 a.m.)
Review request for hive.
Changes
-------
Implemented features for handling HA. HiveServer2 puts HA configurations to a Map in Driver#getFinalDirName() and send it to jdbc clients.
Repository: hive-git
Description
-------
This is a WIP patch for HIVE-11527
* I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
* I still have not considered security such as Kerberos and SSL at present.
* I have not implement Statement#setFetchSize for bypass yet.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 98c6372
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209
jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 7327a42
ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336
service-rpc/if/TCLIService.thrift aa28b6e
service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 7f1d9dd
service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp 3a27a60
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TGetTablesReq.java 1aa3f94
service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
service-rpc/src/gen/thrift/gen-php/Types.php b7df50a
service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py c691781
service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION
service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb 07ed97c
service/src/java/org/apache/hive/service/cli/CLIService.java ab30ae2
service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06
service/src/java/org/apache/hive/service/cli/operation/Operation.java d9a273b
service/src/java/org/apache/hive/service/cli/operation/OperationManager.java 56a9c18
service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 04d816a
service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d
service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 8baecdf
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 62fcde5
Diff: https://reviews.apache.org/r/40867/diff/
Testing
-------
I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
Thanks,
Takanobu Asanuma
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Takanobu Asanuma <ta...@yahoo-corp.jp>.
> On Feb. 3, 2016, 10:41 p.m., Sergey Shelukhin wrote:
> > jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java, line 530
> > <https://reviews.apache.org/r/40867/diff/6/?file=1230821#file1230821line530>
> >
> > actually, do we expect more than one file here, should we handle this case?
I have some questions.
1. Is there any SELECT statement which creates multiple result files in the final directory? I tested some queries, but I wasn't able to find that query.
2. If there are multiple result files, can we assume that all those files' names are numbers and each file's content accords with the series?
- Takanobu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review117697
-----------------------------------------------------------
On Feb. 3, 2016, 9:32 a.m., Takanobu Asanuma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> -----------------------------------------------------------
>
> (Updated Feb. 3, 2016, 9:32 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This is a WIP patch for HIVE-11527
>
> * I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
>
> * I still have not considered security such as Kerberos and SSL at present.
>
> * I have not implement Statement#setFetchSize for bypass yet.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6678de6
> jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209
> jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812
> ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336
> service-rpc/if/TCLIService.thrift 0aa9d13
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 5364293
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp e62e8b7
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
> service-rpc/src/gen/thrift/gen-php/Types.php 76805df
> service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5
> service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION
> service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4
> service/src/gen/thrift/gen-cpp/ThriftHive.h PRE-CREATION
> service/src/gen/thrift/gen-cpp/ThriftHive.cpp PRE-CREATION
> service/src/gen/thrift/gen-cpp/ThriftHive_server.skeleton.cpp PRE-CREATION
> service/src/gen/thrift/gen-cpp/hive_service_constants.h PRE-CREATION
> service/src/gen/thrift/gen-cpp/hive_service_constants.cpp PRE-CREATION
> service/src/gen/thrift/gen-cpp/hive_service_types.h PRE-CREATION
> service/src/gen/thrift/gen-cpp/hive_service_types.cpp PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveClusterStatus.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveServerException.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/JobTrackerState.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java PRE-CREATION
> service/src/gen/thrift/gen-php/ThriftHive.php PRE-CREATION
> service/src/gen/thrift/gen-php/Types.php PRE-CREATION
> service/src/gen/thrift/gen-py/__init__.py PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/ThriftHive.py PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/__init__.py PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/constants.py PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/ttypes.py PRE-CREATION
> service/src/gen/thrift/gen-rb/hive_service_constants.rb PRE-CREATION
> service/src/gen/thrift/gen-rb/hive_service_types.rb PRE-CREATION
> service/src/gen/thrift/gen-rb/thrift_hive.rb PRE-CREATION
> service/src/java/org/apache/hive/service/cli/CLIService.java cc6e138
> service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 0c263cf
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f1ce6f6
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 01b1d3d
> service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 6aee80c
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java d9a9e3b
>
> Diff: https://reviews.apache.org/r/40867/diff/
>
>
> Testing
> -------
>
> I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
>
>
> Thanks,
>
> Takanobu Asanuma
>
>
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review117697
-----------------------------------------------------------
jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java (line 529)
<https://reviews.apache.org/r/40867/#comment178971>
actually, do we expect more than one file here, should we handle this case?
- Sergey Shelukhin
On Feb. 3, 2016, 9:32 a.m., Takanobu Asanuma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> -----------------------------------------------------------
>
> (Updated Feb. 3, 2016, 9:32 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This is a WIP patch for HIVE-11527
>
> * I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
>
> * I still have not considered security such as Kerberos and SSL at present.
>
> * I have not implement Statement#setFetchSize for bypass yet.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6678de6
> jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209
> jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812
> ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336
> service-rpc/if/TCLIService.thrift 0aa9d13
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 5364293
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp e62e8b7
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
> service-rpc/src/gen/thrift/gen-php/Types.php 76805df
> service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5
> service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION
> service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4
> service/src/gen/thrift/gen-cpp/ThriftHive.h PRE-CREATION
> service/src/gen/thrift/gen-cpp/ThriftHive.cpp PRE-CREATION
> service/src/gen/thrift/gen-cpp/ThriftHive_server.skeleton.cpp PRE-CREATION
> service/src/gen/thrift/gen-cpp/hive_service_constants.h PRE-CREATION
> service/src/gen/thrift/gen-cpp/hive_service_constants.cpp PRE-CREATION
> service/src/gen/thrift/gen-cpp/hive_service_types.h PRE-CREATION
> service/src/gen/thrift/gen-cpp/hive_service_types.cpp PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveClusterStatus.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveServerException.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/JobTrackerState.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java PRE-CREATION
> service/src/gen/thrift/gen-php/ThriftHive.php PRE-CREATION
> service/src/gen/thrift/gen-php/Types.php PRE-CREATION
> service/src/gen/thrift/gen-py/__init__.py PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/ThriftHive.py PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/__init__.py PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/constants.py PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/ttypes.py PRE-CREATION
> service/src/gen/thrift/gen-rb/hive_service_constants.rb PRE-CREATION
> service/src/gen/thrift/gen-rb/hive_service_types.rb PRE-CREATION
> service/src/gen/thrift/gen-rb/thrift_hive.rb PRE-CREATION
> service/src/java/org/apache/hive/service/cli/CLIService.java cc6e138
> service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 0c263cf
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f1ce6f6
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 01b1d3d
> service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 6aee80c
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java d9a9e3b
>
> Diff: https://reviews.apache.org/r/40867/diff/
>
>
> Testing
> -------
>
> I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
>
>
> Thanks,
>
> Takanobu Asanuma
>
>
Re: Review Request 40867: HIVE-11527 - bypass HiveServer2 thrift
interface for query results
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40867/#review117680
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/Driver.java (line 1870)
<https://reviews.apache.org/r/40867/#comment178955>
port can be missing in a valid HDFS url, and it will be missing if HA is used. It may be better to use URI class to get the components of the URL.
- Sergey Shelukhin
On Feb. 3, 2016, 9:32 a.m., Takanobu Asanuma wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40867/
> -----------------------------------------------------------
>
> (Updated Feb. 3, 2016, 9:32 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> This is a WIP patch for HIVE-11527
>
> * I added a new configuration whose name is hive.server2.webhdfs.bypass.enabled. The default is false. When this value is true, clients use the bypass.
>
> * I still have not considered security such as Kerberos and SSL at present.
>
> * I have not implement Statement#setFetchSize for bypass yet.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 6678de6
> jdbc/src/java/org/apache/hive/jdbc/HiveQueryResultSet.java 8f67209
> jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java b4dba44
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 4c89812
> ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 0b0c336
> service-rpc/if/TCLIService.thrift 0aa9d13
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.h 5364293
> service-rpc/src/gen/thrift/gen-cpp/TCLIService_types.cpp e62e8b7
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TColumnDesc.java 31472c8
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TExecuteStatementResp.java 7101fa5
> service-rpc/src/gen/thrift/gen-javabean/org/apache/hive/service/rpc/thrift/TProtocolVersion.java 14d50ed
> service-rpc/src/gen/thrift/gen-php/Types.php 76805df
> service-rpc/src/gen/thrift/gen-py/TCLIService/ttypes.py ef5f5f5
> service-rpc/src/gen/thrift/gen-py/__init__.py PRE-CREATION
> service-rpc/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb f004ec4
> service/src/gen/thrift/gen-cpp/ThriftHive.h PRE-CREATION
> service/src/gen/thrift/gen-cpp/ThriftHive.cpp PRE-CREATION
> service/src/gen/thrift/gen-cpp/ThriftHive_server.skeleton.cpp PRE-CREATION
> service/src/gen/thrift/gen-cpp/hive_service_constants.h PRE-CREATION
> service/src/gen/thrift/gen-cpp/hive_service_constants.cpp PRE-CREATION
> service/src/gen/thrift/gen-cpp/hive_service_types.h PRE-CREATION
> service/src/gen/thrift/gen-cpp/hive_service_types.cpp PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveClusterStatus.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/HiveServerException.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/JobTrackerState.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/service/ThriftHive.java PRE-CREATION
> service/src/gen/thrift/gen-php/ThriftHive.php PRE-CREATION
> service/src/gen/thrift/gen-php/Types.php PRE-CREATION
> service/src/gen/thrift/gen-py/__init__.py PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/ThriftHive.py PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/__init__.py PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/constants.py PRE-CREATION
> service/src/gen/thrift/gen-py/hive_service/ttypes.py PRE-CREATION
> service/src/gen/thrift/gen-rb/hive_service_constants.rb PRE-CREATION
> service/src/gen/thrift/gen-rb/hive_service_types.rb PRE-CREATION
> service/src/gen/thrift/gen-rb/thrift_hive.rb PRE-CREATION
> service/src/java/org/apache/hive/service/cli/CLIService.java cc6e138
> service/src/java/org/apache/hive/service/cli/ColumnDescriptor.java 7bd9f06
> service/src/java/org/apache/hive/service/cli/operation/Operation.java 0c263cf
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java f1ce6f6
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 01b1d3d
> service/src/java/org/apache/hive/service/cli/session/HiveSession.java 4f4e92d
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 6aee80c
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java d9a9e3b
>
> Diff: https://reviews.apache.org/r/40867/diff/
>
>
> Testing
> -------
>
> I have tested few simple queries and they worked well. But I think there are some problems for some queries. I'm going to test more queries and fix bugs. I'm also going to add unit tests.
>
>
> Thanks,
>
> Takanobu Asanuma
>
>