You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2020/03/24 10:06:08 UTC
[GitHub] [drill] vvysotskyi opened a new pull request #2038: DRILL-6604:
Upgrade Drill Hive client to Hive3.1 version
vvysotskyi opened a new pull request #2038: DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038
# [DRILL-6604](https://issues.apache.org/jira/browse/DRILL-6604): Upgrade Drill Hive client to Hive3.1 version
## Description
One of the major changes in this PR is cleaning up the output for Hive tests. Now, almost all hive-related stuff is not printed to the stdout.
Additionally, the Hive version was updated to the latest (at the current time) version 3.1.2.
New Hive version introduced new `ObjectInspector` classes for date and timestamp values, so to be able to compile with other versions, code was updated to use correct class names (see changes in `tdd` files and related changes.
As for any other Hive update, Dill wouldn't be able to work with previous Hive versions, but for users, who still want to do it, it is possible to compile manually Drill with setting the following properties:
`hive.version=2.3.2` (or any other version in range [2.3.2-3.1.2]), `freemarker.conf.file=src/main/codegen/config.fmpp` (or `src/main/codegen/configHive3.fmpp` for versions where new date / timestamp classes were introduced).
Example of usage:
```
mvn clean install -DskipTests -Dhive.version=2.3.2 -Dfreemarker.conf.file=src/main/codegen/config.fmpp
```
## Documentation
A new supported Hive version should be documented.
## Testing
Ran all tests suite, checked manually that Drill is able to select from new Hive.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] ihuzenko commented on a change in pull request #2038:
DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
Posted by GitBox <gi...@apache.org>.
ihuzenko commented on a change in pull request #2038: DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#discussion_r399113251
##########
File path: contrib/storage-hive/core/src/test/java/org/apache/drill/exec/impersonation/hive/BaseTestHiveImpersonation.java
##########
@@ -97,11 +99,29 @@ protected static void prepHiveConfAndData() throws Exception {
}
protected static void startHiveMetaStore() throws Exception {
- final int port = MetaStoreUtils.findFreePort();
+ Class<?> metaStoreUtilsClass;
+ Class<?> hadoopThriftAuthBridgeClass;
+ Class<?> confClass;
+ Object hadoopThriftAuthBridge;
+ // TODO: remove reflection stuff when all supported profiles will be switched to Hive 3+ version
Review comment:
It would be good to add Jira ticket to the comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] cgivre removed a comment on issue #2038: DRILL-6604:
Upgrade Drill Hive client to Hive3.1 version
Posted by GitBox <gi...@apache.org>.
cgivre removed a comment on issue #2038: DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#issuecomment-603219947
@vvysotskyi
Thank you for submitting this! I am wondering whether we test for backwards compatibility with older versions of Hive as we upgrade? What is the earliest version we currently support?
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] ihuzenko commented on a change in pull request #2038:
DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
Posted by GitBox <gi...@apache.org>.
ihuzenko commented on a change in pull request #2038: DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#discussion_r399110299
##########
File path: contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/writers/primitive/HiveDateWriter.java
##########
@@ -18,21 +18,23 @@
package org.apache.drill.exec.store.hive.writers.primitive;
import org.apache.drill.exec.vector.complex.writer.DateWriter;
-import org.apache.hadoop.hive.serde2.objectinspector.primitive.DateObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
-public class HiveDateWriter extends AbstractSingleValueWriter<DateObjectInspector, DateWriter> {
+public class HiveDateWriter extends AbstractSingleValueWriter<PrimitiveObjectInspector, DateWriter> {
- public HiveDateWriter(DateObjectInspector inspector, DateWriter writer) {
+ public HiveDateWriter(PrimitiveObjectInspector inspector, DateWriter writer) {
super(inspector, writer);
}
@Override
public void write(Object value) {
- final java.sql.Date dateValue = inspector.getPrimitiveJavaObject(value);
- final DateTime date = new DateTime(dateValue.getTime()).withZoneRetainFields(DateTimeZone.UTC);
- writer.writeDate(date.getMillis());
+ String dateString = PrimitiveObjectInspectorUtils.getString(value, inspector);
+ long time = new DateTime(java.sql.Date.valueOf(dateString).getTime())
Review comment:
```suggestion
long dateMillis = new DateTime(java.sql.Date.valueOf(dateString).getTime())
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] vvysotskyi commented on issue #2038: DRILL-6604: Upgrade
Drill Hive client to Hive3.1 version
Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on issue #2038: DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#issuecomment-603237847
@cgivre,
1. Info about building Drill for different Hive versions is a workaround, we shouldn't recommend doing it for users if other alternatives available, so I don't think that we should document it. Also, when all supported profiles are moved to this version of Hive, code will be cleaned up to remove hacks which allowed using older versions.
2. I don't see a way to achieve this. The issue here is in the interaction between Hive Jars with different versions. I assume besides API changes, also thrift format updates.
3. I don't think so. For updating versions to 2.3.2, code was changed, so I think it still incompatible.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] cgivre commented on issue #2038: DRILL-6604: Upgrade Drill
Hive client to Hive3.1 version
Posted by GitBox <gi...@apache.org>.
cgivre commented on issue #2038: DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#issuecomment-603223087
@vvysotskyi
Thanks for submitting this. A few questions:
1. Can we please include the bit about building Drill for different Hive versions in the documentation?
2. Is there any way to write this such that the users will not have to build a dedicated version of Drill for each version of Hive or are the Hive APIs so different that it is not practical?
3. If we are requiring that Drill is built specially for a given version of Hive, is it possible to support older versions than `2.3.2`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] vvysotskyi commented on a change in pull request #2038:
DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2038: DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#discussion_r399163163
##########
File path: contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/writers/primitive/HiveTimestampWriter.java
##########
@@ -18,21 +18,23 @@
package org.apache.drill.exec.store.hive.writers.primitive;
import org.apache.drill.exec.vector.complex.writer.TimeStampWriter;
-import org.apache.hadoop.hive.serde2.objectinspector.primitive.TimestampObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
-public class HiveTimestampWriter extends AbstractSingleValueWriter<TimestampObjectInspector, TimeStampWriter> {
+public class HiveTimestampWriter extends AbstractSingleValueWriter<PrimitiveObjectInspector, TimeStampWriter> {
- public HiveTimestampWriter(TimestampObjectInspector inspector, TimeStampWriter writer) {
+ public HiveTimestampWriter(PrimitiveObjectInspector inspector, TimeStampWriter writer) {
super(inspector, writer);
}
@Override
public void write(Object value) {
- final java.sql.Timestamp timestampValue = inspector.getPrimitiveJavaObject(value);
- final DateTime ts = new DateTime(timestampValue.getTime()).withZoneRetainFields(DateTimeZone.UTC);
- writer.writeTimeStamp(ts.getMillis());
+ String timestampString = PrimitiveObjectInspectorUtils.getString(value, inspector);
+ long milliseconds = new DateTime(java.sql.Timestamp.valueOf(timestampString).getTime())
Review comment:
Thanks, renamed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] vvysotskyi commented on a change in pull request #2038:
DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2038: DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#discussion_r399169874
##########
File path: contrib/storage-hive/core/src/test/java/org/apache/drill/exec/impersonation/hive/BaseTestHiveImpersonation.java
##########
@@ -97,11 +99,29 @@ protected static void prepHiveConfAndData() throws Exception {
}
protected static void startHiveMetaStore() throws Exception {
- final int port = MetaStoreUtils.findFreePort();
+ Class<?> metaStoreUtilsClass;
+ Class<?> hadoopThriftAuthBridgeClass;
+ Class<?> confClass;
+ Object hadoopThriftAuthBridge;
+ // TODO: remove reflection stuff when all supported profiles will be switched to Hive 3+ version
Review comment:
Currently, we support only a single non-default profile, but it hasn't released Hive 3+ versions, so I think that it would be better to postpone the creation of the Jira to the time when it is released (if it will be released), so we will know the exact version.
But I have created another Jira in Hive - HIVE-23088 to be able to remove workaround with logger class after this issue is fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] cgivre commented on issue #2038: DRILL-6604: Upgrade Drill
Hive client to Hive3.1 version
Posted by GitBox <gi...@apache.org>.
cgivre commented on issue #2038: DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#issuecomment-603219947
@vvysotskyi
Thank you for submitting this! I am wondering whether we test for backwards compatibility with older versions of Hive as we upgrade? What is the earliest version we currently support?
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] vvysotskyi commented on a change in pull request #2038:
DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2038: DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#discussion_r399169874
##########
File path: contrib/storage-hive/core/src/test/java/org/apache/drill/exec/impersonation/hive/BaseTestHiveImpersonation.java
##########
@@ -97,11 +99,29 @@ protected static void prepHiveConfAndData() throws Exception {
}
protected static void startHiveMetaStore() throws Exception {
- final int port = MetaStoreUtils.findFreePort();
+ Class<?> metaStoreUtilsClass;
+ Class<?> hadoopThriftAuthBridgeClass;
+ Class<?> confClass;
+ Object hadoopThriftAuthBridge;
+ // TODO: remove reflection stuff when all supported profiles will be switched to Hive 3+ version
Review comment:
Currently, we support only a single non-default profile, but it hasn't released Hive 3+ versions, so I think that it would be better to postpone the creation of the Jira to the time when it is released (if it will be released), so we will know the exact version.
But I have created another Jira in Hive - HIVE-23088 to fix and remove workaround with logger class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] ihuzenko commented on a change in pull request #2038:
DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
Posted by GitBox <gi...@apache.org>.
ihuzenko commented on a change in pull request #2038: DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#discussion_r399110724
##########
File path: contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/writers/primitive/HiveTimestampWriter.java
##########
@@ -18,21 +18,23 @@
package org.apache.drill.exec.store.hive.writers.primitive;
import org.apache.drill.exec.vector.complex.writer.TimeStampWriter;
-import org.apache.hadoop.hive.serde2.objectinspector.primitive.TimestampObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
-public class HiveTimestampWriter extends AbstractSingleValueWriter<TimestampObjectInspector, TimeStampWriter> {
+public class HiveTimestampWriter extends AbstractSingleValueWriter<PrimitiveObjectInspector, TimeStampWriter> {
- public HiveTimestampWriter(TimestampObjectInspector inspector, TimeStampWriter writer) {
+ public HiveTimestampWriter(PrimitiveObjectInspector inspector, TimeStampWriter writer) {
super(inspector, writer);
}
@Override
public void write(Object value) {
- final java.sql.Timestamp timestampValue = inspector.getPrimitiveJavaObject(value);
- final DateTime ts = new DateTime(timestampValue.getTime()).withZoneRetainFields(DateTimeZone.UTC);
- writer.writeTimeStamp(ts.getMillis());
+ String timestampString = PrimitiveObjectInspectorUtils.getString(value, inspector);
+ long milliseconds = new DateTime(java.sql.Timestamp.valueOf(timestampString).getTime())
Review comment:
```suggestion
long timestampMillis = new DateTime(java.sql.Timestamp.valueOf(timestampString).getTime())
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] asfgit closed pull request #2038: DRILL-6604: Upgrade Drill
Hive client to Hive3.1 version
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #2038: DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] cgivre commented on issue #2038: DRILL-6604: Upgrade Drill
Hive client to Hive3.1 version
Posted by GitBox <gi...@apache.org>.
cgivre commented on issue #2038: DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#issuecomment-603251228
@vvysotskyi
Thanks for the response. The reason I ask is that for many enterprises, they are at the mercy of IT teams and in many cases they are forced to use older versions of tools like Hive. Often, the upgrade schedules are years behind the most current version. (I'm speaking from experience here..;-))
Obviously we should do our best to support the most current version of common platforms like Hive, however, I think you would be surprised at how many large enterprises still use very old versions of these tools.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] vvysotskyi commented on a change in pull request #2038:
DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on a change in pull request #2038: DRILL-6604: Upgrade Drill Hive client to Hive3.1 version
URL: https://github.com/apache/drill/pull/2038#discussion_r399163018
##########
File path: contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/writers/primitive/HiveDateWriter.java
##########
@@ -18,21 +18,23 @@
package org.apache.drill.exec.store.hive.writers.primitive;
import org.apache.drill.exec.vector.complex.writer.DateWriter;
-import org.apache.hadoop.hive.serde2.objectinspector.primitive.DateObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
import org.joda.time.DateTime;
import org.joda.time.DateTimeZone;
-public class HiveDateWriter extends AbstractSingleValueWriter<DateObjectInspector, DateWriter> {
+public class HiveDateWriter extends AbstractSingleValueWriter<PrimitiveObjectInspector, DateWriter> {
- public HiveDateWriter(DateObjectInspector inspector, DateWriter writer) {
+ public HiveDateWriter(PrimitiveObjectInspector inspector, DateWriter writer) {
super(inspector, writer);
}
@Override
public void write(Object value) {
- final java.sql.Date dateValue = inspector.getPrimitiveJavaObject(value);
- final DateTime date = new DateTime(dateValue.getTime()).withZoneRetainFields(DateTimeZone.UTC);
- writer.writeDate(date.getMillis());
+ String dateString = PrimitiveObjectInspectorUtils.getString(value, inspector);
+ long time = new DateTime(java.sql.Date.valueOf(dateString).getTime())
Review comment:
Thanks, fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services