You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Mohit Sabharwal <mo...@cloudera.com> on 2014/12/15 23:33:40 UTC
Review Request 29061: HIVE-9109 : Add support for Java 8 specific
q-test out files
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29061/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-9109
https://issues.apache.org/jira/browse/HIVE-9109
Repository: hive-git
Description
-------
HIVE-9109 : Add support for Java 8 specific q-test out files
Hash function differences between Java 7 and Java 8 lead to result order differences. While we have been able to fix a good number by converting hash maps to insert order hash maps, there are several cases where doing so is either not possible (because changes originate in external APIs) or change leads to even more out file differences.
For example:
(1) In TestCliDriver.testCliDriver_varchar_udf1, for the following query:
select
str_to_map('a:1,b:2,c:3',',',':'),
str_to_map(cast('a:1,b:2,c:3' as varchar(20)),',',':')
from varchar_udf_1 limit 1;”)
the StandardMapObjectInspector used in LazySimpleSerDe to serialize the final output uses a HashMap. Changing it to LinkedHashMap will lead to several other q-test output differences.
(2) In TestCliDriver.testCliDriver_parquet_map_null, data with map column is read from an Avro table into a Parquet table. Avro API, specifically GenericData.Record uses HashMap and returns data in different order.
This patch adds supports to specify a hint called JAVA_VERSION_SPECIFIC_OUTPUT which may be added to a q-test, only if different outputs are expected for Java 7 and Java 8.
Under Java 7, test output file has (original) ".out" extension.
Under Java 8, test output file has ".java8.out" extension.
If hint is not added, we continue to generate a single ".out" file for the test.
Diffs
-----
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java e06d6828aa8924de2b56c8a63cc0955c5bd514d2
ql/src/test/queries/clientpositive/varchar_udf1.q 0a3012b5cd6d3e0cf065e51e7a680af1f0db859d
ql/src/test/results/clientpositive/varchar_udf1.q.java8.out PRE-CREATION
ql/src/test/results/clientpositive/varchar_udf1.q.out 842bd38cb5070994df3a264cc691372384433ae3
Diff: https://reviews.apache.org/r/29061/diff/
Testing
-------
Tested using varchar_udf1.q.
Out file changes for this test are included in the patch
Thanks,
Mohit Sabharwal
Re: Review Request 29061: HIVE-9109 : Add support for Java 8 specific
q-test out files
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29061/#review65155
-----------------------------------------------------------
Hi,
This looks great!! The only thing I found strange is that I don't see a simple path to using -- JAVA_VERSION_SPECIFIC_OUTPUT with java 9. Perhaps we should add an argument. This will make -- JAVA_VERSION_SPECIFIC_OUTPUT work with all jvms. Today -- JAVA_VERSION_SPECIFIC_OUTPUT actually implies java 8, which doesn't make sense to me. e.g.
-- JAVA_VERSION_SPECIFIC_OUTPUT 1.8
- Brock Noland
On Dec. 15, 2014, 10:33 p.m., Mohit Sabharwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29061/
> -----------------------------------------------------------
>
> (Updated Dec. 15, 2014, 10:33 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-9109
> https://issues.apache.org/jira/browse/HIVE-9109
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-9109 : Add support for Java 8 specific q-test out files
>
> Hash function differences between Java 7 and Java 8 lead to result order differences. While we have been able to fix a good number by converting hash maps to insert order hash maps, there are several cases where doing so is either not possible (because changes originate in external APIs) or change leads to even more out file differences.
>
> For example:
> (1) In TestCliDriver.testCliDriver_varchar_udf1, for the following query:
> select
> str_to_map('a:1,b:2,c:3',',',':'),
> str_to_map(cast('a:1,b:2,c:3' as varchar(20)),',',':')
> from varchar_udf_1 limit 1;”)
> the StandardMapObjectInspector used in LazySimpleSerDe to serialize the final output uses a HashMap. Changing it to LinkedHashMap will lead to several other q-test output differences.
>
> (2) In TestCliDriver.testCliDriver_parquet_map_null, data with map column is read from an Avro table into a Parquet table. Avro API, specifically GenericData.Record uses HashMap and returns data in different order.
>
> This patch adds supports to specify a hint called JAVA_VERSION_SPECIFIC_OUTPUT which may be added to a q-test, only if different outputs are expected for Java 7 and Java 8.
>
> Under Java 7, test output file has (original) ".out" extension.
>
> Under Java 8, test output file has ".java8.out" extension.
>
> If hint is not added, we continue to generate a single ".out" file for the test.
>
>
> Diffs
> -----
>
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java e06d6828aa8924de2b56c8a63cc0955c5bd514d2
> ql/src/test/queries/clientpositive/varchar_udf1.q 0a3012b5cd6d3e0cf065e51e7a680af1f0db859d
> ql/src/test/results/clientpositive/varchar_udf1.q.java8.out PRE-CREATION
> ql/src/test/results/clientpositive/varchar_udf1.q.out 842bd38cb5070994df3a264cc691372384433ae3
>
> Diff: https://reviews.apache.org/r/29061/diff/
>
>
> Testing
> -------
>
> Tested using varchar_udf1.q.
> Out file changes for this test are included in the patch
>
>
> Thanks,
>
> Mohit Sabharwal
>
>
Re: Review Request 29061: HIVE-9109 : Add support for Java 8 specific
q-test out files
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29061/#review65208
-----------------------------------------------------------
Ship it!
Ship It!
- Brock Noland
On Dec. 16, 2014, 2:16 a.m., Mohit Sabharwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29061/
> -----------------------------------------------------------
>
> (Updated Dec. 16, 2014, 2:16 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-9109
> https://issues.apache.org/jira/browse/HIVE-9109
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-9109 : Add support for Java 8 specific q-test out files
>
> Hash function differences between Java 7 and Java 8 lead to result order differences. While we have been able to fix a good number by converting hash maps to insert order hash maps, there are several cases where doing so is either not possible (because changes originate in external APIs) or change leads to even more out file differences.
>
> For example:
> (1) In TestCliDriver.testCliDriver_varchar_udf1, for the following query:
> select
> str_to_map('a:1,b:2,c:3',',',':'),
> str_to_map(cast('a:1,b:2,c:3' as varchar(20)),',',':')
> from varchar_udf_1 limit 1;”)
> the StandardMapObjectInspector used in LazySimpleSerDe to serialize the final output uses a HashMap. Changing it to LinkedHashMap will lead to several other q-test output differences.
>
> (2) In TestCliDriver.testCliDriver_parquet_map_null, data with map column is read from an Avro table into a Parquet table. Avro API, specifically GenericData.Record uses HashMap and returns data in different order.
>
> This patch adds supports to specify a hint called JAVA_VERSION_SPECIFIC_OUTPUT which may be added to a q-test, only if different outputs are expected for different Java versions.
>
> Under Java 7, test output file has ".java1.7.out" extension.
>
> Under Java 8, test output file has ".java1.8.out" extension.
>
> If hint is not added, we continue to generate a ".out" file for the test.
>
>
> Diffs
> -----
>
> itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java e06d6828aa8924de2b56c8a63cc0955c5bd514d2
> ql/src/test/queries/clientpositive/varchar_udf1.q 0a3012b5cd6d3e0cf065e51e7a680af1f0db859d
> ql/src/test/results/clientpositive/varchar_udf1.q.java1.7.out PRE-CREATION
> ql/src/test/results/clientpositive/varchar_udf1.q.java1.8.out PRE-CREATION
> ql/src/test/results/clientpositive/varchar_udf1.q.out 842bd38cb5070994df3a264cc691372384433ae3
>
> Diff: https://reviews.apache.org/r/29061/diff/
>
>
> Testing
> -------
>
> Tested using varchar_udf1.q.
> Out file changes for this test are included in the patch
>
>
> Thanks,
>
> Mohit Sabharwal
>
>
Re: Review Request 29061: HIVE-9109 : Add support for Java 8 specific
q-test out files
Posted by Mohit Sabharwal <mo...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29061/
-----------------------------------------------------------
(Updated Dec. 16, 2014, 2:16 a.m.)
Review request for hive.
Changes
-------
Fixed typo.
Bugs: HIVE-9109
https://issues.apache.org/jira/browse/HIVE-9109
Repository: hive-git
Description (updated)
-------
HIVE-9109 : Add support for Java 8 specific q-test out files
Hash function differences between Java 7 and Java 8 lead to result order differences. While we have been able to fix a good number by converting hash maps to insert order hash maps, there are several cases where doing so is either not possible (because changes originate in external APIs) or change leads to even more out file differences.
For example:
(1) In TestCliDriver.testCliDriver_varchar_udf1, for the following query:
select
str_to_map('a:1,b:2,c:3',',',':'),
str_to_map(cast('a:1,b:2,c:3' as varchar(20)),',',':')
from varchar_udf_1 limit 1;”)
the StandardMapObjectInspector used in LazySimpleSerDe to serialize the final output uses a HashMap. Changing it to LinkedHashMap will lead to several other q-test output differences.
(2) In TestCliDriver.testCliDriver_parquet_map_null, data with map column is read from an Avro table into a Parquet table. Avro API, specifically GenericData.Record uses HashMap and returns data in different order.
This patch adds supports to specify a hint called JAVA_VERSION_SPECIFIC_OUTPUT which may be added to a q-test, only if different outputs are expected for different Java versions.
Under Java 7, test output file has ".java1.7.out" extension.
Under Java 8, test output file has ".java1.8.out" extension.
If hint is not added, we continue to generate a ".out" file for the test.
Diffs
-----
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java e06d6828aa8924de2b56c8a63cc0955c5bd514d2
ql/src/test/queries/clientpositive/varchar_udf1.q 0a3012b5cd6d3e0cf065e51e7a680af1f0db859d
ql/src/test/results/clientpositive/varchar_udf1.q.java1.7.out PRE-CREATION
ql/src/test/results/clientpositive/varchar_udf1.q.java1.8.out PRE-CREATION
ql/src/test/results/clientpositive/varchar_udf1.q.out 842bd38cb5070994df3a264cc691372384433ae3
Diff: https://reviews.apache.org/r/29061/diff/
Testing
-------
Tested using varchar_udf1.q.
Out file changes for this test are included in the patch
Thanks,
Mohit Sabharwal
Re: Review Request 29061: HIVE-9109 : Add support for Java 8 specific
q-test out files
Posted by Mohit Sabharwal <mo...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29061/
-----------------------------------------------------------
(Updated Dec. 16, 2014, 2:09 a.m.)
Review request for hive.
Changes
-------
Inc feedback
Bugs: HIVE-9109
https://issues.apache.org/jira/browse/HIVE-9109
Repository: hive-git
Description (updated)
-------
HIVE-9109 : Add support for Java 8 specific q-test out files
Hash function differences between Java 7 and Java 8 lead to result order differences. While we have been able to fix a good number by converting hash maps to insert order hash maps, there are several cases where doing so is either not possible (because changes originate in external APIs) or change leads to even more out file differences.
For example:
(1) In TestCliDriver.testCliDriver_varchar_udf1, for the following query:
select
str_to_map('a:1,b:2,c:3',',',':'),
str_to_map(cast('a:1,b:2,c:3' as varchar(20)),',',':')
from varchar_udf_1 limit 1;”)
the StandardMapObjectInspector used in LazySimpleSerDe to serialize the final output uses a HashMap. Changing it to LinkedHashMap will lead to several other q-test output differences.
(2) In TestCliDriver.testCliDriver_parquet_map_null, data with map column is read from an Avro table into a Parquet table. Avro API, specifically GenericData.Record uses HashMap and returns data in different order.
This patch adds supports to specify a hint called JAVA_VERSION_SPECIFIC_OUTPUT which may be added to a q-test, only if different outputs are expected for different Java versions.
Under Java 7, test output file has ".java1.7.out" extension.
Under Java 8, test output file has ".java1.7.out" extension.
If hint is not added, we continue to generate a ".out" file for the test.
Diffs (updated)
-----
itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java e06d6828aa8924de2b56c8a63cc0955c5bd514d2
ql/src/test/queries/clientpositive/varchar_udf1.q 0a3012b5cd6d3e0cf065e51e7a680af1f0db859d
ql/src/test/results/clientpositive/varchar_udf1.q.java1.7.out PRE-CREATION
ql/src/test/results/clientpositive/varchar_udf1.q.java1.8.out PRE-CREATION
ql/src/test/results/clientpositive/varchar_udf1.q.out 842bd38cb5070994df3a264cc691372384433ae3
Diff: https://reviews.apache.org/r/29061/diff/
Testing
-------
Tested using varchar_udf1.q.
Out file changes for this test are included in the patch
Thanks,
Mohit Sabharwal