You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Deepak Jaiswal <dj...@hortonworks.com> on 2018/01/23 00:54:21 UTC
Review Request 65276: HIVE-18516
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65276/
-----------------------------------------------------------
Review request for hive, Eugene Koifman and Jason Dere.
Bugs: HIVE-18516
https://issues.apache.org/jira/browse/HIVE-18516
Repository: hive-git
Description
-------
load data should rename files consistent with insert statements for ACID Tables.
Includes test change for a missed test.
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 23983d85b3
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java c6a4a8926b
ql/src/test/queries/clientpositive/load_data_acid_rename.q PRE-CREATION
ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496
ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5
ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out PRE-CREATION
ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1
ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913
Diff: https://reviews.apache.org/r/65276/diff/1/
Testing
-------
Thanks,
Deepak Jaiswal
Re: Review Request 65276: HIVE-18516
Posted by Deepak Jaiswal <dj...@hortonworks.com>.
> On Jan. 30, 2018, 3:09 a.m., Jason Dere wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 3821 (patched)
> > <https://reviews.apache.org/r/65276/diff/1-2/?file=1944274#file1944274line3821>
> >
> > So only non-bucketed ACID tables get the file rename to 000000_0 etc? What happens with bucketed ACID tables and the file name?
Yes. As mentioned the change summary, the logic for bucketed ACID tables is part of the larger work I am doing for bucketed tables in general.
> On Jan. 30, 2018, 3:09 a.m., Jason Dere wrote:
> > ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out
> > Lines 29 (patched)
> > <https://reviews.apache.org/r/65276/diff/1-2/?file=1944280#file1944280line29>
> >
> > Is there a smaller table/results you can test with?
The idea was to actually see if we get results or not.
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65276/#review196499
-----------------------------------------------------------
On Jan. 29, 2018, 11:10 p.m., Deepak Jaiswal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65276/
> -----------------------------------------------------------
>
> (Updated Jan. 29, 2018, 11:10 p.m.)
>
>
> Review request for hive, Eugene Koifman and Jason Dere.
>
>
> Bugs: HIVE-18516
> https://issues.apache.org/jira/browse/HIVE-18516
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> load data should rename files consistent with insert statements for ACID Tables.
> Includes test change for a missed test.
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties d86ff58840
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 23983d85b3
> ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java c6a4a8926b
> ql/src/test/queries/clientpositive/load_data_acid_rename.q PRE-CREATION
> ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496
> ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5
> ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out PRE-CREATION
> ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1
> ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913
>
>
> Diff: https://reviews.apache.org/r/65276/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Deepak Jaiswal
>
>
Re: Review Request 65276: HIVE-18516
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65276/#review196499
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 3821 (patched)
<https://reviews.apache.org/r/65276/#comment276178>
So only non-bucketed ACID tables get the file rename to 000000_0 etc? What happens with bucketed ACID tables and the file name?
ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out
Lines 29 (patched)
<https://reviews.apache.org/r/65276/#comment276177>
Is there a smaller table/results you can test with?
- Jason Dere
On Jan. 29, 2018, 11:10 p.m., Deepak Jaiswal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65276/
> -----------------------------------------------------------
>
> (Updated Jan. 29, 2018, 11:10 p.m.)
>
>
> Review request for hive, Eugene Koifman and Jason Dere.
>
>
> Bugs: HIVE-18516
> https://issues.apache.org/jira/browse/HIVE-18516
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> load data should rename files consistent with insert statements for ACID Tables.
> Includes test change for a missed test.
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties d86ff58840
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 23983d85b3
> ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java c6a4a8926b
> ql/src/test/queries/clientpositive/load_data_acid_rename.q PRE-CREATION
> ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496
> ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5
> ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out PRE-CREATION
> ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1
> ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913
>
>
> Diff: https://reviews.apache.org/r/65276/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Deepak Jaiswal
>
>
Re: Review Request 65276: HIVE-18516
Posted by Deepak Jaiswal <dj...@hortonworks.com>.
> On Feb. 1, 2018, 2:24 a.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 3784 (patched)
> > <https://reviews.apache.org/r/65276/diff/5/?file=1951001#file1951001line3794>
> >
> > how is isFullAcidTable different from isAcid ?
> > There should be some java doc if they are indeed different
isAcid is used only when it is ACID and IUD op only. It is false when it is load data which is the case. I will update the variable name to something better and add comments.
> On Feb. 1, 2018, 2:24 a.m., Eugene Koifman wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java
> > Lines 460 (patched)
> > <https://reviews.apache.org/r/65276/diff/5/?file=1951003#file1951003line460>
> >
> > is this left in here intentionally? checkExpected() logs this to the log file laready
Thanks for pointing this out. I will remove it.
> On Feb. 1, 2018, 2:24 a.m., Eugene Koifman wrote:
> > ql/src/test/queries/clientpositive/load_data_acid_rename.q
> > Lines 1 (patched)
> > <https://reviews.apache.org/r/65276/diff/5/?file=1951006#file1951006line1>
> >
> > why is this needed here? this is deperecated in HiveConf
Thanks. I will remove it.
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65276/#review196609
-----------------------------------------------------------
On Jan. 31, 2018, 10 p.m., Deepak Jaiswal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65276/
> -----------------------------------------------------------
>
> (Updated Jan. 31, 2018, 10 p.m.)
>
>
> Review request for hive, Eugene Koifman and Jason Dere.
>
>
> Bugs: HIVE-18516
> https://issues.apache.org/jira/browse/HIVE-18516
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> load data should rename files consistent with insert statements for ACID Tables.
> Includes test change for a missed test.
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties d86ff58840
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 63bcedc000
> ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56
> ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java a9cba456ef
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java c6a4a8926b
> ql/src/test/queries/clientnegative/load_data_into_acid.q 2ac5b561ae
> ql/src/test/queries/clientpositive/load_data_acid_rename.q PRE-CREATION
> ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496
> ql/src/test/results/clientnegative/load_data_into_acid.q.out 46b5cdd2c8
> ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5
> ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out PRE-CREATION
> ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1
> ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913
>
>
> Diff: https://reviews.apache.org/r/65276/diff/5/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Deepak Jaiswal
>
>
Re: Review Request 65276: HIVE-18516
Posted by Eugene Koifman <ek...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65276/#review196609
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 3784 (patched)
<https://reviews.apache.org/r/65276/#comment276302>
how is isFullAcidTable different from isAcid ?
There should be some java doc if they are indeed different
ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java
Lines 460 (patched)
<https://reviews.apache.org/r/65276/#comment276303>
is this left in here intentionally? checkExpected() logs this to the log file laready
ql/src/test/queries/clientpositive/load_data_acid_rename.q
Lines 1 (patched)
<https://reviews.apache.org/r/65276/#comment276299>
why is this needed here? this is deperecated in HiveConf
- Eugene Koifman
On Jan. 31, 2018, 10 p.m., Deepak Jaiswal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65276/
> -----------------------------------------------------------
>
> (Updated Jan. 31, 2018, 10 p.m.)
>
>
> Review request for hive, Eugene Koifman and Jason Dere.
>
>
> Bugs: HIVE-18516
> https://issues.apache.org/jira/browse/HIVE-18516
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> load data should rename files consistent with insert statements for ACID Tables.
> Includes test change for a missed test.
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties d86ff58840
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 63bcedc000
> ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56
> ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java a9cba456ef
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java c6a4a8926b
> ql/src/test/queries/clientnegative/load_data_into_acid.q 2ac5b561ae
> ql/src/test/queries/clientpositive/load_data_acid_rename.q PRE-CREATION
> ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496
> ql/src/test/results/clientnegative/load_data_into_acid.q.out 46b5cdd2c8
> ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5
> ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out PRE-CREATION
> ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1
> ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913
>
>
> Diff: https://reviews.apache.org/r/65276/diff/5/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Deepak Jaiswal
>
>
Re: Review Request 65276: HIVE-18516
Posted by Deepak Jaiswal <dj...@hortonworks.com>.
> On Feb. 2, 2018, 8 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Line 3470 (original), 3484 (patched)
> > <https://reviews.apache.org/r/65276/diff/7/?file=1951308#file1951308line3494>
> >
> > Also, shouldn't it be an error if one of the input files has .gz? This would normally mean someone manyally compressed the file which should not be the case for ORC. Or is this checked elsewhere?
> >
> > Silently ignoring this most likely will mask an error until someone tries to read these files
This is checked elsewhere much earlier.
> On Feb. 2, 2018, 8 p.m., Eugene Koifman wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
> > Lines 3485 (patched)
> > <https://reviews.apache.org/r/65276/diff/7/?file=1951308#file1951308line3495>
> >
> > The logic here seems to be changed.
> > if taskid == -1, then "name" is computed but ignored which wasn't the case previously
Yes. That means it is not ACID table so we keep the existing behavior which is to use the fullname.
The name is only used when the file already exists which is handled right below.
- Deepak
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65276/#review196725
-----------------------------------------------------------
On Feb. 1, 2018, 7:10 p.m., Deepak Jaiswal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65276/
> -----------------------------------------------------------
>
> (Updated Feb. 1, 2018, 7:10 p.m.)
>
>
> Review request for hive, Eugene Koifman and Jason Dere.
>
>
> Bugs: HIVE-18516
> https://issues.apache.org/jira/browse/HIVE-18516
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> load data should rename files consistent with insert statements for ACID Tables.
> Includes test change for a missed test.
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties d86ff58840
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 3b97dac8ca
> ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56
> ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java a9cba456ef
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java c6a4a8926b
> ql/src/test/queries/clientnegative/load_data_into_acid.q 2ac5b561ae
> ql/src/test/queries/clientpositive/load_data_acid_rename.q PRE-CREATION
> ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496
> ql/src/test/results/clientnegative/load_data_into_acid.q.out 46b5cdd2c8
> ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5
> ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out PRE-CREATION
> ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1
> ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913
>
>
> Diff: https://reviews.apache.org/r/65276/diff/7/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Deepak Jaiswal
>
>
Re: Review Request 65276: HIVE-18516
Posted by Eugene Koifman <ek...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65276/#review196725
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Line 3470 (original), 3484 (patched)
<https://reviews.apache.org/r/65276/#comment276534>
Also, shouldn't it be an error if one of the input files has .gz? This would normally mean someone manyally compressed the file which should not be the case for ORC. Or is this checked elsewhere?
Silently ignoring this most likely will mask an error until someone tries to read these files
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 3485 (patched)
<https://reviews.apache.org/r/65276/#comment276532>
The logic here seems to be changed.
if taskid == -1, then "name" is computed but ignored which wasn't the case previously
- Eugene Koifman
On Feb. 1, 2018, 7:10 p.m., Deepak Jaiswal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65276/
> -----------------------------------------------------------
>
> (Updated Feb. 1, 2018, 7:10 p.m.)
>
>
> Review request for hive, Eugene Koifman and Jason Dere.
>
>
> Bugs: HIVE-18516
> https://issues.apache.org/jira/browse/HIVE-18516
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> load data should rename files consistent with insert statements for ACID Tables.
> Includes test change for a missed test.
>
>
> Diffs
> -----
>
> itests/src/test/resources/testconfiguration.properties d86ff58840
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 3b97dac8ca
> ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56
> ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java a9cba456ef
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java c6a4a8926b
> ql/src/test/queries/clientnegative/load_data_into_acid.q 2ac5b561ae
> ql/src/test/queries/clientpositive/load_data_acid_rename.q PRE-CREATION
> ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496
> ql/src/test/results/clientnegative/load_data_into_acid.q.out 46b5cdd2c8
> ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5
> ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out PRE-CREATION
> ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1
> ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913
>
>
> Diff: https://reviews.apache.org/r/65276/diff/7/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Deepak Jaiswal
>
>
Re: Review Request 65276: HIVE-18516
Posted by Deepak Jaiswal <dj...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65276/
-----------------------------------------------------------
(Updated Feb. 1, 2018, 7:10 p.m.)
Review request for hive, Eugene Koifman and Jason Dere.
Changes
-------
Updated with merge conflicts due to another commit in Hive.java
Bugs: HIVE-18516
https://issues.apache.org/jira/browse/HIVE-18516
Repository: hive-git
Description
-------
load data should rename files consistent with insert statements for ACID Tables.
Includes test change for a missed test.
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties d86ff58840
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 3b97dac8ca
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56
ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java a9cba456ef
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java c6a4a8926b
ql/src/test/queries/clientnegative/load_data_into_acid.q 2ac5b561ae
ql/src/test/queries/clientpositive/load_data_acid_rename.q PRE-CREATION
ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496
ql/src/test/results/clientnegative/load_data_into_acid.q.out 46b5cdd2c8
ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5
ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out PRE-CREATION
ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1
ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913
Diff: https://reviews.apache.org/r/65276/diff/7/
Changes: https://reviews.apache.org/r/65276/diff/6-7/
Testing
-------
Thanks,
Deepak Jaiswal
Re: Review Request 65276: HIVE-18516
Posted by Deepak Jaiswal <dj...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65276/
-----------------------------------------------------------
(Updated Feb. 1, 2018, 2:59 a.m.)
Review request for hive, Eugene Koifman and Jason Dere.
Changes
-------
Implemented Eugene's review comments.
Bugs: HIVE-18516
https://issues.apache.org/jira/browse/HIVE-18516
Repository: hive-git
Description
-------
load data should rename files consistent with insert statements for ACID Tables.
Includes test change for a missed test.
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties d86ff58840
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java c8299e2764
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56
ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java a9cba456ef
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java c6a4a8926b
ql/src/test/queries/clientnegative/load_data_into_acid.q 2ac5b561ae
ql/src/test/queries/clientpositive/load_data_acid_rename.q PRE-CREATION
ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496
ql/src/test/results/clientnegative/load_data_into_acid.q.out 46b5cdd2c8
ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5
ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out PRE-CREATION
ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1
ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913
Diff: https://reviews.apache.org/r/65276/diff/6/
Changes: https://reviews.apache.org/r/65276/diff/5-6/
Testing
-------
Thanks,
Deepak Jaiswal
Re: Review Request 65276: HIVE-18516
Posted by Deepak Jaiswal <dj...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65276/
-----------------------------------------------------------
(Updated Jan. 31, 2018, 10 p.m.)
Review request for hive, Eugene Koifman and Jason Dere.
Changes
-------
Before loading, sort the files.
This results in deterministic order of files.
Bugs: HIVE-18516
https://issues.apache.org/jira/browse/HIVE-18516
Repository: hive-git
Description
-------
load data should rename files consistent with insert statements for ACID Tables.
Includes test change for a missed test.
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties d86ff58840
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 63bcedc000
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56
ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java a9cba456ef
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java c6a4a8926b
ql/src/test/queries/clientnegative/load_data_into_acid.q 2ac5b561ae
ql/src/test/queries/clientpositive/load_data_acid_rename.q PRE-CREATION
ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496
ql/src/test/results/clientnegative/load_data_into_acid.q.out 46b5cdd2c8
ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5
ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out PRE-CREATION
ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1
ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913
Diff: https://reviews.apache.org/r/65276/diff/5/
Changes: https://reviews.apache.org/r/65276/diff/4-5/
Testing
-------
Thanks,
Deepak Jaiswal
Re: Review Request 65276: HIVE-18516
Posted by Deepak Jaiswal <dj...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65276/
-----------------------------------------------------------
(Updated Jan. 31, 2018, 7:23 p.m.)
Review request for hive, Eugene Koifman and Jason Dere.
Changes
-------
Updated expected results in TestTxnLoadData based on ptest runs.
Removed load_data_into_acid test as it is not needed anymore.
Added load_data_acid_rename to minillaplocal.only list.
Bugs: HIVE-18516
https://issues.apache.org/jira/browse/HIVE-18516
Repository: hive-git
Description
-------
load data should rename files consistent with insert statements for ACID Tables.
Includes test change for a missed test.
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties d86ff58840
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 63bcedc000
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56
ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java a9cba456ef
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java c6a4a8926b
ql/src/test/queries/clientnegative/load_data_into_acid.q 2ac5b561ae
ql/src/test/queries/clientpositive/load_data_acid_rename.q PRE-CREATION
ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496
ql/src/test/results/clientnegative/load_data_into_acid.q.out 46b5cdd2c8
ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5
ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out PRE-CREATION
ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1
ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913
Diff: https://reviews.apache.org/r/65276/diff/4/
Changes: https://reviews.apache.org/r/65276/diff/3-4/
Testing
-------
Thanks,
Deepak Jaiswal
Re: Review Request 65276: HIVE-18516
Posted by Deepak Jaiswal <dj...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65276/
-----------------------------------------------------------
(Updated Jan. 30, 2018, 6:49 p.m.)
Review request for hive, Eugene Koifman and Jason Dere.
Changes
-------
Fixed the build error.
Also, removed the select query as the goal is to see if the file is loaded in the test.
Bugs: HIVE-18516
https://issues.apache.org/jira/browse/HIVE-18516
Repository: hive-git
Description
-------
load data should rename files consistent with insert statements for ACID Tables.
Includes test change for a missed test.
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties d86ff58840
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 23983d85b3
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java c6a4a8926b
ql/src/test/queries/clientpositive/load_data_acid_rename.q PRE-CREATION
ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496
ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5
ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out PRE-CREATION
ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1
ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913
Diff: https://reviews.apache.org/r/65276/diff/3/
Changes: https://reviews.apache.org/r/65276/diff/2-3/
Testing
-------
Thanks,
Deepak Jaiswal
Re: Review Request 65276: HIVE-18516
Posted by Deepak Jaiswal <dj...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65276/
-----------------------------------------------------------
(Updated Jan. 29, 2018, 11:10 p.m.)
Review request for hive, Eugene Koifman and Jason Dere.
Changes
-------
Implemented review comments.
Since ACID works with ORC files which handle compression internally, the extensions are completely ignored.
The logic is updated only for ACID non-bucketed tables. The fix for ACID bucketed table will come with larger fix for bucketed tables in general.
Bugs: HIVE-18516
https://issues.apache.org/jira/browse/HIVE-18516
Repository: hive-git
Description
-------
load data should rename files consistent with insert statements for ACID Tables.
Includes test change for a missed test.
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties d86ff58840
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 23983d85b3
ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56
ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java c6a4a8926b
ql/src/test/queries/clientpositive/load_data_acid_rename.q PRE-CREATION
ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496
ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5
ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out PRE-CREATION
ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1
ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913
Diff: https://reviews.apache.org/r/65276/diff/2/
Changes: https://reviews.apache.org/r/65276/diff/1-2/
Testing
-------
Thanks,
Deepak Jaiswal
Re: Review Request 65276: HIVE-18516
Posted by Jason Dere <jd...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65276/#review196192
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Lines 3309 (patched)
<https://reviews.apache.org/r/65276/#comment275739>
Maybe add a comment here about the automatic file re-naming convention for acid tables - that the files are listed in order by name, and mapped to bucket 0, 1, etc based on the position of the file name in the list.
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Line 3456 (original), 3471 (patched)
<https://reviews.apache.org/r/65276/#comment275743>
In the acid case, it looks like the file extension is stripped off? Actually I'm not sure if gz/other compressed types will work correctly if the filename is not provided .. can you test that?
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
Line 3471 (original), 3486 (patched)
<https://reviews.apache.org/r/65276/#comment275744>
Actually the extension is added here, so that is an inconsistency between these two lines for acid tables.
- Jason Dere
On Jan. 23, 2018, 12:54 a.m., Deepak Jaiswal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/65276/
> -----------------------------------------------------------
>
> (Updated Jan. 23, 2018, 12:54 a.m.)
>
>
> Review request for hive, Eugene Koifman and Jason Dere.
>
>
> Bugs: HIVE-18516
> https://issues.apache.org/jira/browse/HIVE-18516
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> load data should rename files consistent with insert statements for ACID Tables.
> Includes test change for a missed test.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 23983d85b3
> ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 5868d4dd56
> ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java c6a4a8926b
> ql/src/test/queries/clientpositive/load_data_acid_rename.q PRE-CREATION
> ql/src/test/queries/clientpositive/smb_mapjoin_7.q 4a6afb0496
> ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out 7a6f8c53a5
> ql/src/test/results/clientpositive/llap/load_data_acid_rename.q.out PRE-CREATION
> ql/src/test/results/clientpositive/smb_mapjoin_7.q.out b71c5b87c1
> ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out ac49c02913
>
>
> Diff: https://reviews.apache.org/r/65276/diff/1/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Deepak Jaiswal
>
>