You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Barna Zsombor Klara <zs...@cloudera.com> on 2017/06/26 15:20:33 UTC
Review Request 60433: HIVE-16784: Missing lineage information when
hive.blobstore.optimizations.enabled is true
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60433/
-----------------------------------------------------------
Review request for hive, Marta Kuczora, Sergio Pena, and Sahil Takiar.
Repository: hive-git
Description
-------
HIVE-16784: Missing lineage information when hive.blobstore.optimizations.enabled is true
Diffs
-----
itests/hive-blobstore/src/test/results/clientpositive/insert_blobstore_to_blobstore.q.out 46518999358be9ec5191ad2c3b37f555efbec263
itests/hive-blobstore/src/test/results/clientpositive/insert_empty_into_blobstore.q.out 8e5e09627b3a0f5c66a01fdd096b4b17b0b10689
itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 663a57274840a84a5f04777540d09d9f9bcfb5da
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out 8052fd86eefaff1fe7d024d3f58f6896e7223dd9
itests/hive-blobstore/src/test/results/clientpositive/orc_format_nonpart.q.out f4705687ff4b7b29d9bd6b81dfcb4065364c2965
itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out 1ef98104c5a711ef4b6553ca06fccf9003b99361
itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out 9de4190072cc779b34e556a0e774ce7403324525
itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_nonpart.q.out 44a1f11fa2c7bd32c636b9bad4e4e35d79989dcf
itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out defca3bb78d4d97a327c389650d256a2cd880899
itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out 5db9c7ef90c81524eae3eb6ec5346321e38b2a9e
itests/hive-blobstore/src/test/results/clientpositive/zero_rows_blobstore.q.out 590f947add9b34d3adb05a7e22b0eb7d011c9a0d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 88bf82999953346bc92dac17c07373e6360be232
ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java 0f9506376c283ec6e6ab659b20a5718807268969
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java 68ccda9ef61744e369638ceec5c083c77524c2db
Diff: https://reviews.apache.org/r/60433/diff/1/
Testing
-------
Thanks,
Barna Zsombor Klara
Re: Review Request 60433: HIVE-16784: Missing lineage information when
hive.blobstore.optimizations.enabled is true
Posted by Barna Zsombor Klara <zs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60433/#review181440
-----------------------------------------------------------
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
Lines 57 (patched)
<https://reviews.apache.org/r/60433/#comment257014>
This is needed to prevent an NPE in GenMapRedUtils#mergeMovePaths. If the Session is not initialized Session.get() will return null.
- Barna Zsombor Klara
On July 17, 2017, 2:08 p.m., Barna Zsombor Klara wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60433/
> -----------------------------------------------------------
>
> (Updated July 17, 2017, 2:08 p.m.)
>
>
> Review request for hive, Marta Kuczora, Sergio Pena, and Sahil Takiar.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-16784: Missing lineage information when hive.blobstore.optimizations.enabled is true
>
>
> Diffs
> -----
>
> itests/hive-blobstore/src/test/results/clientpositive/insert_blobstore_to_blobstore.q.out 46518999358be9ec5191ad2c3b37f555efbec263
> itests/hive-blobstore/src/test/results/clientpositive/insert_empty_into_blobstore.q.out 8e5e09627b3a0f5c66a01fdd096b4b17b0b10689
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 663a57274840a84a5f04777540d09d9f9bcfb5da
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out 8052fd86eefaff1fe7d024d3f58f6896e7223dd9
> itests/hive-blobstore/src/test/results/clientpositive/orc_format_nonpart.q.out f4705687ff4b7b29d9bd6b81dfcb4065364c2965
> itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out 1ef98104c5a711ef4b6553ca06fccf9003b99361
> itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out 9de4190072cc779b34e556a0e774ce7403324525
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_nonpart.q.out 44a1f11fa2c7bd32c636b9bad4e4e35d79989dcf
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out defca3bb78d4d97a327c389650d256a2cd880899
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out 5db9c7ef90c81524eae3eb6ec5346321e38b2a9e
> itests/hive-blobstore/src/test/results/clientpositive/zero_rows_blobstore.q.out 590f947add9b34d3adb05a7e22b0eb7d011c9a0d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 88bf82999953346bc92dac17c07373e6360be232
> ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java 0f9506376c283ec6e6ab659b20a5718807268969
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java 68ccda9ef61744e369638ceec5c083c77524c2db
>
>
> Diff: https://reviews.apache.org/r/60433/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Barna Zsombor Klara
>
>
Re: Review Request 60433: HIVE-16784: Missing lineage information when
hive.blobstore.optimizations.enabled is true
Posted by Barna Zsombor Klara <zs...@cloudera.com>.
> On July 25, 2017, 9:29 p.m., Sahil Takiar wrote:
> > ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
> > Lines 57 (patched)
> > <https://reviews.apache.org/r/60433/diff/2/?file=1777640#file1777640line57>
> >
> > Whats this change for?
This is needed to prevent an NPE in GenMapRedUtils#mergeMovePaths. If the Session is not initialized Session.get() will return null.
- Barna Zsombor
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60433/#review181386
-----------------------------------------------------------
On July 17, 2017, 2:08 p.m., Barna Zsombor Klara wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60433/
> -----------------------------------------------------------
>
> (Updated July 17, 2017, 2:08 p.m.)
>
>
> Review request for hive, Marta Kuczora, Sergio Pena, and Sahil Takiar.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-16784: Missing lineage information when hive.blobstore.optimizations.enabled is true
>
>
> Diffs
> -----
>
> itests/hive-blobstore/src/test/results/clientpositive/insert_blobstore_to_blobstore.q.out 46518999358be9ec5191ad2c3b37f555efbec263
> itests/hive-blobstore/src/test/results/clientpositive/insert_empty_into_blobstore.q.out 8e5e09627b3a0f5c66a01fdd096b4b17b0b10689
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 663a57274840a84a5f04777540d09d9f9bcfb5da
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out 8052fd86eefaff1fe7d024d3f58f6896e7223dd9
> itests/hive-blobstore/src/test/results/clientpositive/orc_format_nonpart.q.out f4705687ff4b7b29d9bd6b81dfcb4065364c2965
> itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out 1ef98104c5a711ef4b6553ca06fccf9003b99361
> itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out 9de4190072cc779b34e556a0e774ce7403324525
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_nonpart.q.out 44a1f11fa2c7bd32c636b9bad4e4e35d79989dcf
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out defca3bb78d4d97a327c389650d256a2cd880899
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out 5db9c7ef90c81524eae3eb6ec5346321e38b2a9e
> itests/hive-blobstore/src/test/results/clientpositive/zero_rows_blobstore.q.out 590f947add9b34d3adb05a7e22b0eb7d011c9a0d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 88bf82999953346bc92dac17c07373e6360be232
> ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java 0f9506376c283ec6e6ab659b20a5718807268969
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java 68ccda9ef61744e369638ceec5c083c77524c2db
>
>
> Diff: https://reviews.apache.org/r/60433/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Barna Zsombor Klara
>
>
Re: Review Request 60433: HIVE-16784: Missing lineage information when
hive.blobstore.optimizations.enabled is true
Posted by Sahil Takiar <ta...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60433/#review181386
-----------------------------------------------------------
Fix it, then Ship it!
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java
Lines 57 (patched)
<https://reviews.apache.org/r/60433/#comment256921>
Whats this change for?
- Sahil Takiar
On July 17, 2017, 2:08 p.m., Barna Zsombor Klara wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60433/
> -----------------------------------------------------------
>
> (Updated July 17, 2017, 2:08 p.m.)
>
>
> Review request for hive, Marta Kuczora, Sergio Pena, and Sahil Takiar.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-16784: Missing lineage information when hive.blobstore.optimizations.enabled is true
>
>
> Diffs
> -----
>
> itests/hive-blobstore/src/test/results/clientpositive/insert_blobstore_to_blobstore.q.out 46518999358be9ec5191ad2c3b37f555efbec263
> itests/hive-blobstore/src/test/results/clientpositive/insert_empty_into_blobstore.q.out 8e5e09627b3a0f5c66a01fdd096b4b17b0b10689
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 663a57274840a84a5f04777540d09d9f9bcfb5da
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out 8052fd86eefaff1fe7d024d3f58f6896e7223dd9
> itests/hive-blobstore/src/test/results/clientpositive/orc_format_nonpart.q.out f4705687ff4b7b29d9bd6b81dfcb4065364c2965
> itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out 1ef98104c5a711ef4b6553ca06fccf9003b99361
> itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out 9de4190072cc779b34e556a0e774ce7403324525
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_nonpart.q.out 44a1f11fa2c7bd32c636b9bad4e4e35d79989dcf
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out defca3bb78d4d97a327c389650d256a2cd880899
> itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out 5db9c7ef90c81524eae3eb6ec5346321e38b2a9e
> itests/hive-blobstore/src/test/results/clientpositive/zero_rows_blobstore.q.out 590f947add9b34d3adb05a7e22b0eb7d011c9a0d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 88bf82999953346bc92dac17c07373e6360be232
> ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java 0f9506376c283ec6e6ab659b20a5718807268969
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java 68ccda9ef61744e369638ceec5c083c77524c2db
>
>
> Diff: https://reviews.apache.org/r/60433/diff/2/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Barna Zsombor Klara
>
>
Re: Review Request 60433: HIVE-16784: Missing lineage information when
hive.blobstore.optimizations.enabled is true
Posted by Barna Zsombor Klara <zs...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60433/
-----------------------------------------------------------
(Updated July 17, 2017, 2:08 p.m.)
Review request for hive, Marta Kuczora, Sergio Pena, and Sahil Takiar.
Changes
-------
It seems we cannot just rewrite the path for the lineage information as it may have been used already. Instead we will duplicate it.... any cleaner suggestion would be much appreciated.
Repository: hive-git
Description
-------
HIVE-16784: Missing lineage information when hive.blobstore.optimizations.enabled is true
Diffs (updated)
-----
itests/hive-blobstore/src/test/results/clientpositive/insert_blobstore_to_blobstore.q.out 46518999358be9ec5191ad2c3b37f555efbec263
itests/hive-blobstore/src/test/results/clientpositive/insert_empty_into_blobstore.q.out 8e5e09627b3a0f5c66a01fdd096b4b17b0b10689
itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 663a57274840a84a5f04777540d09d9f9bcfb5da
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out 8052fd86eefaff1fe7d024d3f58f6896e7223dd9
itests/hive-blobstore/src/test/results/clientpositive/orc_format_nonpart.q.out f4705687ff4b7b29d9bd6b81dfcb4065364c2965
itests/hive-blobstore/src/test/results/clientpositive/orc_format_part.q.out 1ef98104c5a711ef4b6553ca06fccf9003b99361
itests/hive-blobstore/src/test/results/clientpositive/orc_nonstd_partitions_loc.q.out 9de4190072cc779b34e556a0e774ce7403324525
itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_nonpart.q.out 44a1f11fa2c7bd32c636b9bad4e4e35d79989dcf
itests/hive-blobstore/src/test/results/clientpositive/rcfile_format_part.q.out defca3bb78d4d97a327c389650d256a2cd880899
itests/hive-blobstore/src/test/results/clientpositive/rcfile_nonstd_partitions_loc.q.out 5db9c7ef90c81524eae3eb6ec5346321e38b2a9e
itests/hive-blobstore/src/test/results/clientpositive/zero_rows_blobstore.q.out 590f947add9b34d3adb05a7e22b0eb7d011c9a0d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 88bf82999953346bc92dac17c07373e6360be232
ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java 0f9506376c283ec6e6ab659b20a5718807268969
ql/src/test/org/apache/hadoop/hive/ql/optimizer/TestGenMapRedUtilsCreateConditionalTask.java 68ccda9ef61744e369638ceec5c083c77524c2db
Diff: https://reviews.apache.org/r/60433/diff/2/
Changes: https://reviews.apache.org/r/60433/diff/1-2/
Testing
-------
Thanks,
Barna Zsombor Klara