You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Suma Shivaprasad <su...@gmail.com> on 2014/08/08 08:21:59 UTC

Review Request 24497: HIVE-7629 - Map joins between two parquet tables failing

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24497/
-----------------------------------------------------------

Review request for hive.


Bugs: HIVE-7629
    https://issues.apache.org/jira/browse/HIVE-7629


Repository: hive-git


Description
-------

Map Joins between 2 parquet tables are failing since the Mapper is trying to access the columns of the first table(bigger table) while trying to load the  second table(smaller map join table). Fixed this by adding a guard on the column indexes passed by hive


Diffs
-----

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ProjectionPusher.java 2f155f6 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java d6be4bd 
  ql/src/test/queries/clientpositive/parquet_join.q PRE-CREATION 
  ql/src/test/results/clientpositive/parquet_join.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/24497/diff/


Testing
-------

parquet_join.q covers most types of joins between 2 parquet tables - Normal, Map join, SMB join


Thanks,

Suma Shivaprasad


Re: Review Request 24497: HIVE-7629 - Map joins between two parquet tables failing

Posted by Suma Shivaprasad <su...@gmail.com>.

> On Aug. 11, 2014, 4:05 p.m., Brock Noland wrote:
> > ql/src/test/queries/clientpositive/parquet_join.q, line 1
> > <https://reviews.apache.org/r/24497/diff/1/?file=656508#file656508line1>
> >
> >     Can you add comments (start with --) which describe how this reproduces the bug?

there is a parsing issue with the comments added. Will update the diff again


- Suma


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24497/#review50182
-----------------------------------------------------------


On Aug. 12, 2014, 6:28 a.m., Suma Shivaprasad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24497/
> -----------------------------------------------------------
> 
> (Updated Aug. 12, 2014, 6:28 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-7629
>     https://issues.apache.org/jira/browse/HIVE-7629
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Map Joins between 2 parquet tables are failing since the Mapper is trying to access the columns of the first table(bigger table) while trying to load the  second table(smaller map join table). Fixed this by adding a guard on the column indexes passed by hive
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java d6be4bd 
>   ql/src/test/queries/clientpositive/parquet_join.q PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/24497/diff/
> 
> 
> Testing
> -------
> 
> parquet_join.q covers most types of joins between 2 parquet tables - Normal, Map join, SMB join
> 
> 
> Thanks,
> 
> Suma Shivaprasad
> 
>


Re: Review Request 24497: HIVE-7629 - Map joins between two parquet tables failing

Posted by Suma Shivaprasad <su...@gmail.com>.

> On Aug. 11, 2014, 4:05 p.m., Brock Noland wrote:
> > Thank you very much! Two comments below.

Thanks for reviewing this quickly


- Suma


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24497/#review50182
-----------------------------------------------------------


On Aug. 12, 2014, 6:28 a.m., Suma Shivaprasad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24497/
> -----------------------------------------------------------
> 
> (Updated Aug. 12, 2014, 6:28 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-7629
>     https://issues.apache.org/jira/browse/HIVE-7629
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Map Joins between 2 parquet tables are failing since the Mapper is trying to access the columns of the first table(bigger table) while trying to load the  second table(smaller map join table). Fixed this by adding a guard on the column indexes passed by hive
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java d6be4bd 
>   ql/src/test/queries/clientpositive/parquet_join.q PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/24497/diff/
> 
> 
> Testing
> -------
> 
> parquet_join.q covers most types of joins between 2 parquet tables - Normal, Map join, SMB join
> 
> 
> Thanks,
> 
> Suma Shivaprasad
> 
>


Re: Review Request 24497: HIVE-7629 - Map joins between two parquet tables failing

Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24497/#review50182
-----------------------------------------------------------


Thank you very much! Two comments below.


ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java
<https://reviews.apache.org/r/24497/#comment87783>

    nit: Missing space between if and (



ql/src/test/queries/clientpositive/parquet_join.q
<https://reviews.apache.org/r/24497/#comment87782>

    Can you add comments (start with --) which describe how this reproduces the bug?


- Brock Noland


On Aug. 8, 2014, 6:21 a.m., Suma Shivaprasad wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24497/
> -----------------------------------------------------------
> 
> (Updated Aug. 8, 2014, 6:21 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-7629
>     https://issues.apache.org/jira/browse/HIVE-7629
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Map Joins between 2 parquet tables are failing since the Mapper is trying to access the columns of the first table(bigger table) while trying to load the  second table(smaller map join table). Fixed this by adding a guard on the column indexes passed by hive
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ProjectionPusher.java 2f155f6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java d6be4bd 
>   ql/src/test/queries/clientpositive/parquet_join.q PRE-CREATION 
>   ql/src/test/results/clientpositive/parquet_join.q.out PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/24497/diff/
> 
> 
> Testing
> -------
> 
> parquet_join.q covers most types of joins between 2 parquet tables - Normal, Map join, SMB join
> 
> 
> Thanks,
> 
> Suma Shivaprasad
> 
>


Re: Review Request 24497: HIVE-7629 - Map joins between two parquet tables failing

Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24497/
-----------------------------------------------------------

(Updated Aug. 12, 2014, 5:38 p.m.)


Review request for hive.


Changes
-------

Fixed parse error


Bugs: HIVE-7629
    https://issues.apache.org/jira/browse/HIVE-7629


Repository: hive-git


Description
-------

Map Joins between 2 parquet tables are failing since the Mapper is trying to access the columns of the first table(bigger table) while trying to load the  second table(smaller map join table). Fixed this by adding a guard on the column indexes passed by hive


Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java d6be4bd 
  ql/src/test/queries/clientpositive/parquet_join.q PRE-CREATION 
  ql/src/test/results/clientpositive/parquet_join.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/24497/diff/


Testing
-------

parquet_join.q covers most types of joins between 2 parquet tables - Normal, Map join, SMB join


Thanks,

Suma Shivaprasad


Re: Review Request 24497: HIVE-7629 - Map joins between two parquet tables failing

Posted by Suma Shivaprasad <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24497/
-----------------------------------------------------------

(Updated Aug. 12, 2014, 6:28 a.m.)


Review request for hive.


Changes
-------

Fixed review comments


Bugs: HIVE-7629
    https://issues.apache.org/jira/browse/HIVE-7629


Repository: hive-git


Description
-------

Map Joins between 2 parquet tables are failing since the Mapper is trying to access the columns of the first table(bigger table) while trying to load the  second table(smaller map join table). Fixed this by adding a guard on the column indexes passed by hive


Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/read/DataWritableReadSupport.java d6be4bd 
  ql/src/test/queries/clientpositive/parquet_join.q PRE-CREATION 

Diff: https://reviews.apache.org/r/24497/diff/


Testing
-------

parquet_join.q covers most types of joins between 2 parquet tables - Normal, Map join, SMB join


Thanks,

Suma Shivaprasad