You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sergey Shelukhin <se...@hortonworks.com> on 2014/03/08 01:31:06 UTC
Review Request 18936: HIVE-6430 MapJoin hash table has large memory overhead
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edc3d38
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3daf7a5
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 61545b5
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java a00aab3
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java a8cb1ae
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 84739ee
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 6ecbcf7
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out bc2c650
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On March 13, 2014, 6 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java, line 31
> > <https://reviews.apache.org/r/18936/diff/2/?file=517002#file517002line31>
> >
> > The ref longs are not just indexes, they contain bits from various components.
index in the array is the hash code
> On March 13, 2014, 6 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java, line 173
> > <https://reviews.apache.org/r/18936/diff/2/?file=517002#file517002line173>
> >
> > reserve -ve bytes?
it was supported, made cleaner
> On March 13, 2014, 6 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java, line 477
> > <https://reviews.apache.org/r/18936/diff/2/?file=517002#file517002line477>
> >
> > Really? :)
yes :) removed comment
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review36939
-----------------------------------------------------------
On March 12, 2014, 2:30 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated March 12, 2014, 2:30 a.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edc3d38
> ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3daf7a5
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 61545b5
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java a00aab3
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java a8cb1ae
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 84739ee
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 6ecbcf7
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984
> ql/src/test/results/clientpositive/tez/ctas.q.out aaac91c
> ql/src/test/results/clientpositive/tez/groupby3.q.out cb8f0e7
> ql/src/test/results/clientpositive/tez/insert1.q.out 7b8819d
> ql/src/test/results/clientpositive/tez/limit_pushdown.q.out 7d6b1c7
> ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out d420caf
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out bc2c650
> ql/src/test/results/clientpositive/tez/subquery_in.q.out c66602b
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Gopal V <go...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review36939
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment68190>
The ref longs are not just indexes, they contain bits from various components.
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment68125>
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment68126>
reserve -ve bytes?
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment68252>
Really? :)
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment68250>
Rehashing involves calculating hashCodes of all keys - this is very expensive CPU wise.
- Gopal V
On March 12, 2014, 2:30 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated March 12, 2014, 2:30 a.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edc3d38
> ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3daf7a5
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 61545b5
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java a00aab3
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java a8cb1ae
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 84739ee
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 6ecbcf7
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984
> ql/src/test/results/clientpositive/tez/ctas.q.out aaac91c
> ql/src/test/results/clientpositive/tez/groupby3.q.out cb8f0e7
> ql/src/test/results/clientpositive/tez/insert1.q.out 7b8819d
> ql/src/test/results/clientpositive/tez/limit_pushdown.q.out 7d6b1c7
> ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out d420caf
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out bc2c650
> ql/src/test/results/clientpositive/tez/subquery_in.q.out c66602b
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review37309
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java
<https://reviews.apache.org/r/18936/#comment68767>
retained from testing, will remove
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java
<https://reviews.apache.org/r/18936/#comment68768>
retained from testing, will remove
- Sergey Shelukhin
On March 15, 2014, 1:22 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated March 15, 2014, 1:22 a.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 56d68f5
> ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3ea9c96
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java a00aab3
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 79af08d
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review37681
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java
<https://reviews.apache.org/r/18936/#comment69316>
should have been changed, will do
- Sergey Shelukhin
On March 19, 2014, 2:40 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated March 19, 2014, 2:40 a.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b0f5c49
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 2cd65cb
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 704fcb9
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java 7dbb8be
> ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3ea9c96
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 79af08d
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java 0fd4983
> ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
> ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984
> ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
> serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
> serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On April 30, 2014, 2:37 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java, line 248
> > <https://reviews.apache.org/r/18936/diff/11/?file=571300#file571300line248>
> >
> > {} block please, I almost missed reading that.
sorry, that is left over from debugging
> On April 30, 2014, 2:37 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java, line 620
> > <https://reviews.apache.org/r/18936/diff/11/?file=571303#file571303line620>
> >
> > mark class as final, so that the method inlining kicks in.
hmm... will do, but is it needed for static methods?
> On April 30, 2014, 2:37 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java, line 140
> > <https://reviews.apache.org/r/18936/diff/11/?file=571306#file571306line140>
> >
> > +1 - this shows in the older profiles as being very heavy.
?
> On April 30, 2014, 2:37 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java, line 119
> > <https://reviews.apache.org/r/18936/diff/11/?file=571311#file571311line119>
> >
> > Is this being checked twice for some reason?
once for each table. I think the key is supposed to be the same now.
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review41798
-----------------------------------------------------------
On April 30, 2014, 2:03 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated April 30, 2014, 2:03 a.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0efe58b
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
> ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 3077d75
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
> ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
> ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/queries/clientpositive/tez_union.q f80d94c
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
> ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
> serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
> serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Gopal V <go...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review41798
-----------------------------------------------------------
Minor comments, mostly related to JIT inlining.
Otherwise I'm on board this patch.
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
<https://reviews.apache.org/r/18936/#comment75410>
Add this to hive-default.xml
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java
<https://reviews.apache.org/r/18936/#comment75411>
nit: whitespace
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
<https://reviews.apache.org/r/18936/#comment75412>
{} block please, I almost missed reading that.
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
<https://reviews.apache.org/r/18936/#comment75413>
+1 - good idea
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
<https://reviews.apache.org/r/18936/#comment75414>
+1 - nice idea.
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java
<https://reviews.apache.org/r/18936/#comment75415>
document this - a protected static looks really odd without a link to caller in the comments.
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment75417>
wrap in isDebugEnabled()
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment75416>
mark class as final, so that the method inlining kicks in.
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java
<https://reviews.apache.org/r/18936/#comment75419>
+1 - this shows in the older profiles as being very heavy.
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java
<https://reviews.apache.org/r/18936/#comment75420>
Is this being checked twice for some reason?
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java
<https://reviews.apache.org/r/18936/#comment75421>
please submit patch to hadoop as well.
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java
<https://reviews.apache.org/r/18936/#comment75422>
final please, this is good specialization.
- Gopal V
On April 30, 2014, 2:03 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated April 30, 2014, 2:03 a.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0efe58b
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
> ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 3077d75
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
> ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
> ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/queries/clientpositive/tez_union.q f80d94c
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
> ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
> serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
> serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Gopal V <go...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review41935
-----------------------------------------------------------
Ship it!
Ship It!
- Gopal V
On May 1, 2014, 2:29 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated May 1, 2014, 2:29 a.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 604bea7
> conf/hive-default.xml.template 2552560
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
> ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 3077d75
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
> ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
> ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/queries/clientpositive/tez_union.q f80d94c
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
> ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
> serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
> serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On May 8, 2014, 10:05 p.m., Gunther Hagleitner wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java, line 405
> > <https://reviews.apache.org/r/18936/diff/13/?file=572109#file572109line405>
> >
> > why do you need this? this seems to do the same thing as tag == -1?
it's more explicit and stays that way if someone resets tag later
> On May 8, 2014, 10:05 p.m., Gunther Hagleitner wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java, line 470
> > <https://reviews.apache.org/r/18936/diff/13/?file=572109#file572109line470>
> >
> > this should exist on the operator, but on the ReduceSinkDesc
when we set it, we are operating on already-created operator
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review42539
-----------------------------------------------------------
On May 9, 2014, 8:16 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated May 9, 2014, 8:16 p.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 604bea7
> conf/hive-default.xml.template 2552560
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
> ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 674ed48
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
> ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
> ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/queries/clientpositive/tez_union.q f80d94c
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
> ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
> serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
> serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Gunther Hagleitner <gh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review42539
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java
<https://reviews.apache.org/r/18936/#comment76332>
This is nice. But should have documentation for class and public methods
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java
<https://reviews.apache.org/r/18936/#comment76333>
There has to be a more portable way to create temp file.
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
<https://reviews.apache.org/r/18936/#comment76334>
Can you make this a jira or drop if it's not important enough?
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java
<https://reviews.apache.org/r/18936/#comment76335>
coding standards
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
<https://reviews.apache.org/r/18936/#comment76337>
why do you need this? this seems to do the same thing as tag == -1?
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
<https://reviews.apache.org/r/18936/#comment76338>
this should exist on the operator, but on the ReduceSinkDesc
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment76340>
needs asf header
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment76341>
can you please use curlies in this file. coding standard again.
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment76342>
same as before. todos shoudl be jiras or removed if not important
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment76336>
if debugenabled?
- Gunther Hagleitner
On May 1, 2014, 2:29 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated May 1, 2014, 2:29 a.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 604bea7
> conf/hive-default.xml.template 2552560
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
> ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 3077d75
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
> ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
> ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/queries/clientpositive/tez_union.q f80d94c
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
> ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
> serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
> serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On May 9, 2014, 1:58 a.m., Gunther Hagleitner wrote:
> > serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java, line 147
> > <https://reviews.apache.org/r/18936/diff/13/?file=572150#file572150line147>
> >
> > randomaccess doesn't extend output?
no
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review42555
-----------------------------------------------------------
On May 1, 2014, 2:29 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated May 1, 2014, 2:29 a.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 604bea7
> conf/hive-default.xml.template 2552560
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
> ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 3077d75
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
> ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
> ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/queries/clientpositive/tez_union.q f80d94c
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
> ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
> serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
> serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Gunther Hagleitner <gh...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review42555
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java
<https://reviews.apache.org/r/18936/#comment76364>
see other comment. shouldn't need at all - or at least not in the operator...
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java
<https://reviews.apache.org/r/18936/#comment76365>
this file also has a bunch of no curly conditions...
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java
<https://reviews.apache.org/r/18936/#comment76363>
randomaccess doesn't extend output?
- Gunther Hagleitner
On May 1, 2014, 2:29 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated May 1, 2014, 2:29 a.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 604bea7
> conf/hive-default.xml.template 2552560
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
> ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 3077d75
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
> ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
> ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/queries/clientpositive/tez_union.q f80d94c
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
> ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
> serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
> serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Gopal V <go...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review41936
-----------------------------------------------------------
Ship it!
Ship It!
- Gopal V
On May 1, 2014, 2:29 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated May 1, 2014, 2:29 a.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 604bea7
> conf/hive-default.xml.template 2552560
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
> ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 3077d75
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
> ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
> ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/queries/clientpositive/tez_union.q f80d94c
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
> ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
> serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
> serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated May 14, 2014, 8:22 p.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 604bea7
conf/hive-default.xml.template 2552560
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 2dbe334
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory.java accc312
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseKeyFactory2.java 1bd2352
ql/src/java/org/apache/hadoop/hive/ql/Driver.java fce77a8
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 674ed48
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/queries/clientpositive/tez_union.q f80d94c
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out cb11b8b
ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 1c16024
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 614a4a6
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java 98a35c7
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated May 9, 2014, 8:16 p.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 604bea7
conf/hive-default.xml.template 2552560
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 674ed48
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/queries/clientpositive/tez_union.q f80d94c
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated May 1, 2014, 2:29 a.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 604bea7
conf/hive-default.xml.template 2552560
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 3077d75
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/queries/clientpositive/tez_union.q f80d94c
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review41892
-----------------------------------------------------------
conf/hive-default.xml.template
<https://reviews.apache.org/r/18936/#comment75550>
dup, will remove
- Sergey Shelukhin
On April 30, 2014, 11:14 p.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated April 30, 2014, 11:14 p.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0efe58b
> conf/hive-default.xml.template 6620bd1
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
> ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 3077d75
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
> ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
> ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/queries/clientpositive/tez_union.q f80d94c
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
> ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
> serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
> serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated April 30, 2014, 11:14 p.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0efe58b
conf/hive-default.xml.template 6620bd1
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 3077d75
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/queries/clientpositive/tez_union.q f80d94c
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated April 30, 2014, 2:03 a.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0efe58b
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 3077d75
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/queries/clientpositive/tez_union.q f80d94c
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated April 26, 2014, 1:53 a.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e0e1339
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 3077d75
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 0e0395e
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated April 18, 2014, 1 a.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Changes
-------
Another iteration
Repository: hive-git
Description
-------
See JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e0e1339
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 2b1438d
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 1104a2b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated April 17, 2014, 1:07 a.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java da45f1a
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 370f6e4
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 2b1438d
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 1104a2b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated March 20, 2014, 11:26 p.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b0f5c49
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 704fcb9
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 7dbb8be
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3ea9c96
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java bf94930
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java 0fd4983
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984
ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated March 19, 2014, 2:40 a.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b0f5c49
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 2cd65cb
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 704fcb9
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 7dbb8be
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3ea9c96
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 79af08d
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java 0fd4983
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984
ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated March 17, 2014, 8:24 a.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 56d68f5
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 2cd65cb
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 704fcb9
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 7dbb8be
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3ea9c96
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java a00aab3
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 79af08d
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java 0fd4983
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated March 15, 2014, 3:35 a.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 56d68f5
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 2cd65cb
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 704fcb9
ql/src/java/org/apache/hadoop/hive/ql/Driver.java 7dbb8be
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3ea9c96
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java a00aab3
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 79af08d
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java 0fd4983
ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated March 15, 2014, 1:22 a.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 56d68f5
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3ea9c96
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java a00aab3
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 79af08d
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated March 15, 2014, 1:17 a.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 56d68f5
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3ea9c96
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java a00aab3
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 79af08d
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/
-----------------------------------------------------------
(Updated March 12, 2014, 2:30 a.m.)
Review request for hive, Gopal V and Gunther Hagleitner.
Repository: hive-git
Description
-------
See JIRA
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edc3d38
ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3daf7a5
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 61545b5
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java a00aab3
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java a8cb1ae
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 84739ee
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 6ecbcf7
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984
ql/src/test/results/clientpositive/tez/ctas.q.out aaac91c
ql/src/test/results/clientpositive/tez/groupby3.q.out cb8f0e7
ql/src/test/results/clientpositive/tez/insert1.q.out 7b8819d
ql/src/test/results/clientpositive/tez/limit_pushdown.q.out 7d6b1c7
ql/src/test/results/clientpositive/tez/load_dyn_part1.q.out d420caf
ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out bc2c650
ql/src/test/results/clientpositive/tez/subquery_in.q.out c66602b
serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
Diff: https://reviews.apache.org/r/18936/diff/
Testing
-------
Thanks,
Sergey Shelukhin
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Sergey Shelukhin <se...@hortonworks.com>.
> On March 11, 2014, 12:30 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java, line 195
> > <https://reviews.apache.org/r/18936/diff/1/?file=513985#file513985line195>
> >
> > Quadriatic probing is much nicer for collisions.
this is quadratic probing. It uses triangular numbers, which are (n+1)*n/2
It "resets" to random slot number when cycling across the end of hashmap, will fix that
> On March 11, 2014, 12:30 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java, line 250
> > <https://reviews.apache.org/r/18936/diff/1/?file=513985#file513985line250>
> >
> > if cmpLength != keylength comparison - cannot be equal if they are not byte-for-byte equal, right?
that's checked first thing in isEqual, but yeah, can be checked earlier
> On March 11, 2014, 12:30 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java, line 169
> > <https://reviews.apache.org/r/18936/diff/1/?file=513994#file513994line169>
> >
> > why is there an init()?
this code is removed
> On March 11, 2014, 12:30 a.m., Gopal V wrote:
> > serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java, line 271
> > <https://reviews.apache.org/r/18936/diff/1/?file=514006#file514006line271>
> >
> > Comment eaten up in diff?
no, it no longer returns
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review36680
-----------------------------------------------------------
On March 8, 2014, 12:31 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated March 8, 2014, 12:31 a.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edc3d38
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3daf7a5
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 61545b5
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java a00aab3
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java a8cb1ae
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 84739ee
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 6ecbcf7
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out bc2c650
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
Posted by Gopal V <go...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review36680
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment67786>
This would be much simpler if you split up the details into 2 groups.
1) Finding the key
2) Finding the value(s)
Because #1 is well understood for closed hashtables.
And #2 is where all the complexity is for this impl, with the multi value linked list via offsets implementation.
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment67737>
This should be an IllegalArgumentException - we don't run asserts in production.
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment67740>
Quadriatic probing is much nicer for collisions.
ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java
<https://reviews.apache.org/r/18936/#comment67742>
if cmpLength != keylength comparison - cannot be equal if they are not byte-for-byte equal, right?
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java
<https://reviews.apache.org/r/18936/#comment67748>
why is there an init()?
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java
<https://reviews.apache.org/r/18936/#comment67736>
Comment eaten up in diff?
- Gopal V
On March 8, 2014, 12:31 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated March 8, 2014, 12:31 a.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java edc3d38
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 170e8c0
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java 3daf7a5
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 61545b5
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java a00aab3
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java a8cb1ae
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 84739ee
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 6ecbcf7
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 755d783
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out d79b984
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out bc2c650
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>