You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Sergey Shelukhin <se...@hortonworks.com> on 2014/05/01 00:35:57 UTC
Re: Review Request 18936: HIVE-6430 MapJoin hash table has large memory
overhead
> On April 30, 2014, 2:37 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java, line 248
> > <https://reviews.apache.org/r/18936/diff/11/?file=571300#file571300line248>
> >
> > {} block please, I almost missed reading that.
sorry, that is left over from debugging
> On April 30, 2014, 2:37 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java, line 620
> > <https://reviews.apache.org/r/18936/diff/11/?file=571303#file571303line620>
> >
> > mark class as final, so that the method inlining kicks in.
hmm... will do, but is it needed for static methods?
> On April 30, 2014, 2:37 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java, line 140
> > <https://reviews.apache.org/r/18936/diff/11/?file=571306#file571306line140>
> >
> > +1 - this shows in the older profiles as being very heavy.
?
> On April 30, 2014, 2:37 a.m., Gopal V wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java, line 119
> > <https://reviews.apache.org/r/18936/diff/11/?file=571311#file571311line119>
> >
> > Is this being checked twice for some reason?
once for each table. I think the key is supposed to be the same now.
- Sergey
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18936/#review41798
-----------------------------------------------------------
On April 30, 2014, 2:03 a.m., Sergey Shelukhin wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18936/
> -----------------------------------------------------------
>
> (Updated April 30, 2014, 2:03 a.m.)
>
>
> Review request for hive, Gopal V and Gunther Hagleitner.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> See JIRA
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 0efe58b
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java 5fe35a5
> hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 142bfd8
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java bf9d4c1
> ql/src/java/org/apache/hadoop/hive/ql/debug/Utils.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java f5d4670
> ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java b93ea7a
> ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 175d3ab
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/AbstractMapJoinTableContainer.java 8854b19
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/BytesBytesMultiHashMap.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/HashMapWrapper.java 9df425b
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinBytesTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinKey.java 64f0be2
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinPersistableTableContainer.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinRowContainer.java 008a8db
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainer.java 988959f
> ql/src/java/org/apache/hadoop/hive/ql/exec/persistence/MapJoinTableContainerSerDe.java 55b7415
> ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java e392592
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java eef7656
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedColumnarSerDe.java d4be78d
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java 3077d75
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/ReduceSinkMapJoinProc.java f7b499b
> ql/src/java/org/apache/hadoop/hive/ql/plan/ReduceSinkDesc.java 157d072
> ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 118b339
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestBytesBytesMultiHashMap.java PRE-CREATION
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinEqualityTableContainer.java 65e3779
> ql/src/test/org/apache/hadoop/hive/ql/exec/persistence/TestMapJoinTableContainer.java 093da55
> ql/src/test/queries/clientpositive/mapjoin_decimal.q b65a7be
> ql/src/test/queries/clientpositive/mapjoin_mapjoin.q 1eb95f6
> ql/src/test/queries/clientpositive/tez_union.q f80d94c
> ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 8350670
> ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out 3c55b5c
> ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out 284cc03
> serde/src/java/org/apache/hadoop/hive/serde2/ByteStream.java 73d9b29
> serde/src/java/org/apache/hadoop/hive/serde2/WriteBuffers.java PRE-CREATION
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 9079b9d
> serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/OutputByteBuffer.java 1b09d41
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/ColumnarSerDe.java 5870884
> serde/src/java/org/apache/hadoop/hive/serde2/columnar/LazyBinaryColumnarSerDe.java bab505e
> serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDe.java 6f344bb
> serde/src/java/org/apache/hadoop/hive/serde2/io/DateWritable.java 1f4ccdd
> serde/src/java/org/apache/hadoop/hive/serde2/io/HiveDecimalWritable.java a99c7b4
> serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java 435d6c6
> serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazySimpleSerDe.java 82c1263
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java b188c3f
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryStruct.java caf3517
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 6c14081
> serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java e5ea452
> serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 06d5c5e
> serde/src/test/org/apache/hadoop/hive/serde2/lazy/TestLazyPrimitive.java 868dd4c
> serde/src/test/org/apache/hadoop/hive/serde2/thrift_test/CreateSequenceFile.java 1fb49e5
>
> Diff: https://reviews.apache.org/r/18936/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Sergey Shelukhin
>
>