svn commit: r903318 - in /hadoop/hive/trunk: CHANGES.txt ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java

[na...@apache.org]

Author: namit
Date: Tue Jan 26 16:45:47 2010
New Revision: 903318

URL: http://svn.apache.org/viewvc?rev=903318&view=rev
Log:
HIVE-1094. Disable streaming last table if there is a skew key in
previous tables. (He Yongqiang via namit)


Modified:
    hadoop/hive/trunk/CHANGES.txt
    hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java

Modified: hadoop/hive/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/hadoop/hive/trunk/CHANGES.txt?rev=903318&r1=903317&r2=903318&view=diff
==============================================================================
--- hadoop/hive/trunk/CHANGES.txt (original)
+++ hadoop/hive/trunk/CHANGES.txt Tue Jan 26 16:45:47 2010
@@ -59,11 +59,14 @@
     HIVE-1072. Show table extended to error out when the partition does not exist.
     (Yongqiang He via zshao)
 
-    HIVE-1085. ColumnarSerde should not be the default Serde when user 
+    HIVE-1085. ColumnarSerde should not be the default Serde when user
     specified a fileformat using 'stored as' (Yongqiang He via Ning Zhang)
 
     HIVE-1086. Add "-Doffline=true" option to ant. (Zheng Shao via Ning Zhang)
 
+    HIVE-1094. Disable streaming last table if there is a skew key in
+    previous tables. (He Yongqiang via namit)
+
 Release 0.5.0 -  Unreleased
 
   INCOMPATIBLE CHANGES
@@ -1508,5 +1511,5 @@
     HIVE-1088. RCFileRecordReader's first split will read duplicate rows if
     the split end is < the first sync mark (Yongqiang He via Ning Zhang)
 
-    HIVE-1087. Let user script write out binary data into a table 
+    HIVE-1087. Let user script write out binary data into a table
     (Zheng Shao via Ning Zhang)

Modified: hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java
URL: http://svn.apache.org/viewvc/hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java?rev=903318&r1=903317&r2=903318&view=diff
==============================================================================
--- hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java (original)
+++ hadoop/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java Tue Jan 26 16:45:47 2010
@@ -74,7 +74,7 @@
       int sz = storage.get(alias).size();
 
       // Are we consuming too much memory
-      if (alias == numAliases - 1) {
+      if (alias == numAliases - 1 && !(handleSkewJoin && skewJoinKeyContext.currBigKeyTag >= 0)) {
         if (sz == joinEmitInterval) {
           // The input is sorted by alias, so if we are already in the last join
           // operand,
@@ -130,7 +130,8 @@
   @Override
   public void jobClose(Configuration hconf, boolean success)
       throws HiveException {
-    if (handleSkewJoin) {
+    int numAliases = conf.getExprs().size();
+    if (conf.getHandleSkewJoin()) {
       try {
         for (int i = 0; i < numAliases; i++) {
           String specPath = conf.getBigKeysDirMap().get((byte) i);