[GitHub] yucai commented on issue #23821: [SPARK-26909][FOLLOWUP][SQL] use unsafeRow.hashCode() as hash value in HashAggregate

[GitBox <gi...@apache.org>]

yucai commented on issue #23821: [SPARK-26909][FOLLOWUP][SQL] use unsafeRow.hashCode() as hash value in HashAggregate
URL: https://github.com/apache/spark/pull/23821#issuecomment-464627362
 
 
   Generated codes:
   
   ```
   /* 054 */     // generate grouping key
   /* 055 */     agg_mutableStateArray_0[0].reset();
   /* 056 */
   /* 057 */     agg_mutableStateArray_0[0].write(0, agg_expr_0_0);
   /* 058 */     int agg_unsafeRowHash_0 = (agg_mutableStateArray_0[0].getRow()).hashCode();
   /* 059 */     if (true) {
   /* 060 */       // try to get the buffer from hash map
   /* 061 */       agg_unsafeRowAggBuffer_0 =
   /* 062 */       agg_hashMap_0.getAggregationBufferFromUnsafeRow((agg_mutableStateArray_0[0].getRow()), agg_unsafeRowHash_0);
   /* 063 */     }
   /* 064 */     // Can't allocate buffer from the hash map. Spill the map and fallback to sort-based
   /* 065 */     // aggregation after processing all input rows.
   /* 066 */     if (agg_unsafeRowAggBuffer_0 == null) {
   /* 067 */       if (agg_sorter_0 == null) {
   /* 068 */         agg_sorter_0 = agg_hashMap_0.destructAndCreateExternalSorter();
   /* 069 */       } else {
   /* 070 */         agg_sorter_0.merge(agg_hashMap_0.destructAndCreateExternalSorter());
   /* 071 */       }
   /* 072 */
   /* 073 */       // the hash map had be spilled, it should have enough memory now,
   /* 074 */       // try to allocate buffer again.
   /* 075 */       agg_unsafeRowAggBuffer_0 = agg_hashMap_0.getAggregationBufferFromUnsafeRow(
   /* 076 */         (agg_mutableStateArray_0[0].getRow()), agg_unsafeRowHash_0);
   /* 077 */       if (agg_unsafeRowAggBuffer_0 == null) {
   /* 078 */         // failed to allocate the first page
   /* 079 */         throw new org.apache.spark.memory.SparkOutOfMemoryError("No enough memory for aggregation");
   /* 080 */       }
   /* 081 */     }
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org