You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "advancedxy (via GitHub)" <gi...@apache.org> on 2024/05/14 13:10:47 UTC

[PR] feat: Add xxhash64 function support [datafusion-comet]

advancedxy opened a new pull request, #424:
URL: https://github.com/apache/datafusion-comet/pull/424

   ## Which issue does this PR close?
   Part of #205 
   Closes #344 
   
   
   ## Rationale for this change
   More function coverage
   
   ## What changes are included in this PR?
   1. include twox-hash as a dep in rust
   2. implement xxhash64 related method in rust side
   3. glue code to bridge the jvm and rust
   
   ## How are these changes tested?
   New added test.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600867066


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -193,27 +241,67 @@ macro_rules! hash_array_decimal {
 fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
     array: &ArrayRef,
     hashes_buffer: &mut [u32],
+    multi_col: bool,
 ) -> Result<()> {
     let dict_array = array.as_any().downcast_ref::<DictionaryArray<K>>().unwrap();
+    if multi_col {
+        // unpack the dictionary array as each row may have a different hash input
+        let unpacked = take(dict_array.values().as_ref(), dict_array.keys(), None)?;
+        create_hashes(&[unpacked], hashes_buffer)?;
+    } else {
+        // Hash each dictionary value once, and then use that computed
+        // hash for each key value to avoid a potentially expensive
+        // redundant hashing for large dictionary elements (e.g. strings)
+        let dict_values = Arc::clone(dict_array.values());
+        // same initial seed as Spark
+        let mut dict_hashes = vec![42; dict_values.len()];
+        create_hashes(&[dict_values], &mut dict_hashes)?;

Review Comment:
   But when we compute hashes, next hash is computed based on current value and  previous hash value. But when we compute hashes for dictionary values, we compute it based on "dictionary values" order, instead of the row value order.
   
   In other words, we compute the hashes of dictionary values like:
   
   ```
   dict_hashes[0] = hash(2, init_seed)
   dict_hashes[1] = hash(1, dict_hashes[0])
   dict_hashes[2] = hash(4, dict_hashes[1]) 
   dict_hashes[3] = hash(3, dict_hashes[2]) 
   dict_hashes[4] = hash(5, dict_hashes[3]) 
   ```
   
   So,
   
   ```
   row_id = 0, dict_key = 1, dict_hashes[1] = hash(1, dict_hashes[0]) but it should be hash(1, init_seed)
   row_id = 1, dict_key = 0, dict_hashes[0] = hash(2, init_seed) but it should be hash(2, hash(1, init_seed))
   row_id = 2, dict_key = 0 dict_hashes[0] = hash(2, init_seed) but it should be hash(2, hash(2, hash(1, init_seed)))
   ...
   ```
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1623197730


##########
spark/inspections/CometTPCHQueriesList-results.txt:
##########
@@ -1,142 +1,129 @@
 Query: q1 TPCH Snappy. Comet Exec: Enabled (CometHashAggregate, CometFilter, CometProject)
 Query: q1 TPCH Snappy: ExplainInfo:
-Shuffle: unsupported Spark partitioning: org.apache.spark.sql.catalyst.plans.physical.RangePartitioning
+Comet shuffle is not enabled: spark.sql.adaptive.coalescePartitions.enabled is enabled and spark.comet.shuffle.enforceMode.enabled is not enabled

Review Comment:
   Thanks,  cherry-picked.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600504627


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -238,55 +241,67 @@ macro_rules! hash_array_decimal {
 fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
     array: &ArrayRef,
     hashes_buffer: &mut [u32],
+    multi_col: bool,
 ) -> Result<()> {
     let dict_array = array.as_any().downcast_ref::<DictionaryArray<K>>().unwrap();
-
-    // Hash each dictionary value once, and then use that computed
-    // hash for each key value to avoid a potentially expensive
-    // redundant hashing for large dictionary elements (e.g. strings)
-    let dict_values = Arc::clone(dict_array.values());
-    let mut dict_hashes = vec![0; dict_values.len()];
-    create_hashes(&[dict_values], &mut dict_hashes)?;
-
-    for (hash, key) in hashes_buffer.iter_mut().zip(dict_array.keys().iter()) {
-        if let Some(key) = key {
-            let idx = key.to_usize().ok_or_else(|| {
-                DataFusionError::Internal(format!(
-                    "Can not convert key value {:?} to usize in dictionary of type {:?}",
-                    key,
-                    dict_array.data_type()
-                ))
-            })?;
-            *hash = dict_hashes[idx]
-        } // no update for Null, consistent with other hashes
+    if multi_col {
+        // unpack the dictionary array as each row may have a different hash input
+        let unpacked = take(dict_array.values().as_ref(), dict_array.keys(), None)?;
+        create_hashes(&[unpacked], hashes_buffer)?;
+    } else {

Review Comment:
   Ah, I see. Because `hashes_buffer` has hashed result for previous columns.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600023085


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -218,6 +263,34 @@ fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
     Ok(())
 }
 
+fn create_xxhash64_hashes_dictionary<K: ArrowDictionaryKeyType>(

Review Comment:
   This is mostly copied and pasted from `create_hashes_dictionary`, maybe  we should rewrite it with macros?



##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -347,6 +513,213 @@ pub fn create_hashes<'a>(
     Ok(hashes_buffer)
 }
 
+pub fn create_xxhash64_hashes<'a>(
+    arrays: &[ArrayRef],
+    hashes_buffer: &'a mut [u64],

Review Comment:
   It's similar as `create_hashes` as well.
   I'm not sure how to define a generic method that take both `&'a mut [u32]` and `&'a mut [u64]`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1619483941


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -244,111 +283,214 @@ fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
 ///
 /// The number of rows to hash is determined by `hashes_buffer.len()`.
 /// `hashes_buffer` should be pre-sized appropriately
-pub fn create_hashes<'a>(
-    arrays: &[ArrayRef],
-    hashes_buffer: &'a mut [u32],
-) -> Result<&'a mut [u32]> {
-    for (i, col) in arrays.iter().enumerate() {
-        let first_col = i == 0;
-        match col.data_type() {
-            DataType::Boolean => {
-                hash_array_boolean!(BooleanArray, col, i32, hashes_buffer);
-            }
-            DataType::Int8 => {
-                hash_array_primitive!(Int8Array, col, i32, hashes_buffer);
-            }
-            DataType::Int16 => {
-                hash_array_primitive!(Int16Array, col, i32, hashes_buffer);
-            }
-            DataType::Int32 => {
-                hash_array_primitive!(Int32Array, col, i32, hashes_buffer);
-            }
-            DataType::Int64 => {
-                hash_array_primitive!(Int64Array, col, i64, hashes_buffer);
-            }
-            DataType::Float32 => {
-                hash_array_primitive_float!(Float32Array, col, f32, i32, hashes_buffer);
-            }
-            DataType::Float64 => {
-                hash_array_primitive_float!(Float64Array, col, f64, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Second, _) => {
-                hash_array_primitive!(TimestampSecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Millisecond, _) => {
-                hash_array_primitive!(TimestampMillisecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Microsecond, _) => {
-                hash_array_primitive!(TimestampMicrosecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Nanosecond, _) => {
-                hash_array_primitive!(TimestampNanosecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Date32 => {
-                hash_array_primitive!(Date32Array, col, i32, hashes_buffer);
-            }
-            DataType::Date64 => {
-                hash_array_primitive!(Date64Array, col, i64, hashes_buffer);
-            }
-            DataType::Utf8 => {
-                hash_array!(StringArray, col, hashes_buffer);
-            }
-            DataType::LargeUtf8 => {
-                hash_array!(LargeStringArray, col, hashes_buffer);
-            }
-            DataType::Binary => {
-                hash_array!(BinaryArray, col, hashes_buffer);
-            }
-            DataType::LargeBinary => {
-                hash_array!(LargeBinaryArray, col, hashes_buffer);
-            }
-            DataType::FixedSizeBinary(_) => {
-                hash_array!(FixedSizeBinaryArray, col, hashes_buffer);
-            }
-            DataType::Decimal128(_, _) => {
-                hash_array_decimal!(Decimal128Array, col, hashes_buffer);
-            }
-            DataType::Dictionary(index_type, _) => match **index_type {
+///
+/// `hash_method` is the hash function to use
+/// `create_dictionary_hash_method` is the function to create hashes for dictionary arrays input

Review Comment:
   nit:
   ```suggestion
   ///
   /// `hash_method` is the hash function to use.
   /// `create_dictionary_hash_method` is the function to create hashes for dictionary arrays input.
   ```
   
   Make it easier to read.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#issuecomment-2110647933

   > Found the issue. The create_hashes_dictionary doesn't handle input hashes correctly, it affects both murmur3hash and this new xxhash64 method.
   
   > Let me try to fix that first.
   
   I have submitted the fix in this PR and waiting for CI passes. I will create a separate PR to include the murmur3 hash fix and depends on your #426 in the morning(in Beijing time) first.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600836820


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -193,27 +241,67 @@ macro_rules! hash_array_decimal {
 fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
     array: &ArrayRef,
     hashes_buffer: &mut [u32],
+    multi_col: bool,
 ) -> Result<()> {
     let dict_array = array.as_any().downcast_ref::<DictionaryArray<K>>().unwrap();
+    if multi_col {
+        // unpack the dictionary array as each row may have a different hash input
+        let unpacked = take(dict_array.values().as_ref(), dict_array.keys(), None)?;
+        create_hashes(&[unpacked], hashes_buffer)?;
+    } else {
+        // Hash each dictionary value once, and then use that computed
+        // hash for each key value to avoid a potentially expensive
+        // redundant hashing for large dictionary elements (e.g. strings)
+        let dict_values = Arc::clone(dict_array.values());
+        // same initial seed as Spark
+        let mut dict_hashes = vec![42; dict_values.len()];
+        create_hashes(&[dict_values], &mut dict_hashes)?;

Review Comment:
   Dictionary values are used to pre-compute hashes here. But the key/value orders could be different to the value orders.
   
   For example, if dictionary values are [2, 1, 4, 3, 5], dictionary keys are [1, 0, 0, 1, 2, 4, 3, 3]. We pre-compute hashes for values in the order of [2, 1, 4, 3, 5], but the unpack values are actually [1, 2, 2, 1, 4, 5, 3, 3].



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1601584368


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -193,27 +241,67 @@ macro_rules! hash_array_decimal {
 fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
     array: &ArrayRef,
     hashes_buffer: &mut [u32],
+    multi_col: bool,
 ) -> Result<()> {
     let dict_array = array.as_any().downcast_ref::<DictionaryArray<K>>().unwrap();
+    if multi_col {
+        // unpack the dictionary array as each row may have a different hash input
+        let unpacked = take(dict_array.values().as_ref(), dict_array.keys(), None)?;
+        create_hashes(&[unpacked], hashes_buffer)?;
+    } else {
+        // Hash each dictionary value once, and then use that computed
+        // hash for each key value to avoid a potentially expensive
+        // redundant hashing for large dictionary elements (e.g. strings)
+        let dict_values = Arc::clone(dict_array.values());
+        // same initial seed as Spark
+        let mut dict_hashes = vec![42; dict_values.len()];
+        create_hashes(&[dict_values], &mut dict_hashes)?;

Review Comment:
   No worries.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600671794


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -738,6 +738,17 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     castTest(generateTimestamps(), DataTypes.DateType)
   }
 
+  // todo: Refactor this
+  test("xxhash64") {
+    val input = generateStrings(timestampPattern, 8).toDF("a")
+    withTempPath { dir =>
+      val data = roundtripParquet(input, dir).coalesce(1)
+      data.createOrReplaceTempView("t")
+      val df = spark.sql(s"select a, xxhash64(a), xxhash64(1, a) from t order by a")
+      checkSparkAnswerAndOperator(df)
+    }
+  }

Review Comment:
   It could be good to also add a specific test with hard-coded values for the problematic use cases though



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600500642


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -738,6 +738,17 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     castTest(generateTimestamps(), DataTypes.DateType)
   }
 
+  // todo: Refactor this
+  test("xxhash64") {
+    val input = generateStrings(timestampPattern, 8).toDF("a")
+    withTempPath { dir =>
+      val data = roundtripParquet(input, dir).coalesce(1)
+      data.createOrReplaceTempView("t")
+      val df = spark.sql(s"select a, xxhash64(a), xxhash64(1, a) from t order by a")
+      checkSparkAnswerAndOperator(df)
+    }
+  }

Review Comment:
   Hmm, this is randomly generated, how do we make sure it definitely produces the data causing the error? Could we have a deterministic test case for the fix?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#issuecomment-2134194528

   @andygrove @viirya @parthchandra and @sunchao would you mind to take a look at this? I think it's ready for review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#issuecomment-2144160556

   Gently ping @andygrove @viirya, do you have any more comments?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1619483941


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -244,111 +283,214 @@ fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
 ///
 /// The number of rows to hash is determined by `hashes_buffer.len()`.
 /// `hashes_buffer` should be pre-sized appropriately
-pub fn create_hashes<'a>(
-    arrays: &[ArrayRef],
-    hashes_buffer: &'a mut [u32],
-) -> Result<&'a mut [u32]> {
-    for (i, col) in arrays.iter().enumerate() {
-        let first_col = i == 0;
-        match col.data_type() {
-            DataType::Boolean => {
-                hash_array_boolean!(BooleanArray, col, i32, hashes_buffer);
-            }
-            DataType::Int8 => {
-                hash_array_primitive!(Int8Array, col, i32, hashes_buffer);
-            }
-            DataType::Int16 => {
-                hash_array_primitive!(Int16Array, col, i32, hashes_buffer);
-            }
-            DataType::Int32 => {
-                hash_array_primitive!(Int32Array, col, i32, hashes_buffer);
-            }
-            DataType::Int64 => {
-                hash_array_primitive!(Int64Array, col, i64, hashes_buffer);
-            }
-            DataType::Float32 => {
-                hash_array_primitive_float!(Float32Array, col, f32, i32, hashes_buffer);
-            }
-            DataType::Float64 => {
-                hash_array_primitive_float!(Float64Array, col, f64, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Second, _) => {
-                hash_array_primitive!(TimestampSecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Millisecond, _) => {
-                hash_array_primitive!(TimestampMillisecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Microsecond, _) => {
-                hash_array_primitive!(TimestampMicrosecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Nanosecond, _) => {
-                hash_array_primitive!(TimestampNanosecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Date32 => {
-                hash_array_primitive!(Date32Array, col, i32, hashes_buffer);
-            }
-            DataType::Date64 => {
-                hash_array_primitive!(Date64Array, col, i64, hashes_buffer);
-            }
-            DataType::Utf8 => {
-                hash_array!(StringArray, col, hashes_buffer);
-            }
-            DataType::LargeUtf8 => {
-                hash_array!(LargeStringArray, col, hashes_buffer);
-            }
-            DataType::Binary => {
-                hash_array!(BinaryArray, col, hashes_buffer);
-            }
-            DataType::LargeBinary => {
-                hash_array!(LargeBinaryArray, col, hashes_buffer);
-            }
-            DataType::FixedSizeBinary(_) => {
-                hash_array!(FixedSizeBinaryArray, col, hashes_buffer);
-            }
-            DataType::Decimal128(_, _) => {
-                hash_array_decimal!(Decimal128Array, col, hashes_buffer);
-            }
-            DataType::Dictionary(index_type, _) => match **index_type {
+///
+/// `hash_method` is the hash function to use
+/// `create_dictionary_hash_method` is the function to create hashes for dictionary arrays input

Review Comment:
   nit:
   ```suggestion
   ///
   /// `hash_method` is the hash function to use.
   /// `create_dictionary_hash_method` is the function to create hashes for dictionary arrays input.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1622930478


##########
spark/inspections/CometTPCHQueriesList-results.txt:
##########
@@ -1,142 +1,129 @@
 Query: q1 TPCH Snappy. Comet Exec: Enabled (CometHashAggregate, CometFilter, CometProject)
 Query: q1 TPCH Snappy: ExplainInfo:
-Shuffle: unsupported Spark partitioning: org.apache.spark.sql.catalyst.plans.physical.RangePartitioning
+Comet shuffle is not enabled: spark.sql.adaptive.coalescePartitions.enabled is enabled and spark.comet.shuffle.enforceMode.enabled is not enabled

Review Comment:
   @advancedxy, here you go - https://github.com/parthchandra/datafusion-comet/commit/95443622c225b7d955bba6261471eed5f82b92b7.patch



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#issuecomment-2110428794

   I'd like to see the tests use some randomly generated inputs.
   
   As a quick hack, I added the following test to `CometCastSuite` and it shows some differences in results between Spark and Comet.
   
   ```scala
     test("xxhash64") {
       val input = generateStrings(timestampPattern, 8).toDF("a")
       withTempPath { dir =>
         val data = roundtripParquet(input, dir).coalesce(1)
         data.createOrReplaceTempView("t")
         val df = spark.sql(s"select a, xxhash64(a) from t order by a")
         checkSparkAnswerAndOperator(df)
       }
     }
   ```
   
   Some differences:
   
   ```
   !== Correct Answer - 1000 ==           == Spark Answer - 1000 ==
    struct<a:string,xxhash64(a):bigint>   struct<a:string,xxhash64(a):bigint>
   ![,-7444071767201028348]               [,-1205034819632174695]
   ![	 23,-1992628079781282865]           [	 23,4312780814362028915]
   ![	31T3,5857608402363468958]           [	31T3,6089516869931970265]
   ```
   
   We could extract the `generate*` methods from `CometCastSuite` into a separate `DataGenerator` class that other test suites can leverage. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600824499


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -104,66 +108,110 @@ fn test_murmur3() {
     let _expected = vec![
         142593372, 1485273170, -97053317, 1322437556, -396302900, 814637928,
     ];
+    assert_eq!(_hashes, _expected)

Review Comment:
   Let me move this test case to the end of this file, which has all the test code.



##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -104,66 +108,110 @@ fn test_murmur3() {
     let _expected = vec![
         142593372, 1485273170, -97053317, 1322437556, -396302900, 814637928,
     ];
+    assert_eq!(_hashes, _expected)
+}
+
+#[inline]
+pub(crate) fn spark_compatible_xxhash64<T: AsRef<[u8]>>(data: T, seed: u64) -> u64 {
+    // TODO: Rewrite with a stateless hasher to reduce stack allocation?
+    let mut hasher = XxHash64::with_seed(seed);
+    hasher.write(data.as_ref());
+    hasher.finish()
+}
+
+#[test]
+fn test_xxhash64() {
+    let _hashes = ["", "a", "ab", "abc", "abcd", "abcde"]
+        .into_iter()
+        .map(|s| spark_compatible_xxhash64(s.as_bytes(), 42) as i64)
+        .collect::<Vec<_>>();
+    let _expected = vec![
+        -7444071767201028348,
+        -8582455328737087284,
+        2710560539726725091,
+        1423657621850124518,
+        -6810745876291105281,
+        -990457398947679591,
+    ];
+    assert_eq!(_hashes, _expected);
 }
 
 macro_rules! hash_array {
-    ($array_type:ident, $column: ident, $hashes: ident) => {
+    ($array_type: ident, $column: ident, $hashes: ident, $hash_method: ident) => {
         let array = $column.as_any().downcast_ref::<$array_type>().unwrap();
         if array.null_count() == 0 {
             for (i, hash) in $hashes.iter_mut().enumerate() {
-                *hash = spark_compatible_murmur3_hash(&array.value(i), *hash);
+                *hash = $hash_method(&array.value(i), *hash);
             }
         } else {
             for (i, hash) in $hashes.iter_mut().enumerate() {
                 if !array.is_null(i) {
-                    *hash = spark_compatible_murmur3_hash(&array.value(i), *hash);
+                    *hash = $hash_method(&array.value(i), *hash);
                 }
             }
         }
     };
 }
 
 macro_rules! hash_array_primitive {
-    ($array_type:ident, $column: ident, $ty: ident, $hashes: ident) => {
+    ($array_type: ident, $column: ident, $ty: ident, $hashes: ident, $hash_method: ident) => {
         let array = $column.as_any().downcast_ref::<$array_type>().unwrap();
         let values = array.values();
 
         if array.null_count() == 0 {
             for (hash, value) in $hashes.iter_mut().zip(values.iter()) {
-                *hash = spark_compatible_murmur3_hash((*value as $ty).to_le_bytes(), *hash);
+                *hash = $hash_method((*value as $ty).to_le_bytes(), *hash);
             }
         } else {
             for (i, (hash, value)) in $hashes.iter_mut().zip(values.iter()).enumerate() {
                 if !array.is_null(i) {
-                    *hash = spark_compatible_murmur3_hash((*value as $ty).to_le_bytes(), *hash);
+                    *hash = $hash_method((*value as $ty).to_le_bytes(), *hash);
+                }
+            }
+        }
+    };
+}
+
+macro_rules! hash_array_primitive_boolean {

Review Comment:
   ok.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#issuecomment-2110516813

   See https://github.com/apache/datafusion-comet/pull/426 for proposed DataGenerator class


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600898381


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -193,27 +241,67 @@ macro_rules! hash_array_decimal {
 fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
     array: &ArrayRef,
     hashes_buffer: &mut [u32],
+    multi_col: bool,
 ) -> Result<()> {
     let dict_array = array.as_any().downcast_ref::<DictionaryArray<K>>().unwrap();
+    if multi_col {
+        // unpack the dictionary array as each row may have a different hash input
+        let unpacked = take(dict_array.values().as_ref(), dict_array.keys(), None)?;
+        create_hashes(&[unpacked], hashes_buffer)?;
+    } else {
+        // Hash each dictionary value once, and then use that computed
+        // hash for each key value to avoid a potentially expensive
+        // redundant hashing for large dictionary elements (e.g. strings)
+        let dict_values = Arc::clone(dict_array.values());
+        // same initial seed as Spark
+        let mut dict_hashes = vec![42; dict_values.len()];
+        create_hashes(&[dict_values], &mut dict_hashes)?;

Review Comment:
   Actually I wrote this part of code last year. I forgot most of the details. 😅 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600892577


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -193,27 +241,67 @@ macro_rules! hash_array_decimal {
 fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
     array: &ArrayRef,
     hashes_buffer: &mut [u32],
+    multi_col: bool,
 ) -> Result<()> {
     let dict_array = array.as_any().downcast_ref::<DictionaryArray<K>>().unwrap();
+    if multi_col {
+        // unpack the dictionary array as each row may have a different hash input
+        let unpacked = take(dict_array.values().as_ref(), dict_array.keys(), None)?;
+        create_hashes(&[unpacked], hashes_buffer)?;
+    } else {
+        // Hash each dictionary value once, and then use that computed
+        // hash for each key value to avoid a potentially expensive
+        // redundant hashing for large dictionary elements (e.g. strings)
+        let dict_values = Arc::clone(dict_array.values());
+        // same initial seed as Spark
+        let mut dict_hashes = vec![42; dict_values.len()];
+        create_hashes(&[dict_values], &mut dict_hashes)?;

Review Comment:
   > so
   ```
   row_id = 0, dict_key = 1, dict_hashes[1] = hash(1, dict_hashes[0]) but it should be hash(1, init_seed)
   row_id = 1, dict_key = 0, dict_hashes[0] = hash(2, init_seed) but it should be hash(2, hash(1, init_seed))
   row_id = 2, dict_key = 0 dict_hashes[0] = hash(2, init_seed) but it should be hash(2, hash(2, hash(1, init_seed)))
   ...
   ```
   
   Hmm, I think you may have confused with row order and column order, which indeed is quite confusing.
   
   For function call like `hash(col_a, col_b)`, the actual hash is `murmur3(col_b, murmur3(col_a, 42))` where 42 is the initial seed, the previous hash is feed as seed, the order is determined the **input cols**(parameters), but not related to the _row order_.
   
   For rows in the RecordBatch/ArrowArray, each hash is computed from scratch, there's no dep between values in different rows.
   
   So the following compute step is not how it currently work
   > dict_hashes[0] = hash(2, init_seed)
   dict_hashes[1] = hash(1, dict_hashes[0])
   dict_hashes[2] = hash(4, dict_hashes[1]) 
   dict_hashes[3] = hash(3, dict_hashes[2]) 
   dict_hashes[4] = hash(5, dict_hashes[3]) 
   
   Instead, currently it works as
   ```
   dict_hashes[0] = hash(2, init_seed)
   dict_hashes[1] = hash(1, init_seed)
   dict_hashes[2] = hash(4, init_seed) 
   dict_hashes[3] = hash(3, init_seed) 
   dict_hashes[4] = hash(5, init_seed) 
   ```
   
   The hashes of the first col are then feed as seed for the next cols in the rows.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#issuecomment-2110446840

   > I'd like to see the tests use some randomly generated inputs.
   > 
   > As a quick hack, I added the following test to `CometCastSuite` and it shows some differences in results between Spark and Comet.
   > 
   > ```scala
   >   test("xxhash64") {
   >     val input = generateStrings(timestampPattern, 8).toDF("a")
   >     withTempPath { dir =>
   >       val data = roundtripParquet(input, dir).coalesce(1)
   >       data.createOrReplaceTempView("t")
   >       val df = spark.sql(s"select a, xxhash64(a) from t order by a")
   >       checkSparkAnswerAndOperator(df)
   >     }
   >   }
   > ```
   > 
   > Some differences:
   > 
   > ```
   > !== Correct Answer - 1000 ==           == Spark Answer - 1000 ==
   >  struct<a:string,xxhash64(a):bigint>   struct<a:string,xxhash64(a):bigint>
   > ![,-7444071767201028348]               [,-1205034819632174695]
   > ![	 23,-1992628079781282865]           [	 23,4312780814362028915]
   > ![	31T3,5857608402363468958]           [	31T3,6089516869931970265]
   > ```
   > 
   > We could extract the `generate*` methods from `CometCastSuite` into a separate `DataGenerator` class that other test suites can leverage.
   
   Good catch, and a good way to make sure the impl is correct. Let me check why the test is failing first. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600856424


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -193,27 +241,67 @@ macro_rules! hash_array_decimal {
 fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
     array: &ArrayRef,
     hashes_buffer: &mut [u32],
+    multi_col: bool,
 ) -> Result<()> {
     let dict_array = array.as_any().downcast_ref::<DictionaryArray<K>>().unwrap();
+    if multi_col {
+        // unpack the dictionary array as each row may have a different hash input
+        let unpacked = take(dict_array.values().as_ref(), dict_array.keys(), None)?;
+        create_hashes(&[unpacked], hashes_buffer)?;
+    } else {
+        // Hash each dictionary value once, and then use that computed
+        // hash for each key value to avoid a potentially expensive
+        // redundant hashing for large dictionary elements (e.g. strings)
+        let dict_values = Arc::clone(dict_array.values());
+        // same initial seed as Spark
+        let mut dict_hashes = vec![42; dict_values.len()];
+        create_hashes(&[dict_values], &mut dict_hashes)?;

Review Comment:
   > We pre-compute hashes for values in the order of [2, 1, 4, 3, 5], but the unpack values are actually [1, 2, 2, 1, 4, 5, 3, 3].
   
   yes, the hashes of dict array are  `dict_hashes = [hash(2), hash(1), hash(4), hash(3), hash5)]`. When computing hashes for this 
   dictionary array, the actual hashes are retrieved from dict keys to idx, which should be:
   
   | row_id | dict_key_idx | actual_hash |
   |--------|--------|--------|
   | 0 | 1 | dict_hashes[1] = `hash(1)` |
   | 1 | 0| dict_hashes[0] = `hash(2)`|
   | 2 | 0 | dict_hashes[0] = `hash(2)`|
   | 3 | 1 | dict_hashes[1] = `hash(1)`| 
   | 4 | 2 | dict_hashes[2] = `hash(4)`|
   |...|...|...|
   
   It should matches the actual value order.



##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -193,27 +241,67 @@ macro_rules! hash_array_decimal {
 fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
     array: &ArrayRef,
     hashes_buffer: &mut [u32],
+    multi_col: bool,
 ) -> Result<()> {
     let dict_array = array.as_any().downcast_ref::<DictionaryArray<K>>().unwrap();
+    if multi_col {
+        // unpack the dictionary array as each row may have a different hash input
+        let unpacked = take(dict_array.values().as_ref(), dict_array.keys(), None)?;
+        create_hashes(&[unpacked], hashes_buffer)?;
+    } else {
+        // Hash each dictionary value once, and then use that computed
+        // hash for each key value to avoid a potentially expensive
+        // redundant hashing for large dictionary elements (e.g. strings)
+        let dict_values = Arc::clone(dict_array.values());
+        // same initial seed as Spark
+        let mut dict_hashes = vec![42; dict_values.len()];
+        create_hashes(&[dict_values], &mut dict_hashes)?;

Review Comment:
   > We pre-compute hashes for values in the order of [2, 1, 4, 3, 5], but the unpack values are actually [1, 2, 2, 1, 4, 5, 3, 3].
   
   yes, the hashes of dict array are  `dict_hashes = [hash(2), hash(1), hash(4), hash(3), hash5)]`. When computing hashes for this 
   dictionary array, the actual hashes are retrieved from dict keys to idx, which should be:
   
   | row_id | dict_key_idx | actual_hash |
   |--------|--------|--------|
   | 0 | 1 | dict_hashes[1] = `hash(1)` |
   | 1 | 0| dict_hashes[0] = `hash(2)`|
   | 2 | 0 | dict_hashes[0] = `hash(2)`|
   | 3 | 1 | dict_hashes[1] = `hash(1)`| 
   | 4 | 2 | dict_hashes[2] = `hash(4)`|
   |...|...|...|
   
   It should match the actual value order.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#issuecomment-2112436642

   @andygrove  @viirya I have created #433 and mark this as a draft. We should merge that first and then come back to this PR . PLAL when you have tome.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600515535


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -193,27 +241,67 @@ macro_rules! hash_array_decimal {
 fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
     array: &ArrayRef,
     hashes_buffer: &mut [u32],
+    multi_col: bool,
 ) -> Result<()> {
     let dict_array = array.as_any().downcast_ref::<DictionaryArray<K>>().unwrap();
+    if multi_col {
+        // unpack the dictionary array as each row may have a different hash input
+        let unpacked = take(dict_array.values().as_ref(), dict_array.keys(), None)?;
+        create_hashes(&[unpacked], hashes_buffer)?;
+    } else {
+        // Hash each dictionary value once, and then use that computed
+        // hash for each key value to avoid a potentially expensive
+        // redundant hashing for large dictionary elements (e.g. strings)
+        let dict_values = Arc::clone(dict_array.values());
+        // same initial seed as Spark
+        let mut dict_hashes = vec![42; dict_values.len()];
+        create_hashes(&[dict_values], &mut dict_hashes)?;

Review Comment:
   Hmm, then I think this approach is actually incorrect, even for first column. We cannot pre-compute hashes for values. The key/value order is not the same as value order.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#issuecomment-2110523513

   > Our `hash` implementation is also not compatible with Spark. I will file an issue for that.
   
   I filed https://github.com/apache/datafusion-comet/issues/427


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "parthchandra (via GitHub)" <gi...@apache.org>.
parthchandra commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1621538973


##########
spark/inspections/CometTPCHQueriesList-results.txt:
##########
@@ -1,142 +1,129 @@
 Query: q1 TPCH Snappy. Comet Exec: Enabled (CometHashAggregate, CometFilter, CometProject)
 Query: q1 TPCH Snappy: ExplainInfo:
-Shuffle: unsupported Spark partitioning: org.apache.spark.sql.catalyst.plans.physical.RangePartitioning
+Comet shuffle is not enabled: spark.sql.adaptive.coalescePartitions.enabled is enabled and spark.comet.shuffle.enforceMode.enabled is not enabled

Review Comment:
   The changes to explain info occur because Spark generates different plans at different scales (especially wrt to broadcast). 
   Let me run with this PR and get an updated version. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1620141162


##########
core/src/execution/datafusion/expressions/scalar_funcs.rs:
##########
@@ -672,6 +676,49 @@ fn spark_murmur3_hash(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusio
     }
 }
 
+fn spark_xxhash64(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {
+    let length = args.len();
+    let seed = &args[length - 1];
+    match seed {
+        ColumnarValue::Scalar(ScalarValue::Int64(Some(seed))) => {
+            // iterate over the arguments to find out the length of the array
+            let num_rows = args[0..args.len() - 1]
+                .iter()
+                .find_map(|arg| match arg {
+                    ColumnarValue::Array(array) => Some(array.len()),
+                    ColumnarValue::Scalar(_) => None,
+                })
+                .unwrap_or(1);
+            let mut hashes: Vec<u64> = vec![0_u64; num_rows];
+            hashes.fill(*seed as u64);
+            let arrays = args[0..args.len() - 1]
+                .iter()
+                .map(|arg| match arg {
+                    ColumnarValue::Array(array) => array.clone(),
+                    ColumnarValue::Scalar(scalar) => {
+                        scalar.clone().to_array_of_size(num_rows).unwrap()
+                    }
+                })
+                .collect::<Vec<ArrayRef>>();

Review Comment:
   hmm. I think we have to compute `num_rows` first?



##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -244,111 +283,214 @@ fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
 ///
 /// The number of rows to hash is determined by `hashes_buffer.len()`.
 /// `hashes_buffer` should be pre-sized appropriately
-pub fn create_hashes<'a>(
-    arrays: &[ArrayRef],
-    hashes_buffer: &'a mut [u32],
-) -> Result<&'a mut [u32]> {
-    for (i, col) in arrays.iter().enumerate() {
-        let first_col = i == 0;
-        match col.data_type() {
-            DataType::Boolean => {
-                hash_array_boolean!(BooleanArray, col, i32, hashes_buffer);
-            }
-            DataType::Int8 => {
-                hash_array_primitive!(Int8Array, col, i32, hashes_buffer);
-            }
-            DataType::Int16 => {
-                hash_array_primitive!(Int16Array, col, i32, hashes_buffer);
-            }
-            DataType::Int32 => {
-                hash_array_primitive!(Int32Array, col, i32, hashes_buffer);
-            }
-            DataType::Int64 => {
-                hash_array_primitive!(Int64Array, col, i64, hashes_buffer);
-            }
-            DataType::Float32 => {
-                hash_array_primitive_float!(Float32Array, col, f32, i32, hashes_buffer);
-            }
-            DataType::Float64 => {
-                hash_array_primitive_float!(Float64Array, col, f64, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Second, _) => {
-                hash_array_primitive!(TimestampSecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Millisecond, _) => {
-                hash_array_primitive!(TimestampMillisecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Microsecond, _) => {
-                hash_array_primitive!(TimestampMicrosecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Nanosecond, _) => {
-                hash_array_primitive!(TimestampNanosecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Date32 => {
-                hash_array_primitive!(Date32Array, col, i32, hashes_buffer);
-            }
-            DataType::Date64 => {
-                hash_array_primitive!(Date64Array, col, i64, hashes_buffer);
-            }
-            DataType::Utf8 => {
-                hash_array!(StringArray, col, hashes_buffer);
-            }
-            DataType::LargeUtf8 => {
-                hash_array!(LargeStringArray, col, hashes_buffer);
-            }
-            DataType::Binary => {
-                hash_array!(BinaryArray, col, hashes_buffer);
-            }
-            DataType::LargeBinary => {
-                hash_array!(LargeBinaryArray, col, hashes_buffer);
-            }
-            DataType::FixedSizeBinary(_) => {
-                hash_array!(FixedSizeBinaryArray, col, hashes_buffer);
-            }
-            DataType::Decimal128(_, _) => {
-                hash_array_decimal!(Decimal128Array, col, hashes_buffer);
-            }
-            DataType::Dictionary(index_type, _) => match **index_type {
+///
+/// `hash_method` is the hash function to use.
+/// `create_dictionary_hash_method` is the function to create hashes for dictionary arrays input.
+macro_rules! create_hashes_internal {
+    ($arrays: ident, $hashes_buffer: ident, $hash_method: ident, $create_dictionary_hash_method: ident) => {
+        for (i, col) in $arrays.iter().enumerate() {
+            let first_col = i == 0;
+            match col.data_type() {
+                DataType::Boolean => {
+                    hash_array_boolean!(BooleanArray, col, i32, $hashes_buffer, $hash_method);
+                }

Review Comment:
   hmm, let me give it a try. I will report back if it's too hard to do that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "kazuyukitanimura (via GitHub)" <gi...@apache.org>.
kazuyukitanimura commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1619940100


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -244,111 +283,214 @@ fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
 ///
 /// The number of rows to hash is determined by `hashes_buffer.len()`.
 /// `hashes_buffer` should be pre-sized appropriately
-pub fn create_hashes<'a>(
-    arrays: &[ArrayRef],
-    hashes_buffer: &'a mut [u32],
-) -> Result<&'a mut [u32]> {
-    for (i, col) in arrays.iter().enumerate() {
-        let first_col = i == 0;
-        match col.data_type() {
-            DataType::Boolean => {
-                hash_array_boolean!(BooleanArray, col, i32, hashes_buffer);
-            }
-            DataType::Int8 => {
-                hash_array_primitive!(Int8Array, col, i32, hashes_buffer);
-            }
-            DataType::Int16 => {
-                hash_array_primitive!(Int16Array, col, i32, hashes_buffer);
-            }
-            DataType::Int32 => {
-                hash_array_primitive!(Int32Array, col, i32, hashes_buffer);
-            }
-            DataType::Int64 => {
-                hash_array_primitive!(Int64Array, col, i64, hashes_buffer);
-            }
-            DataType::Float32 => {
-                hash_array_primitive_float!(Float32Array, col, f32, i32, hashes_buffer);
-            }
-            DataType::Float64 => {
-                hash_array_primitive_float!(Float64Array, col, f64, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Second, _) => {
-                hash_array_primitive!(TimestampSecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Millisecond, _) => {
-                hash_array_primitive!(TimestampMillisecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Microsecond, _) => {
-                hash_array_primitive!(TimestampMicrosecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Nanosecond, _) => {
-                hash_array_primitive!(TimestampNanosecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Date32 => {
-                hash_array_primitive!(Date32Array, col, i32, hashes_buffer);
-            }
-            DataType::Date64 => {
-                hash_array_primitive!(Date64Array, col, i64, hashes_buffer);
-            }
-            DataType::Utf8 => {
-                hash_array!(StringArray, col, hashes_buffer);
-            }
-            DataType::LargeUtf8 => {
-                hash_array!(LargeStringArray, col, hashes_buffer);
-            }
-            DataType::Binary => {
-                hash_array!(BinaryArray, col, hashes_buffer);
-            }
-            DataType::LargeBinary => {
-                hash_array!(LargeBinaryArray, col, hashes_buffer);
-            }
-            DataType::FixedSizeBinary(_) => {
-                hash_array!(FixedSizeBinaryArray, col, hashes_buffer);
-            }
-            DataType::Decimal128(_, _) => {
-                hash_array_decimal!(Decimal128Array, col, hashes_buffer);
-            }
-            DataType::Dictionary(index_type, _) => match **index_type {
+///
+/// `hash_method` is the hash function to use.
+/// `create_dictionary_hash_method` is the function to create hashes for dictionary arrays input.
+macro_rules! create_hashes_internal {
+    ($arrays: ident, $hashes_buffer: ident, $hash_method: ident, $create_dictionary_hash_method: ident) => {
+        for (i, col) in $arrays.iter().enumerate() {
+            let first_col = i == 0;
+            match col.data_type() {
+                DataType::Boolean => {
+                    hash_array_boolean!(BooleanArray, col, i32, $hashes_buffer, $hash_method);
+                }

Review Comment:
   nit: I wonder if we can make `BooleanArray` and `i32` as macro argument, so that we can reduce this large case match...



##########
core/src/execution/datafusion/expressions/scalar_funcs.rs:
##########
@@ -672,6 +676,49 @@ fn spark_murmur3_hash(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusio
     }
 }
 
+fn spark_xxhash64(args: &[ColumnarValue]) -> Result<ColumnarValue, DataFusionError> {
+    let length = args.len();
+    let seed = &args[length - 1];
+    match seed {
+        ColumnarValue::Scalar(ScalarValue::Int64(Some(seed))) => {
+            // iterate over the arguments to find out the length of the array
+            let num_rows = args[0..args.len() - 1]
+                .iter()
+                .find_map(|arg| match arg {
+                    ColumnarValue::Array(array) => Some(array.len()),
+                    ColumnarValue::Scalar(_) => None,
+                })
+                .unwrap_or(1);
+            let mut hashes: Vec<u64> = vec![0_u64; num_rows];
+            hashes.fill(*seed as u64);
+            let arrays = args[0..args.len() - 1]
+                .iter()
+                .map(|arg| match arg {
+                    ColumnarValue::Array(array) => array.clone(),
+                    ColumnarValue::Scalar(scalar) => {
+                        scalar.clone().to_array_of_size(num_rows).unwrap()
+                    }
+                })
+                .collect::<Vec<ArrayRef>>();

Review Comment:
   nit: I feel this can be simplified a little bit
   ```
   let arrays = args[0..args.len() - 1]
      ...;
   let mut hashes: Vec<u64> = vec![0_u64; arrays.len()];
   hashes.fill(*seed as u64);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1619670208


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -244,111 +283,214 @@ fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
 ///
 /// The number of rows to hash is determined by `hashes_buffer.len()`.
 /// `hashes_buffer` should be pre-sized appropriately
-pub fn create_hashes<'a>(
-    arrays: &[ArrayRef],
-    hashes_buffer: &'a mut [u32],
-) -> Result<&'a mut [u32]> {
-    for (i, col) in arrays.iter().enumerate() {
-        let first_col = i == 0;
-        match col.data_type() {
-            DataType::Boolean => {
-                hash_array_boolean!(BooleanArray, col, i32, hashes_buffer);
-            }
-            DataType::Int8 => {
-                hash_array_primitive!(Int8Array, col, i32, hashes_buffer);
-            }
-            DataType::Int16 => {
-                hash_array_primitive!(Int16Array, col, i32, hashes_buffer);
-            }
-            DataType::Int32 => {
-                hash_array_primitive!(Int32Array, col, i32, hashes_buffer);
-            }
-            DataType::Int64 => {
-                hash_array_primitive!(Int64Array, col, i64, hashes_buffer);
-            }
-            DataType::Float32 => {
-                hash_array_primitive_float!(Float32Array, col, f32, i32, hashes_buffer);
-            }
-            DataType::Float64 => {
-                hash_array_primitive_float!(Float64Array, col, f64, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Second, _) => {
-                hash_array_primitive!(TimestampSecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Millisecond, _) => {
-                hash_array_primitive!(TimestampMillisecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Microsecond, _) => {
-                hash_array_primitive!(TimestampMicrosecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Nanosecond, _) => {
-                hash_array_primitive!(TimestampNanosecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Date32 => {
-                hash_array_primitive!(Date32Array, col, i32, hashes_buffer);
-            }
-            DataType::Date64 => {
-                hash_array_primitive!(Date64Array, col, i64, hashes_buffer);
-            }
-            DataType::Utf8 => {
-                hash_array!(StringArray, col, hashes_buffer);
-            }
-            DataType::LargeUtf8 => {
-                hash_array!(LargeStringArray, col, hashes_buffer);
-            }
-            DataType::Binary => {
-                hash_array!(BinaryArray, col, hashes_buffer);
-            }
-            DataType::LargeBinary => {
-                hash_array!(LargeBinaryArray, col, hashes_buffer);
-            }
-            DataType::FixedSizeBinary(_) => {
-                hash_array!(FixedSizeBinaryArray, col, hashes_buffer);
-            }
-            DataType::Decimal128(_, _) => {
-                hash_array_decimal!(Decimal128Array, col, hashes_buffer);
-            }
-            DataType::Dictionary(index_type, _) => match **index_type {
+///
+/// `hash_method` is the hash function to use
+/// `create_dictionary_hash_method` is the function to create hashes for dictionary arrays input

Review Comment:
   Thanks, 🥰.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#issuecomment-2146455431

   Thanks all for reviewing, @


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600867066


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -193,27 +241,67 @@ macro_rules! hash_array_decimal {
 fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
     array: &ArrayRef,
     hashes_buffer: &mut [u32],
+    multi_col: bool,
 ) -> Result<()> {
     let dict_array = array.as_any().downcast_ref::<DictionaryArray<K>>().unwrap();
+    if multi_col {
+        // unpack the dictionary array as each row may have a different hash input
+        let unpacked = take(dict_array.values().as_ref(), dict_array.keys(), None)?;
+        create_hashes(&[unpacked], hashes_buffer)?;
+    } else {
+        // Hash each dictionary value once, and then use that computed
+        // hash for each key value to avoid a potentially expensive
+        // redundant hashing for large dictionary elements (e.g. strings)
+        let dict_values = Arc::clone(dict_array.values());
+        // same initial seed as Spark
+        let mut dict_hashes = vec![42; dict_values.len()];
+        create_hashes(&[dict_values], &mut dict_hashes)?;

Review Comment:
   But when we compute hashes, next hash is computed based on current value and  previous hash value. But when we compute hashes for dictionary values, we compute it based on "dictionary values" order, instead of the row value order.
   
   In other words, we compute the hashes of dictionary values like:
   
   ```
   dict_hashes[0] = hash(2, init_seed)
   dict_hashes[1] = hash(1, dict_hashes[0])
   dict_hashes[2] = hash(4, dict_hashes[1]) 
   dict_hashes[3] = hash(3, dict_hashes[2]) 
   dict_hashes[4] = hash(5, dict_hashes[3]) 
   ```
   
   So,
   
   ```
   row_id = 0, dict_key = 1, dict_hashes[1] = hash(1, dict_hashes[0]) but it should be hash(1, init_seed)
   row_id = 1, dict_key = 0, dict_hashes[0] = hash(2, init_seed) but it should be hash(2, hash(1, init_seed))
   row_id = 2, dict_key = ,0 dict_hashes[0] = hash(2, init_seed) but it should be hash(2, hash(2, hash(1, init_seed)))
   ...
   ```
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#issuecomment-2110508945

   > Let me check why the test is failing first.
   
   Found the issue. The `create_hashes_dictionary` doesn't handle input hashes correctly, it affects both murmur3hash and this new xxhash64 method. 
   
   Let me try to fix that first.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600671478


##########
spark/src/test/scala/org/apache/comet/CometCastSuite.scala:
##########
@@ -738,6 +738,17 @@ class CometCastSuite extends CometTestBase with AdaptiveSparkPlanHelper {
     castTest(generateTimestamps(), DataTypes.DateType)
   }
 
+  // todo: Refactor this
+  test("xxhash64") {
+    val input = generateStrings(timestampPattern, 8).toDF("a")
+    withTempPath { dir =>
+      val data = roundtripParquet(input, dir).coalesce(1)
+      data.createOrReplaceTempView("t")
+      val df = spark.sql(s"select a, xxhash64(a), xxhash64(1, a) from t order by a")
+      checkSparkAnswerAndOperator(df)
+    }
+  }

Review Comment:
   We are using a fixed seed for the data generation, and a single partition, so this is deterministic



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600503628


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -238,55 +241,67 @@ macro_rules! hash_array_decimal {
 fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
     array: &ArrayRef,
     hashes_buffer: &mut [u32],
+    multi_col: bool,
 ) -> Result<()> {
     let dict_array = array.as_any().downcast_ref::<DictionaryArray<K>>().unwrap();
-
-    // Hash each dictionary value once, and then use that computed
-    // hash for each key value to avoid a potentially expensive
-    // redundant hashing for large dictionary elements (e.g. strings)
-    let dict_values = Arc::clone(dict_array.values());
-    let mut dict_hashes = vec![0; dict_values.len()];
-    create_hashes(&[dict_values], &mut dict_hashes)?;
-
-    for (hash, key) in hashes_buffer.iter_mut().zip(dict_array.keys().iter()) {
-        if let Some(key) = key {
-            let idx = key.to_usize().ok_or_else(|| {
-                DataFusionError::Internal(format!(
-                    "Can not convert key value {:?} to usize in dictionary of type {:?}",
-                    key,
-                    dict_array.data_type()
-                ))
-            })?;
-            *hash = dict_hashes[idx]
-        } // no update for Null, consistent with other hashes
+    if multi_col {
+        // unpack the dictionary array as each row may have a different hash input
+        let unpacked = take(dict_array.values().as_ref(), dict_array.keys(), None)?;
+        create_hashes(&[unpacked], hashes_buffer)?;
+    } else {

Review Comment:
   `multi_col` is true for columns after first column. Why we need to unpack the dictionary array for them particularly?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600823779


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -193,27 +241,67 @@ macro_rules! hash_array_decimal {
 fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
     array: &ArrayRef,
     hashes_buffer: &mut [u32],
+    multi_col: bool,
 ) -> Result<()> {
     let dict_array = array.as_any().downcast_ref::<DictionaryArray<K>>().unwrap();
+    if multi_col {
+        // unpack the dictionary array as each row may have a different hash input
+        let unpacked = take(dict_array.values().as_ref(), dict_array.keys(), None)?;
+        create_hashes(&[unpacked], hashes_buffer)?;
+    } else {
+        // Hash each dictionary value once, and then use that computed
+        // hash for each key value to avoid a potentially expensive
+        // redundant hashing for large dictionary elements (e.g. strings)
+        let dict_values = Arc::clone(dict_array.values());
+        // same initial seed as Spark
+        let mut dict_hashes = vec![42; dict_values.len()];
+        create_hashes(&[dict_values], &mut dict_hashes)?;

Review Comment:
   Hmm, I think we can pre-compute hashes for the first dictionary column values?
   
   The `dict_array.keys()` should have the same order of first col in the rows, the index of dict keys is used to retrieve the actual hashes then?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#issuecomment-2110324971

   > Could you also update docs/source/user-guide/expressions.md to add xxhash64 as a supported expression?
   
   Of course, I will update that among other things: such as the review comments and the inspection file: `spark/inspections/CometTPCDSQueriesList-results.txt` in a followup commit.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600518058


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -104,66 +108,110 @@ fn test_murmur3() {
     let _expected = vec![
         142593372, 1485273170, -97053317, 1322437556, -396302900, 814637928,
     ];
+    assert_eq!(_hashes, _expected)

Review Comment:
   We can remove `_` from `_hashes` and `_expected`. I am not sure why we don't compare them before, seems to disable the test so they are prefixed with `_`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1623199710


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -244,111 +283,214 @@ fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
 ///
 /// The number of rows to hash is determined by `hashes_buffer.len()`.
 /// `hashes_buffer` should be pre-sized appropriately
-pub fn create_hashes<'a>(
-    arrays: &[ArrayRef],
-    hashes_buffer: &'a mut [u32],
-) -> Result<&'a mut [u32]> {
-    for (i, col) in arrays.iter().enumerate() {
-        let first_col = i == 0;
-        match col.data_type() {
-            DataType::Boolean => {
-                hash_array_boolean!(BooleanArray, col, i32, hashes_buffer);
-            }
-            DataType::Int8 => {
-                hash_array_primitive!(Int8Array, col, i32, hashes_buffer);
-            }
-            DataType::Int16 => {
-                hash_array_primitive!(Int16Array, col, i32, hashes_buffer);
-            }
-            DataType::Int32 => {
-                hash_array_primitive!(Int32Array, col, i32, hashes_buffer);
-            }
-            DataType::Int64 => {
-                hash_array_primitive!(Int64Array, col, i64, hashes_buffer);
-            }
-            DataType::Float32 => {
-                hash_array_primitive_float!(Float32Array, col, f32, i32, hashes_buffer);
-            }
-            DataType::Float64 => {
-                hash_array_primitive_float!(Float64Array, col, f64, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Second, _) => {
-                hash_array_primitive!(TimestampSecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Millisecond, _) => {
-                hash_array_primitive!(TimestampMillisecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Microsecond, _) => {
-                hash_array_primitive!(TimestampMicrosecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Timestamp(TimeUnit::Nanosecond, _) => {
-                hash_array_primitive!(TimestampNanosecondArray, col, i64, hashes_buffer);
-            }
-            DataType::Date32 => {
-                hash_array_primitive!(Date32Array, col, i32, hashes_buffer);
-            }
-            DataType::Date64 => {
-                hash_array_primitive!(Date64Array, col, i64, hashes_buffer);
-            }
-            DataType::Utf8 => {
-                hash_array!(StringArray, col, hashes_buffer);
-            }
-            DataType::LargeUtf8 => {
-                hash_array!(LargeStringArray, col, hashes_buffer);
-            }
-            DataType::Binary => {
-                hash_array!(BinaryArray, col, hashes_buffer);
-            }
-            DataType::LargeBinary => {
-                hash_array!(LargeBinaryArray, col, hashes_buffer);
-            }
-            DataType::FixedSizeBinary(_) => {
-                hash_array!(FixedSizeBinaryArray, col, hashes_buffer);
-            }
-            DataType::Decimal128(_, _) => {
-                hash_array_decimal!(Decimal128Array, col, hashes_buffer);
-            }
-            DataType::Dictionary(index_type, _) => match **index_type {
+///
+/// `hash_method` is the hash function to use.
+/// `create_dictionary_hash_method` is the function to create hashes for dictionary arrays input.
+macro_rules! create_hashes_internal {
+    ($arrays: ident, $hashes_buffer: ident, $hash_method: ident, $create_dictionary_hash_method: ident) => {
+        for (i, col) in $arrays.iter().enumerate() {
+            let first_col = i == 0;
+            match col.data_type() {
+                DataType::Boolean => {
+                    hash_array_boolean!(BooleanArray, col, i32, $hashes_buffer, $hash_method);
+                }

Review Comment:
   If I understands your proposal correctly, do you mean something like: 
   ``` rust
       match col.data_type() { 
           DataType::Int8 | DataType::Int16: | DataType::Int32 | DataType::Int64 | DataType::UInt8 | DataType::UInt16 | DataType::UInt32 | DataType::UInt64 => {
               hash_array_primitive!(get_array_type_of!(col.data_type()), col, get_input_native_type_of!(col.data_type()), $hashes_buffer, $hash_method);
           }
           ....
       }
   
   ```
   ?
   
   I tried to implement that, but couldn't find a way to do that. The `col.data_type()` is a runtime value, I don't we can infer it in the compile-time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1616472369


##########
spark/inspections/CometTPCHQueriesList-results.txt:
##########
@@ -1,142 +1,129 @@
 Query: q1 TPCH Snappy. Comet Exec: Enabled (CometHashAggregate, CometFilter, CometProject)
 Query: q1 TPCH Snappy: ExplainInfo:
-Shuffle: unsupported Spark partitioning: org.apache.spark.sql.catalyst.plans.physical.RangePartitioning
+Comet shuffle is not enabled: spark.sql.adaptive.coalescePartitions.enabled is enabled and spark.comet.shuffle.enforceMode.enabled is not enabled

Review Comment:
   @parthchandra I don't have enough compute resources to test against 1TB dataset. So, this and the TPCDS result are generated directly from running `org.apache.spark.sql.CometTPCHQueriesList/CometTPCDSQueriesList` locally without modifications.
   It seems a lot of `explain` info has been updated, do you think these changes are reasonable or due to missing some configurations in the scripts.
   
   it would be great that if you could rerun the inspection with this PR or should we update the script to maintain the up-to-date configurations?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#issuecomment-2110308036

   Thanks @advancedxy. I plan on reviewing this PR today. 
   
   Could you also update `docs/source/user-guide/expressions.md` to add xxhash64 as a supported expression?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600518754


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -104,66 +108,110 @@ fn test_murmur3() {
     let _expected = vec![
         142593372, 1485273170, -97053317, 1322437556, -396302900, 814637928,
     ];
+    assert_eq!(_hashes, _expected)
+}
+
+#[inline]
+pub(crate) fn spark_compatible_xxhash64<T: AsRef<[u8]>>(data: T, seed: u64) -> u64 {
+    // TODO: Rewrite with a stateless hasher to reduce stack allocation?
+    let mut hasher = XxHash64::with_seed(seed);
+    hasher.write(data.as_ref());
+    hasher.finish()
+}
+
+#[test]
+fn test_xxhash64() {
+    let _hashes = ["", "a", "ab", "abc", "abcd", "abcde"]
+        .into_iter()
+        .map(|s| spark_compatible_xxhash64(s.as_bytes(), 42) as i64)
+        .collect::<Vec<_>>();
+    let _expected = vec![
+        -7444071767201028348,
+        -8582455328737087284,
+        2710560539726725091,
+        1423657621850124518,
+        -6810745876291105281,
+        -990457398947679591,
+    ];
+    assert_eq!(_hashes, _expected);
 }
 
 macro_rules! hash_array {
-    ($array_type:ident, $column: ident, $hashes: ident) => {
+    ($array_type: ident, $column: ident, $hashes: ident, $hash_method: ident) => {
         let array = $column.as_any().downcast_ref::<$array_type>().unwrap();
         if array.null_count() == 0 {
             for (i, hash) in $hashes.iter_mut().enumerate() {
-                *hash = spark_compatible_murmur3_hash(&array.value(i), *hash);
+                *hash = $hash_method(&array.value(i), *hash);
             }
         } else {
             for (i, hash) in $hashes.iter_mut().enumerate() {
                 if !array.is_null(i) {
-                    *hash = spark_compatible_murmur3_hash(&array.value(i), *hash);
+                    *hash = $hash_method(&array.value(i), *hash);
                 }
             }
         }
     };
 }
 
 macro_rules! hash_array_primitive {
-    ($array_type:ident, $column: ident, $ty: ident, $hashes: ident) => {
+    ($array_type: ident, $column: ident, $ty: ident, $hashes: ident, $hash_method: ident) => {
         let array = $column.as_any().downcast_ref::<$array_type>().unwrap();
         let values = array.values();
 
         if array.null_count() == 0 {
             for (hash, value) in $hashes.iter_mut().zip(values.iter()) {
-                *hash = spark_compatible_murmur3_hash((*value as $ty).to_le_bytes(), *hash);
+                *hash = $hash_method((*value as $ty).to_le_bytes(), *hash);
             }
         } else {
             for (i, (hash, value)) in $hashes.iter_mut().zip(values.iter()).enumerate() {
                 if !array.is_null(i) {
-                    *hash = spark_compatible_murmur3_hash((*value as $ty).to_le_bytes(), *hash);
+                    *hash = $hash_method((*value as $ty).to_le_bytes(), *hash);
+                }
+            }
+        }
+    };
+}
+
+macro_rules! hash_array_primitive_boolean {

Review Comment:
   ```suggestion
   macro_rules! hash_array_boolean {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#issuecomment-2110435886

   Our `hash` implementation is also not compatible with Spark. I will file an issue for that.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "advancedxy (via GitHub)" <gi...@apache.org>.
advancedxy commented on PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#issuecomment-2110554569

   > > Our `hash` implementation is also not compatible with Spark. I will file an issue for that.
   > 
   > I filed #427
   
   Thanks for filing this. I think it's the same issue for both murmur3 hash and xxhash64.  I will submit a pr to fix that first.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: Add xxhash64 function support [datafusion-comet]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #424:
URL: https://github.com/apache/datafusion-comet/pull/424#discussion_r1600896291


##########
core/src/execution/datafusion/spark_hash.rs:
##########
@@ -193,27 +241,67 @@ macro_rules! hash_array_decimal {
 fn create_hashes_dictionary<K: ArrowDictionaryKeyType>(
     array: &ArrayRef,
     hashes_buffer: &mut [u32],
+    multi_col: bool,
 ) -> Result<()> {
     let dict_array = array.as_any().downcast_ref::<DictionaryArray<K>>().unwrap();
+    if multi_col {
+        // unpack the dictionary array as each row may have a different hash input
+        let unpacked = take(dict_array.values().as_ref(), dict_array.keys(), None)?;
+        create_hashes(&[unpacked], hashes_buffer)?;
+    } else {
+        // Hash each dictionary value once, and then use that computed
+        // hash for each key value to avoid a potentially expensive
+        // redundant hashing for large dictionary elements (e.g. strings)
+        let dict_values = Arc::clone(dict_array.values());
+        // same initial seed as Spark
+        let mut dict_hashes = vec![42; dict_values.len()];
+        create_hashes(&[dict_values], &mut dict_hashes)?;

Review Comment:
   Ah, I thought the hash computation is columnar based. It is actually a hash per row...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org