You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "zhztheplayer (via GitHub)" <gi...@apache.org> on 2024/03/27 00:36:57 UTC

[I] [VL] Vanilla Spark broadcast exchange + C2R is slow sometimes [incubator-gluten]

zhztheplayer opened a new issue, #5136:
URL: https://github.com/apache/incubator-gluten/issues/5136

   ### Backend
   
   VL (Velox)
   
   ### Bug description
   
   This is because the code to convert vanilla Spark's hashed relation to Gluten's sometimes produced duplicated rows.
   
   The fix will be incorporated in https://github.com/apache/incubator-gluten/pull/5058 since it can be tested by the ACBO changes.
   
   ### Spark version
   
   None
   
   ### Spark configurations
   
   _No response_
   
   ### System information
   
   _No response_
   
   ### Relevant logs
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Vanilla Spark broadcast exchange + R2C is slow sometimes [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer closed issue #5136: [VL] Vanilla Spark broadcast exchange + R2C is slow sometimes
URL: https://github.com/apache/incubator-gluten/issues/5136


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Vanilla Spark broadcast exchange + R2C is slow sometimes [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #5136:
URL: https://github.com/apache/incubator-gluten/issues/5136#issuecomment-2021919347

   The major issue I have found is that the `flatMap` approach would cause `UnsafeHashedRelation` to produce duplicated rows in my case (TPCDS q14a with current version of ACBO)
   While the `map` approach would cause `LongHashedRelation` to loss rows (TPCDS q2).
   
   The following fix (the same with #5141) can work but I didn't dive into it deeply to find the root reason of the inconsistency (maybe related to `keyIsUnique`? I am not sure).
   
   ```scala
     private def reconstructRows(relation: HashedRelation): Iterator[InternalRow] = {
       // It seems that LongHashedRelation and UnsafeHashedRelation don't follow the same
       //  criteria while getting values from them.
       // Should review the internals of this part of code.
       relation match {
         case relation: LongHashedRelation if relation.keyIsUnique =>
           relation.keys().map(k => relation.getValue(k))
         case relation: LongHashedRelation if !relation.keyIsUnique =>
           relation.keys().flatMap(k => relation.get(k))
         case other => other.valuesWithKeyIndex().map(_.getValue)
       }
     }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Vanilla Spark broadcast exchange + R2C is slow sometimes [incubator-gluten]

Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on issue #5136:
URL: https://github.com/apache/incubator-gluten/issues/5136#issuecomment-2021873724

   Thank you @zhztheplayer It's a good point, columnar broadcast would broadcast the origin binary data but vanilla Spark would broadcast hash relation. So I think this issue is a common case even if there is no r2c.
   
   Is it possbile to create a new pr for this issue ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Vanilla Spark broadcast exchange + R2C is slow sometimes [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #5136:
URL: https://github.com/apache/incubator-gluten/issues/5136#issuecomment-2021916024

   I don't have dedicated UTs for it so it was incorporated into the other PR.
   
   Still I can open one for it if you think it's needed: https://github.com/apache/incubator-gluten/pull/5141.
   
   The change was already tested so I will proceed to merge after code style check is passed if it's OK to you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org


Re: [I] [VL] Vanilla Spark broadcast exchange + R2C is slow sometimes [incubator-gluten]

Posted by "zhztheplayer (via GitHub)" <gi...@apache.org>.
zhztheplayer commented on issue #5136:
URL: https://github.com/apache/incubator-gluten/issues/5136#issuecomment-2022119518

   Fixed in https://github.com/apache/incubator-gluten/pull/5141. I assume we can close this now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org