You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@systemds.apache.org by "e-strauss (via GitHub)" <gi...@apache.org> on 2024/06/03 15:56:00 UTC
[PR] [SYSTEMDS-3695] Fix frame binary append for Spark for misaligned inputs (FrameAppendRSPInstruction) [systemds]
e-strauss opened a new pull request, #2031:
URL: https://github.com/apache/systemds/pull/2031
This patch fixes the binary cbind operation for the spark backend (FrameAppendRSPInstruction) for the edge case when the inputs have different block alignments.
The RHS input is aligned to the LHS by the following steps:
- collect on Index RDD of the LHS, sort the array of indices
- map each FrameBlock from RHS to the correct FrameBlock on the LHS by doing binary search on the index array
- > if necessary slice block on the RHS and map it to multiple Blocks on the LHS
Additionally, this patch adds test cases for different edge cases regarding the alignment. E.g.
cbind
LHS RHS:
+-----+ +---------+
| | +---------+
+-----+ | |
| | +---------+
+-----+ | |
| | +---------+
+-----+ | |
+-----+ +---------+
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SYSTEMDS-3695] Fix frame binary append for Spark for misaligned inputs (FrameAppendRSPInstruction) [systemds]
Posted by "mboehm7 (via GitHub)" <gi...@apache.org>.
mboehm7 commented on PR #2031:
URL: https://github.com/apache/systemds/pull/2031#issuecomment-2150696805
I was thinking it would be very good to add a primitive in `FrameRDDConverterUtils` to check if the distributed collection is already 1K-aligned (by scanning and aggregating all block meta data), another one to convert it to 1K blocks (this is expensive because of shuffling), and then keep this information in the `FrameObject`'s DataCharacteristics (or subclass like FameCharacteristics) as a simple boolean. Whenever we then come to binary operations like the frame cbind, we can check this meta data and potentially resort back to a normal block append. Feel free to have a look into it if you're interested, but it's nice to have right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SYSTEMDS-3695] Fix frame binary append for Spark for misaligned inputs (FrameAppendRSPInstruction) [systemds]
Posted by "e-strauss (via GitHub)" <gi...@apache.org>.
e-strauss commented on PR #2031:
URL: https://github.com/apache/systemds/pull/2031#issuecomment-2149770886
Thanks - I guessed I will be simple way to understand the test case. When you say keep the information do mean store this information in the meta data field for each RDD or a function which checks for alignment, and than conditionally do the alignment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SYSTEMDS-3695] Fix frame binary append for Spark for misaligned inputs (FrameAppendRSPInstruction) [systemds]
Posted by "mboehm7 (via GitHub)" <gi...@apache.org>.
mboehm7 commented on PR #2031:
URL: https://github.com/apache/systemds/pull/2031#issuecomment-2149101123
LGTM - thanks for the additional improvement @e-strauss. I liked the ASCII art in the tests. During the merge I left a TODO comment to keep the information if a frame RDD is already aligned (so we only conditionally align if needed), and fixed the alignment of ASCII art (which where a bit distorted due to a mix of spaces and tabs).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [SYSTEMDS-3695] Fix frame binary append for Spark for misaligned inputs (FrameAppendRSPInstruction) [systemds]
Posted by "mboehm7 (via GitHub)" <gi...@apache.org>.
mboehm7 closed pull request #2031: [SYSTEMDS-3695] Fix frame binary append for Spark for misaligned inputs (FrameAppendRSPInstruction)
URL: https://github.com/apache/systemds/pull/2031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@systemds.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org