You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "chenhao-db (via GitHub)" <gi...@apache.org> on 2026/03/23 02:24:25 UTC

[PR] [SPARK-56134] Make BufferedRowIterator.unsafeRow public to avoid IllegalAccessError. [spark]

chenhao-db opened a new pull request, #54942:
URL: https://github.com/apache/spark/pull/54942

   ### What changes were proposed in this pull request?
   
   This is very similar to https://github.com/apache/spark/pull/20779. When a generated code has split classes and they try to access protected fields in `BufferedRowIterator`, an `IllegalAccessError` will happen.
   
   I am not making `partitionIndex` public too because I cannot find a real example that can trigger an `IllegalAccessError` on it too. In the code base, `partitionIndex` is mostly accessed via `addPartitionInitializationStatement`. Since it is used in the partition initialization code, not in split classes, there should be no issue with it.
   
   ### Why are the changes needed?
   
   Fix runtime error in large queries.
   
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Unit test.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56134][SQL] Make BufferedRowIterator.unsafeRow public to avoid IllegalAccessError. [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #54942: [SPARK-56134][SQL] Make BufferedRowIterator.unsafeRow public to avoid IllegalAccessError.
URL: https://github.com/apache/spark/pull/54942


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56134][SQL] Make BufferedRowIterator.unsafeRow public to avoid IllegalAccessError. [spark]

Posted by "rednaxelafx (via GitHub)" <gi...@apache.org>.
rednaxelafx commented on PR #54942:
URL: https://github.com/apache/spark/pull/54942#issuecomment-4115519875

   Looks good (not official Apache Spark reviewer).
   
   The PR description mentioned that the `partitionIndex` is not included in this PR. Practically that's okay, because currently the partition initialization code is not yet subject to code splitting.
   As can be seen from the init mutable state's case, when these init code grows big, there may be a future point in time that someone decides that partition init  code also need to support splitting. But that might still be okay.
   If I were to do it, I'd probably do specialized splitting so that the split functions specific to partition init would still take `partitionIndex` as the argument, then we still don't need to make it "public" since it's not a field to begin with.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56134] Make BufferedRowIterator.unsafeRow public to avoid IllegalAccessError. [spark]

Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on PR #54942:
URL: https://github.com/apache/spark/pull/54942#issuecomment-4107603536

   @viirya @hvanhovell @rednaxelafx could you take a look? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56134][SQL] Make BufferedRowIterator.unsafeRow public to avoid IllegalAccessError. [spark]

Posted by "chenhao-db (via GitHub)" <gi...@apache.org>.
chenhao-db commented on PR #54942:
URL: https://github.com/apache/spark/pull/54942#issuecomment-4340388828

   @HyukjinKwon could you help merge it? Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56134][SQL] Make BufferedRowIterator.unsafeRow public to avoid IllegalAccessError. [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #54942:
URL: https://github.com/apache/spark/pull/54942#issuecomment-4340703527

   Merged to master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org