You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@cassandra.apache.org by "Francisco Guerrero (Jira)" <ji...@apache.org> on 2025/10/28 23:31:00 UTC

[jira] [Commented] (CASSANALYTICS-89) Create dedicated data class for broadcast variable during bulk write

    [ https://issues.apache.org/jira/browse/CASSANALYTICS-89?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18033708#comment-18033708 ] 

Francisco Guerrero commented on CASSANALYTICS-89:
-------------------------------------------------

+1 thanks for the patch

> Create dedicated data class for broadcast variable during bulk write
> --------------------------------------------------------------------
>
>                 Key: CASSANALYTICS-89
>                 URL: https://issues.apache.org/jira/browse/CASSANALYTICS-89
>             Project: Apache Cassandra Analytics
>          Issue Type: Improvement
>          Components: Writer
>            Reporter: Yifan Cai
>            Assignee: Yifan Cai
>            Priority: Normal
>          Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Bulk write in Analytics uses Spark’s broadcast variable feature to distribute job information (BulkWriterContext) to executors. While this works today, it triggers unnecessary work in Spark’s SizeEstimator, which inspects all fields—including transient ones. Since BulkWriterContext (and the objects it references) contain many transient fields that aren’t meant to be serialized, SizeEstimator still walks them via reflection, wasting CPU cycles.
> A cleaner approach would be to introduce a dedicated data class for the broadcast variable, with only the minimal set of fields required for distribution.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org