[jira] [Assigned] (SPARK-60340) Avoid System.getenv on every JavaUtils.isTesting call

["Hyukjin Kwon (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-60340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon reassigned SPARK-60340:
------------------------------------

    Assignee: Sebastian Hillig

> Avoid System.getenv on every JavaUtils.isTesting call
> -----------------------------------------------------
>
>                 Key: SPARK-60340
>                 URL: https://issues.apache.org/jira/browse/SPARK-60340
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 4.1.3
>            Reporter: Sebastian Hillig
>            Assignee: Sebastian Hillig
>            Priority: Major
>              Labels: pull-request-available
>
> On Linux, {{System.getenv(String)}} goes through {{{}java.lang.ProcessEnvironment{}}}, which wraps the name in a new byte[]-backed {{Variable}} on every lookup, so each {{isTesting()}} call allocates (56 bytes per call in the measurement below). {{isTesting}} is on hot paths. For example, {{AnalysisHelper.assertNotAnalysisRule}} calls it from the {{LogicalPlan}} overrides of {{transformDownWithPruning}} / {{transformUpWithPruning}} / {{{}transformAllExpressionsWithPruning{}}}, which run once per plan node visited by every transform. In a JFR allocation profile of driver-side query planning, these call sites accounted for roughly 1.8% of driver allocation and about 1-2% of CPU.
> A JVM's environment cannot change after startup, so caching the env lookup is safe. The system property cannot be cached: tests set and clear it at runtime ({{{}SparkTestSuite.beforeAll{}}}, {{{}BenchmarkBase{}}}, {{{}KryoSerializerBenchmark{}}}). The only code that mutates the process environment reflectively ({{{}SparkConnectClientSuite.withEnvs{}}}, {{{}JavaXmlSuite.setEnv{}}}) touches {{SPARK_REMOTE}} / {{{}SPARK_LOCAL_IP{}}}, not {{{}SPARK_TESTING{}}}. Code that sets {{SPARK_TESTING}} for a child process via {{ProcessBuilder.environment()}} is unaffected, because the child is a new JVM.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org