[jira] [Updated] (BEAM-12591) Spark Structured Streaming runner sources should be put back to main src folder

["Kenneth Knowles (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-12591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Kenneth Knowles updated BEAM-12591:
-----------------------------------
    Status: Open  (was: Triage Needed)

> Spark Structured Streaming runner sources should be put back to main src folder
> -------------------------------------------------------------------------------
>
>                 Key: BEAM-12591
>                 URL: https://issues.apache.org/jira/browse/BEAM-12591
>             Project: Beam
>          Issue Type: Bug
>          Components: build-system, runner-spark
>            Reporter: Etienne Chauchot
>            Assignee: Etienne Chauchot
>            Priority: P2
>
> When spark 3 was introduced, *the spark structured streaming runner sources were moved to spark/2 module* (https://github.com/apache/beam/pull/14216/commits/9c0f6088078563ece6dc60877b0dcb320f1c9bfe) and the build system was changed. Now spark runners are built per spark version module (2 or 3) and the version specific sources are copied from _spark-version/src_ to _spark-version/build/source-overrides_ and gradle source sets are set to _main-spark src + spark-version/build/source-overrides_. The problem is that  _spark-version/src_ is outside of gradle source sets and thus the IDE does not compile this source, that is not an issue when there is very few classes but in the case of the spark structured streaming runner, *it is the whole runner code that is rendered invisible to the IDE* which is impracticable for runner development
>  
> We should just put back spark structured streaming sources to main spark sources. This will not change the build as it is still done per spark-version so deps will be ok.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)