You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/07/09 13:33:00 UTC
[jira] [Work logged] (BEAM-12591) Spark Structured Streaming runner
sources are not visible from the IDE
[ https://issues.apache.org/jira/browse/BEAM-12591?focusedWorklogId=620980&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-620980 ]
ASF GitHub Bot logged work on BEAM-12591:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 09/Jul/21 13:32
Start Date: 09/Jul/21 13:32
Worklog Time Spent: 10m
Work Description: echauchot commented on pull request #15148:
URL: https://github.com/apache/beam/pull/15148#issuecomment-877190015
Oops, just putting the SS runner sources back to main spark src folder only works for spark 2 but for spark 3 it doesn't as the spark 3 module inheritates the spark 2 related sources from main src folder and DataSourceV2 spark API is not available for Spark 3.
Ideal would be to figure out the equivalent of spark 2 DataSourceV2 API for spark 3 but I fear there is more incompatibilities with spark 3. Maybe a gradle specific conf for spark 2 module would be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 620980)
Time Spent: 20m (was: 10m)
> Spark Structured Streaming runner sources are not visible from the IDE
> ----------------------------------------------------------------------
>
> Key: BEAM-12591
> URL: https://issues.apache.org/jira/browse/BEAM-12591
> Project: Beam
> Issue Type: Bug
> Components: build-system, runner-spark
> Reporter: Etienne Chauchot
> Assignee: Etienne Chauchot
> Priority: P2
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When spark 3 was introduced, *the spark structured streaming runner sources were moved to spark/2 module* ([https://github.com/apache/beam/pull/14216/commits/9c0f6088078563ece6dc60877b0dcb320f1c9bfe]) and the build system was changed. Now spark runners are built per spark version module (2 or 3) and the version specific sources are copied from _spark-version/src_ to _spark-version/build/source-overrides_ and gradle source sets are set to _main-spark src + spark-version/build/source-overrides_. The problem is that _spark-version/src_ is outside of gradle source sets and thus the IDE does not compile this source, that is not an issue when there is very few classes but in the case of the spark structured streaming runner, *it is the whole runner code that is rendered invisible to the IDE* which is impracticable for runner development.
>
> As ide project settings are gerenated with gradle import, changing the project settings manually would not work as the settings would be erased with each gradle import
--
This message was sent by Atlassian Jira
(v8.3.4#803005)