[jira] [Updated] (BAHIR-8) Investigate weather we can use org.apache.bahir packages on Apache Spark extensions

["Christian Kadner (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BAHIR-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Christian Kadner updated BAHIR-8:
---------------------------------
             Assignee: Christian Kadner
    Affects Version/s: 2.0.0
           Issue Type: Task  (was: Bug)

The current set of streaming modules in Bahir (_akka, mqtt, zeromq, twitter_) are using 7 different methods/constructors/fields which are private/packaged-scoped to {{org.apache.spark.*}}.

{noformat}

core/src/main/scala/org/apache/spark/SparkContext.scala
- private[spark] def clean[F <: AnyRef](f: F, checkSerializable: Boolean = true): F = { ...

core/src/main/scala/org/apache/spark/util/Utils.scala
- private[spark] object Utils extends Logging { ... 

core/src/test/scala/org/apache/spark/SparkFunSuite.scala
- private[spark] abstract class SparkFunSuite { ...

core/src/main/scala/org/apache/spark/Logging.scala
- private[spark] trait Logging { ...

streaming/src/main/scala/org/apache/spark/streaming/StreamingContext.scala
- private[streaming] val sc: SparkContext = { ...
- private[streaming] def withNamedScope[U](name: String)(body: => U): U = { ...

streaming/src/main/scala/org/apache/spark/streaming/dstream/InputDStream.scala
- private[streaming] def name: String = { ...

{noformat}

I suggest to create a _shim_ module named *common* or *common-streaming* to the Bahir repository that will contains common utils and the necessary shim classes/objects to make these "private APIs" accessible to the rest of the classes/objects under the {{org.apache.bahir}} namespace. I am working on a PR for that.

> Investigate weather we can use org.apache.bahir packages on Apache Spark extensions
> -----------------------------------------------------------------------------------
>
>                 Key: BAHIR-8
>                 URL: https://issues.apache.org/jira/browse/BAHIR-8
>             Project: Bahir
>          Issue Type: Task
>          Components: Spark Streaming Connectors
>    Affects Versions: 2.0.0
>            Reporter: Luciano Resende
>            Assignee: Christian Kadner
>
> Experiment if we can move the current extensions to org.apache.bahir package. We might see issues related to the private[spark], etc restriction that scala imposes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)