You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by lonikar <lo...@gmail.com> on 2015/09/10 13:10:12 UTC
Spark 1.5.x: Java files in src/main/scala and vice versa
I found these files:
spark-1.5.0/sql/catalyst/*src/main/scala*/org/apache/spark/sql/types/*SQLUserDefinedType.java*
spark-1.5.0/core/src/main/java/org/apache/spark/api/java/function/package.scala
and several java files in spark-1.5.0/core/src/main/scala/.
Is this intentional or inadvertant?
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-1-5-x-Java-files-in-src-main-scala-and-vice-versa-tp14032.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Spark 1.5.x: Java files in src/main/scala and vice versa
Posted by Reynold Xin <rx...@databricks.com>.
Most these files are just package-info.java there for having a good package
index for JavaDoc. If we move them, we will need to create a folder in the
java one for each package that exposes any documentation. And it is very
likely we will forget to update package-info.java when we update
package.scala if the two files are far apart.
Doesn't seem that big of a deal to have them in Scala folder, and the
benefits ain't that big either.
On Sat, Sep 12, 2015 at 2:42 AM, Sean Owen <so...@cloudera.com> wrote:
> There are actually 33 instances of a Java file in src/main/scala -- I
> opened https://issues.apache.org/jira/browse/SPARK-10576 to track a
> discussion and decision.
>
> On Fri, Sep 11, 2015 at 3:10 PM, lonikar <lo...@gmail.com> wrote:
> > It does not cause any problem when building using maven. But when doing
> > eclipse:eclipse, the generated .classpath files contained only
> > <classpathentry kind="src" including="**/*.java" path="src"/>. This
> caused
> > all the .scala sources to be ignored and caused all kinds of eclipse
> build
> > errors. It resolved only when I added prebuild jars in the java build
> path,
> > and it also prevented me from debugging spark code.
> >
> > I understand eclipse:eclipse is not recommended way of creating eclipse
> > projects, but thats how I noticed this issue.
> >
> > As sean said, its a matter of code organization, and its confusing to
> find
> > java files in src/main/scala. In my env, I moved the files and did not
> see
> > notice any issues. Unless there is any specific purpose, it will be
> better
> > if the code is reorganized.
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-1-5-x-Java-files-in-src-main-scala-and-vice-versa-tp14032p14052.html
> > Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > For additional commands, e-mail: dev-help@spark.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>
Re: Spark 1.5.x: Java files in src/main/scala and vice versa
Posted by Sean Owen <so...@cloudera.com>.
There are actually 33 instances of a Java file in src/main/scala -- I
opened https://issues.apache.org/jira/browse/SPARK-10576 to track a
discussion and decision.
On Fri, Sep 11, 2015 at 3:10 PM, lonikar <lo...@gmail.com> wrote:
> It does not cause any problem when building using maven. But when doing
> eclipse:eclipse, the generated .classpath files contained only
> <classpathentry kind="src" including="**/*.java" path="src"/>. This caused
> all the .scala sources to be ignored and caused all kinds of eclipse build
> errors. It resolved only when I added prebuild jars in the java build path,
> and it also prevented me from debugging spark code.
>
> I understand eclipse:eclipse is not recommended way of creating eclipse
> projects, but thats how I noticed this issue.
>
> As sean said, its a matter of code organization, and its confusing to find
> java files in src/main/scala. In my env, I moved the files and did not see
> notice any issues. Unless there is any specific purpose, it will be better
> if the code is reorganized.
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-1-5-x-Java-files-in-src-main-scala-and-vice-versa-tp14032p14052.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Spark 1.5.x: Java files in src/main/scala and vice versa
Posted by lonikar <lo...@gmail.com>.
It does not cause any problem when building using maven. But when doing
eclipse:eclipse, the generated .classpath files contained only
<classpathentry kind="src" including="**/*.java" path="src"/>. This caused
all the .scala sources to be ignored and caused all kinds of eclipse build
errors. It resolved only when I added prebuild jars in the java build path,
and it also prevented me from debugging spark code.
I understand eclipse:eclipse is not recommended way of creating eclipse
projects, but thats how I noticed this issue.
As sean said, its a matter of code organization, and its confusing to find
java files in src/main/scala. In my env, I moved the files and did not see
notice any issues. Unless there is any specific purpose, it will be better
if the code is reorganized.
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-1-5-x-Java-files-in-src-main-scala-and-vice-versa-tp14032p14052.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Spark 1.5.x: Java files in src/main/scala and vice versa
Posted by Sean Owen <so...@cloudera.com>.
This is probably true as the scala plugin actually compiles both
.scala and .java files. Still it seems like the wrong place just as a
matter of style. Can we try moving it and verify it's still OK?
On Fri, Sep 11, 2015 at 12:43 AM, Reynold Xin <rx...@databricks.com> wrote:
> There isn't really any difference I think where you put them. Did you run
> into a problem?
>
>
> On Thu, Sep 10, 2015 at 6:38 AM, Sean Owen <so...@cloudera.com> wrote:
>>
>> I feel like I knew the answer to this but have forgotten. Reynold do
>> you know about this file? looks like you added it.
>>
>> On Thu, Sep 10, 2015 at 1:10 PM, lonikar <lo...@gmail.com> wrote:
>> > I found these files:
>> >
>> > spark-1.5.0/sql/catalyst/*src/main/scala*/org/apache/spark/sql/types/*SQLUserDefinedType.java*
>> >
>> >
>> > spark-1.5.0/core/src/main/java/org/apache/spark/api/java/function/package.scala
>> > and several java files in spark-1.5.0/core/src/main/scala/.
>> >
>> > Is this intentional or inadvertant?
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> > http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-1-5-x-Java-files-in-src-main-scala-and-vice-versa-tp14032.html
>> > Sent from the Apache Spark Developers List mailing list archive at
>> > Nabble.com.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: dev-help@spark.apache.org
>> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Spark 1.5.x: Java files in src/main/scala and vice versa
Posted by Reynold Xin <rx...@databricks.com>.
There isn't really any difference I think where you put them. Did you run
into a problem?
On Thu, Sep 10, 2015 at 6:38 AM, Sean Owen <so...@cloudera.com> wrote:
> I feel like I knew the answer to this but have forgotten. Reynold do
> you know about this file? looks like you added it.
>
> On Thu, Sep 10, 2015 at 1:10 PM, lonikar <lo...@gmail.com> wrote:
> > I found these files:
> >
> spark-1.5.0/sql/catalyst/*src/main/scala*/org/apache/spark/sql/types/*SQLUserDefinedType.java*
> >
> >
> spark-1.5.0/core/src/main/java/org/apache/spark/api/java/function/package.scala
> > and several java files in spark-1.5.0/core/src/main/scala/.
> >
> > Is this intentional or inadvertant?
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-1-5-x-Java-files-in-src-main-scala-and-vice-versa-tp14032.html
> > Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> > For additional commands, e-mail: dev-help@spark.apache.org
> >
>
Re: Spark 1.5.x: Java files in src/main/scala and vice versa
Posted by Sean Owen <so...@cloudera.com>.
I feel like I knew the answer to this but have forgotten. Reynold do
you know about this file? looks like you added it.
On Thu, Sep 10, 2015 at 1:10 PM, lonikar <lo...@gmail.com> wrote:
> I found these files:
> spark-1.5.0/sql/catalyst/*src/main/scala*/org/apache/spark/sql/types/*SQLUserDefinedType.java*
>
> spark-1.5.0/core/src/main/java/org/apache/spark/api/java/function/package.scala
> and several java files in spark-1.5.0/core/src/main/scala/.
>
> Is this intentional or inadvertant?
>
>
>
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-1-5-x-Java-files-in-src-main-scala-and-vice-versa-tp14032.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org