You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Imran Rashid <ir...@cloudera.com> on 2015/02/27 20:10:00 UTC
trouble with sbt building network-* projects?
Has anyone else noticed very strange build behavior in the network-*
projects?
maven seems to the doing the right, but sbt is very inconsistent.
Sometimes when it builds network-shuffle it doesn't know about any of the
code in network-common. Sometimes it will completely skip the java unit
tests. And then some time later, it'll suddenly decide it knows about some
more of the java unit tests. Its not from a simple change, like touching a
test file, or a file the test depends on -- nor a restart of sbt. I am
pretty confused.
maven had issues when I tried to add scala code to network-common, it would
compile the scala code but not make it available to java. I'm working
around that by just coding in java anyhow. I'd really like to be able to
run my tests in sbt, though, it makes the development iterations much
faster.
thanks,
Imran
Re: trouble with sbt building network-* projects?
Posted by Ted Yu <yu...@gmail.com>.
bq. I have to keep cd'ing into network/common, run mvn install, then go
back to network/shuffle and run some other mvn command over there.
Yeah - been through this.
Having continuous testing for maven would be nice.
On Fri, Feb 27, 2015 at 11:31 AM, Imran Rashid <ir...@cloudera.com> wrote:
> well, perhaps I just need to learn to use maven better, but currently I
> find sbt much more convenient for continuously running my tests. I do use
> zinc, but I'm looking for continuous testing. This makes me think I need
> sbt for that:
> http://stackoverflow.com/questions/11347633/is-there-a-java-continuous-testing-plugin-for-maven
>
> 1) I really like that in sbt I can run "~test-only
> com.foo.bar.SomeTestSuite" (or whatever other pattern) and just leave that
> running as I code, without having to go and explicitly trigger "mvn test"
> and wait for the result.
>
> 2) I find sbt's handling of sub-projects much simpler (when it works).
> I'm trying to make changes to network/common & network/shuffle, which means
> I have to keep cd'ing into network/common, run mvn install, then go back to
> network/shuffle and run some other mvn command over there. I don't want to
> run mvn at the root project level, b/c I don't want to wait for it to
> compile all the other projects when I just want to run tests in
> network/common. Even with incremental compiling, in my day-to-day coding I
> want to entirely skip compiling sql, graphx, mllib etc. -- I have to switch
> branches often enough that i end up triggering a full rebuild of those
> projects even when I haven't touched them.
>
>
>
>
>
> On Fri, Feb 27, 2015 at 1:14 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> bq. to be able to run my tests in sbt, though, it makes the development
>> iterations much faster.
>>
>> Was the preference for sbt due to long maven build time ?
>> Have you started Zinc on your machine ?
>>
>> Cheers
>>
>> On Fri, Feb 27, 2015 at 11:10 AM, Imran Rashid <ir...@cloudera.com>
>> wrote:
>>
>>> Has anyone else noticed very strange build behavior in the network-*
>>> projects?
>>>
>>> maven seems to the doing the right, but sbt is very inconsistent.
>>> Sometimes when it builds network-shuffle it doesn't know about any of the
>>> code in network-common. Sometimes it will completely skip the java unit
>>> tests. And then some time later, it'll suddenly decide it knows about
>>> some
>>> more of the java unit tests. Its not from a simple change, like
>>> touching a
>>> test file, or a file the test depends on -- nor a restart of sbt. I am
>>> pretty confused.
>>>
>>>
>>> maven had issues when I tried to add scala code to network-common, it
>>> would
>>> compile the scala code but not make it available to java. I'm working
>>> around that by just coding in java anyhow. I'd really like to be able to
>>> run my tests in sbt, though, it makes the development iterations much
>>> faster.
>>>
>>> thanks,
>>> Imran
>>>
>>
>>
>
Re: trouble with sbt building network-* projects?
Posted by Imran Rashid <ir...@cloudera.com>.
well, perhaps I just need to learn to use maven better, but currently I
find sbt much more convenient for continuously running my tests. I do use
zinc, but I'm looking for continuous testing. This makes me think I need
sbt for that:
http://stackoverflow.com/questions/11347633/is-there-a-java-continuous-testing-plugin-for-maven
1) I really like that in sbt I can run "~test-only
com.foo.bar.SomeTestSuite" (or whatever other pattern) and just leave that
running as I code, without having to go and explicitly trigger "mvn test"
and wait for the result.
2) I find sbt's handling of sub-projects much simpler (when it works). I'm
trying to make changes to network/common & network/shuffle, which means I
have to keep cd'ing into network/common, run mvn install, then go back to
network/shuffle and run some other mvn command over there. I don't want to
run mvn at the root project level, b/c I don't want to wait for it to
compile all the other projects when I just want to run tests in
network/common. Even with incremental compiling, in my day-to-day coding I
want to entirely skip compiling sql, graphx, mllib etc. -- I have to switch
branches often enough that i end up triggering a full rebuild of those
projects even when I haven't touched them.
On Fri, Feb 27, 2015 at 1:14 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. to be able to run my tests in sbt, though, it makes the development
> iterations much faster.
>
> Was the preference for sbt due to long maven build time ?
> Have you started Zinc on your machine ?
>
> Cheers
>
> On Fri, Feb 27, 2015 at 11:10 AM, Imran Rashid <ir...@cloudera.com>
> wrote:
>
>> Has anyone else noticed very strange build behavior in the network-*
>> projects?
>>
>> maven seems to the doing the right, but sbt is very inconsistent.
>> Sometimes when it builds network-shuffle it doesn't know about any of the
>> code in network-common. Sometimes it will completely skip the java unit
>> tests. And then some time later, it'll suddenly decide it knows about
>> some
>> more of the java unit tests. Its not from a simple change, like touching
>> a
>> test file, or a file the test depends on -- nor a restart of sbt. I am
>> pretty confused.
>>
>>
>> maven had issues when I tried to add scala code to network-common, it
>> would
>> compile the scala code but not make it available to java. I'm working
>> around that by just coding in java anyhow. I'd really like to be able to
>> run my tests in sbt, though, it makes the development iterations much
>> faster.
>>
>> thanks,
>> Imran
>>
>
>
Re: trouble with sbt building network-* projects?
Posted by Ted Yu <yu...@gmail.com>.
bq. to be able to run my tests in sbt, though, it makes the development
iterations much faster.
Was the preference for sbt due to long maven build time ?
Have you started Zinc on your machine ?
Cheers
On Fri, Feb 27, 2015 at 11:10 AM, Imran Rashid <ir...@cloudera.com> wrote:
> Has anyone else noticed very strange build behavior in the network-*
> projects?
>
> maven seems to the doing the right, but sbt is very inconsistent.
> Sometimes when it builds network-shuffle it doesn't know about any of the
> code in network-common. Sometimes it will completely skip the java unit
> tests. And then some time later, it'll suddenly decide it knows about some
> more of the java unit tests. Its not from a simple change, like touching a
> test file, or a file the test depends on -- nor a restart of sbt. I am
> pretty confused.
>
>
> maven had issues when I tried to add scala code to network-common, it would
> compile the scala code but not make it available to java. I'm working
> around that by just coding in java anyhow. I'd really like to be able to
> run my tests in sbt, though, it makes the development iterations much
> faster.
>
> thanks,
> Imran
>