Development environment issues

[pierred <pi...@demartines.com>]

Hello all,

I am trying to get productive with Spark and Scala but I haven't figured out
a good development environment yet.

Coming from the eclipse/java/ant/ivy/hadoop world, I understand that I have
a steep learning curve ahead of me, but still I would expect to be able to
relatively quickly settle in an environment that easily lets me do the
following tasks:

1. debug locally  (being able to set breakpoints, inspect variables,
single-step if necessary, etc.)

2. run individual tests and test suites, manually as well as using a CI e.g.
Jenkins

3. deploy to an EC2 cluster and monitor the jobs

So far it has been quite frustrating. I can get things going at the command
line via sbt, but I'd like to do the code-edit-test-debug neatly and
seamlessly into an IDE.

I recently updated my JDK to 1.8.0_11 (breaking a bunch of things in the
process). Ideally I'd like to use Scala 2.11.2 and its new features
(although at the moment there are several libraries that I use that aren't
available yet for scala-2.11).  As a longtime Eclipse user, I am so far kind
of disappointed with its scala/sbt support.  For the Eclipse version I use
(Luna), I found a "milestone" version of ScalaIDE that seems to more or less
work, but I don't seem to be able to run tests (it crashes).

Without getting too deep in the details, to say the least the whole stack
(the way I set it up, anyway) seems a bit wobbly.

So, what is the accepted wisdom in terms of IDE and development environment?
Is there a good tutorial to set things up so that one half of the
libraries/tools doesn't break the other half?

What do you guys use?
scala 2.10 or 2.11?
sbt or maven?
eclipse or idea?
jdk7 or 8?




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Development-environment-issues-tp12611.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Development environment issues

[Daniel Siegmann <da...@velos.io>]

On Thu, Aug 21, 2014 at 6:21 PM, pierred <pi...@demartines.com> wrote:

So, what is the accepted wisdom in terms of IDE and development environment?
>

I don't know what the accepted wisdom is. I've been getting by with the
Scala IDE for Eclipse, though I am using the stable version - as you noted,
this keeps me from upgrading to the latest Eclipse version. The quality of
the Scala IDE is poor, but I have found it generally usable. I generate the
Eclipse project files from SBT. Debugging does work (mostly) - just be
aware you can't easily step into a lambda, so it's easiest to add a
breakpoint inside of it.

As for unit testing, both Specs2 and ScalaTest work, and I can run
individual tests within Eclipse. For Specs2 there is an Eclipse plugin, and
for ScalaTest you can annotate your tests with
@RunWith(classOf[JUnitRunner]) and it'll work in the usual JUnit tools. I
have automated tests running in Bamboo. Took a bit of wrangling to get the
test output picked up, but it works.


> Is there a good tutorial to set things up so that one half of the
> libraries/tools doesn't break the other half?
>

No idea.


> What do you guys use?
> scala 2.10 or 2.11?
> sbt or maven?
> eclipse or idea?
> jdk7 or 8?
>

I'm using Java 7 and Scala 2.10.x (not every framework I use supports later
versions). SBT because I use the Play Framework, but I miss Maven. I
haven't tried IntelliJ's Scala support, but it's probably worth a shot.

The tooling isn't nearly as solid as what Java has, but I make due.

-- 
Daniel Siegmann, Software Developer
Velos
Accelerating Machine Learning

440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001
E: daniel.siegmann@velos.io W: www.velos.io