Re: Running Mahout on a Spark cluster

[Trevor Grant <tr...@gmail.com>]

Hey- sorry for long delay. I've been traveling.

Pat Ferrel was telling me he was having some simlar issues with
Spark+Mahout+SBT recently, and that we need to re-examine our naming
conventions on JARs.

Fwiw- I have several project that use Spark+Mahout in Spark 2.1/Scala-2.11,
and we even test this in our Travis CI tests, but the trick is- we use
Maven for the build. Any chance you could use maven?  If not, maybe Pat can
chime in here, I'm just not an SBT user, so I'm not 100% sure what to tell
you.



On Fri, Sep 22, 2017 at 10:06 PM, Hoa Nguyen <ho...@insightdatascience.com>
wrote:

> Hey all,
>
> Thanks for the offers of help. I've been able to narrow down some of the
> problems to version incompatibility and I just wanted to give an update.
> Just to back track a bit, my initial goal was to run Mahout on a
> distributed cluster whether that was running Hadoop Map Reduce or Spark.
>
> I started out trying to get it to run on Spark, which I have some
> familiarity, but that didn't seem to work. While the error messages seem to
> indicate there weren't enough resources on the workers ("WARN
> scheduler.TaskSchedulerImpl: Initial job has not accepted any resources;
> check your cluster UI to ensure that workers are registered and have
> sufficient memory"), I'm pretty sure that wasn't the case, not only because
> it's a 4 node cluster of m4.xlarges, I was able to run another, simpler
> Spark batch job on that same distributed cluster.
>
> After a bit of wrangling, I was able to narrow down some of the issues. It
> turns out I was kind of blindly using this repo https://github.com/
> pferrel/3-input-cooc as a guide without fully realizing that it was from
> several years ago and based on Mahout 0.10.0, Scala 2.10 and Spark 1.1.1
> That is significantly different from my environment, which has Mahout
> 0.13.0 and Spark 2.1.1 installed, which also means I have to use Scala
> 2.11. After modifying the build.sbt file to account for those versions, I
> now have compile type mismatch issues that I'm just not that savvy to fix
> (see attached screenshot if you're interested).
>
> Anyway, the good news that I was able to finally get Mahout code running
> on Hadoop map-reduce, but also after a bit wrangling. It turned out my
> instances were running Ubuntu 14 and apparently that doesn't play well with
> Hadoop 2.7.4, which prevented me from running any sample Mahout code (from
> here: https://github.com/apache/mahout/tree/master/examples/bin) that
> relied on map-reduce. Those problems went away after I installed Hadoop
> 2.8.1 instead. Now I'm able to get the shell scripts running on a
> distributed Hadoop cluster (yay!).
>
> Anyway, if anyone has more recent and working Spark Scala code that uses
> Mahout that they can point me to, I'd appreciate it.
>
> Many thanks!
> Hoa
>
> On Fri, Sep 22, 2017 at 1:09 AM, Trevor Grant <tr...@gmail.com>
> wrote:
>
>> Hi Hoa,
>>
>> A few things could be happening here, I haven't run across that specific
>> error.
>>
>> 1) Spark 2.x - Mahout 0.13.0: Mahout 0.13.0 WILL run on Spark 2.x, however
>> you need to build from source (not the binaries).  You can do this by
>> downloading mahout source or cloning the repo and building with:
>> mvn clean install -Pspark-2.1,scala-2.11 -DskipTests
>>
>> 2) Have you setup spark with Kryo serialization? How you do this depends
>> on
>> if you're in the shell/zeppelin or using spark submit.
>>
>> However, for both of these cases- it shouldn't have even run local afaik
>> so
>> the fact it did tells me you probably have gotten this far?
>>
>> Assuming you've done 1 and 2, can you share some code? I'll see if I can
>> recreate on my end.
>>
>> Thanks!
>>
>> tg
>>
>> On Thu, Sep 21, 2017 at 9:37 PM, Hoa Nguyen <ho...@insightdatascience.com>
>> wrote:
>>
>> > I apologize in advance if this is too much of a newbie question but I'm
>> > having a hard time running any Mahout example code in a distributed
>> Spark
>> > cluster. The code runs as advertised when Spark is running locally on
>> one
>> > machine but the minute I point Spark to a cluster and master url, I
>> can't
>> > get it to work, drawing the error: "WARN scheduler.TaskSchedulerImpl:
>> > Initial job has not accepted any resources; check your cluster UI to
>> ensure
>> > that workers are registered and have sufficient memory"
>> >
>> > I know my Spark cluster is configured and working correctly because I
>> ran
>> > non-Mahout code and it runs on a distributed cluster fine. What am I
>> doing
>> > wrong? The only thing I can think of is that my Spark version is too
>> recent
>> > -- 2.1.1 -- for the Mahout version I'm using -- 0.13.0. Is that it or
>> am I
>> > doing something else wrong?
>> >
>> > Thanks for any advice,
>> > Hoa
>> >
>>
>
>