Spark fpg large basket

[Sean Barzilay <se...@gmail.com>]

Hi I am currently using spark 1.3.0-snapshot to run the fpg algorithm from
the mllib library. When I am trying to run the algorithm over a large
basket(over 1000 items) the program seems to never finish. Did anyone find
a workaround for this problem?

Re: Spark fpg large basket

[Sean Barzilay <se...@gmail.com>]

My min support is low and after filling out all my space I am applying a
filter on the results to only get item seta that interest me

On Wed, 11 Mar 2015 1:58 pm Sean Owen <so...@cloudera.com> wrote:

> Have you looked at how big your output is? for example, if your min
> support is very low, you will output a massive volume of frequent item
> sets. If that's the case, then it may be expected that it's taking
> ages to write terabytes of data.
>
> On Wed, Mar 11, 2015 at 8:34 AM, Sean Barzilay <se...@gmail.com>
> wrote:
> > The program spends its time when I am writing the output to a text file
> and
> > I am using 70 partitions
> >
> >
> > On Wed, 11 Mar 2015 9:55 am Sean Owen <so...@cloudera.com> wrote:
> >>
> >> I don't think there is enough information here. Where is the program
> >> spending its time? where does it "stop"? how many partitions are
> >> there?
> >>
> >> On Wed, Mar 11, 2015 at 7:10 AM, Akhil Das <ak...@sigmoidanalytics.com>
> >> wrote:
> >> > You need to set spark.cores.max to a number say 16, so that on all 4
> >> > machines the tasks will get distributed evenly, Another thing would be
> >> > to
> >> > set spark.default.parallelism if you haven't tried already.
> >> >
> >> > Thanks
> >> > Best Regards
> >> >
> >> > On Wed, Mar 11, 2015 at 12:27 PM, Sean Barzilay <
> sesnbarzilay@gmail.com>
> >> > wrote:
> >> >>
> >> >> I am running on a 4 workers cluster each having between 16 to 30
> cores
> >> >> and
> >> >> 50 GB of ram
> >> >>
> >> >>
> >> >> On Wed, 11 Mar 2015 8:55 am Akhil Das <ak...@sigmoidanalytics.com>
> >> >> wrote:
> >> >>>
> >> >>> Depending on your cluster setup (cores, memory), you need to specify
> >> >>> the
> >> >>> parallelism/repartition the data.
> >> >>>
> >> >>> Thanks
> >> >>> Best Regards
> >> >>>
> >> >>> On Wed, Mar 11, 2015 at 12:18 PM, Sean Barzilay
> >> >>> <se...@gmail.com>
> >> >>> wrote:
> >> >>>>
> >> >>>> Hi I am currently using spark 1.3.0-snapshot to run the fpg
> algorithm
> >> >>>> from the mllib library. When I am trying to run the algorithm over
> a
> >> >>>> large
> >> >>>> basket(over 1000 items) the program seems to never finish. Did
> anyone
> >> >>>> find a
> >> >>>> workaround for this problem?
> >> >>>
> >> >>>
> >> >
>

Re: Spark fpg large basket

[Sean Owen <so...@cloudera.com>]

Have you looked at how big your output is? for example, if your min
support is very low, you will output a massive volume of frequent item
sets. If that's the case, then it may be expected that it's taking
ages to write terabytes of data.

On Wed, Mar 11, 2015 at 8:34 AM, Sean Barzilay <se...@gmail.com> wrote:
> The program spends its time when I am writing the output to a text file and
> I am using 70 partitions
>
>
> On Wed, 11 Mar 2015 9:55 am Sean Owen <so...@cloudera.com> wrote:
>>
>> I don't think there is enough information here. Where is the program
>> spending its time? where does it "stop"? how many partitions are
>> there?
>>
>> On Wed, Mar 11, 2015 at 7:10 AM, Akhil Das <ak...@sigmoidanalytics.com>
>> wrote:
>> > You need to set spark.cores.max to a number say 16, so that on all 4
>> > machines the tasks will get distributed evenly, Another thing would be
>> > to
>> > set spark.default.parallelism if you haven't tried already.
>> >
>> > Thanks
>> > Best Regards
>> >
>> > On Wed, Mar 11, 2015 at 12:27 PM, Sean Barzilay <se...@gmail.com>
>> > wrote:
>> >>
>> >> I am running on a 4 workers cluster each having between 16 to 30 cores
>> >> and
>> >> 50 GB of ram
>> >>
>> >>
>> >> On Wed, 11 Mar 2015 8:55 am Akhil Das <ak...@sigmoidanalytics.com>
>> >> wrote:
>> >>>
>> >>> Depending on your cluster setup (cores, memory), you need to specify
>> >>> the
>> >>> parallelism/repartition the data.
>> >>>
>> >>> Thanks
>> >>> Best Regards
>> >>>
>> >>> On Wed, Mar 11, 2015 at 12:18 PM, Sean Barzilay
>> >>> <se...@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> Hi I am currently using spark 1.3.0-snapshot to run the fpg algorithm
>> >>>> from the mllib library. When I am trying to run the algorithm over a
>> >>>> large
>> >>>> basket(over 1000 items) the program seems to never finish. Did anyone
>> >>>> find a
>> >>>> workaround for this problem?
>> >>>
>> >>>
>> >

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Spark fpg large basket

[Sean Barzilay <se...@gmail.com>]

The program spends its time when I am writing the output to a text file and
I am using 70 partitions

On Wed, 11 Mar 2015 9:55 am Sean Owen <so...@cloudera.com> wrote:

> I don't think there is enough information here. Where is the program
> spending its time? where does it "stop"? how many partitions are
> there?
>
> On Wed, Mar 11, 2015 at 7:10 AM, Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
> > You need to set spark.cores.max to a number say 16, so that on all 4
> > machines the tasks will get distributed evenly, Another thing would be to
> > set spark.default.parallelism if you haven't tried already.
> >
> > Thanks
> > Best Regards
> >
> > On Wed, Mar 11, 2015 at 12:27 PM, Sean Barzilay <se...@gmail.com>
> > wrote:
> >>
> >> I am running on a 4 workers cluster each having between 16 to 30 cores
> and
> >> 50 GB of ram
> >>
> >>
> >> On Wed, 11 Mar 2015 8:55 am Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
> >>>
> >>> Depending on your cluster setup (cores, memory), you need to specify
> the
> >>> parallelism/repartition the data.
> >>>
> >>> Thanks
> >>> Best Regards
> >>>
> >>> On Wed, Mar 11, 2015 at 12:18 PM, Sean Barzilay <
> sesnbarzilay@gmail.com>
> >>> wrote:
> >>>>
> >>>> Hi I am currently using spark 1.3.0-snapshot to run the fpg algorithm
> >>>> from the mllib library. When I am trying to run the algorithm over a
> large
> >>>> basket(over 1000 items) the program seems to never finish. Did anyone
> find a
> >>>> workaround for this problem?
> >>>
> >>>
> >
>

Re: Spark fpg large basket

[Sean Owen <so...@cloudera.com>]

I don't think there is enough information here. Where is the program
spending its time? where does it "stop"? how many partitions are
there?

On Wed, Mar 11, 2015 at 7:10 AM, Akhil Das <ak...@sigmoidanalytics.com> wrote:
> You need to set spark.cores.max to a number say 16, so that on all 4
> machines the tasks will get distributed evenly, Another thing would be to
> set spark.default.parallelism if you haven't tried already.
>
> Thanks
> Best Regards
>
> On Wed, Mar 11, 2015 at 12:27 PM, Sean Barzilay <se...@gmail.com>
> wrote:
>>
>> I am running on a 4 workers cluster each having between 16 to 30 cores and
>> 50 GB of ram
>>
>>
>> On Wed, 11 Mar 2015 8:55 am Akhil Das <ak...@sigmoidanalytics.com> wrote:
>>>
>>> Depending on your cluster setup (cores, memory), you need to specify the
>>> parallelism/repartition the data.
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Wed, Mar 11, 2015 at 12:18 PM, Sean Barzilay <se...@gmail.com>
>>> wrote:
>>>>
>>>> Hi I am currently using spark 1.3.0-snapshot to run the fpg algorithm
>>>> from the mllib library. When I am trying to run the algorithm over a large
>>>> basket(over 1000 items) the program seems to never finish. Did anyone find a
>>>> workaround for this problem?
>>>
>>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org

Re: Spark fpg large basket

[Akhil Das <ak...@sigmoidanalytics.com>]

You need to set spark.cores.max to a number say 16, so that on all 4
machines the tasks will get distributed evenly, Another thing would be to
set spark.default.parallelism if you haven't tried already.

Thanks
Best Regards

On Wed, Mar 11, 2015 at 12:27 PM, Sean Barzilay <se...@gmail.com>
wrote:

> I am running on a 4 workers cluster each having between 16 to 30 cores and
> 50 GB of ram
>
> On Wed, 11 Mar 2015 8:55 am Akhil Das <ak...@sigmoidanalytics.com> wrote:
>
>> Depending on your cluster setup (cores, memory), you need to specify the
>> parallelism/repartition the data.
>>
>> Thanks
>> Best Regards
>>
>> On Wed, Mar 11, 2015 at 12:18 PM, Sean Barzilay <se...@gmail.com>
>> wrote:
>>
>>> Hi I am currently using spark 1.3.0-snapshot to run the fpg algorithm
>>> from the mllib library. When I am trying to run the algorithm over a large
>>> basket(over 1000 items) the program seems to never finish. Did anyone find
>>> a workaround for this problem?
>>>
>>
>>

Re: Spark fpg large basket

[Akhil Das <ak...@sigmoidanalytics.com>]

Depending on your cluster setup (cores, memory), you need to specify the
parallelism/repartition the data.

Thanks
Best Regards

On Wed, Mar 11, 2015 at 12:18 PM, Sean Barzilay <se...@gmail.com>
wrote:

> Hi I am currently using spark 1.3.0-snapshot to run the fpg algorithm from
> the mllib library. When I am trying to run the algorithm over a large
> basket(over 1000 items) the program seems to never finish. Did anyone find
> a workaround for this problem?
>