You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Sumona Routh <su...@gmail.com> on 2017/05/30 17:29:30 UTC
Re: Random Forest hangs without trace of error
Hi Morten,
Were you able to resolve your issue with RandomForest? I am having similar
issues with a newly trained model (that does have larger number of trees,
smaller minInstancesPerNode, which is by design to produce the best
performing model).
I wanted to get some feedback on how you solved your issue before I post a
separate question.
Thanks!
Sumona
On Sun, Dec 11, 2016 at 4:10 AM Marco Mistroni <mm...@gmail.com> wrote:
> OK. Did u change spark version? Java/scala/python version?
> Have u tried with different versions of any of the above?
> Hope this helps
> Kr
>
> On 10 Dec 2016 10:37 pm, "Morten Hornbech" <mo...@datasolvr.com> wrote:
>
>> I haven’t actually experienced any non-determinism. We have nightly
>> integration tests comparing output from random forests with no variations.
>>
>> The workaround we will probably try is to split the dataset, either
>> randomly or on one of the variables, and then train a forest on each
>> partition, which should then be sufficiently small.
>>
>> I hope to be able to provide a good repro case in some weeks. If the
>> problem was in our own code I will also post it in this thread.
>>
>> Morten
>>
>> Den 10. dec. 2016 kl. 23.25 skrev Marco Mistroni <mm...@gmail.com>:
>>
>> Hello Morten
>> ok.
>> afaik there is a tiny bit of randomness in these ML algorithms (pls
>> anyone correct me if i m wrong).
>> In fact if you run your RDF code multiple times, it will not give you
>> EXACTLY the same results (though accuracy and errors should me more or less
>> similar)..at least this is what i found when playing around with
>> RDF and decision trees and other ML algorithms
>>
>> If RDF is not a must for your usecase, could you try 'scale back' to
>> Decision Trees and see if you still get intermittent failures?
>> this at least to exclude issues with the data
>>
>> hth
>> marco
>>
>> On Sat, Dec 10, 2016 at 5:20 PM, Morten Hornbech <mo...@datasolvr.com>
>> wrote:
>>
>>> Already did. There are no issues with smaller samples. I am running this
>>> in a cluster of three t2.large instances on aws.
>>>
>>> I have tried to find the threshold where the error occurs, but it is not
>>> a single factor causing it. Input size and subsampling rate seems to be
>>> most significant, and number of trees the least.
>>>
>>> I have also tried running on a test frame of randomized numbers with the
>>> same number of rows, and could not reproduce the problem here.
>>>
>>> By the way maxDepth is 5 and maxBins is 32.
>>>
>>> I will probably need to leave this for a few weeks to focus on more
>>> short-term stuff, but I will write here if I solve it or reproduce it more
>>> consistently.
>>>
>>> Morten
>>>
>>> Den 10. dec. 2016 kl. 17.29 skrev Marco Mistroni <mm...@gmail.com>:
>>>
>>> Hi
>>> Bring back samples to 1k range to debug....or as suggested reduce tree
>>> and bins.... had rdd running on same size data with no issues.....or send
>>> me some sample code and data and I try it out on my ec2 instance ...
>>> Kr
>>>
>>> On 10 Dec 2016 3:16 am, "Md. Rezaul Karim" <
>>> rezaul.karim@insight-centre.org> wrote:
>>>
>>>> I had similar experience last week. Even I could not find any error
>>>> trace.
>>>>
>>>> Later on, I did the following to get rid of the problem:
>>>> i) I downgraded to Spark 2.0.0
>>>> ii) Decreased the value of maxBins and maxDepth
>>>>
>>>> Additionally, make sure that you set the featureSubsetStrategy as
>>>> "auto" to let the algorithm choose the best feature subset strategy
>>>> for your data. Finally, set the impurity as "gini" for the information
>>>> gain.
>>>>
>>>> However, setting the value of no. of trees to just 1 does not give you
>>>> either real advantage of the forest neither better predictive performance.
>>>>
>>>>
>>>>
>>>> Best,
>>>> Karim
>>>>
>>>>
>>>> On Dec 9, 2016 11:29 PM, "mhornbech" <mo...@datasolvr.com> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I have spent quite some time trying to debug an issue with the Random
>>>>> Forest
>>>>> algorithm on Spark 2.0.2. The input dataset is relatively large at
>>>>> around
>>>>> 600k rows and 200MB, but I use subsampling to make each tree
>>>>> manageable.
>>>>> However even with only 1 tree and a low sample rate of 0.05 the job
>>>>> hangs at
>>>>> one of the final stages (see attached). I have checked the logs on all
>>>>> executors and the driver and find no traces of error. Could it be a
>>>>> memory
>>>>> issue even though no error appears? The error does seem sporadic to
>>>>> some
>>>>> extent so I also wondered whether it could be a data issue, that only
>>>>> occurs
>>>>> if the subsample includes the bad data rows.
>>>>>
>>>>> Please comment if you have a clue.
>>>>>
>>>>> Morten
>>>>>
>>>>> <
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/file/n28192/Sk%C3%A6rmbillede_2016-12-10_kl.png
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Random-Forest-hangs-without-trace-of-error-tp28192.html
>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>> Nabble.com <http://nabble.com/>.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>>>>
>>>>>
>>>
>>
>>
Re: Random Forest hangs without trace of error
Posted by Morten Hornbech <mo...@datasolvr.com>.
Hi Sumona
I’m afraid I never really resolved the issue. Actually I have just had to rollback an upgrade from 2.1.0 to 2.1.1 because it (for reasons unknown) reintroduced the issue in our nightly integration tests (see http://apache-spark-user-list.1001560.n3.nabble.com/Issue-upgrading-to-Spark-2-1-1-from-2-1-0-tc28660.html <http://apache-spark-user-list.1001560.n3.nabble.com/Issue-upgrading-to-Spark-2-1-1-from-2-1-0-tc28660.html>)
The “solution” for me at the time was to wave my magic Spark wand and hope for the best. That generally means
- trying increasing memory or reducing amount of memory required (smaller datasets, lower sample rate, more partitions, less caching)
- performing random changes to various other parts of the pipeline, including SQL statements and adding/removing stuff such as repartition/coalesce
- flipping around with various spark configuration settings
In the specific case I think it was the subsampling rate that did the trick.
I find issues such as this one extremely demanding to debug because they can generally not be reproduced locally. I guess you basically need to build Spark yourself with appropriate instrumentation added, and even this would probably require a very deep insight into Sparks guts.
Hanging threads are in my opinion the worst possible behaviour of a program, so if anyone can shed some light on this or provide any debugging hints it would be amazing.
Morten
> Den 30. maj 2017 kl. 19.29 skrev Sumona Routh <su...@gmail.com>:
>
> Hi Morten,
> Were you able to resolve your issue with RandomForest? I am having similar issues with a newly trained model (that does have larger number of trees, smaller minInstancesPerNode, which is by design to produce the best performing model).
>
> I wanted to get some feedback on how you solved your issue before I post a separate question.
>
> Thanks!
> Sumona
>
> On Sun, Dec 11, 2016 at 4:10 AM Marco Mistroni <mmistroni@gmail.com <ma...@gmail.com>> wrote:
> OK. Did u change spark version? Java/scala/python version?
> Have u tried with different versions of any of the above?
> Hope this helps
> Kr
>
> On 10 Dec 2016 10:37 pm, "Morten Hornbech" <morten@datasolvr.com <ma...@datasolvr.com>> wrote:
> I haven’t actually experienced any non-determinism. We have nightly integration tests comparing output from random forests with no variations.
>
> The workaround we will probably try is to split the dataset, either randomly or on one of the variables, and then train a forest on each partition, which should then be sufficiently small.
>
> I hope to be able to provide a good repro case in some weeks. If the problem was in our own code I will also post it in this thread.
>
> Morten
>
>> Den 10. dec. 2016 kl. 23.25 skrev Marco Mistroni <mmistroni@gmail.com <ma...@gmail.com>>:
>>
>> Hello Morten
>> ok.
>> afaik there is a tiny bit of randomness in these ML algorithms (pls anyone correct me if i m wrong).
>> In fact if you run your RDF code multiple times, it will not give you EXACTLY the same results (though accuracy and errors should me more or less similar)..at least this is what i found when playing around with
>> RDF and decision trees and other ML algorithms
>>
>> If RDF is not a must for your usecase, could you try 'scale back' to Decision Trees and see if you still get intermittent failures?
>> this at least to exclude issues with the data
>>
>> hth
>> marco
>>
>> On Sat, Dec 10, 2016 at 5:20 PM, Morten Hornbech <morten@datasolvr.com <ma...@datasolvr.com>> wrote:
>> Already did. There are no issues with smaller samples. I am running this in a cluster of three t2.large instances on aws.
>>
>> I have tried to find the threshold where the error occurs, but it is not a single factor causing it. Input size and subsampling rate seems to be most significant, and number of trees the least.
>>
>> I have also tried running on a test frame of randomized numbers with the same number of rows, and could not reproduce the problem here.
>>
>> By the way maxDepth is 5 and maxBins is 32.
>>
>> I will probably need to leave this for a few weeks to focus on more short-term stuff, but I will write here if I solve it or reproduce it more consistently.
>>
>> Morten
>>
>>> Den 10. dec. 2016 kl. 17.29 skrev Marco Mistroni <mmistroni@gmail.com <ma...@gmail.com>>:
>>>
>>> Hi
>>> Bring back samples to 1k range to debug....or as suggested reduce tree and bins.... had rdd running on same size data with no issues.....or send me some sample code and data and I try it out on my ec2 instance ...
>>> Kr
>>>
>>> On 10 Dec 2016 3:16 am, "Md. Rezaul Karim" <rezaul.karim@insight-centre.org <ma...@insight-centre.org>> wrote:
>>> I had similar experience last week. Even I could not find any error trace.
>>>
>>> Later on, I did the following to get rid of the problem:
>>> i) I downgraded to Spark 2.0.0
>>> ii) Decreased the value of maxBins and maxDepth
>>>
>>> Additionally, make sure that you set the featureSubsetStrategy as "auto" to let the algorithm choose the best feature subset strategy for your data. Finally, set the impurity as "gini" for the information gain.
>>>
>>> However, setting the value of no. of trees to just 1 does not give you either real advantage of the forest neither better predictive performance.
>>>
>>>
>>>
>>> Best,
>>> Karim
>>>
>>>
>>> On Dec 9, 2016 11:29 PM, "mhornbech" <morten@datasolvr.com <ma...@datasolvr.com>> wrote:
>>> Hi
>>>
>>> I have spent quite some time trying to debug an issue with the Random Forest
>>> algorithm on Spark 2.0.2. The input dataset is relatively large at around
>>> 600k rows and 200MB, but I use subsampling to make each tree manageable.
>>> However even with only 1 tree and a low sample rate of 0.05 the job hangs at
>>> one of the final stages (see attached). I have checked the logs on all
>>> executors and the driver and find no traces of error. Could it be a memory
>>> issue even though no error appears? The error does seem sporadic to some
>>> extent so I also wondered whether it could be a data issue, that only occurs
>>> if the subsample includes the bad data rows.
>>>
>>> Please comment if you have a clue.
>>>
>>> Morten
>>>
>>> <http://apache-spark-user-list.1001560.n3.nabble.com/file/n28192/Sk%C3%A6rmbillede_2016-12-10_kl.png <http://apache-spark-user-list.1001560.n3.nabble.com/file/n28192/Sk%C3%A6rmbillede_2016-12-10_kl.png>>
>>>
>>>
>>>
>>> --
>>> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Random-Forest-hangs-without-trace-of-error-tp28192.html <http://apache-spark-user-list.1001560.n3.nabble.com/Random-Forest-hangs-without-trace-of-error-tp28192.html>
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com <http://nabble.com/>.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org <ma...@spark.apache.org>
>>>
>>
>>
>