You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ascot Moss <as...@gmail.com> on 2016/07/28 00:49:46 UTC
saveAsTextFile at treeEnsembleModels.scala:447, took 2.513396 s Killed
Hi,
Please help!
When saving the model, I got following error and cannot save the model to
hdfs:
(my source code, my spark is v1.6.2)
my_model.save(sc, "/my_model")
-----------------------------------------
16/07/28 08:36:19 INFO TaskSchedulerImpl: Removed TaskSet 69.0, whose tasks
have all completed, from pool
16/07/28 08:36:19 INFO DAGScheduler: ResultStage 69 (saveAsTextFile at
treeEnsembleModels.scala:447) finished in 0.901 s
16/07/28 08:36:19 INFO DAGScheduler: Job 38 finished: saveAsTextFile at
treeEnsembleModels.scala:447, took 2.513396 s
Killed
-----------------------------------------
Q1: Is there any limitation on saveAsTextFile?
Q2: or where to find the error log file location?
Regards
Re: saveAsTextFile at treeEnsembleModels.scala:447, took 2.513396 s Killed
Posted by Ascot Moss <as...@gmail.com>.
Hi,
Thanks for your reply.
permissions (access) is not an issue in my case, it is because this issue
only happened when the bigger input file was used to generate the model,
i.e. with smaller input(s) all worked well. It seems to me that ".save"
cannot save big file.
Q1: Any idea about the size limit that ".save" can handle?
Q2: Any idea about how to check the size model that will be saved vis
".save" ?
Regards
On Thu, Jul 28, 2016 at 4:19 PM, Spico Florin <sp...@gmail.com> wrote:
> Hi!
> There are many reasons that your task is failed. One could be that you
> don't have proper permissions (access) to hdfs with your user. Please
> check your user rights to write in hdfs. Please have a look also :
>
> http://stackoverflow.com/questions/27427042/spark-unable-to-save-in-hadoop-permission-denied-for-user
> I hope it jelps.
> Florin
>
>
> On Thu, Jul 28, 2016 at 3:49 AM, Ascot Moss <as...@gmail.com> wrote:
>
>>
>> Hi,
>>
>> Please help!
>>
>> When saving the model, I got following error and cannot save the model to
>> hdfs:
>>
>> (my source code, my spark is v1.6.2)
>> my_model.save(sc, "/my_model")
>>
>> -----------------------------------------
>> 16/07/28 08:36:19 INFO TaskSchedulerImpl: Removed TaskSet 69.0, whose
>> tasks have all completed, from pool
>>
>> 16/07/28 08:36:19 INFO DAGScheduler: ResultStage 69 (saveAsTextFile at
>> treeEnsembleModels.scala:447) finished in 0.901 s
>>
>> 16/07/28 08:36:19 INFO DAGScheduler: Job 38 finished: saveAsTextFile at
>> treeEnsembleModels.scala:447, took 2.513396 s
>>
>> Killed
>> -----------------------------------------
>>
>>
>> Q1: Is there any limitation on saveAsTextFile?
>> Q2: or where to find the error log file location?
>>
>> Regards
>>
>>
>>
>>
>>
>