You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crail.apache.org by Jonas Pfefferle <pe...@japf.ch> on 2019/06/19 07:17:30 UTC
Re: Crail used as type 2 storage for TeraSort does not catch
the "finished" signal
Hi David,
Unfortunately, if you use Crail for input/output with Spark this is
expected. The problem is Spark never closes the filesystem correctly. I
haven't look into this lately but if I remember correctly there was no easy
way otherwise to determine Spark is about to close.
Regards,
Jonas
On Tue, 18 Jun 2019 22:17:16 +0000
David Crespi <da...@storedgesystems.com> wrote:
> Hi,
> I’m running Crail as the temporary backend storage for Terasort.
> After each section (TeraGen, TeraSort, TeraVerify)
> the program waits until a Cntl-C is given, then moves on to the next
>section. Is this the expected behavior, or is
> this a bug?
>
> Here’s a small snippet of the output. Terasort waits where the
>bolded “Number of records” is listed, until
> The ^c is given. Each of the three programs does the same, but the
>program does finish without errors.
>
>
> 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:
>TaskSet_1.0, runningTasks: 1
> 19/06/18 15:13:19 INFO TaskSetManager: Finished task 1.0 in stage
>1.0 (TID 3) in 142 ms on 192.168.3.10 (executor 4) (1/2)
> 19/06/18 15:13:19 INFO BlockManagerInfo: Added broadcast_1_piece0 in
>memory on 192.168.3.12:34011 (size: 1825.0 B, free: 366.3 MB)
> 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:
>TaskSet_1.0, runningTasks: 0
> 19/06/18 15:13:19 INFO TaskSetManager: Finished task 0.0 in stage
>1.0 (TID 2) in 977 ms on 192.168.3.12 (executor 3) (2/2)
> 19/06/18 15:13:19 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose
>tasks have all completed, from pool
> 19/06/18 15:13:19 INFO DAGScheduler: ResultStage 1 (count at
>TeraGen.scala:94) finished in 0.995 s
> 19/06/18 15:13:19 DEBUG DAGScheduler: After removal of stage 1,
>remaining stages = 0
> 19/06/18 15:13:19 INFO DAGScheduler: Job 1 finished: count at
>TeraGen.scala:94, took 1.003537 s
> Number of records written: 10000
> ^C19/06/18 15:13:36 INFO SparkContext: Invoking stop() from shutdown
>hook
>
> Regards,
>
> David
>
>
Re: Crail used as type 2 storage for TeraSort does not catch the
"finished" signal
Posted by Patrick Stuedi <ps...@gmail.com>.
Yes, please if you can open a bug JIRA that'd be great!
On Wed, Jun 19, 2019 at 7:51 PM David Crespi <
david.crespi@storedgesystems.com> wrote:
> Patrick, would you like me to open up a bug, or do you have this?
>
>
>
> Regards,
>
>
>
> David
>
>
>
> ________________________________
> From: Patrick Stuedi <ps...@gmail.com>
> Sent: Wednesday, June 19, 2019 6:40:10 AM
> To: dev@crail.apache.org
> Cc: Jonas Pfefferle; dev@crail.incubator.apache.org
> Subject: Re: Crail used as type 2 storage for TeraSort does not catch the
> "finished" signal
>
> Ok. The closing issue you reported is a bug I think. And the changes of
> Adrian are targeting the shuffle plugin only, so they're not fixing the
> closing issue. We need to look into this more carefully, right now there is
> no easy fix I guess, sorry.
>
> -Patrick
>
> On Wed, Jun 19, 2019 at 3:32 PM David Crespi <
> david.crespi@storedgesystems.com> wrote:
>
> > Hi Patrick, I am attempting to use the crail shuffle plugin… different
> > thread.
> >
> > This was just about using crail for input/output. I have HDFS set up
> along
> >
> > side, to also benchmark the different configs.
> >
> >
> >
> > Regards,
> >
> >
> >
> > David
> >
> >
> >
> >
> >
> > ________________________________
> > From: Patrick Stuedi <ps...@gmail.com>
> > Sent: Wednesday, June 19, 2019 6:29:11 AM
> > To: dev@crail.apache.org
> > Cc: Jonas Pfefferle; dev@crail.incubator.apache.org
> > Subject: Re: Crail used as type 2 storage for TeraSort does not catch the
> > "finished" signal
> >
> > The closing issue is related to using Crail for input/output. The changes
> > Adrian made just earlier today are changes on the shuffle plugin. Are you
> > using the Crail shuffle plugin at all? If not then the changes of Adrian
> > are not relevant to you.
> >
> > -Patrick
> >
> > On Wed, Jun 19, 2019 at 3:22 PM David Crespi <
> > david.crespi@storedgesystems.com> wrote:
> >
> > > Thanks for the description. It seemed odd that it behaved this way as
> > > HDFS does close as
> > >
> > > expected, so I wasn’t sure. Wouldn’t this change the Terasort benchmark
> > > numbers?
> > >
> > >
> > >
> > > Regards,
> > >
> > >
> > >
> > > David
> > >
> > >
> > >
> > > C: 714-476-2692
> > >
> > >
> > >
> > > ________________________________
> > > From: Jonas Pfefferle <pe...@japf.ch>
> > > Sent: Wednesday, June 19, 2019 12:17:30 AM
> > > To: dev@crail.apache.org; David Crespi; dev@crail.incubator.apache.org
> > > Subject: Re: Crail used as type 2 storage for TeraSort does not catch
> the
> > > "finished" signal
> > >
> > > Hi David,
> > >
> > >
> > > Unfortunately, if you use Crail for input/output with Spark this is
> > > expected. The problem is Spark never closes the filesystem correctly. I
> > > haven't look into this lately but if I remember correctly there was no
> > easy
> > > way otherwise to determine Spark is about to close.
> > >
> > > Regards,
> > > Jonas
> > >
> > > On Tue, 18 Jun 2019 22:17:16 +0000
> > > David Crespi <da...@storedgesystems.com> wrote:
> > > > Hi,
> > > > I’m running Crail as the temporary backend storage for Terasort.
> > > > After each section (TeraGen, TeraSort, TeraVerify)
> > > > the program waits until a Cntl-C is given, then moves on to the next
> > > >section. Is this the expected behavior, or is
> > > > this a bug?
> > > >
> > > > Here’s a small snippet of the output. Terasort waits where the
> > > >bolded “Number of records” is listed, until
> > > > The ^c is given. Each of the three programs does the same, but the
> > > >program does finish without errors.
> > > >
> > > >
> > > > 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:
> > > >TaskSet_1.0, runningTasks: 1
> > > > 19/06/18 15:13:19 INFO TaskSetManager: Finished task 1.0 in stage
> > > >1.0 (TID 3) in 142 ms on 192.168.3.10 (executor 4) (1/2)
> > > > 19/06/18 15:13:19 INFO BlockManagerInfo: Added broadcast_1_piece0 in
> > > >memory on 192.168.3.12:34011 (size: 1825.0 B, free: 366.3 MB)
> > > > 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:
> > > >TaskSet_1.0, runningTasks: 0
> > > > 19/06/18 15:13:19 INFO TaskSetManager: Finished task 0.0 in stage
> > > >1.0 (TID 2) in 977 ms on 192.168.3.12 (executor 3) (2/2)
> > > > 19/06/18 15:13:19 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose
> > > >tasks have all completed, from pool
> > > > 19/06/18 15:13:19 INFO DAGScheduler: ResultStage 1 (count at
> > > >TeraGen.scala:94) finished in 0.995 s
> > > > 19/06/18 15:13:19 DEBUG DAGScheduler: After removal of stage 1,
> > > >remaining stages = 0
> > > > 19/06/18 15:13:19 INFO DAGScheduler: Job 1 finished: count at
> > > >TeraGen.scala:94, took 1.003537 s
> > > > Number of records written: 10000
> > > > ^C19/06/18 15:13:36 INFO SparkContext: Invoking stop() from shutdown
> > > >hook
> > > >
> > > > Regards,
> > > >
> > > > David
> > > >
> > > >
> > >
> > >
> >
>
RE: Crail used as type 2 storage for TeraSort does not catch the
"finished" signal
Posted by David Crespi <da...@storedgesystems.com>.
Patrick, would you like me to open up a bug, or do you have this?
Regards,
David
________________________________
From: Patrick Stuedi <ps...@gmail.com>
Sent: Wednesday, June 19, 2019 6:40:10 AM
To: dev@crail.apache.org
Cc: Jonas Pfefferle; dev@crail.incubator.apache.org
Subject: Re: Crail used as type 2 storage for TeraSort does not catch the "finished" signal
Ok. The closing issue you reported is a bug I think. And the changes of
Adrian are targeting the shuffle plugin only, so they're not fixing the
closing issue. We need to look into this more carefully, right now there is
no easy fix I guess, sorry.
-Patrick
On Wed, Jun 19, 2019 at 3:32 PM David Crespi <
david.crespi@storedgesystems.com> wrote:
> Hi Patrick, I am attempting to use the crail shuffle plugin… different
> thread.
>
> This was just about using crail for input/output. I have HDFS set up along
>
> side, to also benchmark the different configs.
>
>
>
> Regards,
>
>
>
> David
>
>
>
>
>
> ________________________________
> From: Patrick Stuedi <ps...@gmail.com>
> Sent: Wednesday, June 19, 2019 6:29:11 AM
> To: dev@crail.apache.org
> Cc: Jonas Pfefferle; dev@crail.incubator.apache.org
> Subject: Re: Crail used as type 2 storage for TeraSort does not catch the
> "finished" signal
>
> The closing issue is related to using Crail for input/output. The changes
> Adrian made just earlier today are changes on the shuffle plugin. Are you
> using the Crail shuffle plugin at all? If not then the changes of Adrian
> are not relevant to you.
>
> -Patrick
>
> On Wed, Jun 19, 2019 at 3:22 PM David Crespi <
> david.crespi@storedgesystems.com> wrote:
>
> > Thanks for the description. It seemed odd that it behaved this way as
> > HDFS does close as
> >
> > expected, so I wasn’t sure. Wouldn’t this change the Terasort benchmark
> > numbers?
> >
> >
> >
> > Regards,
> >
> >
> >
> > David
> >
> >
> >
> > C: 714-476-2692
> >
> >
> >
> > ________________________________
> > From: Jonas Pfefferle <pe...@japf.ch>
> > Sent: Wednesday, June 19, 2019 12:17:30 AM
> > To: dev@crail.apache.org; David Crespi; dev@crail.incubator.apache.org
> > Subject: Re: Crail used as type 2 storage for TeraSort does not catch the
> > "finished" signal
> >
> > Hi David,
> >
> >
> > Unfortunately, if you use Crail for input/output with Spark this is
> > expected. The problem is Spark never closes the filesystem correctly. I
> > haven't look into this lately but if I remember correctly there was no
> easy
> > way otherwise to determine Spark is about to close.
> >
> > Regards,
> > Jonas
> >
> > On Tue, 18 Jun 2019 22:17:16 +0000
> > David Crespi <da...@storedgesystems.com> wrote:
> > > Hi,
> > > I’m running Crail as the temporary backend storage for Terasort.
> > > After each section (TeraGen, TeraSort, TeraVerify)
> > > the program waits until a Cntl-C is given, then moves on to the next
> > >section. Is this the expected behavior, or is
> > > this a bug?
> > >
> > > Here’s a small snippet of the output. Terasort waits where the
> > >bolded “Number of records” is listed, until
> > > The ^c is given. Each of the three programs does the same, but the
> > >program does finish without errors.
> > >
> > >
> > > 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:
> > >TaskSet_1.0, runningTasks: 1
> > > 19/06/18 15:13:19 INFO TaskSetManager: Finished task 1.0 in stage
> > >1.0 (TID 3) in 142 ms on 192.168.3.10 (executor 4) (1/2)
> > > 19/06/18 15:13:19 INFO BlockManagerInfo: Added broadcast_1_piece0 in
> > >memory on 192.168.3.12:34011 (size: 1825.0 B, free: 366.3 MB)
> > > 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:
> > >TaskSet_1.0, runningTasks: 0
> > > 19/06/18 15:13:19 INFO TaskSetManager: Finished task 0.0 in stage
> > >1.0 (TID 2) in 977 ms on 192.168.3.12 (executor 3) (2/2)
> > > 19/06/18 15:13:19 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose
> > >tasks have all completed, from pool
> > > 19/06/18 15:13:19 INFO DAGScheduler: ResultStage 1 (count at
> > >TeraGen.scala:94) finished in 0.995 s
> > > 19/06/18 15:13:19 DEBUG DAGScheduler: After removal of stage 1,
> > >remaining stages = 0
> > > 19/06/18 15:13:19 INFO DAGScheduler: Job 1 finished: count at
> > >TeraGen.scala:94, took 1.003537 s
> > > Number of records written: 10000
> > > ^C19/06/18 15:13:36 INFO SparkContext: Invoking stop() from shutdown
> > >hook
> > >
> > > Regards,
> > >
> > > David
> > >
> > >
> >
> >
>
Re: Crail used as type 2 storage for TeraSort does not catch the
"finished" signal
Posted by Patrick Stuedi <ps...@gmail.com>.
Ok. The closing issue you reported is a bug I think. And the changes of
Adrian are targeting the shuffle plugin only, so they're not fixing the
closing issue. We need to look into this more carefully, right now there is
no easy fix I guess, sorry.
-Patrick
On Wed, Jun 19, 2019 at 3:32 PM David Crespi <
david.crespi@storedgesystems.com> wrote:
> Hi Patrick, I am attempting to use the crail shuffle plugin… different
> thread.
>
> This was just about using crail for input/output. I have HDFS set up along
>
> side, to also benchmark the different configs.
>
>
>
> Regards,
>
>
>
> David
>
>
>
>
>
> ________________________________
> From: Patrick Stuedi <ps...@gmail.com>
> Sent: Wednesday, June 19, 2019 6:29:11 AM
> To: dev@crail.apache.org
> Cc: Jonas Pfefferle; dev@crail.incubator.apache.org
> Subject: Re: Crail used as type 2 storage for TeraSort does not catch the
> "finished" signal
>
> The closing issue is related to using Crail for input/output. The changes
> Adrian made just earlier today are changes on the shuffle plugin. Are you
> using the Crail shuffle plugin at all? If not then the changes of Adrian
> are not relevant to you.
>
> -Patrick
>
> On Wed, Jun 19, 2019 at 3:22 PM David Crespi <
> david.crespi@storedgesystems.com> wrote:
>
> > Thanks for the description. It seemed odd that it behaved this way as
> > HDFS does close as
> >
> > expected, so I wasn’t sure. Wouldn’t this change the Terasort benchmark
> > numbers?
> >
> >
> >
> > Regards,
> >
> >
> >
> > David
> >
> >
> >
> > C: 714-476-2692
> >
> >
> >
> > ________________________________
> > From: Jonas Pfefferle <pe...@japf.ch>
> > Sent: Wednesday, June 19, 2019 12:17:30 AM
> > To: dev@crail.apache.org; David Crespi; dev@crail.incubator.apache.org
> > Subject: Re: Crail used as type 2 storage for TeraSort does not catch the
> > "finished" signal
> >
> > Hi David,
> >
> >
> > Unfortunately, if you use Crail for input/output with Spark this is
> > expected. The problem is Spark never closes the filesystem correctly. I
> > haven't look into this lately but if I remember correctly there was no
> easy
> > way otherwise to determine Spark is about to close.
> >
> > Regards,
> > Jonas
> >
> > On Tue, 18 Jun 2019 22:17:16 +0000
> > David Crespi <da...@storedgesystems.com> wrote:
> > > Hi,
> > > I’m running Crail as the temporary backend storage for Terasort.
> > > After each section (TeraGen, TeraSort, TeraVerify)
> > > the program waits until a Cntl-C is given, then moves on to the next
> > >section. Is this the expected behavior, or is
> > > this a bug?
> > >
> > > Here’s a small snippet of the output. Terasort waits where the
> > >bolded “Number of records” is listed, until
> > > The ^c is given. Each of the three programs does the same, but the
> > >program does finish without errors.
> > >
> > >
> > > 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:
> > >TaskSet_1.0, runningTasks: 1
> > > 19/06/18 15:13:19 INFO TaskSetManager: Finished task 1.0 in stage
> > >1.0 (TID 3) in 142 ms on 192.168.3.10 (executor 4) (1/2)
> > > 19/06/18 15:13:19 INFO BlockManagerInfo: Added broadcast_1_piece0 in
> > >memory on 192.168.3.12:34011 (size: 1825.0 B, free: 366.3 MB)
> > > 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:
> > >TaskSet_1.0, runningTasks: 0
> > > 19/06/18 15:13:19 INFO TaskSetManager: Finished task 0.0 in stage
> > >1.0 (TID 2) in 977 ms on 192.168.3.12 (executor 3) (2/2)
> > > 19/06/18 15:13:19 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose
> > >tasks have all completed, from pool
> > > 19/06/18 15:13:19 INFO DAGScheduler: ResultStage 1 (count at
> > >TeraGen.scala:94) finished in 0.995 s
> > > 19/06/18 15:13:19 DEBUG DAGScheduler: After removal of stage 1,
> > >remaining stages = 0
> > > 19/06/18 15:13:19 INFO DAGScheduler: Job 1 finished: count at
> > >TeraGen.scala:94, took 1.003537 s
> > > Number of records written: 10000
> > > ^C19/06/18 15:13:36 INFO SparkContext: Invoking stop() from shutdown
> > >hook
> > >
> > > Regards,
> > >
> > > David
> > >
> > >
> >
> >
>
RE: Crail used as type 2 storage for TeraSort does not catch the
"finished" signal
Posted by David Crespi <da...@storedgesystems.com>.
Hi Patrick, I am attempting to use the crail shuffle plugin… different thread.
This was just about using crail for input/output. I have HDFS set up along
side, to also benchmark the different configs.
Regards,
David
________________________________
From: Patrick Stuedi <ps...@gmail.com>
Sent: Wednesday, June 19, 2019 6:29:11 AM
To: dev@crail.apache.org
Cc: Jonas Pfefferle; dev@crail.incubator.apache.org
Subject: Re: Crail used as type 2 storage for TeraSort does not catch the "finished" signal
The closing issue is related to using Crail for input/output. The changes
Adrian made just earlier today are changes on the shuffle plugin. Are you
using the Crail shuffle plugin at all? If not then the changes of Adrian
are not relevant to you.
-Patrick
On Wed, Jun 19, 2019 at 3:22 PM David Crespi <
david.crespi@storedgesystems.com> wrote:
> Thanks for the description. It seemed odd that it behaved this way as
> HDFS does close as
>
> expected, so I wasn’t sure. Wouldn’t this change the Terasort benchmark
> numbers?
>
>
>
> Regards,
>
>
>
> David
>
>
>
> C: 714-476-2692
>
>
>
> ________________________________
> From: Jonas Pfefferle <pe...@japf.ch>
> Sent: Wednesday, June 19, 2019 12:17:30 AM
> To: dev@crail.apache.org; David Crespi; dev@crail.incubator.apache.org
> Subject: Re: Crail used as type 2 storage for TeraSort does not catch the
> "finished" signal
>
> Hi David,
>
>
> Unfortunately, if you use Crail for input/output with Spark this is
> expected. The problem is Spark never closes the filesystem correctly. I
> haven't look into this lately but if I remember correctly there was no easy
> way otherwise to determine Spark is about to close.
>
> Regards,
> Jonas
>
> On Tue, 18 Jun 2019 22:17:16 +0000
> David Crespi <da...@storedgesystems.com> wrote:
> > Hi,
> > I’m running Crail as the temporary backend storage for Terasort.
> > After each section (TeraGen, TeraSort, TeraVerify)
> > the program waits until a Cntl-C is given, then moves on to the next
> >section. Is this the expected behavior, or is
> > this a bug?
> >
> > Here’s a small snippet of the output. Terasort waits where the
> >bolded “Number of records” is listed, until
> > The ^c is given. Each of the three programs does the same, but the
> >program does finish without errors.
> >
> >
> > 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:
> >TaskSet_1.0, runningTasks: 1
> > 19/06/18 15:13:19 INFO TaskSetManager: Finished task 1.0 in stage
> >1.0 (TID 3) in 142 ms on 192.168.3.10 (executor 4) (1/2)
> > 19/06/18 15:13:19 INFO BlockManagerInfo: Added broadcast_1_piece0 in
> >memory on 192.168.3.12:34011 (size: 1825.0 B, free: 366.3 MB)
> > 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:
> >TaskSet_1.0, runningTasks: 0
> > 19/06/18 15:13:19 INFO TaskSetManager: Finished task 0.0 in stage
> >1.0 (TID 2) in 977 ms on 192.168.3.12 (executor 3) (2/2)
> > 19/06/18 15:13:19 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose
> >tasks have all completed, from pool
> > 19/06/18 15:13:19 INFO DAGScheduler: ResultStage 1 (count at
> >TeraGen.scala:94) finished in 0.995 s
> > 19/06/18 15:13:19 DEBUG DAGScheduler: After removal of stage 1,
> >remaining stages = 0
> > 19/06/18 15:13:19 INFO DAGScheduler: Job 1 finished: count at
> >TeraGen.scala:94, took 1.003537 s
> > Number of records written: 10000
> > ^C19/06/18 15:13:36 INFO SparkContext: Invoking stop() from shutdown
> >hook
> >
> > Regards,
> >
> > David
> >
> >
>
>
Re: Crail used as type 2 storage for TeraSort does not catch the
"finished" signal
Posted by Patrick Stuedi <ps...@gmail.com>.
The closing issue is related to using Crail for input/output. The changes
Adrian made just earlier today are changes on the shuffle plugin. Are you
using the Crail shuffle plugin at all? If not then the changes of Adrian
are not relevant to you.
-Patrick
On Wed, Jun 19, 2019 at 3:22 PM David Crespi <
david.crespi@storedgesystems.com> wrote:
> Thanks for the description. It seemed odd that it behaved this way as
> HDFS does close as
>
> expected, so I wasn’t sure. Wouldn’t this change the Terasort benchmark
> numbers?
>
>
>
> Regards,
>
>
>
> David
>
>
>
> C: 714-476-2692
>
>
>
> ________________________________
> From: Jonas Pfefferle <pe...@japf.ch>
> Sent: Wednesday, June 19, 2019 12:17:30 AM
> To: dev@crail.apache.org; David Crespi; dev@crail.incubator.apache.org
> Subject: Re: Crail used as type 2 storage for TeraSort does not catch the
> "finished" signal
>
> Hi David,
>
>
> Unfortunately, if you use Crail for input/output with Spark this is
> expected. The problem is Spark never closes the filesystem correctly. I
> haven't look into this lately but if I remember correctly there was no easy
> way otherwise to determine Spark is about to close.
>
> Regards,
> Jonas
>
> On Tue, 18 Jun 2019 22:17:16 +0000
> David Crespi <da...@storedgesystems.com> wrote:
> > Hi,
> > I’m running Crail as the temporary backend storage for Terasort.
> > After each section (TeraGen, TeraSort, TeraVerify)
> > the program waits until a Cntl-C is given, then moves on to the next
> >section. Is this the expected behavior, or is
> > this a bug?
> >
> > Here’s a small snippet of the output. Terasort waits where the
> >bolded “Number of records” is listed, until
> > The ^c is given. Each of the three programs does the same, but the
> >program does finish without errors.
> >
> >
> > 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:
> >TaskSet_1.0, runningTasks: 1
> > 19/06/18 15:13:19 INFO TaskSetManager: Finished task 1.0 in stage
> >1.0 (TID 3) in 142 ms on 192.168.3.10 (executor 4) (1/2)
> > 19/06/18 15:13:19 INFO BlockManagerInfo: Added broadcast_1_piece0 in
> >memory on 192.168.3.12:34011 (size: 1825.0 B, free: 366.3 MB)
> > 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:
> >TaskSet_1.0, runningTasks: 0
> > 19/06/18 15:13:19 INFO TaskSetManager: Finished task 0.0 in stage
> >1.0 (TID 2) in 977 ms on 192.168.3.12 (executor 3) (2/2)
> > 19/06/18 15:13:19 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose
> >tasks have all completed, from pool
> > 19/06/18 15:13:19 INFO DAGScheduler: ResultStage 1 (count at
> >TeraGen.scala:94) finished in 0.995 s
> > 19/06/18 15:13:19 DEBUG DAGScheduler: After removal of stage 1,
> >remaining stages = 0
> > 19/06/18 15:13:19 INFO DAGScheduler: Job 1 finished: count at
> >TeraGen.scala:94, took 1.003537 s
> > Number of records written: 10000
> > ^C19/06/18 15:13:36 INFO SparkContext: Invoking stop() from shutdown
> >hook
> >
> > Regards,
> >
> > David
> >
> >
>
>
RE: Crail used as type 2 storage for TeraSort does not catch the
"finished" signal
Posted by David Crespi <da...@storedgesystems.com>.
Thanks for the description. It seemed odd that it behaved this way as HDFS does close as
expected, so I wasn’t sure. Wouldn’t this change the Terasort benchmark numbers?
Regards,
David
C: 714-476-2692
________________________________
From: Jonas Pfefferle <pe...@japf.ch>
Sent: Wednesday, June 19, 2019 12:17:30 AM
To: dev@crail.apache.org; David Crespi; dev@crail.incubator.apache.org
Subject: Re: Crail used as type 2 storage for TeraSort does not catch the "finished" signal
Hi David,
Unfortunately, if you use Crail for input/output with Spark this is
expected. The problem is Spark never closes the filesystem correctly. I
haven't look into this lately but if I remember correctly there was no easy
way otherwise to determine Spark is about to close.
Regards,
Jonas
On Tue, 18 Jun 2019 22:17:16 +0000
David Crespi <da...@storedgesystems.com> wrote:
> Hi,
> I’m running Crail as the temporary backend storage for Terasort.
> After each section (TeraGen, TeraSort, TeraVerify)
> the program waits until a Cntl-C is given, then moves on to the next
>section. Is this the expected behavior, or is
> this a bug?
>
> Here’s a small snippet of the output. Terasort waits where the
>bolded “Number of records” is listed, until
> The ^c is given. Each of the three programs does the same, but the
>program does finish without errors.
>
>
> 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:
>TaskSet_1.0, runningTasks: 1
> 19/06/18 15:13:19 INFO TaskSetManager: Finished task 1.0 in stage
>1.0 (TID 3) in 142 ms on 192.168.3.10 (executor 4) (1/2)
> 19/06/18 15:13:19 INFO BlockManagerInfo: Added broadcast_1_piece0 in
>memory on 192.168.3.12:34011 (size: 1825.0 B, free: 366.3 MB)
> 19/06/18 15:13:19 DEBUG TaskSchedulerImpl: parentName: , name:
>TaskSet_1.0, runningTasks: 0
> 19/06/18 15:13:19 INFO TaskSetManager: Finished task 0.0 in stage
>1.0 (TID 2) in 977 ms on 192.168.3.12 (executor 3) (2/2)
> 19/06/18 15:13:19 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose
>tasks have all completed, from pool
> 19/06/18 15:13:19 INFO DAGScheduler: ResultStage 1 (count at
>TeraGen.scala:94) finished in 0.995 s
> 19/06/18 15:13:19 DEBUG DAGScheduler: After removal of stage 1,
>remaining stages = 0
> 19/06/18 15:13:19 INFO DAGScheduler: Job 1 finished: count at
>TeraGen.scala:94, took 1.003537 s
> Number of records written: 10000
> ^C19/06/18 15:13:36 INFO SparkContext: Invoking stop() from shutdown
>hook
>
> Regards,
>
> David
>
>