You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by RUSHIKESH RAUT <ru...@gmail.com> on 2017/02/17 09:25:10 UTC
Zeppelin unable to respond after some time
Hi all,
I am facing a issue while using Zeppelin. I am trying to load some data(not
that big data) into Zeppelin and then build some visualization on it. The
problem is that when I try to run the code first time it's working but
after some time the same code doesn't work. It remains in running state on
gui, but no logs are generated in Zeppelin logs. Also all further tasks are
hanging in pending state.
As soon as I restart Zeppelin it works. So I am guessing it's some memory
issue. I have read that Zeppelin stores the data in memory so it is
possible that it runs out of memory after some time.
How do I debug this issue? How much is the default memory that Zeppelin
takes at start? Also is there any way that I can run Zeppelin with
specified memory so that I can start the process with more memory. Because
it doesn't make sense to restart Zeppelin after every half hour
Thanks,
Rushikesh Raut
Re: Zeppelin unable to respond after some time
Posted by Shane Lee <sh...@gmail.com>.
My environment is on windows and I see this well using sparkr. It appears the R Session somehow is killed after period of inactivity and it can't be found using task manager. I would need to restart the interpreter in order to get another R Session.
> On Feb 17, 2017, at 9:11 PM, RUSHIKESH RAUT <ru...@gmail.com> wrote:
>
> Yes, it will reproduce the problem. Try running all the tutorials notebook at random, you will see it is in hanging state.
> In my experience it occurs more often with R and sparkR code, even if there is enough memory in the cluster. Just leave the spark session running and after an hour or so it stops responding.
> Happens with me everytime that when I make some visualization on top of it and share it with someone, by the time other person looks at it Zeppelin starts to act weirdly and doesn't response.
>
> On Feb 18, 2017 9:09 AM, "moon soo Lee" <mo...@apache.org> wrote:
> Hi,
>
> Download 0.7.0 -> Run R tutorial notebook repeatedly
>
> will reproduce the problem? Otherwise, can someone clarify instruction to reproduce the problem?
>
>
> Thanks,
> moon
>
>> On Sat, Feb 18, 2017 at 5:45 AM xyunfei@simuwell.com <xy...@simuwell.com> wrote:
>> Within the Scala REPL everything is working fine. Even you application session is down but you run the same code again you will see there is a new job getting created. But zeppelin has such a problem. you can do a test. run a notebook and get the job finished at the end of the notebook. Then re-run the same notebook again then it will get stucked. Running spark code on Scala REPL and Zeppelin are different
>>
>> From: Paul Brenner
>> Date: 2017-02-17 12:37
>> To: users
>> Subject: Re: Re: Zeppelin unable to respond after some time
>>
>> I don’t believe that this explains my issue. Running the Scala REPL also keeps a session alive for as long as the REPL is running. I’ve had REPLs open for days (shhhhh don’t tell anyone) that have correspondingly kept sessions alive for the same period of time with no problem. I only see this issue in zeppelin.
>>
>> We run zeppelin on a server and allow multiple users to connect, each with their own interpreters. We also find that zeppelin memory usage on the server will steadily creep up over time. Executing sys.exit in a spark paragraph, restarting the interpreter, and using yarn application -kill often will cause zeppelin to end the related interpreter process but not always. So over time we find that many zombie processes pile up and eat up resources.
>>
>> The only way to keep on top of this is to regularly login to the zeppelin server and kill zombie jobs. Here is a command that I’ve found helpful. When you know that a specific user has no active zeppelin interpreters running then execute the following:
>>
>> ps aux | grep zeppelin | grep "2BSGYY7S8" | grep java | awk -F " " '{print$2}' | xargs sudo -u yarn kill -9
>>
>> where “2BSGYY7S8" is the interpreter id (found in interpreter.json) and “yarn” is actually the name of the user that originally started zeppelin with:
>> zeppelin-daemon.sh start
>>
>> To kill every interpreter except for a specific users just flip it around with:
>> ps aux | grep zeppelin | grep -v "2BSGYY7S8" |grep -v "zeppelin.server.ZeppelinServer" | grep java | awk -F " " '{print$2}' | xargs sudo -u yarn kill -9
>>
>>
>> If I do this every few days zeppelin keeps humming along pretty smoothly most of the time.
>> Paul Brenner
>> DATA SCIENTIST
>> (217) 390-3033
>>
>>
>>
>> On Fri, Feb 17, 2017 at 3:23 PM "xyunfei@simuwell.com" <">"xyunfei@simuwell.com" > wrote:
>> The problem could be not only the resource, but the session. If you run a chunk of spark code and you should see the a running application in the spark UI, but in your code if you shut it down after the job is finished, then on the spark UI you will see the hob is finished. Within zeppelin, each job will start the spark session only once(different interpreter mode could be set if you want notebooks to share the session or not), if you closed it ,it will never restart it again. The only way to get the same code work again is to restat the interpreter or restart zeppelin. I`m not sure if I explain clearly, but hope it could help
>>
>> From: Paul Brenner
>> Date: 2017-02-17 12:14
>> To: users
>> Subject: Re: Re: Zeppelin unable to respond after some time
>>
>> I’ve definitely had this problem with jobs that don’t take all the resources on the cluster. Also, my experience matches what others have reported: just restarting zeppelin and re-runing the stuck paragraph solves the issue.
>>
>> I’ve also experienced this problem with for loops. Some for loops which write to disk but absolutely don’t have any variables that are increasing in size will hang in Zeppelin. If I run the exact same code in the scala REPL it goes through without problem.
>>
>>
>>
>> Paul Brenner
>> DATA SCIENTIST
>> (217) 390-3033
>>
>>
>>
>> On Fri, Feb 17, 2017 at 2:12 PM "xyunfei@simuwell.com" <">"xyunfei@simuwell.com" > wrote:
>> I have solved the similar issue before. You should check on spark UI and probably you will see your single job is taking all the resources. Therefore further job that submitting to the same cluster will just hang on there. When you restart zeppelin then the old job is killed and all the resource it took will be released
>>
>> xyunfei@simuwell.com
>>
>> From: RUSHIKESH RAUT
>> Date: 2017-02-17 02:29
>> To: users
>> Subject: Re: Zeppelin unable to respond after some time
>> Yes happens with r and spark codes frequently
>>
>> On Feb 17, 2017 3:25 PM, "小野圭二" <on...@gmail.com> wrote:
>> yes, almost every time.
>> There are not any special operations.
>> Just run the tutorial demos.
>> From my feeling, it happens in R demo frequently.
>>
>> 2017-02-17 18:50 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>>
>> Is it easy to reproduce it ?
>>
>> 小野圭二 <on...@gmail.com>于2017年2月17日周五 下午5:47写道：
>> I am facing on the same issue now.
>>
>> 2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT <ru...@gmail.com>:
>> Hi all,
>>
>> I am facing a issue while using Zeppelin. I am trying to load some data(not that big data) into Zeppelin and then build some visualization on it. The problem is that when I try to run the code first time it's working but after some time the same code doesn't work. It remains in running state on gui, but no logs are generated in Zeppelin logs. Also all further tasks are hanging in pending state.
>> As soon as I restart Zeppelin it works. So I am guessing it's some memory issue. I have read that Zeppelin stores the data in memory so it is possible that it runs out of memory after some time.
>> How do I debug this issue? How much is the default memory that Zeppelin takes at start? Also is there any way that I can run Zeppelin with specified memory so that I can start the process with more memory. Because it doesn't make sense to restart Zeppelin after every half hour
>>
>> Thanks,
>> Rushikesh Raut
>>
>>
>
Re: Re: Zeppelin unable to respond after some time
Posted by RUSHIKESH RAUT <ru...@gmail.com>.
Yes, it will reproduce the problem. Try running all the tutorials notebook
at random, you will see it is in hanging state.
In my experience it occurs more often with R and sparkR code, even if there
is enough memory in the cluster. Just leave the spark session running and
after an hour or so it stops responding.
Happens with me everytime that when I make some visualization on top of it
and share it with someone, by the time other person looks at it Zeppelin
starts to act weirdly and doesn't response.
On Feb 18, 2017 9:09 AM, "moon soo Lee" <mo...@apache.org> wrote:
Hi,
Download 0.7.0 -> Run R tutorial notebook repeatedly
will reproduce the problem? Otherwise, can someone clarify instruction to
reproduce the problem?
Thanks,
moon
On Sat, Feb 18, 2017 at 5:45 AM xyunfei@simuwell.com <xy...@simuwell.com>
wrote:
> Within the Scala REPL everything is working fine. Even you application
> session is down but you run the same code again you will see there is a new
> job getting created. But zeppelin has such a problem. you can do a test.
> run a notebook and get the job finished at the end of the notebook. Then
> re-run the same notebook again then it will get stucked. Running spark code
> on Scala REPL and Zeppelin are different
>
>
> *From:* Paul Brenner <pb...@placeiq.com>
> *Date:* 2017-02-17 12:37
>
> *To:* users <us...@zeppelin.apache.org>
> *Subject:* Re: Re: Zeppelin unable to respond after some time
>
> I don’t believe that this explains my issue. Running the Scala REPL also
> keeps a session alive for as long as the REPL is running. I’ve had REPLs
> open for days (shhhhh don’t tell anyone) that have correspondingly kept
> sessions alive for the same period of time with no problem. I only see this
> issue in zeppelin.
>
> We run zeppelin on a server and allow multiple users to connect, each with
> their own interpreters. We also find that zeppelin memory usage on the
> server will steadily creep up over time. Executing sys.exit in a spark
> paragraph, restarting the interpreter, and using yarn application -kill
> often will cause zeppelin to end the related interpreter process but not
> always. So over time we find that many zombie processes pile up and eat up
> resources.
>
> The only way to keep on top of this is to regularly login to the zeppelin
> server and kill zombie jobs. Here is a command that I’ve found helpful.
> When you know that a specific user has no active zeppelin interpreters
> running then execute the following:
>
> ps aux | grep zeppelin | grep "2BSGYY7S8" | grep java | awk -F " "
> '{print$2}' | xargs sudo -u yarn kill -9
>
> where “2BSGYY7S8" is the interpreter id (found in interpreter.json) and
> “yarn” is actually the name of the user that originally started zeppelin
> with:
> zeppelin-daemon.sh start
>
> To kill every interpreter except for a specific users just flip it around
> with:
> ps aux | grep zeppelin | grep -v "2BSGYY7S8" |grep -v "zeppelin.server.ZeppelinServer"
> | grep java | awk -F " " '{print$2}' | xargs sudo -u yarn kill -9
>
>
> If I do this every few days zeppelin keeps humming along pretty smoothly
> most of the time.
> <http://www.placeiq.com/> <http://www.placeiq.com/>
> <http://www.placeiq.com/> Paul Brenner <https://twitter.com/placeiq>
> <https://twitter.com/placeiq> <https://twitter.com/placeiq>
> <https://www.facebook.com/PlaceIQ> <https://www.facebook.com/PlaceIQ>
> <https://www.linkedin.com/company/placeiq>
> <https://www.linkedin.com/company/placeiq>
> DATA SCIENTIST
> *(217) 390-3033 <(217)%20390-3033> *
>
> <http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP>
> <http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/>
> <http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/>[image:
> PlaceIQ:Location Data Accuracy]
> <http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/>
>
> On Fri, Feb 17, 2017 at 3:23 PM "xyunfei@simuwell.com" <">"xyunfei@simuwell.com"
> > wrote:
>
> The problem could be not only the resource, but the session. If you run a
> chunk of spark code and you should see the a running application in the
> spark UI, but in your code if you shut it down after the job is finished,
> then on the spark UI you will see the hob is finished. Within zeppelin,
> each job will start the spark session only once(different interpreter mode
> could be set if you want notebooks to share the session or not), if you
> closed it ,it will never restart it again. The only way to get the same
> code work again is to restat the interpreter or restart zeppelin. I`m not
> sure if I explain clearly, but hope it could help
>
>
>
> From: Paul Brenner <pb...@placeiq.com>
>
> *Date:* 2017-02-17 12:14
> *To:* users <us...@zeppelin.apache.org>
> *Subject:* Re: Re: Zeppelin unable to respond after some time
> I’ve definitely had this problem with jobs that don’t take all the
> resources on the cluster. Also, my experience matches what others have
> reported: just restarting zeppelin and re-runing the stuck paragraph solves
> the issue.
>
> I’ve also experienced this problem with for loops. Some for loops which
> write to disk but absolutely don’t have any variables that are increasing
> in size will hang in Zeppelin. If I run the exact same code in the scala
> REPL it goes through without problem.
>
>
>
> <http://www.placeiq.com/> <http://www.placeiq.com/>
> <http://www.placeiq.com/> Paul Brenner <https://twitter.com/placeiq>
> <https://twitter.com/placeiq> <https://twitter.com/placeiq>
> <https://www.facebook.com/PlaceIQ> <https://www.facebook.com/PlaceIQ>
> <https://www.linkedin.com/company/placeiq>
> <https://www.linkedin.com/company/placeiq>
> DATA SCIENTIST
> *(217) 390-3033 <(217)%20390-3033> *
>
> <http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP>
> <http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/>
> <http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/>[image:
> PlaceIQ:Location Data Accuracy]
> <http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/>
>
> On Fri, Feb 17, 2017 at 2:12 PM "xyunfei@simuwell.com" <">"xyunfei@simuwell.com"
> > wrote:
>
> I have solved the similar issue before. You should check on spark UI and
> probably you will see your single job is taking all the resources.
> Therefore further job that submitting to the same cluster will just hang on
> there. When you restart zeppelin then the old job is killed and all the
> resource it took will be released
>
> ------------------------------
> xyunfei@simuwell.com
>
>
>
> From: RUSHIKESH RAUT <ru...@gmail.com>
>
> *Date:* 2017-02-17 02:29
> *To:* users <us...@zeppelin.apache.org>
> *Subject:* Re: Zeppelin unable to respond after some time
> Yes happens with r and spark codes frequently
>
> On Feb 17, 2017 3:25 PM, "小野圭二" <on...@gmail.com> wrote:
>
> yes, almost every time.
> There are not any special operations.
> Just run the tutorial demos.
> From my feeling, it happens in R demo frequently.
>
> 2017-02-17 18:50 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>
>
> Is it easy to reproduce it ?
>
> 小野圭二 <on...@gmail.com>于2017年2月17日周五 下午5:47写道：
>
> I am facing on the same issue now.
>
> 2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT <ru...@gmail.com>:
>
> Hi all,
>
> I am facing a issue while using Zeppelin. I am trying to load some
> data(not that big data) into Zeppelin and then build some visualization on
> it. The problem is that when I try to run the code first time it's working
> but after some time the same code doesn't work. It remains in running state
> on gui, but no logs are generated in Zeppelin logs. Also all further tasks
> are hanging in pending state.
> As soon as I restart Zeppelin it works. So I am guessing it's some memory
> issue. I have read that Zeppelin stores the data in memory so it is
> possible that it runs out of memory after some time.
> How do I debug this issue? How much is the default memory that Zeppelin
> takes at start? Also is there any way that I can run Zeppelin with
> specified memory so that I can start the process with more memory. Because
> it doesn't make sense to restart Zeppelin after every half hour
>
> Thanks,
> Rushikesh Raut
>
>
>
>
Re: Re: Zeppelin unable to respond after some time
Posted by moon soo Lee <mo...@apache.org>.
Hi,
Download 0.7.0 -> Run R tutorial notebook repeatedly
will reproduce the problem? Otherwise, can someone clarify instruction to
reproduce the problem?
Thanks,
moon
On Sat, Feb 18, 2017 at 5:45 AM xyunfei@simuwell.com <xy...@simuwell.com>
wrote:
> Within the Scala REPL everything is working fine. Even you application
> session is down but you run the same code again you will see there is a new
> job getting created. But zeppelin has such a problem. you can do a test.
> run a notebook and get the job finished at the end of the notebook. Then
> re-run the same notebook again then it will get stucked. Running spark code
> on Scala REPL and Zeppelin are different
>
>
> *From:* Paul Brenner <pb...@placeiq.com>
> *Date:* 2017-02-17 12:37
>
> *To:* users <us...@zeppelin.apache.org>
> *Subject:* Re: Re: Zeppelin unable to respond after some time
>
> I don’t believe that this explains my issue. Running the Scala REPL also
> keeps a session alive for as long as the REPL is running. I’ve had REPLs
> open for days (shhhhh don’t tell anyone) that have correspondingly kept
> sessions alive for the same period of time with no problem. I only see this
> issue in zeppelin.
>
> We run zeppelin on a server and allow multiple users to connect, each with
> their own interpreters. We also find that zeppelin memory usage on the
> server will steadily creep up over time. Executing sys.exit in a spark
> paragraph, restarting the interpreter, and using yarn application -kill
> often will cause zeppelin to end the related interpreter process but not
> always. So over time we find that many zombie processes pile up and eat up
> resources.
>
> The only way to keep on top of this is to regularly login to the zeppelin
> server and kill zombie jobs. Here is a command that I’ve found helpful.
> When you know that a specific user has no active zeppelin interpreters
> running then execute the following:
>
> ps aux | grep zeppelin | grep "2BSGYY7S8" | grep java | awk -F " "
> '{print$2}' | xargs sudo -u yarn kill -9
>
> where “2BSGYY7S8" is the interpreter id (found in interpreter.json) and
> “yarn” is actually the name of the user that originally started zeppelin
> with:
> zeppelin-daemon.sh start
>
> To kill every interpreter except for a specific users just flip it around
> with:
> ps aux | grep zeppelin | grep -v "2BSGYY7S8" |grep -v
> "zeppelin.server.ZeppelinServer" | grep java | awk -F " " '{print$2}' |
> xargs sudo -u yarn kill -9
>
>
> If I do this every few days zeppelin keeps humming along pretty smoothly
> most of the time.
> <http://www.placeiq.com/> <http://www.placeiq.com/>
> <http://www.placeiq.com/> Paul Brenner <https://twitter.com/placeiq>
> <https://twitter.com/placeiq> <https://twitter.com/placeiq>
> <https://www.facebook.com/PlaceIQ> <https://www.facebook.com/PlaceIQ>
> <https://www.linkedin.com/company/placeiq>
> <https://www.linkedin.com/company/placeiq>
> DATA SCIENTIST
> *(217) 390-3033 <(217)%20390-3033> *
>
> <http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP>
> <http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/>
> <http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/>[image:
> PlaceIQ:Location Data Accuracy]
> <http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/>
>
> On Fri, Feb 17, 2017 at 3:23 PM "xyunfei@simuwell.com" <">"xyunfei@simuwell.com"
> > wrote:
>
> The problem could be not only the resource, but the session. If you run a
> chunk of spark code and you should see the a running application in the
> spark UI, but in your code if you shut it down after the job is finished,
> then on the spark UI you will see the hob is finished. Within zeppelin,
> each job will start the spark session only once(different interpreter mode
> could be set if you want notebooks to share the session or not), if you
> closed it ,it will never restart it again. The only way to get the same
> code work again is to restat the interpreter or restart zeppelin. I`m not
> sure if I explain clearly, but hope it could help
>
>
>
> From: Paul Brenner <pb...@placeiq.com>
>
> *Date:* 2017-02-17 12:14
> *To:* users <us...@zeppelin.apache.org>
> *Subject:* Re: Re: Zeppelin unable to respond after some time
> I’ve definitely had this problem with jobs that don’t take all the
> resources on the cluster. Also, my experience matches what others have
> reported: just restarting zeppelin and re-runing the stuck paragraph solves
> the issue.
>
> I’ve also experienced this problem with for loops. Some for loops which
> write to disk but absolutely don’t have any variables that are increasing
> in size will hang in Zeppelin. If I run the exact same code in the scala
> REPL it goes through without problem.
>
>
>
> <http://www.placeiq.com/> <http://www.placeiq.com/>
> <http://www.placeiq.com/> Paul Brenner <https://twitter.com/placeiq>
> <https://twitter.com/placeiq> <https://twitter.com/placeiq>
> <https://www.facebook.com/PlaceIQ> <https://www.facebook.com/PlaceIQ>
> <https://www.linkedin.com/company/placeiq>
> <https://www.linkedin.com/company/placeiq>
> DATA SCIENTIST
> *(217) 390-3033 <(217)%20390-3033> *
>
> <http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/>
> <http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/>
> <http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP>
> <http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/>
> <http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/>[image:
> PlaceIQ:Location Data Accuracy]
> <http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/>
>
> On Fri, Feb 17, 2017 at 2:12 PM "xyunfei@simuwell.com" <">"xyunfei@simuwell.com"
> > wrote:
>
> I have solved the similar issue before. You should check on spark UI and
> probably you will see your single job is taking all the resources.
> Therefore further job that submitting to the same cluster will just hang on
> there. When you restart zeppelin then the old job is killed and all the
> resource it took will be released
>
> ------------------------------
> xyunfei@simuwell.com
>
>
>
> From: RUSHIKESH RAUT <ru...@gmail.com>
>
> *Date:* 2017-02-17 02:29
> *To:* users <us...@zeppelin.apache.org>
> *Subject:* Re: Zeppelin unable to respond after some time
> Yes happens with r and spark codes frequently
>
> On Feb 17, 2017 3:25 PM, "小野圭二" <on...@gmail.com> wrote:
>
> yes, almost every time.
> There are not any special operations.
> Just run the tutorial demos.
> From my feeling, it happens in R demo frequently.
>
> 2017-02-17 18:50 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>
>
> Is it easy to reproduce it ?
>
> 小野圭二 <on...@gmail.com>于2017年2月17日周五 下午5:47写道：
>
> I am facing on the same issue now.
>
> 2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT <ru...@gmail.com>:
>
> Hi all,
>
> I am facing a issue while using Zeppelin. I am trying to load some
> data(not that big data) into Zeppelin and then build some visualization on
> it. The problem is that when I try to run the code first time it's working
> but after some time the same code doesn't work. It remains in running state
> on gui, but no logs are generated in Zeppelin logs. Also all further tasks
> are hanging in pending state.
> As soon as I restart Zeppelin it works. So I am guessing it's some memory
> issue. I have read that Zeppelin stores the data in memory so it is
> possible that it runs out of memory after some time.
> How do I debug this issue? How much is the default memory that Zeppelin
> takes at start? Also is there any way that I can run Zeppelin with
> specified memory so that I can start the process with more memory. Because
> it doesn't make sense to restart Zeppelin after every half hour
>
> Thanks,
> Rushikesh Raut
>
>
>
>
Re: Re: Zeppelin unable to respond after some time
Posted by "xyunfei@simuwell.com" <xy...@simuwell.com>.
Within the Scala REPL everything is working fine. Even you application session is down but you run the same code again you will see there is a new job getting created. But zeppelin has such a problem. you can do a test. run a notebook and get the job finished at the end of the notebook. Then re-run the same notebook again then it will get stucked. Running spark code on Scala REPL and Zeppelin are different
From: Paul Brenner
Date: 2017-02-17 12:37
To: users
Subject: Re: Re: Zeppelin unable to respond after some time
I don’t believe that this explains my issue. Running the Scala REPL also keeps a session alive for as long as the REPL is running. I’ve had REPLs open for days (shhhhh don’t tell anyone) that have correspondingly kept sessions alive for the same period of time with no problem. I only see this issue in zeppelin.
We run zeppelin on a server and allow multiple users to connect, each with their own interpreters. We also find that zeppelin memory usage on the server will steadily creep up over time. Executing sys.exit in a spark paragraph, restarting the interpreter, and using yarn application -kill often will cause zeppelin to end the related interpreter process but not always. So over time we find that many zombie processes pile up and eat up resources.
The only way to keep on top of this is to regularly login to the zeppelin server and kill zombie jobs. Here is a command that I’ve found helpful. When you know that a specific user has no active zeppelin interpreters running then execute the following:
ps aux | grep zeppelin | grep "2BSGYY7S8" | grep java | awk -F " " '{print$2}' | xargs sudo -u yarn kill -9
where “2BSGYY7S8" is the interpreter id (found in interpreter.json) and “yarn” is actually the name of the user that originally started zeppelin with:
zeppelin-daemon.sh start
To kill every interpreter except for a specific users just flip it around with:
ps aux | grep zeppelin | grep -v "2BSGYY7S8" |grep -v "zeppelin.server.ZeppelinServer" | grep java | awk -F " " '{print$2}' | xargs sudo -u yarn kill -9
If I do this every few days zeppelin keeps humming along pretty smoothly most of the time.
Paul Brenner
DATA SCIENTIST
(217) 390-3033
On Fri, Feb 17, 2017 at 3:23 PM "xyunfei@simuwell.com" <">"xyunfei@simuwell.com" > wrote:
The problem could be not only the resource, but the session. If you run a chunk of spark code and you should see the a running application in the spark UI, but in your code if you shut it down after the job is finished, then on the spark UI you will see the hob is finished. Within zeppelin, each job will start the spark session only once(different interpreter mode could be set if you want notebooks to share the session or not), if you closed it ,it will never restart it again. The only way to get the same code work again is to restat the interpreter or restart zeppelin. I`m not sure if I explain clearly, but hope it could help
From: Paul Brenner
Date: 2017-02-17 12:14
To: users
Subject: Re: Re: Zeppelin unable to respond after some time
I’ve definitely had this problem with jobs that don’t take all the resources on the cluster. Also, my experience matches what others have reported: just restarting zeppelin and re-runing the stuck paragraph solves the issue.
I’ve also experienced this problem with for loops. Some for loops which write to disk but absolutely don’t have any variables that are increasing in size will hang in Zeppelin. If I run the exact same code in the scala REPL it goes through without problem.
Paul Brenner
DATA SCIENTIST
(217) 390-3033
On Fri, Feb 17, 2017 at 2:12 PM "xyunfei@simuwell.com" <">"xyunfei@simuwell.com" > wrote:
I have solved the similar issue before. You should check on spark UI and probably you will see your single job is taking all the resources. Therefore further job that submitting to the same cluster will just hang on there. When you restart zeppelin then the old job is killed and all the resource it took will be released
xyunfei@simuwell.com
From: RUSHIKESH RAUT
Date: 2017-02-17 02:29
To: users
Subject: Re: Zeppelin unable to respond after some time
Yes happens with r and spark codes frequently
On Feb 17, 2017 3:25 PM, "小野圭二" <on...@gmail.com> wrote:
yes, almost every time.
There are not any special operations.
Just run the tutorial demos.
From my feeling, it happens in R demo frequently.
2017-02-17 18:50 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
Is it easy to reproduce it ?
小野圭二 <on...@gmail.com>于2017年2月17日周五 下午5:47写道：
I am facing on the same issue now.
2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT <ru...@gmail.com>:
Hi all,
I am facing a issue while using Zeppelin. I am trying to load some data(not that big data) into Zeppelin and then build some visualization on it. The problem is that when I try to run the code first time it's working but after some time the same code doesn't work. It remains in running state on gui, but no logs are generated in Zeppelin logs. Also all further tasks are hanging in pending state.
As soon as I restart Zeppelin it works. So I am guessing it's some memory issue. I have read that Zeppelin stores the data in memory so it is possible that it runs out of memory after some time.
How do I debug this issue? How much is the default memory that Zeppelin takes at start? Also is there any way that I can run Zeppelin with specified memory so that I can start the process with more memory. Because it doesn't make sense to restart Zeppelin after every half hour
Thanks,
Rushikesh Raut
Re: Re: Zeppelin unable to respond after some time
Posted by Paul Brenner <pb...@placeiq.com>.
I don’t believe that this explains my issue. Running the Scala REPL also keeps a session alive for as long as the REPL is running. I’ve had REPLs open for days (shhhhh don’t tell anyone) that have correspondingly kept sessions alive for the same period of time with no problem. I only see this issue in zeppelin.
We run zeppelin on a server and allow multiple users to connect, each with their own interpreters. We also find that zeppelin memory usage on the server will steadily creep up over time. Executing sys.exit in a spark paragraph, restarting the interpreter, and using yarn application -kill often will cause zeppelin to end the related interpreter process but not always. So over time we find that many zombie processes pile up and eat up resources.
The only way to keep on top of this is to regularly login to the zeppelin server and kill zombie jobs. Here is a command that I’ve found helpful. When you know that a specific user has no active zeppelin interpreters running then execute the following:
ps aux | grep zeppelin | grep "2BSGYY7S8" | grep java | awk -F " " '{print$2}' | xargs sudo -u yarn kill -9
where “2BSGYY7S8" is the interpreter id (found in interpreter.json) and “yarn” is actually the name of the user that originally started zeppelin with:
zeppelin-daemon.sh start
To kill every interpreter except for a specific users just flip it around with:
ps aux | grep zeppelin | grep -v "2BSGYY7S8" |grep -v "zeppelin.server.ZeppelinServer" | grep java | awk -F " " '{print$2}' | xargs sudo -u yarn kill -9
If I do this every few days zeppelin keeps humming along pretty smoothly most of the time.
http://www.placeiq.com/ http://www.placeiq.com/ http://www.placeiq.com/
Paul Brenner
https://twitter.com/placeiq https://twitter.com/placeiq https://twitter.com/placeiq
https://www.facebook.com/PlaceIQ https://www.facebook.com/PlaceIQ
https://www.linkedin.com/company/placeiq https://www.linkedin.com/company/placeiq
DATA SCIENTIST
(217) 390-3033
http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/ http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/ http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/ http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/
On Fri, Feb 17, 2017 at 3:23 PM "xyunfei@simuwell.com"
<
mailto:
> wrote:
a, pre, code, a:link, body { word-wrap: break-word !important; }
body { line-height: 1.5; }blockquote { margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em; }div.foxdiv20170217121700549974 { }body { font-size: 10.5pt; font-family: 'Segoe UI'; color: rgb(0, 0, 0); line-height: 1.5; }
The problem could be not only the resource, but the session. If you run a chunk of spark code and you should see the a running application in the spark UI, but in your code if you shut it down after the job is finished, then on the spark UI you will see the hob is finished. Within zeppelin, each job will start the spark session only once(different interpreter mode could be set if you want notebooks to share the session or not), if you closed it ,it will never restart it again. The only way to get the same code work again is to restat the interpreter or restart zeppelin. I`m not sure if I explain clearly, but hope it could help
From:
mailto:pbrenner@placeiq.com
Date:
2017-02-17 12:14
To:
mailto:users@zeppelin.apache.org
Subject:
Re: Re: Zeppelin unable to respond after some time
I’ve definitely had this problem with jobs that don’t take all the resources on the cluster. Also, my experience matches what others have reported: just restarting zeppelin and re-runing the stuck paragraph solves the issue.
I’ve also experienced this problem with for loops. Some for loops which write to disk but absolutely don’t have any variables that are increasing in size will hang in Zeppelin. If I run the exact same code in the scala REPL it goes through without problem.
http://www.placeiq.com/ http://www.placeiq.com/ http://www.placeiq.com/
Paul Brenner
https://twitter.com/placeiq https://twitter.com/placeiq https://twitter.com/placeiq
https://www.facebook.com/PlaceIQ https://www.facebook.com/PlaceIQ
https://www.linkedin.com/company/placeiq https://www.linkedin.com/company/placeiq
DATA SCIENTIST
(217) 390-3033
http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/ http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/ http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/ http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/
On Fri, Feb 17, 2017 at 2:12 PM "xyunfei@simuwell.com"
<
mailto:
> wrote:
I have solved the similar issue before. You should check on spark UI and probably you will see your single job is taking all the resources. Therefore further job that submitting to the same cluster will just hang on there. When you restart zeppelin then the old job is killed and all the resource it took will be released
xyunfei@simuwell.com
From:
mailto:rushikeshraut777@gmail.com
Date:
2017-02-17 02:29
To:
mailto:users@zeppelin.apache.org
Subject:
Re: Zeppelin unable to respond after some time
Yes happens with r and spark codes frequently
On Feb 17, 2017 3:25 PM, "小野圭二" <
mailto:onokeiji@gmail.com
> wrote:
yes, almost every time.
There are not any special operations.
Just run the tutorial demos.
From my feeling, it happens in R demo frequently.
2017-02-17 18:50 GMT+09:00 Jeff Zhang
<
mailto:zjffdu@gmail.com
>
:
Is it easy to reproduce it ?
小野圭二 <
mailto:onokeiji@gmail.com
>于2017年2月17
日周五 下午5:47写道：
I am facing on the same issue now.
2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT
<
mailto:rushikeshraut777@gmail.com
>
:
Hi all,
I am facing a issue while using Zeppelin. I am trying to load some data(not that big data) into Zeppelin and then build some visualization on it. The problem is that when I try to run the code first time it's working but after some time the same code doesn't work. It remains in running state on gui, but no logs are generated in Zeppelin logs. Also all further tasks are hanging in pending state.
As soon as I restart Zeppelin it works. So I am guessing it's some memory issue. I have read that Zeppelin stores the data in memory so it is possible that it runs out of memory after some time.
How do I debug this issue? How much is the default memory that Zeppelin takes at start? Also is there any way that I can run Zeppelin with specified memory so that I can start the process with more memory. Because it doesn't make sense to restart Zeppelin after every half hour
Thanks,
Rushikesh Raut
Re: Re: Zeppelin unable to respond after some time
Posted by "xyunfei@simuwell.com" <xy...@simuwell.com>.
The problem could be not only the resource, but the session. If you run a chunk of spark code and you should see the a running application in the spark UI, but in your code if you shut it down after the job is finished, then on the spark UI you will see the hob is finished. Within zeppelin, each job will start the spark session only once(different interpreter mode could be set if you want notebooks to share the session or not), if you closed it ,it will never restart it again. The only way to get the same code work again is to restat the interpreter or restart zeppelin. I`m not sure if I explain clearly, but hope it could help
From: Paul Brenner
Date: 2017-02-17 12:14
To: users
Subject: Re: Re: Zeppelin unable to respond after some time
I’ve definitely had this problem with jobs that don’t take all the resources on the cluster. Also, my experience matches what others have reported: just restarting zeppelin and re-runing the stuck paragraph solves the issue.
I’ve also experienced this problem with for loops. Some for loops which write to disk but absolutely don’t have any variables that are increasing in size will hang in Zeppelin. If I run the exact same code in the scala REPL it goes through without problem.
Paul Brenner
DATA SCIENTIST
(217) 390-3033
On Fri, Feb 17, 2017 at 2:12 PM "xyunfei@simuwell.com" <">"xyunfei@simuwell.com" > wrote:
I have solved the similar issue before. You should check on spark UI and probably you will see your single job is taking all the resources. Therefore further job that submitting to the same cluster will just hang on there. When you restart zeppelin then the old job is killed and all the resource it took will be released
xyunfei@simuwell.com
From: RUSHIKESH RAUT
Date: 2017-02-17 02:29
To: users
Subject: Re: Zeppelin unable to respond after some time
Yes happens with r and spark codes frequently
On Feb 17, 2017 3:25 PM, "小野圭二" <on...@gmail.com> wrote:
yes, almost every time.
There are not any special operations.
Just run the tutorial demos.
From my feeling, it happens in R demo frequently.
2017-02-17 18:50 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
Is it easy to reproduce it ?
小野圭二 <on...@gmail.com>于2017年2月17日周五 下午5:47写道：
I am facing on the same issue now.
2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT <ru...@gmail.com>:
Hi all,
I am facing a issue while using Zeppelin. I am trying to load some data(not that big data) into Zeppelin and then build some visualization on it. The problem is that when I try to run the code first time it's working but after some time the same code doesn't work. It remains in running state on gui, but no logs are generated in Zeppelin logs. Also all further tasks are hanging in pending state.
As soon as I restart Zeppelin it works. So I am guessing it's some memory issue. I have read that Zeppelin stores the data in memory so it is possible that it runs out of memory after some time.
How do I debug this issue? How much is the default memory that Zeppelin takes at start? Also is there any way that I can run Zeppelin with specified memory so that I can start the process with more memory. Because it doesn't make sense to restart Zeppelin after every half hour
Thanks,
Rushikesh Raut
Re: Re: Zeppelin unable to respond after some time
Posted by Paul Brenner <pb...@placeiq.com>.
I’ve definitely had this problem with jobs that don’t take all the resources on the cluster. Also, my experience matches what others have reported: just restarting zeppelin and re-runing the stuck paragraph solves the issue.
I’ve also experienced this problem with for loops. Some for loops which write to disk but absolutely don’t have any variables that are increasing in size will hang in Zeppelin. If I run the exact same code in the scala REPL it goes through without problem.
http://www.placeiq.com/ http://www.placeiq.com/ http://www.placeiq.com/
Paul Brenner
https://twitter.com/placeiq https://twitter.com/placeiq https://twitter.com/placeiq
https://www.facebook.com/PlaceIQ https://www.facebook.com/PlaceIQ
https://www.linkedin.com/company/placeiq https://www.linkedin.com/company/placeiq
DATA SCIENTIST
(217) 390-3033
http://www.placeiq.com/2015/05/26/placeiq-named-winner-of-prestigious-2015-oracle-data-cloud-activate-award/ http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ http://placeiq.com/2015/12/18/accuracy-vs-precision-in-location-data-mma-webinar/ http://placeiq.com/2016/03/08/measuring-addressable-tv-campaigns-is-now-possible/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://placeiq.com/2016/04/13/placeiq-joins-the-network-advertising-initiative-nai-as-100th-member/ http://pages.placeiq.com/Location-Data-Accuracy-Whitepaper-Download.html?utm_source=Signature&utm_medium=Email&utm_campaign=AccuracyWP http://placeiq.com/2016/08/03/placeiq-bolsters-location-intelligence-platform-with-mastercard-insights/ http://placeiq.com/2016/10/26/the-making-of-a-location-data-industry-milestone/ http://placeiq.com/2016/12/07/placeiq-introduces-landmark-a-groundbreaking-offering-that-delivers-access-to-the-highest-quality-location-data-for-insights-that-fuel-limitless-business-decisions/
On Fri, Feb 17, 2017 at 2:12 PM "xyunfei@simuwell.com"
<
mailto:
> wrote:
a, pre, code, a:link, body { word-wrap: break-word !important; }
body { line-height: 1.5; }blockquote { margin-top: 0px; margin-bottom: 0px; margin-left: 0.5em; }div.foxdiv20170217110832132410 { }body { font-size: 10.5pt; font-family: 'Segoe UI'; color: rgb(0, 0, 0); line-height: 1.5; }
I have solved the similar issue before. You should check on spark UI and probably you will see your single job is taking all the resources. Therefore further job that submitting to the same cluster will just hang on there. When you restart zeppelin then the old job is killed and all the resource it took will be released
xyunfei@simuwell.com
From:
mailto:rushikeshraut777@gmail.com
Date:
2017-02-17 02:29
To:
mailto:users@zeppelin.apache.org
Subject:
Re: Zeppelin unable to respond after some time
Yes happens with r and spark codes frequently
On Feb 17, 2017 3:25 PM, "小野圭二" <
mailto:onokeiji@gmail.com
> wrote:
yes, almost every time.
There are not any special operations.
Just run the tutorial demos.
From my feeling, it happens in R demo frequently.
2017-02-17 18:50 GMT+09:00 Jeff Zhang
<
mailto:zjffdu@gmail.com
>
:
Is it easy to reproduce it ?
小野圭二 <
mailto:onokeiji@gmail.com
>于2017年2月17
日周五 下午5:47写道：
I am facing on the same issue now.
2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT
<
mailto:rushikeshraut777@gmail.com
>
:
Hi all,
I am facing a issue while using Zeppelin. I am trying to load some data(not that big data) into Zeppelin and then build some visualization on it. The problem is that when I try to run the code first time it's working but after some time the same code doesn't work. It remains in running state on gui, but no logs are generated in Zeppelin logs. Also all further tasks are hanging in pending state.
As soon as I restart Zeppelin it works. So I am guessing it's some memory issue. I have read that Zeppelin stores the data in memory so it is possible that it runs out of memory after some time.
How do I debug this issue? How much is the default memory that Zeppelin takes at start? Also is there any way that I can run Zeppelin with specified memory so that I can start the process with more memory. Because it doesn't make sense to restart Zeppelin after every half hour
Thanks,
Rushikesh Raut
Re: Re: Zeppelin unable to respond after some time
Posted by "xyunfei@simuwell.com" <xy...@simuwell.com>.
I have solved the similar issue before. You should check on spark UI and probably you will see your single job is taking all the resources. Therefore further job that submitting to the same cluster will just hang on there. When you restart zeppelin then the old job is killed and all the resource it took will be released
xyunfei@simuwell.com
From: RUSHIKESH RAUT
Date: 2017-02-17 02:29
To: users
Subject: Re: Zeppelin unable to respond after some time
Yes happens with r and spark codes frequently
On Feb 17, 2017 3:25 PM, "小野圭二" <on...@gmail.com> wrote:
yes, almost every time.
There are not any special operations.
Just run the tutorial demos.
From my feeling, it happens in R demo frequently.
2017-02-17 18:50 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
Is it easy to reproduce it ?
小野圭二 <on...@gmail.com>于2017年2月17日周五 下午5:47写道：
I am facing on the same issue now.
2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT <ru...@gmail.com>:
Hi all,
I am facing a issue while using Zeppelin. I am trying to load some data(not that big data) into Zeppelin and then build some visualization on it. The problem is that when I try to run the code first time it's working but after some time the same code doesn't work. It remains in running state on gui, but no logs are generated in Zeppelin logs. Also all further tasks are hanging in pending state.
As soon as I restart Zeppelin it works. So I am guessing it's some memory issue. I have read that Zeppelin stores the data in memory so it is possible that it runs out of memory after some time.
How do I debug this issue? How much is the default memory that Zeppelin takes at start? Also is there any way that I can run Zeppelin with specified memory so that I can start the process with more memory. Because it doesn't make sense to restart Zeppelin after every half hour
Thanks,
Rushikesh Raut
Re: Zeppelin unable to respond after some time
Posted by RUSHIKESH RAUT <ru...@gmail.com>.
Yes happens with r and spark codes frequently
On Feb 17, 2017 3:25 PM, "小野圭二" <on...@gmail.com> wrote:
yes, almost every time.
There are not any special operations.
Just run the tutorial demos.
From my feeling, it happens in R demo frequently.
2017-02-17 18:50 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>
> Is it easy to reproduce it ?
>
> 小野圭二 <on...@gmail.com>于2017年2月17日周五 下午5:47写道：
>
>> I am facing on the same issue now.
>>
>> 2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT <ru...@gmail.com>:
>>
>> Hi all,
>>
>> I am facing a issue while using Zeppelin. I am trying to load some
>> data(not that big data) into Zeppelin and then build some visualization on
>> it. The problem is that when I try to run the code first time it's working
>> but after some time the same code doesn't work. It remains in running state
>> on gui, but no logs are generated in Zeppelin logs. Also all further tasks
>> are hanging in pending state.
>> As soon as I restart Zeppelin it works. So I am guessing it's some
>> memory issue. I have read that Zeppelin stores the data in memory so it is
>> possible that it runs out of memory after some time.
>> How do I debug this issue? How much is the default memory that Zeppelin
>> takes at start? Also is there any way that I can run Zeppelin with
>> specified memory so that I can start the process with more memory. Because
>> it doesn't make sense to restart Zeppelin after every half hour
>>
>> Thanks,
>> Rushikesh Raut
>>
>>
>>
Re: Zeppelin unable to respond after some time
Posted by 小野圭二 <on...@gmail.com>.
yes, almost every time.
There are not any special operations.
Just run the tutorial demos.
From my feeling, it happens in R demo frequently.
2017-02-17 18:50 GMT+09:00 Jeff Zhang <zj...@gmail.com>:
>
> Is it easy to reproduce it ?
>
> 小野圭二 <on...@gmail.com>于2017年2月17日周五 下午5:47写道：
>
>> I am facing on the same issue now.
>>
>> 2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT <ru...@gmail.com>:
>>
>> Hi all,
>>
>> I am facing a issue while using Zeppelin. I am trying to load some
>> data(not that big data) into Zeppelin and then build some visualization on
>> it. The problem is that when I try to run the code first time it's working
>> but after some time the same code doesn't work. It remains in running state
>> on gui, but no logs are generated in Zeppelin logs. Also all further tasks
>> are hanging in pending state.
>> As soon as I restart Zeppelin it works. So I am guessing it's some
>> memory issue. I have read that Zeppelin stores the data in memory so it is
>> possible that it runs out of memory after some time.
>> How do I debug this issue? How much is the default memory that Zeppelin
>> takes at start? Also is there any way that I can run Zeppelin with
>> specified memory so that I can start the process with more memory. Because
>> it doesn't make sense to restart Zeppelin after every half hour
>>
>> Thanks,
>> Rushikesh Raut
>>
>>
>>
Re: Zeppelin unable to respond after some time
Posted by Jeff Zhang <zj...@gmail.com>.
Is it easy to reproduce it ?
小野圭二 <on...@gmail.com>于2017年2月17日周五 下午5:47写道：
> I am facing on the same issue now.
>
> 2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT <ru...@gmail.com>:
>
> Hi all,
>
> I am facing a issue while using Zeppelin. I am trying to load some
> data(not that big data) into Zeppelin and then build some visualization on
> it. The problem is that when I try to run the code first time it's working
> but after some time the same code doesn't work. It remains in running state
> on gui, but no logs are generated in Zeppelin logs. Also all further tasks
> are hanging in pending state.
> As soon as I restart Zeppelin it works. So I am guessing it's some memory
> issue. I have read that Zeppelin stores the data in memory so it is
> possible that it runs out of memory after some time.
> How do I debug this issue? How much is the default memory that Zeppelin
> takes at start? Also is there any way that I can run Zeppelin with
> specified memory so that I can start the process with more memory. Because
> it doesn't make sense to restart Zeppelin after every half hour
>
> Thanks,
> Rushikesh Raut
>
>
>
Re: Zeppelin unable to respond after some time
Posted by 小野圭二 <on...@gmail.com>.
I am facing on the same issue now.
2017-02-17 18:25 GMT+09:00 RUSHIKESH RAUT <ru...@gmail.com>:
> Hi all,
>
> I am facing a issue while using Zeppelin. I am trying to load some
> data(not that big data) into Zeppelin and then build some visualization on
> it. The problem is that when I try to run the code first time it's working
> but after some time the same code doesn't work. It remains in running state
> on gui, but no logs are generated in Zeppelin logs. Also all further tasks
> are hanging in pending state.
> As soon as I restart Zeppelin it works. So I am guessing it's some memory
> issue. I have read that Zeppelin stores the data in memory so it is
> possible that it runs out of memory after some time.
> How do I debug this issue? How much is the default memory that Zeppelin
> takes at start? Also is there any way that I can run Zeppelin with
> specified memory so that I can start the process with more memory. Because
> it doesn't make sense to restart Zeppelin after every half hour
>
> Thanks,
> Rushikesh Raut
>