You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Mich Talebzadeh <mi...@gmail.com> on 2016/10/03 09:21:02 UTC
Using Zeppelin continuously with the same UI open
Hi,
I have been testing Zeppelin in conjunction with Spark sql and Spark
functional programming. I notice that as time gets by and many calls are
made with the same session open with different codes and plots, Zeppelin
becomes very slow and eventually one gets a message "pipe broken" in the
run.
Then in order to make this work I recycle (stop/start) Zeppelin daemon and
things start working.
Is this behavior expected or is this due to excessive memory usage/GC issue.
FYI I am using zeppelin-0.6.1
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
Re: Using Zeppelin continuously with the same UI open
Posted by Mich Talebzadeh <mi...@gmail.com>.
Hi Moon,
I am using Spark/Scala code as below
import java.util.Calendar
import org.joda.time._
val option = z.input("option","2").toString.toInt
val today = new DateTime()
val minutes = -15
val minutesago =
today.plusMinutes(minutes).toString.toString.substring(11,19)
val date = java.time.LocalDate.now.toString
val hour = java.time.LocalTime.now.toString
val location="hdfs://rhes564:9000/data/prices/"+date+"/prices.*"
case class columns(SECURITY: String, TIMECREATED: String, PRICE: String)
val rs = option match {
case 1 => {
println("option = 1")
val df1 = spark.read.option("header", false).csv(location)
val df2 = df1.map(p => columns(p(0).toString,p(1).toString,
p(2).toString))
val rs = df2.filter('security > "").filter('price > "15" &&
substring('TIMECREATED,12,19)>
minutesago).select(substring('TIMECREATED,12,19).as("TIMECREATED"),
'SECURITY, substring('PRICE,1,7).as("PRICE")).orderBy('TIMECREATED)
rs
}
case 2 => {
println("option = 2")
val df2 =
spark.table("test.marketData").select('TIMECREATED,'SECURITY,'PRICE,
'DateStamp)
val rs = df2.filter('security > "").filter('DateStamp >= "${today}" &&
'price > "15" && 'TIMECREATED > date && substring('TIMECREATED,12,19)>
minutesago).select(substring('TIMECREATED,12,19).as("TIMECREATED"),
'SECURITY, substring('PRICE,1,7).as("PRICE")).orderBy('TIMECREATED)
rs
}
case _ => {
sys.error("No valid option provided")
}
}
z.show(rs)
Anything I can do to reduce the effect? The UI is kept open and I just run
the code again and again if I need to
Thanks
Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.
On 5 October 2016 at 15:33, moon soo Lee <mo...@apache.org> wrote:
> Hi,
>
> Thanks for sharing problem.
>
> If you're using Scala in SparkInterpreter, Scala REPL may have some
> problem [1] when there is a lot of calls.
>
> If you're using only SparkSQL or other interpreter, but still experience
> the same, then it is Zeppelin's own problem.
>
> Could you share little more information to reproduce the problem?
>
> Thanks,
> moon
>
> [1] https://issues.scala-lang.org/browse/SI-4331
>
>
> On Mon, Oct 3, 2016 at 6:21 PM Mich Talebzadeh <mi...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I have been testing Zeppelin in conjunction with Spark sql and Spark
>> functional programming. I notice that as time gets by and many calls are
>> made with the same session open with different codes and plots, Zeppelin
>> becomes very slow and eventually one gets a message "pipe broken" in the
>> run.
>>
>> Then in order to make this work I recycle (stop/start) Zeppelin daemon
>> and things start working.
>>
>> Is this behavior expected or is this due to excessive memory usage/GC
>> issue.
>>
>> FYI I am using zeppelin-0.6.1
>>
>> Thanks
>>
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>> *Disclaimer:* Use it at your own risk. Any and all responsibility for
>> any loss, damage or destruction of data or any other property which may
>> arise from relying on this email's technical content is explicitly
>> disclaimed. The author will in no case be liable for any monetary damages
>> arising from such loss, damage or destruction.
>>
>>
>>
>
Re: Using Zeppelin continuously with the same UI open
Posted by moon soo Lee <mo...@apache.org>.
Hi,
Thanks for sharing problem.
If you're using Scala in SparkInterpreter, Scala REPL may have some problem
[1] when there is a lot of calls.
If you're using only SparkSQL or other interpreter, but still experience
the same, then it is Zeppelin's own problem.
Could you share little more information to reproduce the problem?
Thanks,
moon
[1] https://issues.scala-lang.org/browse/SI-4331
On Mon, Oct 3, 2016 at 6:21 PM Mich Talebzadeh <mi...@gmail.com>
wrote:
> Hi,
>
> I have been testing Zeppelin in conjunction with Spark sql and Spark
> functional programming. I notice that as time gets by and many calls are
> made with the same session open with different codes and plots, Zeppelin
> becomes very slow and eventually one gets a message "pipe broken" in the
> run.
>
> Then in order to make this work I recycle (stop/start) Zeppelin daemon and
> things start working.
>
> Is this behavior expected or is this due to excessive memory usage/GC
> issue.
>
> FYI I am using zeppelin-0.6.1
>
> Thanks
>
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>