You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@zeppelin.apache.org by Manuel Sopena Ballesteros <ma...@garvan.org.au> on 2019/11/13 03:31:37 UTC
spark r interpreter resets working directory
Dear Zeppelin community,
I am testing spark r interpreter and realised it does not keep the working directory across notes.
[cid:image001.png@01D59A2F.0E03FB20]
What is the reason behind this behavior?
Thank you very much
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
Re: spark r interpreter resets working directory
Posted by Jeff Zhang <zj...@gmail.com>.
There're several approaches.
1. Use shared mode across notes. So that all the note share the same
interpreter
2. Use distributed resource pool via ZeppelinContext api. e.g. z.get, z.set
I plan to to write document about how to sharing data in Zeppelin, hope
that would be helpful for users.
Manuel Sopena Ballesteros <ma...@garvan.org.au> 于2019年11月13日周三 下午2:02写道：
> Ok, what should I do in order to be able to reuse variables across
> different notes?
>
>
>
> Manuel
>
>
>
> *From:* Jeff Zhang [mailto:zjffdu@gmail.com]
> *Sent:* Wednesday, November 13, 2019 4:57 PM
> *To:* users
> *Subject:* Re: spark r interpreter resets working directory
>
>
>
> In that case, each user use different interpreter process. In your second
> note, the current working directory is the yarn container location which is
> expected
>
>
>
>
>
> Manuel Sopena Ballesteros <ma...@garvan.org.au> 于2019年11月13日周三 下午1:50
> 写道：
>
> Yarn cluster using impersonate (per user + isolated)
>
>
>
> I guess that means each note use different interpreters?
>
>
>
> Manuel
>
>
>
> *From:* Jeff Zhang [mailto:zjffdu@gmail.com]
> *Sent:* Wednesday, November 13, 2019 2:35 PM
> *To:* users
> *Subject:* Re: spark r interpreter resets working directory
>
>
>
> Does your different notes share the same interpreter ? I suspect you are
> using per note isolated or scoped mode.
>
>
>
> Looks like you are local or yarn-client mode for the first note, but using
> yarn-cluster mode for the second note
>
>
>
> Manuel Sopena Ballesteros <ma...@garvan.org.au> 于2019年11月13日周三 上午11:31
> 写道：
>
> Dear Zeppelin community,
>
>
>
> I am testing spark r interpreter and realised it does not keep the working
> directory across notes.
>
>
>
> What is the reason behind this behavior?
>
>
>
> Thank you very much
>
>
>
> NOTICE
>
> Please consider the environment before printing this email. This message
> and any attachments are intended for the addressee named and may contain
> legally privileged/confidential/copyright information. If you are not the
> intended recipient, you should not read, use, disclose, copy or distribute
> this communication. If you have received this message in error please
> notify us at once by return email and then delete both messages. We accept
> no liability for the distribution of viruses or similar in electronic
> communications. This notice should not be removed.
>
>
>
>
> --
>
> Best Regards
>
> Jeff Zhang
>
> NOTICE
>
> Please consider the environment before printing this email. This message
> and any attachments are intended for the addressee named and may contain
> legally privileged/confidential/copyright information. If you are not the
> intended recipient, you should not read, use, disclose, copy or distribute
> this communication. If you have received this message in error please
> notify us at once by return email and then delete both messages. We accept
> no liability for the distribution of viruses or similar in electronic
> communications. This notice should not be removed.
>
>
>
>
> --
>
> Best Regards
>
> Jeff Zhang
> NOTICE
> Please consider the environment before printing this email. This message
> and any attachments are intended for the addressee named and may contain
> legally privileged/confidential/copyright information. If you are not the
> intended recipient, you should not read, use, disclose, copy or distribute
> this communication. If you have received this message in error please
> notify us at once by return email and then delete both messages. We accept
> no liability for the distribution of viruses or similar in electronic
> communications. This notice should not be removed.
>
--
Best Regards
Jeff Zhang
RE: spark r interpreter resets working directory
Posted by Manuel Sopena Ballesteros <ma...@garvan.org.au>.
Ok, what should I do in order to be able to reuse variables across different notes?
Manuel
From: Jeff Zhang [mailto:zjffdu@gmail.com]
Sent: Wednesday, November 13, 2019 4:57 PM
To: users
Subject: Re: spark r interpreter resets working directory
In that case, each user use different interpreter process. In your second note, the current working directory is the yarn container location which is expected
Manuel Sopena Ballesteros <ma...@garvan.org.au>> 于2019年11月13日周三 下午1:50写道：
Yarn cluster using impersonate (per user + isolated)
I guess that means each note use different interpreters?
Manuel
From: Jeff Zhang [mailto:zjffdu@gmail.com<ma...@gmail.com>]
Sent: Wednesday, November 13, 2019 2:35 PM
To: users
Subject: Re: spark r interpreter resets working directory
Does your different notes share the same interpreter ? I suspect you are using per note isolated or scoped mode.
Looks like you are local or yarn-client mode for the first note, but using yarn-cluster mode for the second note
Manuel Sopena Ballesteros <ma...@garvan.org.au>> 于2019年11月13日周三 上午11:31写道：
Dear Zeppelin community,
I am testing spark r interpreter and realised it does not keep the working directory across notes.
[cid:image001.png@01D59A44.11B5FA10]
What is the reason behind this behavior?
Thank you very much
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
--
Best Regards
Jeff Zhang
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
--
Best Regards
Jeff Zhang
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
Re: spark r interpreter resets working directory
Posted by Jeff Zhang <zj...@gmail.com>.
In that case, each user use different interpreter process. In your second
note, the current working directory is the yarn container location which is
expected
Manuel Sopena Ballesteros <ma...@garvan.org.au> 于2019年11月13日周三 下午1:50写道：
> Yarn cluster using impersonate (per user + isolated)
>
>
>
> I guess that means each note use different interpreters?
>
>
>
> Manuel
>
>
>
> *From:* Jeff Zhang [mailto:zjffdu@gmail.com]
> *Sent:* Wednesday, November 13, 2019 2:35 PM
> *To:* users
> *Subject:* Re: spark r interpreter resets working directory
>
>
>
> Does your different notes share the same interpreter ? I suspect you are
> using per note isolated or scoped mode.
>
>
>
> Looks like you are local or yarn-client mode for the first note, but using
> yarn-cluster mode for the second note
>
>
>
> Manuel Sopena Ballesteros <ma...@garvan.org.au> 于2019年11月13日周三 上午11:31
> 写道：
>
> Dear Zeppelin community,
>
>
>
> I am testing spark r interpreter and realised it does not keep the working
> directory across notes.
>
>
>
> What is the reason behind this behavior?
>
>
>
> Thank you very much
>
>
>
> NOTICE
>
> Please consider the environment before printing this email. This message
> and any attachments are intended for the addressee named and may contain
> legally privileged/confidential/copyright information. If you are not the
> intended recipient, you should not read, use, disclose, copy or distribute
> this communication. If you have received this message in error please
> notify us at once by return email and then delete both messages. We accept
> no liability for the distribution of viruses or similar in electronic
> communications. This notice should not be removed.
>
>
>
>
> --
>
> Best Regards
>
> Jeff Zhang
> NOTICE
> Please consider the environment before printing this email. This message
> and any attachments are intended for the addressee named and may contain
> legally privileged/confidential/copyright information. If you are not the
> intended recipient, you should not read, use, disclose, copy or distribute
> this communication. If you have received this message in error please
> notify us at once by return email and then delete both messages. We accept
> no liability for the distribution of viruses or similar in electronic
> communications. This notice should not be removed.
>
--
Best Regards
Jeff Zhang
RE: spark r interpreter resets working directory
Posted by Manuel Sopena Ballesteros <ma...@garvan.org.au>.
Yarn cluster using impersonate (per user + isolated)
I guess that means each note use different interpreters?
Manuel
From: Jeff Zhang [mailto:zjffdu@gmail.com]
Sent: Wednesday, November 13, 2019 2:35 PM
To: users
Subject: Re: spark r interpreter resets working directory
Does your different notes share the same interpreter ? I suspect you are using per note isolated or scoped mode.
Looks like you are local or yarn-client mode for the first note, but using yarn-cluster mode for the second note
Manuel Sopena Ballesteros <ma...@garvan.org.au>> 于2019年11月13日周三 上午11:31写道：
Dear Zeppelin community,
I am testing spark r interpreter and realised it does not keep the working directory across notes.
[cid:image001.png@01D59A42.7279ED90]
What is the reason behind this behavior?
Thank you very much
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
--
Best Regards
Jeff Zhang
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
Re: spark r interpreter resets working directory
Posted by Jeff Zhang <zj...@gmail.com>.
Does your different notes share the same interpreter ? I suspect you are
using per note isolated or scoped mode.
Looks like you are local or yarn-client mode for the first note, but using
yarn-cluster mode for the second note
Manuel Sopena Ballesteros <ma...@garvan.org.au> 于2019年11月13日周三
上午11:31写道：
> Dear Zeppelin community,
>
>
>
> I am testing spark r interpreter and realised it does not keep the working
> directory across notes.
>
>
>
> What is the reason behind this behavior?
>
>
>
> Thank you very much
>
>
> NOTICE
> Please consider the environment before printing this email. This message
> and any attachments are intended for the addressee named and may contain
> legally privileged/confidential/copyright information. If you are not the
> intended recipient, you should not read, use, disclose, copy or distribute
> this communication. If you have received this message in error please
> notify us at once by return email and then delete both messages. We accept
> no liability for the distribution of viruses or similar in electronic
> communications. This notice should not be removed.
>
--
Best Regards
Jeff Zhang
Re: spark r interpreter resets working directory
Posted by Jeff Zhang <zj...@gmail.com>.
Sorry for the confusion, I do be able to reproduce this issue. I think this
is a bug of SparkRInterpreter
Manuel Sopena Ballesteros <ma...@garvan.org.au> 于2019年11月13日周三 下午2:48写道：
> Sorry, I got confused with the terminology (I meant paragraph instead of
> note)
>
>
>
> My interpreter is configured per user +isolated à this means the same
> interpreter process (jvm process) for same user.
>
>
>
> First paragraph
>
>
>
> %anaconda3.r
>
>
>
> setwd("/home/mansop")
>
> getwd()
>
>
>
> output:
>
> [1] “/home/mansop”
>
>
>
> Second paragraph
>
> %anaconda3.r
>
>
>
> getwd()
>
>
>
> output:
>
> [1]
> “/d0/hadoop/yarn/local/usercache/mansop/appcache/application_1572410115474_0106/container_e16_1572410115474_0106_01_000001”
>
>
>
> Why R does not carry the working directory to the second paragraph even if
> both are running in the same interpreter process?
>
>
>
> Thank you
>
>
>
> Manuel
>
>
>
> *From:* Manuel Sopena Ballesteros [mailto:manuel.sb@garvan.org.au]
> *Sent:* Wednesday, November 13, 2019 2:32 PM
> *To:* users@zeppelin.apache.org
> *Subject:* spark r interpreter resets working directory
>
>
>
> Dear Zeppelin community,
>
>
>
> I am testing spark r interpreter and realised it does not keep the working
> directory across notes.
>
>
>
> What is the reason behind this behavior?
>
>
>
> Thank you very much
>
>
>
> NOTICE
>
> Please consider the environment before printing this email. This message
> and any attachments are intended for the addressee named and may contain
> legally privileged/confidential/copyright information. If you are not the
> intended recipient, you should not read, use, disclose, copy or distribute
> this communication. If you have received this message in error please
> notify us at once by return email and then delete both messages. We accept
> no liability for the distribution of viruses or similar in electronic
> communications. This notice should not be removed.
> NOTICE
> Please consider the environment before printing this email. This message
> and any attachments are intended for the addressee named and may contain
> legally privileged/confidential/copyright information. If you are not the
> intended recipient, you should not read, use, disclose, copy or distribute
> this communication. If you have received this message in error please
> notify us at once by return email and then delete both messages. We accept
> no liability for the distribution of viruses or similar in electronic
> communications. This notice should not be removed.
>
--
Best Regards
Jeff Zhang
RE: spark r interpreter resets working directory
Posted by Manuel Sopena Ballesteros <ma...@garvan.org.au>.
Sorry, I got confused with the terminology (I meant paragraph instead of note)
My interpreter is configured per user +isolated --> this means the same interpreter process (jvm process) for same user.
First paragraph
%anaconda3.r
setwd("/home/mansop")
getwd()
output:
[1] "/home/mansop"
Second paragraph
%anaconda3.r
getwd()
output:
[1] "/d0/hadoop/yarn/local/usercache/mansop/appcache/application_1572410115474_0106/container_e16_1572410115474_0106_01_000001"
Why R does not carry the working directory to the second paragraph even if both are running in the same interpreter process?
Thank you
Manuel
From: Manuel Sopena Ballesteros [mailto:manuel.sb@garvan.org.au]
Sent: Wednesday, November 13, 2019 2:32 PM
To: users@zeppelin.apache.org
Subject: spark r interpreter resets working directory
Dear Zeppelin community,
I am testing spark r interpreter and realised it does not keep the working directory across notes.
[cid:image001.png@01D59A4A.938EE2D0]
What is the reason behind this behavior?
Thank you very much
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
NOTICE
Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.