You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@livy.apache.org by Meisam Fathi <me...@gmail.com> on 2019/04/16 03:13:02 UTC
Re: Accessing Detailed Livy Session Information (session name?)
Hi Peter,
Are you using ZooKeeper for recovery store?
If yes, in conf/livy.conf, is livy.server.recovery.zk-state-store.key-prefix
set to different values in different Livy instances? If not, all of Livy
instances will read/write the recovery data from/to the same path, which is
default is /livy/v1 by default.
@dev mailing list:
This behavior is not documented in livy.conf nor on the website. It might
be a good idea to document it somewhere.
Thanks,
Meisam
On Fri, Apr 12, 2019 at 3:20 PM Meisam Fathi <me...@gmail.com> wrote:
> Hi Peter,
>
> Livy 0.6 has a new feature to give each session a name:
> https://github.com/apache/incubator-livy/pull/48
>
> Would this feature be useful in your usecase?
>
> Thanks,
> Meisam
>
> On Fri, Apr 12, 2019, 8:51 AM Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
>> Greetings,
>>
>>
>>
>> I have a custom service that connects to Livy, v0.4 soon to be v0.5 once
>> we go to HDP3. If sessions already exist it logs the session ID’s and
>> starts using them, if sessions don’t exist it creates new ones. The problem
>> is the account used to launch the Livy sessions is not unique to this
>> service, nor is the kind of session. So sometimes it grabs other people’s
>> sessions and absconds off with them. Also, there are multiple instances of
>> the service, running under the same account, and they are not supposed to
>> use each other’s sessions… that’s not working out so well.
>>
>>
>>
>> The service names the sessions, but I can’t find any way to retrieve
>> detailed session data so that I can update the service to check if the Livy
>> Session belongs to the service or not.
>>
>>
>>
>> I found some older comments 2016/2017 about retrieving Livy sessions by
>> name. I don’t really need that, I just want to be able to read the name
>> through the regular sessions REST call.
>>
>>
>>
>> Any REST calls I missed, or undocumented calls… that can help?
>>
>>
>>
>> Thanks,
>>
>> Peter
>>
>>
>>
>> Ref:
>> https://github.com/meisam/livy/wiki/Design-doc-for-Livy-41:-Accessing-sessions-by-name,
>> https://issues.cloudera.org/browse/LIVY-41
>>
>>
>>
>>
>>
>
Re: Accessing Detailed Livy Session Information (session name?)
Posted by Meisam Fathi <me...@gmail.com>.
I noticed a bug in Ambari while playing around with these settings. Ambari
ignores the livy.server.recovery.state-store value and attempts to create a
file bases state store even if state-store is set to zookeeper. The attempt
fails because zkhost1:port,zkhost2:port,zkhost3:port,... is not a valid
path on HDFS. It looks like it is related to this issue:
https://issues.apache.org/jira/browse/AMBARI-25250
Thanks,
Meisam
On Mon, Apr 15, 2019 at 8:39 PM Marco Gaido <ma...@gmail.com> wrote:
> I think in general our website documentation is not complete and I think it
> would be great if we could improve it significantly. For instance, there is
> no place where all configurations are described properly (other than
> reading the code). I think that PRs and help in improving the doc would be
> highly appreciated.
>
> Thanks,
> Marco
>
> On Tue, 16 Apr 2019, 05:13 Meisam Fathi, <me...@gmail.com> wrote:
>
> > Hi Peter,
> >
> > Are you using ZooKeeper for recovery store?
> > If yes, in conf/livy.conf, is
> > livy.server.recovery.zk-state-store.key-prefix
> > set to different values in different Livy instances? If not, all of Livy
> > instances will read/write the recovery data from/to the same path, which
> is
> > default is /livy/v1 by default.
> >
> > @dev mailing list:
> > This behavior is not documented in livy.conf nor on the website. It might
> > be a good idea to document it somewhere.
> >
> > Thanks,
> > Meisam
> >
> > On Fri, Apr 12, 2019 at 3:20 PM Meisam Fathi <me...@gmail.com>
> > wrote:
> >
> > > Hi Peter,
> > >
> > > Livy 0.6 has a new feature to give each session a name:
> > > https://github.com/apache/incubator-livy/pull/48
> > >
> > > Would this feature be useful in your usecase?
> > >
> > > Thanks,
> > > Meisam
> > >
> > > On Fri, Apr 12, 2019, 8:51 AM Peter Wicks (pwicks) <pw...@micron.com>
> > > wrote:
> > >
> > >> Greetings,
> > >>
> > >>
> > >>
> > >> I have a custom service that connects to Livy, v0.4 soon to be v0.5
> once
> > >> we go to HDP3. If sessions already exist it logs the session ID’s and
> > >> starts using them, if sessions don’t exist it creates new ones. The
> > problem
> > >> is the account used to launch the Livy sessions is not unique to this
> > >> service, nor is the kind of session. So sometimes it grabs other
> > people’s
> > >> sessions and absconds off with them. Also, there are multiple
> instances
> > of
> > >> the service, running under the same account, and they are not supposed
> > to
> > >> use each other’s sessions… that’s not working out so well.
> > >>
> > >>
> > >>
> > >> The service names the sessions, but I can’t find any way to retrieve
> > >> detailed session data so that I can update the service to check if the
> > Livy
> > >> Session belongs to the service or not.
> > >>
> > >>
> > >>
> > >> I found some older comments 2016/2017 about retrieving Livy sessions
> by
> > >> name. I don’t really need that, I just want to be able to read the
> name
> > >> through the regular sessions REST call.
> > >>
> > >>
> > >>
> > >> Any REST calls I missed, or undocumented calls… that can help?
> > >>
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> Peter
> > >>
> > >>
> > >>
> > >> Ref:
> > >>
> >
> https://github.com/meisam/livy/wiki/Design-doc-for-Livy-41:-Accessing-sessions-by-name
> > ,
> > >> https://issues.cloudera.org/browse/LIVY-41
> > >>
> > >>
> > >>
> > >>
> > >>
> > >
> >
>
Re: Accessing Detailed Livy Session Information (session name?)
Posted by Marco Gaido <ma...@gmail.com>.
I think in general our website documentation is not complete and I think it
would be great if we could improve it significantly. For instance, there is
no place where all configurations are described properly (other than
reading the code). I think that PRs and help in improving the doc would be
highly appreciated.
Thanks,
Marco
On Tue, 16 Apr 2019, 05:13 Meisam Fathi, <me...@gmail.com> wrote:
> Hi Peter,
>
> Are you using ZooKeeper for recovery store?
> If yes, in conf/livy.conf, is
> livy.server.recovery.zk-state-store.key-prefix
> set to different values in different Livy instances? If not, all of Livy
> instances will read/write the recovery data from/to the same path, which is
> default is /livy/v1 by default.
>
> @dev mailing list:
> This behavior is not documented in livy.conf nor on the website. It might
> be a good idea to document it somewhere.
>
> Thanks,
> Meisam
>
> On Fri, Apr 12, 2019 at 3:20 PM Meisam Fathi <me...@gmail.com>
> wrote:
>
> > Hi Peter,
> >
> > Livy 0.6 has a new feature to give each session a name:
> > https://github.com/apache/incubator-livy/pull/48
> >
> > Would this feature be useful in your usecase?
> >
> > Thanks,
> > Meisam
> >
> > On Fri, Apr 12, 2019, 8:51 AM Peter Wicks (pwicks) <pw...@micron.com>
> > wrote:
> >
> >> Greetings,
> >>
> >>
> >>
> >> I have a custom service that connects to Livy, v0.4 soon to be v0.5 once
> >> we go to HDP3. If sessions already exist it logs the session ID’s and
> >> starts using them, if sessions don’t exist it creates new ones. The
> problem
> >> is the account used to launch the Livy sessions is not unique to this
> >> service, nor is the kind of session. So sometimes it grabs other
> people’s
> >> sessions and absconds off with them. Also, there are multiple instances
> of
> >> the service, running under the same account, and they are not supposed
> to
> >> use each other’s sessions… that’s not working out so well.
> >>
> >>
> >>
> >> The service names the sessions, but I can’t find any way to retrieve
> >> detailed session data so that I can update the service to check if the
> Livy
> >> Session belongs to the service or not.
> >>
> >>
> >>
> >> I found some older comments 2016/2017 about retrieving Livy sessions by
> >> name. I don’t really need that, I just want to be able to read the name
> >> through the regular sessions REST call.
> >>
> >>
> >>
> >> Any REST calls I missed, or undocumented calls… that can help?
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Peter
> >>
> >>
> >>
> >> Ref:
> >>
> https://github.com/meisam/livy/wiki/Design-doc-for-Livy-41:-Accessing-sessions-by-name
> ,
> >> https://issues.cloudera.org/browse/LIVY-41
> >>
> >>
> >>
> >>
> >>
> >
>
Re: Accessing Detailed Livy Session Information (session name?)
Posted by Marco Gaido <ma...@gmail.com>.
I think in general our website documentation is not complete and I think it
would be great if we could improve it significantly. For instance, there is
no place where all configurations are described properly (other than
reading the code). I think that PRs and help in improving the doc would be
highly appreciated.
Thanks,
Marco
On Tue, 16 Apr 2019, 05:13 Meisam Fathi, <me...@gmail.com> wrote:
> Hi Peter,
>
> Are you using ZooKeeper for recovery store?
> If yes, in conf/livy.conf, is
> livy.server.recovery.zk-state-store.key-prefix
> set to different values in different Livy instances? If not, all of Livy
> instances will read/write the recovery data from/to the same path, which is
> default is /livy/v1 by default.
>
> @dev mailing list:
> This behavior is not documented in livy.conf nor on the website. It might
> be a good idea to document it somewhere.
>
> Thanks,
> Meisam
>
> On Fri, Apr 12, 2019 at 3:20 PM Meisam Fathi <me...@gmail.com>
> wrote:
>
> > Hi Peter,
> >
> > Livy 0.6 has a new feature to give each session a name:
> > https://github.com/apache/incubator-livy/pull/48
> >
> > Would this feature be useful in your usecase?
> >
> > Thanks,
> > Meisam
> >
> > On Fri, Apr 12, 2019, 8:51 AM Peter Wicks (pwicks) <pw...@micron.com>
> > wrote:
> >
> >> Greetings,
> >>
> >>
> >>
> >> I have a custom service that connects to Livy, v0.4 soon to be v0.5 once
> >> we go to HDP3. If sessions already exist it logs the session ID’s and
> >> starts using them, if sessions don’t exist it creates new ones. The
> problem
> >> is the account used to launch the Livy sessions is not unique to this
> >> service, nor is the kind of session. So sometimes it grabs other
> people’s
> >> sessions and absconds off with them. Also, there are multiple instances
> of
> >> the service, running under the same account, and they are not supposed
> to
> >> use each other’s sessions… that’s not working out so well.
> >>
> >>
> >>
> >> The service names the sessions, but I can’t find any way to retrieve
> >> detailed session data so that I can update the service to check if the
> Livy
> >> Session belongs to the service or not.
> >>
> >>
> >>
> >> I found some older comments 2016/2017 about retrieving Livy sessions by
> >> name. I don’t really need that, I just want to be able to read the name
> >> through the regular sessions REST call.
> >>
> >>
> >>
> >> Any REST calls I missed, or undocumented calls… that can help?
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Peter
> >>
> >>
> >>
> >> Ref:
> >>
> https://github.com/meisam/livy/wiki/Design-doc-for-Livy-41:-Accessing-sessions-by-name
> ,
> >> https://issues.cloudera.org/browse/LIVY-41
> >>
> >>
> >>
> >>
> >>
> >
>