You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Shashidhar Rao <ra...@gmail.com> on 2014/08/08 21:04:03 UTC
How python integrates with Ambari
Hi,
Anyone please help me if my understanding is correct of how python srcipt
works with Ambari.
Ambari web exposes a rest webservice through jersey. In python, there is
this HTTP client which sends rest request GET,PUT,POST etc through python
rest component and this component is used by Python agent scripts to get
the details.
Please help me out if this architecture is correct.
Thanks in advance.
Regards
shashi
Re: How python integrates with Ambari
Posted by Shashidhar Rao <ra...@gmail.com>.
Thanks Summit, for your help .
On Sun, Aug 10, 2014 at 11:21 PM, Sumit Mohanty <sm...@hortonworks.com>
wrote:
> Ambari relies on jmx for most of the point-in-time data. There are some
> data that are also read from the Ganglia metrics produced by HDFS.
>
> Having said that, Ambari can include Views that are free to employ other
> mechanism. However, in general, HDFS shell is quite adequate for the
> operational needs of core Ambari.
>
>
> On Sun, Aug 10, 2014 at 10:08 AM, Shashidhar Rao <
> raoshashidhar123@gmail.com
> > wrote:
>
> > Thanks for the details of some interactions pointed out.
> >
> > For that confusion on 'Data for HDFS/Map reduce', I mean , say when
> Ambari
> > wants to show the user the total size of all files residing on a
> > particular node or to list all files on a particular data node, I was
> > expecting Ambari to to interact with HDFS java API to get the size or the
> > list files. in other words the size and the list of files become the data
> > for these use cases.Sorry for the confusion.
> >
> >
> > On Sun, Aug 10, 2014 at 9:03 PM, Sumit Mohanty <smohanty@hortonworks.com
> >
> > wrote:
> >
> > > What do you mean by "data for HDFS/Map reduce information"? I can point
> > you
> > > to the code based on what HDFS operation you are looking for.
> > >
> > > For example here are the details of some interactions:
> > > * HDFS service check - Ambari scripts call the "hdfs shell" command
> > > directly to perform some common HDFS operation
> > > * HDFS service configuration - Ambari has scripts that edit the config
> > > files, env.sh files, etc. for HDFS
> > > * HDFS folder creation - The scripts use the shell command to create
> > > directory and perform chmod/chown
> > >
> > >
> > >
> > > On Sun, Aug 10, 2014 at 4:13 AM, Shashidhar Rao <
> > > raoshashidhar123@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I browsed through the ambari source code and I was looking for code
> on
> > > how
> > > > Ambari pulls out data for HDFS/Map reduce information.
> > > >
> > > > I was looking at the ambari source code in the ambari-server
> directory
> > to
> > > > find how ambari interacts with hadoop and was searching for hadoop
> > > related
> > > > classes in the imports section. But I could not find none.
> > > >
> > > > Can somebody let me know on how ambari connects to Hdfs or Map
> reduce.
> > I
> > > am
> > > > sure there has to be some class imports of hadoop related classes.
> > > >
> > > > Thanks in advance.
> > > >
> > > > Regards
> > > > Shashi
> > > >
> > > >
> > > > On Sat, Aug 9, 2014 at 12:45 AM, Dmitry Lysnichenko <
> > > > dlysnichenko@hortonworks.com> wrote:
> > > >
> > > > > Hi Shashidhar,
> > > > >
> > > > > In general, the architecture is:
> > > > >
> > > > > Web UI (JS) -> Rest API (provided by ambari-server java
> > implementation)
> > > > ->
> > > > > ambari-agent (written on python) -> system scripts for service
> > > deployment
> > > > > and control (written on python, launched by ambari-agent and
> executed
> > > in
> > > > > separate processes).
> > > > >
> > > > > Python-client is available as an alternative to Web UI. But it is
> not
> > > > used
> > > > > by agent scripts.
> > > > >
> > > > >
> > > > > On Fri, Aug 8, 2014 at 10:04 PM, Shashidhar Rao <
> > > > > raoshashidhar123@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Anyone please help me if my understanding is correct of how
> python
> > > > srcipt
> > > > > > works with Ambari.
> > > > > >
> > > > > > Ambari web exposes a rest webservice through jersey. In python,
> > there
> > > > is
> > > > > > this HTTP client which sends rest request GET,PUT,POST etc
> through
> > > > python
> > > > > > rest component and this component is used by Python agent scripts
> > to
> > > > get
> > > > > > the details.
> > > > > >
> > > > > > Please help me out if this architecture is correct.
> > > > > >
> > > > > > Thanks in advance.
> > > > > >
> > > > > > Regards
> > > > > > shashi
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Dmitry
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: How python integrates with Ambari
Posted by Sumit Mohanty <sm...@hortonworks.com>.
Ambari relies on jmx for most of the point-in-time data. There are some
data that are also read from the Ganglia metrics produced by HDFS.
Having said that, Ambari can include Views that are free to employ other
mechanism. However, in general, HDFS shell is quite adequate for the
operational needs of core Ambari.
On Sun, Aug 10, 2014 at 10:08 AM, Shashidhar Rao <raoshashidhar123@gmail.com
> wrote:
> Thanks for the details of some interactions pointed out.
>
> For that confusion on 'Data for HDFS/Map reduce', I mean , say when Ambari
> wants to show the user the total size of all files residing on a
> particular node or to list all files on a particular data node, I was
> expecting Ambari to to interact with HDFS java API to get the size or the
> list files. in other words the size and the list of files become the data
> for these use cases.Sorry for the confusion.
>
>
> On Sun, Aug 10, 2014 at 9:03 PM, Sumit Mohanty <sm...@hortonworks.com>
> wrote:
>
> > What do you mean by "data for HDFS/Map reduce information"? I can point
> you
> > to the code based on what HDFS operation you are looking for.
> >
> > For example here are the details of some interactions:
> > * HDFS service check - Ambari scripts call the "hdfs shell" command
> > directly to perform some common HDFS operation
> > * HDFS service configuration - Ambari has scripts that edit the config
> > files, env.sh files, etc. for HDFS
> > * HDFS folder creation - The scripts use the shell command to create
> > directory and perform chmod/chown
> >
> >
> >
> > On Sun, Aug 10, 2014 at 4:13 AM, Shashidhar Rao <
> > raoshashidhar123@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I browsed through the ambari source code and I was looking for code on
> > how
> > > Ambari pulls out data for HDFS/Map reduce information.
> > >
> > > I was looking at the ambari source code in the ambari-server directory
> to
> > > find how ambari interacts with hadoop and was searching for hadoop
> > related
> > > classes in the imports section. But I could not find none.
> > >
> > > Can somebody let me know on how ambari connects to Hdfs or Map reduce.
> I
> > am
> > > sure there has to be some class imports of hadoop related classes.
> > >
> > > Thanks in advance.
> > >
> > > Regards
> > > Shashi
> > >
> > >
> > > On Sat, Aug 9, 2014 at 12:45 AM, Dmitry Lysnichenko <
> > > dlysnichenko@hortonworks.com> wrote:
> > >
> > > > Hi Shashidhar,
> > > >
> > > > In general, the architecture is:
> > > >
> > > > Web UI (JS) -> Rest API (provided by ambari-server java
> implementation)
> > > ->
> > > > ambari-agent (written on python) -> system scripts for service
> > deployment
> > > > and control (written on python, launched by ambari-agent and executed
> > in
> > > > separate processes).
> > > >
> > > > Python-client is available as an alternative to Web UI. But it is not
> > > used
> > > > by agent scripts.
> > > >
> > > >
> > > > On Fri, Aug 8, 2014 at 10:04 PM, Shashidhar Rao <
> > > > raoshashidhar123@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Anyone please help me if my understanding is correct of how python
> > > srcipt
> > > > > works with Ambari.
> > > > >
> > > > > Ambari web exposes a rest webservice through jersey. In python,
> there
> > > is
> > > > > this HTTP client which sends rest request GET,PUT,POST etc through
> > > python
> > > > > rest component and this component is used by Python agent scripts
> to
> > > get
> > > > > the details.
> > > > >
> > > > > Please help me out if this architecture is correct.
> > > > >
> > > > > Thanks in advance.
> > > > >
> > > > > Regards
> > > > > shashi
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Dmitry
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: How python integrates with Ambari
Posted by Shashidhar Rao <ra...@gmail.com>.
Thanks for the details of some interactions pointed out.
For that confusion on 'Data for HDFS/Map reduce', I mean , say when Ambari
wants to show the user the total size of all files residing on a
particular node or to list all files on a particular data node, I was
expecting Ambari to to interact with HDFS java API to get the size or the
list files. in other words the size and the list of files become the data
for these use cases.Sorry for the confusion.
On Sun, Aug 10, 2014 at 9:03 PM, Sumit Mohanty <sm...@hortonworks.com>
wrote:
> What do you mean by "data for HDFS/Map reduce information"? I can point you
> to the code based on what HDFS operation you are looking for.
>
> For example here are the details of some interactions:
> * HDFS service check - Ambari scripts call the "hdfs shell" command
> directly to perform some common HDFS operation
> * HDFS service configuration - Ambari has scripts that edit the config
> files, env.sh files, etc. for HDFS
> * HDFS folder creation - The scripts use the shell command to create
> directory and perform chmod/chown
>
>
>
> On Sun, Aug 10, 2014 at 4:13 AM, Shashidhar Rao <
> raoshashidhar123@gmail.com>
> wrote:
>
> > Hi,
> >
> > I browsed through the ambari source code and I was looking for code on
> how
> > Ambari pulls out data for HDFS/Map reduce information.
> >
> > I was looking at the ambari source code in the ambari-server directory to
> > find how ambari interacts with hadoop and was searching for hadoop
> related
> > classes in the imports section. But I could not find none.
> >
> > Can somebody let me know on how ambari connects to Hdfs or Map reduce. I
> am
> > sure there has to be some class imports of hadoop related classes.
> >
> > Thanks in advance.
> >
> > Regards
> > Shashi
> >
> >
> > On Sat, Aug 9, 2014 at 12:45 AM, Dmitry Lysnichenko <
> > dlysnichenko@hortonworks.com> wrote:
> >
> > > Hi Shashidhar,
> > >
> > > In general, the architecture is:
> > >
> > > Web UI (JS) -> Rest API (provided by ambari-server java implementation)
> > ->
> > > ambari-agent (written on python) -> system scripts for service
> deployment
> > > and control (written on python, launched by ambari-agent and executed
> in
> > > separate processes).
> > >
> > > Python-client is available as an alternative to Web UI. But it is not
> > used
> > > by agent scripts.
> > >
> > >
> > > On Fri, Aug 8, 2014 at 10:04 PM, Shashidhar Rao <
> > > raoshashidhar123@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Anyone please help me if my understanding is correct of how python
> > srcipt
> > > > works with Ambari.
> > > >
> > > > Ambari web exposes a rest webservice through jersey. In python, there
> > is
> > > > this HTTP client which sends rest request GET,PUT,POST etc through
> > python
> > > > rest component and this component is used by Python agent scripts to
> > get
> > > > the details.
> > > >
> > > > Please help me out if this architecture is correct.
> > > >
> > > > Thanks in advance.
> > > >
> > > > Regards
> > > > shashi
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Dmitry
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: How python integrates with Ambari
Posted by Sumit Mohanty <sm...@hortonworks.com>.
What do you mean by "data for HDFS/Map reduce information"? I can point you
to the code based on what HDFS operation you are looking for.
For example here are the details of some interactions:
* HDFS service check - Ambari scripts call the "hdfs shell" command
directly to perform some common HDFS operation
* HDFS service configuration - Ambari has scripts that edit the config
files, env.sh files, etc. for HDFS
* HDFS folder creation - The scripts use the shell command to create
directory and perform chmod/chown
On Sun, Aug 10, 2014 at 4:13 AM, Shashidhar Rao <ra...@gmail.com>
wrote:
> Hi,
>
> I browsed through the ambari source code and I was looking for code on how
> Ambari pulls out data for HDFS/Map reduce information.
>
> I was looking at the ambari source code in the ambari-server directory to
> find how ambari interacts with hadoop and was searching for hadoop related
> classes in the imports section. But I could not find none.
>
> Can somebody let me know on how ambari connects to Hdfs or Map reduce. I am
> sure there has to be some class imports of hadoop related classes.
>
> Thanks in advance.
>
> Regards
> Shashi
>
>
> On Sat, Aug 9, 2014 at 12:45 AM, Dmitry Lysnichenko <
> dlysnichenko@hortonworks.com> wrote:
>
> > Hi Shashidhar,
> >
> > In general, the architecture is:
> >
> > Web UI (JS) -> Rest API (provided by ambari-server java implementation)
> ->
> > ambari-agent (written on python) -> system scripts for service deployment
> > and control (written on python, launched by ambari-agent and executed in
> > separate processes).
> >
> > Python-client is available as an alternative to Web UI. But it is not
> used
> > by agent scripts.
> >
> >
> > On Fri, Aug 8, 2014 at 10:04 PM, Shashidhar Rao <
> > raoshashidhar123@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Anyone please help me if my understanding is correct of how python
> srcipt
> > > works with Ambari.
> > >
> > > Ambari web exposes a rest webservice through jersey. In python, there
> is
> > > this HTTP client which sends rest request GET,PUT,POST etc through
> python
> > > rest component and this component is used by Python agent scripts to
> get
> > > the details.
> > >
> > > Please help me out if this architecture is correct.
> > >
> > > Thanks in advance.
> > >
> > > Regards
> > > shashi
> > >
> >
> >
> >
> > --
> > Thanks,
> > Dmitry
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: How python integrates with Ambari
Posted by Shashidhar Rao <ra...@gmail.com>.
Hi,
I browsed through the ambari source code and I was looking for code on how
Ambari pulls out data for HDFS/Map reduce information.
I was looking at the ambari source code in the ambari-server directory to
find how ambari interacts with hadoop and was searching for hadoop related
classes in the imports section. But I could not find none.
Can somebody let me know on how ambari connects to Hdfs or Map reduce. I am
sure there has to be some class imports of hadoop related classes.
Thanks in advance.
Regards
Shashi
On Sat, Aug 9, 2014 at 12:45 AM, Dmitry Lysnichenko <
dlysnichenko@hortonworks.com> wrote:
> Hi Shashidhar,
>
> In general, the architecture is:
>
> Web UI (JS) -> Rest API (provided by ambari-server java implementation) ->
> ambari-agent (written on python) -> system scripts for service deployment
> and control (written on python, launched by ambari-agent and executed in
> separate processes).
>
> Python-client is available as an alternative to Web UI. But it is not used
> by agent scripts.
>
>
> On Fri, Aug 8, 2014 at 10:04 PM, Shashidhar Rao <
> raoshashidhar123@gmail.com>
> wrote:
>
> > Hi,
> >
> > Anyone please help me if my understanding is correct of how python srcipt
> > works with Ambari.
> >
> > Ambari web exposes a rest webservice through jersey. In python, there is
> > this HTTP client which sends rest request GET,PUT,POST etc through python
> > rest component and this component is used by Python agent scripts to get
> > the details.
> >
> > Please help me out if this architecture is correct.
> >
> > Thanks in advance.
> >
> > Regards
> > shashi
> >
>
>
>
> --
> Thanks,
> Dmitry
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: How python integrates with Ambari
Posted by Dmitry Lysnichenko <dl...@hortonworks.com>.
Hi Shashidhar,
In general, the architecture is:
Web UI (JS) -> Rest API (provided by ambari-server java implementation) ->
ambari-agent (written on python) -> system scripts for service deployment
and control (written on python, launched by ambari-agent and executed in
separate processes).
Python-client is available as an alternative to Web UI. But it is not used
by agent scripts.
On Fri, Aug 8, 2014 at 10:04 PM, Shashidhar Rao <ra...@gmail.com>
wrote:
> Hi,
>
> Anyone please help me if my understanding is correct of how python srcipt
> works with Ambari.
>
> Ambari web exposes a rest webservice through jersey. In python, there is
> this HTTP client which sends rest request GET,PUT,POST etc through python
> rest component and this component is used by Python agent scripts to get
> the details.
>
> Please help me out if this architecture is correct.
>
> Thanks in advance.
>
> Regards
> shashi
>
--
Thanks,
Dmitry
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.