Why command script is written in python?

["hsy541@gmail.com" <hs...@gmail.com>]

May I ask why do you choose python to write the command script. And is
there  any other language? I find difficult to debug because each time if I
find an issue I have to change file package upload to hdfs, rerun and I'm
not very familiar with python


Best
Siyuan

Re: Why command script is written in python?

["hsy541@gmail.com" <hs...@gmail.com>]

Logged a ticket here, https://issues.apache.org/jira/browse/SLIDER-590
feel free to rephrase if anything I said is wrong. Thanks!

Best

On Thu, Oct 30, 2014 at 2:45 PM, Sumit Mohanty <sm...@hortonworks.com>
wrote:

> Agree. In fact, we need to also document how Slider scripts can be debugged
> during development.
>
> If you do not mind can you open a JIRA at
> https://issues.apache.org/jira/browse/SLIDER describing the requirements
> of
> "debugging" during development time and post development. I will use that
> JIRA to drive the issue.
>
> -Sumit
>
> On Thu, Oct 30, 2014 at 2:40 PM, hsy541@gmail.com <hs...@gmail.com>
> wrote:
>
> > Thanks Sumit,
> >
> > The only concern about script language is you don't know the compile
> error
> > until runtime. And in fact it's not arbitrary script it depends on the
> > library in slider for example
> > /python/resource_management/core/resources/system.py
> >
> > Best,
> > Siyuan
> >
> > On Thu, Oct 30, 2014 at 2:24 PM, Sumit Mohanty <su...@gmail.com>
> > wrote:
> >
> > > Actually, you can debug it after deploying it once. This will require
> > > setting some YARN variables to ensure containers are not released even
> if
> > > the application fails. The agent logs report back the commands being
> > > invoked and the parameters used to invoke them. Let me try to write
> that
> > up
> > > on the wiki. *I will need a day or so.*
> > >
> > > The choice of python was mostly a reflection of going for scripting
> > > language that works on Windows and linux. As such it is possible to use
> > any
> > > scripting language as long as the agent code is modified to handle
> > various
> > > script types. Of course, today the only supported/tested one is python.
> > >
> > > On Thu, Oct 30, 2014 at 1:55 PM, hsy541@gmail.com <hs...@gmail.com>
> > > wrote:
> > >
> > > > May I ask why do you choose python to write the command script. And
> is
> > > > there  any other language? I find difficult to debug because each
> time
> > > if I
> > > > find an issue I have to change file package upload to hdfs, rerun and
> > I'm
> > > > not very familiar with python
> > > >
> > > >
> > > > Best
> > > > Siyuan
> > > >
> > >
> > >
> > >
> > > --
> > > thanks
> > > Sumit
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: Why command script is written in python?

[Sumit Mohanty <sm...@hortonworks.com>]

Agree. In fact, we need to also document how Slider scripts can be debugged
during development.

If you do not mind can you open a JIRA at
https://issues.apache.org/jira/browse/SLIDER describing the requirements of
"debugging" during development time and post development. I will use that
JIRA to drive the issue.

-Sumit

On Thu, Oct 30, 2014 at 2:40 PM, hsy541@gmail.com <hs...@gmail.com> wrote:

> Thanks Sumit,
>
> The only concern about script language is you don't know the compile error
> until runtime. And in fact it's not arbitrary script it depends on the
> library in slider for example
> /python/resource_management/core/resources/system.py
>
> Best,
> Siyuan
>
> On Thu, Oct 30, 2014 at 2:24 PM, Sumit Mohanty <su...@gmail.com>
> wrote:
>
> > Actually, you can debug it after deploying it once. This will require
> > setting some YARN variables to ensure containers are not released even if
> > the application fails. The agent logs report back the commands being
> > invoked and the parameters used to invoke them. Let me try to write that
> up
> > on the wiki. *I will need a day or so.*
> >
> > The choice of python was mostly a reflection of going for scripting
> > language that works on Windows and linux. As such it is possible to use
> any
> > scripting language as long as the agent code is modified to handle
> various
> > script types. Of course, today the only supported/tested one is python.
> >
> > On Thu, Oct 30, 2014 at 1:55 PM, hsy541@gmail.com <hs...@gmail.com>
> > wrote:
> >
> > > May I ask why do you choose python to write the command script. And is
> > > there  any other language? I find difficult to debug because each time
> > if I
> > > find an issue I have to change file package upload to hdfs, rerun and
> I'm
> > > not very familiar with python
> > >
> > >
> > > Best
> > > Siyuan
> > >
> >
> >
> >
> > --
> > thanks
> > Sumit
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Re: Why command script is written in python?

["hsy541@gmail.com" <hs...@gmail.com>]

Thanks Sumit,

The only concern about script language is you don't know the compile error
until runtime. And in fact it's not arbitrary script it depends on the
library in slider for example
/python/resource_management/core/resources/system.py

Best,
Siyuan

On Thu, Oct 30, 2014 at 2:24 PM, Sumit Mohanty <su...@gmail.com>
wrote:

> Actually, you can debug it after deploying it once. This will require
> setting some YARN variables to ensure containers are not released even if
> the application fails. The agent logs report back the commands being
> invoked and the parameters used to invoke them. Let me try to write that up
> on the wiki. *I will need a day or so.*
>
> The choice of python was mostly a reflection of going for scripting
> language that works on Windows and linux. As such it is possible to use any
> scripting language as long as the agent code is modified to handle various
> script types. Of course, today the only supported/tested one is python.
>
> On Thu, Oct 30, 2014 at 1:55 PM, hsy541@gmail.com <hs...@gmail.com>
> wrote:
>
> > May I ask why do you choose python to write the command script. And is
> > there  any other language? I find difficult to debug because each time
> if I
> > find an issue I have to change file package upload to hdfs, rerun and I'm
> > not very familiar with python
> >
> >
> > Best
> > Siyuan
> >
>
>
>
> --
> thanks
> Sumit
>

Re: Why command script is written in python?

[Sumit Mohanty <su...@gmail.com>]

Actually, you can debug it after deploying it once. This will require
setting some YARN variables to ensure containers are not released even if
the application fails. The agent logs report back the commands being
invoked and the parameters used to invoke them. Let me try to write that up
on the wiki. *I will need a day or so.*

The choice of python was mostly a reflection of going for scripting
language that works on Windows and linux. As such it is possible to use any
scripting language as long as the agent code is modified to handle various
script types. Of course, today the only supported/tested one is python.

On Thu, Oct 30, 2014 at 1:55 PM, hsy541@gmail.com <hs...@gmail.com> wrote:

> May I ask why do you choose python to write the command script. And is
> there  any other language? I find difficult to debug because each time if I
> find an issue I have to change file package upload to hdfs, rerun and I'm
> not very familiar with python
>
>
> Best
> Siyuan
>



-- 
thanks
Sumit