You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Sandesh Hegde <sa...@datatorrent.com> on 2016/05/23 17:57:39 UTC
Getting various statitstics from StreamingContainers
Hello All,
Getting various information from the StreamingConatainers is a useful
feature to have.
As StreamingContainers are JVMs, various JDK tools can be used to get the
information.
So the idea is to spawn the new process from the streaming containers and
return the information via Stram.
Recently we have added the feature to get stack trace I have modified that
to show the idea I am talking about.
Here is the pull request, the purpose of that is to show the idea, let me
know your thoughts.
https://github.com/apache/incubator-apex-core/pull/340
I have not created a jira yet, wanted to check the viability of the idea.
Thanks
Sandesh
Re: Getting various statitstics from StreamingContainers
Posted by Sandesh Hegde <sa...@datatorrent.com>.
Looked around to see, how others are implementing this, found the feature
implemented in Storm.
Storm supports whitelist of commands to run heap dumps/stack and few
others.
Looked at their implementation, they are also using new process to lauch
the JDK tools to get the information.
https://issues.apache.org/jira/browse/STORM-1157
On Mon, May 23, 2016 at 11:31 AM Sandesh Hegde <sa...@datatorrent.com>
wrote:
> After connecting to the app user will run the following command.
>
> Users will select the container id, jdk tool and the arguments to the tool.
>
> Apex CLI api
> run-jdkTools "Container-id" "Tool-name" "Arguments"
>
> Output
> of the command is interpreted by the user.
>
>
>
> On Mon, May 23, 2016 at 11:31 AM Thomas Weise <th...@datatorrent.com>
> wrote:
>
>> I think it is appropriate to collect the information that the JVM provides
>> using the available API instead of running external processes.
>>
>> For other information, how do you suggest that will be provided to the
>> user?
>>
>> Thanks,
>> Thomas
>>
>>
>>
>> On Mon, May 23, 2016 at 11:27 AM, Sandesh Hegde <sa...@datatorrent.com>
>> wrote:
>>
>> > Users can pass the arguments to the JDK tools. So it exposes all the
>> power
>> > of those tools. If we have to write the code we are doing the duplicate
>> > work.
>> > Also it doesn't evolve with the new features of the JVM, but the tools
>> will
>> > and we just have to change the arguments that we pass.
>> >
>> > On Mon, May 23, 2016 at 11:15 AM Vlad Rozov <v....@datatorrent.com>
>> > wrote:
>> >
>> > > What is the purpose of the new process? Why that information can't be
>> > > collected directly from JVM and passed to app master using heartbeat?
>> > >
>> > > Thank you,
>> > > Vlad
>> > >
>> > > On 5/23/16 10:57, Sandesh Hegde wrote:
>> > > > Hello All,
>> > > >
>> > > > Getting various information from the StreamingConatainers is a
>> useful
>> > > > feature to have.
>> > > > As StreamingContainers are JVMs, various JDK tools can be used to
>> get
>> > the
>> > > > information.
>> > > >
>> > > > So the idea is to spawn the new process from the streaming
>> containers
>> > and
>> > > > return the information via Stram.
>> > > >
>> > > > Recently we have added the feature to get stack trace I have
>> modified
>> > > that
>> > > > to show the idea I am talking about.
>> > > >
>> > > > Here is the pull request, the purpose of that is to show the idea,
>> let
>> > me
>> > > > know your thoughts.
>> > > > https://github.com/apache/incubator-apex-core/pull/340
>> > > >
>> > > > I have not created a jira yet, wanted to check the viability of the
>> > idea.
>> > > >
>> > > > Thanks
>> > > > Sandesh
>> > > >
>> > >
>> > >
>> >
>>
>
Re: Getting various statitstics from StreamingContainers
Posted by Sandesh Hegde <sa...@datatorrent.com>.
After connecting to the app user will run the following command.
Users will select the container id, jdk tool and the arguments to the tool.
Apex CLI api
run-jdkTools "Container-id" "Tool-name" "Arguments"
Output
of the command is interpreted by the user.
On Mon, May 23, 2016 at 11:31 AM Thomas Weise <th...@datatorrent.com>
wrote:
> I think it is appropriate to collect the information that the JVM provides
> using the available API instead of running external processes.
>
> For other information, how do you suggest that will be provided to the
> user?
>
> Thanks,
> Thomas
>
>
>
> On Mon, May 23, 2016 at 11:27 AM, Sandesh Hegde <sa...@datatorrent.com>
> wrote:
>
> > Users can pass the arguments to the JDK tools. So it exposes all the
> power
> > of those tools. If we have to write the code we are doing the duplicate
> > work.
> > Also it doesn't evolve with the new features of the JVM, but the tools
> will
> > and we just have to change the arguments that we pass.
> >
> > On Mon, May 23, 2016 at 11:15 AM Vlad Rozov <v....@datatorrent.com>
> > wrote:
> >
> > > What is the purpose of the new process? Why that information can't be
> > > collected directly from JVM and passed to app master using heartbeat?
> > >
> > > Thank you,
> > > Vlad
> > >
> > > On 5/23/16 10:57, Sandesh Hegde wrote:
> > > > Hello All,
> > > >
> > > > Getting various information from the StreamingConatainers is a useful
> > > > feature to have.
> > > > As StreamingContainers are JVMs, various JDK tools can be used to get
> > the
> > > > information.
> > > >
> > > > So the idea is to spawn the new process from the streaming containers
> > and
> > > > return the information via Stram.
> > > >
> > > > Recently we have added the feature to get stack trace I have modified
> > > that
> > > > to show the idea I am talking about.
> > > >
> > > > Here is the pull request, the purpose of that is to show the idea,
> let
> > me
> > > > know your thoughts.
> > > > https://github.com/apache/incubator-apex-core/pull/340
> > > >
> > > > I have not created a jira yet, wanted to check the viability of the
> > idea.
> > > >
> > > > Thanks
> > > > Sandesh
> > > >
> > >
> > >
> >
>
Re: Getting various statitstics from StreamingContainers
Posted by Thomas Weise <th...@datatorrent.com>.
I think it is appropriate to collect the information that the JVM provides
using the available API instead of running external processes.
For other information, how do you suggest that will be provided to the user?
Thanks,
Thomas
On Mon, May 23, 2016 at 11:27 AM, Sandesh Hegde <sa...@datatorrent.com>
wrote:
> Users can pass the arguments to the JDK tools. So it exposes all the power
> of those tools. If we have to write the code we are doing the duplicate
> work.
> Also it doesn't evolve with the new features of the JVM, but the tools will
> and we just have to change the arguments that we pass.
>
> On Mon, May 23, 2016 at 11:15 AM Vlad Rozov <v....@datatorrent.com>
> wrote:
>
> > What is the purpose of the new process? Why that information can't be
> > collected directly from JVM and passed to app master using heartbeat?
> >
> > Thank you,
> > Vlad
> >
> > On 5/23/16 10:57, Sandesh Hegde wrote:
> > > Hello All,
> > >
> > > Getting various information from the StreamingConatainers is a useful
> > > feature to have.
> > > As StreamingContainers are JVMs, various JDK tools can be used to get
> the
> > > information.
> > >
> > > So the idea is to spawn the new process from the streaming containers
> and
> > > return the information via Stram.
> > >
> > > Recently we have added the feature to get stack trace I have modified
> > that
> > > to show the idea I am talking about.
> > >
> > > Here is the pull request, the purpose of that is to show the idea, let
> me
> > > know your thoughts.
> > > https://github.com/apache/incubator-apex-core/pull/340
> > >
> > > I have not created a jira yet, wanted to check the viability of the
> idea.
> > >
> > > Thanks
> > > Sandesh
> > >
> >
> >
>
Re: Getting various statitstics from StreamingContainers
Posted by Sandesh Hegde <sa...@datatorrent.com>.
Users can pass the arguments to the JDK tools. So it exposes all the power
of those tools. If we have to write the code we are doing the duplicate
work.
Also it doesn't evolve with the new features of the JVM, but the tools will
and we just have to change the arguments that we pass.
On Mon, May 23, 2016 at 11:15 AM Vlad Rozov <v....@datatorrent.com> wrote:
> What is the purpose of the new process? Why that information can't be
> collected directly from JVM and passed to app master using heartbeat?
>
> Thank you,
> Vlad
>
> On 5/23/16 10:57, Sandesh Hegde wrote:
> > Hello All,
> >
> > Getting various information from the StreamingConatainers is a useful
> > feature to have.
> > As StreamingContainers are JVMs, various JDK tools can be used to get the
> > information.
> >
> > So the idea is to spawn the new process from the streaming containers and
> > return the information via Stram.
> >
> > Recently we have added the feature to get stack trace I have modified
> that
> > to show the idea I am talking about.
> >
> > Here is the pull request, the purpose of that is to show the idea, let me
> > know your thoughts.
> > https://github.com/apache/incubator-apex-core/pull/340
> >
> > I have not created a jira yet, wanted to check the viability of the idea.
> >
> > Thanks
> > Sandesh
> >
>
>
Re: Getting various statitstics from StreamingContainers
Posted by Vlad Rozov <v....@datatorrent.com>.
What is the purpose of the new process? Why that information can't be
collected directly from JVM and passed to app master using heartbeat?
Thank you,
Vlad
On 5/23/16 10:57, Sandesh Hegde wrote:
> Hello All,
>
> Getting various information from the StreamingConatainers is a useful
> feature to have.
> As StreamingContainers are JVMs, various JDK tools can be used to get the
> information.
>
> So the idea is to spawn the new process from the streaming containers and
> return the information via Stram.
>
> Recently we have added the feature to get stack trace I have modified that
> to show the idea I am talking about.
>
> Here is the pull request, the purpose of that is to show the idea, let me
> know your thoughts.
> https://github.com/apache/incubator-apex-core/pull/340
>
> I have not created a jira yet, wanted to check the viability of the idea.
>
> Thanks
> Sandesh
>