You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "huipanc@163.com" <hu...@163.com> on 2017/06/26 08:20:31 UTC
A newcomer Question about nifi use
Hi,
I am a newcomer of using nifi. I mean to use nifi to execute remote ssh shell. So I want to query there is a possibilty to use nifi to execute remote ssh shell (just like using oozie ssh to excute remote ssh shell）.
Thanks,
Hollis
Re: Re: A newcomer Question about nifi use
Posted by "huipanc@163.com" <hu...@163.com>.
Hi Koji,
Thank you very much for your help. I will have a try.
Thanks,
Hollis
From: Koji Kawamura
Date: 2017-06-27 13:56
To: users
Subject: Re: Re: A newcomer Question about nifi use
Hi Hollis,
If you haven't done yet, I'd strongly recommend to take a look on the NiFi User Guid doc [1] to get familiar with how to build a NiFi flow.
In case you prefer video, there's a great series of how-to [2].
Then you'll understand how to address those warning messages shown on NiFi UI.
Once you got basic idea how NiFi flow works, you will need to consider "When the remote shell should be executed", because NiFi is more of a streaming data processing application rather than an batch oriented scheduled ETL tool.
For example, if you'd like to call the remote script after putting certain files from local to HDFS, then you can build a NiFi flow looks like:
GetFile -> (MergeContent, optionally to create larger files) -> PutHDFS -> ExecuteStreamCommand
There are different processors those can be used to trigger actions (ListXXX, ListenXXX, ConsumeXXX ... etc) for different use-cases.
[1] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#building-dataflow
[2] https://www.youtube.com/watch?v=Y5znvcJ_NWo&list=PLHre9pIBAgc4e-tiq9OIXkWJX8bVXuqlG
Thanks,
Koji
On Tue, Jun 27, 2017 at 12:45 PM, huipanc@163.com <hu...@163.com> wrote:
Hi Koji,
Thank you much very. I use ExecuteStreamCommand to execute remote ssh shell. But I don't know how to run a full flow. My problem is described as below. Hence what is 'Relationship output strem', 'Relationship original' and 'upstream Connections'. I cannot find any reference in nifi api document till now.
My aim is to execute ssh shell in my hdfs machine.Can you give me a fully example. Thanks.
Thanks,
Hollis
From: Koji Kawamura
Date: 2017-06-26 16:49
To: users
Subject: Re: A newcomer Question about nifi use
Hi Hollis,
To execute a remote shell using ssh, I'd use 'sh' command and its -c
option with ExecuteStreamCommand or ExecuteProcess.
For example, ExecuteStreamCommand configuration would look like:
- Command Path: sh
- Command Arguments: -c;ssh user@host.example.com command-to-execute-on-remote
- Argument Delimiter: ;
Thanks,
Koji
On Mon, Jun 26, 2017 at 5:20 PM, huipanc@163.com <hu...@163.com> wrote:
> Hi,
> I am a newcomer of using nifi. I mean to use nifi to execute remote ssh
> shell. So I want to query there is a possibilty to use nifi to execute
> remote ssh shell (just like using oozie ssh to excute remote ssh shell）.
>
> ________________________________
> Thanks,
> Hollis
Re: Re: A newcomer Question about nifi use
Posted by Koji Kawamura <ij...@gmail.com>.
Hi Hollis,
If you haven't done yet, I'd strongly recommend to take a look on the NiFi
User Guid doc [1] to get familiar with how to build a NiFi flow.
In case you prefer video, there's a great series of how-to [2].
Then you'll understand how to address those warning messages shown on NiFi
UI.
Once you got basic idea how NiFi flow works, you will need to consider
"When the remote shell should be executed", because NiFi is more of a
streaming data processing application rather than an batch oriented
scheduled ETL tool.
For example, if you'd like to call the remote script after putting certain
files from local to HDFS, then you can build a NiFi flow looks like:
GetFile -> (MergeContent, optionally to create larger files) -> PutHDFS ->
ExecuteStreamCommand
There are different processors those can be used to trigger actions
(ListXXX, ListenXXX, ConsumeXXX ... etc) for different use-cases.
[1]
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#building-dataflow
[2]
https://www.youtube.com/watch?v=Y5znvcJ_NWo&list=PLHre9pIBAgc4e-tiq9OIXkWJX8bVXuqlG
Thanks,
Koji
On Tue, Jun 27, 2017 at 12:45 PM, huipanc@163.com <hu...@163.com> wrote:
> Hi Koji,
> Thank you much very. I use ExecuteStreamCommand to execute remote
> ssh shell. But I don't know how to run a full flow. My problem is
> described as below. Hence what is 'Relationship output strem',
> 'Relationship original' and 'upstream Connections'. I cannot find any
> reference in nifi api document till now.
> My aim is to execute ssh shell in my hdfs machine.Can you give me
> a fully example. Thanks.
>
>
> ------------------------------
> Thanks,
> Hollis
>
> *From:* Koji Kawamura <ij...@gmail.com>
> *Date:* 2017-06-26 16:49
> *To:* users <us...@nifi.apache.org>
> *Subject:* Re: A newcomer Question about nifi use
> Hi Hollis,
>
> To execute a remote shell using ssh, I'd use 'sh' command and its -c
> option with ExecuteStreamCommand or ExecuteProcess.
>
> For example, ExecuteStreamCommand configuration would look like:
> - Command Path: sh
> - Command Arguments: -c;ssh user@host.example.com
> command-to-execute-on-remote
> - Argument Delimiter: ;
>
> Thanks,
> Koji
>
> On Mon, Jun 26, 2017 at 5:20 PM, huipanc@163.com <hu...@163.com> wrote:
> > Hi,
> > I am a newcomer of using nifi. I mean to use nifi to execute remote
> ssh
> > shell. So I want to query there is a possibilty to use nifi to execute
> > remote ssh shell (just like using oozie ssh to excute remote ssh shell）.
> >
> > ________________________________
> > Thanks,
> > Hollis
>
>
Re: Re: A newcomer Question about nifi use
Posted by "huipanc@163.com" <hu...@163.com>.
Hi Koji,
Thank you much very. I use ExecuteStreamCommand to execute remote ssh shell. But I don't know how to run a full flow. My problem is described as below. Hence what is 'Relationship output strem', 'Relationship original' and 'upstream Connections'. I cannot find any reference in nifi api document till now.
My aim is to execute ssh shell in my hdfs machine.Can you give me a fully example. Thanks.
Thanks,
Hollis
From: Koji Kawamura
Date: 2017-06-26 16:49
To: users
Subject: Re: A newcomer Question about nifi use
Hi Hollis,
To execute a remote shell using ssh, I'd use 'sh' command and its -c
option with ExecuteStreamCommand or ExecuteProcess.
For example, ExecuteStreamCommand configuration would look like:
- Command Path: sh
- Command Arguments: -c;ssh user@host.example.com command-to-execute-on-remote
- Argument Delimiter: ;
Thanks,
Koji
On Mon, Jun 26, 2017 at 5:20 PM, huipanc@163.com <hu...@163.com> wrote:
> Hi,
> I am a newcomer of using nifi. I mean to use nifi to execute remote ssh
> shell. So I want to query there is a possibilty to use nifi to execute
> remote ssh shell (just like using oozie ssh to excute remote ssh shell）.
>
> ________________________________
> Thanks,
> Hollis
Re: A newcomer Question about nifi use
Posted by Koji Kawamura <ij...@gmail.com>.
Hi Hollis,
To execute a remote shell using ssh, I'd use 'sh' command and its -c
option with ExecuteStreamCommand or ExecuteProcess.
For example, ExecuteStreamCommand configuration would look like:
- Command Path: sh
- Command Arguments: -c;ssh user@host.example.com command-to-execute-on-remote
- Argument Delimiter: ;
Thanks,
Koji
On Mon, Jun 26, 2017 at 5:20 PM, huipanc@163.com <hu...@163.com> wrote:
> Hi,
> I am a newcomer of using nifi. I mean to use nifi to execute remote ssh
> shell. So I want to query there is a possibilty to use nifi to execute
> remote ssh shell (just like using oozie ssh to excute remote ssh shell）.
>
> ________________________________
> Thanks,
> Hollis