You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by sivasankar Reddy <ma...@gmail.com> on 2019/08/08 20:17:32 UTC

nifi deploy automation - variables pass through (incl controller services)

Hi, 

We are looking for nifi deploy automation from regsitry -> higher nifi instance. Quick observations during intial evaluation. 

1. After process group import, controller services are not set in nifi instance
2. How to set variables from source process group to destination process group. 
3. Is there any possibility of definining the positioning of process group to be imported in Nifi Canvas. 

Any other best practices  regarding NiFi deployment automation will be helpful. 

Thanks in Advance!


Re: nifi deploy automation - variables pass through (incl controller services)

Posted by Bryan Bende <bb...@gmail.com>.
Using the REST API to set a property was meant only for sensitive
properties since they cant use variables. Changes to sensitive properties
do not produce a change that is tracked by version control.

All of this will be unnecessary with the new parameters functionality in
1.10.0.

On Mon, Sep 2, 2019 at 10:59 PM Andrew Grande <ap...@gmail.com> wrote:

> I think there is confusion here. Changing *properties* is considered a
> tracked change, the behavior is per design.
>
> There are *variables* on the PG level which are considered non-trackable
> environment settings and won't trigger a dirty flag.
>
> There is a dedicated command in the cli to list and set these *variables*.
>
> Hope it helps,
> Andrew
>
> On Mon, Sep 2, 2019, 4:07 PM sivasankar Reddy <ma...@gmail.com>
> wrote:
>
> > Hi Bryan,
> >
> > Thanks for the options. After pg-import or pg-change-version, if we use
> > nifi rest api and update properties, looks like nifi instance detects
> > changes(compares with bucket flow in registry) and creates * mark.
> >
> > We can change pg-change-version however just for properties setting , may
> > be its not advisable to change version in nifi instance.
> >
> > Any ideas around this will be helpful.
> >
> > Thanks
> > Siva
> >
> > On 2019/08/23 16:23:38, Bryan Bende <bb...@gmail.com> wrote:
> > > Well there are two different encryption parts here -
> > > encryption-at-rest and encryption-in-transit...
> > >
> > > Encryption-at-rest would have to be part of your script that is
> > > calling the REST API. Somehow you create a config file with encrypted
> > > values, and your script needs to read those in an decrypt them in
> > > memory and then put them into the content that will be sent in the
> > > HTTP PUT or POST request.
> > >
> > > Encryption-in-transit would be handled by the fact that you would be
> > > making an HTTPS request to NiFi, so the contents of the POST or PUT
> > > request are encrypted in transit. This is the same thing that is
> > > happening if you were in the NiFi UI and typed in a password into the
> > > sensitive property field.
> > >
> > >
> > > On Fri, Aug 23, 2019 at 11:35 AM sivasankar Reddy <
> mail2msiva@gmail.com>
> > wrote:
> > > >
> > > > Hi Bryan,
> > > >
> > > > Thanks for the reply.
> > > > Able to get the json object and set the property for Password.
> However
> > value of password can be either referred from config file (its plain
> string
> > visible in config). Is there any way that password is encrypted and this
> > set property can take decrypted value.
> > > >
> > > > this way we ensure that password is sensitive. Any other ideas to
> > achieve this
> > > >
> > > >
> > > > Regards,
> > > > Siva
> > > >
> > > >
> > > >
> > > > On 2019/08/22 17:38:37, Bryan Bende <bb...@gmail.com> wrote:
> > > > > The parameters work is on-going and will be in the next release
> which
> > > > > is 1.10.0. Releases don't really have specific timelines, but
> > > > > generally they happen every few months. Most likely 1.10.0 is a
> > couple
> > > > > of weeks away, but depends on when active work is completed and
> when
> > > > > someone volunteers to make a release.
> > > > >
> > > > > There is no timeline for the "set-property" command since it was
> just
> > > > > suggested as a new feature in this email thread :) It requires
> > someone
> > > > > creating a JIRA and deciding to work on it.
> > > > >
> > > > > All of the functionality in the CLI and NiPyAPI, and even NiFi's
> own
> > > > > UI, is based on the REST API. So you can still perform a "set
> > > > > property" by using the REST API to modify the configuration of a
> > > > > processor, the CLI would just make it easier so that you wouldn't
> > have
> > > > > to understand the lower level API details. The best way to
> understand
> > > > > the API calls is to us the UI while you have Chrome Dev Tools open,
> > > > > and then perform the action you are interested in and look at the
> > > > > requests made on the Network tab. You'll be able to see what URLs
> are
> > > > > called and what the request and response bodies look like.
> > > > >
> > > > > On Thu, Aug 22, 2019 at 11:46 AM sivasankar Reddy <
> > mail2msiva@gmail.com> wrote:
> > > > > >
> > > > > > Hi Bryan,
> > > > > >
> > > > > > Thanks for the reply. looks like even set-property will be a new
> > feature, as current CLI doesn't have that.
> > > > > >
> > > > > > Could you please share timelines of these features if its in
> > roadmap.
> > > > > > 1. "set-property"
> > > > > > 2. “parameters”
> > > > > >
> > > > > > The only option currently is set to set sensitive parameters
> > manually? or any other option through CLI
> > > > > >
> > > > > > Regards,
> > > > > > Siva
> > > > > >
> > > > > > On 2019/08/22 04:10:46, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > > Currently there isn’t a good way with the CLI, we would need to
> > add a
> > > > > > > command like set-property that took the id of a component, and
> > the name and
> > > > > > > value for a property.
> > > > > > >
> > > > > > > The next release will have a new feature called “parameters”
> > which will
> > > > > > > solve this problem. You’ll be able to use a parameter in a
> > sensitive
> > > > > > > property and then use CLI to set parameter values.
> > > > > > >
> > > > > > > On Wed, Aug 21, 2019 at 12:29 PM sivasankar Reddy <
> > mail2msiva@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Bryan,
> > > > > > > >
> > > > > > > > Thanks for the update. Could you please mention about
> > > > > > > > sensitive parameters aspect from deployment point of view
> > using CLI.
> > > > > > > >
> > > > > > > > How to set the sensitive parameters values after pg-import.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Siva
> > > > > > > >
> > > > > > > > On 2019/08/21 11:47:49, Bryan Bende <bb...@gmail.com>
> wrote:
> > > > > > > > > It does not, I was saying we could easily improve the
> > command to add
> > > > > > > > those
> > > > > > > > > parameters.
> > > > > > > > >
> > > > > > > > > On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy <
> > mail2msiva@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi  Bryan,
> > > > > > > > > >
> > > > > > > > > > Thanks for the update. Does the existing pg-import takes
> > optional
> > > > > > > > params
> > > > > > > > > > for x/y coordinates. If yes can you share the sample
> > command as
> > > > > > > > pg-import
> > > > > > > > > > --help doesn't has this info.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > siva
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 2019/08/20 22:19:23, Bryan Bende <bb...@gmail.com>
> > wrote:
> > > > > > > > > > > Hello,
> > > > > > > > > > >
> > > > > > > > > > > 1) pg-import will keeping finding a location that does
> > not overlap
> > > > > > > > with
> > > > > > > > > > any
> > > > > > > > > > > existing process groups, so it is more of auto layout.
> > We could
> > > > > > > > easily
> > > > > > > > > > add
> > > > > > > > > > > optional params for x/y coordinates.
> > > > > > > > > > >
> > > > > > > > > > > 2) Both are fine choices.. CLI is more geared towards
> > simple
> > > > > > > > commands,
> > > > > > > > > > > where as with NiPyApi you could more easily write
> > complex logic in
> > > > > > > > > > Python.
> > > > > > > > > > > With CLI you would write a wrapper shell script that
> > made individual
> > > > > > > > > > calls
> > > > > > > > > > > to the CLI. I think it is really just a matter of
> > preference.
> > > > > > > > > > >
> > > > > > > > > > > -Bryan
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy <
> > > > > > > > mail2msiva@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Bryan,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for your input. I would like to know more
> > > > > > > > > > > >
> > > > > > > > > > > > 1. how does CLI pg-import  works if already canvas
> has
> > 100 to 150
> > > > > > > > > > flows.
> > > > > > > > > > > > Will CLI pg-import always place the flow in canvas
> > which can be
> > > > > > > > easily
> > > > > > > > > > > > located. As nipyapi has location(x,y) during deploy
> > flow.
> > > > > > > > > > > >
> > > > > > > > > > > > 2. From enterprise usage, do you recommend toolkit
> CLI
> > vs nipyapi,
> > > > > > > > it
> > > > > > > > > > > > seems toolkit CLI is part of official Nifi.
> > > > > > > > > > > >
> > > > > > > > > > > > Any other points regarding this will be helpful.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Siva
> > > > > > > > > > > >
> > > > > > > > > > > > On 2019/08/08 22:41:43, Bryan Bende <
> bbende@gmail.com>
> > wrote:
> > > > > > > > > > > > > Hello,
> > > > > > > > > > > > >
> > > > > > > > > > > > > A lot depends on whether you are doing the
> promotion
> > with the UI
> > > > > > > > or
> > > > > > > > > > > > > with scripts using something like NiFi CLI or
> > NiPyApi.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1) Currently references to controller services
> > outside the
> > > > > > > > versioned
> > > > > > > > > > > > > process group need to be set after initial import.
> > In the next
> > > > > > > > > > release
> > > > > > > > > > > > > it will auto-select by name as long as there is
> only
> > one service
> > > > > > > > with
> > > > > > > > > > > > > the given name and type in a parent group (name is
> > not unique).
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2) The snapshot of the variables are part of the
> > versioned flow,
> > > > > > > > so
> > > > > > > > > > > > > after importing you would have to set the new
> values
> > in the
> > > > > > > > target
> > > > > > > > > > > > > environment. This can be done in the UI or with
> > scripts.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 3) The REST API allows for positioning of any
> > component that is
> > > > > > > > > > > > > created. i can't speak for how to do it with
> > NiPyApi, but I know
> > > > > > > > the
> > > > > > > > > > > > > CLI pg-import command attempts to automatically
> > organize the
> > > > > > > > imported
> > > > > > > > > > > > > process groups on the root canvas so that they
> don't
> > overlap.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Bryan
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <
> > > > > > > > > > mail2msiva@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We are looking for nifi deploy automation from
> > regsitry ->
> > > > > > > > higher
> > > > > > > > > > nifi
> > > > > > > > > > > > instance. Quick observations during intial
> evaluation.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 1. After process group import, controller
> services
> > are not set
> > > > > > > > in
> > > > > > > > > > nifi
> > > > > > > > > > > > instance
> > > > > > > > > > > > > > 2. How to set variables from source process group
> > to
> > > > > > > > destination
> > > > > > > > > > > > process group.
> > > > > > > > > > > > > > 3. Is there any possibility of definining the
> > positioning of
> > > > > > > > > > process
> > > > > > > > > > > > group to be imported in Nifi Canvas.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Any other best practices  regarding NiFi
> > deployment automation
> > > > > > > > > > will be
> > > > > > > > > > > > helpful.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks in Advance!
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Sent from Gmail Mobile
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Sent from Gmail Mobile
> > > > > > > > >
> > > > > > > >
> > > > > > > --
> > > > > > > Sent from Gmail Mobile
> > > > > > >
> > > > >
> > >
> >
>
-- 
Sent from Gmail Mobile

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by Bryan Bende <bb...@gmail.com>.
The short answer is - please revisit all of this after the 1.10.0
release because all of this will be solved with the new parameters
feature :)

Longer answer...

The CLI has two commands:

nifi pg-get-vars
nifi pg-set-var

These can be used to get and set process group level variables, which
can be referenced in any property that supports EL with at least
VARIABLE_REGISTRY scope.

Sensitive property values are never sent to registry and thus have to
be set one time after first importing the flow from registry. This is
what the discussion of using the REST API was for, since the CLI does
not have a 'set-property' command to set a value of a single property
on a component.

On Tue, Sep 3, 2019 at 4:13 PM Evan Reynolds <ev...@usermind.com.invalid> wrote:
>
> There is a CLI command to change variables ... ? I've moved most of my processor variables to extra-args.property, so it's easy to change them. But there are a few I can't change - for example, DistributedMapCacheClientService controller service has a "Server Hostname" property that doesn't support the expression language, so we have to essentially search and replace the string in the flow.xml file before deploying it. Similarly, the DBCPConnectionPool has a password that is a sensitive values - so even if we set it to an argument, that argument gets wiped when we go through the registry to deploy, so we are currently deploying and then going and manually filling in that password.
>
> I just looked again at the CLI but I didn't see a command that would set them, but I may have very easily missed it! Is there?
>
> ---------------------------------------------------------------
>
> Evan Reynolds
> evan@usermind.com
>
>
> On 9/2/19, 7:59 PM, "Andrew Grande" <ap...@gmail.com> wrote:
>
>     I think there is confusion here. Changing *properties* is considered a
>     tracked change, the behavior is per design.
>
>     There are *variables* on the PG level which are considered non-trackable
>     environment settings and won't trigger a dirty flag.
>
>     There is a dedicated command in the cli to list and set these *variables*.
>
>     Hope it helps,
>     Andrew
>
>     On Mon, Sep 2, 2019, 4:07 PM sivasankar Reddy <ma...@gmail.com> wrote:
>
>     > Hi Bryan,
>     >
>     > Thanks for the options. After pg-import or pg-change-version, if we use
>     > nifi rest api and update properties, looks like nifi instance detects
>     > changes(compares with bucket flow in registry) and creates * mark.
>     >
>     > We can change pg-change-version however just for properties setting , may
>     > be its not advisable to change version in nifi instance.
>     >
>     > Any ideas around this will be helpful.
>     >
>     > Thanks
>     > Siva
>     >
>     > On 2019/08/23 16:23:38, Bryan Bende <bb...@gmail.com> wrote:
>     > > Well there are two different encryption parts here -
>     > > encryption-at-rest and encryption-in-transit...
>     > >
>     > > Encryption-at-rest would have to be part of your script that is
>     > > calling the REST API. Somehow you create a config file with encrypted
>     > > values, and your script needs to read those in an decrypt them in
>     > > memory and then put them into the content that will be sent in the
>     > > HTTP PUT or POST request.
>     > >
>     > > Encryption-in-transit would be handled by the fact that you would be
>     > > making an HTTPS request to NiFi, so the contents of the POST or PUT
>     > > request are encrypted in transit. This is the same thing that is
>     > > happening if you were in the NiFi UI and typed in a password into the
>     > > sensitive property field.
>     > >
>     > >
>     > > On Fri, Aug 23, 2019 at 11:35 AM sivasankar Reddy <ma...@gmail.com>
>     > wrote:
>     > > >
>     > > > Hi Bryan,
>     > > >
>     > > > Thanks for the reply.
>     > > > Able to get the json object and set the property for Password. However
>     > value of password can be either referred from config file (its plain string
>     > visible in config). Is there any way that password is encrypted and this
>     > set property can take decrypted value.
>     > > >
>     > > > this way we ensure that password is sensitive. Any other ideas to
>     > achieve this
>     > > >
>     > > >
>     > > > Regards,
>     > > > Siva
>     > > >
>     > > >
>     > > >
>     > > > On 2019/08/22 17:38:37, Bryan Bende <bb...@gmail.com> wrote:
>     > > > > The parameters work is on-going and will be in the next release which
>     > > > > is 1.10.0. Releases don't really have specific timelines, but
>     > > > > generally they happen every few months. Most likely 1.10.0 is a
>     > couple
>     > > > > of weeks away, but depends on when active work is completed and when
>     > > > > someone volunteers to make a release.
>     > > > >
>     > > > > There is no timeline for the "set-property" command since it was just
>     > > > > suggested as a new feature in this email thread :) It requires
>     > someone
>     > > > > creating a JIRA and deciding to work on it.
>     > > > >
>     > > > > All of the functionality in the CLI and NiPyAPI, and even NiFi's own
>     > > > > UI, is based on the REST API. So you can still perform a "set
>     > > > > property" by using the REST API to modify the configuration of a
>     > > > > processor, the CLI would just make it easier so that you wouldn't
>     > have
>     > > > > to understand the lower level API details. The best way to understand
>     > > > > the API calls is to us the UI while you have Chrome Dev Tools open,
>     > > > > and then perform the action you are interested in and look at the
>     > > > > requests made on the Network tab. You'll be able to see what URLs are
>     > > > > called and what the request and response bodies look like.
>     > > > >
>     > > > > On Thu, Aug 22, 2019 at 11:46 AM sivasankar Reddy <
>     > mail2msiva@gmail.com> wrote:
>     > > > > >
>     > > > > > Hi Bryan,
>     > > > > >
>     > > > > > Thanks for the reply. looks like even set-property will be a new
>     > feature, as current CLI doesn't have that.
>     > > > > >
>     > > > > > Could you please share timelines of these features if its in
>     > roadmap.
>     > > > > > 1. "set-property"
>     > > > > > 2. “parameters”
>     > > > > >
>     > > > > > The only option currently is set to set sensitive parameters
>     > manually? or any other option through CLI
>     > > > > >
>     > > > > > Regards,
>     > > > > > Siva
>     > > > > >
>     > > > > > On 2019/08/22 04:10:46, Bryan Bende <bb...@gmail.com> wrote:
>     > > > > > > Currently there isn’t a good way with the CLI, we would need to
>     > add a
>     > > > > > > command like set-property that took the id of a component, and
>     > the name and
>     > > > > > > value for a property.
>     > > > > > >
>     > > > > > > The next release will have a new feature called “parameters”
>     > which will
>     > > > > > > solve this problem. You’ll be able to use a parameter in a
>     > sensitive
>     > > > > > > property and then use CLI to set parameter values.
>     > > > > > >
>     > > > > > > On Wed, Aug 21, 2019 at 12:29 PM sivasankar Reddy <
>     > mail2msiva@gmail.com>
>     > > > > > > wrote:
>     > > > > > >
>     > > > > > > > Hi Bryan,
>     > > > > > > >
>     > > > > > > > Thanks for the update. Could you please mention about
>     > > > > > > > sensitive parameters aspect from deployment point of view
>     > using CLI.
>     > > > > > > >
>     > > > > > > > How to set the sensitive parameters values after pg-import.
>     > > > > > > >
>     > > > > > > > Regards,
>     > > > > > > > Siva
>     > > > > > > >
>     > > > > > > > On 2019/08/21 11:47:49, Bryan Bende <bb...@gmail.com> wrote:
>     > > > > > > > > It does not, I was saying we could easily improve the
>     > command to add
>     > > > > > > > those
>     > > > > > > > > parameters.
>     > > > > > > > >
>     > > > > > > > > On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy <
>     > mail2msiva@gmail.com>
>     > > > > > > > > wrote:
>     > > > > > > > >
>     > > > > > > > > > Hi  Bryan,
>     > > > > > > > > >
>     > > > > > > > > > Thanks for the update. Does the existing pg-import takes
>     > optional
>     > > > > > > > params
>     > > > > > > > > > for x/y coordinates. If yes can you share the sample
>     > command as
>     > > > > > > > pg-import
>     > > > > > > > > > --help doesn't has this info.
>     > > > > > > > > >
>     > > > > > > > > > Regards,
>     > > > > > > > > > siva
>     > > > > > > > > >
>     > > > > > > > > >
>     > > > > > > > > > On 2019/08/20 22:19:23, Bryan Bende <bb...@gmail.com>
>     > wrote:
>     > > > > > > > > > > Hello,
>     > > > > > > > > > >
>     > > > > > > > > > > 1) pg-import will keeping finding a location that does
>     > not overlap
>     > > > > > > > with
>     > > > > > > > > > any
>     > > > > > > > > > > existing process groups, so it is more of auto layout.
>     > We could
>     > > > > > > > easily
>     > > > > > > > > > add
>     > > > > > > > > > > optional params for x/y coordinates.
>     > > > > > > > > > >
>     > > > > > > > > > > 2) Both are fine choices.. CLI is more geared towards
>     > simple
>     > > > > > > > commands,
>     > > > > > > > > > > where as with NiPyApi you could more easily write
>     > complex logic in
>     > > > > > > > > > Python.
>     > > > > > > > > > > With CLI you would write a wrapper shell script that
>     > made individual
>     > > > > > > > > > calls
>     > > > > > > > > > > to the CLI. I think it is really just a matter of
>     > preference.
>     > > > > > > > > > >
>     > > > > > > > > > > -Bryan
>     > > > > > > > > > >
>     > > > > > > > > > >
>     > > > > > > > > > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy <
>     > > > > > > > mail2msiva@gmail.com>
>     > > > > > > > > > > wrote:
>     > > > > > > > > > >
>     > > > > > > > > > > > Hi Bryan,
>     > > > > > > > > > > >
>     > > > > > > > > > > > Thanks for your input. I would like to know more
>     > > > > > > > > > > >
>     > > > > > > > > > > > 1. how does CLI pg-import  works if already canvas has
>     > 100 to 150
>     > > > > > > > > > flows.
>     > > > > > > > > > > > Will CLI pg-import always place the flow in canvas
>     > which can be
>     > > > > > > > easily
>     > > > > > > > > > > > located. As nipyapi has location(x,y) during deploy
>     > flow.
>     > > > > > > > > > > >
>     > > > > > > > > > > > 2. From enterprise usage, do you recommend toolkit CLI
>     > vs nipyapi,
>     > > > > > > > it
>     > > > > > > > > > > > seems toolkit CLI is part of official Nifi.
>     > > > > > > > > > > >
>     > > > > > > > > > > > Any other points regarding this will be helpful.
>     > > > > > > > > > > >
>     > > > > > > > > > > > Regards,
>     > > > > > > > > > > > Siva
>     > > > > > > > > > > >
>     > > > > > > > > > > > On 2019/08/08 22:41:43, Bryan Bende <bb...@gmail.com>
>     > wrote:
>     > > > > > > > > > > > > Hello,
>     > > > > > > > > > > > >
>     > > > > > > > > > > > > A lot depends on whether you are doing the promotion
>     > with the UI
>     > > > > > > > or
>     > > > > > > > > > > > > with scripts using something like NiFi CLI or
>     > NiPyApi.
>     > > > > > > > > > > > >
>     > > > > > > > > > > > > 1) Currently references to controller services
>     > outside the
>     > > > > > > > versioned
>     > > > > > > > > > > > > process group need to be set after initial import.
>     > In the next
>     > > > > > > > > > release
>     > > > > > > > > > > > > it will auto-select by name as long as there is only
>     > one service
>     > > > > > > > with
>     > > > > > > > > > > > > the given name and type in a parent group (name is
>     > not unique).
>     > > > > > > > > > > > >
>     > > > > > > > > > > > > 2) The snapshot of the variables are part of the
>     > versioned flow,
>     > > > > > > > so
>     > > > > > > > > > > > > after importing you would have to set the new values
>     > in the
>     > > > > > > > target
>     > > > > > > > > > > > > environment. This can be done in the UI or with
>     > scripts.
>     > > > > > > > > > > > >
>     > > > > > > > > > > > > 3) The REST API allows for positioning of any
>     > component that is
>     > > > > > > > > > > > > created. i can't speak for how to do it with
>     > NiPyApi, but I know
>     > > > > > > > the
>     > > > > > > > > > > > > CLI pg-import command attempts to automatically
>     > organize the
>     > > > > > > > imported
>     > > > > > > > > > > > > process groups on the root canvas so that they don't
>     > overlap.
>     > > > > > > > > > > > >
>     > > > > > > > > > > > > Thanks,
>     > > > > > > > > > > > >
>     > > > > > > > > > > > > Bryan
>     > > > > > > > > > > > >
>     > > > > > > > > > > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <
>     > > > > > > > > > mail2msiva@gmail.com>
>     > > > > > > > > > > > wrote:
>     > > > > > > > > > > > > >
>     > > > > > > > > > > > > > Hi,
>     > > > > > > > > > > > > >
>     > > > > > > > > > > > > > We are looking for nifi deploy automation from
>     > regsitry ->
>     > > > > > > > higher
>     > > > > > > > > > nifi
>     > > > > > > > > > > > instance. Quick observations during intial evaluation.
>     > > > > > > > > > > > > >
>     > > > > > > > > > > > > > 1. After process group import, controller services
>     > are not set
>     > > > > > > > in
>     > > > > > > > > > nifi
>     > > > > > > > > > > > instance
>     > > > > > > > > > > > > > 2. How to set variables from source process group
>     > to
>     > > > > > > > destination
>     > > > > > > > > > > > process group.
>     > > > > > > > > > > > > > 3. Is there any possibility of definining the
>     > positioning of
>     > > > > > > > > > process
>     > > > > > > > > > > > group to be imported in Nifi Canvas.
>     > > > > > > > > > > > > >
>     > > > > > > > > > > > > > Any other best practices  regarding NiFi
>     > deployment automation
>     > > > > > > > > > will be
>     > > > > > > > > > > > helpful.
>     > > > > > > > > > > > > >
>     > > > > > > > > > > > > > Thanks in Advance!
>     > > > > > > > > > > > > >
>     > > > > > > > > > > > >
>     > > > > > > > > > > >
>     > > > > > > > > > > --
>     > > > > > > > > > > Sent from Gmail Mobile
>     > > > > > > > > > >
>     > > > > > > > > >
>     > > > > > > > > --
>     > > > > > > > > Sent from Gmail Mobile
>     > > > > > > > >
>     > > > > > > >
>     > > > > > > --
>     > > > > > > Sent from Gmail Mobile
>     > > > > > >
>     > > > >
>     > >
>     >
>

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by Evan Reynolds <ev...@usermind.com.INVALID>.
There is a CLI command to change variables ... ? I've moved most of my processor variables to extra-args.property, so it's easy to change them. But there are a few I can't change - for example, DistributedMapCacheClientService controller service has a "Server Hostname" property that doesn't support the expression language, so we have to essentially search and replace the string in the flow.xml file before deploying it. Similarly, the DBCPConnectionPool has a password that is a sensitive values - so even if we set it to an argument, that argument gets wiped when we go through the registry to deploy, so we are currently deploying and then going and manually filling in that password.

I just looked again at the CLI but I didn't see a command that would set them, but I may have very easily missed it! Is there?

---------------------------------------------------------------
 
Evan Reynolds
evan@usermind.com
 

On 9/2/19, 7:59 PM, "Andrew Grande" <ap...@gmail.com> wrote:

    I think there is confusion here. Changing *properties* is considered a
    tracked change, the behavior is per design.
    
    There are *variables* on the PG level which are considered non-trackable
    environment settings and won't trigger a dirty flag.
    
    There is a dedicated command in the cli to list and set these *variables*.
    
    Hope it helps,
    Andrew
    
    On Mon, Sep 2, 2019, 4:07 PM sivasankar Reddy <ma...@gmail.com> wrote:
    
    > Hi Bryan,
    >
    > Thanks for the options. After pg-import or pg-change-version, if we use
    > nifi rest api and update properties, looks like nifi instance detects
    > changes(compares with bucket flow in registry) and creates * mark.
    >
    > We can change pg-change-version however just for properties setting , may
    > be its not advisable to change version in nifi instance.
    >
    > Any ideas around this will be helpful.
    >
    > Thanks
    > Siva
    >
    > On 2019/08/23 16:23:38, Bryan Bende <bb...@gmail.com> wrote:
    > > Well there are two different encryption parts here -
    > > encryption-at-rest and encryption-in-transit...
    > >
    > > Encryption-at-rest would have to be part of your script that is
    > > calling the REST API. Somehow you create a config file with encrypted
    > > values, and your script needs to read those in an decrypt them in
    > > memory and then put them into the content that will be sent in the
    > > HTTP PUT or POST request.
    > >
    > > Encryption-in-transit would be handled by the fact that you would be
    > > making an HTTPS request to NiFi, so the contents of the POST or PUT
    > > request are encrypted in transit. This is the same thing that is
    > > happening if you were in the NiFi UI and typed in a password into the
    > > sensitive property field.
    > >
    > >
    > > On Fri, Aug 23, 2019 at 11:35 AM sivasankar Reddy <ma...@gmail.com>
    > wrote:
    > > >
    > > > Hi Bryan,
    > > >
    > > > Thanks for the reply.
    > > > Able to get the json object and set the property for Password. However
    > value of password can be either referred from config file (its plain string
    > visible in config). Is there any way that password is encrypted and this
    > set property can take decrypted value.
    > > >
    > > > this way we ensure that password is sensitive. Any other ideas to
    > achieve this
    > > >
    > > >
    > > > Regards,
    > > > Siva
    > > >
    > > >
    > > >
    > > > On 2019/08/22 17:38:37, Bryan Bende <bb...@gmail.com> wrote:
    > > > > The parameters work is on-going and will be in the next release which
    > > > > is 1.10.0. Releases don't really have specific timelines, but
    > > > > generally they happen every few months. Most likely 1.10.0 is a
    > couple
    > > > > of weeks away, but depends on when active work is completed and when
    > > > > someone volunteers to make a release.
    > > > >
    > > > > There is no timeline for the "set-property" command since it was just
    > > > > suggested as a new feature in this email thread :) It requires
    > someone
    > > > > creating a JIRA and deciding to work on it.
    > > > >
    > > > > All of the functionality in the CLI and NiPyAPI, and even NiFi's own
    > > > > UI, is based on the REST API. So you can still perform a "set
    > > > > property" by using the REST API to modify the configuration of a
    > > > > processor, the CLI would just make it easier so that you wouldn't
    > have
    > > > > to understand the lower level API details. The best way to understand
    > > > > the API calls is to us the UI while you have Chrome Dev Tools open,
    > > > > and then perform the action you are interested in and look at the
    > > > > requests made on the Network tab. You'll be able to see what URLs are
    > > > > called and what the request and response bodies look like.
    > > > >
    > > > > On Thu, Aug 22, 2019 at 11:46 AM sivasankar Reddy <
    > mail2msiva@gmail.com> wrote:
    > > > > >
    > > > > > Hi Bryan,
    > > > > >
    > > > > > Thanks for the reply. looks like even set-property will be a new
    > feature, as current CLI doesn't have that.
    > > > > >
    > > > > > Could you please share timelines of these features if its in
    > roadmap.
    > > > > > 1. "set-property"
    > > > > > 2. “parameters”
    > > > > >
    > > > > > The only option currently is set to set sensitive parameters
    > manually? or any other option through CLI
    > > > > >
    > > > > > Regards,
    > > > > > Siva
    > > > > >
    > > > > > On 2019/08/22 04:10:46, Bryan Bende <bb...@gmail.com> wrote:
    > > > > > > Currently there isn’t a good way with the CLI, we would need to
    > add a
    > > > > > > command like set-property that took the id of a component, and
    > the name and
    > > > > > > value for a property.
    > > > > > >
    > > > > > > The next release will have a new feature called “parameters”
    > which will
    > > > > > > solve this problem. You’ll be able to use a parameter in a
    > sensitive
    > > > > > > property and then use CLI to set parameter values.
    > > > > > >
    > > > > > > On Wed, Aug 21, 2019 at 12:29 PM sivasankar Reddy <
    > mail2msiva@gmail.com>
    > > > > > > wrote:
    > > > > > >
    > > > > > > > Hi Bryan,
    > > > > > > >
    > > > > > > > Thanks for the update. Could you please mention about
    > > > > > > > sensitive parameters aspect from deployment point of view
    > using CLI.
    > > > > > > >
    > > > > > > > How to set the sensitive parameters values after pg-import.
    > > > > > > >
    > > > > > > > Regards,
    > > > > > > > Siva
    > > > > > > >
    > > > > > > > On 2019/08/21 11:47:49, Bryan Bende <bb...@gmail.com> wrote:
    > > > > > > > > It does not, I was saying we could easily improve the
    > command to add
    > > > > > > > those
    > > > > > > > > parameters.
    > > > > > > > >
    > > > > > > > > On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy <
    > mail2msiva@gmail.com>
    > > > > > > > > wrote:
    > > > > > > > >
    > > > > > > > > > Hi  Bryan,
    > > > > > > > > >
    > > > > > > > > > Thanks for the update. Does the existing pg-import takes
    > optional
    > > > > > > > params
    > > > > > > > > > for x/y coordinates. If yes can you share the sample
    > command as
    > > > > > > > pg-import
    > > > > > > > > > --help doesn't has this info.
    > > > > > > > > >
    > > > > > > > > > Regards,
    > > > > > > > > > siva
    > > > > > > > > >
    > > > > > > > > >
    > > > > > > > > > On 2019/08/20 22:19:23, Bryan Bende <bb...@gmail.com>
    > wrote:
    > > > > > > > > > > Hello,
    > > > > > > > > > >
    > > > > > > > > > > 1) pg-import will keeping finding a location that does
    > not overlap
    > > > > > > > with
    > > > > > > > > > any
    > > > > > > > > > > existing process groups, so it is more of auto layout.
    > We could
    > > > > > > > easily
    > > > > > > > > > add
    > > > > > > > > > > optional params for x/y coordinates.
    > > > > > > > > > >
    > > > > > > > > > > 2) Both are fine choices.. CLI is more geared towards
    > simple
    > > > > > > > commands,
    > > > > > > > > > > where as with NiPyApi you could more easily write
    > complex logic in
    > > > > > > > > > Python.
    > > > > > > > > > > With CLI you would write a wrapper shell script that
    > made individual
    > > > > > > > > > calls
    > > > > > > > > > > to the CLI. I think it is really just a matter of
    > preference.
    > > > > > > > > > >
    > > > > > > > > > > -Bryan
    > > > > > > > > > >
    > > > > > > > > > >
    > > > > > > > > > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy <
    > > > > > > > mail2msiva@gmail.com>
    > > > > > > > > > > wrote:
    > > > > > > > > > >
    > > > > > > > > > > > Hi Bryan,
    > > > > > > > > > > >
    > > > > > > > > > > > Thanks for your input. I would like to know more
    > > > > > > > > > > >
    > > > > > > > > > > > 1. how does CLI pg-import  works if already canvas has
    > 100 to 150
    > > > > > > > > > flows.
    > > > > > > > > > > > Will CLI pg-import always place the flow in canvas
    > which can be
    > > > > > > > easily
    > > > > > > > > > > > located. As nipyapi has location(x,y) during deploy
    > flow.
    > > > > > > > > > > >
    > > > > > > > > > > > 2. From enterprise usage, do you recommend toolkit CLI
    > vs nipyapi,
    > > > > > > > it
    > > > > > > > > > > > seems toolkit CLI is part of official Nifi.
    > > > > > > > > > > >
    > > > > > > > > > > > Any other points regarding this will be helpful.
    > > > > > > > > > > >
    > > > > > > > > > > > Regards,
    > > > > > > > > > > > Siva
    > > > > > > > > > > >
    > > > > > > > > > > > On 2019/08/08 22:41:43, Bryan Bende <bb...@gmail.com>
    > wrote:
    > > > > > > > > > > > > Hello,
    > > > > > > > > > > > >
    > > > > > > > > > > > > A lot depends on whether you are doing the promotion
    > with the UI
    > > > > > > > or
    > > > > > > > > > > > > with scripts using something like NiFi CLI or
    > NiPyApi.
    > > > > > > > > > > > >
    > > > > > > > > > > > > 1) Currently references to controller services
    > outside the
    > > > > > > > versioned
    > > > > > > > > > > > > process group need to be set after initial import.
    > In the next
    > > > > > > > > > release
    > > > > > > > > > > > > it will auto-select by name as long as there is only
    > one service
    > > > > > > > with
    > > > > > > > > > > > > the given name and type in a parent group (name is
    > not unique).
    > > > > > > > > > > > >
    > > > > > > > > > > > > 2) The snapshot of the variables are part of the
    > versioned flow,
    > > > > > > > so
    > > > > > > > > > > > > after importing you would have to set the new values
    > in the
    > > > > > > > target
    > > > > > > > > > > > > environment. This can be done in the UI or with
    > scripts.
    > > > > > > > > > > > >
    > > > > > > > > > > > > 3) The REST API allows for positioning of any
    > component that is
    > > > > > > > > > > > > created. i can't speak for how to do it with
    > NiPyApi, but I know
    > > > > > > > the
    > > > > > > > > > > > > CLI pg-import command attempts to automatically
    > organize the
    > > > > > > > imported
    > > > > > > > > > > > > process groups on the root canvas so that they don't
    > overlap.
    > > > > > > > > > > > >
    > > > > > > > > > > > > Thanks,
    > > > > > > > > > > > >
    > > > > > > > > > > > > Bryan
    > > > > > > > > > > > >
    > > > > > > > > > > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <
    > > > > > > > > > mail2msiva@gmail.com>
    > > > > > > > > > > > wrote:
    > > > > > > > > > > > > >
    > > > > > > > > > > > > > Hi,
    > > > > > > > > > > > > >
    > > > > > > > > > > > > > We are looking for nifi deploy automation from
    > regsitry ->
    > > > > > > > higher
    > > > > > > > > > nifi
    > > > > > > > > > > > instance. Quick observations during intial evaluation.
    > > > > > > > > > > > > >
    > > > > > > > > > > > > > 1. After process group import, controller services
    > are not set
    > > > > > > > in
    > > > > > > > > > nifi
    > > > > > > > > > > > instance
    > > > > > > > > > > > > > 2. How to set variables from source process group
    > to
    > > > > > > > destination
    > > > > > > > > > > > process group.
    > > > > > > > > > > > > > 3. Is there any possibility of definining the
    > positioning of
    > > > > > > > > > process
    > > > > > > > > > > > group to be imported in Nifi Canvas.
    > > > > > > > > > > > > >
    > > > > > > > > > > > > > Any other best practices  regarding NiFi
    > deployment automation
    > > > > > > > > > will be
    > > > > > > > > > > > helpful.
    > > > > > > > > > > > > >
    > > > > > > > > > > > > > Thanks in Advance!
    > > > > > > > > > > > > >
    > > > > > > > > > > > >
    > > > > > > > > > > >
    > > > > > > > > > > --
    > > > > > > > > > > Sent from Gmail Mobile
    > > > > > > > > > >
    > > > > > > > > >
    > > > > > > > > --
    > > > > > > > > Sent from Gmail Mobile
    > > > > > > > >
    > > > > > > >
    > > > > > > --
    > > > > > > Sent from Gmail Mobile
    > > > > > >
    > > > >
    > >
    >
    

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by Andrew Grande <ap...@gmail.com>.
I think there is confusion here. Changing *properties* is considered a
tracked change, the behavior is per design.

There are *variables* on the PG level which are considered non-trackable
environment settings and won't trigger a dirty flag.

There is a dedicated command in the cli to list and set these *variables*.

Hope it helps,
Andrew

On Mon, Sep 2, 2019, 4:07 PM sivasankar Reddy <ma...@gmail.com> wrote:

> Hi Bryan,
>
> Thanks for the options. After pg-import or pg-change-version, if we use
> nifi rest api and update properties, looks like nifi instance detects
> changes(compares with bucket flow in registry) and creates * mark.
>
> We can change pg-change-version however just for properties setting , may
> be its not advisable to change version in nifi instance.
>
> Any ideas around this will be helpful.
>
> Thanks
> Siva
>
> On 2019/08/23 16:23:38, Bryan Bende <bb...@gmail.com> wrote:
> > Well there are two different encryption parts here -
> > encryption-at-rest and encryption-in-transit...
> >
> > Encryption-at-rest would have to be part of your script that is
> > calling the REST API. Somehow you create a config file with encrypted
> > values, and your script needs to read those in an decrypt them in
> > memory and then put them into the content that will be sent in the
> > HTTP PUT or POST request.
> >
> > Encryption-in-transit would be handled by the fact that you would be
> > making an HTTPS request to NiFi, so the contents of the POST or PUT
> > request are encrypted in transit. This is the same thing that is
> > happening if you were in the NiFi UI and typed in a password into the
> > sensitive property field.
> >
> >
> > On Fri, Aug 23, 2019 at 11:35 AM sivasankar Reddy <ma...@gmail.com>
> wrote:
> > >
> > > Hi Bryan,
> > >
> > > Thanks for the reply.
> > > Able to get the json object and set the property for Password. However
> value of password can be either referred from config file (its plain string
> visible in config). Is there any way that password is encrypted and this
> set property can take decrypted value.
> > >
> > > this way we ensure that password is sensitive. Any other ideas to
> achieve this
> > >
> > >
> > > Regards,
> > > Siva
> > >
> > >
> > >
> > > On 2019/08/22 17:38:37, Bryan Bende <bb...@gmail.com> wrote:
> > > > The parameters work is on-going and will be in the next release which
> > > > is 1.10.0. Releases don't really have specific timelines, but
> > > > generally they happen every few months. Most likely 1.10.0 is a
> couple
> > > > of weeks away, but depends on when active work is completed and when
> > > > someone volunteers to make a release.
> > > >
> > > > There is no timeline for the "set-property" command since it was just
> > > > suggested as a new feature in this email thread :) It requires
> someone
> > > > creating a JIRA and deciding to work on it.
> > > >
> > > > All of the functionality in the CLI and NiPyAPI, and even NiFi's own
> > > > UI, is based on the REST API. So you can still perform a "set
> > > > property" by using the REST API to modify the configuration of a
> > > > processor, the CLI would just make it easier so that you wouldn't
> have
> > > > to understand the lower level API details. The best way to understand
> > > > the API calls is to us the UI while you have Chrome Dev Tools open,
> > > > and then perform the action you are interested in and look at the
> > > > requests made on the Network tab. You'll be able to see what URLs are
> > > > called and what the request and response bodies look like.
> > > >
> > > > On Thu, Aug 22, 2019 at 11:46 AM sivasankar Reddy <
> mail2msiva@gmail.com> wrote:
> > > > >
> > > > > Hi Bryan,
> > > > >
> > > > > Thanks for the reply. looks like even set-property will be a new
> feature, as current CLI doesn't have that.
> > > > >
> > > > > Could you please share timelines of these features if its in
> roadmap.
> > > > > 1. "set-property"
> > > > > 2. “parameters”
> > > > >
> > > > > The only option currently is set to set sensitive parameters
> manually? or any other option through CLI
> > > > >
> > > > > Regards,
> > > > > Siva
> > > > >
> > > > > On 2019/08/22 04:10:46, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > Currently there isn’t a good way with the CLI, we would need to
> add a
> > > > > > command like set-property that took the id of a component, and
> the name and
> > > > > > value for a property.
> > > > > >
> > > > > > The next release will have a new feature called “parameters”
> which will
> > > > > > solve this problem. You’ll be able to use a parameter in a
> sensitive
> > > > > > property and then use CLI to set parameter values.
> > > > > >
> > > > > > On Wed, Aug 21, 2019 at 12:29 PM sivasankar Reddy <
> mail2msiva@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Bryan,
> > > > > > >
> > > > > > > Thanks for the update. Could you please mention about
> > > > > > > sensitive parameters aspect from deployment point of view
> using CLI.
> > > > > > >
> > > > > > > How to set the sensitive parameters values after pg-import.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Siva
> > > > > > >
> > > > > > > On 2019/08/21 11:47:49, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > > > It does not, I was saying we could easily improve the
> command to add
> > > > > > > those
> > > > > > > > parameters.
> > > > > > > >
> > > > > > > > On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy <
> mail2msiva@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi  Bryan,
> > > > > > > > >
> > > > > > > > > Thanks for the update. Does the existing pg-import takes
> optional
> > > > > > > params
> > > > > > > > > for x/y coordinates. If yes can you share the sample
> command as
> > > > > > > pg-import
> > > > > > > > > --help doesn't has this info.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > siva
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 2019/08/20 22:19:23, Bryan Bende <bb...@gmail.com>
> wrote:
> > > > > > > > > > Hello,
> > > > > > > > > >
> > > > > > > > > > 1) pg-import will keeping finding a location that does
> not overlap
> > > > > > > with
> > > > > > > > > any
> > > > > > > > > > existing process groups, so it is more of auto layout.
> We could
> > > > > > > easily
> > > > > > > > > add
> > > > > > > > > > optional params for x/y coordinates.
> > > > > > > > > >
> > > > > > > > > > 2) Both are fine choices.. CLI is more geared towards
> simple
> > > > > > > commands,
> > > > > > > > > > where as with NiPyApi you could more easily write
> complex logic in
> > > > > > > > > Python.
> > > > > > > > > > With CLI you would write a wrapper shell script that
> made individual
> > > > > > > > > calls
> > > > > > > > > > to the CLI. I think it is really just a matter of
> preference.
> > > > > > > > > >
> > > > > > > > > > -Bryan
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy <
> > > > > > > mail2msiva@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Bryan,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your input. I would like to know more
> > > > > > > > > > >
> > > > > > > > > > > 1. how does CLI pg-import  works if already canvas has
> 100 to 150
> > > > > > > > > flows.
> > > > > > > > > > > Will CLI pg-import always place the flow in canvas
> which can be
> > > > > > > easily
> > > > > > > > > > > located. As nipyapi has location(x,y) during deploy
> flow.
> > > > > > > > > > >
> > > > > > > > > > > 2. From enterprise usage, do you recommend toolkit CLI
> vs nipyapi,
> > > > > > > it
> > > > > > > > > > > seems toolkit CLI is part of official Nifi.
> > > > > > > > > > >
> > > > > > > > > > > Any other points regarding this will be helpful.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Siva
> > > > > > > > > > >
> > > > > > > > > > > On 2019/08/08 22:41:43, Bryan Bende <bb...@gmail.com>
> wrote:
> > > > > > > > > > > > Hello,
> > > > > > > > > > > >
> > > > > > > > > > > > A lot depends on whether you are doing the promotion
> with the UI
> > > > > > > or
> > > > > > > > > > > > with scripts using something like NiFi CLI or
> NiPyApi.
> > > > > > > > > > > >
> > > > > > > > > > > > 1) Currently references to controller services
> outside the
> > > > > > > versioned
> > > > > > > > > > > > process group need to be set after initial import.
> In the next
> > > > > > > > > release
> > > > > > > > > > > > it will auto-select by name as long as there is only
> one service
> > > > > > > with
> > > > > > > > > > > > the given name and type in a parent group (name is
> not unique).
> > > > > > > > > > > >
> > > > > > > > > > > > 2) The snapshot of the variables are part of the
> versioned flow,
> > > > > > > so
> > > > > > > > > > > > after importing you would have to set the new values
> in the
> > > > > > > target
> > > > > > > > > > > > environment. This can be done in the UI or with
> scripts.
> > > > > > > > > > > >
> > > > > > > > > > > > 3) The REST API allows for positioning of any
> component that is
> > > > > > > > > > > > created. i can't speak for how to do it with
> NiPyApi, but I know
> > > > > > > the
> > > > > > > > > > > > CLI pg-import command attempts to automatically
> organize the
> > > > > > > imported
> > > > > > > > > > > > process groups on the root canvas so that they don't
> overlap.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Bryan
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <
> > > > > > > > > mail2msiva@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > We are looking for nifi deploy automation from
> regsitry ->
> > > > > > > higher
> > > > > > > > > nifi
> > > > > > > > > > > instance. Quick observations during intial evaluation.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 1. After process group import, controller services
> are not set
> > > > > > > in
> > > > > > > > > nifi
> > > > > > > > > > > instance
> > > > > > > > > > > > > 2. How to set variables from source process group
> to
> > > > > > > destination
> > > > > > > > > > > process group.
> > > > > > > > > > > > > 3. Is there any possibility of definining the
> positioning of
> > > > > > > > > process
> > > > > > > > > > > group to be imported in Nifi Canvas.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Any other best practices  regarding NiFi
> deployment automation
> > > > > > > > > will be
> > > > > > > > > > > helpful.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks in Advance!
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Sent from Gmail Mobile
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > --
> > > > > > > > Sent from Gmail Mobile
> > > > > > > >
> > > > > > >
> > > > > > --
> > > > > > Sent from Gmail Mobile
> > > > > >
> > > >
> >
>

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by sivasankar Reddy <ma...@gmail.com>.
Hi Bryan, 

Thanks for the options. After pg-import or pg-change-version, if we use  nifi rest api and update properties, looks like nifi instance detects changes(compares with bucket flow in registry) and creates * mark. 

We can change pg-change-version however just for properties setting , may be its not advisable to change version in nifi instance. 

Any ideas around this will be helpful. 

Thanks
Siva

On 2019/08/23 16:23:38, Bryan Bende <bb...@gmail.com> wrote: 
> Well there are two different encryption parts here -
> encryption-at-rest and encryption-in-transit...
> 
> Encryption-at-rest would have to be part of your script that is
> calling the REST API. Somehow you create a config file with encrypted
> values, and your script needs to read those in an decrypt them in
> memory and then put them into the content that will be sent in the
> HTTP PUT or POST request.
> 
> Encryption-in-transit would be handled by the fact that you would be
> making an HTTPS request to NiFi, so the contents of the POST or PUT
> request are encrypted in transit. This is the same thing that is
> happening if you were in the NiFi UI and typed in a password into the
> sensitive property field.
> 
> 
> On Fri, Aug 23, 2019 at 11:35 AM sivasankar Reddy <ma...@gmail.com> wrote:
> >
> > Hi Bryan,
> >
> > Thanks for the reply.
> > Able to get the json object and set the property for Password. However value of password can be either referred from config file (its plain string visible in config). Is there any way that password is encrypted and this set property can take decrypted value.
> >
> > this way we ensure that password is sensitive. Any other ideas to achieve this
> >
> >
> > Regards,
> > Siva
> >
> >
> >
> > On 2019/08/22 17:38:37, Bryan Bende <bb...@gmail.com> wrote:
> > > The parameters work is on-going and will be in the next release which
> > > is 1.10.0. Releases don't really have specific timelines, but
> > > generally they happen every few months. Most likely 1.10.0 is a couple
> > > of weeks away, but depends on when active work is completed and when
> > > someone volunteers to make a release.
> > >
> > > There is no timeline for the "set-property" command since it was just
> > > suggested as a new feature in this email thread :) It requires someone
> > > creating a JIRA and deciding to work on it.
> > >
> > > All of the functionality in the CLI and NiPyAPI, and even NiFi's own
> > > UI, is based on the REST API. So you can still perform a "set
> > > property" by using the REST API to modify the configuration of a
> > > processor, the CLI would just make it easier so that you wouldn't have
> > > to understand the lower level API details. The best way to understand
> > > the API calls is to us the UI while you have Chrome Dev Tools open,
> > > and then perform the action you are interested in and look at the
> > > requests made on the Network tab. You'll be able to see what URLs are
> > > called and what the request and response bodies look like.
> > >
> > > On Thu, Aug 22, 2019 at 11:46 AM sivasankar Reddy <ma...@gmail.com> wrote:
> > > >
> > > > Hi Bryan,
> > > >
> > > > Thanks for the reply. looks like even set-property will be a new feature, as current CLI doesn't have that.
> > > >
> > > > Could you please share timelines of these features if its in roadmap.
> > > > 1. "set-property"
> > > > 2. “parameters”
> > > >
> > > > The only option currently is set to set sensitive parameters manually? or any other option through CLI
> > > >
> > > > Regards,
> > > > Siva
> > > >
> > > > On 2019/08/22 04:10:46, Bryan Bende <bb...@gmail.com> wrote:
> > > > > Currently there isn’t a good way with the CLI, we would need to add a
> > > > > command like set-property that took the id of a component, and the name and
> > > > > value for a property.
> > > > >
> > > > > The next release will have a new feature called “parameters” which will
> > > > > solve this problem. You’ll be able to use a parameter in a sensitive
> > > > > property and then use CLI to set parameter values.
> > > > >
> > > > > On Wed, Aug 21, 2019 at 12:29 PM sivasankar Reddy <ma...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Bryan,
> > > > > >
> > > > > > Thanks for the update. Could you please mention about
> > > > > > sensitive parameters aspect from deployment point of view using CLI.
> > > > > >
> > > > > > How to set the sensitive parameters values after pg-import.
> > > > > >
> > > > > > Regards,
> > > > > > Siva
> > > > > >
> > > > > > On 2019/08/21 11:47:49, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > > It does not, I was saying we could easily improve the command to add
> > > > > > those
> > > > > > > parameters.
> > > > > > >
> > > > > > > On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy <ma...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi  Bryan,
> > > > > > > >
> > > > > > > > Thanks for the update. Does the existing pg-import takes optional
> > > > > > params
> > > > > > > > for x/y coordinates. If yes can you share the sample command as
> > > > > > pg-import
> > > > > > > > --help doesn't has this info.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > siva
> > > > > > > >
> > > > > > > >
> > > > > > > > On 2019/08/20 22:19:23, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > > > > Hello,
> > > > > > > > >
> > > > > > > > > 1) pg-import will keeping finding a location that does not overlap
> > > > > > with
> > > > > > > > any
> > > > > > > > > existing process groups, so it is more of auto layout. We could
> > > > > > easily
> > > > > > > > add
> > > > > > > > > optional params for x/y coordinates.
> > > > > > > > >
> > > > > > > > > 2) Both are fine choices.. CLI is more geared towards simple
> > > > > > commands,
> > > > > > > > > where as with NiPyApi you could more easily write complex logic in
> > > > > > > > Python.
> > > > > > > > > With CLI you would write a wrapper shell script that made individual
> > > > > > > > calls
> > > > > > > > > to the CLI. I think it is really just a matter of preference.
> > > > > > > > >
> > > > > > > > > -Bryan
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy <
> > > > > > mail2msiva@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Bryan,
> > > > > > > > > >
> > > > > > > > > > Thanks for your input. I would like to know more
> > > > > > > > > >
> > > > > > > > > > 1. how does CLI pg-import  works if already canvas has 100 to 150
> > > > > > > > flows.
> > > > > > > > > > Will CLI pg-import always place the flow in canvas which can be
> > > > > > easily
> > > > > > > > > > located. As nipyapi has location(x,y) during deploy flow.
> > > > > > > > > >
> > > > > > > > > > 2. From enterprise usage, do you recommend toolkit CLI vs nipyapi,
> > > > > > it
> > > > > > > > > > seems toolkit CLI is part of official Nifi.
> > > > > > > > > >
> > > > > > > > > > Any other points regarding this will be helpful.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Siva
> > > > > > > > > >
> > > > > > > > > > On 2019/08/08 22:41:43, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > > > > > > Hello,
> > > > > > > > > > >
> > > > > > > > > > > A lot depends on whether you are doing the promotion with the UI
> > > > > > or
> > > > > > > > > > > with scripts using something like NiFi CLI or NiPyApi.
> > > > > > > > > > >
> > > > > > > > > > > 1) Currently references to controller services outside the
> > > > > > versioned
> > > > > > > > > > > process group need to be set after initial import. In the next
> > > > > > > > release
> > > > > > > > > > > it will auto-select by name as long as there is only one service
> > > > > > with
> > > > > > > > > > > the given name and type in a parent group (name is not unique).
> > > > > > > > > > >
> > > > > > > > > > > 2) The snapshot of the variables are part of the versioned flow,
> > > > > > so
> > > > > > > > > > > after importing you would have to set the new values in the
> > > > > > target
> > > > > > > > > > > environment. This can be done in the UI or with scripts.
> > > > > > > > > > >
> > > > > > > > > > > 3) The REST API allows for positioning of any component that is
> > > > > > > > > > > created. i can't speak for how to do it with NiPyApi, but I know
> > > > > > the
> > > > > > > > > > > CLI pg-import command attempts to automatically organize the
> > > > > > imported
> > > > > > > > > > > process groups on the root canvas so that they don't overlap.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Bryan
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <
> > > > > > > > mail2msiva@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > We are looking for nifi deploy automation from regsitry ->
> > > > > > higher
> > > > > > > > nifi
> > > > > > > > > > instance. Quick observations during intial evaluation.
> > > > > > > > > > > >
> > > > > > > > > > > > 1. After process group import, controller services are not set
> > > > > > in
> > > > > > > > nifi
> > > > > > > > > > instance
> > > > > > > > > > > > 2. How to set variables from source process group to
> > > > > > destination
> > > > > > > > > > process group.
> > > > > > > > > > > > 3. Is there any possibility of definining the positioning of
> > > > > > > > process
> > > > > > > > > > group to be imported in Nifi Canvas.
> > > > > > > > > > > >
> > > > > > > > > > > > Any other best practices  regarding NiFi deployment automation
> > > > > > > > will be
> > > > > > > > > > helpful.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks in Advance!
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Sent from Gmail Mobile
> > > > > > > > >
> > > > > > > >
> > > > > > > --
> > > > > > > Sent from Gmail Mobile
> > > > > > >
> > > > > >
> > > > > --
> > > > > Sent from Gmail Mobile
> > > > >
> > >
> 

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by Bryan Bende <bb...@gmail.com>.
Well there are two different encryption parts here -
encryption-at-rest and encryption-in-transit...

Encryption-at-rest would have to be part of your script that is
calling the REST API. Somehow you create a config file with encrypted
values, and your script needs to read those in an decrypt them in
memory and then put them into the content that will be sent in the
HTTP PUT or POST request.

Encryption-in-transit would be handled by the fact that you would be
making an HTTPS request to NiFi, so the contents of the POST or PUT
request are encrypted in transit. This is the same thing that is
happening if you were in the NiFi UI and typed in a password into the
sensitive property field.


On Fri, Aug 23, 2019 at 11:35 AM sivasankar Reddy <ma...@gmail.com> wrote:
>
> Hi Bryan,
>
> Thanks for the reply.
> Able to get the json object and set the property for Password. However value of password can be either referred from config file (its plain string visible in config). Is there any way that password is encrypted and this set property can take decrypted value.
>
> this way we ensure that password is sensitive. Any other ideas to achieve this
>
>
> Regards,
> Siva
>
>
>
> On 2019/08/22 17:38:37, Bryan Bende <bb...@gmail.com> wrote:
> > The parameters work is on-going and will be in the next release which
> > is 1.10.0. Releases don't really have specific timelines, but
> > generally they happen every few months. Most likely 1.10.0 is a couple
> > of weeks away, but depends on when active work is completed and when
> > someone volunteers to make a release.
> >
> > There is no timeline for the "set-property" command since it was just
> > suggested as a new feature in this email thread :) It requires someone
> > creating a JIRA and deciding to work on it.
> >
> > All of the functionality in the CLI and NiPyAPI, and even NiFi's own
> > UI, is based on the REST API. So you can still perform a "set
> > property" by using the REST API to modify the configuration of a
> > processor, the CLI would just make it easier so that you wouldn't have
> > to understand the lower level API details. The best way to understand
> > the API calls is to us the UI while you have Chrome Dev Tools open,
> > and then perform the action you are interested in and look at the
> > requests made on the Network tab. You'll be able to see what URLs are
> > called and what the request and response bodies look like.
> >
> > On Thu, Aug 22, 2019 at 11:46 AM sivasankar Reddy <ma...@gmail.com> wrote:
> > >
> > > Hi Bryan,
> > >
> > > Thanks for the reply. looks like even set-property will be a new feature, as current CLI doesn't have that.
> > >
> > > Could you please share timelines of these features if its in roadmap.
> > > 1. "set-property"
> > > 2. “parameters”
> > >
> > > The only option currently is set to set sensitive parameters manually? or any other option through CLI
> > >
> > > Regards,
> > > Siva
> > >
> > > On 2019/08/22 04:10:46, Bryan Bende <bb...@gmail.com> wrote:
> > > > Currently there isn’t a good way with the CLI, we would need to add a
> > > > command like set-property that took the id of a component, and the name and
> > > > value for a property.
> > > >
> > > > The next release will have a new feature called “parameters” which will
> > > > solve this problem. You’ll be able to use a parameter in a sensitive
> > > > property and then use CLI to set parameter values.
> > > >
> > > > On Wed, Aug 21, 2019 at 12:29 PM sivasankar Reddy <ma...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Bryan,
> > > > >
> > > > > Thanks for the update. Could you please mention about
> > > > > sensitive parameters aspect from deployment point of view using CLI.
> > > > >
> > > > > How to set the sensitive parameters values after pg-import.
> > > > >
> > > > > Regards,
> > > > > Siva
> > > > >
> > > > > On 2019/08/21 11:47:49, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > It does not, I was saying we could easily improve the command to add
> > > > > those
> > > > > > parameters.
> > > > > >
> > > > > > On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy <ma...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi  Bryan,
> > > > > > >
> > > > > > > Thanks for the update. Does the existing pg-import takes optional
> > > > > params
> > > > > > > for x/y coordinates. If yes can you share the sample command as
> > > > > pg-import
> > > > > > > --help doesn't has this info.
> > > > > > >
> > > > > > > Regards,
> > > > > > > siva
> > > > > > >
> > > > > > >
> > > > > > > On 2019/08/20 22:19:23, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > 1) pg-import will keeping finding a location that does not overlap
> > > > > with
> > > > > > > any
> > > > > > > > existing process groups, so it is more of auto layout. We could
> > > > > easily
> > > > > > > add
> > > > > > > > optional params for x/y coordinates.
> > > > > > > >
> > > > > > > > 2) Both are fine choices.. CLI is more geared towards simple
> > > > > commands,
> > > > > > > > where as with NiPyApi you could more easily write complex logic in
> > > > > > > Python.
> > > > > > > > With CLI you would write a wrapper shell script that made individual
> > > > > > > calls
> > > > > > > > to the CLI. I think it is really just a matter of preference.
> > > > > > > >
> > > > > > > > -Bryan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy <
> > > > > mail2msiva@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Bryan,
> > > > > > > > >
> > > > > > > > > Thanks for your input. I would like to know more
> > > > > > > > >
> > > > > > > > > 1. how does CLI pg-import  works if already canvas has 100 to 150
> > > > > > > flows.
> > > > > > > > > Will CLI pg-import always place the flow in canvas which can be
> > > > > easily
> > > > > > > > > located. As nipyapi has location(x,y) during deploy flow.
> > > > > > > > >
> > > > > > > > > 2. From enterprise usage, do you recommend toolkit CLI vs nipyapi,
> > > > > it
> > > > > > > > > seems toolkit CLI is part of official Nifi.
> > > > > > > > >
> > > > > > > > > Any other points regarding this will be helpful.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Siva
> > > > > > > > >
> > > > > > > > > On 2019/08/08 22:41:43, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > > > > > Hello,
> > > > > > > > > >
> > > > > > > > > > A lot depends on whether you are doing the promotion with the UI
> > > > > or
> > > > > > > > > > with scripts using something like NiFi CLI or NiPyApi.
> > > > > > > > > >
> > > > > > > > > > 1) Currently references to controller services outside the
> > > > > versioned
> > > > > > > > > > process group need to be set after initial import. In the next
> > > > > > > release
> > > > > > > > > > it will auto-select by name as long as there is only one service
> > > > > with
> > > > > > > > > > the given name and type in a parent group (name is not unique).
> > > > > > > > > >
> > > > > > > > > > 2) The snapshot of the variables are part of the versioned flow,
> > > > > so
> > > > > > > > > > after importing you would have to set the new values in the
> > > > > target
> > > > > > > > > > environment. This can be done in the UI or with scripts.
> > > > > > > > > >
> > > > > > > > > > 3) The REST API allows for positioning of any component that is
> > > > > > > > > > created. i can't speak for how to do it with NiPyApi, but I know
> > > > > the
> > > > > > > > > > CLI pg-import command attempts to automatically organize the
> > > > > imported
> > > > > > > > > > process groups on the root canvas so that they don't overlap.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Bryan
> > > > > > > > > >
> > > > > > > > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <
> > > > > > > mail2msiva@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > We are looking for nifi deploy automation from regsitry ->
> > > > > higher
> > > > > > > nifi
> > > > > > > > > instance. Quick observations during intial evaluation.
> > > > > > > > > > >
> > > > > > > > > > > 1. After process group import, controller services are not set
> > > > > in
> > > > > > > nifi
> > > > > > > > > instance
> > > > > > > > > > > 2. How to set variables from source process group to
> > > > > destination
> > > > > > > > > process group.
> > > > > > > > > > > 3. Is there any possibility of definining the positioning of
> > > > > > > process
> > > > > > > > > group to be imported in Nifi Canvas.
> > > > > > > > > > >
> > > > > > > > > > > Any other best practices  regarding NiFi deployment automation
> > > > > > > will be
> > > > > > > > > helpful.
> > > > > > > > > > >
> > > > > > > > > > > Thanks in Advance!
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > --
> > > > > > > > Sent from Gmail Mobile
> > > > > > > >
> > > > > > >
> > > > > > --
> > > > > > Sent from Gmail Mobile
> > > > > >
> > > > >
> > > > --
> > > > Sent from Gmail Mobile
> > > >
> >

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by sivasankar Reddy <ma...@gmail.com>.
Hi Bryan, 

Thanks for the reply. 
Able to get the json object and set the property for Password. However value of password can be either referred from config file (its plain string visible in config). Is there any way that password is encrypted and this set property can take decrypted value. 

this way we ensure that password is sensitive. Any other ideas to achieve this 


Regards,
Siva



On 2019/08/22 17:38:37, Bryan Bende <bb...@gmail.com> wrote: 
> The parameters work is on-going and will be in the next release which
> is 1.10.0. Releases don't really have specific timelines, but
> generally they happen every few months. Most likely 1.10.0 is a couple
> of weeks away, but depends on when active work is completed and when
> someone volunteers to make a release.
> 
> There is no timeline for the "set-property" command since it was just
> suggested as a new feature in this email thread :) It requires someone
> creating a JIRA and deciding to work on it.
> 
> All of the functionality in the CLI and NiPyAPI, and even NiFi's own
> UI, is based on the REST API. So you can still perform a "set
> property" by using the REST API to modify the configuration of a
> processor, the CLI would just make it easier so that you wouldn't have
> to understand the lower level API details. The best way to understand
> the API calls is to us the UI while you have Chrome Dev Tools open,
> and then perform the action you are interested in and look at the
> requests made on the Network tab. You'll be able to see what URLs are
> called and what the request and response bodies look like.
> 
> On Thu, Aug 22, 2019 at 11:46 AM sivasankar Reddy <ma...@gmail.com> wrote:
> >
> > Hi Bryan,
> >
> > Thanks for the reply. looks like even set-property will be a new feature, as current CLI doesn't have that.
> >
> > Could you please share timelines of these features if its in roadmap.
> > 1. "set-property"
> > 2. “parameters”
> >
> > The only option currently is set to set sensitive parameters manually? or any other option through CLI
> >
> > Regards,
> > Siva
> >
> > On 2019/08/22 04:10:46, Bryan Bende <bb...@gmail.com> wrote:
> > > Currently there isn’t a good way with the CLI, we would need to add a
> > > command like set-property that took the id of a component, and the name and
> > > value for a property.
> > >
> > > The next release will have a new feature called “parameters” which will
> > > solve this problem. You’ll be able to use a parameter in a sensitive
> > > property and then use CLI to set parameter values.
> > >
> > > On Wed, Aug 21, 2019 at 12:29 PM sivasankar Reddy <ma...@gmail.com>
> > > wrote:
> > >
> > > > Hi Bryan,
> > > >
> > > > Thanks for the update. Could you please mention about
> > > > sensitive parameters aspect from deployment point of view using CLI.
> > > >
> > > > How to set the sensitive parameters values after pg-import.
> > > >
> > > > Regards,
> > > > Siva
> > > >
> > > > On 2019/08/21 11:47:49, Bryan Bende <bb...@gmail.com> wrote:
> > > > > It does not, I was saying we could easily improve the command to add
> > > > those
> > > > > parameters.
> > > > >
> > > > > On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy <ma...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi  Bryan,
> > > > > >
> > > > > > Thanks for the update. Does the existing pg-import takes optional
> > > > params
> > > > > > for x/y coordinates. If yes can you share the sample command as
> > > > pg-import
> > > > > > --help doesn't has this info.
> > > > > >
> > > > > > Regards,
> > > > > > siva
> > > > > >
> > > > > >
> > > > > > On 2019/08/20 22:19:23, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > > Hello,
> > > > > > >
> > > > > > > 1) pg-import will keeping finding a location that does not overlap
> > > > with
> > > > > > any
> > > > > > > existing process groups, so it is more of auto layout. We could
> > > > easily
> > > > > > add
> > > > > > > optional params for x/y coordinates.
> > > > > > >
> > > > > > > 2) Both are fine choices.. CLI is more geared towards simple
> > > > commands,
> > > > > > > where as with NiPyApi you could more easily write complex logic in
> > > > > > Python.
> > > > > > > With CLI you would write a wrapper shell script that made individual
> > > > > > calls
> > > > > > > to the CLI. I think it is really just a matter of preference.
> > > > > > >
> > > > > > > -Bryan
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy <
> > > > mail2msiva@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Bryan,
> > > > > > > >
> > > > > > > > Thanks for your input. I would like to know more
> > > > > > > >
> > > > > > > > 1. how does CLI pg-import  works if already canvas has 100 to 150
> > > > > > flows.
> > > > > > > > Will CLI pg-import always place the flow in canvas which can be
> > > > easily
> > > > > > > > located. As nipyapi has location(x,y) during deploy flow.
> > > > > > > >
> > > > > > > > 2. From enterprise usage, do you recommend toolkit CLI vs nipyapi,
> > > > it
> > > > > > > > seems toolkit CLI is part of official Nifi.
> > > > > > > >
> > > > > > > > Any other points regarding this will be helpful.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Siva
> > > > > > > >
> > > > > > > > On 2019/08/08 22:41:43, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > > > > Hello,
> > > > > > > > >
> > > > > > > > > A lot depends on whether you are doing the promotion with the UI
> > > > or
> > > > > > > > > with scripts using something like NiFi CLI or NiPyApi.
> > > > > > > > >
> > > > > > > > > 1) Currently references to controller services outside the
> > > > versioned
> > > > > > > > > process group need to be set after initial import. In the next
> > > > > > release
> > > > > > > > > it will auto-select by name as long as there is only one service
> > > > with
> > > > > > > > > the given name and type in a parent group (name is not unique).
> > > > > > > > >
> > > > > > > > > 2) The snapshot of the variables are part of the versioned flow,
> > > > so
> > > > > > > > > after importing you would have to set the new values in the
> > > > target
> > > > > > > > > environment. This can be done in the UI or with scripts.
> > > > > > > > >
> > > > > > > > > 3) The REST API allows for positioning of any component that is
> > > > > > > > > created. i can't speak for how to do it with NiPyApi, but I know
> > > > the
> > > > > > > > > CLI pg-import command attempts to automatically organize the
> > > > imported
> > > > > > > > > process groups on the root canvas so that they don't overlap.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Bryan
> > > > > > > > >
> > > > > > > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <
> > > > > > mail2msiva@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > We are looking for nifi deploy automation from regsitry ->
> > > > higher
> > > > > > nifi
> > > > > > > > instance. Quick observations during intial evaluation.
> > > > > > > > > >
> > > > > > > > > > 1. After process group import, controller services are not set
> > > > in
> > > > > > nifi
> > > > > > > > instance
> > > > > > > > > > 2. How to set variables from source process group to
> > > > destination
> > > > > > > > process group.
> > > > > > > > > > 3. Is there any possibility of definining the positioning of
> > > > > > process
> > > > > > > > group to be imported in Nifi Canvas.
> > > > > > > > > >
> > > > > > > > > > Any other best practices  regarding NiFi deployment automation
> > > > > > will be
> > > > > > > > helpful.
> > > > > > > > > >
> > > > > > > > > > Thanks in Advance!
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > --
> > > > > > > Sent from Gmail Mobile
> > > > > > >
> > > > > >
> > > > > --
> > > > > Sent from Gmail Mobile
> > > > >
> > > >
> > > --
> > > Sent from Gmail Mobile
> > >
> 

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by Bryan Bende <bb...@gmail.com>.
The parameters work is on-going and will be in the next release which
is 1.10.0. Releases don't really have specific timelines, but
generally they happen every few months. Most likely 1.10.0 is a couple
of weeks away, but depends on when active work is completed and when
someone volunteers to make a release.

There is no timeline for the "set-property" command since it was just
suggested as a new feature in this email thread :) It requires someone
creating a JIRA and deciding to work on it.

All of the functionality in the CLI and NiPyAPI, and even NiFi's own
UI, is based on the REST API. So you can still perform a "set
property" by using the REST API to modify the configuration of a
processor, the CLI would just make it easier so that you wouldn't have
to understand the lower level API details. The best way to understand
the API calls is to us the UI while you have Chrome Dev Tools open,
and then perform the action you are interested in and look at the
requests made on the Network tab. You'll be able to see what URLs are
called and what the request and response bodies look like.

On Thu, Aug 22, 2019 at 11:46 AM sivasankar Reddy <ma...@gmail.com> wrote:
>
> Hi Bryan,
>
> Thanks for the reply. looks like even set-property will be a new feature, as current CLI doesn't have that.
>
> Could you please share timelines of these features if its in roadmap.
> 1. "set-property"
> 2. “parameters”
>
> The only option currently is set to set sensitive parameters manually? or any other option through CLI
>
> Regards,
> Siva
>
> On 2019/08/22 04:10:46, Bryan Bende <bb...@gmail.com> wrote:
> > Currently there isn’t a good way with the CLI, we would need to add a
> > command like set-property that took the id of a component, and the name and
> > value for a property.
> >
> > The next release will have a new feature called “parameters” which will
> > solve this problem. You’ll be able to use a parameter in a sensitive
> > property and then use CLI to set parameter values.
> >
> > On Wed, Aug 21, 2019 at 12:29 PM sivasankar Reddy <ma...@gmail.com>
> > wrote:
> >
> > > Hi Bryan,
> > >
> > > Thanks for the update. Could you please mention about
> > > sensitive parameters aspect from deployment point of view using CLI.
> > >
> > > How to set the sensitive parameters values after pg-import.
> > >
> > > Regards,
> > > Siva
> > >
> > > On 2019/08/21 11:47:49, Bryan Bende <bb...@gmail.com> wrote:
> > > > It does not, I was saying we could easily improve the command to add
> > > those
> > > > parameters.
> > > >
> > > > On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy <ma...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi  Bryan,
> > > > >
> > > > > Thanks for the update. Does the existing pg-import takes optional
> > > params
> > > > > for x/y coordinates. If yes can you share the sample command as
> > > pg-import
> > > > > --help doesn't has this info.
> > > > >
> > > > > Regards,
> > > > > siva
> > > > >
> > > > >
> > > > > On 2019/08/20 22:19:23, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > Hello,
> > > > > >
> > > > > > 1) pg-import will keeping finding a location that does not overlap
> > > with
> > > > > any
> > > > > > existing process groups, so it is more of auto layout. We could
> > > easily
> > > > > add
> > > > > > optional params for x/y coordinates.
> > > > > >
> > > > > > 2) Both are fine choices.. CLI is more geared towards simple
> > > commands,
> > > > > > where as with NiPyApi you could more easily write complex logic in
> > > > > Python.
> > > > > > With CLI you would write a wrapper shell script that made individual
> > > > > calls
> > > > > > to the CLI. I think it is really just a matter of preference.
> > > > > >
> > > > > > -Bryan
> > > > > >
> > > > > >
> > > > > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy <
> > > mail2msiva@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Bryan,
> > > > > > >
> > > > > > > Thanks for your input. I would like to know more
> > > > > > >
> > > > > > > 1. how does CLI pg-import  works if already canvas has 100 to 150
> > > > > flows.
> > > > > > > Will CLI pg-import always place the flow in canvas which can be
> > > easily
> > > > > > > located. As nipyapi has location(x,y) during deploy flow.
> > > > > > >
> > > > > > > 2. From enterprise usage, do you recommend toolkit CLI vs nipyapi,
> > > it
> > > > > > > seems toolkit CLI is part of official Nifi.
> > > > > > >
> > > > > > > Any other points regarding this will be helpful.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Siva
> > > > > > >
> > > > > > > On 2019/08/08 22:41:43, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > A lot depends on whether you are doing the promotion with the UI
> > > or
> > > > > > > > with scripts using something like NiFi CLI or NiPyApi.
> > > > > > > >
> > > > > > > > 1) Currently references to controller services outside the
> > > versioned
> > > > > > > > process group need to be set after initial import. In the next
> > > > > release
> > > > > > > > it will auto-select by name as long as there is only one service
> > > with
> > > > > > > > the given name and type in a parent group (name is not unique).
> > > > > > > >
> > > > > > > > 2) The snapshot of the variables are part of the versioned flow,
> > > so
> > > > > > > > after importing you would have to set the new values in the
> > > target
> > > > > > > > environment. This can be done in the UI or with scripts.
> > > > > > > >
> > > > > > > > 3) The REST API allows for positioning of any component that is
> > > > > > > > created. i can't speak for how to do it with NiPyApi, but I know
> > > the
> > > > > > > > CLI pg-import command attempts to automatically organize the
> > > imported
> > > > > > > > process groups on the root canvas so that they don't overlap.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Bryan
> > > > > > > >
> > > > > > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <
> > > > > mail2msiva@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > We are looking for nifi deploy automation from regsitry ->
> > > higher
> > > > > nifi
> > > > > > > instance. Quick observations during intial evaluation.
> > > > > > > > >
> > > > > > > > > 1. After process group import, controller services are not set
> > > in
> > > > > nifi
> > > > > > > instance
> > > > > > > > > 2. How to set variables from source process group to
> > > destination
> > > > > > > process group.
> > > > > > > > > 3. Is there any possibility of definining the positioning of
> > > > > process
> > > > > > > group to be imported in Nifi Canvas.
> > > > > > > > >
> > > > > > > > > Any other best practices  regarding NiFi deployment automation
> > > > > will be
> > > > > > > helpful.
> > > > > > > > >
> > > > > > > > > Thanks in Advance!
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > --
> > > > > > Sent from Gmail Mobile
> > > > > >
> > > > >
> > > > --
> > > > Sent from Gmail Mobile
> > > >
> > >
> > --
> > Sent from Gmail Mobile
> >

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by sivasankar Reddy <ma...@gmail.com>.
Hi Bryan, 

Thanks for the reply. looks like even set-property will be a new feature, as current CLI doesn't have that. 

Could you please share timelines of these features if its in roadmap. 
1. "set-property"
2. “parameters” 

The only option currently is set to set sensitive parameters manually? or any other option through CLI 

Regards,
Siva

On 2019/08/22 04:10:46, Bryan Bende <bb...@gmail.com> wrote: 
> Currently there isn’t a good way with the CLI, we would need to add a
> command like set-property that took the id of a component, and the name and
> value for a property.
> 
> The next release will have a new feature called “parameters” which will
> solve this problem. You’ll be able to use a parameter in a sensitive
> property and then use CLI to set parameter values.
> 
> On Wed, Aug 21, 2019 at 12:29 PM sivasankar Reddy <ma...@gmail.com>
> wrote:
> 
> > Hi Bryan,
> >
> > Thanks for the update. Could you please mention about
> > sensitive parameters aspect from deployment point of view using CLI.
> >
> > How to set the sensitive parameters values after pg-import.
> >
> > Regards,
> > Siva
> >
> > On 2019/08/21 11:47:49, Bryan Bende <bb...@gmail.com> wrote:
> > > It does not, I was saying we could easily improve the command to add
> > those
> > > parameters.
> > >
> > > On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy <ma...@gmail.com>
> > > wrote:
> > >
> > > > Hi  Bryan,
> > > >
> > > > Thanks for the update. Does the existing pg-import takes optional
> > params
> > > > for x/y coordinates. If yes can you share the sample command as
> > pg-import
> > > > --help doesn't has this info.
> > > >
> > > > Regards,
> > > > siva
> > > >
> > > >
> > > > On 2019/08/20 22:19:23, Bryan Bende <bb...@gmail.com> wrote:
> > > > > Hello,
> > > > >
> > > > > 1) pg-import will keeping finding a location that does not overlap
> > with
> > > > any
> > > > > existing process groups, so it is more of auto layout. We could
> > easily
> > > > add
> > > > > optional params for x/y coordinates.
> > > > >
> > > > > 2) Both are fine choices.. CLI is more geared towards simple
> > commands,
> > > > > where as with NiPyApi you could more easily write complex logic in
> > > > Python.
> > > > > With CLI you would write a wrapper shell script that made individual
> > > > calls
> > > > > to the CLI. I think it is really just a matter of preference.
> > > > >
> > > > > -Bryan
> > > > >
> > > > >
> > > > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy <
> > mail2msiva@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Bryan,
> > > > > >
> > > > > > Thanks for your input. I would like to know more
> > > > > >
> > > > > > 1. how does CLI pg-import  works if already canvas has 100 to 150
> > > > flows.
> > > > > > Will CLI pg-import always place the flow in canvas which can be
> > easily
> > > > > > located. As nipyapi has location(x,y) during deploy flow.
> > > > > >
> > > > > > 2. From enterprise usage, do you recommend toolkit CLI vs nipyapi,
> > it
> > > > > > seems toolkit CLI is part of official Nifi.
> > > > > >
> > > > > > Any other points regarding this will be helpful.
> > > > > >
> > > > > > Regards,
> > > > > > Siva
> > > > > >
> > > > > > On 2019/08/08 22:41:43, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > > Hello,
> > > > > > >
> > > > > > > A lot depends on whether you are doing the promotion with the UI
> > or
> > > > > > > with scripts using something like NiFi CLI or NiPyApi.
> > > > > > >
> > > > > > > 1) Currently references to controller services outside the
> > versioned
> > > > > > > process group need to be set after initial import. In the next
> > > > release
> > > > > > > it will auto-select by name as long as there is only one service
> > with
> > > > > > > the given name and type in a parent group (name is not unique).
> > > > > > >
> > > > > > > 2) The snapshot of the variables are part of the versioned flow,
> > so
> > > > > > > after importing you would have to set the new values in the
> > target
> > > > > > > environment. This can be done in the UI or with scripts.
> > > > > > >
> > > > > > > 3) The REST API allows for positioning of any component that is
> > > > > > > created. i can't speak for how to do it with NiPyApi, but I know
> > the
> > > > > > > CLI pg-import command attempts to automatically organize the
> > imported
> > > > > > > process groups on the root canvas so that they don't overlap.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Bryan
> > > > > > >
> > > > > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <
> > > > mail2msiva@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > We are looking for nifi deploy automation from regsitry ->
> > higher
> > > > nifi
> > > > > > instance. Quick observations during intial evaluation.
> > > > > > > >
> > > > > > > > 1. After process group import, controller services are not set
> > in
> > > > nifi
> > > > > > instance
> > > > > > > > 2. How to set variables from source process group to
> > destination
> > > > > > process group.
> > > > > > > > 3. Is there any possibility of definining the positioning of
> > > > process
> > > > > > group to be imported in Nifi Canvas.
> > > > > > > >
> > > > > > > > Any other best practices  regarding NiFi deployment automation
> > > > will be
> > > > > > helpful.
> > > > > > > >
> > > > > > > > Thanks in Advance!
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > --
> > > > > Sent from Gmail Mobile
> > > > >
> > > >
> > > --
> > > Sent from Gmail Mobile
> > >
> >
> -- 
> Sent from Gmail Mobile
> 

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by Bryan Bende <bb...@gmail.com>.
Currently there isn’t a good way with the CLI, we would need to add a
command like set-property that took the id of a component, and the name and
value for a property.

The next release will have a new feature called “parameters” which will
solve this problem. You’ll be able to use a parameter in a sensitive
property and then use CLI to set parameter values.

On Wed, Aug 21, 2019 at 12:29 PM sivasankar Reddy <ma...@gmail.com>
wrote:

> Hi Bryan,
>
> Thanks for the update. Could you please mention about
> sensitive parameters aspect from deployment point of view using CLI.
>
> How to set the sensitive parameters values after pg-import.
>
> Regards,
> Siva
>
> On 2019/08/21 11:47:49, Bryan Bende <bb...@gmail.com> wrote:
> > It does not, I was saying we could easily improve the command to add
> those
> > parameters.
> >
> > On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy <ma...@gmail.com>
> > wrote:
> >
> > > Hi  Bryan,
> > >
> > > Thanks for the update. Does the existing pg-import takes optional
> params
> > > for x/y coordinates. If yes can you share the sample command as
> pg-import
> > > --help doesn't has this info.
> > >
> > > Regards,
> > > siva
> > >
> > >
> > > On 2019/08/20 22:19:23, Bryan Bende <bb...@gmail.com> wrote:
> > > > Hello,
> > > >
> > > > 1) pg-import will keeping finding a location that does not overlap
> with
> > > any
> > > > existing process groups, so it is more of auto layout. We could
> easily
> > > add
> > > > optional params for x/y coordinates.
> > > >
> > > > 2) Both are fine choices.. CLI is more geared towards simple
> commands,
> > > > where as with NiPyApi you could more easily write complex logic in
> > > Python.
> > > > With CLI you would write a wrapper shell script that made individual
> > > calls
> > > > to the CLI. I think it is really just a matter of preference.
> > > >
> > > > -Bryan
> > > >
> > > >
> > > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy <
> mail2msiva@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Bryan,
> > > > >
> > > > > Thanks for your input. I would like to know more
> > > > >
> > > > > 1. how does CLI pg-import  works if already canvas has 100 to 150
> > > flows.
> > > > > Will CLI pg-import always place the flow in canvas which can be
> easily
> > > > > located. As nipyapi has location(x,y) during deploy flow.
> > > > >
> > > > > 2. From enterprise usage, do you recommend toolkit CLI vs nipyapi,
> it
> > > > > seems toolkit CLI is part of official Nifi.
> > > > >
> > > > > Any other points regarding this will be helpful.
> > > > >
> > > > > Regards,
> > > > > Siva
> > > > >
> > > > > On 2019/08/08 22:41:43, Bryan Bende <bb...@gmail.com> wrote:
> > > > > > Hello,
> > > > > >
> > > > > > A lot depends on whether you are doing the promotion with the UI
> or
> > > > > > with scripts using something like NiFi CLI or NiPyApi.
> > > > > >
> > > > > > 1) Currently references to controller services outside the
> versioned
> > > > > > process group need to be set after initial import. In the next
> > > release
> > > > > > it will auto-select by name as long as there is only one service
> with
> > > > > > the given name and type in a parent group (name is not unique).
> > > > > >
> > > > > > 2) The snapshot of the variables are part of the versioned flow,
> so
> > > > > > after importing you would have to set the new values in the
> target
> > > > > > environment. This can be done in the UI or with scripts.
> > > > > >
> > > > > > 3) The REST API allows for positioning of any component that is
> > > > > > created. i can't speak for how to do it with NiPyApi, but I know
> the
> > > > > > CLI pg-import command attempts to automatically organize the
> imported
> > > > > > process groups on the root canvas so that they don't overlap.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Bryan
> > > > > >
> > > > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <
> > > mail2msiva@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > We are looking for nifi deploy automation from regsitry ->
> higher
> > > nifi
> > > > > instance. Quick observations during intial evaluation.
> > > > > > >
> > > > > > > 1. After process group import, controller services are not set
> in
> > > nifi
> > > > > instance
> > > > > > > 2. How to set variables from source process group to
> destination
> > > > > process group.
> > > > > > > 3. Is there any possibility of definining the positioning of
> > > process
> > > > > group to be imported in Nifi Canvas.
> > > > > > >
> > > > > > > Any other best practices  regarding NiFi deployment automation
> > > will be
> > > > > helpful.
> > > > > > >
> > > > > > > Thanks in Advance!
> > > > > > >
> > > > > >
> > > > >
> > > > --
> > > > Sent from Gmail Mobile
> > > >
> > >
> > --
> > Sent from Gmail Mobile
> >
>
-- 
Sent from Gmail Mobile

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by sivasankar Reddy <ma...@gmail.com>.
Hi Bryan, 

Thanks for the update. Could you please mention about 
sensitive parameters aspect from deployment point of view using CLI. 

How to set the sensitive parameters values after pg-import. 

Regards,
Siva

On 2019/08/21 11:47:49, Bryan Bende <bb...@gmail.com> wrote: 
> It does not, I was saying we could easily improve the command to add those
> parameters.
> 
> On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy <ma...@gmail.com>
> wrote:
> 
> > Hi  Bryan,
> >
> > Thanks for the update. Does the existing pg-import takes optional params
> > for x/y coordinates. If yes can you share the sample command as pg-import
> > --help doesn't has this info.
> >
> > Regards,
> > siva
> >
> >
> > On 2019/08/20 22:19:23, Bryan Bende <bb...@gmail.com> wrote:
> > > Hello,
> > >
> > > 1) pg-import will keeping finding a location that does not overlap with
> > any
> > > existing process groups, so it is more of auto layout. We could easily
> > add
> > > optional params for x/y coordinates.
> > >
> > > 2) Both are fine choices.. CLI is more geared towards simple commands,
> > > where as with NiPyApi you could more easily write complex logic in
> > Python.
> > > With CLI you would write a wrapper shell script that made individual
> > calls
> > > to the CLI. I think it is really just a matter of preference.
> > >
> > > -Bryan
> > >
> > >
> > > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy <ma...@gmail.com>
> > > wrote:
> > >
> > > > Hi Bryan,
> > > >
> > > > Thanks for your input. I would like to know more
> > > >
> > > > 1. how does CLI pg-import  works if already canvas has 100 to 150
> > flows.
> > > > Will CLI pg-import always place the flow in canvas which can be easily
> > > > located. As nipyapi has location(x,y) during deploy flow.
> > > >
> > > > 2. From enterprise usage, do you recommend toolkit CLI vs nipyapi, it
> > > > seems toolkit CLI is part of official Nifi.
> > > >
> > > > Any other points regarding this will be helpful.
> > > >
> > > > Regards,
> > > > Siva
> > > >
> > > > On 2019/08/08 22:41:43, Bryan Bende <bb...@gmail.com> wrote:
> > > > > Hello,
> > > > >
> > > > > A lot depends on whether you are doing the promotion with the UI or
> > > > > with scripts using something like NiFi CLI or NiPyApi.
> > > > >
> > > > > 1) Currently references to controller services outside the versioned
> > > > > process group need to be set after initial import. In the next
> > release
> > > > > it will auto-select by name as long as there is only one service with
> > > > > the given name and type in a parent group (name is not unique).
> > > > >
> > > > > 2) The snapshot of the variables are part of the versioned flow, so
> > > > > after importing you would have to set the new values in the target
> > > > > environment. This can be done in the UI or with scripts.
> > > > >
> > > > > 3) The REST API allows for positioning of any component that is
> > > > > created. i can't speak for how to do it with NiPyApi, but I know the
> > > > > CLI pg-import command attempts to automatically organize the imported
> > > > > process groups on the root canvas so that they don't overlap.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Bryan
> > > > >
> > > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <
> > mail2msiva@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We are looking for nifi deploy automation from regsitry -> higher
> > nifi
> > > > instance. Quick observations during intial evaluation.
> > > > > >
> > > > > > 1. After process group import, controller services are not set in
> > nifi
> > > > instance
> > > > > > 2. How to set variables from source process group to destination
> > > > process group.
> > > > > > 3. Is there any possibility of definining the positioning of
> > process
> > > > group to be imported in Nifi Canvas.
> > > > > >
> > > > > > Any other best practices  regarding NiFi deployment automation
> > will be
> > > > helpful.
> > > > > >
> > > > > > Thanks in Advance!
> > > > > >
> > > > >
> > > >
> > > --
> > > Sent from Gmail Mobile
> > >
> >
> -- 
> Sent from Gmail Mobile
> 

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by Bryan Bende <bb...@gmail.com>.
It does not, I was saying we could easily improve the command to add those
parameters.

On Wed, Aug 21, 2019 at 6:36 AM sivasankar Reddy <ma...@gmail.com>
wrote:

> Hi  Bryan,
>
> Thanks for the update. Does the existing pg-import takes optional params
> for x/y coordinates. If yes can you share the sample command as pg-import
> --help doesn't has this info.
>
> Regards,
> siva
>
>
> On 2019/08/20 22:19:23, Bryan Bende <bb...@gmail.com> wrote:
> > Hello,
> >
> > 1) pg-import will keeping finding a location that does not overlap with
> any
> > existing process groups, so it is more of auto layout. We could easily
> add
> > optional params for x/y coordinates.
> >
> > 2) Both are fine choices.. CLI is more geared towards simple commands,
> > where as with NiPyApi you could more easily write complex logic in
> Python.
> > With CLI you would write a wrapper shell script that made individual
> calls
> > to the CLI. I think it is really just a matter of preference.
> >
> > -Bryan
> >
> >
> > On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy <ma...@gmail.com>
> > wrote:
> >
> > > Hi Bryan,
> > >
> > > Thanks for your input. I would like to know more
> > >
> > > 1. how does CLI pg-import  works if already canvas has 100 to 150
> flows.
> > > Will CLI pg-import always place the flow in canvas which can be easily
> > > located. As nipyapi has location(x,y) during deploy flow.
> > >
> > > 2. From enterprise usage, do you recommend toolkit CLI vs nipyapi, it
> > > seems toolkit CLI is part of official Nifi.
> > >
> > > Any other points regarding this will be helpful.
> > >
> > > Regards,
> > > Siva
> > >
> > > On 2019/08/08 22:41:43, Bryan Bende <bb...@gmail.com> wrote:
> > > > Hello,
> > > >
> > > > A lot depends on whether you are doing the promotion with the UI or
> > > > with scripts using something like NiFi CLI or NiPyApi.
> > > >
> > > > 1) Currently references to controller services outside the versioned
> > > > process group need to be set after initial import. In the next
> release
> > > > it will auto-select by name as long as there is only one service with
> > > > the given name and type in a parent group (name is not unique).
> > > >
> > > > 2) The snapshot of the variables are part of the versioned flow, so
> > > > after importing you would have to set the new values in the target
> > > > environment. This can be done in the UI or with scripts.
> > > >
> > > > 3) The REST API allows for positioning of any component that is
> > > > created. i can't speak for how to do it with NiPyApi, but I know the
> > > > CLI pg-import command attempts to automatically organize the imported
> > > > process groups on the root canvas so that they don't overlap.
> > > >
> > > > Thanks,
> > > >
> > > > Bryan
> > > >
> > > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <
> mail2msiva@gmail.com>
> > > wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > We are looking for nifi deploy automation from regsitry -> higher
> nifi
> > > instance. Quick observations during intial evaluation.
> > > > >
> > > > > 1. After process group import, controller services are not set in
> nifi
> > > instance
> > > > > 2. How to set variables from source process group to destination
> > > process group.
> > > > > 3. Is there any possibility of definining the positioning of
> process
> > > group to be imported in Nifi Canvas.
> > > > >
> > > > > Any other best practices  regarding NiFi deployment automation
> will be
> > > helpful.
> > > > >
> > > > > Thanks in Advance!
> > > > >
> > > >
> > >
> > --
> > Sent from Gmail Mobile
> >
>
-- 
Sent from Gmail Mobile

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by sivasankar Reddy <ma...@gmail.com>.
Hi  Bryan, 

Thanks for the update. Does the existing pg-import takes optional params for x/y coordinates. If yes can you share the sample command as pg-import --help doesn't has this info. 

Regards,
siva


On 2019/08/20 22:19:23, Bryan Bende <bb...@gmail.com> wrote: 
> Hello,
> 
> 1) pg-import will keeping finding a location that does not overlap with any
> existing process groups, so it is more of auto layout. We could easily add
> optional params for x/y coordinates.
> 
> 2) Both are fine choices.. CLI is more geared towards simple commands,
> where as with NiPyApi you could more easily write complex logic in Python.
> With CLI you would write a wrapper shell script that made individual calls
> to the CLI. I think it is really just a matter of preference.
> 
> -Bryan
> 
> 
> On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy <ma...@gmail.com>
> wrote:
> 
> > Hi Bryan,
> >
> > Thanks for your input. I would like to know more
> >
> > 1. how does CLI pg-import  works if already canvas has 100 to 150 flows.
> > Will CLI pg-import always place the flow in canvas which can be easily
> > located. As nipyapi has location(x,y) during deploy flow.
> >
> > 2. From enterprise usage, do you recommend toolkit CLI vs nipyapi, it
> > seems toolkit CLI is part of official Nifi.
> >
> > Any other points regarding this will be helpful.
> >
> > Regards,
> > Siva
> >
> > On 2019/08/08 22:41:43, Bryan Bende <bb...@gmail.com> wrote:
> > > Hello,
> > >
> > > A lot depends on whether you are doing the promotion with the UI or
> > > with scripts using something like NiFi CLI or NiPyApi.
> > >
> > > 1) Currently references to controller services outside the versioned
> > > process group need to be set after initial import. In the next release
> > > it will auto-select by name as long as there is only one service with
> > > the given name and type in a parent group (name is not unique).
> > >
> > > 2) The snapshot of the variables are part of the versioned flow, so
> > > after importing you would have to set the new values in the target
> > > environment. This can be done in the UI or with scripts.
> > >
> > > 3) The REST API allows for positioning of any component that is
> > > created. i can't speak for how to do it with NiPyApi, but I know the
> > > CLI pg-import command attempts to automatically organize the imported
> > > process groups on the root canvas so that they don't overlap.
> > >
> > > Thanks,
> > >
> > > Bryan
> > >
> > > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <ma...@gmail.com>
> > wrote:
> > > >
> > > > Hi,
> > > >
> > > > We are looking for nifi deploy automation from regsitry -> higher nifi
> > instance. Quick observations during intial evaluation.
> > > >
> > > > 1. After process group import, controller services are not set in nifi
> > instance
> > > > 2. How to set variables from source process group to destination
> > process group.
> > > > 3. Is there any possibility of definining the positioning of process
> > group to be imported in Nifi Canvas.
> > > >
> > > > Any other best practices  regarding NiFi deployment automation will be
> > helpful.
> > > >
> > > > Thanks in Advance!
> > > >
> > >
> >
> -- 
> Sent from Gmail Mobile
> 

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by Bryan Bende <bb...@gmail.com>.
Hello,

1) pg-import will keeping finding a location that does not overlap with any
existing process groups, so it is more of auto layout. We could easily add
optional params for x/y coordinates.

2) Both are fine choices.. CLI is more geared towards simple commands,
where as with NiPyApi you could more easily write complex logic in Python.
With CLI you would write a wrapper shell script that made individual calls
to the CLI. I think it is really just a matter of preference.

-Bryan


On Tue, Aug 20, 2019 at 5:11 PM sivasankar Reddy <ma...@gmail.com>
wrote:

> Hi Bryan,
>
> Thanks for your input. I would like to know more
>
> 1. how does CLI pg-import  works if already canvas has 100 to 150 flows.
> Will CLI pg-import always place the flow in canvas which can be easily
> located. As nipyapi has location(x,y) during deploy flow.
>
> 2. From enterprise usage, do you recommend toolkit CLI vs nipyapi, it
> seems toolkit CLI is part of official Nifi.
>
> Any other points regarding this will be helpful.
>
> Regards,
> Siva
>
> On 2019/08/08 22:41:43, Bryan Bende <bb...@gmail.com> wrote:
> > Hello,
> >
> > A lot depends on whether you are doing the promotion with the UI or
> > with scripts using something like NiFi CLI or NiPyApi.
> >
> > 1) Currently references to controller services outside the versioned
> > process group need to be set after initial import. In the next release
> > it will auto-select by name as long as there is only one service with
> > the given name and type in a parent group (name is not unique).
> >
> > 2) The snapshot of the variables are part of the versioned flow, so
> > after importing you would have to set the new values in the target
> > environment. This can be done in the UI or with scripts.
> >
> > 3) The REST API allows for positioning of any component that is
> > created. i can't speak for how to do it with NiPyApi, but I know the
> > CLI pg-import command attempts to automatically organize the imported
> > process groups on the root canvas so that they don't overlap.
> >
> > Thanks,
> >
> > Bryan
> >
> > On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <ma...@gmail.com>
> wrote:
> > >
> > > Hi,
> > >
> > > We are looking for nifi deploy automation from regsitry -> higher nifi
> instance. Quick observations during intial evaluation.
> > >
> > > 1. After process group import, controller services are not set in nifi
> instance
> > > 2. How to set variables from source process group to destination
> process group.
> > > 3. Is there any possibility of definining the positioning of process
> group to be imported in Nifi Canvas.
> > >
> > > Any other best practices  regarding NiFi deployment automation will be
> helpful.
> > >
> > > Thanks in Advance!
> > >
> >
>
-- 
Sent from Gmail Mobile

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by sivasankar Reddy <ma...@gmail.com>.
Hi Bryan, 

Thanks for your input. I would like to know more 

1. how does CLI pg-import  works if already canvas has 100 to 150 flows. Will CLI pg-import always place the flow in canvas which can be easily located. As nipyapi has location(x,y) during deploy flow. 

2. From enterprise usage, do you recommend toolkit CLI vs nipyapi, it seems toolkit CLI is part of official Nifi. 

Any other points regarding this will be helpful. 

Regards,
Siva 

On 2019/08/08 22:41:43, Bryan Bende <bb...@gmail.com> wrote: 
> Hello,
> 
> A lot depends on whether you are doing the promotion with the UI or
> with scripts using something like NiFi CLI or NiPyApi.
> 
> 1) Currently references to controller services outside the versioned
> process group need to be set after initial import. In the next release
> it will auto-select by name as long as there is only one service with
> the given name and type in a parent group (name is not unique).
> 
> 2) The snapshot of the variables are part of the versioned flow, so
> after importing you would have to set the new values in the target
> environment. This can be done in the UI or with scripts.
> 
> 3) The REST API allows for positioning of any component that is
> created. i can't speak for how to do it with NiPyApi, but I know the
> CLI pg-import command attempts to automatically organize the imported
> process groups on the root canvas so that they don't overlap.
> 
> Thanks,
> 
> Bryan
> 
> On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <ma...@gmail.com> wrote:
> >
> > Hi,
> >
> > We are looking for nifi deploy automation from regsitry -> higher nifi instance. Quick observations during intial evaluation.
> >
> > 1. After process group import, controller services are not set in nifi instance
> > 2. How to set variables from source process group to destination process group.
> > 3. Is there any possibility of definining the positioning of process group to be imported in Nifi Canvas.
> >
> > Any other best practices  regarding NiFi deployment automation will be helpful.
> >
> > Thanks in Advance!
> >
> 

Re: nifi deploy automation - variables pass through (incl controller services)

Posted by Bryan Bende <bb...@gmail.com>.
Hello,

A lot depends on whether you are doing the promotion with the UI or
with scripts using something like NiFi CLI or NiPyApi.

1) Currently references to controller services outside the versioned
process group need to be set after initial import. In the next release
it will auto-select by name as long as there is only one service with
the given name and type in a parent group (name is not unique).

2) The snapshot of the variables are part of the versioned flow, so
after importing you would have to set the new values in the target
environment. This can be done in the UI or with scripts.

3) The REST API allows for positioning of any component that is
created. i can't speak for how to do it with NiPyApi, but I know the
CLI pg-import command attempts to automatically organize the imported
process groups on the root canvas so that they don't overlap.

Thanks,

Bryan

On Thu, Aug 8, 2019 at 4:32 PM sivasankar Reddy <ma...@gmail.com> wrote:
>
> Hi,
>
> We are looking for nifi deploy automation from regsitry -> higher nifi instance. Quick observations during intial evaluation.
>
> 1. After process group import, controller services are not set in nifi instance
> 2. How to set variables from source process group to destination process group.
> 3. Is there any possibility of definining the positioning of process group to be imported in Nifi Canvas.
>
> Any other best practices  regarding NiFi deployment automation will be helpful.
>
> Thanks in Advance!
>