You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Rodriguez, Matthew" <Ma...@bestbuy.com> on 2021/06/04 20:17:41 UTC
Re: [CAUTION! EXTERNAL] RE: Problem with the GetFile processor
deleting my entire installation
GetFile is considered a restricted component, when running in secure mode like you are I believe you have to explicitly give users/groups permissions to use restricted components. If your users don’t have a valid use case for those components, or you desire to protect yourself from that scenario you listed, don’t grant them access to use restricted components. All restricted components are called out in the documentation or the UI, they also state which specific restrictions they need if you want to enable the use of only some restricted components.
Permissions to these components are managed globally in the policies portion of the hamburger menu.
[Graphical user interface, text, website Description automatically generated]
[Graphical user interface, application Description automatically generated]
From: Ruth, Thomas <th...@optum.com>
Date: Friday, June 4, 2021 at 3:11 PM
To: users@nifi.apache.org <us...@nifi.apache.org>
Subject: [CAUTION! EXTERNAL] RE: Problem with the GetFile processor deleting my entire installation
⚠ This message is from an external sender and could be a phish. ⚠
Slow down, read carefully and look for signs that it may be a phish. If you think it’s malicious, click the report phish button or forward this email to phishing@bestbuy. com.
No matter how secure I make the cluster, what’s to prevent a user from:
* Using GetFile to copy the NiFi configuration files and keystores into flows.
* Creating flows which contain python scripts that utilize the keys/secrets in the nifi configuration files to decrypt the contents of the keystores.
* Executing those flows to gain access to the private keys used to encrypt data in NiFi
* Using that data to gain unauthorized access to the cluster to further compromise it, possibly by generating their own admin-level user certificates.
I’m genuinely stumped here as to how to create a solution that is production ready and that prevents a complete system failure by a user accidentally supplying the wrong directory to a processor component or generally creating python scripts that execute as the nifi user, being able to basically use a NiFi server as their own script-running playground.
Tom
From: Mike Sofen <ms...@runbox.com>
Sent: Friday, June 4, 2021 2:02 PM
To: users@nifi.apache.org
Subject: RE: Problem with the GetFile processor deleting my entire installation
Setting aside the user auth/permissions issue (which sounds like, from your initial email, that you nailed perfectly with a secured cluster), I’ll address this accidental deletion issue using what I consider a general design pattern for Nifi (or any process that write-touches files):
* I don’t allow physical file deletes (which happen with the default GetFiles settings)
* I create an archive folder for files I’ve processed if I want/need to move them out of the processing folder
* I always set the “leave files” flag
* I use either the timestamp or filename tracker to eliminate repeated processing if I’m not moving the files
* OR – I use Move File flag to the archive folder destination.
* And I advise new users to always spec a small test folder to start, and to double-check that the remove files is turned off.
In some cases, like when I’m processing text files in place in secure repo, I can’t move the files so I use the timestamp eval method.
In others, like log files where I definitely want to archive them out of that processing folder, I use the move model.
Don’t get discouraged about this twist in the road – you’ll find Nifi to be a truly exceptional product for pipeline automation, wildly more powerful/flexible/stable than any other product out there (I’ve used most). I’ve used it IoT to DB to ML processing, document processing, metrology data processing, you name it. In particular, the built data provenance and auditability will be valuable for your situation at Optum. All the best,
Mike
From: Ruth, Thomas <th...@optum.com>>
Sent: Friday, June 04, 2021 12:26 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: Problem with the GetFile processor deleting my entire installation
Is it recommended that after installing NiFi, that I then proceed to remove read permissions from all installation files and directories in order to protect them from removal by users? Will this present a problem running the software if it’s unable to read any files?
So for fun, I loaded up a quick nifi container:
docker run –name nifi -p 8080:8080 -d apache/nifi:latest
Connected to localhost:8080/nifi
Created a GetFile processor, with the directory /opt/nifi. Connected the success to a LogAttributes. Hit Run….
Now I get this:
HTTP ERROR 404 /nifi/canvas.jsp
URI:
/nifi/
STATUS:
404
MESSAGE:
/nifi/canvas.jsp
SERVLET:
jsp
This behavior shouldn’t be possible, in my opinion. It’s putting security in the hands of my developers. I really am looking for a solution to this issue.
The help text says this:
Creates FlowFiles from files in a directory. NiFi will ignore files it doesn't have at least read permissions for.
So as you suggested, I removed the read permission recursively from all files in /opt/nifi. After doing this, nifi no longer starts.
find /opt/nifi -print > /tmp/files.txt
for I in `cat /tmp/files.txt`; do chmod a-r $i; done
I also went to https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator<https://urldefense.com/v3/__https:/nvd.nist.gov/vuln-metrics/cvss/v3-calculator__;!!KtbpKd1p8A!4aX8XRjZKKreeGNa5t2R6_MwRpUiVieStkOv-UX-JszUHN60ElFsBxLNBQ0NeSC3PXMdYRlBXQ$> and attempted to calculate a CVSS score for this vulnerability. I ended up calculating a score of 8.0.
Tom
From: Russell Bateman <ru...@windofkeltia.com>>
Sent: Friday, June 4, 2021 11:20 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Problem with the GetFile processor deleting my entire installation
Oh, sorry, to finish the answer, yes, you do need to be very careful how you specify the Input Directory and File Filter properties; the last one is a regular expression. It's true that the documentation is less than flag-waving or hair-lighting-on-fire as it presents its help on filling those in.
Russ
On 6/4/21 11:16 AM, Russell Bateman wrote:
Sorry for this behavior of GetFile which is purposeful. If you configure to keep the files instead of removing them, you'll keep getting the same files ingested over and over again as flow files. It's just how it is.
The secret was to read the help blurb when configuring this processor.
Hope this helps,
Russ
On 6/4/21 10:44 AM, Ruth, Thomas wrote:
Hello,
I recently built a 3-node NiFi cluster in my organization as a proof-of-concept for some work we are doing. I used version 1.13.2 and installed it onto 3 CentOS 7.9 systems. In my organization, I don’t have root access to the system, so I used a different user called “nfadm” to install and run the product. I don’t remember seeing anything in the documentation that stated that this would be an issue.
I am also new to NiFi, and was relying heavily on the Admin documentation on the web site for instructions to set up the OS and NiFi installations. I configured certificate-based security and distributed them to my users. I also configured policies for groups that I thought were OK for them from a development standpoint.
I had an incident occur yesterday in which a user, who is also new to NiFi, ran a component called “GetFile” for the filesystem “/” with the default settings (Recurse=true, KeepSourceFile=false). Well, this essentially ran “rm -rf /” as the user that owns all the installation files and files in the various repositories, nfadm, the same user running the NiFi processes. This deleted all the installation and configuration files for the entire cluster, making it completely useless now.
I am surprised to find out that NiFi allowed a user to basically wipe out all the files the user running the NiFi server had access to. I would expect much higher security to be present in a default system. I have some questions that hopefully you can help me with:
Is this a known issue in NiFi?
Am I doing something wrong when configuring or installing NiFi?
Is there a section in the documentation that warns me of this type of scenario?
Thanks in advance for your help with this,
Tom Ruth
Sr. Data Engineer
Optum, Inc.
E: Thomas.Ruth@Optum.com<ma...@Optum.com>
This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.
This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.
This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.
Re: [CAUTION! EXTERNAL] RE: Problem with the GetFile processor
deleting my entire installation
Posted by Chris Sampson <ch...@naimuri.com>.
The content repository can be encrypted[1] along with other nifi internal
directory content (see similar sections of the admin guide).
Nifi config itself can be encrypted [2] and sensitive properties within the
flow are encrypted by default/with a configured key in nifi properties.
Individual fields can be hashed (encrypted) using RecordPath expressions
[3] in a Record-based processor such as UpdateRecord [4]. That way you can
read data into nifi (e.g. as json), process each record in the file and
Encrypt individual fields.
Alternatively, entire files can be encrypted [5].
There are plenty of scripted processor options available that may be worth
considering, such as ScriptedTransformRecord [6] if someone wants to write
a custom hash against individual fields within a record but not use the
builtin RecordPath functions. These scripts could be python or groovy (or
other) and may be more constrained (system access wise) than using a
processor such as ExecuteScript (but don't take my word on that).
If you're really concerned about the scripts you're users might write, why
not have a test nifi instance they have access to where they write and test
their scripts. Someone trusted/more experienced can then review the scripts
and flows before they're promoted into the production environment with real
data? In prod, you could use a different policy setup where fewer people
have access and/or very few people can modify the flow/certain restricted
processors, etc.
This is effectively how my team currently operates - we have dev where
anyone on the team with an account can do pretty much anything (if they
break it, they fix it!), then we promote flows via Registry [7] (after
review) through QA and into Prod where those with access can only view the
flows and operate (start/stop), not actually make any modifications by
default.
[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#encrypted-file-system-content-repository-properties
[2]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#encrypt-config_tool
[3] https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html#hash
[4]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.UpdateRecord/index.html
[5]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.CryptographicHashContent/index.html
[6]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-scripting-nar/1.13.2/org.apache.nifi.processors.script.ScriptedTransformRecord/index.html
[7] https://nifi.apache.org/docs/nifi-registry-docs/index.html
Cheers,
Chris Sampson
On Fri, 4 Jun 2021, 21:32 Ruth, Thomas, <th...@optum.com> wrote:
> Excellent, thanks a ton! This is what I was looking for. I have some
> reading to do…
>
>
>
> We are all very new to NiFi here, and it seems like right now, they want
> to create custom python scripts to do column-level encryption on data since
> NiFi is unable to do so natively. The goal is to query certain data
> sources, encrypt fields that contain PII, then send that to Azure.
>
>
>
> It sounds like though, since I’m concerned about mistakes being made that
> causes me to rebuild a whole cluster, that NiFi may not be the solution
> they are looking for in this particular scenario, or at the very least
> perhaps they need to compromise and find a way to secure the PII and still
> send it to Azure by perhaps encrypting the entire flow.
>
>
>
> Tom
>
>
>
> *From:* Joe Witt <jo...@gmail.com>
> *Sent:* Friday, June 4, 2021 2:22 PM
> *To:* users@nifi.apache.org
> *Subject:* Re: [CAUTION! EXTERNAL] RE: Problem with the GetFile processor
> deleting my entire installation
>
>
>
> You've already gotten some great responses here. But to add...
>
>
>
>
>
> A secured NiFi instance allows you to fully control what a
> given authorized user is allowed to do at very granular levels. For
> instance the administrator gets to choose whether user A can use/modify
> processors that manipulate/interact with the file system. That addresses
> very well your statement about developers should not be able to do such
> things. We agree. That feature is critical. So then there is the issue
> of what to do about authorized users having powerful tools at their
> disposal. For authorized users they can write processors with custom code
> on the fly, they can manipulate the file system, etc.. There is no
> mechanism by which we can fully protect authorized users from doing certain
> problematic things (like deleting nifi itself) while still providing those
> general powers I describe. In GetFile itself we of course can put
> guardrails around in the form of various directories to not allow
> capture/delete from. But that does get difficult with portability across
> various systems/sim-links/etc..
>
>
>
> For your specific list:
>
> - Using GetFile to copy the NiFi configuration files and keystores
> into flows.
>
>
> - Creating flows which contain python scripts that utilize the
> keys/secrets in the nifi configuration files to decrypt the contents of the
> keystores.
>
>
> - Executing those flows to gain access to the private keys used to
> encrypt data in NiFi
> - Using that data to gain unauthorized access to the cluster to
> further compromise it, possibly by generating their own admin-level user
> certificates.
>
>
>
> You need to take advantage of NiFi's authorization model and concept of
> restricted processors. In the end there will be someone you
> trust/authorize to use these things and indeed those individuals can do
> some really bad things. Again, that is an authorized user being allowed to
> manipulate the system. If you don't want any users, or certain users to
> have the ability, then restrict them. See [1] [2] [3] and the example
> specific to GetFile as James noted in [4] and commentary around restricted
> components here [5]
>
>
>
> [1]
> http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#restricted
>
> [2]
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Restricted_Components_in_Versioned_Flows
>
> [3]
> http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#UI-with-multi-tenant-authorization
>
> [4]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.GetFile/index.html
>
> [5] http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html
>
>
>
> Your concerns are understandable. But we have a pretty strong set of
> controls for you to use once you better understand the
> model/authorization/restrictions and audits in place and so on. This is
> solvable such that you can control who has authorization to do danger -
> those actions are also audited.
>
>
>
> Thanks
>
> Joe
>
>
>
> On Fri, Jun 4, 2021 at 1:18 PM Rodriguez, Matthew <
> Matthew.Rodriguez2@bestbuy.com> wrote:
>
> GetFile is considered a restricted component, when running in secure mode
> like you are I believe you have to explicitly give users/groups permissions
> to use restricted components. If your users don’t have a valid use case for
> those components, or you desire to protect yourself from that scenario you
> listed, don’t grant them access to use restricted components. All
> restricted components are called out in the documentation or the UI, they
> also state which specific restrictions they need if you want to enable the
> use of only some restricted components.
>
>
>
> Permissions to these components are managed globally in the policies
> portion of the hamburger menu.
>
>
>
> *[image: Graphical user interface, text, website Description automatically
> generated]*
>
>
>
> *[image: Graphical user interface, application Description automatically
> generated]*
>
>
>
>
>
> *From: *Ruth, Thomas <th...@optum.com>
> *Date: *Friday, June 4, 2021 at 3:11 PM
> *To: *users@nifi.apache.org <us...@nifi.apache.org>
> *Subject: *[CAUTION! EXTERNAL] RE: Problem with the GetFile processor
> deleting my entire installation
>
>
>
> ⚠ *This message is from an external sender and could be a phish.* ⚠
>
> Slow down, read carefully and look for signs that it may be a phish. If
> you think it’s malicious, click the report phish button or forward this
> email to phishing@bestbuy. com.
>
>
>
>
>
> No matter how secure I make the cluster, what’s to prevent a user from:
>
> - Using GetFile to copy the NiFi configuration files and keystores
> into flows.
> - Creating flows which contain python scripts that utilize the
> keys/secrets in the nifi configuration files to decrypt the contents of the
> keystores.
> - Executing those flows to gain access to the private keys used to
> encrypt data in NiFi
> - Using that data to gain unauthorized access to the cluster to
> further compromise it, possibly by generating their own admin-level user
> certificates.
>
>
>
> I’m genuinely stumped here as to how to create a solution that is
> production ready and that prevents a complete system failure by a user
> accidentally supplying the wrong directory to a processor component or
> generally creating python scripts that execute as the nifi user, being able
> to basically use a NiFi server as their own script-running playground.
>
>
>
> Tom
>
>
>
> *From:* Mike Sofen <ms...@runbox.com>
> *Sent:* Friday, June 4, 2021 2:02 PM
> *To:* users@nifi.apache.org
> *Subject:* RE: Problem with the GetFile processor deleting my entire
> installation
>
>
>
> Setting aside the user auth/permissions issue (which sounds like, from
> your initial email, that you nailed perfectly with a secured cluster), I’ll
> address this accidental deletion issue using what I consider a general
> design pattern for Nifi (or any process that write-touches files):
>
> - I don’t allow physical file deletes (which happen with the default
> GetFiles settings)
> - I create an archive folder for files I’ve processed if I want/need
> to move them out of the processing folder
> - I always set the “leave files” flag
> - I use either the timestamp or filename tracker to eliminate repeated
> processing if I’m not moving the files
> - OR – I use Move File flag to the archive folder destination.
> - And I advise new users to always spec a small test folder to start,
> and to double-check that the remove files is turned off.
>
>
>
> In some cases, like when I’m processing text files in place in secure
> repo, I can’t move the files so I use the timestamp eval method.
>
>
>
> In others, like log files where I definitely want to archive them out of
> that processing folder, I use the move model.
>
>
>
> Don’t get discouraged about this twist in the road – you’ll find Nifi to
> be a truly exceptional product for pipeline automation, wildly more
> powerful/flexible/stable than any other product out there (I’ve used
> most). I’ve used it IoT to DB to ML processing, document processing,
> metrology data processing, you name it. In particular, the built data
> provenance and auditability will be valuable for your situation at Optum.
> All the best,
>
>
>
> Mike
>
>
>
> *From:* Ruth, Thomas <th...@optum.com>
> *Sent:* Friday, June 04, 2021 12:26 PM
> *To:* users@nifi.apache.org
> *Subject:* RE: Problem with the GetFile processor deleting my entire
> installation
>
>
>
> Is it recommended that after installing NiFi, that I then proceed to
> remove read permissions from all installation files and directories in
> order to protect them from removal by users? Will this present a problem
> running the software if it’s unable to read any files?
>
>
>
> So for fun, I loaded up a quick nifi container:
> docker run –name nifi -p 8080:8080 -d apache/nifi:latest
>
>
>
> Connected to localhost:8080/nifi
>
>
>
> Created a GetFile processor, with the directory /opt/nifi. Connected the
> success to a LogAttributes. Hit Run….
>
>
>
> Now I get this:
> *HTTP ERROR 404 /nifi/canvas.jsp*
>
> *URI:*
>
> /nifi/
>
> *STATUS:*
>
> 404
>
> *MESSAGE:*
>
> /nifi/canvas.jsp
>
> *SERVLET:*
>
> jsp
>
>
>
> This behavior shouldn’t be possible, in my opinion. It’s putting security
> in the hands of my developers. I really am looking for a solution to this
> issue.
>
>
>
> The help text says this:
> Creates FlowFiles from files in a directory. NiFi will ignore files it
> doesn't have at least read permissions for.
>
>
>
> So as you suggested, I removed the read permission recursively from all
> files in /opt/nifi. After doing this, nifi no longer starts.
>
>
>
> find /opt/nifi -print > /tmp/files.txt
>
> for I in `cat /tmp/files.txt`; do chmod a-r $i; done
>
>
>
> I also went to https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator
> <https://urldefense.com/v3/__https:/nvd.nist.gov/vuln-metrics/cvss/v3-calculator__;!!KtbpKd1p8A!4aX8XRjZKKreeGNa5t2R6_MwRpUiVieStkOv-UX-JszUHN60ElFsBxLNBQ0NeSC3PXMdYRlBXQ$>
> and attempted to calculate a CVSS score for this vulnerability. I ended up
> calculating a score of 8.0.
>
>
>
> Tom
>
>
>
> *From:* Russell Bateman <ru...@windofkeltia.com>
> *Sent:* Friday, June 4, 2021 11:20 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: Problem with the GetFile processor deleting my entire
> installation
>
>
>
> Oh, sorry, to finish the answer, yes, you do need to be *very* careful
> how you specify the Input Directory and File Filter properties; the last
> one is a regular expression. It's true that the documentation is less than
> flag-waving or hair-lighting-on-fire as it presents its help on filling
> those in.
>
> Russ
>
> On 6/4/21 11:16 AM, Russell Bateman wrote:
>
> Sorry for this behavior of *GetFile* which is purposeful. If you
> configure to keep the files instead of removing them, you'll keep getting
> the same files ingested over and over again as flow files. It's just how it
> is.
>
> The secret was to read the help blurb when configuring this processor.
>
> Hope this helps,
>
> Russ
>
> On 6/4/21 10:44 AM, Ruth, Thomas wrote:
>
> Hello,
>
>
>
> I recently built a 3-node NiFi cluster in my organization as a
> proof-of-concept for some work we are doing. I used version 1.13.2 and
> installed it onto 3 CentOS 7.9 systems. In my organization, I don’t have
> root access to the system, so I used a different user called “nfadm” to
> install and run the product. I don’t remember seeing anything in the
> documentation that stated that this would be an issue.
>
>
>
> I am also new to NiFi, and was relying heavily on the Admin documentation
> on the web site for instructions to set up the OS and NiFi installations. I
> configured certificate-based security and distributed them to my users. I
> also configured policies for groups that I thought were OK for them from a
> development standpoint.
>
>
>
> I had an incident occur yesterday in which a user, who is also new to
> NiFi, ran a component called “GetFile” for the filesystem “/” with the
> default settings (Recurse=true, KeepSourceFile=false). Well, this
> essentially ran “rm -rf /” as the user that owns all the installation files
> and files in the various repositories, nfadm, the same user running the
> NiFi processes. This deleted all the installation and configuration files
> for the entire cluster, making it completely useless now.
>
>
>
> I am surprised to find out that NiFi allowed a user to basically wipe out
> all the files the user running the NiFi server had access to. I would
> expect much higher security to be present in a default system. I have some
> questions that hopefully you can help me with:
>
>
>
> Is this a known issue in NiFi?
>
>
>
> Am I doing something wrong when configuring or installing NiFi?
>
>
>
> Is there a section in the documentation that warns me of this type of
> scenario?
>
>
>
> Thanks in advance for your help with this,
>
>
>
> Tom Ruth
>
> Sr. Data Engineer
>
> Optum, Inc.
>
> E: Thomas.Ruth@Optum.com
>
>
>
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
>
>
>
>
>
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
>
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
>
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
>
RE: [CAUTION! EXTERNAL] RE: Problem with the GetFile processor
deleting my entire installation
Posted by "Ruth, Thomas" <th...@optum.com>.
Excellent, thanks a ton! This is what I was looking for. I have some reading to do…
We are all very new to NiFi here, and it seems like right now, they want to create custom python scripts to do column-level encryption on data since NiFi is unable to do so natively. The goal is to query certain data sources, encrypt fields that contain PII, then send that to Azure.
It sounds like though, since I’m concerned about mistakes being made that causes me to rebuild a whole cluster, that NiFi may not be the solution they are looking for in this particular scenario, or at the very least perhaps they need to compromise and find a way to secure the PII and still send it to Azure by perhaps encrypting the entire flow.
Tom
From: Joe Witt <jo...@gmail.com>
Sent: Friday, June 4, 2021 2:22 PM
To: users@nifi.apache.org
Subject: Re: [CAUTION! EXTERNAL] RE: Problem with the GetFile processor deleting my entire installation
You've already gotten some great responses here. But to add...
A secured NiFi instance allows you to fully control what a given authorized user is allowed to do at very granular levels. For instance the administrator gets to choose whether user A can use/modify processors that manipulate/interact with the file system. That addresses very well your statement about developers should not be able to do such things. We agree. That feature is critical. So then there is the issue of what to do about authorized users having powerful tools at their disposal. For authorized users they can write processors with custom code on the fly, they can manipulate the file system, etc.. There is no mechanism by which we can fully protect authorized users from doing certain problematic things (like deleting nifi itself) while still providing those general powers I describe. In GetFile itself we of course can put guardrails around in the form of various directories to not allow capture/delete from. But that does get difficult with portability across various systems/sim-links/etc..
For your specific list:
* Using GetFile to copy the NiFi configuration files and keystores into flows.
* Creating flows which contain python scripts that utilize the keys/secrets in the nifi configuration files to decrypt the contents of the keystores.
* Executing those flows to gain access to the private keys used to encrypt data in NiFi
* Using that data to gain unauthorized access to the cluster to further compromise it, possibly by generating their own admin-level user certificates.
You need to take advantage of NiFi's authorization model and concept of restricted processors. In the end there will be someone you trust/authorize to use these things and indeed those individuals can do some really bad things. Again, that is an authorized user being allowed to manipulate the system. If you don't want any users, or certain users to have the ability, then restrict them. See [1] [2] [3] and the example specific to GetFile as James noted in [4] and commentary around restricted components here [5]
[1] http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#restricted
[2] http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Restricted_Components_in_Versioned_Flows
[3] http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#UI-with-multi-tenant-authorization
[4] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.GetFile/index.html
[5] http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html
Your concerns are understandable. But we have a pretty strong set of controls for you to use once you better understand the model/authorization/restrictions and audits in place and so on. This is solvable such that you can control who has authorization to do danger - those actions are also audited.
Thanks
Joe
On Fri, Jun 4, 2021 at 1:18 PM Rodriguez, Matthew <Ma...@bestbuy.com>> wrote:
GetFile is considered a restricted component, when running in secure mode like you are I believe you have to explicitly give users/groups permissions to use restricted components. If your users don’t have a valid use case for those components, or you desire to protect yourself from that scenario you listed, don’t grant them access to use restricted components. All restricted components are called out in the documentation or the UI, they also state which specific restrictions they need if you want to enable the use of only some restricted components.
Permissions to these components are managed globally in the policies portion of the hamburger menu.
[Graphical user interface, text, website Description automatically generated]
[Graphical user interface, application Description automatically generated]
From: Ruth, Thomas <th...@optum.com>>
Date: Friday, June 4, 2021 at 3:11 PM
To: users@nifi.apache.org<ma...@nifi.apache.org> <us...@nifi.apache.org>>
Subject: [CAUTION! EXTERNAL] RE: Problem with the GetFile processor deleting my entire installation
⚠ This message is from an external sender and could be a phish. ⚠
Slow down, read carefully and look for signs that it may be a phish. If you think it’s malicious, click the report phish button or forward this email to phishing@bestbuy. com.
No matter how secure I make the cluster, what’s to prevent a user from:
* Using GetFile to copy the NiFi configuration files and keystores into flows.
* Creating flows which contain python scripts that utilize the keys/secrets in the nifi configuration files to decrypt the contents of the keystores.
* Executing those flows to gain access to the private keys used to encrypt data in NiFi
* Using that data to gain unauthorized access to the cluster to further compromise it, possibly by generating their own admin-level user certificates.
I’m genuinely stumped here as to how to create a solution that is production ready and that prevents a complete system failure by a user accidentally supplying the wrong directory to a processor component or generally creating python scripts that execute as the nifi user, being able to basically use a NiFi server as their own script-running playground.
Tom
From: Mike Sofen <ms...@runbox.com>>
Sent: Friday, June 4, 2021 2:02 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: Problem with the GetFile processor deleting my entire installation
Setting aside the user auth/permissions issue (which sounds like, from your initial email, that you nailed perfectly with a secured cluster), I’ll address this accidental deletion issue using what I consider a general design pattern for Nifi (or any process that write-touches files):
* I don’t allow physical file deletes (which happen with the default GetFiles settings)
* I create an archive folder for files I’ve processed if I want/need to move them out of the processing folder
* I always set the “leave files” flag
* I use either the timestamp or filename tracker to eliminate repeated processing if I’m not moving the files
* OR – I use Move File flag to the archive folder destination.
* And I advise new users to always spec a small test folder to start, and to double-check that the remove files is turned off.
In some cases, like when I’m processing text files in place in secure repo, I can’t move the files so I use the timestamp eval method.
In others, like log files where I definitely want to archive them out of that processing folder, I use the move model.
Don’t get discouraged about this twist in the road – you’ll find Nifi to be a truly exceptional product for pipeline automation, wildly more powerful/flexible/stable than any other product out there (I’ve used most). I’ve used it IoT to DB to ML processing, document processing, metrology data processing, you name it. In particular, the built data provenance and auditability will be valuable for your situation at Optum. All the best,
Mike
From: Ruth, Thomas <th...@optum.com>>
Sent: Friday, June 04, 2021 12:26 PM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: Problem with the GetFile processor deleting my entire installation
Is it recommended that after installing NiFi, that I then proceed to remove read permissions from all installation files and directories in order to protect them from removal by users? Will this present a problem running the software if it’s unable to read any files?
So for fun, I loaded up a quick nifi container:
docker run –name nifi -p 8080:8080 -d apache/nifi:latest
Connected to localhost:8080/nifi
Created a GetFile processor, with the directory /opt/nifi. Connected the success to a LogAttributes. Hit Run….
Now I get this:
HTTP ERROR 404 /nifi/canvas.jsp
URI:
/nifi/
STATUS:
404
MESSAGE:
/nifi/canvas.jsp
SERVLET:
jsp
This behavior shouldn’t be possible, in my opinion. It’s putting security in the hands of my developers. I really am looking for a solution to this issue.
The help text says this:
Creates FlowFiles from files in a directory. NiFi will ignore files it doesn't have at least read permissions for.
So as you suggested, I removed the read permission recursively from all files in /opt/nifi. After doing this, nifi no longer starts.
find /opt/nifi -print > /tmp/files.txt
for I in `cat /tmp/files.txt`; do chmod a-r $i; done
I also went to https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator<https://urldefense.com/v3/__https:/nvd.nist.gov/vuln-metrics/cvss/v3-calculator__;!!KtbpKd1p8A!4aX8XRjZKKreeGNa5t2R6_MwRpUiVieStkOv-UX-JszUHN60ElFsBxLNBQ0NeSC3PXMdYRlBXQ$> and attempted to calculate a CVSS score for this vulnerability. I ended up calculating a score of 8.0.
Tom
From: Russell Bateman <ru...@windofkeltia.com>>
Sent: Friday, June 4, 2021 11:20 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: Problem with the GetFile processor deleting my entire installation
Oh, sorry, to finish the answer, yes, you do need to be very careful how you specify the Input Directory and File Filter properties; the last one is a regular expression. It's true that the documentation is less than flag-waving or hair-lighting-on-fire as it presents its help on filling those in.
Russ
On 6/4/21 11:16 AM, Russell Bateman wrote:
Sorry for this behavior of GetFile which is purposeful. If you configure to keep the files instead of removing them, you'll keep getting the same files ingested over and over again as flow files. It's just how it is.
The secret was to read the help blurb when configuring this processor.
Hope this helps,
Russ
On 6/4/21 10:44 AM, Ruth, Thomas wrote:
Hello,
I recently built a 3-node NiFi cluster in my organization as a proof-of-concept for some work we are doing. I used version 1.13.2 and installed it onto 3 CentOS 7.9 systems. In my organization, I don’t have root access to the system, so I used a different user called “nfadm” to install and run the product. I don’t remember seeing anything in the documentation that stated that this would be an issue.
I am also new to NiFi, and was relying heavily on the Admin documentation on the web site for instructions to set up the OS and NiFi installations. I configured certificate-based security and distributed them to my users. I also configured policies for groups that I thought were OK for them from a development standpoint.
I had an incident occur yesterday in which a user, who is also new to NiFi, ran a component called “GetFile” for the filesystem “/” with the default settings (Recurse=true, KeepSourceFile=false). Well, this essentially ran “rm -rf /” as the user that owns all the installation files and files in the various repositories, nfadm, the same user running the NiFi processes. This deleted all the installation and configuration files for the entire cluster, making it completely useless now.
I am surprised to find out that NiFi allowed a user to basically wipe out all the files the user running the NiFi server had access to. I would expect much higher security to be present in a default system. I have some questions that hopefully you can help me with:
Is this a known issue in NiFi?
Am I doing something wrong when configuring or installing NiFi?
Is there a section in the documentation that warns me of this type of scenario?
Thanks in advance for your help with this,
Tom Ruth
Sr. Data Engineer
Optum, Inc.
E: Thomas.Ruth@Optum.com<ma...@Optum.com>
This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.
This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.
This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.
This e-mail, including attachments, may include confidential and/or
proprietary information, and may be used only by the person or entity
to which it is addressed. If the reader of this e-mail is not the intended
recipient or his or her authorized agent, the reader is hereby notified
that any dissemination, distribution or copying of this e-mail is
prohibited. If you have received this e-mail in error, please notify the
sender by replying to this message and delete this e-mail immediately.
Re: [CAUTION! EXTERNAL] RE: Problem with the GetFile processor
deleting my entire installation
Posted by Joe Witt <jo...@gmail.com>.
You've already gotten some great responses here. But to add...
A secured NiFi instance allows you to fully control what a given authorized
user is allowed to do at very granular levels. For instance the
administrator gets to choose whether user A can use/modify processors that
manipulate/interact with the file system. That addresses very well your
statement about developers should not be able to do such things. We
agree. That feature is critical. So then there is the issue of what to do
about authorized users having powerful tools at their disposal. For
authorized users they can write processors with custom code on the fly,
they can manipulate the file system, etc.. There is no mechanism by which
we can fully protect authorized users from doing certain problematic things
(like deleting nifi itself) while still providing those general powers I
describe. In GetFile itself we of course can put guardrails around in the
form of various directories to not allow capture/delete from. But that
does get difficult with portability across various systems/sim-links/etc..
For your specific list:
- Using GetFile to copy the NiFi configuration files and keystores into
flows.
- Creating flows which contain python scripts that utilize the
keys/secrets in the nifi configuration files to decrypt the contents of the
keystores.
- Executing those flows to gain access to the private keys used to
encrypt data in NiFi
- Using that data to gain unauthorized access to the cluster to further
compromise it, possibly by generating their own admin-level user
certificates.
You need to take advantage of NiFi's authorization model and concept of
restricted processors. In the end there will be someone you
trust/authorize to use these things and indeed those individuals can do
some really bad things. Again, that is an authorized user being allowed to
manipulate the system. If you don't want any users, or certain users to
have the ability, then restrict them. See [1] [2] [3] and the example
specific to GetFile as James noted in [4] and commentary around restricted
components here [5]
[1]
http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#restricted
[2]
http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Restricted_Components_in_Versioned_Flows
[3]
http://nifi.apache.org/docs/nifi-docs/html/user-guide.html#UI-with-multi-tenant-authorization
[4]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.GetFile/index.html
[5] http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html
Your concerns are understandable. But we have a pretty strong set of
controls for you to use once you better understand the
model/authorization/restrictions and audits in place and so on. This is
solvable such that you can control who has authorization to do danger -
those actions are also audited.
Thanks
Joe
On Fri, Jun 4, 2021 at 1:18 PM Rodriguez, Matthew <
Matthew.Rodriguez2@bestbuy.com> wrote:
> GetFile is considered a restricted component, when running in secure mode
> like you are I believe you have to explicitly give users/groups permissions
> to use restricted components. If your users don’t have a valid use case for
> those components, or you desire to protect yourself from that scenario you
> listed, don’t grant them access to use restricted components. All
> restricted components are called out in the documentation or the UI, they
> also state which specific restrictions they need if you want to enable the
> use of only some restricted components.
>
>
>
> Permissions to these components are managed globally in the policies
> portion of the hamburger menu.
>
>
>
> *[image: Graphical user interface, text, website Description automatically
> generated]*
>
>
>
> *[image: Graphical user interface, application Description automatically
> generated]*
>
>
>
>
>
> *From: *Ruth, Thomas <th...@optum.com>
> *Date: *Friday, June 4, 2021 at 3:11 PM
> *To: *users@nifi.apache.org <us...@nifi.apache.org>
> *Subject: *[CAUTION! EXTERNAL] RE: Problem with the GetFile processor
> deleting my entire installation
>
>
>
> ⚠ *This message is from an external sender and could be a phish.* ⚠
>
> Slow down, read carefully and look for signs that it may be a phish. If
> you think it’s malicious, click the report phish button or forward this
> email to phishing@bestbuy. com.
>
>
>
>
>
> No matter how secure I make the cluster, what’s to prevent a user from:
>
> - Using GetFile to copy the NiFi configuration files and keystores
> into flows.
> - Creating flows which contain python scripts that utilize the
> keys/secrets in the nifi configuration files to decrypt the contents of the
> keystores.
> - Executing those flows to gain access to the private keys used to
> encrypt data in NiFi
> - Using that data to gain unauthorized access to the cluster to
> further compromise it, possibly by generating their own admin-level user
> certificates.
>
>
>
> I’m genuinely stumped here as to how to create a solution that is
> production ready and that prevents a complete system failure by a user
> accidentally supplying the wrong directory to a processor component or
> generally creating python scripts that execute as the nifi user, being able
> to basically use a NiFi server as their own script-running playground.
>
>
>
> Tom
>
>
>
> *From:* Mike Sofen <ms...@runbox.com>
> *Sent:* Friday, June 4, 2021 2:02 PM
> *To:* users@nifi.apache.org
> *Subject:* RE: Problem with the GetFile processor deleting my entire
> installation
>
>
>
> Setting aside the user auth/permissions issue (which sounds like, from
> your initial email, that you nailed perfectly with a secured cluster), I’ll
> address this accidental deletion issue using what I consider a general
> design pattern for Nifi (or any process that write-touches files):
>
> - I don’t allow physical file deletes (which happen with the default
> GetFiles settings)
> - I create an archive folder for files I’ve processed if I want/need
> to move them out of the processing folder
> - I always set the “leave files” flag
> - I use either the timestamp or filename tracker to eliminate repeated
> processing if I’m not moving the files
> - OR – I use Move File flag to the archive folder destination.
> - And I advise new users to always spec a small test folder to start,
> and to double-check that the remove files is turned off.
>
>
>
> In some cases, like when I’m processing text files in place in secure
> repo, I can’t move the files so I use the timestamp eval method.
>
>
>
> In others, like log files where I definitely want to archive them out of
> that processing folder, I use the move model.
>
>
>
> Don’t get discouraged about this twist in the road – you’ll find Nifi to
> be a truly exceptional product for pipeline automation, wildly more
> powerful/flexible/stable than any other product out there (I’ve used
> most). I’ve used it IoT to DB to ML processing, document processing,
> metrology data processing, you name it. In particular, the built data
> provenance and auditability will be valuable for your situation at Optum.
> All the best,
>
>
>
> Mike
>
>
>
> *From:* Ruth, Thomas <th...@optum.com>
> *Sent:* Friday, June 04, 2021 12:26 PM
> *To:* users@nifi.apache.org
> *Subject:* RE: Problem with the GetFile processor deleting my entire
> installation
>
>
>
> Is it recommended that after installing NiFi, that I then proceed to
> remove read permissions from all installation files and directories in
> order to protect them from removal by users? Will this present a problem
> running the software if it’s unable to read any files?
>
>
>
> So for fun, I loaded up a quick nifi container:
> docker run –name nifi -p 8080:8080 -d apache/nifi:latest
>
>
>
> Connected to localhost:8080/nifi
>
>
>
> Created a GetFile processor, with the directory /opt/nifi. Connected the
> success to a LogAttributes. Hit Run….
>
>
>
> Now I get this:
> *HTTP ERROR 404 /nifi/canvas.jsp*
>
> *URI:*
>
> /nifi/
>
> *STATUS:*
>
> 404
>
> *MESSAGE:*
>
> /nifi/canvas.jsp
>
> *SERVLET:*
>
> jsp
>
>
>
> This behavior shouldn’t be possible, in my opinion. It’s putting security
> in the hands of my developers. I really am looking for a solution to this
> issue.
>
>
>
> The help text says this:
> Creates FlowFiles from files in a directory. NiFi will ignore files it
> doesn't have at least read permissions for.
>
>
>
> So as you suggested, I removed the read permission recursively from all
> files in /opt/nifi. After doing this, nifi no longer starts.
>
>
>
> find /opt/nifi -print > /tmp/files.txt
>
> for I in `cat /tmp/files.txt`; do chmod a-r $i; done
>
>
>
> I also went to https://nvd.nist.gov/vuln-metrics/cvss/v3-calculator
> <https://urldefense.com/v3/__https:/nvd.nist.gov/vuln-metrics/cvss/v3-calculator__;!!KtbpKd1p8A!4aX8XRjZKKreeGNa5t2R6_MwRpUiVieStkOv-UX-JszUHN60ElFsBxLNBQ0NeSC3PXMdYRlBXQ$>
> and attempted to calculate a CVSS score for this vulnerability. I ended up
> calculating a score of 8.0.
>
>
>
> Tom
>
>
>
> *From:* Russell Bateman <ru...@windofkeltia.com>
> *Sent:* Friday, June 4, 2021 11:20 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: Problem with the GetFile processor deleting my entire
> installation
>
>
>
> Oh, sorry, to finish the answer, yes, you do need to be *very* careful
> how you specify the Input Directory and File Filter properties; the last
> one is a regular expression. It's true that the documentation is less than
> flag-waving or hair-lighting-on-fire as it presents its help on filling
> those in.
>
> Russ
>
> On 6/4/21 11:16 AM, Russell Bateman wrote:
>
> Sorry for this behavior of *GetFile* which is purposeful. If you
> configure to keep the files instead of removing them, you'll keep getting
> the same files ingested over and over again as flow files. It's just how it
> is.
>
> The secret was to read the help blurb when configuring this processor.
>
> Hope this helps,
>
> Russ
>
> On 6/4/21 10:44 AM, Ruth, Thomas wrote:
>
> Hello,
>
>
>
> I recently built a 3-node NiFi cluster in my organization as a
> proof-of-concept for some work we are doing. I used version 1.13.2 and
> installed it onto 3 CentOS 7.9 systems. In my organization, I don’t have
> root access to the system, so I used a different user called “nfadm” to
> install and run the product. I don’t remember seeing anything in the
> documentation that stated that this would be an issue.
>
>
>
> I am also new to NiFi, and was relying heavily on the Admin documentation
> on the web site for instructions to set up the OS and NiFi installations. I
> configured certificate-based security and distributed them to my users. I
> also configured policies for groups that I thought were OK for them from a
> development standpoint.
>
>
>
> I had an incident occur yesterday in which a user, who is also new to
> NiFi, ran a component called “GetFile” for the filesystem “/” with the
> default settings (Recurse=true, KeepSourceFile=false). Well, this
> essentially ran “rm -rf /” as the user that owns all the installation files
> and files in the various repositories, nfadm, the same user running the
> NiFi processes. This deleted all the installation and configuration files
> for the entire cluster, making it completely useless now.
>
>
>
> I am surprised to find out that NiFi allowed a user to basically wipe out
> all the files the user running the NiFi server had access to. I would
> expect much higher security to be present in a default system. I have some
> questions that hopefully you can help me with:
>
>
>
> Is this a known issue in NiFi?
>
>
>
> Am I doing something wrong when configuring or installing NiFi?
>
>
>
> Is there a section in the documentation that warns me of this type of
> scenario?
>
>
>
> Thanks in advance for your help with this,
>
>
>
> Tom Ruth
>
> Sr. Data Engineer
>
> Optum, Inc.
>
> E: Thomas.Ruth@Optum.com
>
>
>
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
>
>
>
>
>
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
>
>
> This e-mail, including attachments, may include confidential and/or
> proprietary information, and may be used only by the person or entity
> to which it is addressed. If the reader of this e-mail is not the intended
> recipient or his or her authorized agent, the reader is hereby notified
> that any dissemination, distribution or copying of this e-mail is
> prohibited. If you have received this e-mail in error, please notify the
> sender by replying to this message and delete this e-mail immediately.
>