You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Tiago Luís Sebastião (DSI)" <ti...@cgd.pt> on 2022/12/06 16:23:40 UTC
RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi Isha,
I had already tried that and didn’t work also.
But I tried the last thing I wrote on the previous email and it worked… Not happy with this solution but the issue (encryption/warnings) is solved I guess.
Having Nifi.properties:
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
With nifi Stopped:
1. Encrypt new files on the side:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f /apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n /apps/nifi-1.18.0/conf/nifi.properties -o /apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s PasswordUsedOnNifiProperties -x -v
2. Backup the flow.json.gz and rename the new one:
mv /apps/nifi-1.18.0/conf/flow.json.gz /apps/nifi-1.18.0/conf/flow_bk202212.json.gz
mv /apps/nifi-1.18.0/conf/flow2.json.gz /apps/nifi-1.18.0/conf/flow.json.gz
3. Change nifi.properties file:
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
to
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
4. Start nifi
No errors found and no warnings…
Tiago Sebastião
Tlf: 218456542 | Ext. 556542 | tiago.luis.sebastiao@cgd.pt<ma...@cgd.pt>
[cid:image002.jpg@01D52816.782219F0]
DSI - Data Office – USI6.2 Unidade de Serviços de Dados
www.cgd.pt<http://www.cgd.pt/>
Antes de imprimir esta mensagem, pense no meio ambiente.
Esta mensagem e-mail, assim como os ficheiros eventualmente anexos, é reservada aos seus destinatários, e pode conter informação confidencial ou estar sujeita a restrições legais. Se não é o seu destinatário ou se recebeu esta mensagem por motivo de erro, solicitamos que não faça qualquer uso ou divulgação do seu conteúdo e proceda à eliminação permanente desta mensagem e respetivos anexos.
Caixa Geral de Depósitos, S.A. | Sede Social: Av. João XXI, 63, 1000-300 LISBOA | Capital Social 3.844.143.735,00 € | CRCL e Contribuinte 500 960 046
From: Isha Lamboo [mailto:isha.lamboo@virtualsciences.nl]
Sent: 25 de novembro de 2022 09:48
To: users@nifi.apache.org
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***
________________________________
Hi Tiago,
I’ve had a similar experience with migrating the flow encryption algorithm and in fact, some of them are still on the old one. The nifi.sh commands to update the sensitive properties key and algorithm are very tricky to use, because they update the nifi.properties file even if the migration fails for the flow.xml.gz and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following ones to fail because it tried to decrypt using the new algorithm. I needed to reset the nifi.properties file everytime.
Another thing I’ve noticed is that it doesn’t support the property file protection scheme that NiFi has. If your sensitive props key is encrypted you need to enter the raw value and make sure the nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.
These steps have mostly worked for me:
1. Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another dir
2. Unprotect the sensitive properties key:
* Replace the encrypted key with the raw one
* Empty this property: nifi.sensitive.props.key.protected
1. Check that the algorithm is still the old one: nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
2. Check the length of the raw key, it needs to be 12 characters or longer to migrate to the new algorithm.
3. If the key is empty or short:
* ./bin/nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties (run from the main dir)
* Check the output for any failures and if they occur, revert the nifi.properties file and fix any errors
* Try to start nifi with the new sensitive properties key
1. If it works, stop nifi and update the algorithm:
* ./bin/nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
* Check the output for any failures and if they occur, revert the nifi.properties file and fix any errors
* Try to start nifi with the new algorithm
1. Stop nifi, encrypt the sensitive properties key (or restore it if you didn’t need to change the password)
* Use the encrypt-config.sh from the nifi-toolkit, specify output files so you can check and compare.
* Beware: this tool doesn’t seem to know about flow.json.gz, so only use it to change the config files.
I hope this helps you find the solution.
Regards,
Isha
Van: Tiago Luís Sebastião (DSI) <ti...@cgd.pt>>
Verzonden: donderdag 24 november 2022 16:19
Aan: users@nifi.apache.org<ma...@nifi.apache.org>
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi again,
Sorry for not following up but other priorities came ahead…
Basically it’s still not working, I’ve tried several combinations and I still keep getting:
“Failed to process Flow Configuration [./conf/flow.xml.gz]
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [AES/GCM/NoPadding]”
After reading some documentation, for this purpose, assuming that the password configured in the nifi.properties file is “PasswordUsedOnNifiProperties”…
I’ve tried and failed:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties
I’ve tried and failed by setting algorithm to empty string:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties
I’ve tried and failed using the new toolkit (I was using toolkit version 1.13.3):
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f /apps/nifi-1.18.0/conf/flow.xml.gz -n /apps/nifi-1.18.0/conf/nifi.properties -s PasswordUsedOnNifiProperties -A NIFI_ARGON2_AES_GCM_256 -x -v
I’ve tried and failed doing the same but generating new files to debug:
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f /apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n /apps/nifi-1.18.0/conf/nifi.properties -o /apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s PasswordUsedOnNifiProperties -x -v
In this last one I noticed that the file flow2.json.gz got its passwords encrypted differently and a longer encrypt also.
I’m kind of wondering if I can use this last command to generate these files on the side and then manipulate the nifi.properties file by changing the algorithm to NIFI_PBKDF2_AES_GCM_256 since it seems it is already encrypted despite the known errors/warnings, then I would rename these new files to the older ones and start nifi with:
1. New flow.json.gz file (apparently encrypted with NIFI_PBKDF2_AES_GCM_256 algorithm)
2. New nifi.properties file (with nifi.sensitive.props.algorithm property manipulated to NIFI_PBKDF2_AES_GCM_256)
Since, unfortunately<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Fsearch%3Fsxsrf%3DALiCzsYtztPHersBtg21lqlpGlc7DZ9CUw%3A1669302812266%26q%3Dunfortunately%26spell%3D1%26sa%3DX%26ved%3D2ahUKEwiq4pbJjcf7AhXKzqQKHfK8DNwQkeECKAB6BAgGEAE&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C14fdf6863d254d1163dd08dace2f467c%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638049003651712131%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=QkSjVNecwZDyUuzgbK4F9n4DneE8e5vD81BjfHtf224%3D&reserved=0>, Im getting nowhere with this and I need to migrate to version 1.18.0 in order to apply the bugfix that changes the serverConnectorFactory.setNeedClientAuth(wantClientAuth) to serverConnectorFactory.setWantClientAuth(wantClientAuth) I am needing help in a consequent situation.
In order to reduce the size of the log generated from the deprecation warnings (WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] generate salt requested) I’ve tried to LEVEL OFF that Warning from stateless-logback.xml file without success.
On the tag <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> and <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender"> I changed the tag <pattern> so that I could see the full class name (although without success also…)
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
to
<pattern>%date %level [%thread] %logger{140} %msg%n</pattern>
On the stateless-logback.xml I inserted the following:
<logger name="org.apache.nifi" level="INFO"/>
<logger name="deprecation.org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider" level="OFF" />
It’s not working and I don’t understand why, the class name seems to be correct but I keep getting the same WARN.
Sorry for the long email…
Regards.
Tiago Sebastião
From: Tiago Luís Sebastião (DSI)
Sent: 28 de outubro de 2022 09:48
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi David,
It’s a standalone deployment and runs directly on the server.
Yes the command updated the flow.xml.gz/flow.json.gz and nifi.properties settings.
Maybe I messed up the nifi.sensitive.props.key, I’ll run some more tests.
Thanks for your help.
Tiago
From: David Handermann [mailto:exceptionfactory@apache.org]
Sent: 27 de outubro de 2022 16:50
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi Tiago,
The initial warning for the Insecure Cipher Provider Algorithm indicates the use of the deprecated setting as mentioned previously.
The set-sensitive-properties-algorithm command looks correct, and should have updated the flow.xml.gz, flow.json.gz, and nifi.properties settings.
The Decryption Failed message indicates that the nifi.sensitive.props.key value does not match the value used to encrypt the flow configuration, or that the algorithm does not match.
Can you provide some additional details about the NiFi installation? Is this a standalone or clustered deployment, and is it running in a containerized environment, or directly on a server?
Regards,
David Handermann
On Thu, Oct 27, 2022 at 10:35 AM Tiago Luís Sebastião (DSI) <ti...@cgd.pt>> wrote:
Hi all,
I'm having the same “problem”.
I upgraded nifi version from 1.17.0 to 1.18.0 and that same warning started to appear 500k times a day.
"
WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] generate salt requested
"
A already had nifi.sensitive.props.key value defined from when we migrated to 1.15.3.
With Nifi STOPPED and without changing any configuration on nifi.properties I executed the following:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
No errors found there, then I started Nifi and received the following errors:
"
WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [AES/GCM/NoPadding]
"
Since Nifi could not start anymore I reversed it...
Now Im kind of stuck with this warning...
Anyone knows what Im doing wrong?
Tiago
From: David Handermann [mailto:exceptionfactory@apache.org<ma...@apache.org>]
Sent: 19 de outubro de 2022 13:41
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi Mike,
The deprecation warning is not related to NIFI-10567 or Sensitive Dynamic Properties.
Deprecation logging is a new feature added in NiFi 1.18.0 to highlight components and features that are targeted for removal in future major releases. The current administrator's guide has more details on deprecation logging. [1] Deprecation warnings do not impact operational behavior, but they do identify configuration settings that should be changed.
In this particular case, the deprecation is related to the use of the insecure algorithm. NiFi 1.14.0 and following introduced new Sensitive Properties Key Algorithm settings, which should be used instead of the historical default value indicated in the warning. The new default value is NIFI_PBKDF2_AES_GCM_256, additional supported options are listed in the administrator's guide, [2] along with the command that can be run to update the Sensitive Properties Key Algorithm. [3]
Feel free to follow up if you have additional questions.
Regards,
David Handermann
[1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#deprecation-logging<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23deprecation-logging&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C14fdf6863d254d1163dd08dace2f467c%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638049003651712131%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=e3TvRGGVBqr3SufZI4QqwBMIZcjrZYfatDeSG1BgBlw%3D&reserved=0>
[2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#property-encryption-algorithms<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23property-encryption-algorithms&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C14fdf6863d254d1163dd08dace2f467c%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638049003651712131%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=l9DP6913pDQq1wOp0x00iO982WpN21hdwB%2FjnlzKhhk%3D&reserved=0>
[3] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#updating-the-sensitive-properties-algorithm<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23updating-the-sensitive-properties-algorithm&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C14fdf6863d254d1163dd08dace2f467c%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638049003651712131%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YfPuNMaiCCRNLz4eHiUO%2BvN7l2rh%2B7Ce9yCdfOFn%2B3w%3D&reserved=0>
On Wed, Oct 19, 2022 at 7:28 AM Mike S <88...@gmail.com>> wrote:
I upgraded from 1.16.2 to 1.18.0 and now see this warning in the log file.
WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] generate salt requested
org.apache.nifi.deprecation.log.DeprecationException: Reference Class [org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider] ClassLoader [org.apache.nifi.nar.NarClassLoader[./work/nar/framework/nifi-framework-nar-1.18.0.nar-unpacked]]
I read this here.
NIFI-10567<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-10567&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C14fdf6863d254d1163dd08dace2f467c%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638049003651712131%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=vyvgkBNyto%2Bt7YdHFPugG3kRP9QLGA%2BlCCsQVWE1Gm0%3D&reserved=0> Corrects the parsing of Sensitive Dynamic Properties read from the XML version of the flow configuration, in absence of the JSON version.
The issue surfaces when upgrading to NiFi 1.17.0 or 1.18.0 from a version older than 1.16.0. The issue also requires the presence of a Parameter Context with a Sensitive value assigned to a component with a Sensitive Property. Upgrading from 1.16.0 and following is not a problem.
It appears that all my ListS3 processors using sensitive properties are working.
Is this related since 1.16.2 has the latest flow.json.gz file?
Mike
Re: NiFi 1.18.0 Sensitive Property broken after Upgrade
Posted by Mark Payne <ma...@hotmail.com>.
I’ve not followed everything goin on in this thread. But just to offer a bit of clarification: If the flow.json.gz exists then the flow.xml.gz is ignored. But if the flow.json.gz file is not found, it will automatically fall back to the flow.xml.gz
So if you were to remove/rename the flow.json.gz it would definitely pick up the XML file and process that. Otherwise, it will ignore it.
Thanks
-Mark
On Dec 7, 2022, at 4:24 AM, Tiago Luís Sebastião (DSI) <ti...@cgd.pt> wrote:
Hi Isha,
I did not delete the flow.xml.gz because I think that since the version 1.16.0, if I’m not mistaken, when the flow.json.xml was released it replaced completely the flow.xml.gz, so any changes made on flow.xml.gz wouldn’t make any difference, so knowing that, I assumed there shouldn’t be anything configured/dependent on the xml file. Also, the error didn’t seem related at all to the flow, it seemed related to not being able to decrypt the flow.json.gz with the old algorithm and encrypt with the new algorithm.
Thanks for the help.
Regards.
Tiago Sebastião
From: Isha Lamboo [mailto:isha.lamboo@virtualsciences.nl]
Sent: 7 de dezembro de 2022 08:42
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***
________________________________
Hi Tiago,
Thanks for updating and sharing your solution.
I noticed you only encrypted the flow.json.gz file. Did you also delete the xml version in previous attempts?
It makes me wonder if the migrating fails for the flow.xml.gz specifically, then deleting the xml during migration might be an easier fix.
Regards,
Isha
Van: Tiago Luís Sebastião (DSI) <ti...@cgd.pt>>
Verzonden: dinsdag 6 december 2022 17:24
Aan: users@nifi.apache.org<ma...@nifi.apache.org>
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi Isha,
I had already tried that and didn’t work also.
But I tried the last thing I wrote on the previous email and it worked… Not happy with this solution but the issue (encryption/warnings) is solved I guess.
Having Nifi.properties:
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
With nifi Stopped:
1. Encrypt new files on the side:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f /apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n /apps/nifi-1.18.0/conf/nifi.properties -o /apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -sPasswordUsedOnNifiProperties -x -v
1. Backup the flow.json.gz and rename the new one:
mv /apps/nifi-1.18.0/conf/flow.json.gz /apps/nifi-1.18.0/conf/flow_bk202212.json.gz
mv /apps/nifi-1.18.0/conf/flow2.json.gz /apps/nifi-1.18.0/conf/flow.json.gz
1. Change nifi.properties file:
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
to
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
1. Start nifi
No errors found and no warnings…
Tiago Sebastião
From: Isha Lamboo [mailto:isha.lamboo@virtualsciences.nl]
Sent: 25 de novembro de 2022 09:48
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***
________________________________
Hi Tiago,
I’ve had a similar experience with migrating the flow encryption algorithm and in fact, some of them are still on the old one. The nifi.sh commands to update the sensitive properties key and algorithm are very tricky to use, because they update the nifi.properties file even if the migration fails for the flow.xml.gz and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following ones to fail because it tried to decrypt using the new algorithm. I needed to reset the nifi.properties file everytime.
Another thing I’ve noticed is that it doesn’t support the property file protection scheme that NiFi has. If your sensitive props key is encrypted you need to enter the raw value and make sure the nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.
These steps have mostly worked for me:
1. Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another dir
2. Unprotect the sensitive properties key:
* Replace the encrypted key with the raw one
* Empty this property: nifi.sensitive.props.key.protected
1. Check that the algorithm is still the old one: nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
2. Check the length of the raw key, it needs to be 12 characters or longer to migrate to the new algorithm.
3. If the key is empty or short:
* ./bin/nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties (run from the main dir)
* Check the output for any failures and if they occur, revert the nifi.properties file and fix any errors
* Try to start nifi with the new sensitive properties key
1. If it works, stop nifi and update the algorithm:
* ./bin/nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
* Check the output for any failures and if they occur, revert the nifi.properties file and fix any errors
* Try to start nifi with the new algorithm
1. Stop nifi, encrypt the sensitive properties key (or restore it if you didn’t need to change the password)
* Use the encrypt-config.sh from the nifi-toolkit, specify output files so you can check and compare.
* Beware: this tool doesn’t seem to know about flow.json.gz, so only use it to change the config files.
I hope this helps you find the solution.
Regards,
Isha
Van: Tiago Luís Sebastião (DSI) <ti...@cgd.pt>>
Verzonden: donderdag 24 november 2022 16:19
Aan: users@nifi.apache.org<ma...@nifi.apache.org>
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi again,
Sorry for not following up but other priorities came ahead…
Basically it’s still not working, I’ve tried several combinations and I still keep getting:
“Failed to process Flow Configuration [./conf/flow.xml.gz]
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [AES/GCM/NoPadding]”
After reading some documentation, for this purpose, assuming that the password configured in the nifi.properties file is “PasswordUsedOnNifiProperties”…
I’ve tried and failed:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties
I’ve tried and failed by setting algorithm to empty string:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties
I’ve tried and failed using the new toolkit (I was using toolkit version 1.13.3):
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f /apps/nifi-1.18.0/conf/flow.xml.gz -n /apps/nifi-1.18.0/conf/nifi.properties -s PasswordUsedOnNifiProperties -A NIFI_ARGON2_AES_GCM_256 -x -v
I’ve tried and failed doing the same but generating new files to debug:
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f /apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n /apps/nifi-1.18.0/conf/nifi.properties -o /apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s PasswordUsedOnNifiProperties -x -v
In this last one I noticed that the file flow2.json.gz got its passwords encrypted differently and a longer encrypt also.
I’m kind of wondering if I can use this last command to generate these files on the side and then manipulate the nifi.properties file by changing the algorithm to NIFI_PBKDF2_AES_GCM_256 since it seems it is already encrypted despite the known errors/warnings, then I would rename these new files to the older ones and start nifi with:
1. New flow.json.gz file (apparently encrypted with NIFI_PBKDF2_AES_GCM_256 algorithm)
2. New nifi.properties file (with nifi.sensitive.props.algorithm property manipulated to NIFI_PBKDF2_AES_GCM_256)
Since, unfortunately<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Fsearch%3Fsxsrf%3DALiCzsYtztPHersBtg21lqlpGlc7DZ9CUw%3A1669302812266%26q%3Dunfortunately%26spell%3D1%26sa%3DX%26ved%3D2ahUKEwiq4pbJjcf7AhXKzqQKHfK8DNwQkeECKAB6BAgGEAE&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3sFUfMRIjK%2BMvhhh8dKaNEXgpua8C0E5Cqed3pWjrww%3D&reserved=0>, Im getting nowhere with this and I need to migrate to version 1.18.0 in order to apply the bugfix that changes the serverConnectorFactory.setNeedClientAuth(wantClientAuth) to serverConnectorFactory.setWantClientAuth(wantClientAuth) I am needing help in a consequent situation.
In order to reduce the size of the log generated from the deprecation warnings (WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] generate salt requested) I’ve tried to LEVEL OFF that Warning from stateless-logback.xml file without success.
On the tag <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> and <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender"> I changed the tag <pattern> so that I could see the full class name (although without success also…)
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
to
<pattern>%date %level [%thread] %logger{140} %msg%n</pattern>
On the stateless-logback.xml I inserted the following:
<logger name="org.apache.nifi" level="INFO"/>
<logger name="deprecation.org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider" level="OFF" />
It’s not working and I don’t understand why, the class name seems to be correct but I keep getting the same WARN.
Sorry for the long email…
Regards.
Tiago Sebastião
From: Tiago Luís Sebastião (DSI)
Sent: 28 de outubro de 2022 09:48
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi David,
It’s a standalone deployment and runs directly on the server.
Yes the command updated the flow.xml.gz/flow.json.gz and nifi.properties settings.
Maybe I messed up the nifi.sensitive.props.key, I’ll run some more tests.
Thanks for your help.
Tiago
From: David Handermann [mailto:exceptionfactory@apache.org]
Sent: 27 de outubro de 2022 16:50
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi Tiago,
The initial warning for the Insecure Cipher Provider Algorithm indicates the use of the deprecated setting as mentioned previously.
The set-sensitive-properties-algorithm command looks correct, and should have updated the flow.xml.gz, flow.json.gz, and nifi.properties settings.
The Decryption Failed message indicates that the nifi.sensitive.props.key value does not match the value used to encrypt the flow configuration, or that the algorithm does not match.
Can you provide some additional details about the NiFi installation? Is this a standalone or clustered deployment, and is it running in a containerized environment, or directly on a server?
Regards,
David Handermann
On Thu, Oct 27, 2022 at 10:35 AM Tiago Luís Sebastião (DSI) <ti...@cgd.pt>> wrote:
Hi all,
I'm having the same “problem”.
I upgraded nifi version from 1.17.0 to 1.18.0 and that same warning started to appear 500k times a day.
"
WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] generate salt requested
"
A already had nifi.sensitive.props.key value defined from when we migrated to 1.15.3.
With Nifi STOPPED and without changing any configuration on nifi.properties I executed the following:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
No errors found there, then I started Nifi and received the following errors:
"
WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [AES/GCM/NoPadding]
"
Since Nifi could not start anymore I reversed it...
Now Im kind of stuck with this warning...
Anyone knows what Im doing wrong?
Tiago
From: David Handermann [mailto:exceptionfactory@apache.org<ma...@apache.org>]
Sent: 19 de outubro de 2022 13:41
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi Mike,
The deprecation warning is not related to NIFI-10567 or Sensitive Dynamic Properties.
Deprecation logging is a new feature added in NiFi 1.18.0 to highlight components and features that are targeted for removal in future major releases. The current administrator's guide has more details on deprecation logging. [1] Deprecation warnings do not impact operational behavior, but they do identify configuration settings that should be changed.
In this particular case, the deprecation is related to the use of the insecure algorithm. NiFi 1.14.0 and following introduced new Sensitive Properties Key Algorithm settings, which should be used instead of the historical default value indicated in the warning. The new default value is NIFI_PBKDF2_AES_GCM_256, additional supported options are listed in the administrator's guide, [2] along with the command that can be run to update the Sensitive Properties Key Algorithm. [3]
Feel free to follow up if you have additional questions.
Regards,
David Handermann
[1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#deprecation-logging<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23deprecation-logging&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hziWAtka2q2neF9xJhAaApmx1LGfaF%2BgakH8igfLBEA%3D&reserved=0>
[2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#property-encryption-algorithms<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23property-encryption-algorithms&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XG8GjHn6OlNwWMPN6HdKg2NpJUE1XQLo3PY1%2BHY8GVA%3D&reserved=0>
[3] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#updating-the-sensitive-properties-algorithm<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23updating-the-sensitive-properties-algorithm&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=f9QbD7x0STWtvVIgNAhJ%2BXEtQlgDBQ6LS3w%2BWehlYpc%3D&reserved=0>
On Wed, Oct 19, 2022 at 7:28 AM Mike S <88...@gmail.com>> wrote:
I upgraded from 1.16.2 to 1.18.0 and now see this warning in the log file.
WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] generate salt requested
org.apache.nifi.deprecation.log.DeprecationException: Reference Class [org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider] ClassLoader [org.apache.nifi.nar.NarClassLoader[./work/nar/framework/nifi-framework-nar-1.18.0.nar-unpacked]]
I read this here.
NIFI-10567<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-10567&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867354005%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BMM0MYueQm3Yqu7daAhuo7I%2F41uK6O54URVDf7%2F1C9A%3D&reserved=0> Corrects the parsing of Sensitive Dynamic Properties read from the XML version of the flow configuration, in absence of the JSON version.
The issue surfaces when upgrading to NiFi 1.17.0 or 1.18.0 from a version older than 1.16.0. The issue also requires the presence of a Parameter Context with a Sensitive value assigned to a component with a Sensitive Property. Upgrading from 1.16.0 and following is not a problem.
It appears that all my ListS3 processors using sensitive properties are working.
Is this related since 1.16.2 has the latest flow.json.gz file?
Mike
RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Posted by "Tiago Luís Sebastião (DSI)" <ti...@cgd.pt>.
Hi Isha,
I did not delete the flow.xml.gz because I think that since the version 1.16.0, if I’m not mistaken, when the flow.json.xml was released it replaced completely the flow.xml.gz, so any changes made on flow.xml.gz wouldn’t make any difference, so knowing that, I assumed there shouldn’t be anything configured/dependent on the xml file. Also, the error didn’t seem related at all to the flow, it seemed related to not being able to decrypt the flow.json.gz with the old algorithm and encrypt with the new algorithm.
Thanks for the help.
Regards.
Tiago Sebastião
From: Isha Lamboo [mailto:isha.lamboo@virtualsciences.nl]
Sent: 7 de dezembro de 2022 08:42
To: users@nifi.apache.org
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***
________________________________
Hi Tiago,
Thanks for updating and sharing your solution.
I noticed you only encrypted the flow.json.gz file. Did you also delete the xml version in previous attempts?
It makes me wonder if the migrating fails for the flow.xml.gz specifically, then deleting the xml during migration might be an easier fix.
Regards,
Isha
Van: Tiago Luís Sebastião (DSI) <ti...@cgd.pt>>
Verzonden: dinsdag 6 december 2022 17:24
Aan: users@nifi.apache.org<ma...@nifi.apache.org>
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi Isha,
I had already tried that and didn’t work also.
But I tried the last thing I wrote on the previous email and it worked… Not happy with this solution but the issue (encryption/warnings) is solved I guess.
Having Nifi.properties:
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
With nifi Stopped:
1. Encrypt new files on the side:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f /apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n /apps/nifi-1.18.0/conf/nifi.properties -o /apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s PasswordUsedOnNifiProperties -x -v
1. Backup the flow.json.gz and rename the new one:
mv /apps/nifi-1.18.0/conf/flow.json.gz /apps/nifi-1.18.0/conf/flow_bk202212.json.gz
mv /apps/nifi-1.18.0/conf/flow2.json.gz /apps/nifi-1.18.0/conf/flow.json.gz
1. Change nifi.properties file:
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
to
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
1. Start nifi
No errors found and no warnings…
Tiago Sebastião
From: Isha Lamboo [mailto:isha.lamboo@virtualsciences.nl]
Sent: 25 de novembro de 2022 09:48
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***
________________________________
Hi Tiago,
I’ve had a similar experience with migrating the flow encryption algorithm and in fact, some of them are still on the old one. The nifi.sh commands to update the sensitive properties key and algorithm are very tricky to use, because they update the nifi.properties file even if the migration fails for the flow.xml.gz and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following ones to fail because it tried to decrypt using the new algorithm. I needed to reset the nifi.properties file everytime.
Another thing I’ve noticed is that it doesn’t support the property file protection scheme that NiFi has. If your sensitive props key is encrypted you need to enter the raw value and make sure the nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.
These steps have mostly worked for me:
1. Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another dir
2. Unprotect the sensitive properties key:
* Replace the encrypted key with the raw one
* Empty this property: nifi.sensitive.props.key.protected
1. Check that the algorithm is still the old one: nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
2. Check the length of the raw key, it needs to be 12 characters or longer to migrate to the new algorithm.
3. If the key is empty or short:
* ./bin/nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties (run from the main dir)
* Check the output for any failures and if they occur, revert the nifi.properties file and fix any errors
* Try to start nifi with the new sensitive properties key
1. If it works, stop nifi and update the algorithm:
* ./bin/nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
* Check the output for any failures and if they occur, revert the nifi.properties file and fix any errors
* Try to start nifi with the new algorithm
1. Stop nifi, encrypt the sensitive properties key (or restore it if you didn’t need to change the password)
* Use the encrypt-config.sh from the nifi-toolkit, specify output files so you can check and compare.
* Beware: this tool doesn’t seem to know about flow.json.gz, so only use it to change the config files.
I hope this helps you find the solution.
Regards,
Isha
Van: Tiago Luís Sebastião (DSI) <ti...@cgd.pt>>
Verzonden: donderdag 24 november 2022 16:19
Aan: users@nifi.apache.org<ma...@nifi.apache.org>
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi again,
Sorry for not following up but other priorities came ahead…
Basically it’s still not working, I’ve tried several combinations and I still keep getting:
“Failed to process Flow Configuration [./conf/flow.xml.gz]
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [AES/GCM/NoPadding]”
After reading some documentation, for this purpose, assuming that the password configured in the nifi.properties file is “PasswordUsedOnNifiProperties”…
I’ve tried and failed:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties
I’ve tried and failed by setting algorithm to empty string:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties
I’ve tried and failed using the new toolkit (I was using toolkit version 1.13.3):
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f /apps/nifi-1.18.0/conf/flow.xml.gz -n /apps/nifi-1.18.0/conf/nifi.properties -s PasswordUsedOnNifiProperties -A NIFI_ARGON2_AES_GCM_256 -x -v
I’ve tried and failed doing the same but generating new files to debug:
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f /apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n /apps/nifi-1.18.0/conf/nifi.properties -o /apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s PasswordUsedOnNifiProperties -x -v
In this last one I noticed that the file flow2.json.gz got its passwords encrypted differently and a longer encrypt also.
I’m kind of wondering if I can use this last command to generate these files on the side and then manipulate the nifi.properties file by changing the algorithm to NIFI_PBKDF2_AES_GCM_256 since it seems it is already encrypted despite the known errors/warnings, then I would rename these new files to the older ones and start nifi with:
1. New flow.json.gz file (apparently encrypted with NIFI_PBKDF2_AES_GCM_256 algorithm)
2. New nifi.properties file (with nifi.sensitive.props.algorithm property manipulated to NIFI_PBKDF2_AES_GCM_256)
Since, unfortunately<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Fsearch%3Fsxsrf%3DALiCzsYtztPHersBtg21lqlpGlc7DZ9CUw%3A1669302812266%26q%3Dunfortunately%26spell%3D1%26sa%3DX%26ved%3D2ahUKEwiq4pbJjcf7AhXKzqQKHfK8DNwQkeECKAB6BAgGEAE&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3sFUfMRIjK%2BMvhhh8dKaNEXgpua8C0E5Cqed3pWjrww%3D&reserved=0>, Im getting nowhere with this and I need to migrate to version 1.18.0 in order to apply the bugfix that changes the serverConnectorFactory.setNeedClientAuth(wantClientAuth) to serverConnectorFactory.setWantClientAuth(wantClientAuth) I am needing help in a consequent situation.
In order to reduce the size of the log generated from the deprecation warnings (WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] generate salt requested) I’ve tried to LEVEL OFF that Warning from stateless-logback.xml file without success.
On the tag <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> and <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender"> I changed the tag <pattern> so that I could see the full class name (although without success also…)
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
to
<pattern>%date %level [%thread] %logger{140} %msg%n</pattern>
On the stateless-logback.xml I inserted the following:
<logger name="org.apache.nifi" level="INFO"/>
<logger name="deprecation.org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider" level="OFF" />
It’s not working and I don’t understand why, the class name seems to be correct but I keep getting the same WARN.
Sorry for the long email…
Regards.
Tiago Sebastião
From: Tiago Luís Sebastião (DSI)
Sent: 28 de outubro de 2022 09:48
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi David,
It’s a standalone deployment and runs directly on the server.
Yes the command updated the flow.xml.gz/flow.json.gz and nifi.properties settings.
Maybe I messed up the nifi.sensitive.props.key, I’ll run some more tests.
Thanks for your help.
Tiago
From: David Handermann [mailto:exceptionfactory@apache.org]
Sent: 27 de outubro de 2022 16:50
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi Tiago,
The initial warning for the Insecure Cipher Provider Algorithm indicates the use of the deprecated setting as mentioned previously.
The set-sensitive-properties-algorithm command looks correct, and should have updated the flow.xml.gz, flow.json.gz, and nifi.properties settings.
The Decryption Failed message indicates that the nifi.sensitive.props.key value does not match the value used to encrypt the flow configuration, or that the algorithm does not match.
Can you provide some additional details about the NiFi installation? Is this a standalone or clustered deployment, and is it running in a containerized environment, or directly on a server?
Regards,
David Handermann
On Thu, Oct 27, 2022 at 10:35 AM Tiago Luís Sebastião (DSI) <ti...@cgd.pt>> wrote:
Hi all,
I'm having the same “problem”.
I upgraded nifi version from 1.17.0 to 1.18.0 and that same warning started to appear 500k times a day.
"
WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] generate salt requested
"
A already had nifi.sensitive.props.key value defined from when we migrated to 1.15.3.
With Nifi STOPPED and without changing any configuration on nifi.properties I executed the following:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
No errors found there, then I started Nifi and received the following errors:
"
WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [AES/GCM/NoPadding]
"
Since Nifi could not start anymore I reversed it...
Now Im kind of stuck with this warning...
Anyone knows what Im doing wrong?
Tiago
From: David Handermann [mailto:exceptionfactory@apache.org<ma...@apache.org>]
Sent: 19 de outubro de 2022 13:41
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi Mike,
The deprecation warning is not related to NIFI-10567 or Sensitive Dynamic Properties.
Deprecation logging is a new feature added in NiFi 1.18.0 to highlight components and features that are targeted for removal in future major releases. The current administrator's guide has more details on deprecation logging. [1] Deprecation warnings do not impact operational behavior, but they do identify configuration settings that should be changed.
In this particular case, the deprecation is related to the use of the insecure algorithm. NiFi 1.14.0 and following introduced new Sensitive Properties Key Algorithm settings, which should be used instead of the historical default value indicated in the warning. The new default value is NIFI_PBKDF2_AES_GCM_256, additional supported options are listed in the administrator's guide, [2] along with the command that can be run to update the Sensitive Properties Key Algorithm. [3]
Feel free to follow up if you have additional questions.
Regards,
David Handermann
[1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#deprecation-logging<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23deprecation-logging&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hziWAtka2q2neF9xJhAaApmx1LGfaF%2BgakH8igfLBEA%3D&reserved=0>
[2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#property-encryption-algorithms<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23property-encryption-algorithms&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XG8GjHn6OlNwWMPN6HdKg2NpJUE1XQLo3PY1%2BHY8GVA%3D&reserved=0>
[3] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#updating-the-sensitive-properties-algorithm<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23updating-the-sensitive-properties-algorithm&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=f9QbD7x0STWtvVIgNAhJ%2BXEtQlgDBQ6LS3w%2BWehlYpc%3D&reserved=0>
On Wed, Oct 19, 2022 at 7:28 AM Mike S <88...@gmail.com>> wrote:
I upgraded from 1.16.2 to 1.18.0 and now see this warning in the log file.
WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] generate salt requested
org.apache.nifi.deprecation.log.DeprecationException: Reference Class [org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider] ClassLoader [org.apache.nifi.nar.NarClassLoader[./work/nar/framework/nifi-framework-nar-1.18.0.nar-unpacked]]
I read this here.
NIFI-10567<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-10567&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867354005%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BMM0MYueQm3Yqu7daAhuo7I%2F41uK6O54URVDf7%2F1C9A%3D&reserved=0> Corrects the parsing of Sensitive Dynamic Properties read from the XML version of the flow configuration, in absence of the JSON version.
The issue surfaces when upgrading to NiFi 1.17.0 or 1.18.0 from a version older than 1.16.0. The issue also requires the presence of a Parameter Context with a Sensitive value assigned to a component with a Sensitive Property. Upgrading from 1.16.0 and following is not a problem.
It appears that all my ListS3 processors using sensitive properties are working.
Is this related since 1.16.2 has the latest flow.json.gz file?
Mike
RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Posted by Isha Lamboo <is...@virtualsciences.nl>.
Hi Tiago,
Thanks for updating and sharing your solution.
I noticed you only encrypted the flow.json.gz file. Did you also delete the xml version in previous attempts?
It makes me wonder if the migrating fails for the flow.xml.gz specifically, then deleting the xml during migration might be an easier fix.
Regards,
Isha
Van: Tiago Luís Sebastião (DSI) <ti...@cgd.pt>
Verzonden: dinsdag 6 december 2022 17:24
Aan: users@nifi.apache.org
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi Isha,
I had already tried that and didn’t work also.
But I tried the last thing I wrote on the previous email and it worked… Not happy with this solution but the issue (encryption/warnings) is solved I guess.
Having Nifi.properties:
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
With nifi Stopped:
1. Encrypt new files on the side:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f /apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n /apps/nifi-1.18.0/conf/nifi.properties -o /apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s PasswordUsedOnNifiProperties -x -v
1. Backup the flow.json.gz and rename the new one:
mv /apps/nifi-1.18.0/conf/flow.json.gz /apps/nifi-1.18.0/conf/flow_bk202212.json.gz
mv /apps/nifi-1.18.0/conf/flow2.json.gz /apps/nifi-1.18.0/conf/flow.json.gz
1. Change nifi.properties file:
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
to
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
1. Start nifi
No errors found and no warnings…
Tiago Sebastião
Tlf: 218456542 | Ext. 556542 | tiago.luis.sebastiao@cgd.pt<ma...@cgd.pt>
[cid:image001.jpg@01D90A1F.72B3A4E0]
DSI - Data Office – USI6.2 Unidade de Serviços de Dados
www.cgd.pt<https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.cgd.pt%2F&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=37%2FWY7BK17LoCCmr7ii1I0w7FcW65j%2F2A43ckkt1mok%3D&reserved=0>
Antes de imprimir esta mensagem, pense no meio ambiente.
Esta mensagem e-mail, assim como os ficheiros eventualmente anexos, é reservada aos seus destinatários, e pode conter informação confidencial ou estar sujeita a restrições legais. Se não é o seu destinatário ou se recebeu esta mensagem por motivo de erro, solicitamos que não faça qualquer uso ou divulgação do seu conteúdo e proceda à eliminação permanente desta mensagem e respetivos anexos.
Caixa Geral de Depósitos, S.A. | Sede Social: Av. João XXI, 63, 1000-300 LISBOA | Capital Social 3.844.143.735,00 € | CRCL e Contribuinte 500 960 046
From: Isha Lamboo [mailto:isha.lamboo@virtualsciences.nl]
Sent: 25 de novembro de 2022 09:48
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
*** ATENÇÃO: esta mensagem de e-mail tem origem externa!
A cibersegurança é uma responsabilidade partilhada. Não aceda a links nem anexos de mensagens suspeitas ou inesperadas.
CSIRT CGD ***
________________________________
Hi Tiago,
I’ve had a similar experience with migrating the flow encryption algorithm and in fact, some of them are still on the old one. The nifi.sh commands to update the sensitive properties key and algorithm are very tricky to use, because they update the nifi.properties file even if the migration fails for the flow.xml.gz and flow.json.gz.
It took me a while to realize my first failed attempt caused all the following ones to fail because it tried to decrypt using the new algorithm. I needed to reset the nifi.properties file everytime.
Another thing I’ve noticed is that it doesn’t support the property file protection scheme that NiFi has. If your sensitive props key is encrypted you need to enter the raw value and make sure the nifi.sensitive.props.key.protected is empty. You can re-encrypt afterwards.
These steps have mostly worked for me:
1. Backup your conf dir and flow.xml.gz/flow.json.gz if they are in another dir
2. Unprotect the sensitive properties key:
* Replace the encrypted key with the raw one
* Empty this property: nifi.sensitive.props.key.protected
1. Check that the algorithm is still the old one: nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
2. Check the length of the raw key, it needs to be 12 characters or longer to migrate to the new algorithm.
3. If the key is empty or short:
* ./bin/nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties (run from the main dir)
* Check the output for any failures and if they occur, revert the nifi.properties file and fix any errors
* Try to start nifi with the new sensitive properties key
1. If it works, stop nifi and update the algorithm:
* ./bin/nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
* Check the output for any failures and if they occur, revert the nifi.properties file and fix any errors
* Try to start nifi with the new algorithm
1. Stop nifi, encrypt the sensitive properties key (or restore it if you didn’t need to change the password)
* Use the encrypt-config.sh from the nifi-toolkit, specify output files so you can check and compare.
* Beware: this tool doesn’t seem to know about flow.json.gz, so only use it to change the config files.
I hope this helps you find the solution.
Regards,
Isha
Van: Tiago Luís Sebastião (DSI) <ti...@cgd.pt>>
Verzonden: donderdag 24 november 2022 16:19
Aan: users@nifi.apache.org<ma...@nifi.apache.org>
Onderwerp: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi again,
Sorry for not following up but other priorities came ahead…
Basically it’s still not working, I’ve tried several combinations and I still keep getting:
“Failed to process Flow Configuration [./conf/flow.xml.gz]
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [AES/GCM/NoPadding]”
After reading some documentation, for this purpose, assuming that the password configured in the nifi.properties file is “PasswordUsedOnNifiProperties”…
I’ve tried and failed:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties
I’ve tried and failed by setting algorithm to empty string:
File: nifi.properties
nifi.sensitive.props.key=PasswordUsedOnNifiProperties
nifi.sensitive.props.algorithm=
Cmd:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
./nifi.sh set-sensitive-properties-key PasswordUsedOnNifiProperties
I’ve tried and failed using the new toolkit (I was using toolkit version 1.13.3):
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f /apps/nifi-1.18.0/conf/flow.xml.gz -n /apps/nifi-1.18.0/conf/nifi.properties -s PasswordUsedOnNifiProperties -A NIFI_ARGON2_AES_GCM_256 -x -v
I’ve tried and failed doing the same but generating new files to debug:
Cmd:
/apps/nifi-toolkit-1.18.0/bin/encrypt-config.sh -f /apps/nifi-1.18.0/conf/flow.json.gz -g /apps/nifi-1.18.0/conf/flow2.json.gz -n /apps/nifi-1.18.0/conf/nifi.properties -o /apps/nifi-1.18.0/conf/nifi2.properties -A NIFI_PBKDF2_AES_GCM_256 -s PasswordUsedOnNifiProperties -x -v
In this last one I noticed that the file flow2.json.gz got its passwords encrypted differently and a longer encrypt also.
I’m kind of wondering if I can use this last command to generate these files on the side and then manipulate the nifi.properties file by changing the algorithm to NIFI_PBKDF2_AES_GCM_256 since it seems it is already encrypted despite the known errors/warnings, then I would rename these new files to the older ones and start nifi with:
1. New flow.json.gz file (apparently encrypted with NIFI_PBKDF2_AES_GCM_256 algorithm)
2. New nifi.properties file (with nifi.sensitive.props.algorithm property manipulated to NIFI_PBKDF2_AES_GCM_256)
Since, unfortunately<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.google.com%2Fsearch%3Fsxsrf%3DALiCzsYtztPHersBtg21lqlpGlc7DZ9CUw%3A1669302812266%26q%3Dunfortunately%26spell%3D1%26sa%3DX%26ved%3D2ahUKEwiq4pbJjcf7AhXKzqQKHfK8DNwQkeECKAB6BAgGEAE&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3sFUfMRIjK%2BMvhhh8dKaNEXgpua8C0E5Cqed3pWjrww%3D&reserved=0>, Im getting nowhere with this and I need to migrate to version 1.18.0 in order to apply the bugfix that changes the serverConnectorFactory.setNeedClientAuth(wantClientAuth) to serverConnectorFactory.setWantClientAuth(wantClientAuth) I am needing help in a consequent situation.
In order to reduce the size of the log generated from the deprecation warnings (WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] generate salt requested) I’ve tried to LEVEL OFF that Warning from stateless-logback.xml file without success.
On the tag <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> and <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender"> I changed the tag <pattern> so that I could see the full class name (although without success also…)
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
to
<pattern>%date %level [%thread] %logger{140} %msg%n</pattern>
On the stateless-logback.xml I inserted the following:
<logger name="org.apache.nifi" level="INFO"/>
<logger name="deprecation.org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider" level="OFF" />
It’s not working and I don’t understand why, the class name seems to be correct but I keep getting the same WARN.
Sorry for the long email…
Regards.
Tiago Sebastião
From: Tiago Luís Sebastião (DSI)
Sent: 28 de outubro de 2022 09:48
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: RE: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi David,
It’s a standalone deployment and runs directly on the server.
Yes the command updated the flow.xml.gz/flow.json.gz and nifi.properties settings.
Maybe I messed up the nifi.sensitive.props.key, I’ll run some more tests.
Thanks for your help.
Tiago
From: David Handermann [mailto:exceptionfactory@apache.org]
Sent: 27 de outubro de 2022 16:50
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi Tiago,
The initial warning for the Insecure Cipher Provider Algorithm indicates the use of the deprecated setting as mentioned previously.
The set-sensitive-properties-algorithm command looks correct, and should have updated the flow.xml.gz, flow.json.gz, and nifi.properties settings.
The Decryption Failed message indicates that the nifi.sensitive.props.key value does not match the value used to encrypt the flow configuration, or that the algorithm does not match.
Can you provide some additional details about the NiFi installation? Is this a standalone or clustered deployment, and is it running in a containerized environment, or directly on a server?
Regards,
David Handermann
On Thu, Oct 27, 2022 at 10:35 AM Tiago Luís Sebastião (DSI) <ti...@cgd.pt>> wrote:
Hi all,
I'm having the same “problem”.
I upgraded nifi version from 1.17.0 to 1.18.0 and that same warning started to appear 500k times a day.
"
WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] generate salt requested
"
A already had nifi.sensitive.props.key value defined from when we migrated to 1.15.3.
With Nifi STOPPED and without changing any configuration on nifi.properties I executed the following:
./nifi.sh set-sensitive-properties-algorithm NIFI_PBKDF2_AES_GCM_256
No errors found there, then I started Nifi and received the following errors:
"
WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
org.apache.nifi.encrypt.EncryptionException: Decryption Failed with Algorithm [AES/GCM/NoPadding]
"
Since Nifi could not start anymore I reversed it...
Now Im kind of stuck with this warning...
Anyone knows what Im doing wrong?
Tiago
From: David Handermann [mailto:exceptionfactory@apache.org<ma...@apache.org>]
Sent: 19 de outubro de 2022 13:41
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: NiFi 1.18.0 Sensitive Property broken after Upgrade
Hi Mike,
The deprecation warning is not related to NIFI-10567 or Sensitive Dynamic Properties.
Deprecation logging is a new feature added in NiFi 1.18.0 to highlight components and features that are targeted for removal in future major releases. The current administrator's guide has more details on deprecation logging. [1] Deprecation warnings do not impact operational behavior, but they do identify configuration settings that should be changed.
In this particular case, the deprecation is related to the use of the insecure algorithm. NiFi 1.14.0 and following introduced new Sensitive Properties Key Algorithm settings, which should be used instead of the historical default value indicated in the warning. The new default value is NIFI_PBKDF2_AES_GCM_256, additional supported options are listed in the administrator's guide, [2] along with the command that can be run to update the Sensitive Properties Key Algorithm. [3]
Feel free to follow up if you have additional questions.
Regards,
David Handermann
[1] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#deprecation-logging<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23deprecation-logging&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=hziWAtka2q2neF9xJhAaApmx1LGfaF%2BgakH8igfLBEA%3D&reserved=0>
[2] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#property-encryption-algorithms<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23property-encryption-algorithms&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=XG8GjHn6OlNwWMPN6HdKg2NpJUE1XQLo3PY1%2BHY8GVA%3D&reserved=0>
[3] https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#updating-the-sensitive-properties-algorithm<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnifi.apache.org%2Fdocs%2Fnifi-docs%2Fhtml%2Fadministration-guide.html%23updating-the-sensitive-properties-algorithm&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867197792%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=f9QbD7x0STWtvVIgNAhJ%2BXEtQlgDBQ6LS3w%2BWehlYpc%3D&reserved=0>
On Wed, Oct 19, 2022 at 7:28 AM Mike S <88...@gmail.com>> wrote:
I upgraded from 1.16.2 to 1.18.0 and now see this warning in the log file.
WARN [Flow Service Tasks Thread-1] d.o.a.n.s.u.c.NiFiLegacyCipherProvider Insecure Cipher Provider Algorithm [PBEWITHMD5AND256BITAES-CBC-OPENSSL] generate salt requested
org.apache.nifi.deprecation.log.DeprecationException: Reference Class [org.apache.nifi.security.util.crypto.NiFiLegacyCipherProvider] ClassLoader [org.apache.nifi.nar.NarClassLoader[./work/nar/framework/nifi-framework-nar-1.18.0.nar-unpacked]]
I read this here.
NIFI-10567<https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FNIFI-10567&data=05%7C01%7Cisha.lamboo%40virtualsciences.nl%7C6c07eab2e0d84bfc8de908dad7a6489e%7C21429da9e4ad45f99a6fcd126a64274b%7C0%7C0%7C638059407867354005%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=BMM0MYueQm3Yqu7daAhuo7I%2F41uK6O54URVDf7%2F1C9A%3D&reserved=0> Corrects the parsing of Sensitive Dynamic Properties read from the XML version of the flow configuration, in absence of the JSON version.
The issue surfaces when upgrading to NiFi 1.17.0 or 1.18.0 from a version older than 1.16.0. The issue also requires the presence of a Parameter Context with a Sensitive value assigned to a component with a Sensitive Property. Upgrading from 1.16.0 and following is not a problem.
It appears that all my ListS3 processors using sensitive properties are working.
Is this related since 1.16.2 has the latest flow.json.gz file?
Mike