You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Jorge Machado <jo...@me.com> on 2018/06/04 07:57:43 UTC
KerberosProperties.validatePrincipalAndKeytab Error ?
Hi Guys,
I’m facing the issue that I cannot start the DeleteHDFS with the error: "Kerberos Principal must be provided when using a secure configuration”
I’m able to reproduce this with this test:
@Test
public void testKerberosOptionsWithCredentialServices() throws Exception {
SimpleHadoopProcessor processor = new SimpleHadoopProcessor(kerberosProperties);
TestRunner runner = TestRunners.newTestRunner(processor);
// initialize the runner with EL for the kerberos properties
runner.setProperty(AbstractHadoopProcessor.HADOOP_CONFIGURATION_RESOURCES, "${variableHadoopConfigResources}");
runner.setProperty(kerberosProperties.getKerberosPrincipal(), "${variablePrincipal}");
runner.setProperty(kerberosProperties.getKerberosKeytab(), "${variableKeytab}");
// add variables for all the kerberos properties except for the keytab
runner.setVariable("variableHadoopConfigResources", "src/test/resources/core-site-security.xml");
runner.assertValid();
}
In our case the ${principal} and ${keytab} is coming as a attribute on the incomming the Flowfile, the problem is that this validation of the attribute is happening before.
Should this work like this? In all other places if we are using a variable it can be evaluate at run time...
Jorge Machado
Re: KerberosProperties.validatePrincipalAndKeytab Error ?
Posted by Jorge Machado <jo...@me.com>.
Ok that makes sinn. It was really confusing because all other processors when they say Supports language expression it works on the incoming flow files.
Thanks for the heads up.
Jorge Machado
> On 4 Jun 2018, at 14:34, Mark Payne <ma...@hotmail.com> wrote:
>
> Jorge,
>
> These properties do support Expression Language. However, they do not support evaluating FlowFile Attributes,
> only values from the Variable Registry. So this is going to be invalid unless you define a variable in the variable
> registry for both the Principal and the Keytab.
>
> Unfortunately, in past versions of NiFi, this was not made very clear, when a property supported Expression Language
> but only against the Variable Registry. Fortunately, in the next version, 1.7.0, the UI will clearly indicate whether Expression
> Language can be evaluated against FlowFile Attributes or only the Variable Registry, so it should help to clear up
> some of this confusion.
>
> Thanks
> -Mark
>
>
>> On Jun 4, 2018, at 6:43 AM, Jorge Machado <jo...@me.com> wrote:
>>
>> Sivaprasanna,
>> Yes I have that property set, the Unit test is failing IMHO it should not.
>>
>>
>> Jorge Machado
>> Best Regards
>>
>>
>>> On 4 Jun 2018, at 12:09, Sivaprasanna <si...@gmail.com> wrote:
>>>
>>> Jorge,
>>>
>>> Both 'Kerberos Principal' abd 'Kerberos Keytab' support NiFi expression language so ${principal} and ${keytab} is valid here. Can you check if the property "nifi.kerberos.krb5.file" is set in nifi.properties file? Looks like this has to be set according to the description of those properties.
>>>
>>> -
>>> Sivaprasanna
>>>
>>> On Mon, Jun 4, 2018 at 1:27 PM, Jorge Machado <jomach@me.com <ma...@me.com>> wrote:
>>> Hi Guys,
>>>
>>> I’m facing the issue that I cannot start the DeleteHDFS with the error: "Kerberos Principal must be provided when using a secure configuration”
>>>
>>>
>>> I’m able to reproduce this with this test:
>>> @Test
>>> public void testKerberosOptionsWithCredentialServices() throws Exception {
>>> SimpleHadoopProcessor processor = new SimpleHadoopProcessor(kerberosProperties);
>>> TestRunner runner = TestRunners.newTestRunner(processor);
>>>
>>> // initialize the runner with EL for the kerberos properties
>>> runner.setProperty(AbstractHadoopProcessor.HADOOP_CONFIGURATION_RESOURCES, "${variableHadoopConfigResources}");
>>> runner.setProperty(kerberosProperties.getKerberosPrincipal(), "${variablePrincipal}");
>>> runner.setProperty(kerberosProperties.getKerberosKeytab(), "${variableKeytab}");
>>>
>>> // add variables for all the kerberos properties except for the keytab
>>> runner.setVariable("variableHadoopConfigResources", "src/test/resources/core-site-security.xml");
>>> runner.assertValid();
>>> }
>>> In our case the ${principal} and ${keytab} is coming as a attribute on the incomming the Flowfile, the problem is that this validation of the attribute is happening before.
>>> Should this work like this? In all other places if we are using a variable it can be evaluate at run time...
>>>
>>> Jorge Machado
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
Re: KerberosProperties.validatePrincipalAndKeytab Error ?
Posted by Mark Payne <ma...@hotmail.com>.
Jorge,
These properties do support Expression Language. However, they do not support evaluating FlowFile Attributes,
only values from the Variable Registry. So this is going to be invalid unless you define a variable in the variable
registry for both the Principal and the Keytab.
Unfortunately, in past versions of NiFi, this was not made very clear, when a property supported Expression Language
but only against the Variable Registry. Fortunately, in the next version, 1.7.0, the UI will clearly indicate whether Expression
Language can be evaluated against FlowFile Attributes or only the Variable Registry, so it should help to clear up
some of this confusion.
Thanks
-Mark
> On Jun 4, 2018, at 6:43 AM, Jorge Machado <jo...@me.com> wrote:
>
> Sivaprasanna,
> Yes I have that property set, the Unit test is failing IMHO it should not.
>
>
> Jorge Machado
> Best Regards
>
>
>> On 4 Jun 2018, at 12:09, Sivaprasanna <si...@gmail.com> wrote:
>>
>> Jorge,
>>
>> Both 'Kerberos Principal' abd 'Kerberos Keytab' support NiFi expression language so ${principal} and ${keytab} is valid here. Can you check if the property "nifi.kerberos.krb5.file" is set in nifi.properties file? Looks like this has to be set according to the description of those properties.
>>
>> -
>> Sivaprasanna
>>
>> On Mon, Jun 4, 2018 at 1:27 PM, Jorge Machado <jomach@me.com <ma...@me.com>> wrote:
>> Hi Guys,
>>
>> I’m facing the issue that I cannot start the DeleteHDFS with the error: "Kerberos Principal must be provided when using a secure configuration”
>>
>>
>> I’m able to reproduce this with this test:
>> @Test
>> public void testKerberosOptionsWithCredentialServices() throws Exception {
>> SimpleHadoopProcessor processor = new SimpleHadoopProcessor(kerberosProperties);
>> TestRunner runner = TestRunners.newTestRunner(processor);
>>
>> // initialize the runner with EL for the kerberos properties
>> runner.setProperty(AbstractHadoopProcessor.HADOOP_CONFIGURATION_RESOURCES, "${variableHadoopConfigResources}");
>> runner.setProperty(kerberosProperties.getKerberosPrincipal(), "${variablePrincipal}");
>> runner.setProperty(kerberosProperties.getKerberosKeytab(), "${variableKeytab}");
>>
>> // add variables for all the kerberos properties except for the keytab
>> runner.setVariable("variableHadoopConfigResources", "src/test/resources/core-site-security.xml");
>> runner.assertValid();
>> }
>> In our case the ${principal} and ${keytab} is coming as a attribute on the incomming the Flowfile, the problem is that this validation of the attribute is happening before.
>> Should this work like this? In all other places if we are using a variable it can be evaluate at run time...
>>
>> Jorge Machado
>>
>>
>>
>>
>>
>>
>
Re: KerberosProperties.validatePrincipalAndKeytab Error ?
Posted by Jorge Machado <jo...@me.com>.
Sivaprasanna,
Yes I have that property set, the Unit test is failing IMHO it should not.
Jorge Machado
Best Regards
> On 4 Jun 2018, at 12:09, Sivaprasanna <si...@gmail.com> wrote:
>
> Jorge,
>
> Both 'Kerberos Principal' abd 'Kerberos Keytab' support NiFi expression language so ${principal} and ${keytab} is valid here. Can you check if the property "nifi.kerberos.krb5.file" is set in nifi.properties file? Looks like this has to be set according to the description of those properties.
>
> -
> Sivaprasanna
>
> On Mon, Jun 4, 2018 at 1:27 PM, Jorge Machado <jomach@me.com <ma...@me.com>> wrote:
> Hi Guys,
>
> I’m facing the issue that I cannot start the DeleteHDFS with the error: "Kerberos Principal must be provided when using a secure configuration”
>
>
> I’m able to reproduce this with this test:
> @Test
> public void testKerberosOptionsWithCredentialServices() throws Exception {
> SimpleHadoopProcessor processor = new SimpleHadoopProcessor(kerberosProperties);
> TestRunner runner = TestRunners.newTestRunner(processor);
>
> // initialize the runner with EL for the kerberos properties
> runner.setProperty(AbstractHadoopProcessor.HADOOP_CONFIGURATION_RESOURCES, "${variableHadoopConfigResources}");
> runner.setProperty(kerberosProperties.getKerberosPrincipal(), "${variablePrincipal}");
> runner.setProperty(kerberosProperties.getKerberosKeytab(), "${variableKeytab}");
>
> // add variables for all the kerberos properties except for the keytab
> runner.setVariable("variableHadoopConfigResources", "src/test/resources/core-site-security.xml");
> runner.assertValid();
> }
> In our case the ${principal} and ${keytab} is coming as a attribute on the incomming the Flowfile, the problem is that this validation of the attribute is happening before.
> Should this work like this? In all other places if we are using a variable it can be evaluate at run time...
>
> Jorge Machado
>
>
>
>
>
>
Re: KerberosProperties.validatePrincipalAndKeytab Error ?
Posted by Sivaprasanna <si...@gmail.com>.
Jorge,
Both 'Kerberos Principal' abd 'Kerberos Keytab' support NiFi expression
language so ${principal} and ${keytab} is valid here. Can you check if the
property "nifi.kerberos.krb5.file" is set in nifi.properties file? Looks
like this has to be set according to the description of those properties.
-
Sivaprasanna
On Mon, Jun 4, 2018 at 1:27 PM, Jorge Machado <jo...@me.com> wrote:
> Hi Guys,
>
> I’m facing the issue that I cannot start the DeleteHDFS with the error:
> "Kerberos Principal must be provided when using a secure configuration”
>
> I’m able to reproduce this with this test:
>
> @Test
> public void testKerberosOptionsWithCredentialServices() throws Exception {
> SimpleHadoopProcessor processor = new SimpleHadoopProcessor(kerberosProperties);
> TestRunner runner = TestRunners.newTestRunner(processor);
>
> // initialize the runner with EL for the kerberos properties
> runner.setProperty(AbstractHadoopProcessor.HADOOP_CONFIGURATION_RESOURCES, "${variableHadoopConfigResources}");
> runner.setProperty(kerberosProperties.getKerberosPrincipal(), "${variablePrincipal}");
> runner.setProperty(kerberosProperties.getKerberosKeytab(), "${variableKeytab}");
>
> // add variables for all the kerberos properties except for the keytab
> runner.setVariable("variableHadoopConfigResources", "src/test/resources/core-site-security.xml");
> runner.assertValid();
> }
>
> In our case the ${principal} and ${keytab} is coming as a attribute on the
> incomming the Flowfile, the problem is that this validation of the
> attribute is happening before.
> Should this work like this? In all other places if we are using a variable
> it can be evaluate at run time...
>
> Jorge Machado
>
>
>
>
>
>