You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Juan Pablo Gardella <ga...@gmail.com> on 2018/10/18 14:53:07 UTC
ReplaceText cannot consume messages if Regex does not match
Hi all,
I'm seeing that ReplaceText is not able to consume messages that does not
match regex. It keeps all the messages in the input queue instead of
sending them to failure relationship. Is this the intended behavior or I
have to file a ticket in order to be fixed? In that way, the processor is
not able to process bad messages and converts in the bottleneck of a flow
Juan
RE: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
That looks like your expression has a syntax error but I’d have to see the whole message.
Thanks
Shawn
From: Juan Pablo Gardella <ga...@gmail.com>
Sent: Thursday, October 18, 2018 10:01 AM
To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Hortonworks nifi based on 1.5.0:
[image.png]
Configuration:
[image.png]
Thanks
On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Hi Juan,
What version of NiFi are you running on?
What mode are you running ReplaceText in, all text or line by line?
Other settings that might be important? What’s your RegEx look like (if your able to share).
--Peter
From: Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com<ma...@gmail.com>]
Sent: Thursday, October 18, 2018 8:53 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: [EXT] ReplaceText cannot consume messages if Regex does not match
Hi all,
I'm seeing that ReplaceText is not able to consume messages that does not match regex. It keeps all the messages in the input queue instead of sending them to failure relationship. Is this the intended behavior or I have to file a ticket in order to be fixed? In that way, the processor is not able to process bad messages and converts in the bottleneck of a flow
Juan
Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Jeff <jt...@gmail.com>.
Is the actualSettlementDate attribute's value in the "MM/dd/yyyy" format,
with no other text in front of the date? For instance, "10/18/2018
12:30:00" is parsable by the "MM/dd/yyyy" format, but "12:30:00 10/18/2018"
is not.
On Thu, Oct 18, 2018 at 12:02 PM Juan Pablo Gardella <
gardellajuanpablo@gmail.com> wrote:
> At *search value*:(?s)(^.*$)
>
> At *Replacement value*:
>
> *<?xml version="1.0"
> encoding="UTF-8"?><SettleDate>${actualSettlementDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</SettleDate>*
>
> The actualSettlementDate is a flowfile attribute. The problem is the
> replacement value is evaluated inside the processor and the *toDate *method
> fails.
>
> Hope it's clear now.
>
>
> On Thu, 18 Oct 2018 at 12:51 Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
>> I’m still trying to understand your actual issue, can your provide a
>> screenshot of the ReplaceText config like the attached, I need to see
>> exactly where you’re putting the expression. A template would also be
>> really helpful.
>>
>>
>>
>> Thanks
>>
>> Shawn Weeks
>>
>>
>>
>> *From:* Juan Pablo Gardella <ga...@gmail.com>
>>
>> *Sent:* Thursday, October 18, 2018 10:45 AM
>>
>>
>> *To:* users@nifi.apache.org
>> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
>> not match
>>
>>
>>
>> At ReplaceText
>> <https://raw.githubusercontent.com/apache/nifi/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java>processor
>> we have:
>>
>>
>>
>> [image: image.png]
>>
>> As you can see, only if *StackOverflowError *is raised during the
>> evaluation, the flowfile is send to failure relationship. I would like to
>> update the code to use Exception or NifiExpressionFailedException (if it
>> exits).
>>
>>
>>
>> Juan
>>
>>
>>
>> On Thu, 18 Oct 2018 at 12:33 Shawn Weeks <sw...@weeksconsulting.us>
>> wrote:
>>
>> What processor are you defining your expression in? I also may be
>> misunderstanding the problem because I don’t see any regular expressions
>> anywhere. Can you create a sample workflow showing your issue so I can take
>> a look at it.
>>
>>
>>
>> Thanks
>>
>> Shawn Weeks
>>
>>
>>
>> *From:* Juan Pablo Gardella <ga...@gmail.com>
>> *Sent:* Thursday, October 18, 2018 10:27 AM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
>> not match
>>
>>
>>
>> No, it's not a valid date. I would like if it an error happens, I would
>> like to throw the flowfile to failure and continue.
>>
>>
>>
>> On Thu, 18 Oct 2018 at 12:19 Shawn Weeks <sw...@weeksconsulting.us>
>> wrote:
>>
>> Any expression language syntax has to be correct or the processor won’t
>> run. I’m not sure there is any way to work around that except to explicitly
>> check that the value you are trying to evaluate is valid. Is the attribute
>> “tradeDate” coming from the contents of a flow file or is it defined
>> somewhere else. Can you ensure it is a valid date in that format before
>> hand?
>>
>>
>>
>> Thanks
>>
>> Shawn Weeks
>>
>>
>>
>> *From:* Juan Pablo Gardella <ga...@gmail.com>
>>
>> *Sent:* Thursday, October 18, 2018 10:13 AM
>>
>>
>> *To:* users@nifi.apache.org
>> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
>> not match
>>
>>
>>
>> Hi, the error is not in the processor itself. It's in the expression used
>> against flowfile attributes. For example inside the text, I have:
>>
>>
>>
>>
>> <EventDate>${tradeDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</EventDate>
>>
>> And that is the root issue. If it's unable to convert it, the flow cannot
>> be consumed. How can I evaluate attributes in a non-blocker way?
>>
>>
>>
>> Juan
>>
>>
>>
>> On Thu, 18 Oct 2018 at 12:07 Shawn Weeks <sw...@weeksconsulting.us>
>> wrote:
>>
>> Where is your expression? That’s not the entire configuration for that
>> processor.
>>
>>
>>
>> Thanks
>>
>> Shawn Weeks
>>
>>
>>
>> *From:* Juan Pablo Gardella <ga...@gmail.com>
>> *Sent:* Thursday, October 18, 2018 10:03 AM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
>> not match
>>
>>
>>
>> Configuration:
>>
>> Replacement Strategy: Always replace
>>
>> EvaluationMode: Entire text
>>
>>
>>
>>
>>
>> On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <
>> gardellajuanpablo@gmail.com> wrote:
>>
>> Hortonworks nifi based on 1.5.0:
>>
>>
>>
>> Configuration:
>>
>> Thanks
>>
>>
>>
>> On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com>
>> wrote:
>>
>> Hi Juan,
>>
>>
>>
>> What version of NiFi are you running on?
>>
>> What mode are you running ReplaceText in, all text or line by line?
>>
>> Other settings that might be important? What’s your RegEx look like (if
>> your able to share).
>>
>>
>>
>> --Peter
>>
>>
>>
>>
>>
>> *From:* Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com]
>> *Sent:* Thursday, October 18, 2018 8:53 AM
>> *To:* users@nifi.apache.org
>> *Subject:* [EXT] ReplaceText cannot consume messages if Regex does not
>> match
>>
>>
>>
>> Hi all,
>>
>>
>>
>> I'm seeing that ReplaceText is not able to consume messages that does not
>> match regex. It keeps all the messages in the input queue instead of
>> sending them to failure relationship. Is this the intended behavior or I
>> have to file a ticket in order to be fixed? In that way, the processor is
>> not able to process bad messages and converts in the bottleneck of a flow
>>
>>
>>
>> Juan
>>
>>
Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Juan Pablo Gardella <ga...@gmail.com>.
Patch available at https://issues.apache.org/jira/browse/NIFI-5761
<https://www.google.com/url?q=https://issues.apache.org/jira/browse/NIFI-5761&sa=D&source=hangouts&ust=1540669801593000&usg=AFQjCNHVVj1whk8fb1Z3Ic51w2TdwvrC1Q>
On Thu, 18 Oct 2018 at 13:33 Juan Pablo Gardella <
gardellajuanpablo@gmail.com> wrote:
> Exactly, I know it is not an issue in the replace text code, but it
> happens inside it. If we are using the ReplaceText in multiple places, it
> increases the flow design complexity. We need to evaluate all expressions
> before sending to the processor, to be sure will not fail in ReplaceText
> processor.
>
> Notice it's impossible if you have to process content dynamically. I would
> happy to file a ticket and the patch as I mention.
>
> Juan
>
>
>
> On Thu, 18 Oct 2018 at 13:18 Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
>> I understand the issue now, I’m not sure that a failure of ReplaceText is
>> the best place to catch this though. The reason I’m not sure it’s the best
>> place is what happens if there are multiple failures because you had
>> multiple expressions, just having them all routed to the same failure
>> wouldn’t help you make decisions on what to do with a single attribute.
>> Perhaps a better solution would be to use a RouteOnAttribute to check if
>> the attributes match a certain pattern before sending them to ReplaceText.
>> A possible expression could be
>> “${actualSettlementDate:matches('[0-9]{2}/[0-9]{2}/[0-9]{4}')}” however
>> that would not catch things that look like dates but aren’t valid.
>>
>>
>>
>> Thanks
>>
>> Shawn Weeks
>>
>>
>>
>> *From:* Juan Pablo Gardella <ga...@gmail.com>
>>
>> *Sent:* Thursday, October 18, 2018 11:03 AM
>>
>>
>> *To:* users@nifi.apache.org
>> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
>> not match
>>
>>
>>
>> At *search value*:(?s)(^.*$)
>>
>>
>>
>> At *Replacement value*:
>>
>>
>> *<?xml version="1.0" encoding="UTF-8"?>
>> <SettleDate>${actualSettlementDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</SettleDate>*
>>
>>
>>
>> The actualSettlementDate is a flowfile attribute. The problem is the
>> replacement value is evaluated inside the processor and the *toDate *method
>> fails.
>>
>>
>>
>> Hope it's clear now.
>>
>>
>>
>>
>>
>> On Thu, 18 Oct 2018 at 12:51 Shawn Weeks <sw...@weeksconsulting.us>
>> wrote:
>>
>> I’m still trying to understand your actual issue, can your provide a
>> screenshot of the ReplaceText config like the attached, I need to see
>> exactly where you’re putting the expression. A template would also be
>> really helpful.
>>
>>
>>
>> Thanks
>>
>> Shawn Weeks
>>
>>
>>
>> *From:* Juan Pablo Gardella <ga...@gmail.com>
>>
>> *Sent:* Thursday, October 18, 2018 10:45 AM
>>
>>
>> *To:* users@nifi.apache.org
>> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
>> not match
>>
>>
>>
>> At ReplaceText
>> <https://raw.githubusercontent.com/apache/nifi/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java>processor
>> we have:
>>
>>
>>
>> [image: image.png]
>>
>> As you can see, only if *StackOverflowError *is raised during the
>> evaluation, the flowfile is send to failure relationship. I would like to
>> update the code to use Exception or NifiExpressionFailedException (if it
>> exits).
>>
>>
>>
>> Juan
>>
>>
>>
>> On Thu, 18 Oct 2018 at 12:33 Shawn Weeks <sw...@weeksconsulting.us>
>> wrote:
>>
>> What processor are you defining your expression in? I also may be
>> misunderstanding the problem because I don’t see any regular expressions
>> anywhere. Can you create a sample workflow showing your issue so I can take
>> a look at it.
>>
>>
>>
>> Thanks
>>
>> Shawn Weeks
>>
>>
>>
>> *From:* Juan Pablo Gardella <ga...@gmail.com>
>> *Sent:* Thursday, October 18, 2018 10:27 AM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
>> not match
>>
>>
>>
>> No, it's not a valid date. I would like if it an error happens, I would
>> like to throw the flowfile to failure and continue.
>>
>>
>>
>> On Thu, 18 Oct 2018 at 12:19 Shawn Weeks <sw...@weeksconsulting.us>
>> wrote:
>>
>> Any expression language syntax has to be correct or the processor won’t
>> run. I’m not sure there is any way to work around that except to explicitly
>> check that the value you are trying to evaluate is valid. Is the attribute
>> “tradeDate” coming from the contents of a flow file or is it defined
>> somewhere else. Can you ensure it is a valid date in that format before
>> hand?
>>
>>
>>
>> Thanks
>>
>> Shawn Weeks
>>
>>
>>
>> *From:* Juan Pablo Gardella <ga...@gmail.com>
>>
>> *Sent:* Thursday, October 18, 2018 10:13 AM
>>
>>
>> *To:* users@nifi.apache.org
>> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
>> not match
>>
>>
>>
>> Hi, the error is not in the processor itself. It's in the expression used
>> against flowfile attributes. For example inside the text, I have:
>>
>>
>>
>>
>> <EventDate>${tradeDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</EventDate>
>>
>> And that is the root issue. If it's unable to convert it, the flow cannot
>> be consumed. How can I evaluate attributes in a non-blocker way?
>>
>>
>>
>> Juan
>>
>>
>>
>> On Thu, 18 Oct 2018 at 12:07 Shawn Weeks <sw...@weeksconsulting.us>
>> wrote:
>>
>> Where is your expression? That’s not the entire configuration for that
>> processor.
>>
>>
>>
>> Thanks
>>
>> Shawn Weeks
>>
>>
>>
>> *From:* Juan Pablo Gardella <ga...@gmail.com>
>> *Sent:* Thursday, October 18, 2018 10:03 AM
>> *To:* users@nifi.apache.org
>> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
>> not match
>>
>>
>>
>> Configuration:
>>
>> Replacement Strategy: Always replace
>>
>> EvaluationMode: Entire text
>>
>>
>>
>>
>>
>> On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <
>> gardellajuanpablo@gmail.com> wrote:
>>
>> Hortonworks nifi based on 1.5.0:
>>
>>
>>
>> Configuration:
>>
>> Thanks
>>
>>
>>
>> On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com>
>> wrote:
>>
>> Hi Juan,
>>
>>
>>
>> What version of NiFi are you running on?
>>
>> What mode are you running ReplaceText in, all text or line by line?
>>
>> Other settings that might be important? What’s your RegEx look like (if
>> your able to share).
>>
>>
>>
>> --Peter
>>
>>
>>
>>
>>
>> *From:* Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com]
>> *Sent:* Thursday, October 18, 2018 8:53 AM
>> *To:* users@nifi.apache.org
>> *Subject:* [EXT] ReplaceText cannot consume messages if Regex does not
>> match
>>
>>
>>
>> Hi all,
>>
>>
>>
>> I'm seeing that ReplaceText is not able to consume messages that does not
>> match regex. It keeps all the messages in the input queue instead of
>> sending them to failure relationship. Is this the intended behavior or I
>> have to file a ticket in order to be fixed? In that way, the processor is
>> not able to process bad messages and converts in the bottleneck of a flow
>>
>>
>>
>> Juan
>>
>>
Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Juan Pablo Gardella <ga...@gmail.com>.
Exactly, I know it is not an issue in the replace text code, but it happens
inside it. If we are using the ReplaceText in multiple places, it increases
the flow design complexity. We need to evaluate all expressions before
sending to the processor, to be sure will not fail in ReplaceText
processor.
Notice it's impossible if you have to process content dynamically. I would
happy to file a ticket and the patch as I mention.
Juan
On Thu, 18 Oct 2018 at 13:18 Shawn Weeks <sw...@weeksconsulting.us> wrote:
> I understand the issue now, I’m not sure that a failure of ReplaceText is
> the best place to catch this though. The reason I’m not sure it’s the best
> place is what happens if there are multiple failures because you had
> multiple expressions, just having them all routed to the same failure
> wouldn’t help you make decisions on what to do with a single attribute.
> Perhaps a better solution would be to use a RouteOnAttribute to check if
> the attributes match a certain pattern before sending them to ReplaceText.
> A possible expression could be
> “${actualSettlementDate:matches('[0-9]{2}/[0-9]{2}/[0-9]{4}')}” however
> that would not catch things that look like dates but aren’t valid.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
>
> *Sent:* Thursday, October 18, 2018 11:03 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> At *search value*:(?s)(^.*$)
>
>
>
> At *Replacement value*:
>
>
> *<?xml version="1.0" encoding="UTF-8"?>
> <SettleDate>${actualSettlementDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</SettleDate>*
>
>
>
> The actualSettlementDate is a flowfile attribute. The problem is the
> replacement value is evaluated inside the processor and the *toDate *method
> fails.
>
>
>
> Hope it's clear now.
>
>
>
>
>
> On Thu, 18 Oct 2018 at 12:51 Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> I’m still trying to understand your actual issue, can your provide a
> screenshot of the ReplaceText config like the attached, I need to see
> exactly where you’re putting the expression. A template would also be
> really helpful.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
>
> *Sent:* Thursday, October 18, 2018 10:45 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> At ReplaceText
> <https://raw.githubusercontent.com/apache/nifi/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java>processor
> we have:
>
>
>
> [image: image.png]
>
> As you can see, only if *StackOverflowError *is raised during the
> evaluation, the flowfile is send to failure relationship. I would like to
> update the code to use Exception or NifiExpressionFailedException (if it
> exits).
>
>
>
> Juan
>
>
>
> On Thu, 18 Oct 2018 at 12:33 Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> What processor are you defining your expression in? I also may be
> misunderstanding the problem because I don’t see any regular expressions
> anywhere. Can you create a sample workflow showing your issue so I can take
> a look at it.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
> *Sent:* Thursday, October 18, 2018 10:27 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> No, it's not a valid date. I would like if it an error happens, I would
> like to throw the flowfile to failure and continue.
>
>
>
> On Thu, 18 Oct 2018 at 12:19 Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> Any expression language syntax has to be correct or the processor won’t
> run. I’m not sure there is any way to work around that except to explicitly
> check that the value you are trying to evaluate is valid. Is the attribute
> “tradeDate” coming from the contents of a flow file or is it defined
> somewhere else. Can you ensure it is a valid date in that format before
> hand?
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
>
> *Sent:* Thursday, October 18, 2018 10:13 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Hi, the error is not in the processor itself. It's in the expression used
> against flowfile attributes. For example inside the text, I have:
>
>
>
>
> <EventDate>${tradeDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</EventDate>
>
> And that is the root issue. If it's unable to convert it, the flow cannot
> be consumed. How can I evaluate attributes in a non-blocker way?
>
>
>
> Juan
>
>
>
> On Thu, 18 Oct 2018 at 12:07 Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> Where is your expression? That’s not the entire configuration for that
> processor.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
> *Sent:* Thursday, October 18, 2018 10:03 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Configuration:
>
> Replacement Strategy: Always replace
>
> EvaluationMode: Entire text
>
>
>
>
>
> On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <
> gardellajuanpablo@gmail.com> wrote:
>
> Hortonworks nifi based on 1.5.0:
>
>
>
> Configuration:
>
> Thanks
>
>
>
> On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> Hi Juan,
>
>
>
> What version of NiFi are you running on?
>
> What mode are you running ReplaceText in, all text or line by line?
>
> Other settings that might be important? What’s your RegEx look like (if
> your able to share).
>
>
>
> --Peter
>
>
>
>
>
> *From:* Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com]
> *Sent:* Thursday, October 18, 2018 8:53 AM
> *To:* users@nifi.apache.org
> *Subject:* [EXT] ReplaceText cannot consume messages if Regex does not
> match
>
>
>
> Hi all,
>
>
>
> I'm seeing that ReplaceText is not able to consume messages that does not
> match regex. It keeps all the messages in the input queue instead of
> sending them to failure relationship. Is this the intended behavior or I
> have to file a ticket in order to be fixed? In that way, the processor is
> not able to process bad messages and converts in the bottleneck of a flow
>
>
>
> Juan
>
>
RE: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
I understand the issue now, I’m not sure that a failure of ReplaceText is the best place to catch this though. The reason I’m not sure it’s the best place is what happens if there are multiple failures because you had multiple expressions, just having them all routed to the same failure wouldn’t help you make decisions on what to do with a single attribute. Perhaps a better solution would be to use a RouteOnAttribute to check if the attributes match a certain pattern before sending them to ReplaceText. A possible expression could be “${actualSettlementDate:matches('[0-9]{2}/[0-9]{2}/[0-9]{4}')}” however that would not catch things that look like dates but aren’t valid.
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>
Sent: Thursday, October 18, 2018 11:03 AM
To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
At search value:(?s)(^.*$)
At Replacement value:
<?xml version="1.0" encoding="UTF-8"?>
<SettleDate>${actualSettlementDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</SettleDate>
The actualSettlementDate is a flowfile attribute. The problem is the replacement value is evaluated inside the processor and the toDate method fails.
Hope it's clear now.
On Thu, 18 Oct 2018 at 12:51 Shawn Weeks <sw...@weeksconsulting.us>> wrote:
I’m still trying to understand your actual issue, can your provide a screenshot of the ReplaceText config like the attached, I need to see exactly where you’re putting the expression. A template would also be really helpful.
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>>
Sent: Thursday, October 18, 2018 10:45 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
At ReplaceText <https://raw.githubusercontent.com/apache/nifi/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java> processor we have:
[image.png]
As you can see, only if StackOverflowError is raised during the evaluation, the flowfile is send to failure relationship. I would like to update the code to use Exception or NifiExpressionFailedException (if it exits).
Juan
On Thu, 18 Oct 2018 at 12:33 Shawn Weeks <sw...@weeksconsulting.us>> wrote:
What processor are you defining your expression in? I also may be misunderstanding the problem because I don’t see any regular expressions anywhere. Can you create a sample workflow showing your issue so I can take a look at it.
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>>
Sent: Thursday, October 18, 2018 10:27 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
No, it's not a valid date. I would like if it an error happens, I would like to throw the flowfile to failure and continue.
On Thu, 18 Oct 2018 at 12:19 Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Any expression language syntax has to be correct or the processor won’t run. I’m not sure there is any way to work around that except to explicitly check that the value you are trying to evaluate is valid. Is the attribute “tradeDate” coming from the contents of a flow file or is it defined somewhere else. Can you ensure it is a valid date in that format before hand?
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>>
Sent: Thursday, October 18, 2018 10:13 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Hi, the error is not in the processor itself. It's in the expression used against flowfile attributes. For example inside the text, I have:
<EventDate>${tradeDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</EventDate>
And that is the root issue. If it's unable to convert it, the flow cannot be consumed. How can I evaluate attributes in a non-blocker way?
Juan
On Thu, 18 Oct 2018 at 12:07 Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Where is your expression? That’s not the entire configuration for that processor.
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>>
Sent: Thursday, October 18, 2018 10:03 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Configuration:
Replacement Strategy: Always replace
EvaluationMode: Entire text
On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <ga...@gmail.com>> wrote:
Hortonworks nifi based on 1.5.0:
Configuration:
Thanks
On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Hi Juan,
What version of NiFi are you running on?
What mode are you running ReplaceText in, all text or line by line?
Other settings that might be important? What’s your RegEx look like (if your able to share).
--Peter
From: Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com<ma...@gmail.com>]
Sent: Thursday, October 18, 2018 8:53 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: [EXT] ReplaceText cannot consume messages if Regex does not match
Hi all,
I'm seeing that ReplaceText is not able to consume messages that does not match regex. It keeps all the messages in the input queue instead of sending them to failure relationship. Is this the intended behavior or I have to file a ticket in order to be fixed? In that way, the processor is not able to process bad messages and converts in the bottleneck of a flow
Juan
Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Juan Pablo Gardella <ga...@gmail.com>.
At *search value*:(?s)(^.*$)
At *Replacement value*:
*<?xml version="1.0"
encoding="UTF-8"?><SettleDate>${actualSettlementDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</SettleDate>*
The actualSettlementDate is a flowfile attribute. The problem is the
replacement value is evaluated inside the processor and the *toDate *method
fails.
Hope it's clear now.
On Thu, 18 Oct 2018 at 12:51 Shawn Weeks <sw...@weeksconsulting.us> wrote:
> I’m still trying to understand your actual issue, can your provide a
> screenshot of the ReplaceText config like the attached, I need to see
> exactly where you’re putting the expression. A template would also be
> really helpful.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
>
> *Sent:* Thursday, October 18, 2018 10:45 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> At ReplaceText
> <https://raw.githubusercontent.com/apache/nifi/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java>processor
> we have:
>
>
>
> [image: image.png]
>
> As you can see, only if *StackOverflowError *is raised during the
> evaluation, the flowfile is send to failure relationship. I would like to
> update the code to use Exception or NifiExpressionFailedException (if it
> exits).
>
>
>
> Juan
>
>
>
> On Thu, 18 Oct 2018 at 12:33 Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> What processor are you defining your expression in? I also may be
> misunderstanding the problem because I don’t see any regular expressions
> anywhere. Can you create a sample workflow showing your issue so I can take
> a look at it.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
> *Sent:* Thursday, October 18, 2018 10:27 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> No, it's not a valid date. I would like if it an error happens, I would
> like to throw the flowfile to failure and continue.
>
>
>
> On Thu, 18 Oct 2018 at 12:19 Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> Any expression language syntax has to be correct or the processor won’t
> run. I’m not sure there is any way to work around that except to explicitly
> check that the value you are trying to evaluate is valid. Is the attribute
> “tradeDate” coming from the contents of a flow file or is it defined
> somewhere else. Can you ensure it is a valid date in that format before
> hand?
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
>
> *Sent:* Thursday, October 18, 2018 10:13 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Hi, the error is not in the processor itself. It's in the expression used
> against flowfile attributes. For example inside the text, I have:
>
>
>
>
> <EventDate>${tradeDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</EventDate>
>
> And that is the root issue. If it's unable to convert it, the flow cannot
> be consumed. How can I evaluate attributes in a non-blocker way?
>
>
>
> Juan
>
>
>
> On Thu, 18 Oct 2018 at 12:07 Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> Where is your expression? That’s not the entire configuration for that
> processor.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
> *Sent:* Thursday, October 18, 2018 10:03 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Configuration:
>
> Replacement Strategy: Always replace
>
> EvaluationMode: Entire text
>
>
>
>
>
> On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <
> gardellajuanpablo@gmail.com> wrote:
>
> Hortonworks nifi based on 1.5.0:
>
>
>
> Configuration:
>
> Thanks
>
>
>
> On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> Hi Juan,
>
>
>
> What version of NiFi are you running on?
>
> What mode are you running ReplaceText in, all text or line by line?
>
> Other settings that might be important? What’s your RegEx look like (if
> your able to share).
>
>
>
> --Peter
>
>
>
>
>
> *From:* Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com]
> *Sent:* Thursday, October 18, 2018 8:53 AM
> *To:* users@nifi.apache.org
> *Subject:* [EXT] ReplaceText cannot consume messages if Regex does not
> match
>
>
>
> Hi all,
>
>
>
> I'm seeing that ReplaceText is not able to consume messages that does not
> match regex. It keeps all the messages in the input queue instead of
> sending them to failure relationship. Is this the intended behavior or I
> have to file a ticket in order to be fixed? In that way, the processor is
> not able to process bad messages and converts in the bottleneck of a flow
>
>
>
> Juan
>
>
RE: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
I’m still trying to understand your actual issue, can your provide a screenshot of the ReplaceText config like the attached, I need to see exactly where you’re putting the expression. A template would also be really helpful.
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>
Sent: Thursday, October 18, 2018 10:45 AM
To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
At ReplaceText <https://raw.githubusercontent.com/apache/nifi/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java> processor we have:
[image.png]
As you can see, only if StackOverflowError is raised during the evaluation, the flowfile is send to failure relationship. I would like to update the code to use Exception or NifiExpressionFailedException (if it exits).
Juan
On Thu, 18 Oct 2018 at 12:33 Shawn Weeks <sw...@weeksconsulting.us>> wrote:
What processor are you defining your expression in? I also may be misunderstanding the problem because I don’t see any regular expressions anywhere. Can you create a sample workflow showing your issue so I can take a look at it.
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>>
Sent: Thursday, October 18, 2018 10:27 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
No, it's not a valid date. I would like if it an error happens, I would like to throw the flowfile to failure and continue.
On Thu, 18 Oct 2018 at 12:19 Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Any expression language syntax has to be correct or the processor won’t run. I’m not sure there is any way to work around that except to explicitly check that the value you are trying to evaluate is valid. Is the attribute “tradeDate” coming from the contents of a flow file or is it defined somewhere else. Can you ensure it is a valid date in that format before hand?
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>>
Sent: Thursday, October 18, 2018 10:13 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Hi, the error is not in the processor itself. It's in the expression used against flowfile attributes. For example inside the text, I have:
<EventDate>${tradeDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</EventDate>
And that is the root issue. If it's unable to convert it, the flow cannot be consumed. How can I evaluate attributes in a non-blocker way?
Juan
On Thu, 18 Oct 2018 at 12:07 Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Where is your expression? That’s not the entire configuration for that processor.
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>>
Sent: Thursday, October 18, 2018 10:03 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Configuration:
Replacement Strategy: Always replace
EvaluationMode: Entire text
On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <ga...@gmail.com>> wrote:
Hortonworks nifi based on 1.5.0:
Configuration:
Thanks
On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Hi Juan,
What version of NiFi are you running on?
What mode are you running ReplaceText in, all text or line by line?
Other settings that might be important? What’s your RegEx look like (if your able to share).
--Peter
From: Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com<ma...@gmail.com>]
Sent: Thursday, October 18, 2018 8:53 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: [EXT] ReplaceText cannot consume messages if Regex does not match
Hi all,
I'm seeing that ReplaceText is not able to consume messages that does not match regex. It keeps all the messages in the input queue instead of sending them to failure relationship. Is this the intended behavior or I have to file a ticket in order to be fixed? In that way, the processor is not able to process bad messages and converts in the bottleneck of a flow
Juan
Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Juan Pablo Gardella <ga...@gmail.com>.
At ReplaceText
<https://raw.githubusercontent.com/apache/nifi/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ReplaceText.java>processor
we have:
[image: image.png]
As you can see, only if *StackOverflowError *is raised during the
evaluation, the flowfile is send to failure relationship. I would like to
update the code to use Exception or NifiExpressionFailedException (if it
exits).
Juan
On Thu, 18 Oct 2018 at 12:33 Shawn Weeks <sw...@weeksconsulting.us> wrote:
> What processor are you defining your expression in? I also may be
> misunderstanding the problem because I don’t see any regular expressions
> anywhere. Can you create a sample workflow showing your issue so I can take
> a look at it.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
> *Sent:* Thursday, October 18, 2018 10:27 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> No, it's not a valid date. I would like if it an error happens, I would
> like to throw the flowfile to failure and continue.
>
>
>
> On Thu, 18 Oct 2018 at 12:19 Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> Any expression language syntax has to be correct or the processor won’t
> run. I’m not sure there is any way to work around that except to explicitly
> check that the value you are trying to evaluate is valid. Is the attribute
> “tradeDate” coming from the contents of a flow file or is it defined
> somewhere else. Can you ensure it is a valid date in that format before
> hand?
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
>
> *Sent:* Thursday, October 18, 2018 10:13 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Hi, the error is not in the processor itself. It's in the expression used
> against flowfile attributes. For example inside the text, I have:
>
>
>
>
> <EventDate>${tradeDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</EventDate>
>
> And that is the root issue. If it's unable to convert it, the flow cannot
> be consumed. How can I evaluate attributes in a non-blocker way?
>
>
>
> Juan
>
>
>
> On Thu, 18 Oct 2018 at 12:07 Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> Where is your expression? That’s not the entire configuration for that
> processor.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
> *Sent:* Thursday, October 18, 2018 10:03 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Configuration:
>
> Replacement Strategy: Always replace
>
> EvaluationMode: Entire text
>
>
>
>
>
> On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <
> gardellajuanpablo@gmail.com> wrote:
>
> Hortonworks nifi based on 1.5.0:
>
>
>
> Configuration:
>
> Thanks
>
>
>
> On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> Hi Juan,
>
>
>
> What version of NiFi are you running on?
>
> What mode are you running ReplaceText in, all text or line by line?
>
> Other settings that might be important? What’s your RegEx look like (if
> your able to share).
>
>
>
> --Peter
>
>
>
>
>
> *From:* Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com]
> *Sent:* Thursday, October 18, 2018 8:53 AM
> *To:* users@nifi.apache.org
> *Subject:* [EXT] ReplaceText cannot consume messages if Regex does not
> match
>
>
>
> Hi all,
>
>
>
> I'm seeing that ReplaceText is not able to consume messages that does not
> match regex. It keeps all the messages in the input queue instead of
> sending them to failure relationship. Is this the intended behavior or I
> have to file a ticket in order to be fixed? In that way, the processor is
> not able to process bad messages and converts in the bottleneck of a flow
>
>
>
> Juan
>
>
RE: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
What processor are you defining your expression in? I also may be misunderstanding the problem because I don’t see any regular expressions anywhere. Can you create a sample workflow showing your issue so I can take a look at it.
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>
Sent: Thursday, October 18, 2018 10:27 AM
To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
No, it's not a valid date. I would like if it an error happens, I would like to throw the flowfile to failure and continue.
On Thu, 18 Oct 2018 at 12:19 Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Any expression language syntax has to be correct or the processor won’t run. I’m not sure there is any way to work around that except to explicitly check that the value you are trying to evaluate is valid. Is the attribute “tradeDate” coming from the contents of a flow file or is it defined somewhere else. Can you ensure it is a valid date in that format before hand?
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>>
Sent: Thursday, October 18, 2018 10:13 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Hi, the error is not in the processor itself. It's in the expression used against flowfile attributes. For example inside the text, I have:
<EventDate>${tradeDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</EventDate>
And that is the root issue. If it's unable to convert it, the flow cannot be consumed. How can I evaluate attributes in a non-blocker way?
Juan
On Thu, 18 Oct 2018 at 12:07 Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Where is your expression? That’s not the entire configuration for that processor.
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>>
Sent: Thursday, October 18, 2018 10:03 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Configuration:
Replacement Strategy: Always replace
EvaluationMode: Entire text
On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <ga...@gmail.com>> wrote:
Hortonworks nifi based on 1.5.0:
Configuration:
Thanks
On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Hi Juan,
What version of NiFi are you running on?
What mode are you running ReplaceText in, all text or line by line?
Other settings that might be important? What’s your RegEx look like (if your able to share).
--Peter
From: Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com<ma...@gmail.com>]
Sent: Thursday, October 18, 2018 8:53 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: [EXT] ReplaceText cannot consume messages if Regex does not match
Hi all,
I'm seeing that ReplaceText is not able to consume messages that does not match regex. It keeps all the messages in the input queue instead of sending them to failure relationship. Is this the intended behavior or I have to file a ticket in order to be fixed? In that way, the processor is not able to process bad messages and converts in the bottleneck of a flow
Juan
Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Juan Pablo Gardella <ga...@gmail.com>.
No, it's not a valid date. I would like if it an error happens, I would
like to throw the flowfile to failure and continue.
On Thu, 18 Oct 2018 at 12:19 Shawn Weeks <sw...@weeksconsulting.us> wrote:
> Any expression language syntax has to be correct or the processor won’t
> run. I’m not sure there is any way to work around that except to explicitly
> check that the value you are trying to evaluate is valid. Is the attribute
> “tradeDate” coming from the contents of a flow file or is it defined
> somewhere else. Can you ensure it is a valid date in that format before
> hand?
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
>
> *Sent:* Thursday, October 18, 2018 10:13 AM
>
>
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Hi, the error is not in the processor itself. It's in the expression used
> against flowfile attributes. For example inside the text, I have:
>
>
>
>
> <EventDate>${tradeDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</EventDate>
>
> And that is the root issue. If it's unable to convert it, the flow cannot
> be consumed. How can I evaluate attributes in a non-blocker way?
>
>
>
> Juan
>
>
>
> On Thu, 18 Oct 2018 at 12:07 Shawn Weeks <sw...@weeksconsulting.us>
> wrote:
>
> Where is your expression? That’s not the entire configuration for that
> processor.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
> *Sent:* Thursday, October 18, 2018 10:03 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Configuration:
>
> Replacement Strategy: Always replace
>
> EvaluationMode: Entire text
>
>
>
>
>
> On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <
> gardellajuanpablo@gmail.com> wrote:
>
> Hortonworks nifi based on 1.5.0:
>
> [image: image.png]
>
>
>
> Configuration:
>
> Thanks
>
>
>
> On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> Hi Juan,
>
>
>
> What version of NiFi are you running on?
>
> What mode are you running ReplaceText in, all text or line by line?
>
> Other settings that might be important? What’s your RegEx look like (if
> your able to share).
>
>
>
> --Peter
>
>
>
>
>
> *From:* Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com]
> *Sent:* Thursday, October 18, 2018 8:53 AM
> *To:* users@nifi.apache.org
> *Subject:* [EXT] ReplaceText cannot consume messages if Regex does not
> match
>
>
>
> Hi all,
>
>
>
> I'm seeing that ReplaceText is not able to consume messages that does not
> match regex. It keeps all the messages in the input queue instead of
> sending them to failure relationship. Is this the intended behavior or I
> have to file a ticket in order to be fixed? In that way, the processor is
> not able to process bad messages and converts in the bottleneck of a flow
>
>
>
> Juan
>
>
RE: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
Any expression language syntax has to be correct or the processor won’t run. I’m not sure there is any way to work around that except to explicitly check that the value you are trying to evaluate is valid. Is the attribute “tradeDate” coming from the contents of a flow file or is it defined somewhere else. Can you ensure it is a valid date in that format before hand?
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>
Sent: Thursday, October 18, 2018 10:13 AM
To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Hi, the error is not in the processor itself. It's in the expression used against flowfile attributes. For example inside the text, I have:
<EventDate>${tradeDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</EventDate>
And that is the root issue. If it's unable to convert it, the flow cannot be consumed. How can I evaluate attributes in a non-blocker way?
Juan
On Thu, 18 Oct 2018 at 12:07 Shawn Weeks <sw...@weeksconsulting.us>> wrote:
Where is your expression? That’s not the entire configuration for that processor.
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>>
Sent: Thursday, October 18, 2018 10:03 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Configuration:
Replacement Strategy: Always replace
EvaluationMode: Entire text
On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <ga...@gmail.com>> wrote:
Hortonworks nifi based on 1.5.0:
[image.png]
Configuration:
Thanks
On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Hi Juan,
What version of NiFi are you running on?
What mode are you running ReplaceText in, all text or line by line?
Other settings that might be important? What’s your RegEx look like (if your able to share).
--Peter
From: Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com<ma...@gmail.com>]
Sent: Thursday, October 18, 2018 8:53 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: [EXT] ReplaceText cannot consume messages if Regex does not match
Hi all,
I'm seeing that ReplaceText is not able to consume messages that does not match regex. It keeps all the messages in the input queue instead of sending them to failure relationship. Is this the intended behavior or I have to file a ticket in order to be fixed? In that way, the processor is not able to process bad messages and converts in the bottleneck of a flow
Juan
Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Juan Pablo Gardella <ga...@gmail.com>.
Hi, the error is not in the processor itself. It's in the expression used
against flowfile attributes. For example inside the text, I have:
<EventDate>${tradeDate:toDate('MM/dd/yyyy'):format("yyyy-MM-dd'T'00:00:00.0000000")}</EventDate>
And that is the root issue. If it's unable to convert it, the flow cannot
be consumed. How can I evaluate attributes in a non-blocker way?
Juan
On Thu, 18 Oct 2018 at 12:07 Shawn Weeks <sw...@weeksconsulting.us> wrote:
> Where is your expression? That’s not the entire configuration for that
> processor.
>
>
>
> Thanks
>
> Shawn Weeks
>
>
>
> *From:* Juan Pablo Gardella <ga...@gmail.com>
> *Sent:* Thursday, October 18, 2018 10:03 AM
> *To:* users@nifi.apache.org
> *Subject:* Re: [EXT] ReplaceText cannot consume messages if Regex does
> not match
>
>
>
> Configuration:
>
> Replacement Strategy: Always replace
>
> EvaluationMode: Entire text
>
>
>
>
>
> On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <
> gardellajuanpablo@gmail.com> wrote:
>
> Hortonworks nifi based on 1.5.0:
>
> [image: image.png]
>
>
>
> Configuration:
>
> Thanks
>
>
>
> On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
> Hi Juan,
>
>
>
> What version of NiFi are you running on?
>
> What mode are you running ReplaceText in, all text or line by line?
>
> Other settings that might be important? What’s your RegEx look like (if
> your able to share).
>
>
>
> --Peter
>
>
>
>
>
> *From:* Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com]
> *Sent:* Thursday, October 18, 2018 8:53 AM
> *To:* users@nifi.apache.org
> *Subject:* [EXT] ReplaceText cannot consume messages if Regex does not
> match
>
>
>
> Hi all,
>
>
>
> I'm seeing that ReplaceText is not able to consume messages that does not
> match regex. It keeps all the messages in the input queue instead of
> sending them to failure relationship. Is this the intended behavior or I
> have to file a ticket in order to be fixed? In that way, the processor is
> not able to process bad messages and converts in the bottleneck of a flow
>
>
>
> Juan
>
>
RE: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Shawn Weeks <sw...@weeksconsulting.us>.
Where is your expression? That’s not the entire configuration for that processor.
Thanks
Shawn Weeks
From: Juan Pablo Gardella <ga...@gmail.com>
Sent: Thursday, October 18, 2018 10:03 AM
To: users@nifi.apache.org
Subject: Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Configuration:
Replacement Strategy: Always replace
EvaluationMode: Entire text
On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <ga...@gmail.com>> wrote:
Hortonworks nifi based on 1.5.0:
[image.png]
Configuration:
Thanks
On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com>> wrote:
Hi Juan,
What version of NiFi are you running on?
What mode are you running ReplaceText in, all text or line by line?
Other settings that might be important? What’s your RegEx look like (if your able to share).
--Peter
From: Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com<ma...@gmail.com>]
Sent: Thursday, October 18, 2018 8:53 AM
To: users@nifi.apache.org<ma...@nifi.apache.org>
Subject: [EXT] ReplaceText cannot consume messages if Regex does not match
Hi all,
I'm seeing that ReplaceText is not able to consume messages that does not match regex. It keeps all the messages in the input queue instead of sending them to failure relationship. Is this the intended behavior or I have to file a ticket in order to be fixed? In that way, the processor is not able to process bad messages and converts in the bottleneck of a flow
Juan
Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Juan Pablo Gardella <ga...@gmail.com>.
Configuration:
Replacement Strategy: Always replace
EvaluationMode: Entire text
On Thu, 18 Oct 2018 at 12:01 Juan Pablo Gardella <
gardellajuanpablo@gmail.com> wrote:
> Hortonworks nifi based on 1.5.0:
> [image: image.png]
>
> Configuration:
> [image: image.png]
> Thanks
>
> On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com>
> wrote:
>
>> Hi Juan,
>>
>>
>>
>> What version of NiFi are you running on?
>>
>> What mode are you running ReplaceText in, all text or line by line?
>>
>> Other settings that might be important? What’s your RegEx look like (if
>> your able to share).
>>
>>
>>
>> --Peter
>>
>>
>>
>>
>>
>> *From:* Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com]
>> *Sent:* Thursday, October 18, 2018 8:53 AM
>> *To:* users@nifi.apache.org
>> *Subject:* [EXT] ReplaceText cannot consume messages if Regex does not
>> match
>>
>>
>>
>> Hi all,
>>
>>
>>
>> I'm seeing that ReplaceText is not able to consume messages that does not
>> match regex. It keeps all the messages in the input queue instead of
>> sending them to failure relationship. Is this the intended behavior or I
>> have to file a ticket in order to be fixed? In that way, the processor is
>> not able to process bad messages and converts in the bottleneck of a flow
>>
>>
>>
>> Juan
>>
>
Re: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by Juan Pablo Gardella <ga...@gmail.com>.
Hortonworks nifi based on 1.5.0:
[image: image.png]
Configuration:
[image: image.png]
Thanks
On Thu, 18 Oct 2018 at 11:56 Peter Wicks (pwicks) <pw...@micron.com> wrote:
> Hi Juan,
>
>
>
> What version of NiFi are you running on?
>
> What mode are you running ReplaceText in, all text or line by line?
>
> Other settings that might be important? What’s your RegEx look like (if
> your able to share).
>
>
>
> --Peter
>
>
>
>
>
> *From:* Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com]
> *Sent:* Thursday, October 18, 2018 8:53 AM
> *To:* users@nifi.apache.org
> *Subject:* [EXT] ReplaceText cannot consume messages if Regex does not
> match
>
>
>
> Hi all,
>
>
>
> I'm seeing that ReplaceText is not able to consume messages that does not
> match regex. It keeps all the messages in the input queue instead of
> sending them to failure relationship. Is this the intended behavior or I
> have to file a ticket in order to be fixed? In that way, the processor is
> not able to process bad messages and converts in the bottleneck of a flow
>
>
>
> Juan
>
RE: [EXT] ReplaceText cannot consume messages if Regex does not match
Posted by "Peter Wicks (pwicks)" <pw...@micron.com>.
Hi Juan,
What version of NiFi are you running on?
What mode are you running ReplaceText in, all text or line by line?
Other settings that might be important? What’s your RegEx look like (if your able to share).
--Peter
From: Juan Pablo Gardella [mailto:gardellajuanpablo@gmail.com]
Sent: Thursday, October 18, 2018 8:53 AM
To: users@nifi.apache.org
Subject: [EXT] ReplaceText cannot consume messages if Regex does not match
Hi all,
I'm seeing that ReplaceText is not able to consume messages that does not match regex. It keeps all the messages in the input queue instead of sending them to failure relationship. Is this the intended behavior or I have to file a ticket in order to be fixed? In that way, the processor is not able to process bad messages and converts in the bottleneck of a flow
Juan