You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@nifi.apache.org by Igor Kravzov <ig...@gmail.com> on 2016/06/09 14:37:52 UTC

Best way to compare dates

Hi,

In my workflow data is coming form Kafka Topic in JSON format and saved ot
HDFS directory by day.

HDFS directory path constructed as /mypath/${now():format("yyyyMMdd")}
JSON file contains attribute {... "adddate":"2016-06-07 17:06:17" }

I want to avoid over-spill of a previous date files into "today's"
directory.
The thing is if add date is the same as taday's date, a want file merged to
a bigger file and save to HDFS. Previous file not merged and saved as is.

I am thinking to use EvaluateJsonPath to grab add date and to use
RouteOnAttribute to direct to appropriate  path.

How comparison would look like?

Thanks in advance.

Re: Best way to compare dates

Posted by Igor Kravzov <ig...@gmail.com>.

Thanks James. That's exactly what I ended up with :)

On Thu, Jun 9, 2016 at 12:48 PM, James Wing <jv...@gmail.com> wrote:

> Igor,
>
> One way would be to format both dates as strings (like "20160609" in your
> HDFS paths) first, then compare the two strings for equality.  In a
> RouteOnAttribute expression:
>
> ${now():format("yyyyMMdd"):equals(${entryDate:format("yyyyMMdd")})}
>
> If your goal is to merge the records into larger files, you might be able
> to do this with MergeContent, if you have the Correlation Attribute set to
> an attribute populated with the yyyyMMdd-formatteddate so it bins files by
> day.
>
> Thanks,
>
> James
>
>
> On Thu, Jun 9, 2016 at 7:37 AM, Igor Kravzov <ig...@gmail.com>
> wrote:
>
>> Hi,
>>
>> In my workflow data is coming form Kafka Topic in JSON format and saved
>> ot HDFS directory by day.
>>
>> HDFS directory path constructed as /mypath/${now():format("yyyyMMdd")}
>> JSON file contains attribute {... "adddate":"2016-06-07 17:06:17" }
>>
>> I want to avoid over-spill of a previous date files into "today's"
>> directory.
>> The thing is if add date is the same as taday's date, a want file merged
>> to a bigger file and save to HDFS. Previous file not merged and saved as is.
>>
>> I am thinking to use EvaluateJsonPath to grab add date and to use
>> RouteOnAttribute to direct to appropriate  path.
>>
>> How comparison would look like?
>>
>> Thanks in advance.
>>
>
>

Re: Best way to compare dates

Posted by James Wing <jv...@gmail.com>.

Igor,

One way would be to format both dates as strings (like "20160609" in your
HDFS paths) first, then compare the two strings for equality.  In a
RouteOnAttribute expression:

${now():format("yyyyMMdd"):equals(${entryDate:format("yyyyMMdd")})}

If your goal is to merge the records into larger files, you might be able
to do this with MergeContent, if you have the Correlation Attribute set to
an attribute populated with the yyyyMMdd-formatteddate so it bins files by
day.

Thanks,

James

On Thu, Jun 9, 2016 at 7:37 AM, Igor Kravzov <ig...@gmail.com> wrote:

> Hi,
>
> In my workflow data is coming form Kafka Topic in JSON format and saved ot
> HDFS directory by day.
>
> HDFS directory path constructed as /mypath/${now():format("yyyyMMdd")}
> JSON file contains attribute {... "adddate":"2016-06-07 17:06:17" }
>
> I want to avoid over-spill of a previous date files into "today's"
> directory.
> The thing is if add date is the same as taday's date, a want file merged
> to a bigger file and save to HDFS. Previous file not merged and saved as is.
>
> I am thinking to use EvaluateJsonPath to grab add date and to use
> RouteOnAttribute to direct to appropriate  path.
>
> How comparison would look like?
>
> Thanks in advance.
>