You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Martin Zink <ma...@apache.org> on 2023/03/02 12:09:03 UTC
RFC3339 parsing with Expression Language
When dealing with timestamps coming from third parties, sometimes we cannot
rely on exact timestamp formats, so it is difficult to parse them using
toDate
<https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#todate>.
(e.g. Syslog messages allow a subset of RFC3339 for timestamps).
To circumvent this I started working on an expression language function
that could parse all RFC3339 compliant timestamps with expression
language in MiNiFi C++, but I feel this functionality would also be a great
addition to NiFi.
It would be great if the expression language would remain consistent
between MiNiFi C++ and NiFi, and I'm not quite sure where this
functionality should be accessible from.
- We could either change the current toDate
<https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#todate>
to
accept zero arguments and calling it without argument would try to parse
the string as RFC3339 compliant.
- or we could introduce a new expression language function e.g.
parseRfc3339
What's the community's stance on this?
Thanks,
Martin
Re: RFC3339 parsing with Expression Language
Posted by Marton Szasz <sz...@apache.org>.
We've already talked about this with Martin and other contributors,
but leaving my 2c here for the wider discussion.
I'm leaning towards the second option, having a separate EL function,
because parsing the RFC-compliant dates does not directly map to
strptime-like format strings used in toDate. In python, they also have
separate functions for parsing like strptime [1] (like the EL toDate)
and parsing RFC 3339 or ISO 8601 dates [2]. Most other MiNiFi C++ devs
are leaning towards the first option, extending toDate with an
overload that doesn't take a format string.
Ultimately I think we should follow whatever interface the NiFi devs
prefer, so if the NiFi Expression Language is ever extended with a
similar RFC/ISO date parser facility in the future, the MiNiFi C++
Expression Language would match that. This way users have an easier
time converting their flows between the two implementations.
Thanks,
Marton
[1] https://docs.python.org/3/library/datetime.html#datetime.datetime.strptime
[2] https://docs.python.org/3/library/datetime.html#datetime.datetime.fromisoformat
and https://stackoverflow.com/a/15228038
On Thu, 2 Mar 2023 at 13:09, Martin Zink <ma...@apache.org> wrote:
>
> When dealing with timestamps coming from third parties, sometimes we cannot
> rely on exact timestamp formats, so it is difficult to parse them using
> toDate
> <https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#todate>.
> (e.g. Syslog messages allow a subset of RFC3339 for timestamps).
> To circumvent this I started working on an expression language function
> that could parse all RFC3339 compliant timestamps with expression
> language in MiNiFi C++, but I feel this functionality would also be a great
> addition to NiFi.
>
> It would be great if the expression language would remain consistent
> between MiNiFi C++ and NiFi, and I'm not quite sure where this
> functionality should be accessible from.
>
> - We could either change the current toDate
> <https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#todate>
> to
> accept zero arguments and calling it without argument would try to parse
> the string as RFC3339 compliant.
> - or we could introduce a new expression language function e.g.
> parseRfc3339
>
>
> What's the community's stance on this?
>
> Thanks,
> Martin