You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Juan Sequeiros <he...@gmail.com> on 2018/08/07 15:02:55 UTC
JoltTransformJson issues with date enrichment
Hello all,
I am trying to "enrich" my json.
My problem happens when I try to give it a timestamp.
My requirement is to give it time in this format:
${now():format("yyyy-MM-dd HH:mm",'Z')}
So at the top level and every element of my json should have an entry
called
EX:
"createTime":"2018-08-07 14:12"
To do this I am creating an attribute "createTime" and setting it to
"${now():format("yyyy-MM-dd HH:mm",'Z')}"
I then pass that to JoltTransform but it fails with:
"Unable to unmarshal JSON to object"
If I set my createTime attribute to hard code EX: "2018008-07 09:15"
Then it works.
I've also tried setting my createTime toString but that does not help.
What helps is if I set my createTime to ${createTime:escapeJson()}
But then elasticSearch does not like it since it has a "\n" in the end and
the elasticSearch index uses schema validation.
At this stage, we have not sprint planned to use records so when I ingest
to ES I am using PutElasticsearchHttp
TLDR:
JoltTransform errors out when I pass it an attribute set by calling the
now() function.
My Jolt spec looks like this:
Jolt Transformation DSL
Chain
[{
"operation": "default",
"spec": {
"isVerified":"TRUE",
"recentResultDate" : "",
"createDate" : "${createDate}",
"*":{
"resultCode":"GREEN",
"resultDate":"${createDate}"
}
}
}]
--
Juan Carlos Sequeiros
Re: JoltTransformJson issues with date enrichment
Posted by Juan Sequeiros <he...@gmail.com>.
Matt thanks for the quick reply.
So I changed the attribute name and set it to now()
and it worked.
I then changed it back to ${now():format("yyyy-MM-dd HH:mm",'Z')}
and it worked!!! too
Either I did have something funky with my attribute name on the spec or I
wonder if the setting in JoltTransformJSON "Transform Cache Size" had
anything to do with it since before reaching out to you all I was trying
all kinds of things.
I'm on 1.4 by the way but all is well.
Thanks!
On Tue, Aug 7, 2018 at 11:16 AM Matt Burgess <ma...@apache.org> wrote:
> Juan,
>
> What version of NiFi are you using? I tried with the latest master,
> using a GenerateFlowFile (where I set the attribute using the now()
> expression) into JoltTransformJSON with your spec above, and it works
> fine. Are you sure there's not a double-quote missing somewhere, or
> something extra in your EL expression?
>
> Regards,
> Matt
>
> On Tue, Aug 7, 2018 at 11:03 AM Juan Sequeiros <he...@gmail.com>
> wrote:
> >
> > Hello all,
> >
> > I am trying to "enrich" my json.
> > My problem happens when I try to give it a timestamp.
> > My requirement is to give it time in this format:
> > ${now():format("yyyy-MM-dd HH:mm",'Z')}
> >
> > So at the top level and every element of my json should have an entry
> called
> >
> > EX:
> >
> > "createTime":"2018-08-07 14:12"
> >
> > To do this I am creating an attribute "createTime" and setting it to
> "${now():format("yyyy-MM-dd HH:mm",'Z')}"
> >
> > I then pass that to JoltTransform but it fails with:
> >
> > "Unable to unmarshal JSON to object"
> >
> > If I set my createTime attribute to hard code EX: "2018008-07 09:15"
> > Then it works.
> >
> > I've also tried setting my createTime toString but that does not help.
> > What helps is if I set my createTime to ${createTime:escapeJson()}
> >
> > But then elasticSearch does not like it since it has a "\n" in the end
> and the elasticSearch index uses schema validation.
> > At this stage, we have not sprint planned to use records so when I
> ingest to ES I am using PutElasticsearchHttp
> >
> > TLDR:
> >
> > JoltTransform errors out when I pass it an attribute set by calling the
> now() function.
> >
> > My Jolt spec looks like this:
> >
> > Jolt Transformation DSL
> > Chain
> > [{
> > "operation": "default",
> > "spec": {
> > "isVerified":"TRUE",
> > "recentResultDate" : "",
> > "createDate" : "${createDate}",
> >
> > "*":{
> > "resultCode":"GREEN",
> > "resultDate":"${createDate}"
> > }
> > }
> > }]
> > --
> > Juan Carlos Sequeiros
>
--
Juan Carlos Sequeiros
Re: JoltTransformJson issues with date enrichment
Posted by Matt Burgess <ma...@apache.org>.
Juan,
What version of NiFi are you using? I tried with the latest master,
using a GenerateFlowFile (where I set the attribute using the now()
expression) into JoltTransformJSON with your spec above, and it works
fine. Are you sure there's not a double-quote missing somewhere, or
something extra in your EL expression?
Regards,
Matt
On Tue, Aug 7, 2018 at 11:03 AM Juan Sequeiros <he...@gmail.com> wrote:
>
> Hello all,
>
> I am trying to "enrich" my json.
> My problem happens when I try to give it a timestamp.
> My requirement is to give it time in this format:
> ${now():format("yyyy-MM-dd HH:mm",'Z')}
>
> So at the top level and every element of my json should have an entry called
>
> EX:
>
> "createTime":"2018-08-07 14:12"
>
> To do this I am creating an attribute "createTime" and setting it to "${now():format("yyyy-MM-dd HH:mm",'Z')}"
>
> I then pass that to JoltTransform but it fails with:
>
> "Unable to unmarshal JSON to object"
>
> If I set my createTime attribute to hard code EX: "2018008-07 09:15"
> Then it works.
>
> I've also tried setting my createTime toString but that does not help.
> What helps is if I set my createTime to ${createTime:escapeJson()}
>
> But then elasticSearch does not like it since it has a "\n" in the end and the elasticSearch index uses schema validation.
> At this stage, we have not sprint planned to use records so when I ingest to ES I am using PutElasticsearchHttp
>
> TLDR:
>
> JoltTransform errors out when I pass it an attribute set by calling the now() function.
>
> My Jolt spec looks like this:
>
> Jolt Transformation DSL
> Chain
> [{
> "operation": "default",
> "spec": {
> "isVerified":"TRUE",
> "recentResultDate" : "",
> "createDate" : "${createDate}",
>
> "*":{
> "resultCode":"GREEN",
> "resultDate":"${createDate}"
> }
> }
> }]
> --
> Juan Carlos Sequeiros