You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Mike Thomsen <mi...@gmail.com> on 2017/06/23 13:06:30 UTC
Representing date queries properly with GetMongo
I am trying to write a query for GetMongo that gives me documents added in
the last five minutes. It looks like this:
{
"ts": {
"$gte": new Date(ISODate().getTime() - (1000 * 60 * 5))
}
}
The processor goes to an invalid state because it says "query validated
against [that query] is invalid because org.bosn.json.JsonParseException."
Does anyone know how to convert that query from the Mongo command
shell-acceptable syntax into one NiFi can accept?
Thanks,
Mike
Re: Representing date queries properly with GetMongo
Posted by Andy LoPresto <al...@apache.org>.
Mike,
I was unclear — the expression language has to be evaluated during the UpdateAttribute processor currently, but with the resolution of the ticket I referenced, you would be able to evaluate it inline in the GetMongo processor.
Glad you found a solution that works for you and thank you for sharing.
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 23, 2017, at 11:45 AM, Mike Thomsen <mi...@gmail.com> wrote:
>
> Thanks Andy. Didn't quite do the job because EL is not usable on the query field in my version, but this is what I did in case someone else wants to try it:
>
> Mongo console:
>
> db.system.js.save({
> "_id": "lastFiveMinutes",
> "value": function() {
> return new Date(ISODate().getTime() - (1000 * 60 * 5));
> }
> });
> db.loadServerScripts();
>
> Query field:
>
> {
> "$where": "obj.ts >= lastFiveMinutes()"
> }
>
> On Fri, Jun 23, 2017 at 10:08 AM, Andy LoPresto <alopresto@apache.org <ma...@apache.org>> wrote:
> Mike,
>
> The query is failing JSON validation because the value of the “$gte” key is not valid JSON. You should be able to use a combination of NiFi Expression Language [1] and the UpdateAttribute processor [2] to populate the expected date value in your query. Once NIFI-4082 [3] is implemented, you won’t need the UpdateAttribute processor anymore.
>
> [1] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html <https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html>
> [2] https://stackoverflow.com/a/44585760/70465 <https://stackoverflow.com/a/44585760/70465>
> [3] https://issues.apache.org/jira/browse/NIFI-4082 <https://issues.apache.org/jira/browse/NIFI-4082>
>
> Andy LoPresto
> alopresto@apache.org <ma...@apache.org>
> alopresto.apache@gmail.com <ma...@gmail.com>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
>> On Jun 23, 2017, at 9:06 AM, Mike Thomsen <mikerthomsen@gmail.com <ma...@gmail.com>> wrote:
>>
>> I am trying to write a query for GetMongo that gives me documents added in
>> the last five minutes. It looks like this:
>>
>> {
>> "ts": {
>> "$gte": new Date(ISODate().getTime() - (1000 * 60 * 5))
>> }
>> }
>>
>> The processor goes to an invalid state because it says "query validated
>> against [that query] is invalid because org.bosn.json.JsonParseException."
>> Does anyone know how to convert that query from the Mongo command
>> shell-acceptable syntax into one NiFi can accept?
>>
>> Thanks,
>>
>> Mike
>
>
Re: Representing date queries properly with GetMongo
Posted by Mike Thomsen <mi...@gmail.com>.
Thanks Andy. Didn't quite do the job because EL is not usable on the query
field in my version, but this is what I did in case someone else wants to
try it:
Mongo console:
db.system.js.save({
"_id": "lastFiveMinutes",
"value": function() {
return new Date(ISODate().getTime() - (1000 * 60 * 5));
}
});
db.loadServerScripts();
Query field:
{
"$where": "obj.ts >= lastFiveMinutes()"
}
On Fri, Jun 23, 2017 at 10:08 AM, Andy LoPresto <al...@apache.org>
wrote:
> Mike,
>
> The query is failing JSON validation because the value of the “$gte” key
> is not valid JSON. You should be able to use a combination of NiFi
> Expression Language [1] and the UpdateAttribute processor [2] to populate
> the expected date value in your query. Once NIFI-4082 [3] is implemented,
> you won’t need the UpdateAttribute processor anymore.
>
> [1] https://nifi.apache.org/docs/nifi-docs/html/
> expression-language-guide.html
> [2] https://stackoverflow.com/a/44585760/70465
> [3] https://issues.apache.org/jira/browse/NIFI-4082
>
> Andy LoPresto
> alopresto@apache.org
> *alopresto.apache@gmail.com <al...@gmail.com>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> On Jun 23, 2017, at 9:06 AM, Mike Thomsen <mi...@gmail.com> wrote:
>
> I am trying to write a query for GetMongo that gives me documents added in
> the last five minutes. It looks like this:
>
> {
> "ts": {
> "$gte": new Date(ISODate().getTime() - (1000 * 60 * 5))
> }
> }
>
> The processor goes to an invalid state because it says "query validated
> against [that query] is invalid because org.bosn.json.JsonParseException."
> Does anyone know how to convert that query from the Mongo command
> shell-acceptable syntax into one NiFi can accept?
>
> Thanks,
>
> Mike
>
>
>
Re: Representing date queries properly with GetMongo
Posted by Andy LoPresto <al...@apache.org>.
Mike,
The query is failing JSON validation because the value of the “$gte” key is not valid JSON. You should be able to use a combination of NiFi Expression Language [1] and the UpdateAttribute processor [2] to populate the expected date value in your query. Once NIFI-4082 [3] is implemented, you won’t need the UpdateAttribute processor anymore.
[1] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html
[2] https://stackoverflow.com/a/44585760/70465
[3] https://issues.apache.org/jira/browse/NIFI-4082 <https://issues.apache.org/jira/browse/NIFI-4082>
Andy LoPresto
alopresto@apache.org
alopresto.apache@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Jun 23, 2017, at 9:06 AM, Mike Thomsen <mi...@gmail.com> wrote:
>
> I am trying to write a query for GetMongo that gives me documents added in
> the last five minutes. It looks like this:
>
> {
> "ts": {
> "$gte": new Date(ISODate().getTime() - (1000 * 60 * 5))
> }
> }
>
> The processor goes to an invalid state because it says "query validated
> against [that query] is invalid because org.bosn.json.JsonParseException."
> Does anyone know how to convert that query from the Mongo command
> shell-acceptable syntax into one NiFi can accept?
>
> Thanks,
>
> Mike