You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Ryan Hendrickson <ry...@gmail.com> on 2018/08/07 16:54:43 UTC
GetMongo Truncates Dates
Hi all,
I'm using GetMongo configured with JSON Type of "Standard JSON". The
document I've got in Mongo has a date field that looks like the following:
{
...
"date" : ISODate("2018-08-06T16:20:10.912Z"
...
}
When GetMongo spits it out, the date comes out as:
"2018-08-06T16:20:10Z", noticeably missing the milliseconds.
I've created a bug ticket here:
https://issues.apache.org/jira/browse/NIFI-5495.
I'm not sure if there's a work around or anything like that. If anyone
else has suggestions to either add back the missing milliseconds, even if
it's just tagging on ".000" to get the format back, I'm all ears.
Thanks,
Ryan
Re: GetMongo Truncates Dates
Posted by Otto Fowler <ot...@gmail.com>.
I commented the jira about that. We do have validators, or at least the
start of some, but they are in the serialization libraries and would need
to be moved to be usable by both processors and records.
On August 7, 2018 at 13:58:27, Ryan Hendrickson (
ryan.andrew.hendrickson@gmail.com) wrote:
Yea, just finished testing, it works.. I added an additional property
called "Date Format" and I passed in a java date format there. Now I'm
just wondering if there's a way to validate it's a good date format before
it tries to use it when it's passing docs through. Maybe a custom
validator?
Ryan
On Tue, Aug 7, 2018 at 1:53 PM Ryan Hendrickson <
ryan.andrew.hendrickson@gmail.com> wrote:
> Haha, yea I'm testing out some code-hacking here to see what I can do
> too. If it works, I can try to submit a Pull Request for it. I haven't
> done one before, and this seems pretty easy to make it configurable.
>
> Ryan
>
> On Tue, Aug 7, 2018 at 1:48 PM Mike Thomsen <mi...@gmail.com>
> wrote:
>
>> Annnndd, I should have read your comment on the Jira ticket because it's
>> even easier than that!
>>
>> On Tue, Aug 7, 2018 at 1:47 PM Mike Thomsen <mi...@gmail.com>
>> wrote:
>>
>>> I just checked the code, and it's using the default Jackson mapping
>>> behavior for that. The Mongo driver returns a Date, and looks like Jackson
>>> is just turning that into an ISO8601 string without that level of
>>> precision. A custom mapper for Date objects should be able to solve that.
>>> I'll work it when I get some free time from the daily grind.
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>> On Tue, Aug 7, 2018 at 12:55 PM Ryan Hendrickson <
>>> ryan.andrew.hendrickson@gmail.com> wrote:
>>>
>>>> Hi all,
>>>> I'm using GetMongo configured with JSON Type of "Standard JSON".
>>>> The document I've got in Mongo has a date field that looks like the
>>>> following:
>>>> {
>>>> ...
>>>> "date" : ISODate("2018-08-06T16:20:10.912Z"
>>>> ...
>>>> }
>>>>
>>>> When GetMongo spits it out, the date comes out as:
>>>> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
>>>>
>>>> I've created a bug ticket here:
>>>> https://issues.apache.org/jira/browse/NIFI-5495.
>>>>
>>>> I'm not sure if there's a work around or anything like that. If
>>>> anyone else has suggestions to either add back the missing milliseconds,
>>>> even if it's just tagging on ".000" to get the format back, I'm all ears.
>>>>
>>>> Thanks,
>>>> Ryan
>>>>
>>>
Re: GetMongo Truncates Dates
Posted by Ryan Hendrickson <ry...@gmail.com>.
Thanks Mike, I missed the email (all my NiFi goes to a folder). Glad to
see it's in there. I'll get my first PR with something else!
Ryan
On Mon, Aug 27, 2018 at 8:26 AM Mike Thomsen <mi...@gmail.com> wrote:
> Pushed a PR for this.
>
> On Thu, Aug 23, 2018 at 9:23 PM Mike Thomsen <mi...@gmail.com>
> wrote:
>
>> Ryan,
>>
>> Didn't see you put out a pull request in the last two weeks. Let me know
>> if you're actively working this, if not I can do my own patch.
>>
>> Thanks,
>>
>> Mike
>>
>> On Wed, Aug 8, 2018 at 6:03 AM Mike Thomsen <mi...@gmail.com>
>> wrote:
>>
>>> Yes. Use a custom validator.
>>>
>>> On Tue, Aug 7, 2018 at 1:58 PM Ryan Hendrickson <
>>> ryan.andrew.hendrickson@gmail.com> wrote:
>>>
>>>> Yea, just finished testing, it works.. I added an additional property
>>>> called "Date Format" and I passed in a java date format there. Now I'm
>>>> just wondering if there's a way to validate it's a good date format before
>>>> it tries to use it when it's passing docs through. Maybe a custom
>>>> validator?
>>>>
>>>> Ryan
>>>>
>>>> On Tue, Aug 7, 2018 at 1:53 PM Ryan Hendrickson <
>>>> ryan.andrew.hendrickson@gmail.com> wrote:
>>>>
>>>>> Haha, yea I'm testing out some code-hacking here to see what I can do
>>>>> too. If it works, I can try to submit a Pull Request for it. I haven't
>>>>> done one before, and this seems pretty easy to make it configurable.
>>>>>
>>>>> Ryan
>>>>>
>>>>> On Tue, Aug 7, 2018 at 1:48 PM Mike Thomsen <mi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Annnndd, I should have read your comment on the Jira ticket because
>>>>>> it's even easier than that!
>>>>>>
>>>>>> On Tue, Aug 7, 2018 at 1:47 PM Mike Thomsen <mi...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I just checked the code, and it's using the default Jackson mapping
>>>>>>> behavior for that. The Mongo driver returns a Date, and looks like Jackson
>>>>>>> is just turning that into an ISO8601 string without that level of
>>>>>>> precision. A custom mapper for Date objects should be able to solve that.
>>>>>>> I'll work it when I get some free time from the daily grind.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Mike
>>>>>>>
>>>>>>> On Tue, Aug 7, 2018 at 12:55 PM Ryan Hendrickson <
>>>>>>> ryan.andrew.hendrickson@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>> I'm using GetMongo configured with JSON Type of "Standard
>>>>>>>> JSON". The document I've got in Mongo has a date field that looks like the
>>>>>>>> following:
>>>>>>>> {
>>>>>>>> ...
>>>>>>>> "date" : ISODate("2018-08-06T16:20:10.912Z"
>>>>>>>> ...
>>>>>>>> }
>>>>>>>>
>>>>>>>> When GetMongo spits it out, the date comes out as:
>>>>>>>> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
>>>>>>>>
>>>>>>>> I've created a bug ticket here:
>>>>>>>> https://issues.apache.org/jira/browse/NIFI-5495.
>>>>>>>>
>>>>>>>> I'm not sure if there's a work around or anything like that. If
>>>>>>>> anyone else has suggestions to either add back the missing milliseconds,
>>>>>>>> even if it's just tagging on ".000" to get the format back, I'm all ears.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Ryan
>>>>>>>>
>>>>>>>
Re: GetMongo Truncates Dates
Posted by Mike Thomsen <mi...@gmail.com>.
Pushed a PR for this.
On Thu, Aug 23, 2018 at 9:23 PM Mike Thomsen <mi...@gmail.com> wrote:
> Ryan,
>
> Didn't see you put out a pull request in the last two weeks. Let me know
> if you're actively working this, if not I can do my own patch.
>
> Thanks,
>
> Mike
>
> On Wed, Aug 8, 2018 at 6:03 AM Mike Thomsen <mi...@gmail.com>
> wrote:
>
>> Yes. Use a custom validator.
>>
>> On Tue, Aug 7, 2018 at 1:58 PM Ryan Hendrickson <
>> ryan.andrew.hendrickson@gmail.com> wrote:
>>
>>> Yea, just finished testing, it works.. I added an additional property
>>> called "Date Format" and I passed in a java date format there. Now I'm
>>> just wondering if there's a way to validate it's a good date format before
>>> it tries to use it when it's passing docs through. Maybe a custom
>>> validator?
>>>
>>> Ryan
>>>
>>> On Tue, Aug 7, 2018 at 1:53 PM Ryan Hendrickson <
>>> ryan.andrew.hendrickson@gmail.com> wrote:
>>>
>>>> Haha, yea I'm testing out some code-hacking here to see what I can do
>>>> too. If it works, I can try to submit a Pull Request for it. I haven't
>>>> done one before, and this seems pretty easy to make it configurable.
>>>>
>>>> Ryan
>>>>
>>>> On Tue, Aug 7, 2018 at 1:48 PM Mike Thomsen <mi...@gmail.com>
>>>> wrote:
>>>>
>>>>> Annnndd, I should have read your comment on the Jira ticket because
>>>>> it's even easier than that!
>>>>>
>>>>> On Tue, Aug 7, 2018 at 1:47 PM Mike Thomsen <mi...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I just checked the code, and it's using the default Jackson mapping
>>>>>> behavior for that. The Mongo driver returns a Date, and looks like Jackson
>>>>>> is just turning that into an ISO8601 string without that level of
>>>>>> precision. A custom mapper for Date objects should be able to solve that.
>>>>>> I'll work it when I get some free time from the daily grind.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Mike
>>>>>>
>>>>>> On Tue, Aug 7, 2018 at 12:55 PM Ryan Hendrickson <
>>>>>> ryan.andrew.hendrickson@gmail.com> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>> I'm using GetMongo configured with JSON Type of "Standard JSON".
>>>>>>> The document I've got in Mongo has a date field that looks like the
>>>>>>> following:
>>>>>>> {
>>>>>>> ...
>>>>>>> "date" : ISODate("2018-08-06T16:20:10.912Z"
>>>>>>> ...
>>>>>>> }
>>>>>>>
>>>>>>> When GetMongo spits it out, the date comes out as:
>>>>>>> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
>>>>>>>
>>>>>>> I've created a bug ticket here:
>>>>>>> https://issues.apache.org/jira/browse/NIFI-5495.
>>>>>>>
>>>>>>> I'm not sure if there's a work around or anything like that. If
>>>>>>> anyone else has suggestions to either add back the missing milliseconds,
>>>>>>> even if it's just tagging on ".000" to get the format back, I'm all ears.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Ryan
>>>>>>>
>>>>>>
Re: GetMongo Truncates Dates
Posted by Mike Thomsen <mi...@gmail.com>.
Ryan,
Didn't see you put out a pull request in the last two weeks. Let me know if
you're actively working this, if not I can do my own patch.
Thanks,
Mike
On Wed, Aug 8, 2018 at 6:03 AM Mike Thomsen <mi...@gmail.com> wrote:
> Yes. Use a custom validator.
>
> On Tue, Aug 7, 2018 at 1:58 PM Ryan Hendrickson <
> ryan.andrew.hendrickson@gmail.com> wrote:
>
>> Yea, just finished testing, it works.. I added an additional property
>> called "Date Format" and I passed in a java date format there. Now I'm
>> just wondering if there's a way to validate it's a good date format before
>> it tries to use it when it's passing docs through. Maybe a custom
>> validator?
>>
>> Ryan
>>
>> On Tue, Aug 7, 2018 at 1:53 PM Ryan Hendrickson <
>> ryan.andrew.hendrickson@gmail.com> wrote:
>>
>>> Haha, yea I'm testing out some code-hacking here to see what I can do
>>> too. If it works, I can try to submit a Pull Request for it. I haven't
>>> done one before, and this seems pretty easy to make it configurable.
>>>
>>> Ryan
>>>
>>> On Tue, Aug 7, 2018 at 1:48 PM Mike Thomsen <mi...@gmail.com>
>>> wrote:
>>>
>>>> Annnndd, I should have read your comment on the Jira ticket because
>>>> it's even easier than that!
>>>>
>>>> On Tue, Aug 7, 2018 at 1:47 PM Mike Thomsen <mi...@gmail.com>
>>>> wrote:
>>>>
>>>>> I just checked the code, and it's using the default Jackson mapping
>>>>> behavior for that. The Mongo driver returns a Date, and looks like Jackson
>>>>> is just turning that into an ISO8601 string without that level of
>>>>> precision. A custom mapper for Date objects should be able to solve that.
>>>>> I'll work it when I get some free time from the daily grind.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Mike
>>>>>
>>>>> On Tue, Aug 7, 2018 at 12:55 PM Ryan Hendrickson <
>>>>> ryan.andrew.hendrickson@gmail.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>> I'm using GetMongo configured with JSON Type of "Standard JSON".
>>>>>> The document I've got in Mongo has a date field that looks like the
>>>>>> following:
>>>>>> {
>>>>>> ...
>>>>>> "date" : ISODate("2018-08-06T16:20:10.912Z"
>>>>>> ...
>>>>>> }
>>>>>>
>>>>>> When GetMongo spits it out, the date comes out as:
>>>>>> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
>>>>>>
>>>>>> I've created a bug ticket here:
>>>>>> https://issues.apache.org/jira/browse/NIFI-5495.
>>>>>>
>>>>>> I'm not sure if there's a work around or anything like that. If
>>>>>> anyone else has suggestions to either add back the missing milliseconds,
>>>>>> even if it's just tagging on ".000" to get the format back, I'm all ears.
>>>>>>
>>>>>> Thanks,
>>>>>> Ryan
>>>>>>
>>>>>
Re: GetMongo Truncates Dates
Posted by Mike Thomsen <mi...@gmail.com>.
Yes. Use a custom validator.
On Tue, Aug 7, 2018 at 1:58 PM Ryan Hendrickson <
ryan.andrew.hendrickson@gmail.com> wrote:
> Yea, just finished testing, it works.. I added an additional property
> called "Date Format" and I passed in a java date format there. Now I'm
> just wondering if there's a way to validate it's a good date format before
> it tries to use it when it's passing docs through. Maybe a custom
> validator?
>
> Ryan
>
> On Tue, Aug 7, 2018 at 1:53 PM Ryan Hendrickson <
> ryan.andrew.hendrickson@gmail.com> wrote:
>
>> Haha, yea I'm testing out some code-hacking here to see what I can do
>> too. If it works, I can try to submit a Pull Request for it. I haven't
>> done one before, and this seems pretty easy to make it configurable.
>>
>> Ryan
>>
>> On Tue, Aug 7, 2018 at 1:48 PM Mike Thomsen <mi...@gmail.com>
>> wrote:
>>
>>> Annnndd, I should have read your comment on the Jira ticket because it's
>>> even easier than that!
>>>
>>> On Tue, Aug 7, 2018 at 1:47 PM Mike Thomsen <mi...@gmail.com>
>>> wrote:
>>>
>>>> I just checked the code, and it's using the default Jackson mapping
>>>> behavior for that. The Mongo driver returns a Date, and looks like Jackson
>>>> is just turning that into an ISO8601 string without that level of
>>>> precision. A custom mapper for Date objects should be able to solve that.
>>>> I'll work it when I get some free time from the daily grind.
>>>>
>>>> Thanks,
>>>>
>>>> Mike
>>>>
>>>> On Tue, Aug 7, 2018 at 12:55 PM Ryan Hendrickson <
>>>> ryan.andrew.hendrickson@gmail.com> wrote:
>>>>
>>>>> Hi all,
>>>>> I'm using GetMongo configured with JSON Type of "Standard JSON".
>>>>> The document I've got in Mongo has a date field that looks like the
>>>>> following:
>>>>> {
>>>>> ...
>>>>> "date" : ISODate("2018-08-06T16:20:10.912Z"
>>>>> ...
>>>>> }
>>>>>
>>>>> When GetMongo spits it out, the date comes out as:
>>>>> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
>>>>>
>>>>> I've created a bug ticket here:
>>>>> https://issues.apache.org/jira/browse/NIFI-5495.
>>>>>
>>>>> I'm not sure if there's a work around or anything like that. If
>>>>> anyone else has suggestions to either add back the missing milliseconds,
>>>>> even if it's just tagging on ".000" to get the format back, I'm all ears.
>>>>>
>>>>> Thanks,
>>>>> Ryan
>>>>>
>>>>
Re: GetMongo Truncates Dates
Posted by Ryan Hendrickson <ry...@gmail.com>.
Yea, just finished testing, it works.. I added an additional property
called "Date Format" and I passed in a java date format there. Now I'm
just wondering if there's a way to validate it's a good date format before
it tries to use it when it's passing docs through. Maybe a custom
validator?
Ryan
On Tue, Aug 7, 2018 at 1:53 PM Ryan Hendrickson <
ryan.andrew.hendrickson@gmail.com> wrote:
> Haha, yea I'm testing out some code-hacking here to see what I can do
> too. If it works, I can try to submit a Pull Request for it. I haven't
> done one before, and this seems pretty easy to make it configurable.
>
> Ryan
>
> On Tue, Aug 7, 2018 at 1:48 PM Mike Thomsen <mi...@gmail.com>
> wrote:
>
>> Annnndd, I should have read your comment on the Jira ticket because it's
>> even easier than that!
>>
>> On Tue, Aug 7, 2018 at 1:47 PM Mike Thomsen <mi...@gmail.com>
>> wrote:
>>
>>> I just checked the code, and it's using the default Jackson mapping
>>> behavior for that. The Mongo driver returns a Date, and looks like Jackson
>>> is just turning that into an ISO8601 string without that level of
>>> precision. A custom mapper for Date objects should be able to solve that.
>>> I'll work it when I get some free time from the daily grind.
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>> On Tue, Aug 7, 2018 at 12:55 PM Ryan Hendrickson <
>>> ryan.andrew.hendrickson@gmail.com> wrote:
>>>
>>>> Hi all,
>>>> I'm using GetMongo configured with JSON Type of "Standard JSON".
>>>> The document I've got in Mongo has a date field that looks like the
>>>> following:
>>>> {
>>>> ...
>>>> "date" : ISODate("2018-08-06T16:20:10.912Z"
>>>> ...
>>>> }
>>>>
>>>> When GetMongo spits it out, the date comes out as:
>>>> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
>>>>
>>>> I've created a bug ticket here:
>>>> https://issues.apache.org/jira/browse/NIFI-5495.
>>>>
>>>> I'm not sure if there's a work around or anything like that. If
>>>> anyone else has suggestions to either add back the missing milliseconds,
>>>> even if it's just tagging on ".000" to get the format back, I'm all ears.
>>>>
>>>> Thanks,
>>>> Ryan
>>>>
>>>
Re: GetMongo Truncates Dates
Posted by Ryan Hendrickson <ry...@gmail.com>.
Haha, yea I'm testing out some code-hacking here to see what I can do too.
If it works, I can try to submit a Pull Request for it. I haven't done one
before, and this seems pretty easy to make it configurable.
Ryan
On Tue, Aug 7, 2018 at 1:48 PM Mike Thomsen <mi...@gmail.com> wrote:
> Annnndd, I should have read your comment on the Jira ticket because it's
> even easier than that!
>
> On Tue, Aug 7, 2018 at 1:47 PM Mike Thomsen <mi...@gmail.com>
> wrote:
>
>> I just checked the code, and it's using the default Jackson mapping
>> behavior for that. The Mongo driver returns a Date, and looks like Jackson
>> is just turning that into an ISO8601 string without that level of
>> precision. A custom mapper for Date objects should be able to solve that.
>> I'll work it when I get some free time from the daily grind.
>>
>> Thanks,
>>
>> Mike
>>
>> On Tue, Aug 7, 2018 at 12:55 PM Ryan Hendrickson <
>> ryan.andrew.hendrickson@gmail.com> wrote:
>>
>>> Hi all,
>>> I'm using GetMongo configured with JSON Type of "Standard JSON". The
>>> document I've got in Mongo has a date field that looks like the following:
>>> {
>>> ...
>>> "date" : ISODate("2018-08-06T16:20:10.912Z"
>>> ...
>>> }
>>>
>>> When GetMongo spits it out, the date comes out as:
>>> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
>>>
>>> I've created a bug ticket here:
>>> https://issues.apache.org/jira/browse/NIFI-5495.
>>>
>>> I'm not sure if there's a work around or anything like that. If
>>> anyone else has suggestions to either add back the missing milliseconds,
>>> even if it's just tagging on ".000" to get the format back, I'm all ears.
>>>
>>> Thanks,
>>> Ryan
>>>
>>
Re: GetMongo Truncates Dates
Posted by Mike Thomsen <mi...@gmail.com>.
Annnndd, I should have read your comment on the Jira ticket because it's
even easier than that!
On Tue, Aug 7, 2018 at 1:47 PM Mike Thomsen <mi...@gmail.com> wrote:
> I just checked the code, and it's using the default Jackson mapping
> behavior for that. The Mongo driver returns a Date, and looks like Jackson
> is just turning that into an ISO8601 string without that level of
> precision. A custom mapper for Date objects should be able to solve that.
> I'll work it when I get some free time from the daily grind.
>
> Thanks,
>
> Mike
>
> On Tue, Aug 7, 2018 at 12:55 PM Ryan Hendrickson <
> ryan.andrew.hendrickson@gmail.com> wrote:
>
>> Hi all,
>> I'm using GetMongo configured with JSON Type of "Standard JSON". The
>> document I've got in Mongo has a date field that looks like the following:
>> {
>> ...
>> "date" : ISODate("2018-08-06T16:20:10.912Z"
>> ...
>> }
>>
>> When GetMongo spits it out, the date comes out as:
>> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
>>
>> I've created a bug ticket here:
>> https://issues.apache.org/jira/browse/NIFI-5495.
>>
>> I'm not sure if there's a work around or anything like that. If
>> anyone else has suggestions to either add back the missing milliseconds,
>> even if it's just tagging on ".000" to get the format back, I'm all ears.
>>
>> Thanks,
>> Ryan
>>
>
Re: GetMongo Truncates Dates
Posted by Mike Thomsen <mi...@gmail.com>.
I just checked the code, and it's using the default Jackson mapping
behavior for that. The Mongo driver returns a Date, and looks like Jackson
is just turning that into an ISO8601 string without that level of
precision. A custom mapper for Date objects should be able to solve that.
I'll work it when I get some free time from the daily grind.
Thanks,
Mike
On Tue, Aug 7, 2018 at 12:55 PM Ryan Hendrickson <
ryan.andrew.hendrickson@gmail.com> wrote:
> Hi all,
> I'm using GetMongo configured with JSON Type of "Standard JSON". The
> document I've got in Mongo has a date field that looks like the following:
> {
> ...
> "date" : ISODate("2018-08-06T16:20:10.912Z"
> ...
> }
>
> When GetMongo spits it out, the date comes out as:
> "2018-08-06T16:20:10Z", noticeably missing the milliseconds.
>
> I've created a bug ticket here:
> https://issues.apache.org/jira/browse/NIFI-5495.
>
> I'm not sure if there's a work around or anything like that. If anyone
> else has suggestions to either add back the missing milliseconds, even if
> it's just tagging on ".000" to get the format back, I'm all ears.
>
> Thanks,
> Ryan
>