You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Oxenberg, Jeff" <je...@hpe.com> on 2016/07/01 02:54:02 UTC
RE: PutCassandraQL failing on ISO-8601-formatted timestamp
Hey Matt,
Thanks for responding, I just had the time to update Nifi and retry. I updated to 1.0 as of today. I'm still unable to insert timestamps - if I omit the timestamp column (and rework the query/table to just be key and value), I can insert just fine.
Here's the error I'm getting now:
com.datastax.driver.core.exceptions.CodecNotFoundException: Codec not found for requested operation: [timestamp <-> java.lang.String]
Here's the relevant attributes of the flowfile:
FlowFile Attribute Map Content
Key: 'cql.args.1.type'
Value: 'text'
Key: 'cql.args.1.value'
Value: 'temp3'
Key: 'cql.args.2.type'
Value: 'timestamp'
Key: 'cql.args.2.value'
Value: '2016-06-30T20:04:36Z'
Key: 'cql.args.3.type'
Value: 'float'
Key: 'cql.args.3.value'
Value: '6.7'
Key: 'j.id'
Value: 'temp3'
Key: 'j.ts'
Value: '2016-06-30T20:04:36Z'
Key: 'j.value'
Value: '6.7'
--------------------------------------------------
INSERT INTO test.test2 (sensor, ts, value) VALUES(?,?,?)
Thanks,
Jeff Oxenberg
-----Original Message-----
From: Matt Burgess [mailto:mattyb149@gmail.com]
Sent: Tuesday, June 21, 2016 8:52 PM
To: users@nifi.apache.org
Subject: Re: PutCassandraQL failing on ISO-8601-formatted timestamp
Jeff,
That appears to be a correct ISO-8601 date, so I'm not sure what's going on there. I checked the NiFi code and the Cassandra Java driver Jira and didn't see anything related (that wasn't already fixed, in the latter case). The upcoming 0.7.0 release has an updated Cassandra driver, perhaps that will solve your problem; if it doesn't, please feel free to file a Jira about this.
Regards,
Matt
On Tue, Jun 21, 2016 at 9:32 PM, Oxenberg, Jeff <je...@hpe.com> wrote:
> Hey,
>
>
>
> As a learning exercise, I’ve created a flow that parses a kafka topic
> of json messages and inserts them into Cassandra. It’s a three column table:
> id (text), ts (timestamp), and value (float). I set cql.args.x.type
> to the proper data types for each column.
>
>
>
> PutCassandraQL is failing with the following:
>
>
>
> org.apache.nifi.processor.exception.ProcessException: The value of the
> cql.args.2.value is '2016-06-21T20:23:41Z', which cannot be converted
> into the necessary data type: timestamp
>
>
>
> This happens for each timestamp, but I’m pretty sure the format is correct.
> It’s in ISO-8601, and I’m able to insert them manually into the c* table.
> I’m on Nifi 0.6.1.
>
>
>
> Any help would be appreciated, thanks!
RE: PutCassandraQL failing on ISO-8601-formatted timestamp
Posted by "Oxenberg, Jeff" <je...@hpe.com>.
No problem! This is my first contribution to Nifi. Please let me know what else I need to do.
NIFI-2165 - PutCassandraQL is handling timestamps as strings: https://issues.apache.org/jira/browse/NIFI-2165
PR: https://github.com/apache/nifi/pull/602
Thanks,
Jeff Oxenberg
-----Original Message-----
From: Matt Burgess [mailto:mattyb149@gmail.com]
Sent: Friday, July 01, 2016 12:30 PM
To: users@nifi.apache.org
Subject: Re: PutCassandraQL failing on ISO-8601-formatted timestamp
Awesome, thanks! I'll keep an eye out (and will review) the PR.
On Fri, Jul 1, 2016 at 1:11 PM, Oxenberg, Jeff <je...@hpe.com> wrote:
> Hey Matt,
>
> Yeah, I will submit a Jira and a pull request, I found and fixed the issue in the code, and it works now.
>
> Standard FlowFile Attributes
> Key: 'entryDate'
> Value: 'Fri Jul 01 12:13:08 UTC 2016'
> Key: 'lineageStartDate'
> Value: 'Fri Jul 01 12:13:08 UTC 2016'
> Key: 'fileSize'
> Value: '56'
> FlowFile Attribute Map Content
> Key: 'cql.args.1.type'
> Value: 'text'
> Key: 'cql.args.1.value'
> Value: 'temp3'
> Key: 'cql.args.2.type'
> Value: 'timestamp'
> Key: 'cql.args.2.value'
> Value: '2016-07-01T12:05:20Z'
> Key: 'cql.args.3.type'
> Value: 'float'
> Key: 'cql.args.3.value'
> Value: '7.38'
> Key: 'filename'
> Value: '180661344546505'
> Key: 'j.id'
> Value: 'temp3'
> Key: 'j.ts'
> Value: '2016-07-01T12:05:20Z'
> Key: 'j.value'
> Value: '7.38'
> Key: 'path'
> Value: './'
> Key: 'uuid'
> Value: 'b01c67dc-7bcf-4f77-8e38-3e6f8b2eeb2c'
> --------------------------------------------------
> INSERT INTO test.test2 (sensor, ts, value) VALUES(?,?,?)
> 2016-07-01 12:13:17,932 INFO [Provenance Maintenance Thread-2]
> o.a.n.p.PersistentProvenanceRepository Created new Provenance Event
> Writers for events starting with ID 16
> 2016-07-01 12:13:18,095 INFO [Provenance Repository Rollover Thread-1]
> o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal
> files (8 records) into single Provenance Log File
> ./provenance_repository/8.prov in 165 milliseconds
> 2016-07-01 12:13:18,096 INFO [Provenance Repository Rollover Thread-1]
> o.a.n.p.PersistentProvenanceRepository Successfully Rolled over
> Provenance Event file containing 89 records
>
>
> Thanks,
>
>
> Jeff Oxenberg
>
> -----Original Message-----
> From: Matt Burgess [mailto:mattyb149@gmail.com]
> Sent: Friday, July 01, 2016 11:06 AM
> To: users@nifi.apache.org
> Subject: Re: PutCassandraQL failing on ISO-8601-formatted timestamp
>
> Jeff,
>
> This looks like a bug. PutCassandraQL is handling timestamps as strings, and thus is not using the timestamp codec to parse the incoming value. Mind writing a Jira for this?
>
> Thanks,
> Matt
>
> On Thu, Jun 30, 2016 at 10:54 PM, Oxenberg, Jeff <je...@hpe.com> wrote:
>> Hey Matt,
>>
>> Thanks for responding, I just had the time to update Nifi and retry. I updated to 1.0 as of today. I'm still unable to insert timestamps - if I omit the timestamp column (and rework the query/table to just be key and value), I can insert just fine.
>>
>> Here's the error I'm getting now:
>>
>> com.datastax.driver.core.exceptions.CodecNotFoundException: Codec not
>> found for requested operation: [timestamp <-> java.lang.String] Here's the relevant attributes of the flowfile:
>>
>> FlowFile Attribute Map Content
>> Key: 'cql.args.1.type'
>> Value: 'text'
>> Key: 'cql.args.1.value'
>> Value: 'temp3'
>> Key: 'cql.args.2.type'
>> Value: 'timestamp'
>> Key: 'cql.args.2.value'
>> Value: '2016-06-30T20:04:36Z'
>> Key: 'cql.args.3.type'
>> Value: 'float'
>> Key: 'cql.args.3.value'
>> Value: '6.7'
>> Key: 'j.id'
>> Value: 'temp3'
>> Key: 'j.ts'
>> Value: '2016-06-30T20:04:36Z'
>> Key: 'j.value'
>> Value: '6.7'
>> --------------------------------------------------
>> INSERT INTO test.test2 (sensor, ts, value) VALUES(?,?,?)
>>
>> Thanks,
>>
>>
>> Jeff Oxenberg
>>
>> -----Original Message-----
>> From: Matt Burgess [mailto:mattyb149@gmail.com]
>> Sent: Tuesday, June 21, 2016 8:52 PM
>> To: users@nifi.apache.org
>> Subject: Re: PutCassandraQL failing on ISO-8601-formatted timestamp
>>
>> Jeff,
>>
>> That appears to be a correct ISO-8601 date, so I'm not sure what's going on there. I checked the NiFi code and the Cassandra Java driver Jira and didn't see anything related (that wasn't already fixed, in the latter case). The upcoming 0.7.0 release has an updated Cassandra driver, perhaps that will solve your problem; if it doesn't, please feel free to file a Jira about this.
>>
>> Regards,
>> Matt
>>
>> On Tue, Jun 21, 2016 at 9:32 PM, Oxenberg, Jeff <je...@hpe.com> wrote:
>>> Hey,
>>>
>>>
>>>
>>> As a learning exercise, I’ve created a flow that parses a kafka
>>> topic of json messages and inserts them into Cassandra. It’s a three column table:
>>> id (text), ts (timestamp), and value (float). I set cql.args.x.type
>>> to the proper data types for each column.
>>>
>>>
>>>
>>> PutCassandraQL is failing with the following:
>>>
>>>
>>>
>>> org.apache.nifi.processor.exception.ProcessException: The value of
>>> the cql.args.2.value is '2016-06-21T20:23:41Z', which cannot be
>>> converted into the necessary data type: timestamp
>>>
>>>
>>>
>>> This happens for each timestamp, but I’m pretty sure the format is correct.
>>> It’s in ISO-8601, and I’m able to insert them manually into the c* table.
>>> I’m on Nifi 0.6.1.
>>>
>>>
>>>
>>> Any help would be appreciated, thanks!
Re: PutCassandraQL failing on ISO-8601-formatted timestamp
Posted by Matt Burgess <ma...@gmail.com>.
Awesome, thanks! I'll keep an eye out (and will review) the PR.
On Fri, Jul 1, 2016 at 1:11 PM, Oxenberg, Jeff <je...@hpe.com> wrote:
> Hey Matt,
>
> Yeah, I will submit a Jira and a pull request, I found and fixed the issue in the code, and it works now.
>
> Standard FlowFile Attributes
> Key: 'entryDate'
> Value: 'Fri Jul 01 12:13:08 UTC 2016'
> Key: 'lineageStartDate'
> Value: 'Fri Jul 01 12:13:08 UTC 2016'
> Key: 'fileSize'
> Value: '56'
> FlowFile Attribute Map Content
> Key: 'cql.args.1.type'
> Value: 'text'
> Key: 'cql.args.1.value'
> Value: 'temp3'
> Key: 'cql.args.2.type'
> Value: 'timestamp'
> Key: 'cql.args.2.value'
> Value: '2016-07-01T12:05:20Z'
> Key: 'cql.args.3.type'
> Value: 'float'
> Key: 'cql.args.3.value'
> Value: '7.38'
> Key: 'filename'
> Value: '180661344546505'
> Key: 'j.id'
> Value: 'temp3'
> Key: 'j.ts'
> Value: '2016-07-01T12:05:20Z'
> Key: 'j.value'
> Value: '7.38'
> Key: 'path'
> Value: './'
> Key: 'uuid'
> Value: 'b01c67dc-7bcf-4f77-8e38-3e6f8b2eeb2c'
> --------------------------------------------------
> INSERT INTO test.test2 (sensor, ts, value) VALUES(?,?,?)
> 2016-07-01 12:13:17,932 INFO [Provenance Maintenance Thread-2] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 16
> 2016-07-01 12:13:18,095 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (8 records) into single Provenance Log File ./provenance_repository/8.prov in 165 milliseconds
> 2016-07-01 12:13:18,096 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 89 records
>
>
> Thanks,
>
>
> Jeff Oxenberg
>
> -----Original Message-----
> From: Matt Burgess [mailto:mattyb149@gmail.com]
> Sent: Friday, July 01, 2016 11:06 AM
> To: users@nifi.apache.org
> Subject: Re: PutCassandraQL failing on ISO-8601-formatted timestamp
>
> Jeff,
>
> This looks like a bug. PutCassandraQL is handling timestamps as strings, and thus is not using the timestamp codec to parse the incoming value. Mind writing a Jira for this?
>
> Thanks,
> Matt
>
> On Thu, Jun 30, 2016 at 10:54 PM, Oxenberg, Jeff <je...@hpe.com> wrote:
>> Hey Matt,
>>
>> Thanks for responding, I just had the time to update Nifi and retry. I updated to 1.0 as of today. I'm still unable to insert timestamps - if I omit the timestamp column (and rework the query/table to just be key and value), I can insert just fine.
>>
>> Here's the error I'm getting now:
>>
>> com.datastax.driver.core.exceptions.CodecNotFoundException: Codec not
>> found for requested operation: [timestamp <-> java.lang.String] Here's the relevant attributes of the flowfile:
>>
>> FlowFile Attribute Map Content
>> Key: 'cql.args.1.type'
>> Value: 'text'
>> Key: 'cql.args.1.value'
>> Value: 'temp3'
>> Key: 'cql.args.2.type'
>> Value: 'timestamp'
>> Key: 'cql.args.2.value'
>> Value: '2016-06-30T20:04:36Z'
>> Key: 'cql.args.3.type'
>> Value: 'float'
>> Key: 'cql.args.3.value'
>> Value: '6.7'
>> Key: 'j.id'
>> Value: 'temp3'
>> Key: 'j.ts'
>> Value: '2016-06-30T20:04:36Z'
>> Key: 'j.value'
>> Value: '6.7'
>> --------------------------------------------------
>> INSERT INTO test.test2 (sensor, ts, value) VALUES(?,?,?)
>>
>> Thanks,
>>
>>
>> Jeff Oxenberg
>>
>> -----Original Message-----
>> From: Matt Burgess [mailto:mattyb149@gmail.com]
>> Sent: Tuesday, June 21, 2016 8:52 PM
>> To: users@nifi.apache.org
>> Subject: Re: PutCassandraQL failing on ISO-8601-formatted timestamp
>>
>> Jeff,
>>
>> That appears to be a correct ISO-8601 date, so I'm not sure what's going on there. I checked the NiFi code and the Cassandra Java driver Jira and didn't see anything related (that wasn't already fixed, in the latter case). The upcoming 0.7.0 release has an updated Cassandra driver, perhaps that will solve your problem; if it doesn't, please feel free to file a Jira about this.
>>
>> Regards,
>> Matt
>>
>> On Tue, Jun 21, 2016 at 9:32 PM, Oxenberg, Jeff <je...@hpe.com> wrote:
>>> Hey,
>>>
>>>
>>>
>>> As a learning exercise, I’ve created a flow that parses a kafka topic
>>> of json messages and inserts them into Cassandra. It’s a three column table:
>>> id (text), ts (timestamp), and value (float). I set cql.args.x.type
>>> to the proper data types for each column.
>>>
>>>
>>>
>>> PutCassandraQL is failing with the following:
>>>
>>>
>>>
>>> org.apache.nifi.processor.exception.ProcessException: The value of
>>> the cql.args.2.value is '2016-06-21T20:23:41Z', which cannot be
>>> converted into the necessary data type: timestamp
>>>
>>>
>>>
>>> This happens for each timestamp, but I’m pretty sure the format is correct.
>>> It’s in ISO-8601, and I’m able to insert them manually into the c* table.
>>> I’m on Nifi 0.6.1.
>>>
>>>
>>>
>>> Any help would be appreciated, thanks!
RE: PutCassandraQL failing on ISO-8601-formatted timestamp
Posted by "Oxenberg, Jeff" <je...@hpe.com>.
Hey Matt,
Yeah, I will submit a Jira and a pull request, I found and fixed the issue in the code, and it works now.
Standard FlowFile Attributes
Key: 'entryDate'
Value: 'Fri Jul 01 12:13:08 UTC 2016'
Key: 'lineageStartDate'
Value: 'Fri Jul 01 12:13:08 UTC 2016'
Key: 'fileSize'
Value: '56'
FlowFile Attribute Map Content
Key: 'cql.args.1.type'
Value: 'text'
Key: 'cql.args.1.value'
Value: 'temp3'
Key: 'cql.args.2.type'
Value: 'timestamp'
Key: 'cql.args.2.value'
Value: '2016-07-01T12:05:20Z'
Key: 'cql.args.3.type'
Value: 'float'
Key: 'cql.args.3.value'
Value: '7.38'
Key: 'filename'
Value: '180661344546505'
Key: 'j.id'
Value: 'temp3'
Key: 'j.ts'
Value: '2016-07-01T12:05:20Z'
Key: 'j.value'
Value: '7.38'
Key: 'path'
Value: './'
Key: 'uuid'
Value: 'b01c67dc-7bcf-4f77-8e38-3e6f8b2eeb2c'
--------------------------------------------------
INSERT INTO test.test2 (sensor, ts, value) VALUES(?,?,?)
2016-07-01 12:13:17,932 INFO [Provenance Maintenance Thread-2] o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 16
2016-07-01 12:13:18,095 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (8 records) into single Provenance Log File ./provenance_repository/8.prov in 165 milliseconds
2016-07-01 12:13:18,096 INFO [Provenance Repository Rollover Thread-1] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 89 records
Thanks,
Jeff Oxenberg
-----Original Message-----
From: Matt Burgess [mailto:mattyb149@gmail.com]
Sent: Friday, July 01, 2016 11:06 AM
To: users@nifi.apache.org
Subject: Re: PutCassandraQL failing on ISO-8601-formatted timestamp
Jeff,
This looks like a bug. PutCassandraQL is handling timestamps as strings, and thus is not using the timestamp codec to parse the incoming value. Mind writing a Jira for this?
Thanks,
Matt
On Thu, Jun 30, 2016 at 10:54 PM, Oxenberg, Jeff <je...@hpe.com> wrote:
> Hey Matt,
>
> Thanks for responding, I just had the time to update Nifi and retry. I updated to 1.0 as of today. I'm still unable to insert timestamps - if I omit the timestamp column (and rework the query/table to just be key and value), I can insert just fine.
>
> Here's the error I'm getting now:
>
> com.datastax.driver.core.exceptions.CodecNotFoundException: Codec not
> found for requested operation: [timestamp <-> java.lang.String] Here's the relevant attributes of the flowfile:
>
> FlowFile Attribute Map Content
> Key: 'cql.args.1.type'
> Value: 'text'
> Key: 'cql.args.1.value'
> Value: 'temp3'
> Key: 'cql.args.2.type'
> Value: 'timestamp'
> Key: 'cql.args.2.value'
> Value: '2016-06-30T20:04:36Z'
> Key: 'cql.args.3.type'
> Value: 'float'
> Key: 'cql.args.3.value'
> Value: '6.7'
> Key: 'j.id'
> Value: 'temp3'
> Key: 'j.ts'
> Value: '2016-06-30T20:04:36Z'
> Key: 'j.value'
> Value: '6.7'
> --------------------------------------------------
> INSERT INTO test.test2 (sensor, ts, value) VALUES(?,?,?)
>
> Thanks,
>
>
> Jeff Oxenberg
>
> -----Original Message-----
> From: Matt Burgess [mailto:mattyb149@gmail.com]
> Sent: Tuesday, June 21, 2016 8:52 PM
> To: users@nifi.apache.org
> Subject: Re: PutCassandraQL failing on ISO-8601-formatted timestamp
>
> Jeff,
>
> That appears to be a correct ISO-8601 date, so I'm not sure what's going on there. I checked the NiFi code and the Cassandra Java driver Jira and didn't see anything related (that wasn't already fixed, in the latter case). The upcoming 0.7.0 release has an updated Cassandra driver, perhaps that will solve your problem; if it doesn't, please feel free to file a Jira about this.
>
> Regards,
> Matt
>
> On Tue, Jun 21, 2016 at 9:32 PM, Oxenberg, Jeff <je...@hpe.com> wrote:
>> Hey,
>>
>>
>>
>> As a learning exercise, I’ve created a flow that parses a kafka topic
>> of json messages and inserts them into Cassandra. It’s a three column table:
>> id (text), ts (timestamp), and value (float). I set cql.args.x.type
>> to the proper data types for each column.
>>
>>
>>
>> PutCassandraQL is failing with the following:
>>
>>
>>
>> org.apache.nifi.processor.exception.ProcessException: The value of
>> the cql.args.2.value is '2016-06-21T20:23:41Z', which cannot be
>> converted into the necessary data type: timestamp
>>
>>
>>
>> This happens for each timestamp, but I’m pretty sure the format is correct.
>> It’s in ISO-8601, and I’m able to insert them manually into the c* table.
>> I’m on Nifi 0.6.1.
>>
>>
>>
>> Any help would be appreciated, thanks!
Re: PutCassandraQL failing on ISO-8601-formatted timestamp
Posted by Matt Burgess <ma...@gmail.com>.
Jeff,
This looks like a bug. PutCassandraQL is handling timestamps as
strings, and thus is not using the timestamp codec to parse the
incoming value. Mind writing a Jira for this?
Thanks,
Matt
On Thu, Jun 30, 2016 at 10:54 PM, Oxenberg, Jeff <je...@hpe.com> wrote:
> Hey Matt,
>
> Thanks for responding, I just had the time to update Nifi and retry. I updated to 1.0 as of today. I'm still unable to insert timestamps - if I omit the timestamp column (and rework the query/table to just be key and value), I can insert just fine.
>
> Here's the error I'm getting now:
>
> com.datastax.driver.core.exceptions.CodecNotFoundException: Codec not found for requested operation: [timestamp <-> java.lang.String]
> Here's the relevant attributes of the flowfile:
>
> FlowFile Attribute Map Content
> Key: 'cql.args.1.type'
> Value: 'text'
> Key: 'cql.args.1.value'
> Value: 'temp3'
> Key: 'cql.args.2.type'
> Value: 'timestamp'
> Key: 'cql.args.2.value'
> Value: '2016-06-30T20:04:36Z'
> Key: 'cql.args.3.type'
> Value: 'float'
> Key: 'cql.args.3.value'
> Value: '6.7'
> Key: 'j.id'
> Value: 'temp3'
> Key: 'j.ts'
> Value: '2016-06-30T20:04:36Z'
> Key: 'j.value'
> Value: '6.7'
> --------------------------------------------------
> INSERT INTO test.test2 (sensor, ts, value) VALUES(?,?,?)
>
> Thanks,
>
>
> Jeff Oxenberg
>
> -----Original Message-----
> From: Matt Burgess [mailto:mattyb149@gmail.com]
> Sent: Tuesday, June 21, 2016 8:52 PM
> To: users@nifi.apache.org
> Subject: Re: PutCassandraQL failing on ISO-8601-formatted timestamp
>
> Jeff,
>
> That appears to be a correct ISO-8601 date, so I'm not sure what's going on there. I checked the NiFi code and the Cassandra Java driver Jira and didn't see anything related (that wasn't already fixed, in the latter case). The upcoming 0.7.0 release has an updated Cassandra driver, perhaps that will solve your problem; if it doesn't, please feel free to file a Jira about this.
>
> Regards,
> Matt
>
> On Tue, Jun 21, 2016 at 9:32 PM, Oxenberg, Jeff <je...@hpe.com> wrote:
>> Hey,
>>
>>
>>
>> As a learning exercise, I’ve created a flow that parses a kafka topic
>> of json messages and inserts them into Cassandra. It’s a three column table:
>> id (text), ts (timestamp), and value (float). I set cql.args.x.type
>> to the proper data types for each column.
>>
>>
>>
>> PutCassandraQL is failing with the following:
>>
>>
>>
>> org.apache.nifi.processor.exception.ProcessException: The value of the
>> cql.args.2.value is '2016-06-21T20:23:41Z', which cannot be converted
>> into the necessary data type: timestamp
>>
>>
>>
>> This happens for each timestamp, but I’m pretty sure the format is correct.
>> It’s in ISO-8601, and I’m able to insert them manually into the c* table.
>> I’m on Nifi 0.6.1.
>>
>>
>>
>> Any help would be appreciated, thanks!