NiFi PutBigQueryBatch and AVRO logical types

["John W. Phillips" <jo...@looker.com>]

I'm using NiFi 1.9.2 QueryDatabaseTable->PutBigQueryBatch to attempt to
replicate a MySQL table to BigQuery.  In QueryDatabaseTable I've configured
'Use Avro Logical Types=true', so I have a MySQL DATETIME which is encoded
in Avro as a Long with logical type timestamp-millis.  The PutBigQueryBatch
does not support the BigQuery use_avro_logical_types option, so I have an
explicit schema where I cast the Long to a BigQuery TIMESTAMP.

The issue, though, is the Long is being interpreted by BQ as a Timestamp
with microseconds, and so the resulting Timestamp is off by x1000.  Does
anyone have a suggestion for a workaround?  I've tried an UpdateRecord
processor with :multiply(1000), but it has a type conversion exception when
writing the new Avro file.



--
Sent from: http://apache-nifi-users-list.2361937.n4.nabble.com/