HiveQL processors in Nifi 1.7.0

[jeff whybark <je...@gmail.com>]

Hello,

We recently upgraded from Nifi 1.4 to Nifi 1.7.0 and are suddenly having
some issues with both the SelectHiveQL and PutHiveQL processors.  Currently
we are hosting Nifi 1.7.0 in AWS and are connecting to Hive in AWS EMR
(Hive Version 2.1.1).


Since the upgrade we’ve noticed that the SelectHiveQL and PutHiveQL
processors have been acting a bit erratically with a series of different
strange patterns.


SelectHiveQL Processor -

1.  Processors Fails yet the flow file is passed to the Successful
Connection.  Typically error is something like "Error during database query
of conversion of records”

2.  Simple Query fails with an error erratically "transfer relationship not
specified”

3.  Queries that yields zero rows erratically fail.


PutHiveQL Processor -

1.  Query fails and flow file is not routed to failure connection. Instead
the flow file is retried over and over even though the “Retry” connection
is not specified.

2.  An "Insert into" query yielded zero records yet advanced to
“Successful” connection.  Unfortunately I cannot prove exactly what
happened here, but it is my belief that the query must have failed yet
advanced to “Successful”.  When rerun a second time, rows were successfully
written out to the target as it should have.


Mainly, I just wanted to see if anyone else was seeing this kind of odd
behavior with Nifi 1.7.0 and HiveQL processors.  Since these are all a bit
erratic, it has been difficult to troubleshoot and recreate on the fly.
Also, if anyone didn’t see any behavior problems when upgrading to Nifi
1.7.0 in AWS, I’d be really interested to hear which version of EMR/Hive
you are using.


Thank you for any suggestions,

Jeff