Phoenix upsert failing on stale connection

[Ryan Templeton <rt...@hortonworks.com>]

Hey Everyone!

I have a Storm demo topology I’ve been working on for a while. Recently I adapted the final sink bolt to write data to my Phoenix cdr_fact table and initially everything works as expected wrt being able to insert data into the table. The problem comes when the topology sits idle for a period of time (I.e. No data arriving in the graph for a few minutes) records fail to insert into the table but no exceptions are being thrown. I thought this was due to some kind of timeout error and recently added code to check the validity of the connection but the validity check passes yet records are just not being inserted. I looked in the Phoenix logs but don’t find any entries for failed UPSERTS. I’m stuck on what to try next in order to track this down.

The code for the bolt can be found here: https://github.com/rtempleton/StreamingCDR/blob/master/src/main/java/com/github/rtempleton/cdr_storm/bolts/PhoenixFactWriter.java

Here’s a snippet from the my app log file

2016-11-11 18:26:23.278 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.292 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.309 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.326 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.339 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.353 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.368 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.384 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.402 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.415 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.538 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.554 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.571 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.586 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.598 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.611 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.626 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.639 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.653 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.675 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:26:23.692 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 100

2016-11-11 18:27:14.776 o.a.s.k.ZkCoordinator [INFO] Task [1/2] Refreshing partition manager connections

2016-11-11 18:27:14.779 o.a.s.k.ZkCoordinator [INFO] Task [2/2] Refreshing partition manager connections

2016-11-11 18:27:14.784 o.a.s.k.DynamicBrokersReader [INFO] Read partition info from zookeeper: GlobalPartitionInformation{topic=cdr-storm, partitionMap={0=rtempleton-3.field.hortonworks.com:6667, 1=rtempleton-1.field.hortonworks.com:6667}}


I wait a little while before pushing more data onto the Kafka queue – during this time when no data was flowing through the graph, the state of the Phoenix writer somehow changes and UPSERTS start failing…


2016-11-11 18:53:45.112 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 0

2016-11-11 18:53:45.120 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 0

2016-11-11 18:53:45.222 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 0

2016-11-11 18:53:45.403 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 0

2016-11-11 18:53:45.589 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 0

2016-11-11 18:53:45.689 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 0

2016-11-11 18:53:45.788 c.g.r.c.b.PhoenixFactWriter [INFO] Updates added in one batch - 0


Thanks,
Ryan

Ryan Templeton | Technical Account Mgr | Mobile: 512-694-1965
[cid:497D6A42-9263-466E-BD5B-B0CAFE2C49BC]