[jira] [Created] (SENTRY-2110) send HDFS full updates in incremental chunks, to overcome Thrift 2Gb message size limit

["Vadim Spector (JIRA)" <ji...@apache.org>]

Vadim Spector created SENTRY-2110:
-------------------------------------

             Summary: send HDFS full updates in incremental chunks, to overcome Thrift 2Gb message size limit
                 Key: SENTRY-2110
                 URL: https://issues.apache.org/jira/browse/SENTRY-2110
             Project: Sentry
          Issue Type: Improvement
            Reporter: Vadim Spector


Thrift messages are limited to 2 Gb. Sending full update for millions of partitions from Sentry to HDFS plugin can at some point exceed the message size limit. Sample figures: for 15 million partitions, 2 Gb translates to 143 bytes per partition record which is not too much.

Full update can be split into several pieces. It will require 

1. adding some additional fields to Thrift message schema, like the sequence number of the incremental full update message and the total number of incremental full update messages (e.g. messge #2 of 10 total)

2. Logic on Sentry side (SentryPlugin) to split full update into chunks and manage sending full updates in those chunks, watching for acknowledgements from HDFS plugin and resending chunks if needed.

3. Logic on HDFS plugin to assemble incremental chunks into full updates, watching for chunk numbers, and asking SentryPlugin to resend a chunk if needed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)