You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Rayman (JIRA)" <ji...@apache.org> on 2019/06/18 22:45:00 UTC
[jira] [Updated] (SAMZA-2255) Smart value writes in
TaskSideInputStorageManager
[ https://issues.apache.org/jira/browse/SAMZA-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rayman updated SAMZA-2255:
--------------------------
Description:
TaskSideInputStorageManager converts each IME into the desired set of records to be written by invoking the respective sideInputsProcessor.
For example,
List<Record> entriesToBeWritten = sideInputsProcessor.process(IME.message);
Then it iterates over this list, and if the entry to be written has a null value then the TaskSideInputStorageManager issues a delete to the KV Store, otherwise it issues a put.
This can be optimized as follows:
For a given list of entriesToBeWritten, the TaskSideInputStorageManager should first
a. Do a O( n ) pass over it and retain only the last record for each key.
b. Now given the list in a, it should apply all records with a null value by using the deleteAll, and
all records with a non-null value by using a put-All.
a. is easy to do by simply iterating over the list in reverse order and retaining the first record encountered for each key. b. is straightforward.
was:
TaskSideInputStorageManager converts each IME into the desired set of records to be written by invoking the respective sideInputsProcessor.
For example,
List<Record> entriesToBeWritten = sideInputsProcessor.process(IME.message);
Then it iterates over this list, and if the entry to be written has a null value then the TaskSideInputStorageManager issues a delete to the KV Store, otherwise it issues a put.
This can be optimized as follows:
For a given list of entriesToBeWritten, the TaskSideInputStorageManager should first
a. Do a O(n) pass over it and retain only the last record for each key.
b. Now given the list in a, it should apply all records with a null value by using the deleteAll, and
all records with a non-null value by using a put-All.
a. is easy to do by simply iterating over the list in reverse order and retaining the first record encountered for each key. b. is straightforward.
> Smart value writes in TaskSideInputStorageManager
> -------------------------------------------------
>
> Key: SAMZA-2255
> URL: https://issues.apache.org/jira/browse/SAMZA-2255
> Project: Samza
> Issue Type: Improvement
> Reporter: Rayman
> Priority: Major
>
> TaskSideInputStorageManager converts each IME into the desired set of records to be written by invoking the respective sideInputsProcessor.
> For example,
> List<Record> entriesToBeWritten = sideInputsProcessor.process(IME.message);
> Then it iterates over this list, and if the entry to be written has a null value then the TaskSideInputStorageManager issues a delete to the KV Store, otherwise it issues a put.
> This can be optimized as follows:
> For a given list of entriesToBeWritten, the TaskSideInputStorageManager should first
> a. Do a O( n ) pass over it and retain only the last record for each key.
> b. Now given the list in a, it should apply all records with a null value by using the deleteAll, and
> all records with a non-null value by using a put-All.
> a. is easy to do by simply iterating over the list in reverse order and retaining the first record encountered for each key. b. is straightforward.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)