You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by mairbek <gi...@git.apache.org> on 2017/08/18 01:12:07 UTC
[GitHub] beam pull request #3729: [BEAM-1542] Added a preprocessing step to the Cloud...
GitHub user mairbek opened a pull request:
https://github.com/apache/beam/pull/3729
[BEAM-1542] Added a preprocessing step to the Cloud Spanner sink.
The general intuition we follow here: if mutations are presorted by the primary key before batching, it is more likely that mutations in the batch will end up in the same partition. It minimizes the number of participants in the distributed transaction on the Cloud Spanner side and leads to a better throughput.
Mutations are encoded before running other steps to avoid paying the serialization price. Primary keys are encoded using OrderedCode library, and ApproximateQuantiles transform is used to sample keys.
Once primary keys are sampled, for each mutation we assign the index of the closest primary key as a key and group by that key. Range deletes are submitted separately.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mairbek/beam prepro-pr
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/3729.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3729
----
commit 7aeb0b0d02c308c690fb598b69a7aec649e4bb89
Author: Mairbek Khadikov <ma...@google.com>
Date: 2017-07-20T23:22:04Z
Added a preprocessing step to the Cloud Spanner sink.
The general intuition we follow here: if mutations are presorted by the primary key before batching, it is more likely that mutations in the batch will end up in the same partition. It minimizes the number of participants in the distributed transaction on the Cloud Spanner side and leads to a better throughput.
Mutations are encoded before running other steps to avoid paying the serialization price. Primary keys are encoded using OrderedCode library, and ApproximateQuantiles transform is used to sample keys.
Once primary keys are sampled, for each mutation we assign the index of the closest primary key as a key and group by that key. Range deletes are submitted separately.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] beam pull request #3729: [BEAM-1542] Added a preprocessing step to the Cloud...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/beam/pull/3729
---