You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by chamikaramj <gi...@git.apache.org> on 2016/07/07 04:49:01 UTC
[GitHub] incubator-beam pull request #599: [BEAM-360] Some updates related to dynamic...
GitHub user chamikaramj opened a pull request:
https://github.com/apache/incubator-beam/pull/599
[BEAM-360] Some updates related to dynamic work rebalancing of custom sources.
Adds a class 'iobase.BoundedSourceSplit' to represent dynamic work rebalancing results of custom sources.
Updates Dataflow runner specific code (apiclient.py) to support dynamic work rebalancing custom sources.
Updates 'OffsetRangeTracker' so that the result of 'position_at_fraction()'' is a 'long' instead of a 'float'.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chamikaramj/incubator-beam custom_sources_dwr
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/599.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #599
----
commit e51d4acf12133a79671c567c9ff709c941c54f8c
Author: Chamikara Jayalath <ch...@apache.org>
Date: 2016-06-21T01:09:50Z
Implements a framework for developing sources for new file types.
Module 'filebasedsource' provides a framework for creating sources for new file types. This framework readily implements several features common to many sources based on files.
Additionally, module 'avroio' contains a new source, 'AvroSource', that is implemented using the framework described above. 'AvroSource' is a source for reading Avro files.
Adds many unit tests for 'filebasedsource' and 'avroio' modules.
commit cacb613448b47592f8415570f7b64bc6de797f91
Author: Chamikara Jayalath <ch...@apache.org>
Date: 2016-07-07T03:25:04Z
Adds a class 'iobase.BoundedSourceSplit' to represent dynamic work rebalancing result of custom sources.
Updates Dataflow runner specific code (apiclient.py) to support dynamic work rebalancing custom sources.
Updates 'OffsetRangeTracker' so that the result of 'position_at_fraction()'' is a 'long' instead of a 'float'.
commit 264b4afc17c255e568a490e02ce47e9fb4b1e17a
Author: Chamikara Jayalath <ch...@apache.org>
Date: 2016-07-07T03:34:21Z
Adds more comments.
commit 49e097f9c5c3d8c2bca48d3416b4934a4d86ed34
Author: Chamikara Jayalath <ch...@google.com>
Date: 2016-07-07T04:41:06Z
Some updates related to dynamic work rebalancing custom sources.
Adds a class 'iobase.BoundedSourceSplit' to represent dynamic work rebalancing result of custom sources.
Updates Dataflow runner specific code (apiclient.py) to support dynamic work rebalancing custom sources.
Updates 'OffsetRangeTracker' so that the result of 'position_at_fraction()'' is a 'long' instead of a 'float'.
commit c9696c9e17c9c7a6fc13d53d4da21ac9b325c73c
Author: Chamikara Jayalath <ch...@google.com>
Date: 2016-07-07T04:41:20Z
Some updates related to dynamic work rebalancing custom sources.
Adds a class 'iobase.BoundedSourceSplit' to represent dynamic work rebalancing result of custom sources.
Updates Dataflow runner specific code (apiclient.py) to support dynamic work rebalancing custom sources.
Updates 'OffsetRangeTracker' so that the result of 'position_at_fraction()'' is a 'long' instead of a 'float'.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-beam pull request #599: [BEAM-360] Some updates related to dynamic...
Posted by chamikaramj <gi...@git.apache.org>.
Github user chamikaramj closed the pull request at:
https://github.com/apache/incubator-beam/pull/599
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-beam pull request #599: [BEAM-360] Some updates related to dynamic...
Posted by chamikaramj <gi...@git.apache.org>.
GitHub user chamikaramj reopened a pull request:
https://github.com/apache/incubator-beam/pull/599
[BEAM-360] Some updates related to dynamic work rebalancing of custom sources.
Adds a class 'iobase.BoundedSourceSplit' to represent dynamic work rebalancing results of custom sources.
Updates Dataflow runner specific code (apiclient.py) to support dynamic work rebalancing custom sources.
Updates 'OffsetRangeTracker' so that the result of 'position_at_fraction()'' is a 'long' instead of a 'float'.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chamikaramj/incubator-beam custom_sources_dwr
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-beam/pull/599.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #599
----
commit 19a41ccf5bcf00192e3646258eae0cbce85da23b
Author: Chamikara Jayalath <ch...@apache.org>
Date: 2016-07-07T03:25:04Z
Adds a class 'iobase.BoundedSourceSplit' to represent dynamic work rebalancing result of custom sources.
Updates Dataflow runner specific code (apiclient.py) to support dynamic work rebalancing custom sources.
Updates 'OffsetRangeTracker' so that the result of 'position_at_fraction()'' is a 'long' instead of a 'float'.
commit 4415989ef0dfd656643e6e8575b6e2090b4437b5
Author: Chamikara Jayalath <ch...@apache.org>
Date: 2016-07-07T03:34:21Z
Adds more comments.
commit 6aa697465e88f827a3121a1de8bad1b810d904da
Author: Chamikara Jayalath <ch...@google.com>
Date: 2016-07-07T04:41:20Z
Some updates related to dynamic work rebalancing custom sources.
Adds a class 'iobase.BoundedSourceSplit' to represent dynamic work rebalancing result of custom sources.
Updates Dataflow runner specific code (apiclient.py) to support dynamic work rebalancing custom sources.
Updates 'OffsetRangeTracker' so that the result of 'position_at_fraction()'' is a 'long' instead of a 'float'.
commit 1e01b1f5cd70e5b39cd064577110898c623e524a
Author: Chamikara Jayalath <ch...@google.com>
Date: 2016-07-08T19:01:42Z
Reverting some updates.
commit 171df1ec3333edd51c7c72db309d526dfa9badf1
Author: Chamikara Jayalath <ch...@google.com>
Date: 2016-07-08T22:34:52Z
Adds a method 'fileio.ChannelFactory.size_in_bytes()'' that can be used to determine the size of a single file.
Updates 'filebasedsource' to use this method when determining size of files.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-beam pull request #599: [BEAM-360] Some updates related to dynamic...
Posted by chamikaramj <gi...@git.apache.org>.
Github user chamikaramj closed the pull request at:
https://github.com/apache/incubator-beam/pull/599
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---