You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 18:03:44 UTC
[GitHub] [beam] damccorm opened a new issue, #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
damccorm opened a new issue, #20528:
URL: https://github.com/apache/beam/issues/20528
not only there are more than 1 result per window, results for each window got duplicated as well.
here is some code I made to reproduce the issue, just run it with and without `*.with_fanout*`
if running with Dataflow runner, add appropriate `*gs://path/*` in `*WriteToText*`
```
import apache_beam as beam
from apache_beam.transforms import window
from apache_beam.utils.timestamp
import Timestamp
class ListFn(beam.CombineFn):
def create_accumulator(self):
return []
def add_input(self, mutable_accumulator, element):
return mutable_accumulator + [element]
def merge_accumulators(self, accumulators):
res = []
for accu in accumulators:
res
= res + accu
return res
def extract_output(self, accumulator):
return accumulator
p
= beam.Pipeline()
(
p
| beam.Create([
window.TimestampedValue(1, Timestamp(seconds=1596216396)),
window.TimestampedValue(2, Timestamp(seconds=1596216397)),
window.TimestampedValue(3, Timestamp(seconds=1596216398)),
window.TimestampedValue(4, Timestamp(seconds=1596216399)),
window.TimestampedValue(5, Timestamp(seconds=1596216400)),
window.TimestampedValue(6, Timestamp(seconds=1596216402)),
window.TimestampedValue(7, Timestamp(seconds=1596216403)),
window.TimestampedValue(8, Timestamp(seconds=1596216405))])
| beam.WindowInto(window.SlidingWindows(10,
5))
| beam.CombineGlobally(ListFn()).without_defaults().with_fanout(5)
| beam.Map(repr)
| beam.io.WriteToText("py-test-result", file_name_suffix='.json', num_shards=1))
p.run()
```
Imported from Jira [BEAM-10617](https://issues.apache.org/jira/browse/BEAM-10617). Original Jira may contain additional context.
Reported by: leiyiz.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1292421836
I discussed the fix yesterday w/ Robert and we will pursue a slightly different fix, which should be ready shortly. however, this bug has been there from the very first commit implementing `with_fanout`, so I wouldn't delay this specific release on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1292434196
Ack. Removing this from the milestone for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1527817394
we had a tentative fix but it has caused a performance regression and was rolled back; since then it remains in the backlog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1329364119
current fix caused an OOM in one of customers' pipelines , planning to revert and investigate further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
robertwb commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1281676183
If we can't get someone to fix this we should at least throw an error ASAP, before the next release, e.g. rejecting all but known good windows (globals and fixed windows).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1290748220
> OK, the problem is that we re-window to avoid stacked accumulating mode:
Can we re-window only if we detect accumulating mode? See: https://github.com/apache/beam/pull/23828
Alternatively, we could not rewindow if we detect slidng windows. disabling sliding windows is straightforward too if we want to go that route for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1286245282
started working on this today, no PR yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1343497799
Actually simply disabling it would be a reasonable way to protect user data and we should do that in the immediate term.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1343497183
OK given this context I am going to remove the release milestone. We can document the range of releases for which it does not work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1334449582
Is it possible that the OOM is unavoidable because of the fix? This is a pretty seriouis data corruption problem, no? I suppose it is not a regression but I would very much like 2.44.0 to have correct results in this situation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1341108177
> Is it possible that the OOM is unavoidable because of the fix?
it's unlikely
> This is a pretty seriouis data corruption problem, no? I suppose it is not a regression
that's correct. it has been there since when with_fanout was added.
> but I would very much like 2.44.0 to have correct results in this situation.
I am planning to come back to it next week when I am on a rotational duty again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1272016484
Sure, we can investigate as P1. P2 was based on the suspicion that this behavior has been like this from initial implementation.
As next step, I think we should identify whether with_fanout by nature of its implementation only works for particular classes of combiners/accumulators, for example: Top or Mean where repeating elements twice doesn't affect the end result, or whether this is a bug that can and should be fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1270873751
it may be neither actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1286243144
@tvalentyn any updates here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1292395069
This is the only issue currently blocking the 2.43.0 release without an existing cherry-pick request.
Can we push this to the next release ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ee07dazn commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by "ee07dazn (via GitHub)" <gi...@apache.org>.
ee07dazn commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1527795000
any progress on this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1246019980
If this is still true, then it should be very easy to reproduce. @robertwb or @tvalentyn or @pabloem do you know?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1261669118
@BjornPrime can you try to reproduce this ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1428714766
If it isn't actively being worked on, I suggest downgrading and unassigning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1527854994
Ack. this feedback would help w/ prioritization.
I'm not sure I can answer based on limited information about the usecase, but for experimentation processes you could apply https://github.com/apache/beam/pull/23828 locally. These changes only apply at job submission.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1346684300
The mitigation has been cherrypicked, so I am removing it from the 2.44.0 milestone. The bug remains open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1289370791
Yea, it probably has to do with any windowfn that duplicates an element into multiple windows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
robertwb commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1275311372
OK, the problem is that we re-window to avoid stacked accumulating mode:
https://github.com/apache/beam/blob/release-2.40.0/sdks/python/apache_beam/transforms/core.py#L2742
This re-duplicates the windows for sliding windows WindowFns. (It will do the wrong thing for sessions as well.) What we need to do is set the new windowing to be the same thing without accumulating mode, but without applying the windowfn.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1272102898
I mean, disable with_fanout when sliding windows.
BTW is this on all runners?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1428714290
If this is now disabled, is it perhaps P2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1271953481
It sounds like data loss, which should certainly be P1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1272102560
If we cannot quick fix, we should remove with_fanout until we can fix it, or at least warn somehow, and add to release notes for released SDKs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1270489945
I feel like the labels `P1` and `good first issue` should be mutually exclusive?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1289118236
> This re-duplicates the windows for sliding windows WindowFns. (It will do the wrong thing for sessions as well.)
Does not seem to repro with sessions. Passing test:
```
def test_combining_with_session_windows_and_fanout(self):
import logging
class ListFn(beam.CombineFn):
def create_accumulator(self):
return []
def add_input(self, mutable_accumulator, element):
return mutable_accumulator + [element]
def merge_accumulators(self, accumulators):
res = []
for accu in accumulators:
res = res + accu
return res
def extract_output(self, accumulator):
return accumulator
options = PipelineOptions()
options.view_as(StandardOptions).streaming = True
with TestPipeline(options=options) as p:
def has_expected_values(actual):
from hamcrest.core import assert_that as hamcrest_assert
from hamcrest.library.collection import contains_exactly
from hamcrest.library.collection import only_contains
hamcrest_assert(ordered, contains_exactly([0, 1, 2, 3], [5, 6, 7, 8]))
result = (
p
| beam.Create([
window.TimestampedValue(0, Timestamp(seconds=0)),
window.TimestampedValue(1, Timestamp(seconds=1)),
window.TimestampedValue(2, Timestamp(seconds=2)),
window.TimestampedValue(3, Timestamp(seconds=3)),
window.TimestampedValue(5, Timestamp(seconds=5)),
window.TimestampedValue(6, Timestamp(seconds=6)),
window.TimestampedValue(7, Timestamp(seconds=7)),
window.TimestampedValue(8, Timestamp(seconds=8))])
| beam.WindowInto(window.Sessions(2))
| beam.CombineGlobally(ListFn()).without_defaults().with_fanout(5)
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1281683499
ack, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] ee07dazn commented on issue #20528: python CombineGlobally().with_fanout() cause duplicate combine results for sliding windows
Posted by "ee07dazn (via GitHub)" <gi...@apache.org>.
ee07dazn commented on issue #20528:
URL: https://github.com/apache/beam/issues/20528#issuecomment-1527820497
I need the combiner because group-by is ineffective. And i needed it to further fanout as its streaming kafka input at quite a high rate.
any possible workaround approaches ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org