You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by be...@gmail.com on 2022/12/06 10:02:48 UTC
Beam High Priority Issue Report (60)
This is your daily summary of Beam's current high priority issues that may need attention.
See https://beam.apache.org/contribute/issue-priorities for the meaning and expectations around issue priorities.
Unassigned P1 Issues:
https://github.com/apache/beam/issues/24537 [Bug]: python flink runner is not compatible with Azure blob file system in Java
https://github.com/apache/beam/issues/24535 [Bug]: Bigquery Load jobs with WRITE_TRUNCATE disposition may truncate valid records.
https://github.com/apache/beam/issues/24415 [Bug]: Cannot find a matching Calcite SqlTypeName for Beam type: LOGICAL_TYPE seen in 2.44.0 SNAPSHOT
https://github.com/apache/beam/issues/24383 [Bug]: Daemon will be stopped at the end of the build after the daemon was no longer found in the daemon registry
https://github.com/apache/beam/issues/24367 [Bug]: workflow.tar.gz cannot be passed to flink runner
https://github.com/apache/beam/issues/24313 [Flaky]: apache_beam/runners/portability/portable_runner_test.py::PortableRunnerTestWithSubprocesses::test_pardo_state_with_custom_key_coder
https://github.com/apache/beam/issues/24267 [Failing Test]: Timeout waiting to lock gradle
https://github.com/apache/beam/issues/24263 [Bug]: Remote call on apache-beam-jenkins-3 failed. The channel is closing down or has closed down
https://github.com/apache/beam/issues/23944 beam_PreCommit_Python_Cron regularily failing - test_pardo_large_input flaky
https://github.com/apache/beam/issues/23709 [Flake]: Spark batch flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement and ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
https://github.com/apache/beam/issues/22969 Discrepancy in behavior of `DoFn.process()` when `yield` is combined with `return` statement, or vice versa
https://github.com/apache/beam/issues/22961 [Bug]: WriteToBigQuery silently skips most of records without job fail
https://github.com/apache/beam/issues/22913 [Bug]: beam_PostCommit_Java_ValidatesRunner_Flink is flakes in org.apache.beam.sdk.transforms.GroupByKeyTest$BasicTests.testAfterProcessingTimeContinuationTriggerUsingState
https://github.com/apache/beam/issues/22321 PortableRunnerTestWithExternalEnv.test_pardo_large_input is regularly failing on jenkins
https://github.com/apache/beam/issues/21713 404s in BigQueryIO don't get output to Failed Inserts PCollection
https://github.com/apache/beam/issues/21561 ExternalPythonTransformTest.trivialPythonTransform flaky
https://github.com/apache/beam/issues/21480 flake: FlinkRunnerTest.testEnsureStdoutStdErrIsRestored
https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink flaky: Connection refused
https://github.com/apache/beam/issues/21462 Flake in org.apache.beam.sdk.io.mqtt.MqttIOTest.testReadObject: Address already in use
https://github.com/apache/beam/issues/21261 org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer is flaky
https://github.com/apache/beam/issues/21260 Python DirectRunner does not emit data at GC time
https://github.com/apache/beam/issues/21121 apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT.test_streaming_wordcount_it flakey
https://github.com/apache/beam/issues/21113 testTwoTimersSettingEachOtherWithCreateAsInputBounded flaky
https://github.com/apache/beam/issues/20976 apache_beam.runners.portability.flink_runner_test.FlinkRunnerTestOptimized.test_flink_metrics is flaky
https://github.com/apache/beam/issues/20975 org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming: false] is flaky
https://github.com/apache/beam/issues/20974 Python GHA PreCommits flake with grpc.FutureTimeoutError on SDK harness startup
https://github.com/apache/beam/issues/20689 Kafka commitOffsetsInFinalize OOM on Flink
https://github.com/apache/beam/issues/20108 Python direct runner doesn't emit empty pane when it should
https://github.com/apache/beam/issues/19814 Flink streaming flakes in ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful and ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
https://github.com/apache/beam/issues/19734 WatchTest.testMultiplePollsWithManyResults flake: Outputs must be in timestamp order (sickbayed)
https://github.com/apache/beam/issues/19465 Explore possibilities to lower in-use IP address quota footprint.
https://github.com/apache/beam/issues/19241 Python Dataflow integration tests should export the pipeline Job ID and console output to Jenkins Test Result section
P1 Issues with no update in the last week:
https://github.com/apache/beam/issues/24100 [Bug]: `Filter.whereFieldName` appears in docs but not available
https://github.com/apache/beam/issues/23906 [Bug]: Dataflow jpms tests fail on the 2.43.0 release branch
https://github.com/apache/beam/issues/23875 [Bug]: beam.Row.__eq__ returns true for unequal rows
https://github.com/apache/beam/issues/23525 [Bug]: Default PubsubMessage coder will drop message id and orderingKey
https://github.com/apache/beam/issues/23489 [Bug]: add DebeziumIO to the connectors page
https://github.com/apache/beam/issues/23306 [Bug]: BigQueryBatchFileLoads in python loses data when using WRITE_TRUNCATE
https://github.com/apache/beam/issues/23286 [Bug]: beam_PerformanceTests_InfluxDbIO_IT Flaky > 50 % Fail
https://github.com/apache/beam/issues/22891 [Bug]: beam_PostCommit_XVR_PythonUsingJavaDataflow is flaky
https://github.com/apache/beam/issues/22605 [Bug]: Beam Python failure for dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest.test_metrics_it
https://github.com/apache/beam/issues/22115 [Bug]: apache_beam.runners.portability.portable_runner_test.PortableRunnerTestWithSubprocesses is flaky
https://github.com/apache/beam/issues/22011 [Bug]: org.apache.beam.sdk.io.aws2.kinesis.KinesisIOWriteTest.testWriteFailure flaky
https://github.com/apache/beam/issues/21709 beam_PostCommit_Java_ValidatesRunner_Samza Failing
https://github.com/apache/beam/issues/21708 beam_PostCommit_Java_DataflowV2, testBigQueryStorageWrite30MProto failing consistently
https://github.com/apache/beam/issues/21706 Flaky timeout in github Python unit test action StatefulDoFnOnDirectRunnerTest.test_dynamic_timer_clear_then_set_timer
https://github.com/apache/beam/issues/21700 --dataflowServiceOptions=use_runner_v2 is broken
https://github.com/apache/beam/issues/21695 DataflowPipelineResult does not raise exception for unsuccessful states.
https://github.com/apache/beam/issues/21645 beam_PostCommit_XVR_GoUsingJava_Dataflow fails on some test transforms
https://github.com/apache/beam/issues/21643 FnRunnerTest with non-trivial (order 1000 elements) numpy input flakes in non-cython environment
https://github.com/apache/beam/issues/21476 WriteToBigQuery Dynamic table destinations returns wrong tableId
https://github.com/apache/beam/issues/21474 Flaky tests: Gradle build daemon disappeared unexpectedly
https://github.com/apache/beam/issues/21368 Reduce number of Gax related threads, likely by providing common executor to GAX clients
https://github.com/apache/beam/issues/21333 Flink testParDoRequiresStableInput flaky
https://github.com/apache/beam/issues/21262 Python AfterAny, AfterAll do not follow spec
https://github.com/apache/beam/issues/21111 Java creates an incorrect pipeline proto when core-construction-java jar is not in the CLASSPATH
https://github.com/apache/beam/issues/21104 Flaky: apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTestWithGrpcAndMultiWorkers
https://github.com/apache/beam/issues/20819 Java build flakes: "Memory constraints are impeding performance"
https://github.com/apache/beam/issues/20814 JmsIO is not acknowledging messages correctly
https://github.com/apache/beam/issues/20812 Cross-language consistency (RequiresStableInputs) is quietly broken (at least on portable flink runner)
Re: Beam High Priority Issue Report (60)
Posted by Kenneth Knowles <ke...@apache.org>.
Now that we've managed to triage all our incoming issues, perhaps the next
easiest step is to get updates on the *assigned* P1s. After all, in theory
people are working on these and they are quite urgent. If you are assigned
a P1 and not working on it, go ahead and unassign it so we have an accurate
view of the state of bugs.
On Tue, Dec 6, 2022 at 2:03 AM <be...@gmail.com> wrote:
> This is your daily summary of Beam's current high priority issues that may
> need attention.
>
> See https://beam.apache.org/contribute/issue-priorities for the
> meaning and expectations around issue priorities.
>
> Unassigned P1 Issues:
>
> https://github.com/apache/beam/issues/24537 [Bug]: python flink runner is
> not compatible with Azure blob file system in Java
> https://github.com/apache/beam/issues/24535 [Bug]: Bigquery Load jobs
> with WRITE_TRUNCATE disposition may truncate valid records.
> https://github.com/apache/beam/issues/24415 [Bug]: Cannot find a matching
> Calcite SqlTypeName for Beam type: LOGICAL_TYPE seen in 2.44.0 SNAPSHOT
> https://github.com/apache/beam/issues/24383 [Bug]: Daemon will be stopped
> at the end of the build after the daemon was no longer found in the daemon
> registry
> https://github.com/apache/beam/issues/24367 [Bug]: workflow.tar.gz cannot
> be passed to flink runner
> https://github.com/apache/beam/issues/24313 [Flaky]:
> apache_beam/runners/portability/portable_runner_test.py::PortableRunnerTestWithSubprocesses::test_pardo_state_with_custom_key_coder
> https://github.com/apache/beam/issues/24267 [Failing Test]: Timeout
> waiting to lock gradle
> https://github.com/apache/beam/issues/24263 [Bug]: Remote call on
> apache-beam-jenkins-3 failed. The channel is closing down or has closed down
> https://github.com/apache/beam/issues/23944 beam_PreCommit_Python_Cron
> regularily failing - test_pardo_large_input flaky
> https://github.com/apache/beam/issues/23709 [Flake]: Spark batch flakes
> in ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElement and
> ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundle
> https://github.com/apache/beam/issues/22969 Discrepancy in behavior of
> `DoFn.process()` when `yield` is combined with `return` statement, or vice
> versa
> https://github.com/apache/beam/issues/22961 [Bug]: WriteToBigQuery
> silently skips most of records without job fail
> https://github.com/apache/beam/issues/22913 [Bug]:
> beam_PostCommit_Java_ValidatesRunner_Flink is flakes in
> org.apache.beam.sdk.transforms.GroupByKeyTest$BasicTests.testAfterProcessingTimeContinuationTriggerUsingState
> https://github.com/apache/beam/issues/22321
> PortableRunnerTestWithExternalEnv.test_pardo_large_input is regularly
> failing on jenkins
> https://github.com/apache/beam/issues/21713 404s in BigQueryIO don't get
> output to Failed Inserts PCollection
> https://github.com/apache/beam/issues/21561
> ExternalPythonTransformTest.trivialPythonTransform flaky
> https://github.com/apache/beam/issues/21480 flake:
> FlinkRunnerTest.testEnsureStdoutStdErrIsRestored
> https://github.com/apache/beam/issues/21469 beam_PostCommit_XVR_Flink
> flaky: Connection refused
> https://github.com/apache/beam/issues/21462 Flake in
> org.apache.beam.sdk.io.mqtt.MqttIOTest.testReadObject: Address already in
> use
> https://github.com/apache/beam/issues/21261
> org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer
> is flaky
> https://github.com/apache/beam/issues/21260 Python DirectRunner does not
> emit data at GC time
> https://github.com/apache/beam/issues/21121
> apache_beam.examples.streaming_wordcount_it_test.StreamingWordCountIT.test_streaming_wordcount_it
> flakey
> https://github.com/apache/beam/issues/21113
> testTwoTimersSettingEachOtherWithCreateAsInputBounded flaky
> https://github.com/apache/beam/issues/20976
> apache_beam.runners.portability.flink_runner_test.FlinkRunnerTestOptimized.test_flink_metrics
> is flaky
> https://github.com/apache/beam/issues/20975
> org.apache.beam.runners.flink.ReadSourcePortableTest.testExecution[streaming:
> false] is flaky
> https://github.com/apache/beam/issues/20974 Python GHA PreCommits flake
> with grpc.FutureTimeoutError on SDK harness startup
> https://github.com/apache/beam/issues/20689 Kafka commitOffsetsInFinalize
> OOM on Flink
> https://github.com/apache/beam/issues/20108 Python direct runner doesn't
> emit empty pane when it should
> https://github.com/apache/beam/issues/19814 Flink streaming flakes in
> ParDoLifecycleTest.testTeardownCalledAfterExceptionInStartBundleStateful
> and
> ParDoLifecycleTest.testTeardownCalledAfterExceptionInProcessElementStateful
> https://github.com/apache/beam/issues/19734
> WatchTest.testMultiplePollsWithManyResults flake: Outputs must be in
> timestamp order (sickbayed)
> https://github.com/apache/beam/issues/19465 Explore possibilities to
> lower in-use IP address quota footprint.
> https://github.com/apache/beam/issues/19241 Python Dataflow integration
> tests should export the pipeline Job ID and console output to Jenkins Test
> Result section
>
>
> P1 Issues with no update in the last week:
>
> https://github.com/apache/beam/issues/24100 [Bug]:
> `Filter.whereFieldName` appears in docs but not available
> https://github.com/apache/beam/issues/23906 [Bug]: Dataflow jpms tests
> fail on the 2.43.0 release branch
> https://github.com/apache/beam/issues/23875 [Bug]: beam.Row.__eq__
> returns true for unequal rows
> https://github.com/apache/beam/issues/23525 [Bug]: Default PubsubMessage
> coder will drop message id and orderingKey
> https://github.com/apache/beam/issues/23489 [Bug]: add DebeziumIO to the
> connectors page
> https://github.com/apache/beam/issues/23306 [Bug]: BigQueryBatchFileLoads
> in python loses data when using WRITE_TRUNCATE
> https://github.com/apache/beam/issues/23286 [Bug]:
> beam_PerformanceTests_InfluxDbIO_IT Flaky > 50 % Fail
> https://github.com/apache/beam/issues/22891 [Bug]:
> beam_PostCommit_XVR_PythonUsingJavaDataflow is flaky
> https://github.com/apache/beam/issues/22605 [Bug]: Beam Python failure
> for
> dataflow_exercise_metrics_pipeline_test.ExerciseMetricsPipelineTest.test_metrics_it
> https://github.com/apache/beam/issues/22115 [Bug]:
> apache_beam.runners.portability.portable_runner_test.PortableRunnerTestWithSubprocesses
> is flaky
> https://github.com/apache/beam/issues/22011 [Bug]:
> org.apache.beam.sdk.io.aws2.kinesis.KinesisIOWriteTest.testWriteFailure
> flaky
> https://github.com/apache/beam/issues/21709
> beam_PostCommit_Java_ValidatesRunner_Samza Failing
> https://github.com/apache/beam/issues/21708
> beam_PostCommit_Java_DataflowV2, testBigQueryStorageWrite30MProto failing
> consistently
> https://github.com/apache/beam/issues/21706 Flaky timeout in github
> Python unit test action
> StatefulDoFnOnDirectRunnerTest.test_dynamic_timer_clear_then_set_timer
> https://github.com/apache/beam/issues/21700
> --dataflowServiceOptions=use_runner_v2 is broken
> https://github.com/apache/beam/issues/21695 DataflowPipelineResult does
> not raise exception for unsuccessful states.
> https://github.com/apache/beam/issues/21645
> beam_PostCommit_XVR_GoUsingJava_Dataflow fails on some test transforms
> https://github.com/apache/beam/issues/21643 FnRunnerTest with non-trivial
> (order 1000 elements) numpy input flakes in non-cython environment
> https://github.com/apache/beam/issues/21476 WriteToBigQuery Dynamic table
> destinations returns wrong tableId
> https://github.com/apache/beam/issues/21474 Flaky tests: Gradle build
> daemon disappeared unexpectedly
> https://github.com/apache/beam/issues/21368 Reduce number of Gax related
> threads, likely by providing common executor to GAX clients
> https://github.com/apache/beam/issues/21333 Flink
> testParDoRequiresStableInput flaky
> https://github.com/apache/beam/issues/21262 Python AfterAny, AfterAll do
> not follow spec
> https://github.com/apache/beam/issues/21111 Java creates an incorrect
> pipeline proto when core-construction-java jar is not in the CLASSPATH
> https://github.com/apache/beam/issues/21104 Flaky:
> apache_beam.runners.portability.fn_api_runner.fn_runner_test.FnApiRunnerTestWithGrpcAndMultiWorkers
> https://github.com/apache/beam/issues/20819 Java build flakes: "Memory
> constraints are impeding performance"
> https://github.com/apache/beam/issues/20814 JmsIO is not acknowledging
> messages correctly
> https://github.com/apache/beam/issues/20812 Cross-language consistency
> (RequiresStableInputs) is quietly broken (at least on portable flink runner)
>
>
>