You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Luke Cwik <lc...@google.com> on 2020/04/21 20:34:18 UTC
Re: Running NexMark Tests
+dev
When would we not want --shutdownSourcesOnFinalWatermark=true ?
On Tue, Apr 21, 2020 at 1:22 PM Ismaël Mejía <ie...@gmail.com> wrote:
> You need to instruct the Flink runner to shutdown the the source
> otherwise it will stay waiting.
> You can this by adding the extra
> argument`--shutdownSourcesOnFinalWatermark=true`
> And if that works and you want to open a PR to update our
> documentation that would be greatly appreciated.
>
> Regards,
> Ismaël
>
>
> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> <sr...@gmail.com> wrote:
> >
> > Hello,
> >
> > I am trying to run nexmark queries using flink runner streaming.
> Followed the documentation and used the command
> > ./gradlew :sdks:java:testing:nexmark:run \
> >
> > -Pnexmark.runner=":runners:flink:1.10" \
> > -Pnexmark.args="
> > --runner=FlinkRunner
> > --suite=SMOKE
> > --streamTimeout=60
> > --streaming=true
> > --manageResources=false
> > --monitorJobs=true
> > --flinkMaster=[local]"
> >
> >
> > But after the events are read from the source, there is no further
> progress and the job is always stuck at 99%. Is there any configuration
> that I am missing?
> >
> > Regards,
> > Sruthi
>
Re: Running NexMark Tests
Posted by Maximilian Michels <mx...@apache.org>.
The flag is needed when checkpointing is enabled because Flink is unable
to create a new checkpoint when not all operators are running.
By default, operators shut down when all input has been read. That will
trigger sending out the maximum (final) watermark at the sources. The
flag name is a bit confusing in this regard because shutting down the
sources triggers sending out the watermark, not the other way around.
-Max
On 22.04.20 06:26, Kenneth Knowles wrote:
> We should always want to shut down sources on final watermark. All
> incoming data should be dropped anyhow.
>
> Kenn
>
> On Tue, Apr 21, 2020 at 1:34 PM Luke Cwik <lcwik@google.com
> <ma...@google.com>> wrote:
>
> +dev
>
> When would we not want --shutdownSourcesOnFinalWatermark=true ?
>
> On Tue, Apr 21, 2020 at 1:22 PM Ismaël Mejía <iemejia@gmail.com
> <ma...@gmail.com>> wrote:
>
> You need to instruct the Flink runner to shutdown the the source
> otherwise it will stay waiting.
> You can this by adding the extra
> argument`--shutdownSourcesOnFinalWatermark=true`
> And if that works and you want to open a PR to update our
> documentation that would be greatly appreciated.
>
> Regards,
> Ismaël
>
>
> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> <sruthisreekumar2712@gmail.com
> <ma...@gmail.com>> wrote:
> >
> > Hello,
> >
> > I am trying to run nexmark queries using flink runner
> streaming. Followed the documentation and used the command
> > ./gradlew :sdks:java:testing:nexmark:run \
> >
> > -Pnexmark.runner=":runners:flink:1.10" \
> > -Pnexmark.args="
> > --runner=FlinkRunner
> > --suite=SMOKE
> > --streamTimeout=60
> > --streaming=true
> > --manageResources=false
> > --monitorJobs=true
> > --flinkMaster=[local]"
> >
> >
> > But after the events are read from the source, there is no
> further progress and the job is always stuck at 99%. Is there
> any configuration that I am missing?
> >
> > Regards,
> > Sruthi
>
Re: Running NexMark Tests
Posted by Maximilian Michels <mx...@apache.org>.
The flag is needed when checkpointing is enabled because Flink is unable
to create a new checkpoint when not all operators are running.
By default, operators shut down when all input has been read. That will
trigger sending out the maximum (final) watermark at the sources. The
flag name is a bit confusing in this regard because shutting down the
sources triggers sending out the watermark, not the other way around.
-Max
On 22.04.20 06:26, Kenneth Knowles wrote:
> We should always want to shut down sources on final watermark. All
> incoming data should be dropped anyhow.
>
> Kenn
>
> On Tue, Apr 21, 2020 at 1:34 PM Luke Cwik <lcwik@google.com
> <ma...@google.com>> wrote:
>
> +dev
>
> When would we not want --shutdownSourcesOnFinalWatermark=true ?
>
> On Tue, Apr 21, 2020 at 1:22 PM Ismaël Mejía <iemejia@gmail.com
> <ma...@gmail.com>> wrote:
>
> You need to instruct the Flink runner to shutdown the the source
> otherwise it will stay waiting.
> You can this by adding the extra
> argument`--shutdownSourcesOnFinalWatermark=true`
> And if that works and you want to open a PR to update our
> documentation that would be greatly appreciated.
>
> Regards,
> Ismaël
>
>
> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> <sruthisreekumar2712@gmail.com
> <ma...@gmail.com>> wrote:
> >
> > Hello,
> >
> > I am trying to run nexmark queries using flink runner
> streaming. Followed the documentation and used the command
> > ./gradlew :sdks:java:testing:nexmark:run \
> >
> > -Pnexmark.runner=":runners:flink:1.10" \
> > -Pnexmark.args="
> > --runner=FlinkRunner
> > --suite=SMOKE
> > --streamTimeout=60
> > --streaming=true
> > --manageResources=false
> > --monitorJobs=true
> > --flinkMaster=[local]"
> >
> >
> > But after the events are read from the source, there is no
> further progress and the job is always stuck at 99%. Is there
> any configuration that I am missing?
> >
> > Regards,
> > Sruthi
>
Re: Running NexMark Tests
Posted by Kenneth Knowles <ke...@apache.org>.
We should always want to shut down sources on final watermark. All incoming
data should be dropped anyhow.
Kenn
On Tue, Apr 21, 2020 at 1:34 PM Luke Cwik <lc...@google.com> wrote:
> +dev
>
> When would we not want --shutdownSourcesOnFinalWatermark=true ?
>
> On Tue, Apr 21, 2020 at 1:22 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> You need to instruct the Flink runner to shutdown the the source
>> otherwise it will stay waiting.
>> You can this by adding the extra
>> argument`--shutdownSourcesOnFinalWatermark=true`
>> And if that works and you want to open a PR to update our
>> documentation that would be greatly appreciated.
>>
>> Regards,
>> Ismaël
>>
>>
>> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
>> <sr...@gmail.com> wrote:
>> >
>> > Hello,
>> >
>> > I am trying to run nexmark queries using flink runner streaming.
>> Followed the documentation and used the command
>> > ./gradlew :sdks:java:testing:nexmark:run \
>> >
>> > -Pnexmark.runner=":runners:flink:1.10" \
>> > -Pnexmark.args="
>> > --runner=FlinkRunner
>> > --suite=SMOKE
>> > --streamTimeout=60
>> > --streaming=true
>> > --manageResources=false
>> > --monitorJobs=true
>> > --flinkMaster=[local]"
>> >
>> >
>> > But after the events are read from the source, there is no further
>> progress and the job is always stuck at 99%. Is there any configuration
>> that I am missing?
>> >
>> > Regards,
>> > Sruthi
>>
>
Re: Running NexMark Tests
Posted by Kenneth Knowles <ke...@apache.org>.
We should always want to shut down sources on final watermark. All incoming
data should be dropped anyhow.
Kenn
On Tue, Apr 21, 2020 at 1:34 PM Luke Cwik <lc...@google.com> wrote:
> +dev
>
> When would we not want --shutdownSourcesOnFinalWatermark=true ?
>
> On Tue, Apr 21, 2020 at 1:22 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> You need to instruct the Flink runner to shutdown the the source
>> otherwise it will stay waiting.
>> You can this by adding the extra
>> argument`--shutdownSourcesOnFinalWatermark=true`
>> And if that works and you want to open a PR to update our
>> documentation that would be greatly appreciated.
>>
>> Regards,
>> Ismaël
>>
>>
>> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
>> <sr...@gmail.com> wrote:
>> >
>> > Hello,
>> >
>> > I am trying to run nexmark queries using flink runner streaming.
>> Followed the documentation and used the command
>> > ./gradlew :sdks:java:testing:nexmark:run \
>> >
>> > -Pnexmark.runner=":runners:flink:1.10" \
>> > -Pnexmark.args="
>> > --runner=FlinkRunner
>> > --suite=SMOKE
>> > --streamTimeout=60
>> > --streaming=true
>> > --manageResources=false
>> > --monitorJobs=true
>> > --flinkMaster=[local]"
>> >
>> >
>> > But after the events are read from the source, there is no further
>> progress and the job is always stuck at 99%. Is there any configuration
>> that I am missing?
>> >
>> > Regards,
>> > Sruthi
>>
>