You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Sruthi Sree Kumar <sr...@gmail.com> on 2020/04/21 20:04:34 UTC
Running NexMark Tests
Hello,
I am trying to run nexmark queries using flink runner streaming. Followed
the documentation and used the command
./gradlew :sdks:java:testing:nexmark:run \
-Pnexmark.runner=":runners:flink:1.10" \
-Pnexmark.args="
--runner=FlinkRunner
--suite=SMOKE
--streamTimeout=60
--streaming=true
--manageResources=false
--monitorJobs=true
--flinkMaster=[local]"
But after the events are read from the source, there is no further
progress and the job is always stuck at 99%. Is there any configuration
that I am missing?
Regards,
Sruthi
Re: Running NexMark Tests
Posted by Sruthi Sree Kumar <sr...@gmail.com>.
PR for the update: https://github.com/apache/beam/pull/11751
Regards,
Sruthi
On Tue, May 19, 2020 at 3:51 PM Maximilian Michels <mx...@apache.org> wrote:
> Looks like an accidental change to me. Running with either version, 1.9
> or 1.10 works, but this should be changed back to using the latest version.
>
> Do you mind creating a PR?
>
> Thanks,
> Max
>
> On 19.05.20 13:02, Sruthi Sree Kumar wrote:
> > On the documentation, the version of Flink runner is changed to 1.9
> > which was 1.10(latest)
> > before
> https://github.com/apache/beam/commit/1d2700818474c008eaa324ac1b5c49c9d2857298#diff-0e75160f4b09a1a300671557930589d9
> .
> >
> > Is this an accidental change or is there any particular reason for this
> > downgrade of version?
> >
> > Regards,
> > Sruthi
> >
> > On Tue, May 12, 2020 at 7:21 PM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > A heads-up if anybody else sees this, we have removed the flag:
> > https://jira.apache.org/jira/browse/BEAM-9900
> >
> > Further contributions are very welcome :)
> >
> > -Max
> >
> > On 11.05.20 17:05, Sruthi Sree Kumar wrote:
> > > I have opened a PR with the documentation change.
> > > https://github.com/apache/beam/pull/11662
> > >
> > > Regards,
> > > Sruthi
> > >
> > > On 2020/04/21 20:22:17, Ismaël Mejía <iemejia@gmail.com
> > <ma...@gmail.com>> wrote:
> > >> You need to instruct the Flink runner to shutdown the the source
> > >> otherwise it will stay waiting.
> > >> You can this by adding the extra
> > >> argument`--shutdownSourcesOnFinalWatermark=true`
> > >> And if that works and you want to open a PR to update our
> > >> documentation that would be greatly appreciated.
> > >>
> > >> Regards,
> > >> Ismaël
> > >>
> > >>
> > >> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> > >> <sruthisreekumar2712@gmail.com
> > <ma...@gmail.com>> wrote:
> > >>>
> > >>> Hello,
> > >>>
> > >>> I am trying to run nexmark queries using flink runner streaming.
> > Followed the documentation and used the command
> > >>> ./gradlew :sdks:java:testing:nexmark:run \
> > >>>
> > >>> -Pnexmark.runner=":runners:flink:1.10" \
> > >>> -Pnexmark.args="
> > >>> --runner=FlinkRunner
> > >>> --suite=SMOKE
> > >>> --streamTimeout=60
> > >>> --streaming=true
> > >>> --manageResources=false
> > >>> --monitorJobs=true
> > >>> --flinkMaster=[local]"
> > >>>
> > >>>
> > >>> But after the events are read from the source, there is no
> > further progress and the job is always stuck at 99%. Is there any
> > configuration that I am missing?
> > >>>
> > >>> Regards,
> > >>> Sruthi
> > >>
> >
>
Re: Running NexMark Tests
Posted by Sruthi Sree Kumar <sr...@gmail.com>.
PR for the update: https://github.com/apache/beam/pull/11751
Regards,
Sruthi
On Tue, May 19, 2020 at 3:51 PM Maximilian Michels <mx...@apache.org> wrote:
> Looks like an accidental change to me. Running with either version, 1.9
> or 1.10 works, but this should be changed back to using the latest version.
>
> Do you mind creating a PR?
>
> Thanks,
> Max
>
> On 19.05.20 13:02, Sruthi Sree Kumar wrote:
> > On the documentation, the version of Flink runner is changed to 1.9
> > which was 1.10(latest)
> > before
> https://github.com/apache/beam/commit/1d2700818474c008eaa324ac1b5c49c9d2857298#diff-0e75160f4b09a1a300671557930589d9
> .
> >
> > Is this an accidental change or is there any particular reason for this
> > downgrade of version?
> >
> > Regards,
> > Sruthi
> >
> > On Tue, May 12, 2020 at 7:21 PM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > A heads-up if anybody else sees this, we have removed the flag:
> > https://jira.apache.org/jira/browse/BEAM-9900
> >
> > Further contributions are very welcome :)
> >
> > -Max
> >
> > On 11.05.20 17:05, Sruthi Sree Kumar wrote:
> > > I have opened a PR with the documentation change.
> > > https://github.com/apache/beam/pull/11662
> > >
> > > Regards,
> > > Sruthi
> > >
> > > On 2020/04/21 20:22:17, Ismaël Mejía <iemejia@gmail.com
> > <ma...@gmail.com>> wrote:
> > >> You need to instruct the Flink runner to shutdown the the source
> > >> otherwise it will stay waiting.
> > >> You can this by adding the extra
> > >> argument`--shutdownSourcesOnFinalWatermark=true`
> > >> And if that works and you want to open a PR to update our
> > >> documentation that would be greatly appreciated.
> > >>
> > >> Regards,
> > >> Ismaël
> > >>
> > >>
> > >> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> > >> <sruthisreekumar2712@gmail.com
> > <ma...@gmail.com>> wrote:
> > >>>
> > >>> Hello,
> > >>>
> > >>> I am trying to run nexmark queries using flink runner streaming.
> > Followed the documentation and used the command
> > >>> ./gradlew :sdks:java:testing:nexmark:run \
> > >>>
> > >>> -Pnexmark.runner=":runners:flink:1.10" \
> > >>> -Pnexmark.args="
> > >>> --runner=FlinkRunner
> > >>> --suite=SMOKE
> > >>> --streamTimeout=60
> > >>> --streaming=true
> > >>> --manageResources=false
> > >>> --monitorJobs=true
> > >>> --flinkMaster=[local]"
> > >>>
> > >>>
> > >>> But after the events are read from the source, there is no
> > further progress and the job is always stuck at 99%. Is there any
> > configuration that I am missing?
> > >>>
> > >>> Regards,
> > >>> Sruthi
> > >>
> >
>
Re: Running NexMark Tests
Posted by Maximilian Michels <mx...@apache.org>.
Looks like an accidental change to me. Running with either version, 1.9
or 1.10 works, but this should be changed back to using the latest version.
Do you mind creating a PR?
Thanks,
Max
On 19.05.20 13:02, Sruthi Sree Kumar wrote:
> On the documentation, the version of Flink runner is changed to 1.9
> which was 1.10(latest)
> before https://github.com/apache/beam/commit/1d2700818474c008eaa324ac1b5c49c9d2857298#diff-0e75160f4b09a1a300671557930589d9.
>
> Is this an accidental change or is there any particular reason for this
> downgrade of version?
>
> Regards,
> Sruthi
>
> On Tue, May 12, 2020 at 7:21 PM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> A heads-up if anybody else sees this, we have removed the flag:
> https://jira.apache.org/jira/browse/BEAM-9900
>
> Further contributions are very welcome :)
>
> -Max
>
> On 11.05.20 17:05, Sruthi Sree Kumar wrote:
> > I have opened a PR with the documentation change.
> > https://github.com/apache/beam/pull/11662
> >
> > Regards,
> > Sruthi
> >
> > On 2020/04/21 20:22:17, Ismaël Mejía <iemejia@gmail.com
> <ma...@gmail.com>> wrote:
> >> You need to instruct the Flink runner to shutdown the the source
> >> otherwise it will stay waiting.
> >> You can this by adding the extra
> >> argument`--shutdownSourcesOnFinalWatermark=true`
> >> And if that works and you want to open a PR to update our
> >> documentation that would be greatly appreciated.
> >>
> >> Regards,
> >> Ismaël
> >>
> >>
> >> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> >> <sruthisreekumar2712@gmail.com
> <ma...@gmail.com>> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am trying to run nexmark queries using flink runner streaming.
> Followed the documentation and used the command
> >>> ./gradlew :sdks:java:testing:nexmark:run \
> >>>
> >>> -Pnexmark.runner=":runners:flink:1.10" \
> >>> -Pnexmark.args="
> >>> --runner=FlinkRunner
> >>> --suite=SMOKE
> >>> --streamTimeout=60
> >>> --streaming=true
> >>> --manageResources=false
> >>> --monitorJobs=true
> >>> --flinkMaster=[local]"
> >>>
> >>>
> >>> But after the events are read from the source, there is no
> further progress and the job is always stuck at 99%. Is there any
> configuration that I am missing?
> >>>
> >>> Regards,
> >>> Sruthi
> >>
>
Re: Running NexMark Tests
Posted by Maximilian Michels <mx...@apache.org>.
Looks like an accidental change to me. Running with either version, 1.9
or 1.10 works, but this should be changed back to using the latest version.
Do you mind creating a PR?
Thanks,
Max
On 19.05.20 13:02, Sruthi Sree Kumar wrote:
> On the documentation, the version of Flink runner is changed to 1.9
> which was 1.10(latest)
> before https://github.com/apache/beam/commit/1d2700818474c008eaa324ac1b5c49c9d2857298#diff-0e75160f4b09a1a300671557930589d9.
>
> Is this an accidental change or is there any particular reason for this
> downgrade of version?
>
> Regards,
> Sruthi
>
> On Tue, May 12, 2020 at 7:21 PM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> A heads-up if anybody else sees this, we have removed the flag:
> https://jira.apache.org/jira/browse/BEAM-9900
>
> Further contributions are very welcome :)
>
> -Max
>
> On 11.05.20 17:05, Sruthi Sree Kumar wrote:
> > I have opened a PR with the documentation change.
> > https://github.com/apache/beam/pull/11662
> >
> > Regards,
> > Sruthi
> >
> > On 2020/04/21 20:22:17, Ismaël Mejía <iemejia@gmail.com
> <ma...@gmail.com>> wrote:
> >> You need to instruct the Flink runner to shutdown the the source
> >> otherwise it will stay waiting.
> >> You can this by adding the extra
> >> argument`--shutdownSourcesOnFinalWatermark=true`
> >> And if that works and you want to open a PR to update our
> >> documentation that would be greatly appreciated.
> >>
> >> Regards,
> >> Ismaël
> >>
> >>
> >> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> >> <sruthisreekumar2712@gmail.com
> <ma...@gmail.com>> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am trying to run nexmark queries using flink runner streaming.
> Followed the documentation and used the command
> >>> ./gradlew :sdks:java:testing:nexmark:run \
> >>>
> >>> -Pnexmark.runner=":runners:flink:1.10" \
> >>> -Pnexmark.args="
> >>> --runner=FlinkRunner
> >>> --suite=SMOKE
> >>> --streamTimeout=60
> >>> --streaming=true
> >>> --manageResources=false
> >>> --monitorJobs=true
> >>> --flinkMaster=[local]"
> >>>
> >>>
> >>> But after the events are read from the source, there is no
> further progress and the job is always stuck at 99%. Is there any
> configuration that I am missing?
> >>>
> >>> Regards,
> >>> Sruthi
> >>
>
Re: Running NexMark Tests
Posted by Sruthi Sree Kumar <sr...@gmail.com>.
On the documentation, the version of Flink runner is changed to 1.9 which
was 1.10(latest) before
https://github.com/apache/beam/commit/1d2700818474c008eaa324ac1b5c49c9d2857298#diff-0e75160f4b09a1a300671557930589d9
.
Is this an accidental change or is there any particular reason for this
downgrade of version?
Regards,
Sruthi
On Tue, May 12, 2020 at 7:21 PM Maximilian Michels <mx...@apache.org> wrote:
> A heads-up if anybody else sees this, we have removed the flag:
> https://jira.apache.org/jira/browse/BEAM-9900
>
> Further contributions are very welcome :)
>
> -Max
>
> On 11.05.20 17:05, Sruthi Sree Kumar wrote:
> > I have opened a PR with the documentation change.
> > https://github.com/apache/beam/pull/11662
> >
> > Regards,
> > Sruthi
> >
> > On 2020/04/21 20:22:17, Ismaël Mejía <ie...@gmail.com> wrote:
> >> You need to instruct the Flink runner to shutdown the the source
> >> otherwise it will stay waiting.
> >> You can this by adding the extra
> >> argument`--shutdownSourcesOnFinalWatermark=true`
> >> And if that works and you want to open a PR to update our
> >> documentation that would be greatly appreciated.
> >>
> >> Regards,
> >> Ismaël
> >>
> >>
> >> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> >> <sr...@gmail.com> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am trying to run nexmark queries using flink runner streaming.
> Followed the documentation and used the command
> >>> ./gradlew :sdks:java:testing:nexmark:run \
> >>>
> >>> -Pnexmark.runner=":runners:flink:1.10" \
> >>> -Pnexmark.args="
> >>> --runner=FlinkRunner
> >>> --suite=SMOKE
> >>> --streamTimeout=60
> >>> --streaming=true
> >>> --manageResources=false
> >>> --monitorJobs=true
> >>> --flinkMaster=[local]"
> >>>
> >>>
> >>> But after the events are read from the source, there is no further
> progress and the job is always stuck at 99%. Is there any configuration
> that I am missing?
> >>>
> >>> Regards,
> >>> Sruthi
> >>
>
Re: Running NexMark Tests
Posted by Sruthi Sree Kumar <sr...@gmail.com>.
On the documentation, the version of Flink runner is changed to 1.9 which
was 1.10(latest) before
https://github.com/apache/beam/commit/1d2700818474c008eaa324ac1b5c49c9d2857298#diff-0e75160f4b09a1a300671557930589d9
.
Is this an accidental change or is there any particular reason for this
downgrade of version?
Regards,
Sruthi
On Tue, May 12, 2020 at 7:21 PM Maximilian Michels <mx...@apache.org> wrote:
> A heads-up if anybody else sees this, we have removed the flag:
> https://jira.apache.org/jira/browse/BEAM-9900
>
> Further contributions are very welcome :)
>
> -Max
>
> On 11.05.20 17:05, Sruthi Sree Kumar wrote:
> > I have opened a PR with the documentation change.
> > https://github.com/apache/beam/pull/11662
> >
> > Regards,
> > Sruthi
> >
> > On 2020/04/21 20:22:17, Ismaël Mejía <ie...@gmail.com> wrote:
> >> You need to instruct the Flink runner to shutdown the the source
> >> otherwise it will stay waiting.
> >> You can this by adding the extra
> >> argument`--shutdownSourcesOnFinalWatermark=true`
> >> And if that works and you want to open a PR to update our
> >> documentation that would be greatly appreciated.
> >>
> >> Regards,
> >> Ismaël
> >>
> >>
> >> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> >> <sr...@gmail.com> wrote:
> >>>
> >>> Hello,
> >>>
> >>> I am trying to run nexmark queries using flink runner streaming.
> Followed the documentation and used the command
> >>> ./gradlew :sdks:java:testing:nexmark:run \
> >>>
> >>> -Pnexmark.runner=":runners:flink:1.10" \
> >>> -Pnexmark.args="
> >>> --runner=FlinkRunner
> >>> --suite=SMOKE
> >>> --streamTimeout=60
> >>> --streaming=true
> >>> --manageResources=false
> >>> --monitorJobs=true
> >>> --flinkMaster=[local]"
> >>>
> >>>
> >>> But after the events are read from the source, there is no further
> progress and the job is always stuck at 99%. Is there any configuration
> that I am missing?
> >>>
> >>> Regards,
> >>> Sruthi
> >>
>
Re: Running NexMark Tests
Posted by Maximilian Michels <mx...@apache.org>.
A heads-up if anybody else sees this, we have removed the flag:
https://jira.apache.org/jira/browse/BEAM-9900
Further contributions are very welcome :)
-Max
On 11.05.20 17:05, Sruthi Sree Kumar wrote:
> I have opened a PR with the documentation change.
> https://github.com/apache/beam/pull/11662
>
> Regards,
> Sruthi
>
> On 2020/04/21 20:22:17, Ismaël Mejía <ie...@gmail.com> wrote:
>> You need to instruct the Flink runner to shutdown the the source
>> otherwise it will stay waiting.
>> You can this by adding the extra
>> argument`--shutdownSourcesOnFinalWatermark=true`
>> And if that works and you want to open a PR to update our
>> documentation that would be greatly appreciated.
>>
>> Regards,
>> Ismaël
>>
>>
>> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
>> <sr...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> I am trying to run nexmark queries using flink runner streaming. Followed the documentation and used the command
>>> ./gradlew :sdks:java:testing:nexmark:run \
>>>
>>> -Pnexmark.runner=":runners:flink:1.10" \
>>> -Pnexmark.args="
>>> --runner=FlinkRunner
>>> --suite=SMOKE
>>> --streamTimeout=60
>>> --streaming=true
>>> --manageResources=false
>>> --monitorJobs=true
>>> --flinkMaster=[local]"
>>>
>>>
>>> But after the events are read from the source, there is no further progress and the job is always stuck at 99%. Is there any configuration that I am missing?
>>>
>>> Regards,
>>> Sruthi
>>
Re: Running NexMark Tests
Posted by Maximilian Michels <mx...@apache.org>.
A heads-up if anybody else sees this, we have removed the flag:
https://jira.apache.org/jira/browse/BEAM-9900
Further contributions are very welcome :)
-Max
On 11.05.20 17:05, Sruthi Sree Kumar wrote:
> I have opened a PR with the documentation change.
> https://github.com/apache/beam/pull/11662
>
> Regards,
> Sruthi
>
> On 2020/04/21 20:22:17, Ismaël Mejía <ie...@gmail.com> wrote:
>> You need to instruct the Flink runner to shutdown the the source
>> otherwise it will stay waiting.
>> You can this by adding the extra
>> argument`--shutdownSourcesOnFinalWatermark=true`
>> And if that works and you want to open a PR to update our
>> documentation that would be greatly appreciated.
>>
>> Regards,
>> Ismaël
>>
>>
>> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
>> <sr...@gmail.com> wrote:
>>>
>>> Hello,
>>>
>>> I am trying to run nexmark queries using flink runner streaming. Followed the documentation and used the command
>>> ./gradlew :sdks:java:testing:nexmark:run \
>>>
>>> -Pnexmark.runner=":runners:flink:1.10" \
>>> -Pnexmark.args="
>>> --runner=FlinkRunner
>>> --suite=SMOKE
>>> --streamTimeout=60
>>> --streaming=true
>>> --manageResources=false
>>> --monitorJobs=true
>>> --flinkMaster=[local]"
>>>
>>>
>>> But after the events are read from the source, there is no further progress and the job is always stuck at 99%. Is there any configuration that I am missing?
>>>
>>> Regards,
>>> Sruthi
>>
Re: Running NexMark Tests
Posted by Sruthi Sree Kumar <sr...@gmail.com>.
I have opened a PR with the documentation change.
https://github.com/apache/beam/pull/11662
Regards,
Sruthi
On 2020/04/21 20:22:17, Ismaël Mejía <ie...@gmail.com> wrote:
> You need to instruct the Flink runner to shutdown the the source
> otherwise it will stay waiting.
> You can this by adding the extra
> argument`--shutdownSourcesOnFinalWatermark=true`
> And if that works and you want to open a PR to update our
> documentation that would be greatly appreciated.
>
> Regards,
> Ismaël
>
>
> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> <sr...@gmail.com> wrote:
> >
> > Hello,
> >
> > I am trying to run nexmark queries using flink runner streaming. Followed the documentation and used the command
> > ./gradlew :sdks:java:testing:nexmark:run \
> >
> > -Pnexmark.runner=":runners:flink:1.10" \
> > -Pnexmark.args="
> > --runner=FlinkRunner
> > --suite=SMOKE
> > --streamTimeout=60
> > --streaming=true
> > --manageResources=false
> > --monitorJobs=true
> > --flinkMaster=[local]"
> >
> >
> > But after the events are read from the source, there is no further progress and the job is always stuck at 99%. Is there any configuration that I am missing?
> >
> > Regards,
> > Sruthi
>
Re: Running NexMark Tests
Posted by Sruthi Sree Kumar <sr...@gmail.com>.
Thank you. It worked with the argument '--shutdownSourcesOnFinalWatermark=true'
I will open PR to update the documentation.
Regards,
Sruthi
On 2020/04/21 20:22:17, Ismaël Mejía <ie...@gmail.com> wrote:
> You need to instruct the Flink runner to shutdown the the source
> otherwise it will stay waiting.
> You can this by adding the extra
> argument`--shutdownSourcesOnFinalWatermark=true`
> And if that works and you want to open a PR to update our
> documentation that would be greatly appreciated.
>
> Regards,
> Ismaël
>
>
> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> <sr...@gmail.com> wrote:
> >
> > Hello,
> >
> > I am trying to run nexmark queries using flink runner streaming. Followed the documentation and used the command
> > ./gradlew :sdks:java:testing:nexmark:run \
> >
> > -Pnexmark.runner=":runners:flink:1.10" \
> > -Pnexmark.args="
> > --runner=FlinkRunner
> > --suite=SMOKE
> > --streamTimeout=60
> > --streaming=true
> > --manageResources=false
> > --monitorJobs=true
> > --flinkMaster=[local]"
> >
> >
> > But after the events are read from the source, there is no further progress and the job is always stuck at 99%. Is there any configuration that I am missing?
> >
> > Regards,
> > Sruthi
>
Re: Running NexMark Tests
Posted by Maximilian Michels <mx...@apache.org>.
The flag is needed when checkpointing is enabled because Flink is unable
to create a new checkpoint when not all operators are running.
By default, operators shut down when all input has been read. That will
trigger sending out the maximum (final) watermark at the sources. The
flag name is a bit confusing in this regard because shutting down the
sources triggers sending out the watermark, not the other way around.
-Max
On 22.04.20 06:26, Kenneth Knowles wrote:
> We should always want to shut down sources on final watermark. All
> incoming data should be dropped anyhow.
>
> Kenn
>
> On Tue, Apr 21, 2020 at 1:34 PM Luke Cwik <lcwik@google.com
> <ma...@google.com>> wrote:
>
> +dev
>
> When would we not want --shutdownSourcesOnFinalWatermark=true ?
>
> On Tue, Apr 21, 2020 at 1:22 PM Ismaël Mejía <iemejia@gmail.com
> <ma...@gmail.com>> wrote:
>
> You need to instruct the Flink runner to shutdown the the source
> otherwise it will stay waiting.
> You can this by adding the extra
> argument`--shutdownSourcesOnFinalWatermark=true`
> And if that works and you want to open a PR to update our
> documentation that would be greatly appreciated.
>
> Regards,
> Ismaël
>
>
> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> <sruthisreekumar2712@gmail.com
> <ma...@gmail.com>> wrote:
> >
> > Hello,
> >
> > I am trying to run nexmark queries using flink runner
> streaming. Followed the documentation and used the command
> > ./gradlew :sdks:java:testing:nexmark:run \
> >
> > -Pnexmark.runner=":runners:flink:1.10" \
> > -Pnexmark.args="
> > --runner=FlinkRunner
> > --suite=SMOKE
> > --streamTimeout=60
> > --streaming=true
> > --manageResources=false
> > --monitorJobs=true
> > --flinkMaster=[local]"
> >
> >
> > But after the events are read from the source, there is no
> further progress and the job is always stuck at 99%. Is there
> any configuration that I am missing?
> >
> > Regards,
> > Sruthi
>
Re: Running NexMark Tests
Posted by Maximilian Michels <mx...@apache.org>.
The flag is needed when checkpointing is enabled because Flink is unable
to create a new checkpoint when not all operators are running.
By default, operators shut down when all input has been read. That will
trigger sending out the maximum (final) watermark at the sources. The
flag name is a bit confusing in this regard because shutting down the
sources triggers sending out the watermark, not the other way around.
-Max
On 22.04.20 06:26, Kenneth Knowles wrote:
> We should always want to shut down sources on final watermark. All
> incoming data should be dropped anyhow.
>
> Kenn
>
> On Tue, Apr 21, 2020 at 1:34 PM Luke Cwik <lcwik@google.com
> <ma...@google.com>> wrote:
>
> +dev
>
> When would we not want --shutdownSourcesOnFinalWatermark=true ?
>
> On Tue, Apr 21, 2020 at 1:22 PM Ismaël Mejía <iemejia@gmail.com
> <ma...@gmail.com>> wrote:
>
> You need to instruct the Flink runner to shutdown the the source
> otherwise it will stay waiting.
> You can this by adding the extra
> argument`--shutdownSourcesOnFinalWatermark=true`
> And if that works and you want to open a PR to update our
> documentation that would be greatly appreciated.
>
> Regards,
> Ismaël
>
>
> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> <sruthisreekumar2712@gmail.com
> <ma...@gmail.com>> wrote:
> >
> > Hello,
> >
> > I am trying to run nexmark queries using flink runner
> streaming. Followed the documentation and used the command
> > ./gradlew :sdks:java:testing:nexmark:run \
> >
> > -Pnexmark.runner=":runners:flink:1.10" \
> > -Pnexmark.args="
> > --runner=FlinkRunner
> > --suite=SMOKE
> > --streamTimeout=60
> > --streaming=true
> > --manageResources=false
> > --monitorJobs=true
> > --flinkMaster=[local]"
> >
> >
> > But after the events are read from the source, there is no
> further progress and the job is always stuck at 99%. Is there
> any configuration that I am missing?
> >
> > Regards,
> > Sruthi
>
Re: Running NexMark Tests
Posted by Kenneth Knowles <ke...@apache.org>.
We should always want to shut down sources on final watermark. All incoming
data should be dropped anyhow.
Kenn
On Tue, Apr 21, 2020 at 1:34 PM Luke Cwik <lc...@google.com> wrote:
> +dev
>
> When would we not want --shutdownSourcesOnFinalWatermark=true ?
>
> On Tue, Apr 21, 2020 at 1:22 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> You need to instruct the Flink runner to shutdown the the source
>> otherwise it will stay waiting.
>> You can this by adding the extra
>> argument`--shutdownSourcesOnFinalWatermark=true`
>> And if that works and you want to open a PR to update our
>> documentation that would be greatly appreciated.
>>
>> Regards,
>> Ismaël
>>
>>
>> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
>> <sr...@gmail.com> wrote:
>> >
>> > Hello,
>> >
>> > I am trying to run nexmark queries using flink runner streaming.
>> Followed the documentation and used the command
>> > ./gradlew :sdks:java:testing:nexmark:run \
>> >
>> > -Pnexmark.runner=":runners:flink:1.10" \
>> > -Pnexmark.args="
>> > --runner=FlinkRunner
>> > --suite=SMOKE
>> > --streamTimeout=60
>> > --streaming=true
>> > --manageResources=false
>> > --monitorJobs=true
>> > --flinkMaster=[local]"
>> >
>> >
>> > But after the events are read from the source, there is no further
>> progress and the job is always stuck at 99%. Is there any configuration
>> that I am missing?
>> >
>> > Regards,
>> > Sruthi
>>
>
Re: Running NexMark Tests
Posted by Kenneth Knowles <ke...@apache.org>.
We should always want to shut down sources on final watermark. All incoming
data should be dropped anyhow.
Kenn
On Tue, Apr 21, 2020 at 1:34 PM Luke Cwik <lc...@google.com> wrote:
> +dev
>
> When would we not want --shutdownSourcesOnFinalWatermark=true ?
>
> On Tue, Apr 21, 2020 at 1:22 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> You need to instruct the Flink runner to shutdown the the source
>> otherwise it will stay waiting.
>> You can this by adding the extra
>> argument`--shutdownSourcesOnFinalWatermark=true`
>> And if that works and you want to open a PR to update our
>> documentation that would be greatly appreciated.
>>
>> Regards,
>> Ismaël
>>
>>
>> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
>> <sr...@gmail.com> wrote:
>> >
>> > Hello,
>> >
>> > I am trying to run nexmark queries using flink runner streaming.
>> Followed the documentation and used the command
>> > ./gradlew :sdks:java:testing:nexmark:run \
>> >
>> > -Pnexmark.runner=":runners:flink:1.10" \
>> > -Pnexmark.args="
>> > --runner=FlinkRunner
>> > --suite=SMOKE
>> > --streamTimeout=60
>> > --streaming=true
>> > --manageResources=false
>> > --monitorJobs=true
>> > --flinkMaster=[local]"
>> >
>> >
>> > But after the events are read from the source, there is no further
>> progress and the job is always stuck at 99%. Is there any configuration
>> that I am missing?
>> >
>> > Regards,
>> > Sruthi
>>
>
Re: Running NexMark Tests
Posted by Luke Cwik <lc...@google.com>.
+dev
When would we not want --shutdownSourcesOnFinalWatermark=true ?
On Tue, Apr 21, 2020 at 1:22 PM Ismaël Mejía <ie...@gmail.com> wrote:
> You need to instruct the Flink runner to shutdown the the source
> otherwise it will stay waiting.
> You can this by adding the extra
> argument`--shutdownSourcesOnFinalWatermark=true`
> And if that works and you want to open a PR to update our
> documentation that would be greatly appreciated.
>
> Regards,
> Ismaël
>
>
> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> <sr...@gmail.com> wrote:
> >
> > Hello,
> >
> > I am trying to run nexmark queries using flink runner streaming.
> Followed the documentation and used the command
> > ./gradlew :sdks:java:testing:nexmark:run \
> >
> > -Pnexmark.runner=":runners:flink:1.10" \
> > -Pnexmark.args="
> > --runner=FlinkRunner
> > --suite=SMOKE
> > --streamTimeout=60
> > --streaming=true
> > --manageResources=false
> > --monitorJobs=true
> > --flinkMaster=[local]"
> >
> >
> > But after the events are read from the source, there is no further
> progress and the job is always stuck at 99%. Is there any configuration
> that I am missing?
> >
> > Regards,
> > Sruthi
>
Re: Running NexMark Tests
Posted by Luke Cwik <lc...@google.com>.
+dev
When would we not want --shutdownSourcesOnFinalWatermark=true ?
On Tue, Apr 21, 2020 at 1:22 PM Ismaël Mejía <ie...@gmail.com> wrote:
> You need to instruct the Flink runner to shutdown the the source
> otherwise it will stay waiting.
> You can this by adding the extra
> argument`--shutdownSourcesOnFinalWatermark=true`
> And if that works and you want to open a PR to update our
> documentation that would be greatly appreciated.
>
> Regards,
> Ismaël
>
>
> On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
> <sr...@gmail.com> wrote:
> >
> > Hello,
> >
> > I am trying to run nexmark queries using flink runner streaming.
> Followed the documentation and used the command
> > ./gradlew :sdks:java:testing:nexmark:run \
> >
> > -Pnexmark.runner=":runners:flink:1.10" \
> > -Pnexmark.args="
> > --runner=FlinkRunner
> > --suite=SMOKE
> > --streamTimeout=60
> > --streaming=true
> > --manageResources=false
> > --monitorJobs=true
> > --flinkMaster=[local]"
> >
> >
> > But after the events are read from the source, there is no further
> progress and the job is always stuck at 99%. Is there any configuration
> that I am missing?
> >
> > Regards,
> > Sruthi
>
Re: Running NexMark Tests
Posted by Ismaël Mejía <ie...@gmail.com>.
You need to instruct the Flink runner to shutdown the the source
otherwise it will stay waiting.
You can this by adding the extra
argument`--shutdownSourcesOnFinalWatermark=true`
And if that works and you want to open a PR to update our
documentation that would be greatly appreciated.
Regards,
Ismaël
On Tue, Apr 21, 2020 at 10:04 PM Sruthi Sree Kumar
<sr...@gmail.com> wrote:
>
> Hello,
>
> I am trying to run nexmark queries using flink runner streaming. Followed the documentation and used the command
> ./gradlew :sdks:java:testing:nexmark:run \
>
> -Pnexmark.runner=":runners:flink:1.10" \
> -Pnexmark.args="
> --runner=FlinkRunner
> --suite=SMOKE
> --streamTimeout=60
> --streaming=true
> --manageResources=false
> --monitorJobs=true
> --flinkMaster=[local]"
>
>
> But after the events are read from the source, there is no further progress and the job is always stuck at 99%. Is there any configuration that I am missing?
>
> Regards,
> Sruthi