You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Naum Gjorgjeski <na...@hotmail.com> on 2018/08/10 20:25:03 UTC
Re: Yahoo Streaming Benchmark on a Flink 1.5 cluster
Hi all,
I still cannot resolve the problem. Can I please get any advice for it? Thank you.
Best regards,
Naum Gjorgjeski
________________________________
From: Naum Gjorgjeski <na...@hotmail.com>
Sent: Wednesday, August 1, 2018 1:05 AM
To: user@flink.apache.org
Subject: Yahoo Streaming Benchmark on a Flink 1.5 cluster
Hi,
I am trying to run the data Artisans version of the Yahoo Streaming Benchmark. The benchmark applications are written for Flink 1.0.1. However, I need them to run on a Flink 1.5 cluster. When I try to build the benchmark applications with any version of Flink from 1.3.0 or higher, I get many compile errors. The compile errors state that some of the classes and methods cannot be found (because part of the Flink API has changed in recent versions).
The classes that cannot be found are:
org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner
org.apache.flink.api.common.state.OperatorState
org.apache.flink.api.common.state.StateBackend
org.apache.flink.runtime.state.AsynchronousStateHandle
org.apache.flink.runtime.state.StateHandle
org.apache.flink.streaming.runtime.tasks.StreamTaskState
org.apache.flink.runtime.state.AbstractStateBackend.CheckpointStateOutputView
The methods that cannot be found are:
org.apache.flink.streaming.api.operators.StreamOperator.snapshotOperatorState(long,long)
org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStateBackend()
org.apache.flink.runtime.state.AbstractStateBackend.createCheckpointStateOutputView(long,long)
I was able to find a substitution for few of them (e.g. using FlinkFixedPartitioner instead of FlinkPartitioner), but for most of them there are no straightforward substitutions. Could you please give me an advice on how to resolve this problem? Thank you in advance.
Best regards,
Naum
Re: Yahoo Streaming Benchmark on a Flink 1.5 cluster
Posted by Chesnay Schepler <ch...@apache.org>.
@vino Please take a deeper look at the question before answering them.
The benchmark implements custom operators that make use of several
non-Public APIs,
so API backwards compatibility simply doesn't apply here.
Updating the benchmark will require a bit of work. Below are some hints
i gathered, but given that we're jumping
5 versions at once (and 1.0 being really old) I can't guarantee anything
about the correctness.
The replacement for StateBackend is StateBinder. (apparently)
The OperatorState usage should be replaced with a ValueState usage.
The code in QueryableWindowOperatorEvicting#snapshotOperatorState can be
ported to a keyed ValueState,
have a look at the WindowOperator#open().
The StateHandle/StreamTaskState/CheckpointStateOutputView stuff should
become irrelevant after porting the operator to 1.5.
On 11.08.2018 04:14, vino yang wrote:
> Hi Namu,
>
> I don't think you need to pay attention to the internals of the Flink
> API.
> Its interface is backward compatible. If you update the dependent
> version of the API and the corresponding version of the Flink system
> so that their versions are consistent, there should be no problems.
> Please ensure that the version of Flink is consistent with the version
> of the API you are using.
>
> Thanks, vino.
>
> Naum Gjorgjeski <naum_gj@hotmail.com <ma...@hotmail.com>>
> 于2018年8月11日周六 上午4:25写道：
>
> Hi all,
>
>
> I still cannot resolve the problem. Can I please get any advice
> for it? Thank you.
>
>
> Best regards,
>
> Naum Gjorgjeski
>
>
> ------------------------------------------------------------------------
> *From:* Naum Gjorgjeski <naum_gj@hotmail.com
> <ma...@hotmail.com>>
> *Sent:* Wednesday, August 1, 2018 1:05 AM
> *To:* user@flink.apache.org <ma...@flink.apache.org>
> *Subject:* Yahoo Streaming Benchmark on a Flink 1.5 cluster
> Hi,
>
> I am trying to run the data Artisans version of the Yahoo
> Streaming Benchmark. The benchmark applications are written for
> Flink 1.0.1. However, I need them to run on a Flink 1.5 cluster.
> When I try to build the benchmark applications with any version
> of Flink from 1.3.0 or higher, I get many compile errors. The
> compile errors state that some of the classes and methods cannot
> be found (because part of the Flink API has changed in recent
> versions).
>
> The classes that cannot be found are:
> org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner
> org.apache.flink.api.common.state.OperatorState
> org.apache.flink.api.common.state.StateBackend
> org.apache.flink.runtime.state.AsynchronousStateHandle
> org.apache.flink.runtime.state.StateHandle
> org.apache.flink.streaming.runtime.tasks.StreamTaskState
> org.apache.flink.runtime.state.AbstractStateBackend.CheckpointStateOutputView
>
> The methods that cannot be found are:
> org.apache.flink.streaming.api.operators.StreamOperator.snapshotOperatorState(long,long)
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStateBackend()
> org.apache.flink.runtime.state.AbstractStateBackend.createCheckpointStateOutputView(long,long)
>
> I was able to find a substitution for few of them (e.g. using
> FlinkFixedPartitioner instead of FlinkPartitioner), but for most
> of them there are no straightforward substitutions. Could you
> please give me an advice on how to resolve this problem? Thank
> you in advance.
>
> Best regards,
> Naum
>
Re: Yahoo Streaming Benchmark on a Flink 1.5 cluster
Posted by vino yang <ya...@gmail.com>.
Hi Namu,
I don't think you need to pay attention to the internals of the Flink API.
Its interface is backward compatible. If you update the dependent version
of the API and the corresponding version of the Flink system so that their
versions are consistent, there should be no problems.
Please ensure that the version of Flink is consistent with the version of
the API you are using.
Thanks, vino.
Naum Gjorgjeski <na...@hotmail.com> 于2018年8月11日周六 上午4:25写道：
> Hi all,
>
>
> I still cannot resolve the problem. Can I please get any advice for it?
> Thank you.
>
>
> Best regards,
>
> Naum Gjorgjeski
>
> ------------------------------
> *From:* Naum Gjorgjeski <na...@hotmail.com>
> *Sent:* Wednesday, August 1, 2018 1:05 AM
> *To:* user@flink.apache.org
> *Subject:* Yahoo Streaming Benchmark on a Flink 1.5 cluster
>
>
> Hi,
>
> I am trying to run the data Artisans version of the Yahoo Streaming
> Benchmark. The benchmark applications are written for Flink 1.0.1.
> However, I need them to run on a Flink 1.5 cluster. When I try to build
> the benchmark applications with any version of Flink from 1.3.0 or higher,
> I get many compile errors. The compile errors state that some of the
> classes and methods cannot be found (because part of the Flink API has
> changed in recent versions).
>
> The classes that cannot be found are:
> org.apache.flink.streaming.connectors.kafka.partitioner.FixedPartitioner
> org.apache.flink.api.common.state.OperatorState
> org.apache.flink.api.common.state.StateBackend
> org.apache.flink.runtime.state.AsynchronousStateHandle
> org.apache.flink.runtime.state.StateHandle
> org.apache.flink.streaming.runtime.tasks.StreamTaskState
>
> org.apache.flink.runtime.state.AbstractStateBackend.CheckpointStateOutputView
>
> The methods that cannot be found are:
>
> org.apache.flink.streaming.api.operators.StreamOperator.snapshotOperatorState(long,long)
>
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStateBackend()
>
> org.apache.flink.runtime.state.AbstractStateBackend.createCheckpointStateOutputView(long,long)
>
> I was able to find a substitution for few of them (e.g. using
> FlinkFixedPartitioner instead of FlinkPartitioner), but for most of them
> there are no straightforward substitutions. Could you please give me an
> advice on how to resolve this problem? Thank you in advance.
>
> Best regards,
> Naum
>
>