You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by AndreaKinn <ki...@hotmail.it> on 2017/08/29 11:19:10 UTC
datastream.print() doesn't works
Hi,
I have a simple datastream of a Tuple2. Unfortunately when I call the
print() method. No one output is showed although no errors or exceptions are
raised.
I want to highlight that I have also other data streams which are correctly
printed.
This is the stream:
/DataStream<Tuple2<Double,Double>> result = HTM.learn(kafkaStream, new
Harness.AnomalyNetwork())
.select(new InferenceSelectFunction<Harness.KafkaRecord,
Tuple2<Double, Double>>() {
@Override
public Tuple2<Double,Double>
select(Tuple2<Harness.KafkaRecord, NetworkInference> inference) throws
Exception {
return new Tuple2<Double, Double>(3.14444,
inference.f1.getAnomalyScore());
}
});/
I'm quite sure the problem isn't in the particular function described above
because effectively in inner functions of getAnomalyScore for example the
return value exists.
Anyway I'm not able to find the problem which doesn't allow to print the
stream so I thought to write here to discover if there are typical situation
where streams are not printed. How can I verify and debug the problem? Some
hints about it ? I really tried everything, also tried to print something in
the function select() for example, with no results
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: datastream.print() doesn't works
Posted by AndreaKinn <ki...@hotmail.it>.
Using mvn clean now I obtain:
Error: Could not find or load main class
org.apache.flink.mainProgram.StreamingJob
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223p15239.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: datastream.print() doesn't works
Posted by AndreaKinn <ki...@hotmail.it>.
I verified I use just one environment. Unfortunately, (also without using
start-local.sh) /callingcreateLocalEnvironmentWithWebUI()/ and run the
program from the IDE anyway no one running jobs is listed in the dashboard
at /http://localhost:8081/#/overview/.
In the ide it is correctly executed meanwhile.
I pasted my code here pastebin <https://pastebin.com/raw/1WFFvfJF> . Tell
if you need other classes.
Thank you
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: datastream.print() doesn't works
Posted by Chesnay Schepler <ch...@apache.org>.
If you call createLocalEnvironmentWithWebUI you don't need to start a
cluster with the start-local.sh script,
you can run it from the IDE and it will start the web UI. If you submit
a job to a cluster that was started outside the IDE
you can call getExecutionEnvironment as usual.
Not sure why the errors are different, but the message generally occurs
when you call execute() on an environment
where you never defined any operators on. Please check that you're not
creating multiple environments.
If possible it might also be helpful to provide us (or me personally)
with the code of your program.
On 30.08.2017 19:08, AndreaKinn wrote:
> Hi, in the night uninstalling and re-installing maven and flink I solved my
> issue.
>
> I started the web dashboard using start-local.sh script and used
> /createLocalEnvironmentWithWebUI(new Configuration())/ as you suggested.
> Anyway when I start it in eclipse in the ui dashboard no running jobs are
> showed (as I expected).
> To run it via console with the console command:
>
> /./bin/flink run -c org.apache.flink.quickstart.StreamingJob
> target/flink-java-project-0.1.jar/
>
> but an error appears:
>
> /No operators defined in streaming topology. Cannot execute.
> /
>
> Strangely, if I execute it in eclipse no errors appear. Do you know which
> operators it refers??
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>
Re: datastream.print() doesn't works
Posted by AndreaKinn <ki...@hotmail.it>.
Hi, in the night uninstalling and re-installing maven and flink I solved my
issue.
I started the web dashboard using start-local.sh script and used
/createLocalEnvironmentWithWebUI(new Configuration())/ as you suggested.
Anyway when I start it in eclipse in the ui dashboard no running jobs are
showed (as I expected).
To run it via console with the console command:
/./bin/flink run -c org.apache.flink.quickstart.StreamingJob
target/flink-java-project-0.1.jar/
but an error appears:
/No operators defined in streaming topology. Cannot execute.
/
Strangely, if I execute it in eclipse no errors appear. Do you know which
operators it refers??
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: datastream.print() doesn't works
Posted by Chesnay Schepler <ch...@apache.org>.
I'm afraid I don't know anything about eclipse.
Running mvn clean package will first delete all files in the /target
directory, and then recompile your code to that very location.
It shouldn't affect an IDE in a way that isn't resolvable by rebuilding
the project in it.
You could also try re-importing your code as a new project and see if it
helps.
Note that you can start a Flink job with a functioning web UI from the
IDE by calling
`StreamExecutionEnvironment#createLocalEnvironmentWithWebUI()`, it is
not necessary to build a jar.
On 29.08.2017 22:00, AndreaKinn wrote:
> Chesnay Schepler wrote
>> The easiest explanation is that there is nothing to print.
>>
>> Since print statements within the select function don't appear in the
>> logs I assume
>> that the result of HTM.learn is empty.
>>
>> Please check via the webUI or metrics whether any of these operations
>> actually return records.
>>
>> On 29.08.2017 13:19, AndreaKinn wrote:
>>> Hi,
>>> I have a simple datastream of a Tuple2. Unfortunately when I call the
>>> print() method. No one output is showed although no errors or exceptions
>>> are
>>> raised.
>>> I want to highlight that I have also other data streams which are
>>> correctly
>>> printed.
>>>
>>> This is the stream:
>>>
>>> /DataStream<Tuple2&lt;Double,Double>> result =
>>> HTM.learn(kafkaStream, new
>>> Harness.AnomalyNetwork())
>>> .select(new InferenceSelectFunction<Harness.KafkaRecord,
>> > Tuple2<Double, Double>>() {
>>> @Override
>>> public
>>> Tuple2<Double,Double>
>>> select(Tuple2<Harness.KafkaRecord, NetworkInference> inference)
>>> throws
>>> Exception {
>>> return new Tuple2<Double,
>>> Double>(3.14444,
>>> inference.f1.getAnomalyScore());
>>> }
>>> });/
>>>
>>> I'm quite sure the problem isn't in the particular function described
>>> above
>>> because effectively in inner functions of getAnomalyScore for example the
>>> return value exists.
>>>
>>> Anyway I'm not able to find the problem which doesn't allow to print the
>>> stream so I thought to write here to discover if there are typical
>>> situation
>>> where streams are not printed. How can I verify and debug the problem?
>>> Some
>>> hints about it ? I really tried everything, also tried to print something
>>> in
>>> the function select() for example, with no results
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223.html
>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive at Nabble.com.
>>>
> I'm losing my head in 6 hours...
> To try to start flink job in webUI I tried to generate the jar of my flink
> program using mvn clean package command.
> I had to solve several problems about external libraries, then the command
> worked but now in Eclipse everything is fucked. I have many errors something
> like ClassNotFoundException. I tried everything but I'm not able to rewind
> the program to the last functioning configuration. I'm desperate...
>
> I'm not an expert of maven, what it does exactly and how can I cancel the
> clean operation to go back to the previous config?
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223p15237.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>
Re: datastream.print() doesn't works
Posted by AndreaKinn <ki...@hotmail.it>.
Chesnay Schepler wrote
> The easiest explanation is that there is nothing to print.
>
> Since print statements within the select function don't appear in the
> logs I assume
> that the result of HTM.learn is empty.
>
> Please check via the webUI or metrics whether any of these operations
> actually return records.
>
> On 29.08.2017 13:19, AndreaKinn wrote:
>> Hi,
>> I have a simple datastream of a Tuple2. Unfortunately when I call the
>> print() method. No one output is showed although no errors or exceptions
>> are
>> raised.
>> I want to highlight that I have also other data streams which are
>> correctly
>> printed.
>>
>> This is the stream:
>>
>> /DataStream<Tuple2&lt;Double,Double>> result =
>> HTM.learn(kafkaStream, new
>> Harness.AnomalyNetwork())
>> .select(new InferenceSelectFunction<Harness.KafkaRecord,
> > Tuple2<Double, Double>>() {
>> @Override
>> public
>> Tuple2<Double,Double>
>> select(Tuple2<Harness.KafkaRecord, NetworkInference> inference)
>> throws
>> Exception {
>> return new Tuple2<Double,
>> Double>(3.14444,
>> inference.f1.getAnomalyScore());
>> }
>> });/
>>
>> I'm quite sure the problem isn't in the particular function described
>> above
>> because effectively in inner functions of getAnomalyScore for example the
>> return value exists.
>>
>> Anyway I'm not able to find the problem which doesn't allow to print the
>> stream so I thought to write here to discover if there are typical
>> situation
>> where streams are not printed. How can I verify and debug the problem?
>> Some
>> hints about it ? I really tried everything, also tried to print something
>> in
>> the function select() for example, with no results
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
I'm losing my head in 6 hours...
To try to start flink job in webUI I tried to generate the jar of my flink
program using mvn clean package command.
I had to solve several problems about external libraries, then the command
worked but now in Eclipse everything is fucked. I have many errors something
like ClassNotFoundException. I tried everything but I'm not able to rewind
the program to the last functioning configuration. I'm desperate...
I'm not an expert of maven, what it does exactly and how can I cancel the
clean operation to go back to the previous config?
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223p15237.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: datastream.print() doesn't works
Posted by AndreaKinn <ki...@hotmail.it>.
I call env.execute().
What do you mean for "configure the logger?"
Timo Walther wrote
> Don't forget to call env.execute() at the end and make sure you have
> configured your logger correctly.
>
> Regards,
> Timo
>
> Am 29.08.17 um 14:59 schrieb Chesnay Schepler:
>> The easiest explanation is that there is nothing to print.
>>
>> Since print statements within the select function don't appear in the
>> logs I assume
>> that the result of HTM.learn is empty.
>>
>> Please check via the webUI or metrics whether any of these operations
>> actually return records.
>>
>> On 29.08.2017 13:19, AndreaKinn wrote:
>>> Hi,
>>> I have a simple datastream of a Tuple2. Unfortunately when I call the
>>> print() method. No one output is showed although no errors or
>>> exceptions are
>>> raised.
>>> I want to highlight that I have also other data streams which are
>>> correctly
>>> printed.
>>>
>>> This is the stream:
>>>
>>> /DataStream<Tuple2&lt;Double,Double>> result =
>>> HTM.learn(kafkaStream,
>>> new
>>> Harness.AnomalyNetwork())
>>> .select(new
>>> InferenceSelectFunction<Harness.KafkaRecord,
> >> Tuple2<Double, Double>>() {
>>> @Override
>>> public
>>> Tuple2<Double,Double>
>>> select(Tuple2<Harness.KafkaRecord, NetworkInference> inference)
>>> throws
>>> Exception {
>>> return new Tuple2<Double,
> >> Double>(3.14444,
>>> inference.f1.getAnomalyScore());
>>> }
>>> });/
>>>
>>> I'm quite sure the problem isn't in the particular function described
>>> above
>>> because effectively in inner functions of getAnomalyScore for example
>>> the
>>> return value exists.
>>>
>>> Anyway I'm not able to find the problem which doesn't allow to print the
>>> stream so I thought to write here to discover if there are typical
>>> situation
>>> where streams are not printed. How can I verify and debug the
>>> problem? Some
>>> hints about it ? I really tried everything, also tried to print
>>> something in
>>> the function select() for example, with no results
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223.html
>>> Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive at Nabble.com.
>>>
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: datastream.print() doesn't works
Posted by Timo Walther <tw...@apache.org>.
Don't forget to call env.execute() at the end and make sure you have
configured your logger correctly.
Regards,
Timo
Am 29.08.17 um 14:59 schrieb Chesnay Schepler:
> The easiest explanation is that there is nothing to print.
>
> Since print statements within the select function don't appear in the
> logs I assume
> that the result of HTM.learn is empty.
>
> Please check via the webUI or metrics whether any of these operations
> actually return records.
>
> On 29.08.2017 13:19, AndreaKinn wrote:
>> Hi,
>> I have a simple datastream of a Tuple2. Unfortunately when I call the
>> print() method. No one output is showed although no errors or
>> exceptions are
>> raised.
>> I want to highlight that I have also other data streams which are
>> correctly
>> printed.
>>
>> This is the stream:
>>
>> /DataStream<Tuple2<Double,Double>> result = HTM.learn(kafkaStream,
>> new
>> Harness.AnomalyNetwork())
>> .select(new InferenceSelectFunction<Harness.KafkaRecord,
>> Tuple2<Double, Double>>() {
>> @Override
>> public Tuple2<Double,Double>
>> select(Tuple2<Harness.KafkaRecord, NetworkInference> inference) throws
>> Exception {
>> return new Tuple2<Double,
>> Double>(3.14444,
>> inference.f1.getAnomalyScore());
>> }
>> });/
>>
>> I'm quite sure the problem isn't in the particular function described
>> above
>> because effectively in inner functions of getAnomalyScore for example
>> the
>> return value exists.
>>
>> Anyway I'm not able to find the problem which doesn't allow to print the
>> stream so I thought to write here to discover if there are typical
>> situation
>> where streams are not printed. How can I verify and debug the
>> problem? Some
>> hints about it ? I really tried everything, also tried to print
>> something in
>> the function select() for example, with no results
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223.html
>> Sent from the Apache Flink User Mailing List archive. mailing list
>> archive at Nabble.com.
>>
Re: datastream.print() doesn't works
Posted by Chesnay Schepler <ch...@apache.org>.
The easiest explanation is that there is nothing to print.
Since print statements within the select function don't appear in the
logs I assume
that the result of HTM.learn is empty.
Please check via the webUI or metrics whether any of these operations
actually return records.
On 29.08.2017 13:19, AndreaKinn wrote:
> Hi,
> I have a simple datastream of a Tuple2. Unfortunately when I call the
> print() method. No one output is showed although no errors or exceptions are
> raised.
> I want to highlight that I have also other data streams which are correctly
> printed.
>
> This is the stream:
>
> /DataStream<Tuple2<Double,Double>> result = HTM.learn(kafkaStream, new
> Harness.AnomalyNetwork())
> .select(new InferenceSelectFunction<Harness.KafkaRecord,
> Tuple2<Double, Double>>() {
> @Override
> public Tuple2<Double,Double>
> select(Tuple2<Harness.KafkaRecord, NetworkInference> inference) throws
> Exception {
> return new Tuple2<Double, Double>(3.14444,
> inference.f1.getAnomalyScore());
> }
> });/
>
> I'm quite sure the problem isn't in the particular function described above
> because effectively in inner functions of getAnomalyScore for example the
> return value exists.
>
> Anyway I'm not able to find the problem which doesn't allow to print the
> stream so I thought to write here to discover if there are typical situation
> where streams are not printed. How can I verify and debug the problem? Some
> hints about it ? I really tried everything, also tried to print something in
> the function select() for example, with no results
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>