You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by "Sandybayev, Turar (CAI - Atlanta)" <Tu...@coxautoinc.com> on 2018/05/31 20:23:22 UTC

Looking for a working POM file example for EMR cluster

Hi,

I'm looking for a sample POM file that works when running on EMR cluster. I'm new to Flink and EMR, so I'm simply following AWS EMR documentation on Flink and I am creating a Step and submitting my program JAR file. My program is just a slight modification of the Wikipedia example. 

I was trying to follow an example from AWS reference architecture for their Taxi events example: https://github.com/aws-samples/flink-stream-processing-refarch/blob/master/flink-taxi-stream-processor/pom.xml 

However, I've been seeing various errors having to do with dependencies and ClassNotFoundExceptions for basic common Flink dependencies. I tried removing excludes from the maven-shade-plugin section of the POM file from the reference architecture, and now I'm seeing the following exception:

Exception in thread "main" com.typesafe.config.ConfigException$UnresolvedSubstitution: Could not resolve substitution to a value: ${akka.stream.materializer}

If I run a local Flink cluster and submit my JAR, I'm not seeing any issues with pretty much any way I modify the POM file. I would greatly appreciate if someone can point me to a working POM example.

Thanks!
Turar

Re: Looking for a working POM file example for EMR cluster

Posted by "Sandybayev, Turar (CAI - Atlanta)" <Tu...@coxautoinc.com>.

Thanks Georgi,

I ended up chucking the POM example from AWS ref architecture and re-generating a new pom from Flink 1.4 archetype, and then logging into a master node via SSH and submitting a job directly. Using Steps on EMR Console doesn’t seem to be quite the same thing I guess.

Thanks for your help!!

Turar

From: Georgi Stoyanov <gs...@live.com>
Date: Friday, June 1, 2018 at 2:18 AM
To: "Sandybayev, Turar (CAI - Atlanta)" <Tu...@coxautoinc.com>, "user@flink.apache.org" <us...@flink.apache.org>
Subject: RE: Looking for a working POM file example for EMR cluster

Hi,

Did you check solutions from here - https://stackoverflow.com/questions/48904881/could-not-resolve-substitution-to-a-value-akka-stream-materializer-in-aws-la

Regards,

Georgi Stoyanov

________________________________
From: Sandybayev, Turar (CAI - Atlanta) <Tu...@coxautoinc.com>
Sent: Thursday, May 31, 2018 11:23:22 PM
To: user@flink.apache.org
Subject: Looking for a working POM file example for EMR cluster

Hi,

I'm looking for a sample POM file that works when running on EMR cluster. I'm new to Flink and EMR, so I'm simply following AWS EMR documentation on Flink and I am creating a Step and submitting my program JAR file. My program is just a slight modification of the Wikipedia example.

I was trying to follow an example from AWS reference architecture for their Taxi events example: https://github.com/aws-samples/flink-stream-processing-refarch/blob/master/flink-taxi-stream-processor/pom.xml

However, I've been seeing various errors having to do with dependencies and ClassNotFoundExceptions for basic common Flink dependencies. I tried removing excludes from the maven-shade-plugin section of the POM file from the reference architecture, and now I'm seeing the following exception:

Exception in thread "main" com.typesafe.config.ConfigException$UnresolvedSubstitution: Could not resolve substitution to a value: ${akka.stream.materializer}

If I run a local Flink cluster and submit my JAR, I'm not seeing any issues with pretty much any way I modify the POM file. I would greatly appreciate if someone can point me to a working POM example.

Thanks!
Turar

RE: Looking for a working POM file example for EMR cluster

Posted by Georgi Stoyanov <gs...@live.com>.


Hi,



Did you check solutions from here - https://stackoverflow.com/questions/48904881/could-not-resolve-substitution-to-a-value-akka-stream-materializer-in-aws-la



Regards,

Georgi Stoyanov



________________________________
From: Sandybayev, Turar (CAI - Atlanta) <Tu...@coxautoinc.com>
Sent: Thursday, May 31, 2018 11:23:22 PM
To: user@flink.apache.org
Subject: Looking for a working POM file example for EMR cluster

Hi,

I'm looking for a sample POM file that works when running on EMR cluster. I'm new to Flink and EMR, so I'm simply following AWS EMR documentation on Flink and I am creating a Step and submitting my program JAR file. My program is just a slight modification of the Wikipedia example.

I was trying to follow an example from AWS reference architecture for their Taxi events example: https://github.com/aws-samples/flink-stream-processing-refarch/blob/master/flink-taxi-stream-processor/pom.xml

However, I've been seeing various errors having to do with dependencies and ClassNotFoundExceptions for basic common Flink dependencies. I tried removing excludes from the maven-shade-plugin section of the POM file from the reference architecture, and now I'm seeing the following exception:

Exception in thread "main" com.typesafe.config.ConfigException$UnresolvedSubstitution: Could not resolve substitution to a value: ${akka.stream.materializer}

If I run a local Flink cluster and submit my JAR, I'm not seeing any issues with pretty much any way I modify the POM file. I would greatly appreciate if someone can point me to a working POM example.

Thanks!
Turar