You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Boris Lublinsky <bo...@lightbend.com> on 2019/02/18 00:07:36 UTC
Starting Flink cluster and running a job
Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker <https://github.com/apache/flink/tree/release-1.7/flink-container/docker>
I have created an entry point, which looks like follows:
#!/bin/sh
################################################################################
# from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh
# and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh
################################################################################
# If unspecified, the hostname of the container is taken as the JobManager address
JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
drop_privs_cmd() {
if [ $(id -u) != 0 ]; then
# Don't need to drop privs if EUID != 0
return
elif [ -x /sbin/su-exec ]; then
# Alpine
echo su-exec flink
else
# Others
echo gosu flink
fi
}
JOB_MANAGER="jobmanager"
TASK_MANAGER="taskmanager"
CMD="$1"
shift
if [ "${CMD}" = "help" ]; then
echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
exit 0
elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
if [ "${CMD}" = "${TASK_MANAGER}" ]; then
TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
echo "Starting Task Manager"
echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
else
sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
if [ -z "$1" ]; then
exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
else
exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
fi
fi
fi
exec "$@"
It does work for all the cases, except running standalone job.
The problem, the way I understand it, is a racing condition.
In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh
tries to start a job immediately once the cluster is created (before connection is established).
Is there a better option to implement it starting a job on container startup?
Re: Starting Flink cluster and running a job
Posted by Konstantin Knauf <ko...@ververica.com>.
Hi Boris,
in the first case it is linked into the *lib* directory, in the second case
it is located in the lib directory. Both work. My job uses both the Kafka
producer as well the Kafka consumer and both are processing records.
Best,
Konstantin
On Wed, Mar 6, 2019 at 3:16 PM Boris Lublinsky <
boris.lublinsky@lightbend.com> wrote:
> Thanks Konstanin,
> A couple of question?
> 1. Is your fat jar in lib directory? If it is not it works fine
> 2. The error occurs only when you start pushing traffic on Kafka
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Mar 6, 2019, at 4:27 AM, Konstantin Knauf <ko...@ververica.com>
> wrote:
>
> Hi Boris,
>
> I was not able tot reproduce your problem. I am using the
> docker-entrypoint.sh and Dockerfile from the flink-container project [1].
>
> The content of my fat-jar:
>
> flinkShadowJar
> +--- project :fe-flink-common
> +--- org.apache.flink:flink-metrics-influxdb_2.11:1.8-SNAPSHOT
> +--- org.apache.flink:flink-metrics-slf4j:1.7.2
> +--- org.apache.flink:flink-connector-kafka_2.11:1.7.2
> | +--- org.apache.flink:flink-connector-kafka-base_2.11:1.7.2
> | \--- org.apache.kafka:kafka-clients:2.0.1
> | +--- org.lz4:lz4-java:1.4.1
> | \--- org.xerial.snappy:snappy-java:1.1.7.1
> \--- org.apache.flink:flink-s3-fs-presto:1.7.2
>
> In the standard Docker image the job.jar is "only" linked into the Flink
> lib directory. I also tested it with a modified Dokerfile, which copies the
> job.jar into the lib directory of Flink.
>
> bash-4.4$ ls -lia lib
> total 135140
> 611154 drwxrwxr-x 1 flink flink 4096 Mar 6 10:20 .
> 606092 drwxrwxr-x 1 flink flink 4096 Feb 11 15:38 ..
> 26477916 -rw-r--r-- 1 flink flink 93445474 Feb 11 15:38
> flink-dist_2.11-1.7.2.jar
> 26477917 -rw-r--r-- 1 flink flink 141937 Feb 11 15:37
> flink-python_2.11-1.7.2.jar
> 26477525 -rw-r--r-- 1 flink flink 44278062 Mar 6 10:20 job.jar
> 26477918 -rw-rw-r-- 1 flink flink 489884 Feb 11 14:32
> log4j-1.2.17.jar
> 26477919 -rw-rw-r-- 1 flink flink 9931 Feb 11 14:32
> slf4j-log4j12-1.7.15.jar
>
> This also works. In that case you only need to make sure, that the job.jar
> is owned by the *flink* user by epxlicitly changing the ownership in the
> Dockerfile. Given your description of the Dockerfile, I suspect this is the
> problem.
>
> Cheers,
>
> Konstantin
>
> [1] https://github.com/apache/flink/tree/release-1.7.2/flink-container
>
> On Wed, Feb 27, 2019 at 11:19 PM Boris Lublinsky <
> boris.lublinsky@lightbend.com> wrote:
>
>> I am not including Kafka per se - I am using Flink Kafka connector, that
>> pulls in Kafka.
>> I tried exclude and it failed.
>>
>> I can’t quite shade Kafka and it does not seem like the right thing to do.
>> I tried to see where it is loaded from, and its my jar, but I see its
>> being loaded twice when it is in Lib.
>>
>> You guys should be able to easily reproduce it on any Kafka sample that
>> you have
>>
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com
>> https://www.lightbend.com/
>>
>> On Feb 27, 2019, at 2:10 PM, Ken Krugler <kk...@transpac.com>
>> wrote:
>>
>> Hi Boris,
>>
>> 1. I assume you’ve tried marking the Kafka dependencies as “provided”,
>> and that also failed, yes?
>>
>> 2. Did you try shading the Kafka dependencies?
>>
>> — Ken
>>
>> PS - Moving this conversation back to the user list.
>>
>>
>> On Feb 27, 2019, at 2:03 PM, Boris Lublinsky <
>> boris.lublinsky@lightbend.com> wrote:
>>
>> Guys, please,
>> I do still need help with this one
>> It does work if I put my jar into a random directory, but fails if I put
>> it into Lib.
>> And I do need to put it into lib, because standalone-job.sh does not take
>> a jar, it assumes a jar to be in lib
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com
>> https://www.lightbend.com/
>>
>> On Feb 26, 2019, at 7:33 AM, Boris Lublinsky <
>> boris.lublinsky@lightbend.com> wrote:
>>
>> Any progress on this one?
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com
>> https://www.lightbend.com/
>>
>> On Feb 24, 2019, at 5:16 PM, Boris Lublinsky <
>> boris.lublinsky@lightbend.com> wrote:
>>
>> And there are more explanations and concerns there
>> It turns out that the execution depends on where my jar is located.
>> If I put it in lib directory (I need this to be able to run
>> bin/standalone-job.sh )
>> It fails. With the same error.
>>
>>
>> Currently I have the following in this directory
>> fdp-flink-taxiride-assembly-2.0.0.jar
>> flink-metrics-prometheus-1.7.2.jar
>> flink-queryable-state-runtime_2.11-1.7.2.jar log4j-1.2.17.jar
>> flink-dist_2.11-1.7.2.jar flink-python_2.11-1.7.2.jar
>> flink-table_2.11-1.7.2.jar slf4j-log4j12-1.7.15.jar
>>
>> And no one of them seem to directly contain Kafka (except for my jar
>> fdp-flink-taxiride-assembly-2.0.0.jar)
>>
>> But id I move my jar in any other directory it works fine.
>>
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com
>> https://www.lightbend.com/
>>
>> On Feb 23, 2019, at 11:40 AM, Boris Lublinsky <
>> boris.lublinsky@lightbend.com> wrote:
>>
>> And its working now.
>> Sorry about all confusion.
>> The root cause of the problem was that in “standard” Link image directory
>> /opt/flink/log does not have write permissions and as a result the local
>> cluster was never created
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com
>> https://www.lightbend.com/
>>
>> On Feb 22, 2019, at 9:13 AM, Dawid Wysakowicz <dw...@apache.org>
>> wrote:
>>
>> Hi,
>>
>> One additional question, how do you actually build the docker image? How
>> do you put the user jar into the container? Maybe you added extra kafka
>> connector to the cluster classpath? Have you checked what is on the
>> classpath of a running taskmanager?
>>
>> Best,
>>
>> Dawid
>> On 22/02/2019 15:44, Boris Lublinsky wrote:
>>
>> Gordon, I double checked it several times. Here is the list
>>
>> Columns are
>> [info] - Jar-Size including dependencies
>> [info] - Jar-Size
>> [info] - Number of transitive dependencies
>> [info] - Number of direct dependencies
>> [info] - ModuleID
>> [info] Done updating.
>> [info] TotSize JarSize #TDe #Dep Module
>> [info] 27.988 MB ------- MB 33 2
>> lightbend:fdp-flink-taxiride_2.11:2.0.0
>> [info] 19.632 MB ------- MB 24 6 lightbend:support_2.11:2.0.0
>> [info] 17.675 MB 0.000 MB 16 9 org.typelevel:cats_2.11:0.9.0
>> [info] 17.389 MB 1.182 MB 13 9 org.typelevel:cats-laws_2.11:0.9.0
>> [info] 13.115 MB 0.286 MB 7 4 org.typelevel:cats-free_2.11:0.9.0
>> [info] 12.830 MB 0.000 MB 7 4 org.typelevel:cats-jvm_2.11:0.9.0
>> [info] 12.830 MB 3.403 MB 6 4 org.typelevel:cats-core_2.11:0.9.0
>> [info] 8.397 MB 0.068 MB 8 5
>> org.apache.flink:flink-connector-kafka_2.11:1.7.1
>> [info] 8.175 MB 1.889 MB 4 4
>> org.apache.kafka:kafka-clients:2.1.0
>> [info] 7.973 MB 0.212 MB 7 4
>> org.typelevel:cats-kernel-laws_2.11:0.9.0
>> [info] 4.834 MB 0.007 MB 4 2
>> org.typelevel:cats-macros_2.11:0.9.0
>> [info] 4.746 MB 0.082 MB 2 2
>> com.typesafe.scala-logging:scala-logging_2.11:3.9.0
>> [info] 4.658 MB 0.035 MB 1 1 org.typelevel:machinist_2.11:0.6.1
>> [info] 4.623 MB 4.623 MB 0 0
>> org.scala-lang:scala-reflect:2.11.12
>> [info] 4.592 MB 4.592 MB 0 0
>> org.typelevel:cats-kernel_2.11:0.9.0
>> [info] 3.714 MB 3.714 MB 0 0 com.github.luben:zstd-jni:1.3.5-4
>> [info] 3.152 MB 0.043 MB 2 1 org.typelevel:discipline_2.11:0.7.2
>> [info] 3.109 MB 3.094 MB 1 1
>> org.scalacheck:scalacheck_2.11:1.13.4
>> [info] 2.019 MB 2.019 MB 0 0
>> org.xerial.snappy:snappy-java:1.1.7.2
>> [info] 0.803 MB 0.290 MB 2 2
>> ch.qos.logback:logback-classic:1.2.3
>> [info] 0.641 MB 0.641 MB 0 0 joda-time:joda-time:2.10.1
>> [info] 0.512 MB 0.512 MB 0 0 org.lz4:lz4-java:1.5.0
>> [info] 0.472 MB 0.472 MB 0 0 ch.qos.logback:logback-core:1.2.3
>> [info] 0.286 MB 0.286 MB 0 0 com.typesafe:config:1.3.3
>> [info] 0.195 MB 0.114 MB 3 3
>> org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>> [info] 0.170 MB 0.167 MB 1 1
>> com.github.mpilquist:simulacrum_2.11:0.10.0
>> [info] 0.145 MB 0.145 MB 0 0 org.joda:joda-convert:2.1.2
>> [info] 0.041 MB 0.041 MB 0 0 org.slf4j:slf4j-api:1.7.25
>> [info] 0.033 MB 0.033 MB 0 0
>> com.google.code.findbugs:jsr305:1.3.9
>> [info] 0.016 MB 0.002 MB 2 1
>> org.typelevel:catalysts-platform_2.11:0.0.5
>> [info] 0.015 MB 0.012 MB 1 1
>> org.typelevel:catalysts-macros_2.11:0.0.5
>> [info] 0.015 MB 0.015 MB 0 0 org.scala-sbt:test-interface:1.0
>> [info] 0.007 MB 0.007 MB 0 0
>> org.apache.flink:force-shading:1.7.1
>> [info] 0.003 MB 0.003 MB 0 0
>> org.typelevel:macro-compat_2.11:1.1.1
>>
>>
>> And here is another view
>>
>> [info] lightbend:fdp-flink-taxiride_2.11:2.0.0 [S]
>> [info] +-lightbend:support_2.11:2.0.0 [S]
>> [info] | +-ch.qos.logback:logback-classic:1.2.3
>> [info] | | +-ch.qos.logback:logback-core:1.2.3
>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>> [info] | |
>> [info] | +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>> [info] | |
>> [info] | +-com.typesafe:config:1.3.3
>> [info] | +-joda-time:joda-time:2.10.1
>> [info] | +-org.joda:joda-convert:2.1.2
>> [info] | +-org.typelevel:cats_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-free_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | | |
>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | | |
>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | | |
>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | | |
>> [info] | | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.apache.flink:flink-connector-kafka_2.11:1.7.1
>> [info] +-com.google.code.findbugs:jsr305:1.3.9
>> [info] +-org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>> [info] | +-com.google.code.findbugs:jsr305:1.3.9
>> [info] | +-org.apache.flink:force-shading:1.7.1
>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] | +-org.slf4j:slf4j-api:1.7.25
>> [info] |
>> [info] +-org.apache.flink:force-shading:1.7.1
>> [info] +-org.apache.kafka:kafka-clients:2.1.0
>> [info] | +-com.github.luben:zstd-jni:1.3.5-4
>> [info] | +-org.lz4:lz4-java:1.5.0
>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] | +-org.slf4j:slf4j-api:1.7.25
>> [info] | +-org.xerial.snappy:snappy-java:1.1.7.2
>> [info] |
>> [info] +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] +-org.slf4j:slf4j-api:1.7.25
>> [info]
>> [info] Done updating.
>> [info] lightbend:support_2.11:2.0.0 [S]
>> [info] +-ch.qos.logback:logback-classic:1.2.3
>> [info] | +-ch.qos.logback:logback-core:1.2.3
>> [info] | +-org.slf4j:slf4j-api:1.7.25
>> [info] |
>> [info] +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | +-org.slf4j:slf4j-api:1.7.25
>> [info] |
>> [info] +-com.typesafe:config:1.3.3
>> [info] +-joda-time:joda-time:2.10.1
>> [info] +-org.joda:joda-convert:2.1.2
>> [info] +-org.typelevel:cats_2.11:0.9.0 [S]
>> [info] +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] |
>> [info] +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:cats-free_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | +-org.scala-sbt:test-interface:1.0
>> [info] |
>> [info] +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info]
>>
>>
>>
>>
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com
>> https://www.lightbend.com/
>>
>> On Feb 22, 2019, at 12:33 AM, Tzu-Li (Gordon) Tai <go...@ververica.com>
>> wrote:
>>
>> Hi,
>>
>> I haven't seen this problem for Flink 1.6.x / 1.7.x, so it shouldn't be a
>> reoccurrence of FLINK-8741.
>> I've double checked the used classloaders in the Kafka connector, they
>> seem to be correct.
>>
>> The fact that it works correctly in IntelliJ, but not when packaged,
>> suggests that there could be some conflicting dependencies in the packaged
>> jar.
>>
>> Could you check the actual resolved dependency tree of the project, and
>> see if there are multiple versions of some dependency related to Kafka
>> being pulled in? For Maven for example, that would be "mvn dependency:tree".
>>
>> Cheers,
>> Gordon
>>
>> On Thu, Feb 21, 2019 at 10:49 PM Boris Lublinsky <
>> boris.lublinsky@lightbend.com> wrote:
>>
>>> The relevant dependencies are
>>>
>>> val flinkScala = "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"val flinkStreamingScala = "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"val flinkKafka = "org.apache.flink" %% "flink-connector-kafka" % flinkVersion exclude("org.slf4j", "slf4j-log4j12")
>>>
>>> I am using SBT
>>> I tried both connector-kafka and connector-kaka-11 - same result
>>>
>>>
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com
>>> https://www.lightbend.com/
>>>
>>> On Feb 21, 2019, at 1:38 AM, Konstantin Knauf <ko...@ververica.com>
>>> wrote:
>>>
>>> Hi Boris,
>>>
>>> can you the relevant parts (dependencies) of your pom.xml? Did you also
>>> try without fixing the Kafka version, i.e. running with the Kafka client
>>> version provided by the Kafka connector of Flink? Gordon (cc) dealt with
>>> FLINK-8741.
>>>
>>> @Gordon: have you seen this issue with 1.6/1.7 before?
>>>
>>> Cheers,
>>>
>>> Konstantin
>>>
>>> On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky <
>>> boris.lublinsky@lightbend.com> wrote:
>>>
>>>> I found some more details on this
>>>> The same error for the same application was reported about a year ago
>>>> http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E
>>>> And was due to https://issues.apache.org/jira/browse/FLINK-8741
>>>>
>>>> It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with
>>>> both latest kaffka-connector
>>>> And Kafka-connector-011
>>>>
>>>> Boris Lublinsky
>>>> FDP Architect
>>>> boris.lublinsky@lightbend.com
>>>> https://www.lightbend.com/
>>>>
>>>> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kk...@transpac.com>
>>>> wrote:
>>>>
>>>> Hi Boris,
>>>>
>>>> I haven’t seen this exact error, but I have seen similar errors caused
>>>> by multiple versions of jars on the classpath.
>>>>
>>>> When I’ve run into this particular "XXX is not an instance of YYY"
>>>> problem, it often seems to be caused by a jar that I should have marked as
>>>> provided in my pom.
>>>>
>>>> Though I’m typically running on a YARN cluster, not w/K8s, so maybe
>>>> this doesn’t apply.
>>>>
>>>> — Ken
>>>>
>>>> PS - I assume you’ve been reading
>>>> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html
>>>>
>>>>
>>>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <
>>>> boris.lublinsky@lightbend.com> wrote:
>>>>
>>>> Konstantin,
>>>> After experimenting with this for a while, I got to the root cause of
>>>> the problem
>>>> I am running a version of a Taxi ride travel prediction as my sample.
>>>> It works fine in Intellij,
>>>> But when I am trying to put it in the docker (standard Debian 1.7 image)
>>>> It fails with a following error
>>>>
>>>>
>>>> The program finished with the following exception:
>>>>
>>>> org.apache.flink.client.program.ProgramInvocationException: Job failed.
>>>> (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>>>> at
>>>> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>>>> at
>>>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>>>> at
>>>> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>>>> at
>>>> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>>>> at
>>>> com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>>>> at
>>>> com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at
>>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>>>> at
>>>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>>>> at
>>>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>>>> at
>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>>>> at
>>>> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job
>>>> execution failed.
>>>> at
>>>> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>>>> at
>>>> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>>>> ... 19 more
>>>> Caused by: org.apache.kafka.common.KafkaException: Failed to construct
>>>> kafka producer
>>>> at
>>>> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>>>> at
>>>> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>>>> at
>>>> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>>>> at
>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>>>> at
>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>>>> at
>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>>>> at
>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>>>> at
>>>> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>>>> at
>>>> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>>>> at
>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>>>> at
>>>> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>>>> at
>>>> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>>>> at
>>>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>>>> at
>>>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>>>> at
>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>>>> at
>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Caused by: org.apache.kafka.common.KafkaException:
>>>> org.apache.kafka.common.serialization.ByteArraySerializer is not an
>>>> instance of org.apache.kafka.common.serialization.Serializer
>>>> at
>>>> org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>>>> at
>>>> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>>>> ... 17 more
>>>>
>>>> The closest that I found
>>>> https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no
>>>> Which talks about class loader. (I tried there solution, but it did not
>>>> help)
>>>> I looked at the loading and I see that the pair of these 2 classes is
>>>> loaded from my uber jar, but twice.
>>>>
>>>> Have you guys seen this error before?
>>>> Any suggestion?
>>>>
>>>> Boris Lublinsky
>>>> FDP Architect
>>>> boris.lublinsky@lightbend.com
>>>> https://www.lightbend.com/
>>>>
>>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <ko...@ververica.com>
>>>> wrote:
>>>>
>>>> Hi Boris,
>>>>
>>>> without looking at the entrypoint in much detail, generally there
>>>> should not be a race condition there:
>>>>
>>>> * if the taskmanagers can not connect to the resourcemanager they will
>>>> retry (per default the timeout is 5 mins)
>>>> * if the JobManager does not get enough resources from the
>>>> ResourceManager it will also wait for the resources/slots to provided. The
>>>> timeout there is also 5 minutes, I think.
>>>>
>>>> So, this should actually be pretty robust as long as the Taskmanager
>>>> containers can reach the Jobmanager eventually.
>>>>
>>>> Could you provide the Taskmanager/JobManager logs for such a failure
>>>> case?
>>>>
>>>> Cheers,
>>>>
>>>> Konstantin
>>>>
>>>>
>>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <
>>>> boris.lublinsky@lightbend.com> wrote:
>>>>
>>>>> Following
>>>>> https://github.com/apache/flink/tree/release-1.7/flink-container/docker
>>>>> I have created an entry point, which looks like follows:
>>>>>
>>>>> #!/bin/sh################################################################################# from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh# and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh################################################################################# If unspecified, the hostname of the container is taken as the JobManager addressJOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>>>> drop_privs_cmd() {
>>>>> if [ $(id -u) != 0 ]; then # Don't need to drop privs if EUID != 0 return elif [ -x /sbin/su-exec ]; then # Alpine echo su-exec flink
>>>>> else # Others echo gosu flink
>>>>> fi}
>>>>>
>>>>> JOB_MANAGER="jobmanager"TASK_MANAGER="taskmanager"CMD="$1"shiftif [ "${CMD}" = "help" ]; then echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)" exit 0elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then if [ "${CMD}" = "${TASK_MANAGER}" ]; then TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>>>>
>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml" sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml" echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml" echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml" echo "Starting Task Manager" echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml" exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>>>> else sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml" echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml" echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml" echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml" if [ -z "$1" ]; then exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@" else exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@" fi fifiexec "$@"
>>>>>
>>>>> It does work for all the cases, except running standalone job.
>>>>> The problem, the way I understand it, is a racing condition.
>>>>> In kubernetes it takes several attempts for establish connection
>>>>> between Job and Task manager, while standalone-job.sh
>>>>> tries to start a job immediately once the cluster is created (before
>>>>> connection is established).
>>>>> Is there a better option to implement it starting a job on container
>>>>> startup?
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Konstantin Knauf | Solutions Architect
>>>> +49 160 91394525
>>>>
>>>>
>>>> --------------------------
>>>> Ken Krugler
>>>> +1 530-210-6378
>>>> http://www.scaleunlimited.com
>>>> Custom big data solutions & training
>>>> Flink, Solr, Hadoop, Cascading & Cassandra
>>>>
>>>>
>>>>
>>>
>>> --
>>> Konstantin Knauf | Solutions Architect
>>> +49 160 91394525
>>> <https://www.ververica.com/>
>>>
>>> Follow us @VervericaData
>>> --
>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>>> Conference
>>> Stream Processing | Event Driven | Real Time
>>> --
>>> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>> --
>>> Data Artisans GmbH Registered at Amtsgericht Charlottenburg: HRB 158244
>>> B Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>>>
>>>
>>>
>>
>>
>>
>>
>>
>> --------------------------
>> Ken Krugler
>> +1 530-210-6378
>> http://www.scaleunlimited.com
>> Custom big data solutions & training
>> Flink, Solr, Hadoop, Cascading & Cassandra
>>
>>
>>
>
> --
> Konstantin Knauf | Solutions Architect
> +49 160 91394525
> <https://www.ververica.com/>
>
> Follow us @VervericaData
> --
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> Conference
> Stream Processing | Event Driven | Real Time
> --
> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> --
> Data Artisans GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>
>
>
--
Konstantin Knauf | Solutions Architect
+49 160 91394525
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
Re: Starting Flink cluster and running a job
Posted by Boris Lublinsky <bo...@lightbend.com>.
Thanks Konstanin,
A couple of question?
1. Is your fat jar in lib directory? If it is not it works fine
2. The error occurs only when you start pushing traffic on Kafka
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
> On Mar 6, 2019, at 4:27 AM, Konstantin Knauf <ko...@ververica.com> wrote:
>
> Hi Boris,
>
> I was not able tot reproduce your problem. I am using the docker-entrypoint.sh and Dockerfile from the flink-container project [1].
>
> The content of my fat-jar:
>
> flinkShadowJar
> +--- project :fe-flink-common
> +--- org.apache.flink:flink-metrics-influxdb_2.11:1.8-SNAPSHOT
> +--- org.apache.flink:flink-metrics-slf4j:1.7.2
> +--- org.apache.flink:flink-connector-kafka_2.11:1.7.2
> | +--- org.apache.flink:flink-connector-kafka-base_2.11:1.7.2
> | \--- org.apache.kafka:kafka-clients:2.0.1
> | +--- org.lz4:lz4-java:1.4.1
> | \--- org.xerial.snappy:snappy-java:1.1.7.1
> \--- org.apache.flink:flink-s3-fs-presto:1.7.2
>
> In the standard Docker image the job.jar is "only" linked into the Flink lib directory. I also tested it with a modified Dokerfile, which copies the job.jar into the lib directory of Flink.
>
> bash-4.4$ ls -lia lib
> total 135140
> 611154 drwxrwxr-x 1 flink flink 4096 Mar 6 10:20 .
> 606092 drwxrwxr-x 1 flink flink 4096 Feb 11 15:38 ..
> 26477916 -rw-r--r-- 1 flink flink 93445474 Feb 11 15:38 flink-dist_2.11-1.7.2.jar
> 26477917 -rw-r--r-- 1 flink flink 141937 Feb 11 15:37 flink-python_2.11-1.7.2.jar
> 26477525 -rw-r--r-- 1 flink flink 44278062 Mar 6 10:20 job.jar
> 26477918 -rw-rw-r-- 1 flink flink 489884 Feb 11 14:32 log4j-1.2.17.jar
> 26477919 -rw-rw-r-- 1 flink flink 9931 Feb 11 14:32 slf4j-log4j12-1.7.15.jar
>
> This also works. In that case you only need to make sure, that the job.jar is owned by the flink user by epxlicitly changing the ownership in the Dockerfile. Given your description of the Dockerfile, I suspect this is the problem.
>
> Cheers,
>
> Konstantin
>
> [1] https://github.com/apache/flink/tree/release-1.7.2/flink-container <https://github.com/apache/flink/tree/release-1.7.2/flink-container>
>
> On Wed, Feb 27, 2019 at 11:19 PM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
> I am not including Kafka per se - I am using Flink Kafka connector, that pulls in Kafka.
> I tried exclude and it failed.
>
> I can’t quite shade Kafka and it does not seem like the right thing to do.
> I tried to see where it is loaded from, and its my jar, but I see its being loaded twice when it is in Lib.
>
> You guys should be able to easily reproduce it on any Kafka sample that you have
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com <ma...@lightbend.com>
> https://www.lightbend.com/ <https://www.lightbend.com/>
>> On Feb 27, 2019, at 2:10 PM, Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>> wrote:
>>
>> Hi Boris,
>>
>> 1. I assume you’ve tried marking the Kafka dependencies as “provided”, and that also failed, yes?
>>
>> 2. Did you try shading the Kafka dependencies?
>>
>> — Ken
>>
>> PS - Moving this conversation back to the user list.
>>
>>
>>> On Feb 27, 2019, at 2:03 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>
>>> Guys, please,
>>> I do still need help with this one
>>> It does work if I put my jar into a random directory, but fails if I put it into Lib.
>>> And I do need to put it into lib, because standalone-job.sh does not take a jar, it assumes a jar to be in lib
>>>
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>> On Feb 26, 2019, at 7:33 AM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>
>>>> Any progress on this one?
>>>> Boris Lublinsky
>>>> FDP Architect
>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>> On Feb 24, 2019, at 5:16 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>
>>>>> And there are more explanations and concerns there
>>>>> It turns out that the execution depends on where my jar is located.
>>>>> If I put it in lib directory (I need this to be able to run bin/standalone-job.sh )
>>>>> It fails. With the same error.
>>>>>
>>>>>
>>>>> Currently I have the following in this directory
>>>>> fdp-flink-taxiride-assembly-2.0.0.jar flink-metrics-prometheus-1.7.2.jar flink-queryable-state-runtime_2.11-1.7.2.jar log4j-1.2.17.jar
>>>>> flink-dist_2.11-1.7.2.jar flink-python_2.11-1.7.2.jar flink-table_2.11-1.7.2.jar slf4j-log4j12-1.7.15.jar
>>>>>
>>>>> And no one of them seem to directly contain Kafka (except for my jar fdp-flink-taxiride-assembly-2.0.0.jar)
>>>>>
>>>>> But id I move my jar in any other directory it works fine.
>>>>>
>>>>>
>>>>> Boris Lublinsky
>>>>> FDP Architect
>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>> On Feb 23, 2019, at 11:40 AM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>
>>>>>> And its working now.
>>>>>> Sorry about all confusion.
>>>>>> The root cause of the problem was that in “standard” Link image directory /opt/flink/log does not have write permissions and as a result the local cluster was never created
>>>>>>
>>>>>> Boris Lublinsky
>>>>>> FDP Architect
>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>> On Feb 22, 2019, at 9:13 AM, Dawid Wysakowicz <dwysakowicz@apache.org <ma...@apache.org>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> One additional question, how do you actually build the docker image? How do you put the user jar into the container? Maybe you added extra kafka connector to the cluster classpath? Have you checked what is on the classpath of a running taskmanager?
>>>>>>> Best,
>>>>>>>
>>>>>>> Dawid
>>>>>>> On 22/02/2019 15:44, Boris Lublinsky wrote:
>>>>>>>> Gordon, I double checked it several times. Here is the list
>>>>>>>>
>>>>>>>> Columns are
>>>>>>>> [info] - Jar-Size including dependencies
>>>>>>>> [info] - Jar-Size
>>>>>>>> [info] - Number of transitive dependencies
>>>>>>>> [info] - Number of direct dependencies
>>>>>>>> [info] - ModuleID
>>>>>>>> [info] Done updating.
>>>>>>>> [info] TotSize JarSize #TDe #Dep Module
>>>>>>>> [info] 27.988 MB ------- MB 33 2 lightbend:fdp-flink-taxiride_2.11:2.0.0
>>>>>>>> [info] 19.632 MB ------- MB 24 6 lightbend:support_2.11:2.0.0
>>>>>>>> [info] 17.675 MB 0.000 MB 16 9 org.typelevel:cats_2.11:0.9.0
>>>>>>>> [info] 17.389 MB 1.182 MB 13 9 org.typelevel:cats-laws_2.11:0.9.0
>>>>>>>> [info] 13.115 MB 0.286 MB 7 4 org.typelevel:cats-free_2.11:0.9.0
>>>>>>>> [info] 12.830 MB 0.000 MB 7 4 org.typelevel:cats-jvm_2.11:0.9.0
>>>>>>>> [info] 12.830 MB 3.403 MB 6 4 org.typelevel:cats-core_2.11:0.9.0
>>>>>>>> [info] 8.397 MB 0.068 MB 8 5 org.apache.flink:flink-connector-kafka_2.11:1.7.1
>>>>>>>> [info] 8.175 MB 1.889 MB 4 4 org.apache.kafka:kafka-clients:2.1.0
>>>>>>>> [info] 7.973 MB 0.212 MB 7 4 org.typelevel:cats-kernel-laws_2.11:0.9.0
>>>>>>>> [info] 4.834 MB 0.007 MB 4 2 org.typelevel:cats-macros_2.11:0.9.0
>>>>>>>> [info] 4.746 MB 0.082 MB 2 2 com.typesafe.scala-logging:scala-logging_2.11:3.9.0
>>>>>>>> [info] 4.658 MB 0.035 MB 1 1 org.typelevel:machinist_2.11:0.6.1
>>>>>>>> [info] 4.623 MB 4.623 MB 0 0 org.scala-lang:scala-reflect:2.11.12
>>>>>>>> [info] 4.592 MB 4.592 MB 0 0 org.typelevel:cats-kernel_2.11:0.9.0
>>>>>>>> [info] 3.714 MB 3.714 MB 0 0 com.github.luben:zstd-jni:1.3.5-4
>>>>>>>> [info] 3.152 MB 0.043 MB 2 1 org.typelevel:discipline_2.11:0.7.2
>>>>>>>> [info] 3.109 MB 3.094 MB 1 1 org.scalacheck:scalacheck_2.11:1.13.4
>>>>>>>> [info] 2.019 MB 2.019 MB 0 0 org.xerial.snappy:snappy-java:1.1.7.2
>>>>>>>> [info] 0.803 MB 0.290 MB 2 2 ch.qos.logback:logback-classic:1.2.3
>>>>>>>> [info] 0.641 MB 0.641 MB 0 0 joda-time:joda-time:2.10.1
>>>>>>>> [info] 0.512 MB 0.512 MB 0 0 org.lz4:lz4-java:1.5.0
>>>>>>>> [info] 0.472 MB 0.472 MB 0 0 ch.qos.logback:logback-core:1.2.3
>>>>>>>> [info] 0.286 MB 0.286 MB 0 0 com.typesafe:config:1.3.3
>>>>>>>> [info] 0.195 MB 0.114 MB 3 3 org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>>>>>>>> [info] 0.170 MB 0.167 MB 1 1 com.github.mpilquist:simulacrum_2.11:0.10.0
>>>>>>>> [info] 0.145 MB 0.145 MB 0 0 org.joda:joda-convert:2.1.2
>>>>>>>> [info] 0.041 MB 0.041 MB 0 0 org.slf4j:slf4j-api:1.7.25
>>>>>>>> [info] 0.033 MB 0.033 MB 0 0 com.google.code.findbugs:jsr305:1.3.9
>>>>>>>> [info] 0.016 MB 0.002 MB 2 1 org.typelevel:catalysts-platform_2.11:0.0.5
>>>>>>>> [info] 0.015 MB 0.012 MB 1 1 org.typelevel:catalysts-macros_2.11:0.0.5
>>>>>>>> [info] 0.015 MB 0.015 MB 0 0 org.scala-sbt:test-interface:1.0
>>>>>>>> [info] 0.007 MB 0.007 MB 0 0 org.apache.flink:force-shading:1.7.1
>>>>>>>> [info] 0.003 MB 0.003 MB 0 0 org.typelevel:macro-compat_2.11:1.1.1
>>>>>>>>
>>>>>>>>
>>>>>>>> And here is another view
>>>>>>>>
>>>>>>>> [info] lightbend:fdp-flink-taxiride_2.11:2.0.0 [S]
>>>>>>>> [info] +-lightbend:support_2.11:2.0.0 [S]
>>>>>>>> [info] | +-ch.qos.logback:logback-classic:1.2.3
>>>>>>>> [info] | | +-ch.qos.logback:logback-core:1.2.3
>>>>>>>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>>>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>>>>>>>> [info] | |
>>>>>>>> [info] | +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>>>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>>>>>>>> [info] | |
>>>>>>>> [info] | +-com.typesafe:config:1.3.3
>>>>>>>> [info] | +-joda-time:joda-time:2.10.1
>>>>>>>> [info] | +-org.joda:joda-convert:2.1.2
>>>>>>>> [info] | +-org.typelevel:cats_2.11:0.9.0 [S]
>>>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-free_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | | | |
>>>>>>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | | | |
>>>>>>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | | | |
>>>>>>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>>> [info] | | | | +-org.scala-sbt:test-interface:1.0
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>>>> [info] | | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] |
>>>>>>>> [info] +-org.apache.flink:flink-connector-kafka_2.11:1.7.1
>>>>>>>> [info] +-com.google.code.findbugs:jsr305:1.3.9
>>>>>>>> [info] +-org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>>>>>>>> [info] | +-com.google.code.findbugs:jsr305:1.3.9
>>>>>>>> [info] | +-org.apache.flink:force-shading:1.7.1
>>>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>>>>>> [info] |
>>>>>>>> [info] +-org.apache.flink:force-shading:1.7.1
>>>>>>>> [info] +-org.apache.kafka:kafka-clients:2.1.0
>>>>>>>> [info] | +-com.github.luben:zstd-jni:1.3.5-4
>>>>>>>> [info] | +-org.lz4:lz4-java:1.5.0
>>>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>>>>>> [info] | +-org.xerial.snappy:snappy-java:1.1.7.2
>>>>>>>> [info] |
>>>>>>>> [info] +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>>>> [info] +-org.slf4j:slf4j-api:1.7.25
>>>>>>>> [info]
>>>>>>>> [info] Done updating.
>>>>>>>> [info] lightbend:support_2.11:2.0.0 [S]
>>>>>>>> [info] +-ch.qos.logback:logback-classic:1.2.3
>>>>>>>> [info] | +-ch.qos.logback:logback-core:1.2.3
>>>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>>>>>> [info] |
>>>>>>>> [info] +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>>>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>>>>>> [info] |
>>>>>>>> [info] +-com.typesafe:config:1.3.3
>>>>>>>> [info] +-joda-time:joda-time:2.10.1
>>>>>>>> [info] +-org.joda:joda-convert:2.1.2
>>>>>>>> [info] +-org.typelevel:cats_2.11:0.9.0 [S]
>>>>>>>> [info] +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] |
>>>>>>>> [info] +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] |
>>>>>>>> [info] +-org.typelevel:cats-free_2.11:0.9.0 [S]
>>>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] |
>>>>>>>> [info] +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>>>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] |
>>>>>>>> [info] +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>>>>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>>>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>>> [info] | +-org.scala-sbt:test-interface:1.0
>>>>>>>> [info] |
>>>>>>>> [info] +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>>>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>>>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | | |
>>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | | |
>>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] |
>>>>>>>> [info] +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>>> [info] | |
>>>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info] |
>>>>>>>> [info] +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>>> [info] +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>>> [info]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Boris Lublinsky
>>>>>>>> FDP Architect
>>>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>>>> On Feb 22, 2019, at 12:33 AM, Tzu-Li (Gordon) Tai <gordon@ververica.com <ma...@ververica.com>> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I haven't seen this problem for Flink 1.6.x / 1.7.x, so it shouldn't be a reoccurrence of FLINK-8741.
>>>>>>>>> I've double checked the used classloaders in the Kafka connector, they seem to be correct.
>>>>>>>>>
>>>>>>>>> The fact that it works correctly in IntelliJ, but not when packaged, suggests that there could be some conflicting dependencies in the packaged jar.
>>>>>>>>>
>>>>>>>>> Could you check the actual resolved dependency tree of the project, and see if there are multiple versions of some dependency related to Kafka being pulled in? For Maven for example, that would be "mvn dependency:tree".
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Gordon
>>>>>>>>>
>>>>>>>>> On Thu, Feb 21, 2019 at 10:49 PM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>>> The relevant dependencies are
>>>>>>>>> val flinkScala = "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
>>>>>>>>> val flinkStreamingScala = "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
>>>>>>>>> val flinkKafka = "org.apache.flink" %% "flink-connector-kafka" % flinkVersion exclude("org.slf4j", "slf4j-log4j12")
>>>>>>>>> I am using SBT
>>>>>>>>> I tried both connector-kafka and connector-kaka-11 - same result
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Boris Lublinsky
>>>>>>>>> FDP Architect
>>>>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>>>>> On Feb 21, 2019, at 1:38 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Boris,
>>>>>>>>>>
>>>>>>>>>> can you the relevant parts (dependencies) of your pom.xml? Did you also try without fixing the Kafka version, i.e. running with the Kafka client version provided by the Kafka connector of Flink? Gordon (cc) dealt with FLINK-8741.
>>>>>>>>>>
>>>>>>>>>> @Gordon: have you seen this issue with 1.6/1.7 before?
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Konstantin
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>>>> I found some more details on this
>>>>>>>>>> The same error for the same application was reported about a year ago http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E>
>>>>>>>>>> And was due to https://issues.apache.org/jira/browse/FLINK-8741 <https://issues.apache.org/jira/browse/FLINK-8741>
>>>>>>>>>>
>>>>>>>>>> It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with both latest kaffka-connector
>>>>>>>>>> And Kafka-connector-011
>>>>>>>>>>
>>>>>>>>>> Boris Lublinsky
>>>>>>>>>> FDP Architect
>>>>>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>>>>>> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Boris,
>>>>>>>>>>>
>>>>>>>>>>> I haven’t seen this exact error, but I have seen similar errors caused by multiple versions of jars on the classpath.
>>>>>>>>>>>
>>>>>>>>>>> When I’ve run into this particular "XXX is not an instance of YYY" problem, it often seems to be caused by a jar that I should have marked as provided in my pom.
>>>>>>>>>>>
>>>>>>>>>>> Though I’m typically running on a YARN cluster, not w/K8s, so maybe this doesn’t apply.
>>>>>>>>>>>
>>>>>>>>>>> — Ken
>>>>>>>>>>>
>>>>>>>>>>> PS - I assume you’ve been reading https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Konstantin,
>>>>>>>>>>>> After experimenting with this for a while, I got to the root cause of the problem
>>>>>>>>>>>> I am running a version of a Taxi ride travel prediction as my sample.
>>>>>>>>>>>> It works fine in Intellij,
>>>>>>>>>>>> But when I am trying to put it in the docker (standard Debian 1.7 image)
>>>>>>>>>>>> It fails with a following error
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The program finished with the following exception:
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>>>>>>>>>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>>>>>>>>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>>>>>>>>>>>> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>>>>>>>>>>>> at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>>>>>>>>>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>>>>>>>>>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>>>>>>>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>>>>>>>>>>>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>>>>>>>>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>>>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>>>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>>>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>>>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>>>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>>>>>>>>>>>> at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>>>>>>>>>>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>>>>>>>>>>>> at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>>>>>>>>>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>>>>>>>>>>>> ... 19 more
>>>>>>>>>>>> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
>>>>>>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>>>>>>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>>>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>>>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>>>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>>>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>>>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>>>>>>>>>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>>>>>>>>>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>>>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>>>>>>>>>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>>>>>>>>>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>>>>>>>>>>>> at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>>>>>>>>>>>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>>>>>>>>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>>>>>>>>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>>>>>>>>>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>>> Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.serialization.ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer
>>>>>>>>>>>> at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>>>>>>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>>>>>>>>>>>> ... 17 more
>>>>>>>>>>>>
>>>>>>>>>>>> The closest that I found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no <https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no>
>>>>>>>>>>>> Which talks about class loader. (I tried there solution, but it did not help)
>>>>>>>>>>>> I looked at the loading and I see that the pair of these 2 classes is loaded from my uber jar, but twice.
>>>>>>>>>>>>
>>>>>>>>>>>> Have you guys seen this error before?
>>>>>>>>>>>> Any suggestion?
>>>>>>>>>>>>
>>>>>>>>>>>> Boris Lublinsky
>>>>>>>>>>>> FDP Architect
>>>>>>>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>>>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>>>>>>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Boris,
>>>>>>>>>>>>>
>>>>>>>>>>>>> without looking at the entrypoint in much detail, generally there should not be a race condition there:
>>>>>>>>>>>>>
>>>>>>>>>>>>> * if the taskmanagers can not connect to the resourcemanager they will retry (per default the timeout is 5 mins)
>>>>>>>>>>>>> * if the JobManager does not get enough resources from the ResourceManager it will also wait for the resources/slots to provided. The timeout there is also 5 minutes, I think.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So, this should actually be pretty robust as long as the Taskmanager containers can reach the Jobmanager eventually.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Could you provide the Taskmanager/JobManager logs for such a failure case?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Konstantin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>>>>>>> Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker <https://github.com/apache/flink/tree/release-1.7/flink-container/docker>
>>>>>>>>>>>>> I have created an entry point, which looks like follows:
>>>>>>>>>>>>> #!/bin/sh
>>>>>>>>>>>>>
>>>>>>>>>>>>> ################################################################################
>>>>>>>>>>>>> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh>
>>>>>>>>>>>>> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh <https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh>
>>>>>>>>>>>>> ################################################################################
>>>>>>>>>>>>>
>>>>>>>>>>>>> # If unspecified, the hostname of the container is taken as the JobManager address
>>>>>>>>>>>>> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>>>>>>>>>>>>
>>>>>>>>>>>>> drop_privs_cmd() {
>>>>>>>>>>>>> if [ $(id -u) != 0 ]; then
>>>>>>>>>>>>> # Don't need to drop privs if EUID != 0
>>>>>>>>>>>>> return
>>>>>>>>>>>>> elif [ -x /sbin/su-exec ]; then
>>>>>>>>>>>>> # Alpine
>>>>>>>>>>>>> echo su-exec flink
>>>>>>>>>>>>> else
>>>>>>>>>>>>> # Others
>>>>>>>>>>>>> echo gosu flink
>>>>>>>>>>>>> fi
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> JOB_MANAGER="jobmanager"
>>>>>>>>>>>>> TASK_MANAGER="taskmanager"
>>>>>>>>>>>>>
>>>>>>>>>>>>> CMD="$1"
>>>>>>>>>>>>> shift
>>>>>>>>>>>>>
>>>>>>>>>>>>> if [ "${CMD}" = "help" ]; then
>>>>>>>>>>>>> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
>>>>>>>>>>>>> exit 0
>>>>>>>>>>>>> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>>>>>>>>>> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>>>>>>>>>> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>>>>>>>>>>>>
>>>>>>>>>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>>> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>>>
>>>>>>>>>>>>> echo "Starting Task Manager"
>>>>>>>>>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>>>>>>>>>>>> else
>>>>>>>>>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>>>
>>>>>>>>>>>>> if [ -z "$1" ]; then
>>>>>>>>>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
>>>>>>>>>>>>> else
>>>>>>>>>>>>> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
>>>>>>>>>>>>> fi
>>>>>>>>>>>>> fi
>>>>>>>>>>>>> fi
>>>>>>>>>>>>>
>>>>>>>>>>>>> exec "$@"
>>>>>>>>>>>>> It does work for all the cases, except running standalone job.
>>>>>>>>>>>>> The problem, the way I understand it, is a racing condition.
>>>>>>>>>>>>> In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh
>>>>>>>>>>>>> tries to start a job immediately once the cluster is created (before connection is established).
>>>>>>>>>>>>> Is there a better option to implement it starting a job on container startup?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Konstantin Knauf | Solutions Architect
>>>>>>>>>>>>> +49 160 91394525
>>>>>>>>>>>
>>>>>>>>>>> --------------------------
>>>>>>>>>>> Ken Krugler
>>>>>>>>>>> +1 530-210-6378
>>>>>>>>>>> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
>>>>>>>>>>> Custom big data solutions & training
>>>>>>>>>>> Flink, Solr, Hadoop, Cascading & Cassandra
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Konstantin Knauf | Solutions Architect
>>>>>>>>>> +49 160 91394525
>>>>>>>>>> <https://www.ververica.com/>
>>>>>>>>>> Follow us @VervericaData
>>>>>>>>>> --
>>>>>>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference
>>>>>>>>>> Stream Processing | Event Driven | Real Time
>>>>>>>>>> --
>>>>>>>>>> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>>>>>>>> --
>>>>>>>>>> Data Artisans GmbH
>>>>>>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>>>>>>>> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>> --------------------------
>> Ken Krugler
>> +1 530-210-6378
>> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
>> Custom big data solutions & training
>> Flink, Solr, Hadoop, Cascading & Cassandra
>>
>
>
>
> --
> Konstantin Knauf | Solutions Architect
> +49 160 91394525
> <https://www.ververica.com/>
> Follow us @VervericaData
> --
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference
> Stream Processing | Event Driven | Real Time
> --
> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> --
> Data Artisans GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
Re: Starting Flink cluster and running a job
Posted by Konstantin Knauf <ko...@ververica.com>.
Hi Boris,
I was not able tot reproduce your problem. I am using the
docker-entrypoint.sh and Dockerfile from the flink-container project [1].
The content of my fat-jar:
flinkShadowJar
+--- project :fe-flink-common
+--- org.apache.flink:flink-metrics-influxdb_2.11:1.8-SNAPSHOT
+--- org.apache.flink:flink-metrics-slf4j:1.7.2
+--- org.apache.flink:flink-connector-kafka_2.11:1.7.2
| +--- org.apache.flink:flink-connector-kafka-base_2.11:1.7.2
| \--- org.apache.kafka:kafka-clients:2.0.1
| +--- org.lz4:lz4-java:1.4.1
| \--- org.xerial.snappy:snappy-java:1.1.7.1
\--- org.apache.flink:flink-s3-fs-presto:1.7.2
In the standard Docker image the job.jar is "only" linked into the Flink
lib directory. I also tested it with a modified Dokerfile, which copies the
job.jar into the lib directory of Flink.
bash-4.4$ ls -lia lib
total 135140
611154 drwxrwxr-x 1 flink flink 4096 Mar 6 10:20 .
606092 drwxrwxr-x 1 flink flink 4096 Feb 11 15:38 ..
26477916 -rw-r--r-- 1 flink flink 93445474 Feb 11 15:38
flink-dist_2.11-1.7.2.jar
26477917 -rw-r--r-- 1 flink flink 141937 Feb 11 15:37
flink-python_2.11-1.7.2.jar
26477525 -rw-r--r-- 1 flink flink 44278062 Mar 6 10:20 job.jar
26477918 -rw-rw-r-- 1 flink flink 489884 Feb 11 14:32
log4j-1.2.17.jar
26477919 -rw-rw-r-- 1 flink flink 9931 Feb 11 14:32
slf4j-log4j12-1.7.15.jar
This also works. In that case you only need to make sure, that the job.jar
is owned by the *flink* user by epxlicitly changing the ownership in the
Dockerfile. Given your description of the Dockerfile, I suspect this is the
problem.
Cheers,
Konstantin
[1] https://github.com/apache/flink/tree/release-1.7.2/flink-container
On Wed, Feb 27, 2019 at 11:19 PM Boris Lublinsky <
boris.lublinsky@lightbend.com> wrote:
> I am not including Kafka per se - I am using Flink Kafka connector, that
> pulls in Kafka.
> I tried exclude and it failed.
>
> I can’t quite shade Kafka and it does not seem like the right thing to do.
> I tried to see where it is loaded from, and its my jar, but I see its
> being loaded twice when it is in Lib.
>
> You guys should be able to easily reproduce it on any Kafka sample that
> you have
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Feb 27, 2019, at 2:10 PM, Ken Krugler <kk...@transpac.com>
> wrote:
>
> Hi Boris,
>
> 1. I assume you’ve tried marking the Kafka dependencies as “provided”, and
> that also failed, yes?
>
> 2. Did you try shading the Kafka dependencies?
>
> — Ken
>
> PS - Moving this conversation back to the user list.
>
>
> On Feb 27, 2019, at 2:03 PM, Boris Lublinsky <
> boris.lublinsky@lightbend.com> wrote:
>
> Guys, please,
> I do still need help with this one
> It does work if I put my jar into a random directory, but fails if I put
> it into Lib.
> And I do need to put it into lib, because standalone-job.sh does not take
> a jar, it assumes a jar to be in lib
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Feb 26, 2019, at 7:33 AM, Boris Lublinsky <
> boris.lublinsky@lightbend.com> wrote:
>
> Any progress on this one?
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Feb 24, 2019, at 5:16 PM, Boris Lublinsky <
> boris.lublinsky@lightbend.com> wrote:
>
> And there are more explanations and concerns there
> It turns out that the execution depends on where my jar is located.
> If I put it in lib directory (I need this to be able to run
> bin/standalone-job.sh )
> It fails. With the same error.
>
>
> Currently I have the following in this directory
> fdp-flink-taxiride-assembly-2.0.0.jar flink-metrics-prometheus-1.7.2.jar
> flink-queryable-state-runtime_2.11-1.7.2.jar log4j-1.2.17.jar
> flink-dist_2.11-1.7.2.jar flink-python_2.11-1.7.2.jar
> flink-table_2.11-1.7.2.jar slf4j-log4j12-1.7.15.jar
>
> And no one of them seem to directly contain Kafka (except for my jar
> fdp-flink-taxiride-assembly-2.0.0.jar)
>
> But id I move my jar in any other directory it works fine.
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Feb 23, 2019, at 11:40 AM, Boris Lublinsky <
> boris.lublinsky@lightbend.com> wrote:
>
> And its working now.
> Sorry about all confusion.
> The root cause of the problem was that in “standard” Link image directory
> /opt/flink/log does not have write permissions and as a result the local
> cluster was never created
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Feb 22, 2019, at 9:13 AM, Dawid Wysakowicz <dw...@apache.org>
> wrote:
>
> Hi,
>
> One additional question, how do you actually build the docker image? How
> do you put the user jar into the container? Maybe you added extra kafka
> connector to the cluster classpath? Have you checked what is on the
> classpath of a running taskmanager?
>
> Best,
>
> Dawid
> On 22/02/2019 15:44, Boris Lublinsky wrote:
>
> Gordon, I double checked it several times. Here is the list
>
> Columns are
> [info] - Jar-Size including dependencies
> [info] - Jar-Size
> [info] - Number of transitive dependencies
> [info] - Number of direct dependencies
> [info] - ModuleID
> [info] Done updating.
> [info] TotSize JarSize #TDe #Dep Module
> [info] 27.988 MB ------- MB 33 2
> lightbend:fdp-flink-taxiride_2.11:2.0.0
> [info] 19.632 MB ------- MB 24 6 lightbend:support_2.11:2.0.0
> [info] 17.675 MB 0.000 MB 16 9 org.typelevel:cats_2.11:0.9.0
> [info] 17.389 MB 1.182 MB 13 9 org.typelevel:cats-laws_2.11:0.9.0
> [info] 13.115 MB 0.286 MB 7 4 org.typelevel:cats-free_2.11:0.9.0
> [info] 12.830 MB 0.000 MB 7 4 org.typelevel:cats-jvm_2.11:0.9.0
> [info] 12.830 MB 3.403 MB 6 4 org.typelevel:cats-core_2.11:0.9.0
> [info] 8.397 MB 0.068 MB 8 5
> org.apache.flink:flink-connector-kafka_2.11:1.7.1
> [info] 8.175 MB 1.889 MB 4 4 org.apache.kafka:kafka-clients:2.1.0
> [info] 7.973 MB 0.212 MB 7 4
> org.typelevel:cats-kernel-laws_2.11:0.9.0
> [info] 4.834 MB 0.007 MB 4 2 org.typelevel:cats-macros_2.11:0.9.0
> [info] 4.746 MB 0.082 MB 2 2
> com.typesafe.scala-logging:scala-logging_2.11:3.9.0
> [info] 4.658 MB 0.035 MB 1 1 org.typelevel:machinist_2.11:0.6.1
> [info] 4.623 MB 4.623 MB 0 0 org.scala-lang:scala-reflect:2.11.12
> [info] 4.592 MB 4.592 MB 0 0 org.typelevel:cats-kernel_2.11:0.9.0
> [info] 3.714 MB 3.714 MB 0 0 com.github.luben:zstd-jni:1.3.5-4
> [info] 3.152 MB 0.043 MB 2 1 org.typelevel:discipline_2.11:0.7.2
> [info] 3.109 MB 3.094 MB 1 1
> org.scalacheck:scalacheck_2.11:1.13.4
> [info] 2.019 MB 2.019 MB 0 0
> org.xerial.snappy:snappy-java:1.1.7.2
> [info] 0.803 MB 0.290 MB 2 2 ch.qos.logback:logback-classic:1.2.3
> [info] 0.641 MB 0.641 MB 0 0 joda-time:joda-time:2.10.1
> [info] 0.512 MB 0.512 MB 0 0 org.lz4:lz4-java:1.5.0
> [info] 0.472 MB 0.472 MB 0 0 ch.qos.logback:logback-core:1.2.3
> [info] 0.286 MB 0.286 MB 0 0 com.typesafe:config:1.3.3
> [info] 0.195 MB 0.114 MB 3 3
> org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
> [info] 0.170 MB 0.167 MB 1 1
> com.github.mpilquist:simulacrum_2.11:0.10.0
> [info] 0.145 MB 0.145 MB 0 0 org.joda:joda-convert:2.1.2
> [info] 0.041 MB 0.041 MB 0 0 org.slf4j:slf4j-api:1.7.25
> [info] 0.033 MB 0.033 MB 0 0
> com.google.code.findbugs:jsr305:1.3.9
> [info] 0.016 MB 0.002 MB 2 1
> org.typelevel:catalysts-platform_2.11:0.0.5
> [info] 0.015 MB 0.012 MB 1 1
> org.typelevel:catalysts-macros_2.11:0.0.5
> [info] 0.015 MB 0.015 MB 0 0 org.scala-sbt:test-interface:1.0
> [info] 0.007 MB 0.007 MB 0 0 org.apache.flink:force-shading:1.7.1
> [info] 0.003 MB 0.003 MB 0 0
> org.typelevel:macro-compat_2.11:1.1.1
>
>
> And here is another view
>
> [info] lightbend:fdp-flink-taxiride_2.11:2.0.0 [S]
> [info] +-lightbend:support_2.11:2.0.0 [S]
> [info] | +-ch.qos.logback:logback-classic:1.2.3
> [info] | | +-ch.qos.logback:logback-core:1.2.3
> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info] | | +-org.slf4j:slf4j-api:1.7.25
> [info] | |
> [info] | +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info] | | +-org.slf4j:slf4j-api:1.7.25
> [info] | |
> [info] | +-com.typesafe:config:1.3.3
> [info] | +-joda-time:joda-time:2.10.1
> [info] | +-org.joda:joda-convert:2.1.2
> [info] | +-org.typelevel:cats_2.11:0.9.0 [S]
> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:cats-free_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | | |
> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | | |
> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | | +-org.scala-sbt:test-interface:1.0
> [info] | | |
> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | +-org.scala-sbt:test-interface:1.0
> [info] | |
> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | +-org.typelevel:cats-laws_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | | +-org.scala-sbt:test-interface:1.0
> [info] | | |
> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | | |
> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | | | +-org.scala-sbt:test-interface:1.0
> [info] | | | |
> [info] | | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info] | | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | | +-org.scala-sbt:test-interface:1.0
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | | +-org.scala-sbt:test-interface:1.0
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] |
> [info] +-org.apache.flink:flink-connector-kafka_2.11:1.7.1
> [info] +-com.google.code.findbugs:jsr305:1.3.9
> [info] +-org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
> [info] | +-com.google.code.findbugs:jsr305:1.3.9
> [info] | +-org.apache.flink:force-shading:1.7.1
> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info] | +-org.slf4j:slf4j-api:1.7.25
> [info] |
> [info] +-org.apache.flink:force-shading:1.7.1
> [info] +-org.apache.kafka:kafka-clients:2.1.0
> [info] | +-com.github.luben:zstd-jni:1.3.5-4
> [info] | +-org.lz4:lz4-java:1.5.0
> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info] | +-org.slf4j:slf4j-api:1.7.25
> [info] | +-org.xerial.snappy:snappy-java:1.1.7.2
> [info] |
> [info] +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info] +-org.slf4j:slf4j-api:1.7.25
> [info]
> [info] Done updating.
> [info] lightbend:support_2.11:2.0.0 [S]
> [info] +-ch.qos.logback:logback-classic:1.2.3
> [info] | +-ch.qos.logback:logback-core:1.2.3
> [info] | +-org.slf4j:slf4j-api:1.7.25
> [info] |
> [info] +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | +-org.slf4j:slf4j-api:1.7.25
> [info] |
> [info] +-com.typesafe:config:1.3.3
> [info] +-joda-time:joda-time:2.10.1
> [info] +-org.joda:joda-convert:2.1.2
> [info] +-org.typelevel:cats_2.11:0.9.0 [S]
> [info] +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] |
> [info] +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] |
> [info] +-org.typelevel:cats-free_2.11:0.9.0 [S]
> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] |
> [info] +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] |
> [info] +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | +-org.scala-sbt:test-interface:1.0
> [info] | |
> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | +-org.scala-sbt:test-interface:1.0
> [info] |
> [info] +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] +-org.typelevel:cats-laws_2.11:0.9.0 [S]
> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | +-org.scala-sbt:test-interface:1.0
> [info] | |
> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | | +-org.scala-sbt:test-interface:1.0
> [info] | | |
> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | +-org.scala-sbt:test-interface:1.0
> [info] | |
> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | +-org.scala-sbt:test-interface:1.0
> [info] | |
> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] |
> [info] +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] |
> [info] +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]
>
>
>
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Feb 22, 2019, at 12:33 AM, Tzu-Li (Gordon) Tai <go...@ververica.com>
> wrote:
>
> Hi,
>
> I haven't seen this problem for Flink 1.6.x / 1.7.x, so it shouldn't be a
> reoccurrence of FLINK-8741.
> I've double checked the used classloaders in the Kafka connector, they
> seem to be correct.
>
> The fact that it works correctly in IntelliJ, but not when packaged,
> suggests that there could be some conflicting dependencies in the packaged
> jar.
>
> Could you check the actual resolved dependency tree of the project, and
> see if there are multiple versions of some dependency related to Kafka
> being pulled in? For Maven for example, that would be "mvn dependency:tree".
>
> Cheers,
> Gordon
>
> On Thu, Feb 21, 2019 at 10:49 PM Boris Lublinsky <
> boris.lublinsky@lightbend.com> wrote:
>
>> The relevant dependencies are
>>
>> val flinkScala = "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"val flinkStreamingScala = "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"val flinkKafka = "org.apache.flink" %% "flink-connector-kafka" % flinkVersion exclude("org.slf4j", "slf4j-log4j12")
>>
>> I am using SBT
>> I tried both connector-kafka and connector-kaka-11 - same result
>>
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com
>> https://www.lightbend.com/
>>
>> On Feb 21, 2019, at 1:38 AM, Konstantin Knauf <ko...@ververica.com>
>> wrote:
>>
>> Hi Boris,
>>
>> can you the relevant parts (dependencies) of your pom.xml? Did you also
>> try without fixing the Kafka version, i.e. running with the Kafka client
>> version provided by the Kafka connector of Flink? Gordon (cc) dealt with
>> FLINK-8741.
>>
>> @Gordon: have you seen this issue with 1.6/1.7 before?
>>
>> Cheers,
>>
>> Konstantin
>>
>> On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky <
>> boris.lublinsky@lightbend.com> wrote:
>>
>>> I found some more details on this
>>> The same error for the same application was reported about a year ago
>>> http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E
>>> And was due to https://issues.apache.org/jira/browse/FLINK-8741
>>>
>>> It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with
>>> both latest kaffka-connector
>>> And Kafka-connector-011
>>>
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com
>>> https://www.lightbend.com/
>>>
>>> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kk...@transpac.com>
>>> wrote:
>>>
>>> Hi Boris,
>>>
>>> I haven’t seen this exact error, but I have seen similar errors caused
>>> by multiple versions of jars on the classpath.
>>>
>>> When I’ve run into this particular "XXX is not an instance of YYY"
>>> problem, it often seems to be caused by a jar that I should have marked as
>>> provided in my pom.
>>>
>>> Though I’m typically running on a YARN cluster, not w/K8s, so maybe this
>>> doesn’t apply.
>>>
>>> — Ken
>>>
>>> PS - I assume you’ve been reading
>>> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html
>>>
>>>
>>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <
>>> boris.lublinsky@lightbend.com> wrote:
>>>
>>> Konstantin,
>>> After experimenting with this for a while, I got to the root cause of
>>> the problem
>>> I am running a version of a Taxi ride travel prediction as my sample.
>>> It works fine in Intellij,
>>> But when I am trying to put it in the docker (standard Debian 1.7 image)
>>> It fails with a following error
>>>
>>>
>>> The program finished with the following exception:
>>>
>>> org.apache.flink.client.program.ProgramInvocationException: Job failed.
>>> (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>>> at
>>> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>>> at
>>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>>> at
>>> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>>> at
>>> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>>> at
>>> com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>>> at
>>> com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at
>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>>> at
>>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>>> at
>>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>>> at
>>> org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>>> at
>>> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job
>>> execution failed.
>>> at
>>> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>>> at
>>> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>>> ... 19 more
>>> Caused by: org.apache.kafka.common.KafkaException: Failed to construct
>>> kafka producer
>>> at
>>> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>>> at
>>> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>>> at
>>> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>>> at
>>> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>>> at
>>> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>>> at
>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>>> at
>>> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>>> at
>>> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>>> at
>>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>>> at
>>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>>> at
>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.kafka.common.KafkaException:
>>> org.apache.kafka.common.serialization.ByteArraySerializer is not an
>>> instance of org.apache.kafka.common.serialization.Serializer
>>> at
>>> org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>>> at
>>> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>>> ... 17 more
>>>
>>> The closest that I found
>>> https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no
>>> Which talks about class loader. (I tried there solution, but it did not
>>> help)
>>> I looked at the loading and I see that the pair of these 2 classes is
>>> loaded from my uber jar, but twice.
>>>
>>> Have you guys seen this error before?
>>> Any suggestion?
>>>
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com
>>> https://www.lightbend.com/
>>>
>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <ko...@ververica.com>
>>> wrote:
>>>
>>> Hi Boris,
>>>
>>> without looking at the entrypoint in much detail, generally there should
>>> not be a race condition there:
>>>
>>> * if the taskmanagers can not connect to the resourcemanager they will
>>> retry (per default the timeout is 5 mins)
>>> * if the JobManager does not get enough resources from the
>>> ResourceManager it will also wait for the resources/slots to provided. The
>>> timeout there is also 5 minutes, I think.
>>>
>>> So, this should actually be pretty robust as long as the Taskmanager
>>> containers can reach the Jobmanager eventually.
>>>
>>> Could you provide the Taskmanager/JobManager logs for such a failure
>>> case?
>>>
>>> Cheers,
>>>
>>> Konstantin
>>>
>>>
>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <
>>> boris.lublinsky@lightbend.com> wrote:
>>>
>>>> Following
>>>> https://github.com/apache/flink/tree/release-1.7/flink-container/docker
>>>> I have created an entry point, which looks like follows:
>>>>
>>>> #!/bin/sh################################################################################# from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh# and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh################################################################################# If unspecified, the hostname of the container is taken as the JobManager addressJOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>>> drop_privs_cmd() {
>>>> if [ $(id -u) != 0 ]; then # Don't need to drop privs if EUID != 0 return elif [ -x /sbin/su-exec ]; then # Alpine echo su-exec flink
>>>> else # Others echo gosu flink
>>>> fi}
>>>>
>>>> JOB_MANAGER="jobmanager"TASK_MANAGER="taskmanager"CMD="$1"shiftif [ "${CMD}" = "help" ]; then echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)" exit 0elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then if [ "${CMD}" = "${TASK_MANAGER}" ]; then TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>>>
>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml" sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml" echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml" echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml" echo "Starting Task Manager" echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml" exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>>> else sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml" echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml" echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml" echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml" if [ -z "$1" ]; then exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@" else exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@" fi fifiexec "$@"
>>>>
>>>> It does work for all the cases, except running standalone job.
>>>> The problem, the way I understand it, is a racing condition.
>>>> In kubernetes it takes several attempts for establish connection
>>>> between Job and Task manager, while standalone-job.sh
>>>> tries to start a job immediately once the cluster is created (before
>>>> connection is established).
>>>> Is there a better option to implement it starting a job on container
>>>> startup?
>>>>
>>>>
>>>
>>>
>>> --
>>> Konstantin Knauf | Solutions Architect
>>> +49 160 91394525
>>>
>>>
>>> --------------------------
>>> Ken Krugler
>>> +1 530-210-6378
>>> http://www.scaleunlimited.com
>>> Custom big data solutions & training
>>> Flink, Solr, Hadoop, Cascading & Cassandra
>>>
>>>
>>>
>>
>> --
>> Konstantin Knauf | Solutions Architect
>> +49 160 91394525
>> <https://www.ververica.com/>
>>
>> Follow us @VervericaData
>> --
>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink
>> Conference
>> Stream Processing | Event Driven | Real Time
>> --
>> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>> --
>> Data Artisans GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing
>> Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>>
>>
>>
>
>
>
>
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassandra
>
>
>
--
Konstantin Knauf | Solutions Architect
+49 160 91394525
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
Re: Starting Flink cluster and running a job
Posted by Boris Lublinsky <bo...@lightbend.com>.
I am not including Kafka per se - I am using Flink Kafka connector, that pulls in Kafka.
I tried exclude and it failed.
I can’t quite shade Kafka and it does not seem like the right thing to do.
I tried to see where it is loaded from, and its my jar, but I see its being loaded twice when it is in Lib.
You guys should be able to easily reproduce it on any Kafka sample that you have
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
> On Feb 27, 2019, at 2:10 PM, Ken Krugler <kk...@transpac.com> wrote:
>
> Hi Boris,
>
> 1. I assume you’ve tried marking the Kafka dependencies as “provided”, and that also failed, yes?
>
> 2. Did you try shading the Kafka dependencies?
>
> — Ken
>
> PS - Moving this conversation back to the user list.
>
>
>> On Feb 27, 2019, at 2:03 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>
>> Guys, please,
>> I do still need help with this one
>> It does work if I put my jar into a random directory, but fails if I put it into Lib.
>> And I do need to put it into lib, because standalone-job.sh does not take a jar, it assumes a jar to be in lib
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>> On Feb 26, 2019, at 7:33 AM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>
>>> Any progress on this one?
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>> On Feb 24, 2019, at 5:16 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>
>>>> And there are more explanations and concerns there
>>>> It turns out that the execution depends on where my jar is located.
>>>> If I put it in lib directory (I need this to be able to run bin/standalone-job.sh )
>>>> It fails. With the same error.
>>>>
>>>>
>>>> Currently I have the following in this directory
>>>> fdp-flink-taxiride-assembly-2.0.0.jar flink-metrics-prometheus-1.7.2.jar flink-queryable-state-runtime_2.11-1.7.2.jar log4j-1.2.17.jar
>>>> flink-dist_2.11-1.7.2.jar flink-python_2.11-1.7.2.jar flink-table_2.11-1.7.2.jar slf4j-log4j12-1.7.15.jar
>>>>
>>>> And no one of them seem to directly contain Kafka (except for my jar fdp-flink-taxiride-assembly-2.0.0.jar)
>>>>
>>>> But id I move my jar in any other directory it works fine.
>>>>
>>>>
>>>> Boris Lublinsky
>>>> FDP Architect
>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>> On Feb 23, 2019, at 11:40 AM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>
>>>>> And its working now.
>>>>> Sorry about all confusion.
>>>>> The root cause of the problem was that in “standard” Link image directory /opt/flink/log does not have write permissions and as a result the local cluster was never created
>>>>>
>>>>> Boris Lublinsky
>>>>> FDP Architect
>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>> On Feb 22, 2019, at 9:13 AM, Dawid Wysakowicz <dwysakowicz@apache.org <ma...@apache.org>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> One additional question, how do you actually build the docker image? How do you put the user jar into the container? Maybe you added extra kafka connector to the cluster classpath? Have you checked what is on the classpath of a running taskmanager?
>>>>>> Best,
>>>>>>
>>>>>> Dawid
>>>>>> On 22/02/2019 15:44, Boris Lublinsky wrote:
>>>>>>> Gordon, I double checked it several times. Here is the list
>>>>>>>
>>>>>>> Columns are
>>>>>>> [info] - Jar-Size including dependencies
>>>>>>> [info] - Jar-Size
>>>>>>> [info] - Number of transitive dependencies
>>>>>>> [info] - Number of direct dependencies
>>>>>>> [info] - ModuleID
>>>>>>> [info] Done updating.
>>>>>>> [info] TotSize JarSize #TDe #Dep Module
>>>>>>> [info] 27.988 MB ------- MB 33 2 lightbend:fdp-flink-taxiride_2.11:2.0.0
>>>>>>> [info] 19.632 MB ------- MB 24 6 lightbend:support_2.11:2.0.0
>>>>>>> [info] 17.675 MB 0.000 MB 16 9 org.typelevel:cats_2.11:0.9.0
>>>>>>> [info] 17.389 MB 1.182 MB 13 9 org.typelevel:cats-laws_2.11:0.9.0
>>>>>>> [info] 13.115 MB 0.286 MB 7 4 org.typelevel:cats-free_2.11:0.9.0
>>>>>>> [info] 12.830 MB 0.000 MB 7 4 org.typelevel:cats-jvm_2.11:0.9.0
>>>>>>> [info] 12.830 MB 3.403 MB 6 4 org.typelevel:cats-core_2.11:0.9.0
>>>>>>> [info] 8.397 MB 0.068 MB 8 5 org.apache.flink:flink-connector-kafka_2.11:1.7.1
>>>>>>> [info] 8.175 MB 1.889 MB 4 4 org.apache.kafka:kafka-clients:2.1.0
>>>>>>> [info] 7.973 MB 0.212 MB 7 4 org.typelevel:cats-kernel-laws_2.11:0.9.0
>>>>>>> [info] 4.834 MB 0.007 MB 4 2 org.typelevel:cats-macros_2.11:0.9.0
>>>>>>> [info] 4.746 MB 0.082 MB 2 2 com.typesafe.scala-logging:scala-logging_2.11:3.9.0
>>>>>>> [info] 4.658 MB 0.035 MB 1 1 org.typelevel:machinist_2.11:0.6.1
>>>>>>> [info] 4.623 MB 4.623 MB 0 0 org.scala-lang:scala-reflect:2.11.12
>>>>>>> [info] 4.592 MB 4.592 MB 0 0 org.typelevel:cats-kernel_2.11:0.9.0
>>>>>>> [info] 3.714 MB 3.714 MB 0 0 com.github.luben:zstd-jni:1.3.5-4
>>>>>>> [info] 3.152 MB 0.043 MB 2 1 org.typelevel:discipline_2.11:0.7.2
>>>>>>> [info] 3.109 MB 3.094 MB 1 1 org.scalacheck:scalacheck_2.11:1.13.4
>>>>>>> [info] 2.019 MB 2.019 MB 0 0 org.xerial.snappy:snappy-java:1.1.7.2
>>>>>>> [info] 0.803 MB 0.290 MB 2 2 ch.qos.logback:logback-classic:1.2.3
>>>>>>> [info] 0.641 MB 0.641 MB 0 0 joda-time:joda-time:2.10.1
>>>>>>> [info] 0.512 MB 0.512 MB 0 0 org.lz4:lz4-java:1.5.0
>>>>>>> [info] 0.472 MB 0.472 MB 0 0 ch.qos.logback:logback-core:1.2.3
>>>>>>> [info] 0.286 MB 0.286 MB 0 0 com.typesafe:config:1.3.3
>>>>>>> [info] 0.195 MB 0.114 MB 3 3 org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>>>>>>> [info] 0.170 MB 0.167 MB 1 1 com.github.mpilquist:simulacrum_2.11:0.10.0
>>>>>>> [info] 0.145 MB 0.145 MB 0 0 org.joda:joda-convert:2.1.2
>>>>>>> [info] 0.041 MB 0.041 MB 0 0 org.slf4j:slf4j-api:1.7.25
>>>>>>> [info] 0.033 MB 0.033 MB 0 0 com.google.code.findbugs:jsr305:1.3.9
>>>>>>> [info] 0.016 MB 0.002 MB 2 1 org.typelevel:catalysts-platform_2.11:0.0.5
>>>>>>> [info] 0.015 MB 0.012 MB 1 1 org.typelevel:catalysts-macros_2.11:0.0.5
>>>>>>> [info] 0.015 MB 0.015 MB 0 0 org.scala-sbt:test-interface:1.0
>>>>>>> [info] 0.007 MB 0.007 MB 0 0 org.apache.flink:force-shading:1.7.1
>>>>>>> [info] 0.003 MB 0.003 MB 0 0 org.typelevel:macro-compat_2.11:1.1.1
>>>>>>>
>>>>>>>
>>>>>>> And here is another view
>>>>>>>
>>>>>>> [info] lightbend:fdp-flink-taxiride_2.11:2.0.0 [S]
>>>>>>> [info] +-lightbend:support_2.11:2.0.0 [S]
>>>>>>> [info] | +-ch.qos.logback:logback-classic:1.2.3
>>>>>>> [info] | | +-ch.qos.logback:logback-core:1.2.3
>>>>>>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>>>>>>> [info] | |
>>>>>>> [info] | +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>>>>>>> [info] | |
>>>>>>> [info] | +-com.typesafe:config:1.3.3
>>>>>>> [info] | +-joda-time:joda-time:2.10.1
>>>>>>> [info] | +-org.joda:joda-convert:2.1.2
>>>>>>> [info] | +-org.typelevel:cats_2.11:0.9.0 [S]
>>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-free_2.11:0.9.0 [S]
>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | | | |
>>>>>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | | | |
>>>>>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | | | |
>>>>>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>> [info] | | | | +-org.scala-sbt:test-interface:1.0
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>>> [info] | | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] |
>>>>>>> [info] +-org.apache.flink:flink-connector-kafka_2.11:1.7.1
>>>>>>> [info] +-com.google.code.findbugs:jsr305:1.3.9
>>>>>>> [info] +-org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>>>>>>> [info] | +-com.google.code.findbugs:jsr305:1.3.9
>>>>>>> [info] | +-org.apache.flink:force-shading:1.7.1
>>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>>>>> [info] |
>>>>>>> [info] +-org.apache.flink:force-shading:1.7.1
>>>>>>> [info] +-org.apache.kafka:kafka-clients:2.1.0
>>>>>>> [info] | +-com.github.luben:zstd-jni:1.3.5-4
>>>>>>> [info] | +-org.lz4:lz4-java:1.5.0
>>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>>>>> [info] | +-org.xerial.snappy:snappy-java:1.1.7.2
>>>>>>> [info] |
>>>>>>> [info] +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>>> [info] +-org.slf4j:slf4j-api:1.7.25
>>>>>>> [info]
>>>>>>> [info] Done updating.
>>>>>>> [info] lightbend:support_2.11:2.0.0 [S]
>>>>>>> [info] +-ch.qos.logback:logback-classic:1.2.3
>>>>>>> [info] | +-ch.qos.logback:logback-core:1.2.3
>>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>>>>> [info] |
>>>>>>> [info] +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>>>>> [info] |
>>>>>>> [info] +-com.typesafe:config:1.3.3
>>>>>>> [info] +-joda-time:joda-time:2.10.1
>>>>>>> [info] +-org.joda:joda-convert:2.1.2
>>>>>>> [info] +-org.typelevel:cats_2.11:0.9.0 [S]
>>>>>>> [info] +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] |
>>>>>>> [info] +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] |
>>>>>>> [info] +-org.typelevel:cats-free_2.11:0.9.0 [S]
>>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] |
>>>>>>> [info] +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] |
>>>>>>> [info] +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>>>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>> [info] | +-org.scala-sbt:test-interface:1.0
>>>>>>> [info] |
>>>>>>> [info] +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | | |
>>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | | |
>>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] |
>>>>>>> [info] +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>>> [info] | |
>>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info] |
>>>>>>> [info] +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>>> [info] +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>>> [info]
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Boris Lublinsky
>>>>>>> FDP Architect
>>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>>> On Feb 22, 2019, at 12:33 AM, Tzu-Li (Gordon) Tai <gordon@ververica.com <ma...@ververica.com>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I haven't seen this problem for Flink 1.6.x / 1.7.x, so it shouldn't be a reoccurrence of FLINK-8741.
>>>>>>>> I've double checked the used classloaders in the Kafka connector, they seem to be correct.
>>>>>>>>
>>>>>>>> The fact that it works correctly in IntelliJ, but not when packaged, suggests that there could be some conflicting dependencies in the packaged jar.
>>>>>>>>
>>>>>>>> Could you check the actual resolved dependency tree of the project, and see if there are multiple versions of some dependency related to Kafka being pulled in? For Maven for example, that would be "mvn dependency:tree".
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Gordon
>>>>>>>>
>>>>>>>> On Thu, Feb 21, 2019 at 10:49 PM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>> The relevant dependencies are
>>>>>>>> val flinkScala = "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
>>>>>>>> val flinkStreamingScala = "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
>>>>>>>> val flinkKafka = "org.apache.flink" %% "flink-connector-kafka" % flinkVersion exclude("org.slf4j", "slf4j-log4j12")
>>>>>>>> I am using SBT
>>>>>>>> I tried both connector-kafka and connector-kaka-11 - same result
>>>>>>>>
>>>>>>>>
>>>>>>>> Boris Lublinsky
>>>>>>>> FDP Architect
>>>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>>>> On Feb 21, 2019, at 1:38 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Boris,
>>>>>>>>>
>>>>>>>>> can you the relevant parts (dependencies) of your pom.xml? Did you also try without fixing the Kafka version, i.e. running with the Kafka client version provided by the Kafka connector of Flink? Gordon (cc) dealt with FLINK-8741.
>>>>>>>>>
>>>>>>>>> @Gordon: have you seen this issue with 1.6/1.7 before?
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Konstantin
>>>>>>>>>
>>>>>>>>> On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>>> I found some more details on this
>>>>>>>>> The same error for the same application was reported about a year ago http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E>
>>>>>>>>> And was due to https://issues.apache.org/jira/browse/FLINK-8741 <https://issues.apache.org/jira/browse/FLINK-8741>
>>>>>>>>>
>>>>>>>>> It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with both latest kaffka-connector
>>>>>>>>> And Kafka-connector-011
>>>>>>>>>
>>>>>>>>> Boris Lublinsky
>>>>>>>>> FDP Architect
>>>>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>>>>> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Boris,
>>>>>>>>>>
>>>>>>>>>> I haven’t seen this exact error, but I have seen similar errors caused by multiple versions of jars on the classpath.
>>>>>>>>>>
>>>>>>>>>> When I’ve run into this particular "XXX is not an instance of YYY" problem, it often seems to be caused by a jar that I should have marked as provided in my pom.
>>>>>>>>>>
>>>>>>>>>> Though I’m typically running on a YARN cluster, not w/K8s, so maybe this doesn’t apply.
>>>>>>>>>>
>>>>>>>>>> — Ken
>>>>>>>>>>
>>>>>>>>>> PS - I assume you’ve been reading https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Konstantin,
>>>>>>>>>>> After experimenting with this for a while, I got to the root cause of the problem
>>>>>>>>>>> I am running a version of a Taxi ride travel prediction as my sample.
>>>>>>>>>>> It works fine in Intellij,
>>>>>>>>>>> But when I am trying to put it in the docker (standard Debian 1.7 image)
>>>>>>>>>>> It fails with a following error
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The program finished with the following exception:
>>>>>>>>>>>
>>>>>>>>>>> org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>>>>>>>>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>>>>>>>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>>>>>>>>>>> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>>>>>>>>>>> at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>>>>>>>>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>>>>>>>>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>>>>>>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>>>>>>>>>>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>>>>>>>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>>>>>>>>>>> at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>>>>>>>>>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>>>>>>>>>>> at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>>>>>>>>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>>>>>>>>>>> ... 19 more
>>>>>>>>>>> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
>>>>>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>>>>>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>>>>>>>>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>>>>>>>>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>>>>>>>>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>>>>>>>>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>>>>>>>>>>> at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>>>>>>>>>>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>>>>>>>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>>>>>>>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>>>>>>>>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>>> Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.serialization.ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer
>>>>>>>>>>> at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>>>>>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>>>>>>>>>>> ... 17 more
>>>>>>>>>>>
>>>>>>>>>>> The closest that I found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no <https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no>
>>>>>>>>>>> Which talks about class loader. (I tried there solution, but it did not help)
>>>>>>>>>>> I looked at the loading and I see that the pair of these 2 classes is loaded from my uber jar, but twice.
>>>>>>>>>>>
>>>>>>>>>>> Have you guys seen this error before?
>>>>>>>>>>> Any suggestion?
>>>>>>>>>>>
>>>>>>>>>>> Boris Lublinsky
>>>>>>>>>>> FDP Architect
>>>>>>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>>>>>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi Boris,
>>>>>>>>>>>>
>>>>>>>>>>>> without looking at the entrypoint in much detail, generally there should not be a race condition there:
>>>>>>>>>>>>
>>>>>>>>>>>> * if the taskmanagers can not connect to the resourcemanager they will retry (per default the timeout is 5 mins)
>>>>>>>>>>>> * if the JobManager does not get enough resources from the ResourceManager it will also wait for the resources/slots to provided. The timeout there is also 5 minutes, I think.
>>>>>>>>>>>>
>>>>>>>>>>>> So, this should actually be pretty robust as long as the Taskmanager containers can reach the Jobmanager eventually.
>>>>>>>>>>>>
>>>>>>>>>>>> Could you provide the Taskmanager/JobManager logs for such a failure case?
>>>>>>>>>>>>
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>
>>>>>>>>>>>> Konstantin
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>>>>>> Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker <https://github.com/apache/flink/tree/release-1.7/flink-container/docker>
>>>>>>>>>>>> I have created an entry point, which looks like follows:
>>>>>>>>>>>> #!/bin/sh
>>>>>>>>>>>>
>>>>>>>>>>>> ################################################################################
>>>>>>>>>>>> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh>
>>>>>>>>>>>> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh <https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh>
>>>>>>>>>>>> ################################################################################
>>>>>>>>>>>>
>>>>>>>>>>>> # If unspecified, the hostname of the container is taken as the JobManager address
>>>>>>>>>>>> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>>>>>>>>>>>
>>>>>>>>>>>> drop_privs_cmd() {
>>>>>>>>>>>> if [ $(id -u) != 0 ]; then
>>>>>>>>>>>> # Don't need to drop privs if EUID != 0
>>>>>>>>>>>> return
>>>>>>>>>>>> elif [ -x /sbin/su-exec ]; then
>>>>>>>>>>>> # Alpine
>>>>>>>>>>>> echo su-exec flink
>>>>>>>>>>>> else
>>>>>>>>>>>> # Others
>>>>>>>>>>>> echo gosu flink
>>>>>>>>>>>> fi
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> JOB_MANAGER="jobmanager"
>>>>>>>>>>>> TASK_MANAGER="taskmanager"
>>>>>>>>>>>>
>>>>>>>>>>>> CMD="$1"
>>>>>>>>>>>> shift
>>>>>>>>>>>>
>>>>>>>>>>>> if [ "${CMD}" = "help" ]; then
>>>>>>>>>>>> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
>>>>>>>>>>>> exit 0
>>>>>>>>>>>> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>>>>>>>>> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>>>>>>>>> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>>>>>>>>>>>
>>>>>>>>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>>
>>>>>>>>>>>> echo "Starting Task Manager"
>>>>>>>>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>>>>>>>>>>> else
>>>>>>>>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>>
>>>>>>>>>>>> if [ -z "$1" ]; then
>>>>>>>>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
>>>>>>>>>>>> else
>>>>>>>>>>>> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
>>>>>>>>>>>> fi
>>>>>>>>>>>> fi
>>>>>>>>>>>> fi
>>>>>>>>>>>>
>>>>>>>>>>>> exec "$@"
>>>>>>>>>>>> It does work for all the cases, except running standalone job.
>>>>>>>>>>>> The problem, the way I understand it, is a racing condition.
>>>>>>>>>>>> In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh
>>>>>>>>>>>> tries to start a job immediately once the cluster is created (before connection is established).
>>>>>>>>>>>> Is there a better option to implement it starting a job on container startup?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Konstantin Knauf | Solutions Architect
>>>>>>>>>>>> +49 160 91394525
>>>>>>>>>>
>>>>>>>>>> --------------------------
>>>>>>>>>> Ken Krugler
>>>>>>>>>> +1 530-210-6378
>>>>>>>>>> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
>>>>>>>>>> Custom big data solutions & training
>>>>>>>>>> Flink, Solr, Hadoop, Cascading & Cassandra
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Konstantin Knauf | Solutions Architect
>>>>>>>>> +49 160 91394525
>>>>>>>>> <https://www.ververica.com/>
>>>>>>>>> Follow us @VervericaData
>>>>>>>>> --
>>>>>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference
>>>>>>>>> Stream Processing | Event Driven | Real Time
>>>>>>>>> --
>>>>>>>>> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>>>>>>> --
>>>>>>>>> Data Artisans GmbH
>>>>>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>>>>>>> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassandra
>
Re: Starting Flink cluster and running a job
Posted by Ken Krugler <kk...@transpac.com>.
Hi Boris,
1. I assume you’ve tried marking the Kafka dependencies as “provided”, and that also failed, yes?
2. Did you try shading the Kafka dependencies?
— Ken
PS - Moving this conversation back to the user list.
> On Feb 27, 2019, at 2:03 PM, Boris Lublinsky <bo...@lightbend.com> wrote:
>
> Guys, please,
> I do still need help with this one
> It does work if I put my jar into a random directory, but fails if I put it into Lib.
> And I do need to put it into lib, because standalone-job.sh does not take a jar, it assumes a jar to be in lib
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com <ma...@lightbend.com>
> https://www.lightbend.com/
>
>> On Feb 26, 2019, at 7:33 AM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>
>> Any progress on this one?
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>> On Feb 24, 2019, at 5:16 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>
>>> And there are more explanations and concerns there
>>> It turns out that the execution depends on where my jar is located.
>>> If I put it in lib directory (I need this to be able to run bin/standalone-job.sh )
>>> It fails. With the same error.
>>>
>>>
>>> Currently I have the following in this directory
>>> fdp-flink-taxiride-assembly-2.0.0.jar flink-metrics-prometheus-1.7.2.jar flink-queryable-state-runtime_2.11-1.7.2.jar log4j-1.2.17.jar
>>> flink-dist_2.11-1.7.2.jar flink-python_2.11-1.7.2.jar flink-table_2.11-1.7.2.jar slf4j-log4j12-1.7.15.jar
>>>
>>> And no one of them seem to directly contain Kafka (except for my jar fdp-flink-taxiride-assembly-2.0.0.jar)
>>>
>>> But id I move my jar in any other directory it works fine.
>>>
>>>
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>> On Feb 23, 2019, at 11:40 AM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>
>>>> And its working now.
>>>> Sorry about all confusion.
>>>> The root cause of the problem was that in “standard” Link image directory /opt/flink/log does not have write permissions and as a result the local cluster was never created
>>>>
>>>> Boris Lublinsky
>>>> FDP Architect
>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>> On Feb 22, 2019, at 9:13 AM, Dawid Wysakowicz <dwysakowicz@apache.org <ma...@apache.org>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> One additional question, how do you actually build the docker image? How do you put the user jar into the container? Maybe you added extra kafka connector to the cluster classpath? Have you checked what is on the classpath of a running taskmanager?
>>>>> Best,
>>>>>
>>>>> Dawid
>>>>> On 22/02/2019 15:44, Boris Lublinsky wrote:
>>>>>> Gordon, I double checked it several times. Here is the list
>>>>>>
>>>>>> Columns are
>>>>>> [info] - Jar-Size including dependencies
>>>>>> [info] - Jar-Size
>>>>>> [info] - Number of transitive dependencies
>>>>>> [info] - Number of direct dependencies
>>>>>> [info] - ModuleID
>>>>>> [info] Done updating.
>>>>>> [info] TotSize JarSize #TDe #Dep Module
>>>>>> [info] 27.988 MB ------- MB 33 2 lightbend:fdp-flink-taxiride_2.11:2.0.0
>>>>>> [info] 19.632 MB ------- MB 24 6 lightbend:support_2.11:2.0.0
>>>>>> [info] 17.675 MB 0.000 MB 16 9 org.typelevel:cats_2.11:0.9.0
>>>>>> [info] 17.389 MB 1.182 MB 13 9 org.typelevel:cats-laws_2.11:0.9.0
>>>>>> [info] 13.115 MB 0.286 MB 7 4 org.typelevel:cats-free_2.11:0.9.0
>>>>>> [info] 12.830 MB 0.000 MB 7 4 org.typelevel:cats-jvm_2.11:0.9.0
>>>>>> [info] 12.830 MB 3.403 MB 6 4 org.typelevel:cats-core_2.11:0.9.0
>>>>>> [info] 8.397 MB 0.068 MB 8 5 org.apache.flink:flink-connector-kafka_2.11:1.7.1
>>>>>> [info] 8.175 MB 1.889 MB 4 4 org.apache.kafka:kafka-clients:2.1.0
>>>>>> [info] 7.973 MB 0.212 MB 7 4 org.typelevel:cats-kernel-laws_2.11:0.9.0
>>>>>> [info] 4.834 MB 0.007 MB 4 2 org.typelevel:cats-macros_2.11:0.9.0
>>>>>> [info] 4.746 MB 0.082 MB 2 2 com.typesafe.scala-logging:scala-logging_2.11:3.9.0
>>>>>> [info] 4.658 MB 0.035 MB 1 1 org.typelevel:machinist_2.11:0.6.1
>>>>>> [info] 4.623 MB 4.623 MB 0 0 org.scala-lang:scala-reflect:2.11.12
>>>>>> [info] 4.592 MB 4.592 MB 0 0 org.typelevel:cats-kernel_2.11:0.9.0
>>>>>> [info] 3.714 MB 3.714 MB 0 0 com.github.luben:zstd-jni:1.3.5-4
>>>>>> [info] 3.152 MB 0.043 MB 2 1 org.typelevel:discipline_2.11:0.7.2
>>>>>> [info] 3.109 MB 3.094 MB 1 1 org.scalacheck:scalacheck_2.11:1.13.4
>>>>>> [info] 2.019 MB 2.019 MB 0 0 org.xerial.snappy:snappy-java:1.1.7.2
>>>>>> [info] 0.803 MB 0.290 MB 2 2 ch.qos.logback:logback-classic:1.2.3
>>>>>> [info] 0.641 MB 0.641 MB 0 0 joda-time:joda-time:2.10.1
>>>>>> [info] 0.512 MB 0.512 MB 0 0 org.lz4:lz4-java:1.5.0
>>>>>> [info] 0.472 MB 0.472 MB 0 0 ch.qos.logback:logback-core:1.2.3
>>>>>> [info] 0.286 MB 0.286 MB 0 0 com.typesafe:config:1.3.3
>>>>>> [info] 0.195 MB 0.114 MB 3 3 org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>>>>>> [info] 0.170 MB 0.167 MB 1 1 com.github.mpilquist:simulacrum_2.11:0.10.0
>>>>>> [info] 0.145 MB 0.145 MB 0 0 org.joda:joda-convert:2.1.2
>>>>>> [info] 0.041 MB 0.041 MB 0 0 org.slf4j:slf4j-api:1.7.25
>>>>>> [info] 0.033 MB 0.033 MB 0 0 com.google.code.findbugs:jsr305:1.3.9
>>>>>> [info] 0.016 MB 0.002 MB 2 1 org.typelevel:catalysts-platform_2.11:0.0.5
>>>>>> [info] 0.015 MB 0.012 MB 1 1 org.typelevel:catalysts-macros_2.11:0.0.5
>>>>>> [info] 0.015 MB 0.015 MB 0 0 org.scala-sbt:test-interface:1.0
>>>>>> [info] 0.007 MB 0.007 MB 0 0 org.apache.flink:force-shading:1.7.1
>>>>>> [info] 0.003 MB 0.003 MB 0 0 org.typelevel:macro-compat_2.11:1.1.1
>>>>>>
>>>>>>
>>>>>> And here is another view
>>>>>>
>>>>>> [info] lightbend:fdp-flink-taxiride_2.11:2.0.0 [S]
>>>>>> [info] +-lightbend:support_2.11:2.0.0 [S]
>>>>>> [info] | +-ch.qos.logback:logback-classic:1.2.3
>>>>>> [info] | | +-ch.qos.logback:logback-core:1.2.3
>>>>>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>>>>>> [info] | |
>>>>>> [info] | +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>>>>>> [info] | |
>>>>>> [info] | +-com.typesafe:config:1.3.3
>>>>>> [info] | +-joda-time:joda-time:2.10.1
>>>>>> [info] | +-org.joda:joda-convert:2.1.2
>>>>>> [info] | +-org.typelevel:cats_2.11:0.9.0 [S]
>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-free_2.11:0.9.0 [S]
>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | | | |
>>>>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | | | |
>>>>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | | | |
>>>>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>> [info] | | | | +-org.scala-sbt:test-interface:1.0
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>> [info] | | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] |
>>>>>> [info] +-org.apache.flink:flink-connector-kafka_2.11:1.7.1
>>>>>> [info] +-com.google.code.findbugs:jsr305:1.3.9
>>>>>> [info] +-org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>>>>>> [info] | +-com.google.code.findbugs:jsr305:1.3.9
>>>>>> [info] | +-org.apache.flink:force-shading:1.7.1
>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>>>> [info] |
>>>>>> [info] +-org.apache.flink:force-shading:1.7.1
>>>>>> [info] +-org.apache.kafka:kafka-clients:2.1.0
>>>>>> [info] | +-com.github.luben:zstd-jni:1.3.5-4
>>>>>> [info] | +-org.lz4:lz4-java:1.5.0
>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>>>> [info] | +-org.xerial.snappy:snappy-java:1.1.7.2
>>>>>> [info] |
>>>>>> [info] +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>>>> [info] +-org.slf4j:slf4j-api:1.7.25
>>>>>> [info]
>>>>>> [info] Done updating.
>>>>>> [info] lightbend:support_2.11:2.0.0 [S]
>>>>>> [info] +-ch.qos.logback:logback-classic:1.2.3
>>>>>> [info] | +-ch.qos.logback:logback-core:1.2.3
>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>>>> [info] |
>>>>>> [info] +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>>>> [info] |
>>>>>> [info] +-com.typesafe:config:1.3.3
>>>>>> [info] +-joda-time:joda-time:2.10.1
>>>>>> [info] +-org.joda:joda-convert:2.1.2
>>>>>> [info] +-org.typelevel:cats_2.11:0.9.0 [S]
>>>>>> [info] +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] |
>>>>>> [info] +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] |
>>>>>> [info] +-org.typelevel:cats-free_2.11:0.9.0 [S]
>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] |
>>>>>> [info] +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] |
>>>>>> [info] +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>> [info] | +-org.scala-sbt:test-interface:1.0
>>>>>> [info] |
>>>>>> [info] +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | | |
>>>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | | |
>>>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] |
>>>>>> [info] +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>>>> [info] | |
>>>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info] |
>>>>>> [info] +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>>>> [info] +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>>>> [info]
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Boris Lublinsky
>>>>>> FDP Architect
>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>> On Feb 22, 2019, at 12:33 AM, Tzu-Li (Gordon) Tai <gordon@ververica.com <ma...@ververica.com>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I haven't seen this problem for Flink 1.6.x / 1.7.x, so it shouldn't be a reoccurrence of FLINK-8741.
>>>>>>> I've double checked the used classloaders in the Kafka connector, they seem to be correct.
>>>>>>>
>>>>>>> The fact that it works correctly in IntelliJ, but not when packaged, suggests that there could be some conflicting dependencies in the packaged jar.
>>>>>>>
>>>>>>> Could you check the actual resolved dependency tree of the project, and see if there are multiple versions of some dependency related to Kafka being pulled in? For Maven for example, that would be "mvn dependency:tree".
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Gordon
>>>>>>>
>>>>>>> On Thu, Feb 21, 2019 at 10:49 PM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>> The relevant dependencies are
>>>>>>> val flinkScala = "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
>>>>>>> val flinkStreamingScala = "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
>>>>>>> val flinkKafka = "org.apache.flink" %% "flink-connector-kafka" % flinkVersion exclude("org.slf4j", "slf4j-log4j12")
>>>>>>> I am using SBT
>>>>>>> I tried both connector-kafka and connector-kaka-11 - same result
>>>>>>>
>>>>>>>
>>>>>>> Boris Lublinsky
>>>>>>> FDP Architect
>>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>>> On Feb 21, 2019, at 1:38 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>>>>>
>>>>>>>> Hi Boris,
>>>>>>>>
>>>>>>>> can you the relevant parts (dependencies) of your pom.xml? Did you also try without fixing the Kafka version, i.e. running with the Kafka client version provided by the Kafka connector of Flink? Gordon (cc) dealt with FLINK-8741.
>>>>>>>>
>>>>>>>> @Gordon: have you seen this issue with 1.6/1.7 before?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Konstantin
>>>>>>>>
>>>>>>>> On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>> I found some more details on this
>>>>>>>> The same error for the same application was reported about a year ago http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E>
>>>>>>>> And was due to https://issues.apache.org/jira/browse/FLINK-8741 <https://issues.apache.org/jira/browse/FLINK-8741>
>>>>>>>>
>>>>>>>> It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with both latest kaffka-connector
>>>>>>>> And Kafka-connector-011
>>>>>>>>
>>>>>>>> Boris Lublinsky
>>>>>>>> FDP Architect
>>>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>>>> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Boris,
>>>>>>>>>
>>>>>>>>> I haven’t seen this exact error, but I have seen similar errors caused by multiple versions of jars on the classpath.
>>>>>>>>>
>>>>>>>>> When I’ve run into this particular "XXX is not an instance of YYY" problem, it often seems to be caused by a jar that I should have marked as provided in my pom.
>>>>>>>>>
>>>>>>>>> Though I’m typically running on a YARN cluster, not w/K8s, so maybe this doesn’t apply.
>>>>>>>>>
>>>>>>>>> — Ken
>>>>>>>>>
>>>>>>>>> PS - I assume you’ve been reading https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>>>>
>>>>>>>>>> Konstantin,
>>>>>>>>>> After experimenting with this for a while, I got to the root cause of the problem
>>>>>>>>>> I am running a version of a Taxi ride travel prediction as my sample.
>>>>>>>>>> It works fine in Intellij,
>>>>>>>>>> But when I am trying to put it in the docker (standard Debian 1.7 image)
>>>>>>>>>> It fails with a following error
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The program finished with the following exception:
>>>>>>>>>>
>>>>>>>>>> org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>>>>>>>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>>>>>>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>>>>>>>>>> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>>>>>>>>>> at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>>>>>>>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>>>>>>>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>>>>>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>>>>>>>>>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>>>>>>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>>>>>>>>>> at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>>>>>>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>>>>>>>>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>>>>>>>>>> at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>>>>>>>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>>>>>>>>>> ... 19 more
>>>>>>>>>> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
>>>>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>>>>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>>>>>>>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>>>>>>>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>>>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>>>>>>>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>>>>>>>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>>>>>>>>>> at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>>>>>>>>>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>>>>>>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>>>>>>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>>>>>>>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>>>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>>>> Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.serialization.ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer
>>>>>>>>>> at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>>>>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>>>>>>>>>> ... 17 more
>>>>>>>>>>
>>>>>>>>>> The closest that I found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no <https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no>
>>>>>>>>>> Which talks about class loader. (I tried there solution, but it did not help)
>>>>>>>>>> I looked at the loading and I see that the pair of these 2 classes is loaded from my uber jar, but twice.
>>>>>>>>>>
>>>>>>>>>> Have you guys seen this error before?
>>>>>>>>>> Any suggestion?
>>>>>>>>>>
>>>>>>>>>> Boris Lublinsky
>>>>>>>>>> FDP Architect
>>>>>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>>>>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Boris,
>>>>>>>>>>>
>>>>>>>>>>> without looking at the entrypoint in much detail, generally there should not be a race condition there:
>>>>>>>>>>>
>>>>>>>>>>> * if the taskmanagers can not connect to the resourcemanager they will retry (per default the timeout is 5 mins)
>>>>>>>>>>> * if the JobManager does not get enough resources from the ResourceManager it will also wait for the resources/slots to provided. The timeout there is also 5 minutes, I think.
>>>>>>>>>>>
>>>>>>>>>>> So, this should actually be pretty robust as long as the Taskmanager containers can reach the Jobmanager eventually.
>>>>>>>>>>>
>>>>>>>>>>> Could you provide the Taskmanager/JobManager logs for such a failure case?
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Konstantin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>>>>> Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker <https://github.com/apache/flink/tree/release-1.7/flink-container/docker>
>>>>>>>>>>> I have created an entry point, which looks like follows:
>>>>>>>>>>> #!/bin/sh
>>>>>>>>>>>
>>>>>>>>>>> ################################################################################
>>>>>>>>>>> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh>
>>>>>>>>>>> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh <https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh>
>>>>>>>>>>> ################################################################################
>>>>>>>>>>>
>>>>>>>>>>> # If unspecified, the hostname of the container is taken as the JobManager address
>>>>>>>>>>> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>>>>>>>>>>
>>>>>>>>>>> drop_privs_cmd() {
>>>>>>>>>>> if [ $(id -u) != 0 ]; then
>>>>>>>>>>> # Don't need to drop privs if EUID != 0
>>>>>>>>>>> return
>>>>>>>>>>> elif [ -x /sbin/su-exec ]; then
>>>>>>>>>>> # Alpine
>>>>>>>>>>> echo su-exec flink
>>>>>>>>>>> else
>>>>>>>>>>> # Others
>>>>>>>>>>> echo gosu flink
>>>>>>>>>>> fi
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> JOB_MANAGER="jobmanager"
>>>>>>>>>>> TASK_MANAGER="taskmanager"
>>>>>>>>>>>
>>>>>>>>>>> CMD="$1"
>>>>>>>>>>> shift
>>>>>>>>>>>
>>>>>>>>>>> if [ "${CMD}" = "help" ]; then
>>>>>>>>>>> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
>>>>>>>>>>> exit 0
>>>>>>>>>>> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>>>>>>>> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>>>>>>>> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>>>>>>>>>>
>>>>>>>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>
>>>>>>>>>>> echo "Starting Task Manager"
>>>>>>>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>>>>>>>>>> else
>>>>>>>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>>>
>>>>>>>>>>> if [ -z "$1" ]; then
>>>>>>>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
>>>>>>>>>>> else
>>>>>>>>>>> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
>>>>>>>>>>> fi
>>>>>>>>>>> fi
>>>>>>>>>>> fi
>>>>>>>>>>>
>>>>>>>>>>> exec "$@"
>>>>>>>>>>> It does work for all the cases, except running standalone job.
>>>>>>>>>>> The problem, the way I understand it, is a racing condition.
>>>>>>>>>>> In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh
>>>>>>>>>>> tries to start a job immediately once the cluster is created (before connection is established).
>>>>>>>>>>> Is there a better option to implement it starting a job on container startup?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Konstantin Knauf | Solutions Architect
>>>>>>>>>>> +49 160 91394525
>>>>>>>>>
>>>>>>>>> --------------------------
>>>>>>>>> Ken Krugler
>>>>>>>>> +1 530-210-6378
>>>>>>>>> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
>>>>>>>>> Custom big data solutions & training
>>>>>>>>> Flink, Solr, Hadoop, Cascading & Cassandra
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Konstantin Knauf | Solutions Architect
>>>>>>>> +49 160 91394525
>>>>>>>> <https://www.ververica.com/>
>>>>>>>> Follow us @VervericaData
>>>>>>>> --
>>>>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference
>>>>>>>> Stream Processing | Event Driven | Real Time
>>>>>>>> --
>>>>>>>> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>>>>>> --
>>>>>>>> Data Artisans GmbH
>>>>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>>>>>> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>>>>>>>
>>>>>>
>>>>
>>>
>>
>
--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra
Re: Starting Flink cluster and running a job
Posted by Boris Lublinsky <bo...@lightbend.com>.
Any progress on this one?
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
> On Feb 24, 2019, at 5:16 PM, Boris Lublinsky <bo...@lightbend.com> wrote:
>
> And there are more explanations and concerns there
> It turns out that the execution depends on where my jar is located.
> If I put it in lib directory (I need this to be able to run bin/standalone-job.sh )
> It fails. With the same error.
>
>
> Currently I have the following in this directory
> fdp-flink-taxiride-assembly-2.0.0.jar flink-metrics-prometheus-1.7.2.jar flink-queryable-state-runtime_2.11-1.7.2.jar log4j-1.2.17.jar
> flink-dist_2.11-1.7.2.jar flink-python_2.11-1.7.2.jar flink-table_2.11-1.7.2.jar slf4j-log4j12-1.7.15.jar
>
> And no one of them seem to directly contain Kafka (except for my jar fdp-flink-taxiride-assembly-2.0.0.jar)
>
> But id I move my jar in any other directory it works fine.
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com <ma...@lightbend.com>
> https://www.lightbend.com/
>
>> On Feb 23, 2019, at 11:40 AM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>
>> And its working now.
>> Sorry about all confusion.
>> The root cause of the problem was that in “standard” Link image directory /opt/flink/log does not have write permissions and as a result the local cluster was never created
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>> On Feb 22, 2019, at 9:13 AM, Dawid Wysakowicz <dwysakowicz@apache.org <ma...@apache.org>> wrote:
>>>
>>> Hi,
>>>
>>> One additional question, how do you actually build the docker image? How do you put the user jar into the container? Maybe you added extra kafka connector to the cluster classpath? Have you checked what is on the classpath of a running taskmanager?
>>> Best,
>>>
>>> Dawid
>>> On 22/02/2019 15:44, Boris Lublinsky wrote:
>>>> Gordon, I double checked it several times. Here is the list
>>>>
>>>> Columns are
>>>> [info] - Jar-Size including dependencies
>>>> [info] - Jar-Size
>>>> [info] - Number of transitive dependencies
>>>> [info] - Number of direct dependencies
>>>> [info] - ModuleID
>>>> [info] Done updating.
>>>> [info] TotSize JarSize #TDe #Dep Module
>>>> [info] 27.988 MB ------- MB 33 2 lightbend:fdp-flink-taxiride_2.11:2.0.0
>>>> [info] 19.632 MB ------- MB 24 6 lightbend:support_2.11:2.0.0
>>>> [info] 17.675 MB 0.000 MB 16 9 org.typelevel:cats_2.11:0.9.0
>>>> [info] 17.389 MB 1.182 MB 13 9 org.typelevel:cats-laws_2.11:0.9.0
>>>> [info] 13.115 MB 0.286 MB 7 4 org.typelevel:cats-free_2.11:0.9.0
>>>> [info] 12.830 MB 0.000 MB 7 4 org.typelevel:cats-jvm_2.11:0.9.0
>>>> [info] 12.830 MB 3.403 MB 6 4 org.typelevel:cats-core_2.11:0.9.0
>>>> [info] 8.397 MB 0.068 MB 8 5 org.apache.flink:flink-connector-kafka_2.11:1.7.1
>>>> [info] 8.175 MB 1.889 MB 4 4 org.apache.kafka:kafka-clients:2.1.0
>>>> [info] 7.973 MB 0.212 MB 7 4 org.typelevel:cats-kernel-laws_2.11:0.9.0
>>>> [info] 4.834 MB 0.007 MB 4 2 org.typelevel:cats-macros_2.11:0.9.0
>>>> [info] 4.746 MB 0.082 MB 2 2 com.typesafe.scala-logging:scala-logging_2.11:3.9.0
>>>> [info] 4.658 MB 0.035 MB 1 1 org.typelevel:machinist_2.11:0.6.1
>>>> [info] 4.623 MB 4.623 MB 0 0 org.scala-lang:scala-reflect:2.11.12
>>>> [info] 4.592 MB 4.592 MB 0 0 org.typelevel:cats-kernel_2.11:0.9.0
>>>> [info] 3.714 MB 3.714 MB 0 0 com.github.luben:zstd-jni:1.3.5-4
>>>> [info] 3.152 MB 0.043 MB 2 1 org.typelevel:discipline_2.11:0.7.2
>>>> [info] 3.109 MB 3.094 MB 1 1 org.scalacheck:scalacheck_2.11:1.13.4
>>>> [info] 2.019 MB 2.019 MB 0 0 org.xerial.snappy:snappy-java:1.1.7.2
>>>> [info] 0.803 MB 0.290 MB 2 2 ch.qos.logback:logback-classic:1.2.3
>>>> [info] 0.641 MB 0.641 MB 0 0 joda-time:joda-time:2.10.1
>>>> [info] 0.512 MB 0.512 MB 0 0 org.lz4:lz4-java:1.5.0
>>>> [info] 0.472 MB 0.472 MB 0 0 ch.qos.logback:logback-core:1.2.3
>>>> [info] 0.286 MB 0.286 MB 0 0 com.typesafe:config:1.3.3
>>>> [info] 0.195 MB 0.114 MB 3 3 org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>>>> [info] 0.170 MB 0.167 MB 1 1 com.github.mpilquist:simulacrum_2.11:0.10.0
>>>> [info] 0.145 MB 0.145 MB 0 0 org.joda:joda-convert:2.1.2
>>>> [info] 0.041 MB 0.041 MB 0 0 org.slf4j:slf4j-api:1.7.25
>>>> [info] 0.033 MB 0.033 MB 0 0 com.google.code.findbugs:jsr305:1.3.9
>>>> [info] 0.016 MB 0.002 MB 2 1 org.typelevel:catalysts-platform_2.11:0.0.5
>>>> [info] 0.015 MB 0.012 MB 1 1 org.typelevel:catalysts-macros_2.11:0.0.5
>>>> [info] 0.015 MB 0.015 MB 0 0 org.scala-sbt:test-interface:1.0
>>>> [info] 0.007 MB 0.007 MB 0 0 org.apache.flink:force-shading:1.7.1
>>>> [info] 0.003 MB 0.003 MB 0 0 org.typelevel:macro-compat_2.11:1.1.1
>>>>
>>>>
>>>> And here is another view
>>>>
>>>> [info] lightbend:fdp-flink-taxiride_2.11:2.0.0 [S]
>>>> [info] +-lightbend:support_2.11:2.0.0 [S]
>>>> [info] | +-ch.qos.logback:logback-classic:1.2.3
>>>> [info] | | +-ch.qos.logback:logback-core:1.2.3
>>>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>>>> [info] | |
>>>> [info] | +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>>>> [info] | |
>>>> [info] | +-com.typesafe:config:1.3.3
>>>> [info] | +-joda-time:joda-time:2.10.1
>>>> [info] | +-org.joda:joda-convert:2.1.2
>>>> [info] | +-org.typelevel:cats_2.11:0.9.0 [S]
>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-free_2.11:0.9.0 [S]
>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | | | |
>>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | | | |
>>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | | | |
>>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>> [info] | | | | +-org.scala-sbt:test-interface:1.0
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>> [info] | | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] |
>>>> [info] +-org.apache.flink:flink-connector-kafka_2.11:1.7.1
>>>> [info] +-com.google.code.findbugs:jsr305:1.3.9
>>>> [info] +-org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>>>> [info] | +-com.google.code.findbugs:jsr305:1.3.9
>>>> [info] | +-org.apache.flink:force-shading:1.7.1
>>>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>> [info] |
>>>> [info] +-org.apache.flink:force-shading:1.7.1
>>>> [info] +-org.apache.kafka:kafka-clients:2.1.0
>>>> [info] | +-com.github.luben:zstd-jni:1.3.5-4
>>>> [info] | +-org.lz4:lz4-java:1.5.0
>>>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>> [info] | +-org.xerial.snappy:snappy-java:1.1.7.2
>>>> [info] |
>>>> [info] +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>>> [info] +-org.slf4j:slf4j-api:1.7.25
>>>> [info]
>>>> [info] Done updating.
>>>> [info] lightbend:support_2.11:2.0.0 [S]
>>>> [info] +-ch.qos.logback:logback-classic:1.2.3
>>>> [info] | +-ch.qos.logback:logback-core:1.2.3
>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>> [info] |
>>>> [info] +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>>> [info] |
>>>> [info] +-com.typesafe:config:1.3.3
>>>> [info] +-joda-time:joda-time:2.10.1
>>>> [info] +-org.joda:joda-convert:2.1.2
>>>> [info] +-org.typelevel:cats_2.11:0.9.0 [S]
>>>> [info] +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] |
>>>> [info] +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] |
>>>> [info] +-org.typelevel:cats-free_2.11:0.9.0 [S]
>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] |
>>>> [info] +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] |
>>>> [info] +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>> [info] | |
>>>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>> [info] | +-org.scala-sbt:test-interface:1.0
>>>> [info] |
>>>> [info] +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | |
>>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>> [info] | |
>>>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | | |
>>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>> [info] | |
>>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | | |
>>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>>> [info] | |
>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] |
>>>> [info] +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>>> [info] | |
>>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info] |
>>>> [info] +-org.typelevel:machinist_2.11:0.6.1 [S]
>>>> [info] +-org.scala-lang:scala-reflect:2.11.12 [S]
>>>> [info]
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Boris Lublinsky
>>>> FDP Architect
>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>> On Feb 22, 2019, at 12:33 AM, Tzu-Li (Gordon) Tai <gordon@ververica.com <ma...@ververica.com>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I haven't seen this problem for Flink 1.6.x / 1.7.x, so it shouldn't be a reoccurrence of FLINK-8741.
>>>>> I've double checked the used classloaders in the Kafka connector, they seem to be correct.
>>>>>
>>>>> The fact that it works correctly in IntelliJ, but not when packaged, suggests that there could be some conflicting dependencies in the packaged jar.
>>>>>
>>>>> Could you check the actual resolved dependency tree of the project, and see if there are multiple versions of some dependency related to Kafka being pulled in? For Maven for example, that would be "mvn dependency:tree".
>>>>>
>>>>> Cheers,
>>>>> Gordon
>>>>>
>>>>> On Thu, Feb 21, 2019 at 10:49 PM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>> The relevant dependencies are
>>>>> val flinkScala = "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
>>>>> val flinkStreamingScala = "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
>>>>> val flinkKafka = "org.apache.flink" %% "flink-connector-kafka" % flinkVersion exclude("org.slf4j", "slf4j-log4j12")
>>>>> I am using SBT
>>>>> I tried both connector-kafka and connector-kaka-11 - same result
>>>>>
>>>>>
>>>>> Boris Lublinsky
>>>>> FDP Architect
>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>> On Feb 21, 2019, at 1:38 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>>>
>>>>>> Hi Boris,
>>>>>>
>>>>>> can you the relevant parts (dependencies) of your pom.xml? Did you also try without fixing the Kafka version, i.e. running with the Kafka client version provided by the Kafka connector of Flink? Gordon (cc) dealt with FLINK-8741.
>>>>>>
>>>>>> @Gordon: have you seen this issue with 1.6/1.7 before?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Konstantin
>>>>>>
>>>>>> On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>> I found some more details on this
>>>>>> The same error for the same application was reported about a year ago http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E>
>>>>>> And was due to https://issues.apache.org/jira/browse/FLINK-8741 <https://issues.apache.org/jira/browse/FLINK-8741>
>>>>>>
>>>>>> It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with both latest kaffka-connector
>>>>>> And Kafka-connector-011
>>>>>>
>>>>>> Boris Lublinsky
>>>>>> FDP Architect
>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>> wrote:
>>>>>>>
>>>>>>> Hi Boris,
>>>>>>>
>>>>>>> I haven’t seen this exact error, but I have seen similar errors caused by multiple versions of jars on the classpath.
>>>>>>>
>>>>>>> When I’ve run into this particular "XXX is not an instance of YYY" problem, it often seems to be caused by a jar that I should have marked as provided in my pom.
>>>>>>>
>>>>>>> Though I’m typically running on a YARN cluster, not w/K8s, so maybe this doesn’t apply.
>>>>>>>
>>>>>>> — Ken
>>>>>>>
>>>>>>> PS - I assume you’ve been reading https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html>
>>>>>>>
>>>>>>>
>>>>>>>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>>
>>>>>>>> Konstantin,
>>>>>>>> After experimenting with this for a while, I got to the root cause of the problem
>>>>>>>> I am running a version of a Taxi ride travel prediction as my sample.
>>>>>>>> It works fine in Intellij,
>>>>>>>> But when I am trying to put it in the docker (standard Debian 1.7 image)
>>>>>>>> It fails with a following error
>>>>>>>>
>>>>>>>>
>>>>>>>> The program finished with the following exception:
>>>>>>>>
>>>>>>>> org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>>>>>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>>>>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>>>>>>>> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>>>>>>>> at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>>>>>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>>>>>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>>>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>>>>>>>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>>>>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>>>>>>>> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>>>>>>>> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>>>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>>>>>>>> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>>>>>>>> at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>>>>>>>> at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>>>>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>>>>>>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>>>>>>>> at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>>>>>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>>>>>>>> ... 19 more
>>>>>>>> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
>>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>>>>>>>> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>>>>>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>>>>>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>>>>>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>>>>>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>>>>>>>> at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>>>>>>>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>>>>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>>>>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>>>>>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>>> Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.serialization.ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer
>>>>>>>> at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>>>>>>>> ... 17 more
>>>>>>>>
>>>>>>>> The closest that I found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no <https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no>
>>>>>>>> Which talks about class loader. (I tried there solution, but it did not help)
>>>>>>>> I looked at the loading and I see that the pair of these 2 classes is loaded from my uber jar, but twice.
>>>>>>>>
>>>>>>>> Have you guys seen this error before?
>>>>>>>> Any suggestion?
>>>>>>>>
>>>>>>>> Boris Lublinsky
>>>>>>>> FDP Architect
>>>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Boris,
>>>>>>>>>
>>>>>>>>> without looking at the entrypoint in much detail, generally there should not be a race condition there:
>>>>>>>>>
>>>>>>>>> * if the taskmanagers can not connect to the resourcemanager they will retry (per default the timeout is 5 mins)
>>>>>>>>> * if the JobManager does not get enough resources from the ResourceManager it will also wait for the resources/slots to provided. The timeout there is also 5 minutes, I think.
>>>>>>>>>
>>>>>>>>> So, this should actually be pretty robust as long as the Taskmanager containers can reach the Jobmanager eventually.
>>>>>>>>>
>>>>>>>>> Could you provide the Taskmanager/JobManager logs for such a failure case?
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Konstantin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>>> Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker <https://github.com/apache/flink/tree/release-1.7/flink-container/docker>
>>>>>>>>> I have created an entry point, which looks like follows:
>>>>>>>>> #!/bin/sh
>>>>>>>>>
>>>>>>>>> ################################################################################
>>>>>>>>> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh>
>>>>>>>>> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh <https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh>
>>>>>>>>> ################################################################################
>>>>>>>>>
>>>>>>>>> # If unspecified, the hostname of the container is taken as the JobManager address
>>>>>>>>> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>>>>>>>>
>>>>>>>>> drop_privs_cmd() {
>>>>>>>>> if [ $(id -u) != 0 ]; then
>>>>>>>>> # Don't need to drop privs if EUID != 0
>>>>>>>>> return
>>>>>>>>> elif [ -x /sbin/su-exec ]; then
>>>>>>>>> # Alpine
>>>>>>>>> echo su-exec flink
>>>>>>>>> else
>>>>>>>>> # Others
>>>>>>>>> echo gosu flink
>>>>>>>>> fi
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> JOB_MANAGER="jobmanager"
>>>>>>>>> TASK_MANAGER="taskmanager"
>>>>>>>>>
>>>>>>>>> CMD="$1"
>>>>>>>>> shift
>>>>>>>>>
>>>>>>>>> if [ "${CMD}" = "help" ]; then
>>>>>>>>> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
>>>>>>>>> exit 0
>>>>>>>>> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>>>>>> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>>>>>> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>>>>>>>>
>>>>>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>
>>>>>>>>> echo "Starting Task Manager"
>>>>>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>>>>>>>> else
>>>>>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>>
>>>>>>>>> if [ -z "$1" ]; then
>>>>>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
>>>>>>>>> else
>>>>>>>>> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
>>>>>>>>> fi
>>>>>>>>> fi
>>>>>>>>> fi
>>>>>>>>>
>>>>>>>>> exec "$@"
>>>>>>>>> It does work for all the cases, except running standalone job.
>>>>>>>>> The problem, the way I understand it, is a racing condition.
>>>>>>>>> In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh
>>>>>>>>> tries to start a job immediately once the cluster is created (before connection is established).
>>>>>>>>> Is there a better option to implement it starting a job on container startup?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Konstantin Knauf | Solutions Architect
>>>>>>>>> +49 160 91394525
>>>>>>>
>>>>>>> --------------------------
>>>>>>> Ken Krugler
>>>>>>> +1 530-210-6378
>>>>>>> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
>>>>>>> Custom big data solutions & training
>>>>>>> Flink, Solr, Hadoop, Cascading & Cassandra
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Konstantin Knauf | Solutions Architect
>>>>>> +49 160 91394525
>>>>>> <https://www.ververica.com/>
>>>>>> Follow us @VervericaData
>>>>>> --
>>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference
>>>>>> Stream Processing | Event Driven | Real Time
>>>>>> --
>>>>>> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>>>> --
>>>>>> Data Artisans GmbH
>>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>>>> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>>>>>
>>>>
>>
>
Re: Starting Flink cluster and running a job
Posted by Boris Lublinsky <bo...@lightbend.com>.
And there are more explanations and concerns there
It turns out that the execution depends on where my jar is located.
If I put it in lib directory (I need this to be able to run bin/standalone-job.sh )
It fails. With the same error.
Currently I have the following in this directory
fdp-flink-taxiride-assembly-2.0.0.jar flink-metrics-prometheus-1.7.2.jar flink-queryable-state-runtime_2.11-1.7.2.jar log4j-1.2.17.jar
flink-dist_2.11-1.7.2.jar flink-python_2.11-1.7.2.jar flink-table_2.11-1.7.2.jar slf4j-log4j12-1.7.15.jar
And no one of them seem to directly contain Kafka (except for my jar fdp-flink-taxiride-assembly-2.0.0.jar)
But id I move my jar in any other directory it works fine.
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
> On Feb 23, 2019, at 11:40 AM, Boris Lublinsky <bo...@lightbend.com> wrote:
>
> And its working now.
> Sorry about all confusion.
> The root cause of the problem was that in “standard” Link image directory /opt/flink/log does not have write permissions and as a result the local cluster was never created
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com <ma...@lightbend.com>
> https://www.lightbend.com/
>
>> On Feb 22, 2019, at 9:13 AM, Dawid Wysakowicz <dwysakowicz@apache.org <ma...@apache.org>> wrote:
>>
>> Hi,
>>
>> One additional question, how do you actually build the docker image? How do you put the user jar into the container? Maybe you added extra kafka connector to the cluster classpath? Have you checked what is on the classpath of a running taskmanager?
>> Best,
>>
>> Dawid
>> On 22/02/2019 15:44, Boris Lublinsky wrote:
>>> Gordon, I double checked it several times. Here is the list
>>>
>>> Columns are
>>> [info] - Jar-Size including dependencies
>>> [info] - Jar-Size
>>> [info] - Number of transitive dependencies
>>> [info] - Number of direct dependencies
>>> [info] - ModuleID
>>> [info] Done updating.
>>> [info] TotSize JarSize #TDe #Dep Module
>>> [info] 27.988 MB ------- MB 33 2 lightbend:fdp-flink-taxiride_2.11:2.0.0
>>> [info] 19.632 MB ------- MB 24 6 lightbend:support_2.11:2.0.0
>>> [info] 17.675 MB 0.000 MB 16 9 org.typelevel:cats_2.11:0.9.0
>>> [info] 17.389 MB 1.182 MB 13 9 org.typelevel:cats-laws_2.11:0.9.0
>>> [info] 13.115 MB 0.286 MB 7 4 org.typelevel:cats-free_2.11:0.9.0
>>> [info] 12.830 MB 0.000 MB 7 4 org.typelevel:cats-jvm_2.11:0.9.0
>>> [info] 12.830 MB 3.403 MB 6 4 org.typelevel:cats-core_2.11:0.9.0
>>> [info] 8.397 MB 0.068 MB 8 5 org.apache.flink:flink-connector-kafka_2.11:1.7.1
>>> [info] 8.175 MB 1.889 MB 4 4 org.apache.kafka:kafka-clients:2.1.0
>>> [info] 7.973 MB 0.212 MB 7 4 org.typelevel:cats-kernel-laws_2.11:0.9.0
>>> [info] 4.834 MB 0.007 MB 4 2 org.typelevel:cats-macros_2.11:0.9.0
>>> [info] 4.746 MB 0.082 MB 2 2 com.typesafe.scala-logging:scala-logging_2.11:3.9.0
>>> [info] 4.658 MB 0.035 MB 1 1 org.typelevel:machinist_2.11:0.6.1
>>> [info] 4.623 MB 4.623 MB 0 0 org.scala-lang:scala-reflect:2.11.12
>>> [info] 4.592 MB 4.592 MB 0 0 org.typelevel:cats-kernel_2.11:0.9.0
>>> [info] 3.714 MB 3.714 MB 0 0 com.github.luben:zstd-jni:1.3.5-4
>>> [info] 3.152 MB 0.043 MB 2 1 org.typelevel:discipline_2.11:0.7.2
>>> [info] 3.109 MB 3.094 MB 1 1 org.scalacheck:scalacheck_2.11:1.13.4
>>> [info] 2.019 MB 2.019 MB 0 0 org.xerial.snappy:snappy-java:1.1.7.2
>>> [info] 0.803 MB 0.290 MB 2 2 ch.qos.logback:logback-classic:1.2.3
>>> [info] 0.641 MB 0.641 MB 0 0 joda-time:joda-time:2.10.1
>>> [info] 0.512 MB 0.512 MB 0 0 org.lz4:lz4-java:1.5.0
>>> [info] 0.472 MB 0.472 MB 0 0 ch.qos.logback:logback-core:1.2.3
>>> [info] 0.286 MB 0.286 MB 0 0 com.typesafe:config:1.3.3
>>> [info] 0.195 MB 0.114 MB 3 3 org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>>> [info] 0.170 MB 0.167 MB 1 1 com.github.mpilquist:simulacrum_2.11:0.10.0
>>> [info] 0.145 MB 0.145 MB 0 0 org.joda:joda-convert:2.1.2
>>> [info] 0.041 MB 0.041 MB 0 0 org.slf4j:slf4j-api:1.7.25
>>> [info] 0.033 MB 0.033 MB 0 0 com.google.code.findbugs:jsr305:1.3.9
>>> [info] 0.016 MB 0.002 MB 2 1 org.typelevel:catalysts-platform_2.11:0.0.5
>>> [info] 0.015 MB 0.012 MB 1 1 org.typelevel:catalysts-macros_2.11:0.0.5
>>> [info] 0.015 MB 0.015 MB 0 0 org.scala-sbt:test-interface:1.0
>>> [info] 0.007 MB 0.007 MB 0 0 org.apache.flink:force-shading:1.7.1
>>> [info] 0.003 MB 0.003 MB 0 0 org.typelevel:macro-compat_2.11:1.1.1
>>>
>>>
>>> And here is another view
>>>
>>> [info] lightbend:fdp-flink-taxiride_2.11:2.0.0 [S]
>>> [info] +-lightbend:support_2.11:2.0.0 [S]
>>> [info] | +-ch.qos.logback:logback-classic:1.2.3
>>> [info] | | +-ch.qos.logback:logback-core:1.2.3
>>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>>> [info] | |
>>> [info] | +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>>> [info] | |
>>> [info] | +-com.typesafe:config:1.3.3
>>> [info] | +-joda-time:joda-time:2.10.1
>>> [info] | +-org.joda:joda-convert:2.1.2
>>> [info] | +-org.typelevel:cats_2.11:0.9.0 [S]
>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:cats-free_2.11:0.9.0 [S]
>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | | | |
>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | | | |
>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>> [info] | | |
>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>> [info] | |
>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>> [info] | | |
>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | | | |
>>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>> [info] | | | | +-org.scala-sbt:test-interface:1.0
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>> [info] | | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>> [info] | | |
>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>> [info] | | |
>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] |
>>> [info] +-org.apache.flink:flink-connector-kafka_2.11:1.7.1
>>> [info] +-com.google.code.findbugs:jsr305:1.3.9
>>> [info] +-org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>>> [info] | +-com.google.code.findbugs:jsr305:1.3.9
>>> [info] | +-org.apache.flink:force-shading:1.7.1
>>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>> [info] |
>>> [info] +-org.apache.flink:force-shading:1.7.1
>>> [info] +-org.apache.kafka:kafka-clients:2.1.0
>>> [info] | +-com.github.luben:zstd-jni:1.3.5-4
>>> [info] | +-org.lz4:lz4-java:1.5.0
>>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>> [info] | +-org.xerial.snappy:snappy-java:1.1.7.2
>>> [info] |
>>> [info] +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>>> [info] +-org.slf4j:slf4j-api:1.7.25
>>> [info]
>>> [info] Done updating.
>>> [info] lightbend:support_2.11:2.0.0 [S]
>>> [info] +-ch.qos.logback:logback-classic:1.2.3
>>> [info] | +-ch.qos.logback:logback-core:1.2.3
>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>> [info] |
>>> [info] +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | +-org.slf4j:slf4j-api:1.7.25
>>> [info] |
>>> [info] +-com.typesafe:config:1.3.3
>>> [info] +-joda-time:joda-time:2.10.1
>>> [info] +-org.joda:joda-convert:2.1.2
>>> [info] +-org.typelevel:cats_2.11:0.9.0 [S]
>>> [info] +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] |
>>> [info] +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] |
>>> [info] +-org.typelevel:cats-free_2.11:0.9.0 [S]
>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] |
>>> [info] +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] |
>>> [info] +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>> [info] | |
>>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>> [info] | +-org.scala-sbt:test-interface:1.0
>>> [info] |
>>> [info] +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | |
>>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>> [info] | |
>>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | | |
>>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>> [info] | | | +-org.scala-sbt:test-interface:1.0
>>> [info] | | |
>>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>> [info] | |
>>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | | |
>>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>>> [info] | | +-org.scala-sbt:test-interface:1.0
>>> [info] | |
>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] |
>>> [info] +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>>> [info] | |
>>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info] |
>>> [info] +-org.typelevel:machinist_2.11:0.6.1 [S]
>>> [info] +-org.scala-lang:scala-reflect:2.11.12 [S]
>>> [info]
>>>
>>>
>>>
>>>
>>>
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>> On Feb 22, 2019, at 12:33 AM, Tzu-Li (Gordon) Tai <gordon@ververica.com <ma...@ververica.com>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I haven't seen this problem for Flink 1.6.x / 1.7.x, so it shouldn't be a reoccurrence of FLINK-8741.
>>>> I've double checked the used classloaders in the Kafka connector, they seem to be correct.
>>>>
>>>> The fact that it works correctly in IntelliJ, but not when packaged, suggests that there could be some conflicting dependencies in the packaged jar.
>>>>
>>>> Could you check the actual resolved dependency tree of the project, and see if there are multiple versions of some dependency related to Kafka being pulled in? For Maven for example, that would be "mvn dependency:tree".
>>>>
>>>> Cheers,
>>>> Gordon
>>>>
>>>> On Thu, Feb 21, 2019 at 10:49 PM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>> The relevant dependencies are
>>>> val flinkScala = "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
>>>> val flinkStreamingScala = "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
>>>> val flinkKafka = "org.apache.flink" %% "flink-connector-kafka" % flinkVersion exclude("org.slf4j", "slf4j-log4j12")
>>>> I am using SBT
>>>> I tried both connector-kafka and connector-kaka-11 - same result
>>>>
>>>>
>>>> Boris Lublinsky
>>>> FDP Architect
>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>> On Feb 21, 2019, at 1:38 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>>
>>>>> Hi Boris,
>>>>>
>>>>> can you the relevant parts (dependencies) of your pom.xml? Did you also try without fixing the Kafka version, i.e. running with the Kafka client version provided by the Kafka connector of Flink? Gordon (cc) dealt with FLINK-8741.
>>>>>
>>>>> @Gordon: have you seen this issue with 1.6/1.7 before?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Konstantin
>>>>>
>>>>> On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>> I found some more details on this
>>>>> The same error for the same application was reported about a year ago http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E>
>>>>> And was due to https://issues.apache.org/jira/browse/FLINK-8741 <https://issues.apache.org/jira/browse/FLINK-8741>
>>>>>
>>>>> It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with both latest kaffka-connector
>>>>> And Kafka-connector-011
>>>>>
>>>>> Boris Lublinsky
>>>>> FDP Architect
>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>> wrote:
>>>>>>
>>>>>> Hi Boris,
>>>>>>
>>>>>> I haven’t seen this exact error, but I have seen similar errors caused by multiple versions of jars on the classpath.
>>>>>>
>>>>>> When I’ve run into this particular "XXX is not an instance of YYY" problem, it often seems to be caused by a jar that I should have marked as provided in my pom.
>>>>>>
>>>>>> Though I’m typically running on a YARN cluster, not w/K8s, so maybe this doesn’t apply.
>>>>>>
>>>>>> — Ken
>>>>>>
>>>>>> PS - I assume you’ve been reading https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html>
>>>>>>
>>>>>>
>>>>>>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>
>>>>>>> Konstantin,
>>>>>>> After experimenting with this for a while, I got to the root cause of the problem
>>>>>>> I am running a version of a Taxi ride travel prediction as my sample.
>>>>>>> It works fine in Intellij,
>>>>>>> But when I am trying to put it in the docker (standard Debian 1.7 image)
>>>>>>> It fails with a following error
>>>>>>>
>>>>>>>
>>>>>>> The program finished with the following exception:
>>>>>>>
>>>>>>> org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>>>>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>>>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>>>>>>> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>>>>>>> at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>>>>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>>>>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>>>>>>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>>>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>>>>>>> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>>>>>>> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>>>>>>> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>>>>>>> at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>>>>>>> at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>>>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>>>>>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>>>>>>> at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>>>>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>>>>>>> ... 19 more
>>>>>>> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>>>>>>> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>>>>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>>>>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>>>>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>>>>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>>>>>>> at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>>>>>>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>>>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>>>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>>>>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>>> Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.serialization.ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer
>>>>>>> at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>>>>>>> ... 17 more
>>>>>>>
>>>>>>> The closest that I found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no <https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no>
>>>>>>> Which talks about class loader. (I tried there solution, but it did not help)
>>>>>>> I looked at the loading and I see that the pair of these 2 classes is loaded from my uber jar, but twice.
>>>>>>>
>>>>>>> Have you guys seen this error before?
>>>>>>> Any suggestion?
>>>>>>>
>>>>>>> Boris Lublinsky
>>>>>>> FDP Architect
>>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>>>>>
>>>>>>>> Hi Boris,
>>>>>>>>
>>>>>>>> without looking at the entrypoint in much detail, generally there should not be a race condition there:
>>>>>>>>
>>>>>>>> * if the taskmanagers can not connect to the resourcemanager they will retry (per default the timeout is 5 mins)
>>>>>>>> * if the JobManager does not get enough resources from the ResourceManager it will also wait for the resources/slots to provided. The timeout there is also 5 minutes, I think.
>>>>>>>>
>>>>>>>> So, this should actually be pretty robust as long as the Taskmanager containers can reach the Jobmanager eventually.
>>>>>>>>
>>>>>>>> Could you provide the Taskmanager/JobManager logs for such a failure case?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Konstantin
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>>> Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker <https://github.com/apache/flink/tree/release-1.7/flink-container/docker>
>>>>>>>> I have created an entry point, which looks like follows:
>>>>>>>> #!/bin/sh
>>>>>>>>
>>>>>>>> ################################################################################
>>>>>>>> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh>
>>>>>>>> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh <https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh>
>>>>>>>> ################################################################################
>>>>>>>>
>>>>>>>> # If unspecified, the hostname of the container is taken as the JobManager address
>>>>>>>> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>>>>>>>
>>>>>>>> drop_privs_cmd() {
>>>>>>>> if [ $(id -u) != 0 ]; then
>>>>>>>> # Don't need to drop privs if EUID != 0
>>>>>>>> return
>>>>>>>> elif [ -x /sbin/su-exec ]; then
>>>>>>>> # Alpine
>>>>>>>> echo su-exec flink
>>>>>>>> else
>>>>>>>> # Others
>>>>>>>> echo gosu flink
>>>>>>>> fi
>>>>>>>> }
>>>>>>>>
>>>>>>>> JOB_MANAGER="jobmanager"
>>>>>>>> TASK_MANAGER="taskmanager"
>>>>>>>>
>>>>>>>> CMD="$1"
>>>>>>>> shift
>>>>>>>>
>>>>>>>> if [ "${CMD}" = "help" ]; then
>>>>>>>> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
>>>>>>>> exit 0
>>>>>>>> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>>>>> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>>>>> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>>>>>>>
>>>>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>
>>>>>>>> echo "Starting Task Manager"
>>>>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>>>>>>> else
>>>>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>>
>>>>>>>> if [ -z "$1" ]; then
>>>>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
>>>>>>>> else
>>>>>>>> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
>>>>>>>> fi
>>>>>>>> fi
>>>>>>>> fi
>>>>>>>>
>>>>>>>> exec "$@"
>>>>>>>> It does work for all the cases, except running standalone job.
>>>>>>>> The problem, the way I understand it, is a racing condition.
>>>>>>>> In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh
>>>>>>>> tries to start a job immediately once the cluster is created (before connection is established).
>>>>>>>> Is there a better option to implement it starting a job on container startup?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Konstantin Knauf | Solutions Architect
>>>>>>>> +49 160 91394525
>>>>>>
>>>>>> --------------------------
>>>>>> Ken Krugler
>>>>>> +1 530-210-6378
>>>>>> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
>>>>>> Custom big data solutions & training
>>>>>> Flink, Solr, Hadoop, Cascading & Cassandra
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Konstantin Knauf | Solutions Architect
>>>>> +49 160 91394525
>>>>> <https://www.ververica.com/>
>>>>> Follow us @VervericaData
>>>>> --
>>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference
>>>>> Stream Processing | Event Driven | Real Time
>>>>> --
>>>>> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>>> --
>>>>> Data Artisans GmbH
>>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>>> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>>>>
>>>
>
Re: Starting Flink cluster and running a job
Posted by Boris Lublinsky <bo...@lightbend.com>.
And its working now.
Sorry about all confusion.
The root cause of the problem was that in “standard” Link image directory /opt/flink/log does not have write permissions and as a result the local cluster was never created
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
> On Feb 22, 2019, at 9:13 AM, Dawid Wysakowicz <dw...@apache.org> wrote:
>
> Hi,
>
> One additional question, how do you actually build the docker image? How do you put the user jar into the container? Maybe you added extra kafka connector to the cluster classpath? Have you checked what is on the classpath of a running taskmanager?
> Best,
>
> Dawid
> On 22/02/2019 15:44, Boris Lublinsky wrote:
>> Gordon, I double checked it several times. Here is the list
>>
>> Columns are
>> [info] - Jar-Size including dependencies
>> [info] - Jar-Size
>> [info] - Number of transitive dependencies
>> [info] - Number of direct dependencies
>> [info] - ModuleID
>> [info] Done updating.
>> [info] TotSize JarSize #TDe #Dep Module
>> [info] 27.988 MB ------- MB 33 2 lightbend:fdp-flink-taxiride_2.11:2.0.0
>> [info] 19.632 MB ------- MB 24 6 lightbend:support_2.11:2.0.0
>> [info] 17.675 MB 0.000 MB 16 9 org.typelevel:cats_2.11:0.9.0
>> [info] 17.389 MB 1.182 MB 13 9 org.typelevel:cats-laws_2.11:0.9.0
>> [info] 13.115 MB 0.286 MB 7 4 org.typelevel:cats-free_2.11:0.9.0
>> [info] 12.830 MB 0.000 MB 7 4 org.typelevel:cats-jvm_2.11:0.9.0
>> [info] 12.830 MB 3.403 MB 6 4 org.typelevel:cats-core_2.11:0.9.0
>> [info] 8.397 MB 0.068 MB 8 5 org.apache.flink:flink-connector-kafka_2.11:1.7.1
>> [info] 8.175 MB 1.889 MB 4 4 org.apache.kafka:kafka-clients:2.1.0
>> [info] 7.973 MB 0.212 MB 7 4 org.typelevel:cats-kernel-laws_2.11:0.9.0
>> [info] 4.834 MB 0.007 MB 4 2 org.typelevel:cats-macros_2.11:0.9.0
>> [info] 4.746 MB 0.082 MB 2 2 com.typesafe.scala-logging:scala-logging_2.11:3.9.0
>> [info] 4.658 MB 0.035 MB 1 1 org.typelevel:machinist_2.11:0.6.1
>> [info] 4.623 MB 4.623 MB 0 0 org.scala-lang:scala-reflect:2.11.12
>> [info] 4.592 MB 4.592 MB 0 0 org.typelevel:cats-kernel_2.11:0.9.0
>> [info] 3.714 MB 3.714 MB 0 0 com.github.luben:zstd-jni:1.3.5-4
>> [info] 3.152 MB 0.043 MB 2 1 org.typelevel:discipline_2.11:0.7.2
>> [info] 3.109 MB 3.094 MB 1 1 org.scalacheck:scalacheck_2.11:1.13.4
>> [info] 2.019 MB 2.019 MB 0 0 org.xerial.snappy:snappy-java:1.1.7.2
>> [info] 0.803 MB 0.290 MB 2 2 ch.qos.logback:logback-classic:1.2.3
>> [info] 0.641 MB 0.641 MB 0 0 joda-time:joda-time:2.10.1
>> [info] 0.512 MB 0.512 MB 0 0 org.lz4:lz4-java:1.5.0
>> [info] 0.472 MB 0.472 MB 0 0 ch.qos.logback:logback-core:1.2.3
>> [info] 0.286 MB 0.286 MB 0 0 com.typesafe:config:1.3.3
>> [info] 0.195 MB 0.114 MB 3 3 org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>> [info] 0.170 MB 0.167 MB 1 1 com.github.mpilquist:simulacrum_2.11:0.10.0
>> [info] 0.145 MB 0.145 MB 0 0 org.joda:joda-convert:2.1.2
>> [info] 0.041 MB 0.041 MB 0 0 org.slf4j:slf4j-api:1.7.25
>> [info] 0.033 MB 0.033 MB 0 0 com.google.code.findbugs:jsr305:1.3.9
>> [info] 0.016 MB 0.002 MB 2 1 org.typelevel:catalysts-platform_2.11:0.0.5
>> [info] 0.015 MB 0.012 MB 1 1 org.typelevel:catalysts-macros_2.11:0.0.5
>> [info] 0.015 MB 0.015 MB 0 0 org.scala-sbt:test-interface:1.0
>> [info] 0.007 MB 0.007 MB 0 0 org.apache.flink:force-shading:1.7.1
>> [info] 0.003 MB 0.003 MB 0 0 org.typelevel:macro-compat_2.11:1.1.1
>>
>>
>> And here is another view
>>
>> [info] lightbend:fdp-flink-taxiride_2.11:2.0.0 [S]
>> [info] +-lightbend:support_2.11:2.0.0 [S]
>> [info] | +-ch.qos.logback:logback-classic:1.2.3
>> [info] | | +-ch.qos.logback:logback-core:1.2.3
>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>> [info] | |
>> [info] | +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>> [info] | |
>> [info] | +-com.typesafe:config:1.3.3
>> [info] | +-joda-time:joda-time:2.10.1
>> [info] | +-org.joda:joda-convert:2.1.2
>> [info] | +-org.typelevel:cats_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-free_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | | |
>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | | |
>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | | |
>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | | |
>> [info] | | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.apache.flink:flink-connector-kafka_2.11:1.7.1
>> [info] +-com.google.code.findbugs:jsr305:1.3.9
>> [info] +-org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>> [info] | +-com.google.code.findbugs:jsr305:1.3.9
>> [info] | +-org.apache.flink:force-shading:1.7.1
>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] | +-org.slf4j:slf4j-api:1.7.25
>> [info] |
>> [info] +-org.apache.flink:force-shading:1.7.1
>> [info] +-org.apache.kafka:kafka-clients:2.1.0
>> [info] | +-com.github.luben:zstd-jni:1.3.5-4
>> [info] | +-org.lz4:lz4-java:1.5.0
>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] | +-org.slf4j:slf4j-api:1.7.25
>> [info] | +-org.xerial.snappy:snappy-java:1.1.7.2
>> [info] |
>> [info] +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] +-org.slf4j:slf4j-api:1.7.25
>> [info]
>> [info] Done updating.
>> [info] lightbend:support_2.11:2.0.0 [S]
>> [info] +-ch.qos.logback:logback-classic:1.2.3
>> [info] | +-ch.qos.logback:logback-core:1.2.3
>> [info] | +-org.slf4j:slf4j-api:1.7.25
>> [info] |
>> [info] +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | +-org.slf4j:slf4j-api:1.7.25
>> [info] |
>> [info] +-com.typesafe:config:1.3.3
>> [info] +-joda-time:joda-time:2.10.1
>> [info] +-org.joda:joda-convert:2.1.2
>> [info] +-org.typelevel:cats_2.11:0.9.0 [S]
>> [info] +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] |
>> [info] +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:cats-free_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | +-org.scala-sbt:test-interface:1.0
>> [info] |
>> [info] +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info]
>>
>>
>>
>>
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>> On Feb 22, 2019, at 12:33 AM, Tzu-Li (Gordon) Tai <gordon@ververica.com <ma...@ververica.com>> wrote:
>>>
>>> Hi,
>>>
>>> I haven't seen this problem for Flink 1.6.x / 1.7.x, so it shouldn't be a reoccurrence of FLINK-8741.
>>> I've double checked the used classloaders in the Kafka connector, they seem to be correct.
>>>
>>> The fact that it works correctly in IntelliJ, but not when packaged, suggests that there could be some conflicting dependencies in the packaged jar.
>>>
>>> Could you check the actual resolved dependency tree of the project, and see if there are multiple versions of some dependency related to Kafka being pulled in? For Maven for example, that would be "mvn dependency:tree".
>>>
>>> Cheers,
>>> Gordon
>>>
>>> On Thu, Feb 21, 2019 at 10:49 PM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>> The relevant dependencies are
>>> val flinkScala = "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
>>> val flinkStreamingScala = "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
>>> val flinkKafka = "org.apache.flink" %% "flink-connector-kafka" % flinkVersion exclude("org.slf4j", "slf4j-log4j12")
>>> I am using SBT
>>> I tried both connector-kafka and connector-kaka-11 - same result
>>>
>>>
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>> On Feb 21, 2019, at 1:38 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>
>>>> Hi Boris,
>>>>
>>>> can you the relevant parts (dependencies) of your pom.xml? Did you also try without fixing the Kafka version, i.e. running with the Kafka client version provided by the Kafka connector of Flink? Gordon (cc) dealt with FLINK-8741.
>>>>
>>>> @Gordon: have you seen this issue with 1.6/1.7 before?
>>>>
>>>> Cheers,
>>>>
>>>> Konstantin
>>>>
>>>> On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>> I found some more details on this
>>>> The same error for the same application was reported about a year ago http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E>
>>>> And was due to https://issues.apache.org/jira/browse/FLINK-8741 <https://issues.apache.org/jira/browse/FLINK-8741>
>>>>
>>>> It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with both latest kaffka-connector
>>>> And Kafka-connector-011
>>>>
>>>> Boris Lublinsky
>>>> FDP Architect
>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>> wrote:
>>>>>
>>>>> Hi Boris,
>>>>>
>>>>> I haven’t seen this exact error, but I have seen similar errors caused by multiple versions of jars on the classpath.
>>>>>
>>>>> When I’ve run into this particular "XXX is not an instance of YYY" problem, it often seems to be caused by a jar that I should have marked as provided in my pom.
>>>>>
>>>>> Though I’m typically running on a YARN cluster, not w/K8s, so maybe this doesn’t apply.
>>>>>
>>>>> — Ken
>>>>>
>>>>> PS - I assume you’ve been reading https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html>
>>>>>
>>>>>
>>>>>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>
>>>>>> Konstantin,
>>>>>> After experimenting with this for a while, I got to the root cause of the problem
>>>>>> I am running a version of a Taxi ride travel prediction as my sample.
>>>>>> It works fine in Intellij,
>>>>>> But when I am trying to put it in the docker (standard Debian 1.7 image)
>>>>>> It fails with a following error
>>>>>>
>>>>>>
>>>>>> The program finished with the following exception:
>>>>>>
>>>>>> org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>>>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>>>>>> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>>>>>> at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>>>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>>>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>>>>>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>>>>>> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>>>>>> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>>>>>> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>>>>>> at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>>>>>> at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>>>>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>>>>>> at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>>>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>>>>>> ... 19 more
>>>>>> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>>>>>> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>>>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>>>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>>>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>>>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>>>>>> at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>>>>>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>>>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>> Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.serialization.ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer
>>>>>> at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>>>>>> ... 17 more
>>>>>>
>>>>>> The closest that I found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no <https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no>
>>>>>> Which talks about class loader. (I tried there solution, but it did not help)
>>>>>> I looked at the loading and I see that the pair of these 2 classes is loaded from my uber jar, but twice.
>>>>>>
>>>>>> Have you guys seen this error before?
>>>>>> Any suggestion?
>>>>>>
>>>>>> Boris Lublinsky
>>>>>> FDP Architect
>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>>>>
>>>>>>> Hi Boris,
>>>>>>>
>>>>>>> without looking at the entrypoint in much detail, generally there should not be a race condition there:
>>>>>>>
>>>>>>> * if the taskmanagers can not connect to the resourcemanager they will retry (per default the timeout is 5 mins)
>>>>>>> * if the JobManager does not get enough resources from the ResourceManager it will also wait for the resources/slots to provided. The timeout there is also 5 minutes, I think.
>>>>>>>
>>>>>>> So, this should actually be pretty robust as long as the Taskmanager containers can reach the Jobmanager eventually.
>>>>>>>
>>>>>>> Could you provide the Taskmanager/JobManager logs for such a failure case?
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Konstantin
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>> Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker <https://github.com/apache/flink/tree/release-1.7/flink-container/docker>
>>>>>>> I have created an entry point, which looks like follows:
>>>>>>> #!/bin/sh
>>>>>>>
>>>>>>> ################################################################################
>>>>>>> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh>
>>>>>>> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh <https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh>
>>>>>>> ################################################################################
>>>>>>>
>>>>>>> # If unspecified, the hostname of the container is taken as the JobManager address
>>>>>>> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>>>>>>
>>>>>>> drop_privs_cmd() {
>>>>>>> if [ $(id -u) != 0 ]; then
>>>>>>> # Don't need to drop privs if EUID != 0
>>>>>>> return
>>>>>>> elif [ -x /sbin/su-exec ]; then
>>>>>>> # Alpine
>>>>>>> echo su-exec flink
>>>>>>> else
>>>>>>> # Others
>>>>>>> echo gosu flink
>>>>>>> fi
>>>>>>> }
>>>>>>>
>>>>>>> JOB_MANAGER="jobmanager"
>>>>>>> TASK_MANAGER="taskmanager"
>>>>>>>
>>>>>>> CMD="$1"
>>>>>>> shift
>>>>>>>
>>>>>>> if [ "${CMD}" = "help" ]; then
>>>>>>> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
>>>>>>> exit 0
>>>>>>> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>>>> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>>>> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>>>>>>
>>>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>
>>>>>>> echo "Starting Task Manager"
>>>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>>>>>> else
>>>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>
>>>>>>> if [ -z "$1" ]; then
>>>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
>>>>>>> else
>>>>>>> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
>>>>>>> fi
>>>>>>> fi
>>>>>>> fi
>>>>>>>
>>>>>>> exec "$@"
>>>>>>> It does work for all the cases, except running standalone job.
>>>>>>> The problem, the way I understand it, is a racing condition.
>>>>>>> In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh
>>>>>>> tries to start a job immediately once the cluster is created (before connection is established).
>>>>>>> Is there a better option to implement it starting a job on container startup?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Konstantin Knauf | Solutions Architect
>>>>>>> +49 160 91394525
>>>>>
>>>>> --------------------------
>>>>> Ken Krugler
>>>>> +1 530-210-6378
>>>>> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
>>>>> Custom big data solutions & training
>>>>> Flink, Solr, Hadoop, Cascading & Cassandra
>>>>
>>>>
>>>>
>>>> --
>>>> Konstantin Knauf | Solutions Architect
>>>> +49 160 91394525
>>>> <https://www.ververica.com/>
>>>> Follow us @VervericaData
>>>> --
>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference
>>>> Stream Processing | Event Driven | Real Time
>>>> --
>>>> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>> --
>>>> Data Artisans GmbH
>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>>>
>>
Re: Starting Flink cluster and running a job
Posted by Boris Lublinsky <bo...@lightbend.com>.
I am doing this using sbt native packager and sbt docker.
ef sbtdockerFlinkAppBase(id: String)(base: String = id) = projectBase(id)(base)
.enablePlugins(sbtdocker.DockerPlugin)
.settings(
dockerfile in docker := {
val artifact: File = assembly.value
// This location is fixed. the jar has to go there so that Flink can load it
val artifactTargetPath = s"/opt/flink/lib/${artifact.name}"
new Dockerfile {
from ("lightbend/flink:1.7.1-scala_2.11")
add(artifact, artifactTargetPath)
}
},
// Set name for the image
imageNames in docker := Seq(
ImageName(namespace = Some(organization.value),
repository = name.value.toLowerCase,
tag = Some(version.value))
),
buildOptions in docker := BuildOptions(cache = false)
)
It produce docker file as follows
FROM lightbend/flink:1.7.1-scala_2.11
ADD 0/fdp-flink-taxiride-assembly-2.0.0.jar /opt/flink/lib/fdp-flink-taxiride-assembly-2.0.0.jar
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
> On Feb 22, 2019, at 9:13 AM, Dawid Wysakowicz <dw...@apache.org> wrote:
>
> Hi,
>
> One additional question, how do you actually build the docker image? How do you put the user jar into the container? Maybe you added extra kafka connector to the cluster classpath? Have you checked what is on the classpath of a running taskmanager?
> Best,
>
> Dawid
> On 22/02/2019 15:44, Boris Lublinsky wrote:
>> Gordon, I double checked it several times. Here is the list
>>
>> Columns are
>> [info] - Jar-Size including dependencies
>> [info] - Jar-Size
>> [info] - Number of transitive dependencies
>> [info] - Number of direct dependencies
>> [info] - ModuleID
>> [info] Done updating.
>> [info] TotSize JarSize #TDe #Dep Module
>> [info] 27.988 MB ------- MB 33 2 lightbend:fdp-flink-taxiride_2.11:2.0.0
>> [info] 19.632 MB ------- MB 24 6 lightbend:support_2.11:2.0.0
>> [info] 17.675 MB 0.000 MB 16 9 org.typelevel:cats_2.11:0.9.0
>> [info] 17.389 MB 1.182 MB 13 9 org.typelevel:cats-laws_2.11:0.9.0
>> [info] 13.115 MB 0.286 MB 7 4 org.typelevel:cats-free_2.11:0.9.0
>> [info] 12.830 MB 0.000 MB 7 4 org.typelevel:cats-jvm_2.11:0.9.0
>> [info] 12.830 MB 3.403 MB 6 4 org.typelevel:cats-core_2.11:0.9.0
>> [info] 8.397 MB 0.068 MB 8 5 org.apache.flink:flink-connector-kafka_2.11:1.7.1
>> [info] 8.175 MB 1.889 MB 4 4 org.apache.kafka:kafka-clients:2.1.0
>> [info] 7.973 MB 0.212 MB 7 4 org.typelevel:cats-kernel-laws_2.11:0.9.0
>> [info] 4.834 MB 0.007 MB 4 2 org.typelevel:cats-macros_2.11:0.9.0
>> [info] 4.746 MB 0.082 MB 2 2 com.typesafe.scala-logging:scala-logging_2.11:3.9.0
>> [info] 4.658 MB 0.035 MB 1 1 org.typelevel:machinist_2.11:0.6.1
>> [info] 4.623 MB 4.623 MB 0 0 org.scala-lang:scala-reflect:2.11.12
>> [info] 4.592 MB 4.592 MB 0 0 org.typelevel:cats-kernel_2.11:0.9.0
>> [info] 3.714 MB 3.714 MB 0 0 com.github.luben:zstd-jni:1.3.5-4
>> [info] 3.152 MB 0.043 MB 2 1 org.typelevel:discipline_2.11:0.7.2
>> [info] 3.109 MB 3.094 MB 1 1 org.scalacheck:scalacheck_2.11:1.13.4
>> [info] 2.019 MB 2.019 MB 0 0 org.xerial.snappy:snappy-java:1.1.7.2
>> [info] 0.803 MB 0.290 MB 2 2 ch.qos.logback:logback-classic:1.2.3
>> [info] 0.641 MB 0.641 MB 0 0 joda-time:joda-time:2.10.1
>> [info] 0.512 MB 0.512 MB 0 0 org.lz4:lz4-java:1.5.0
>> [info] 0.472 MB 0.472 MB 0 0 ch.qos.logback:logback-core:1.2.3
>> [info] 0.286 MB 0.286 MB 0 0 com.typesafe:config:1.3.3
>> [info] 0.195 MB 0.114 MB 3 3 org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>> [info] 0.170 MB 0.167 MB 1 1 com.github.mpilquist:simulacrum_2.11:0.10.0
>> [info] 0.145 MB 0.145 MB 0 0 org.joda:joda-convert:2.1.2
>> [info] 0.041 MB 0.041 MB 0 0 org.slf4j:slf4j-api:1.7.25
>> [info] 0.033 MB 0.033 MB 0 0 com.google.code.findbugs:jsr305:1.3.9
>> [info] 0.016 MB 0.002 MB 2 1 org.typelevel:catalysts-platform_2.11:0.0.5
>> [info] 0.015 MB 0.012 MB 1 1 org.typelevel:catalysts-macros_2.11:0.0.5
>> [info] 0.015 MB 0.015 MB 0 0 org.scala-sbt:test-interface:1.0
>> [info] 0.007 MB 0.007 MB 0 0 org.apache.flink:force-shading:1.7.1
>> [info] 0.003 MB 0.003 MB 0 0 org.typelevel:macro-compat_2.11:1.1.1
>>
>>
>> And here is another view
>>
>> [info] lightbend:fdp-flink-taxiride_2.11:2.0.0 [S]
>> [info] +-lightbend:support_2.11:2.0.0 [S]
>> [info] | +-ch.qos.logback:logback-classic:1.2.3
>> [info] | | +-ch.qos.logback:logback-core:1.2.3
>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>> [info] | |
>> [info] | +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] | | +-org.slf4j:slf4j-api:1.7.25
>> [info] | |
>> [info] | +-com.typesafe:config:1.3.3
>> [info] | +-joda-time:joda-time:2.10.1
>> [info] | +-org.joda:joda-convert:2.1.2
>> [info] | +-org.typelevel:cats_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-free_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | | |
>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | | |
>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | | |
>> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | | |
>> [info] | | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.apache.flink:flink-connector-kafka_2.11:1.7.1
>> [info] +-com.google.code.findbugs:jsr305:1.3.9
>> [info] +-org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
>> [info] | +-com.google.code.findbugs:jsr305:1.3.9
>> [info] | +-org.apache.flink:force-shading:1.7.1
>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] | +-org.slf4j:slf4j-api:1.7.25
>> [info] |
>> [info] +-org.apache.flink:force-shading:1.7.1
>> [info] +-org.apache.kafka:kafka-clients:2.1.0
>> [info] | +-com.github.luben:zstd-jni:1.3.5-4
>> [info] | +-org.lz4:lz4-java:1.5.0
>> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] | +-org.slf4j:slf4j-api:1.7.25
>> [info] | +-org.xerial.snappy:snappy-java:1.1.7.2
>> [info] |
>> [info] +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
>> [info] +-org.slf4j:slf4j-api:1.7.25
>> [info]
>> [info] Done updating.
>> [info] lightbend:support_2.11:2.0.0 [S]
>> [info] +-ch.qos.logback:logback-classic:1.2.3
>> [info] | +-ch.qos.logback:logback-core:1.2.3
>> [info] | +-org.slf4j:slf4j-api:1.7.25
>> [info] |
>> [info] +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | +-org.slf4j:slf4j-api:1.7.25
>> [info] |
>> [info] +-com.typesafe:config:1.3.3
>> [info] +-joda-time:joda-time:2.10.1
>> [info] +-org.joda:joda-convert:2.1.2
>> [info] +-org.typelevel:cats_2.11:0.9.0 [S]
>> [info] +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] |
>> [info] +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:cats-free_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | +-org.scala-sbt:test-interface:1.0
>> [info] |
>> [info] +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] +-org.typelevel:cats-laws_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | | |
>> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | | +-org.scala-sbt:test-interface:1.0
>> [info] | | |
>> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
>> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | | |
>> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] | |
>> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
>> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
>> [info] | | +-org.scala-sbt:test-interface:1.0
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:cats-macros_2.11:0.9.0 [S]
>> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
>> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
>> [info] | |
>> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info] |
>> [info] +-org.typelevel:machinist_2.11:0.6.1 [S]
>> [info] +-org.scala-lang:scala-reflect:2.11.12 [S]
>> [info]
>>
>>
>>
>>
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>> On Feb 22, 2019, at 12:33 AM, Tzu-Li (Gordon) Tai <gordon@ververica.com <ma...@ververica.com>> wrote:
>>>
>>> Hi,
>>>
>>> I haven't seen this problem for Flink 1.6.x / 1.7.x, so it shouldn't be a reoccurrence of FLINK-8741.
>>> I've double checked the used classloaders in the Kafka connector, they seem to be correct.
>>>
>>> The fact that it works correctly in IntelliJ, but not when packaged, suggests that there could be some conflicting dependencies in the packaged jar.
>>>
>>> Could you check the actual resolved dependency tree of the project, and see if there are multiple versions of some dependency related to Kafka being pulled in? For Maven for example, that would be "mvn dependency:tree".
>>>
>>> Cheers,
>>> Gordon
>>>
>>> On Thu, Feb 21, 2019 at 10:49 PM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>> The relevant dependencies are
>>> val flinkScala = "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
>>> val flinkStreamingScala = "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
>>> val flinkKafka = "org.apache.flink" %% "flink-connector-kafka" % flinkVersion exclude("org.slf4j", "slf4j-log4j12")
>>> I am using SBT
>>> I tried both connector-kafka and connector-kaka-11 - same result
>>>
>>>
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>> On Feb 21, 2019, at 1:38 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>
>>>> Hi Boris,
>>>>
>>>> can you the relevant parts (dependencies) of your pom.xml? Did you also try without fixing the Kafka version, i.e. running with the Kafka client version provided by the Kafka connector of Flink? Gordon (cc) dealt with FLINK-8741.
>>>>
>>>> @Gordon: have you seen this issue with 1.6/1.7 before?
>>>>
>>>> Cheers,
>>>>
>>>> Konstantin
>>>>
>>>> On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>> I found some more details on this
>>>> The same error for the same application was reported about a year ago http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E>
>>>> And was due to https://issues.apache.org/jira/browse/FLINK-8741 <https://issues.apache.org/jira/browse/FLINK-8741>
>>>>
>>>> It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with both latest kaffka-connector
>>>> And Kafka-connector-011
>>>>
>>>> Boris Lublinsky
>>>> FDP Architect
>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>> wrote:
>>>>>
>>>>> Hi Boris,
>>>>>
>>>>> I haven’t seen this exact error, but I have seen similar errors caused by multiple versions of jars on the classpath.
>>>>>
>>>>> When I’ve run into this particular "XXX is not an instance of YYY" problem, it often seems to be caused by a jar that I should have marked as provided in my pom.
>>>>>
>>>>> Though I’m typically running on a YARN cluster, not w/K8s, so maybe this doesn’t apply.
>>>>>
>>>>> — Ken
>>>>>
>>>>> PS - I assume you’ve been reading https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html>
>>>>>
>>>>>
>>>>>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>
>>>>>> Konstantin,
>>>>>> After experimenting with this for a while, I got to the root cause of the problem
>>>>>> I am running a version of a Taxi ride travel prediction as my sample.
>>>>>> It works fine in Intellij,
>>>>>> But when I am trying to put it in the docker (standard Debian 1.7 image)
>>>>>> It fails with a following error
>>>>>>
>>>>>>
>>>>>> The program finished with the following exception:
>>>>>>
>>>>>> org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>>>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>>>>>> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>>>>>> at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>>>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>>>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>>>>>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>>>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>>>>>> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>>>>>> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>>>>>> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>>>>>> at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>>>>>> at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>>>>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>>>>>> at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>>>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>>>>>> ... 19 more
>>>>>> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>>>>>> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>>>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>>>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>>>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>>>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>>>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>>>>>> at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>>>>>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>>>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>>>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>>> Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.serialization.ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer
>>>>>> at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>>>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>>>>>> ... 17 more
>>>>>>
>>>>>> The closest that I found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no <https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no>
>>>>>> Which talks about class loader. (I tried there solution, but it did not help)
>>>>>> I looked at the loading and I see that the pair of these 2 classes is loaded from my uber jar, but twice.
>>>>>>
>>>>>> Have you guys seen this error before?
>>>>>> Any suggestion?
>>>>>>
>>>>>> Boris Lublinsky
>>>>>> FDP Architect
>>>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>>>>
>>>>>>> Hi Boris,
>>>>>>>
>>>>>>> without looking at the entrypoint in much detail, generally there should not be a race condition there:
>>>>>>>
>>>>>>> * if the taskmanagers can not connect to the resourcemanager they will retry (per default the timeout is 5 mins)
>>>>>>> * if the JobManager does not get enough resources from the ResourceManager it will also wait for the resources/slots to provided. The timeout there is also 5 minutes, I think.
>>>>>>>
>>>>>>> So, this should actually be pretty robust as long as the Taskmanager containers can reach the Jobmanager eventually.
>>>>>>>
>>>>>>> Could you provide the Taskmanager/JobManager logs for such a failure case?
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Konstantin
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>>>> Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker <https://github.com/apache/flink/tree/release-1.7/flink-container/docker>
>>>>>>> I have created an entry point, which looks like follows:
>>>>>>> #!/bin/sh
>>>>>>>
>>>>>>> ################################################################################
>>>>>>> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh>
>>>>>>> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh <https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh>
>>>>>>> ################################################################################
>>>>>>>
>>>>>>> # If unspecified, the hostname of the container is taken as the JobManager address
>>>>>>> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>>>>>>
>>>>>>> drop_privs_cmd() {
>>>>>>> if [ $(id -u) != 0 ]; then
>>>>>>> # Don't need to drop privs if EUID != 0
>>>>>>> return
>>>>>>> elif [ -x /sbin/su-exec ]; then
>>>>>>> # Alpine
>>>>>>> echo su-exec flink
>>>>>>> else
>>>>>>> # Others
>>>>>>> echo gosu flink
>>>>>>> fi
>>>>>>> }
>>>>>>>
>>>>>>> JOB_MANAGER="jobmanager"
>>>>>>> TASK_MANAGER="taskmanager"
>>>>>>>
>>>>>>> CMD="$1"
>>>>>>> shift
>>>>>>>
>>>>>>> if [ "${CMD}" = "help" ]; then
>>>>>>> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
>>>>>>> exit 0
>>>>>>> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>>>> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>>>> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>>>>>>
>>>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>
>>>>>>> echo "Starting Task Manager"
>>>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>>>>>> else
>>>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>>>
>>>>>>> if [ -z "$1" ]; then
>>>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
>>>>>>> else
>>>>>>> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
>>>>>>> fi
>>>>>>> fi
>>>>>>> fi
>>>>>>>
>>>>>>> exec "$@"
>>>>>>> It does work for all the cases, except running standalone job.
>>>>>>> The problem, the way I understand it, is a racing condition.
>>>>>>> In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh
>>>>>>> tries to start a job immediately once the cluster is created (before connection is established).
>>>>>>> Is there a better option to implement it starting a job on container startup?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Konstantin Knauf | Solutions Architect
>>>>>>> +49 160 91394525
>>>>>
>>>>> --------------------------
>>>>> Ken Krugler
>>>>> +1 530-210-6378
>>>>> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
>>>>> Custom big data solutions & training
>>>>> Flink, Solr, Hadoop, Cascading & Cassandra
>>>>
>>>>
>>>>
>>>> --
>>>> Konstantin Knauf | Solutions Architect
>>>> +49 160 91394525
>>>> <https://www.ververica.com/>
>>>> Follow us @VervericaData
>>>> --
>>>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference
>>>> Stream Processing | Event Driven | Real Time
>>>> --
>>>> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>>> --
>>>> Data Artisans GmbH
>>>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>>>> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>>>
>>
Re: Starting Flink cluster and running a job
Posted by Dawid Wysakowicz <dw...@apache.org>.
Hi,
One additional question, how do you actually build the docker image? How
do you put the user jar into the container? Maybe you added extra kafka
connector to the cluster classpath? Have you checked what is on the
classpath of a running taskmanager?
Best,
Dawid
On 22/02/2019 15:44, Boris Lublinsky wrote:
> Gordon, I double checked it several times. Here is the list
>
> Columns are
> [info] - Jar-Size including dependencies
> [info] - Jar-Size
> [info] - Number of transitive dependencies
> [info] - Number of direct dependencies
> [info] - ModuleID
> [info] Done updating.
> [info] TotSize JarSize #TDe #Dep Module
> [info] 27.988 MB ------- MB 33 2
> lightbend:fdp-flink-taxiride_2.11:2.0.0
> [info] 19.632 MB ------- MB 24 6 lightbend:support_2.11:2.0.0
> [info] 17.675 MB 0.000 MB 16 9 org.typelevel:cats_2.11:0.9.0
> [info] 17.389 MB 1.182 MB 13 9 org.typelevel:cats-laws_2.11:0.9.0
> [info] 13.115 MB 0.286 MB 7 4 org.typelevel:cats-free_2.11:0.9.0
> [info] 12.830 MB 0.000 MB 7 4 org.typelevel:cats-jvm_2.11:0.9.0
> [info] 12.830 MB 3.403 MB 6 4 org.typelevel:cats-core_2.11:0.9.0
> [info] 8.397 MB 0.068 MB 8 5
> org.apache.flink:flink-connector-kafka_2.11:1.7.1
> [info] 8.175 MB 1.889 MB 4 4
> org.apache.kafka:kafka-clients:2.1.0
> [info] 7.973 MB 0.212 MB 7 4
> org.typelevel:cats-kernel-laws_2.11:0.9.0
> [info] 4.834 MB 0.007 MB 4 2
> org.typelevel:cats-macros_2.11:0.9.0
> [info] 4.746 MB 0.082 MB 2 2
> com.typesafe.scala-logging:scala-logging_2.11:3.9.0
> [info] 4.658 MB 0.035 MB 1 1 org.typelevel:machinist_2.11:0.6.1
> [info] 4.623 MB 4.623 MB 0 0
> org.scala-lang:scala-reflect:2.11.12
> [info] 4.592 MB 4.592 MB 0 0
> org.typelevel:cats-kernel_2.11:0.9.0
> [info] 3.714 MB 3.714 MB 0 0 com.github.luben:zstd-jni:1.3.5-4
> [info] 3.152 MB 0.043 MB 2 1 org.typelevel:discipline_2.11:0.7.2
> [info] 3.109 MB 3.094 MB 1 1
> org.scalacheck:scalacheck_2.11:1.13.4
> [info] 2.019 MB 2.019 MB 0 0
> org.xerial.snappy:snappy-java:1.1.7.2
> [info] 0.803 MB 0.290 MB 2 2
> ch.qos.logback:logback-classic:1.2.3
> [info] 0.641 MB 0.641 MB 0 0 joda-time:joda-time:2.10.1
> [info] 0.512 MB 0.512 MB 0 0 org.lz4:lz4-java:1.5.0
> [info] 0.472 MB 0.472 MB 0 0 ch.qos.logback:logback-core:1.2.3
> [info] 0.286 MB 0.286 MB 0 0 com.typesafe:config:1.3.3
> [info] 0.195 MB 0.114 MB 3 3
> org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
> [info] 0.170 MB 0.167 MB 1 1
> com.github.mpilquist:simulacrum_2.11:0.10.0
> [info] 0.145 MB 0.145 MB 0 0 org.joda:joda-convert:2.1.2
> [info] 0.041 MB 0.041 MB 0 0 org.slf4j:slf4j-api:1.7.25
> [info] 0.033 MB 0.033 MB 0 0
> com.google.code.findbugs:jsr305:1.3.9
> [info] 0.016 MB 0.002 MB 2 1
> org.typelevel:catalysts-platform_2.11:0.0.5
> [info] 0.015 MB 0.012 MB 1 1
> org.typelevel:catalysts-macros_2.11:0.0.5
> [info] 0.015 MB 0.015 MB 0 0 org.scala-sbt:test-interface:1.0
> [info] 0.007 MB 0.007 MB 0 0
> org.apache.flink:force-shading:1.7.1
> [info] 0.003 MB 0.003 MB 0 0
> org.typelevel:macro-compat_2.11:1.1.1
>
>
> And here is another view
>
> [info] lightbend:fdp-flink-taxiride_2.11:2.0.0 [S]
> [info] +-lightbend:support_2.11:2.0.0 [S]
> [info] | +-ch.qos.logback:logback-classic:1.2.3
> [info] | | +-ch.qos.logback:logback-core:1.2.3
> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info] | | +-org.slf4j:slf4j-api:1.7.25
> [info] | |
> [info] | +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info] | | +-org.slf4j:slf4j-api:1.7.25
> [info] | |
> [info] | +-com.typesafe:config:1.3.3
> [info] | +-joda-time:joda-time:2.10.1
> [info] | +-org.joda:joda-convert:2.1.2
> [info] | +-org.typelevel:cats_2.11:0.9.0 [S]
> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:cats-free_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | | |
> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | | |
> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | | +-org.scala-sbt:test-interface:1.0
> [info] | | |
> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | +-org.scala-sbt:test-interface:1.0
> [info] | |
> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | +-org.typelevel:cats-laws_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | | +-org.scala-sbt:test-interface:1.0
> [info] | | |
> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | | |
> [info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | | | +-org.scala-sbt:test-interface:1.0
> [info] | | | |
> [info] | | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info] | | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | | +-org.scala-sbt:test-interface:1.0
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | | +-org.scala-sbt:test-interface:1.0
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] |
> [info] +-org.apache.flink:flink-connector-kafka_2.11:1.7.1
> [info] +-com.google.code.findbugs:jsr305:1.3.9
> [info] +-org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
> [info] | +-com.google.code.findbugs:jsr305:1.3.9
> [info] | +-org.apache.flink:force-shading:1.7.1
> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info] | +-org.slf4j:slf4j-api:1.7.25
> [info] |
> [info] +-org.apache.flink:force-shading:1.7.1
> [info] +-org.apache.kafka:kafka-clients:2.1.0
> [info] | +-com.github.luben:zstd-jni:1.3.5-4
> [info] | +-org.lz4:lz4-java:1.5.0
> [info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info] | +-org.slf4j:slf4j-api:1.7.25
> [info] | +-org.xerial.snappy:snappy-java:1.1.7.2
> [info] |
> [info] +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
> [info] +-org.slf4j:slf4j-api:1.7.25
> [info]
> [info] Done updating.
> [info] lightbend:support_2.11:2.0.0 [S]
> [info] +-ch.qos.logback:logback-classic:1.2.3
> [info] | +-ch.qos.logback:logback-core:1.2.3
> [info] | +-org.slf4j:slf4j-api:1.7.25
> [info] |
> [info] +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | +-org.slf4j:slf4j-api:1.7.25
> [info] |
> [info] +-com.typesafe:config:1.3.3
> [info] +-joda-time:joda-time:2.10.1
> [info] +-org.joda:joda-convert:2.1.2
> [info] +-org.typelevel:cats_2.11:0.9.0 [S]
> [info] +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] |
> [info] +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] |
> [info] +-org.typelevel:cats-free_2.11:0.9.0 [S]
> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] |
> [info] +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] |
> [info] +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | +-org.scala-sbt:test-interface:1.0
> [info] | |
> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | +-org.scala-sbt:test-interface:1.0
> [info] |
> [info] +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] +-org.typelevel:cats-laws_2.11:0.9.0 [S]
> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | +-org.scala-sbt:test-interface:1.0
> [info] | |
> [info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | | |
> [info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | | +-org.scala-sbt:test-interface:1.0
> [info] | | |
> [info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
> [info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | +-org.scala-sbt:test-interface:1.0
> [info] | |
> [info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
> [info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | | |
> [info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] | |
> [info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
> [info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
> [info] | | +-org.scala-sbt:test-interface:1.0
> [info] | |
> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] |
> [info] +-org.typelevel:cats-macros_2.11:0.9.0 [S]
> [info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
> [info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
> [info] | |
> [info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info] |
> [info] +-org.typelevel:machinist_2.11:0.6.1 [S]
> [info] +-org.scala-lang:scala-reflect:2.11.12 [S]
> [info]
>
>
>
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com <ma...@lightbend.com>
> https://www.lightbend.com/
>
>> On Feb 22, 2019, at 12:33 AM, Tzu-Li (Gordon) Tai
>> <gordon@ververica.com <ma...@ververica.com>> wrote:
>>
>> Hi,
>>
>> I haven't seen this problem for Flink 1.6.x / 1.7.x, so it shouldn't
>> be a reoccurrence of FLINK-8741.
>> I've double checked the used classloaders in the Kafka connector,
>> they seem to be correct.
>>
>> The fact that it works correctly in IntelliJ, but not when packaged,
>> suggests that there could be some conflicting dependencies in the
>> packaged jar.
>>
>> Could you check the actual resolved dependency tree of the project,
>> and see if there are multiple versions of some dependency related to
>> Kafka being pulled in? For Maven for example, that would be "mvn
>> dependency:tree".
>>
>> Cheers,
>> Gordon
>>
>> On Thu, Feb 21, 2019 at 10:49 PM Boris Lublinsky
>> <boris.lublinsky@lightbend.com
>> <ma...@lightbend.com>> wrote:
>>
>> The relevant dependencies are
>>
>> val flinkScala = "org.apache.flink" %% "flink-scala" % flinkVersion % "provided" val flinkStreamingScala = "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided" val flinkKafka = "org.apache.flink" %% "flink-connector-kafka" % flinkVersion exclude("org.slf4j", "slf4j-log4j12")
>>
>> I am using SBT
>> I tried both connector-kafka and connector-kaka-11 - same result
>>
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>> https://www.lightbend.com/
>>
>>> On Feb 21, 2019, at 1:38 AM, Konstantin Knauf
>>> <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>
>>> Hi Boris,
>>>
>>> can you the relevant parts (dependencies) of your pom.xml? Did
>>> you also try without fixing the Kafka version, i.e. running with
>>> the Kafka client version provided by the Kafka connector of
>>> Flink? Gordon (cc) dealt with FLINK-8741.
>>>
>>> @Gordon: have you seen this issue with 1.6/1.7 before?
>>>
>>> Cheers,
>>>
>>> Konstantin
>>>
>>> On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky
>>> <boris.lublinsky@lightbend.com
>>> <ma...@lightbend.com>> wrote:
>>>
>>> I found some more details on this
>>> The same error for the same application was reported about a
>>> year
>>> ago http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E
>>> And was due to https://issues.apache.org/jira/browse/FLINK-8741
>>>
>>> It looks like the same issue is back in 1.7.1 and 1.6.3. I
>>> tried with both latest kaffka-connector
>>> And Kafka-connector-011
>>>
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com
>>> <ma...@lightbend.com>
>>> https://www.lightbend.com/
>>>
>>>> On Feb 19, 2019, at 7:02 PM, Ken Krugler
>>>> <kkrugler_lists@transpac.com
>>>> <ma...@transpac.com>> wrote:
>>>>
>>>> Hi Boris,
>>>>
>>>> I haven’t seen this exact error, but I have seen similar
>>>> errors caused by multiple versions of jars on the classpath.
>>>>
>>>> When I’ve run into this particular "XXX is not an instance
>>>> of YYY" problem, it often seems to be caused by a jar that
>>>> I should have marked as provided in my pom.
>>>>
>>>> Though I’m typically running on a YARN cluster, not w/K8s,
>>>> so maybe this doesn’t apply.
>>>>
>>>> — Ken
>>>>
>>>> PS - I assume you’ve been
>>>> reading https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html
>>>>
>>>>
>>>>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky
>>>>> <boris.lublinsky@lightbend.com
>>>>> <ma...@lightbend.com>> wrote:
>>>>>
>>>>> Konstantin,
>>>>> After experimenting with this for a while, I got to the
>>>>> root cause of the problem
>>>>> I am running a version of a Taxi ride travel prediction as
>>>>> my sample.
>>>>> It works fine in Intellij,
>>>>> But when I am trying to put it in the docker (standard
>>>>> Debian 1.7 image)
>>>>> It fails with a following error
>>>>>
>>>>>
>>>>> The program finished with the following exception:
>>>>>
>>>>> org.apache.flink.client.program.ProgramInvocationException:
>>>>> Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>>>>> at
>>>>> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>>>>> at
>>>>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>>>>> at
>>>>> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>>>>> at
>>>>> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>>>>> at
>>>>> com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>>>>> at
>>>>> com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>>> at
>>>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>>>>> at
>>>>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>>>>> at
>>>>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>>>>> at
>>>>> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>>>> at
>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>>>>> Caused by:
>>>>> org.apache.flink.runtime.client.JobExecutionException: Job
>>>>> execution failed.
>>>>> at
>>>>> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>>>>> at
>>>>> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>>>>> ... 19 more
>>>>> Caused by: org.apache.kafka.common.KafkaException: Failed
>>>>> to construct kafka producer
>>>>> at
>>>>> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>>>>> at
>>>>> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>>>>> at
>>>>> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>>>>> at
>>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>>>>> at
>>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>>>>> at
>>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>>>>> at
>>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>>>>> at
>>>>> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>>>>> at
>>>>> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>>>>> at
>>>>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>>>>> at
>>>>> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>>>>> at
>>>>> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>>>>> at
>>>>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>>>>> at
>>>>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>>>>> at
>>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>>>>> at
>>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>>>>> at
>>>>> org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>>>>> at java.lang.Thread.run(Thread.java:748)
>>>>> Caused by: org.apache.kafka.common.KafkaException:
>>>>> org.apache.kafka.common.serialization.ByteArraySerializer
>>>>> is not an instance of
>>>>> org.apache.kafka.common.serialization.Serializer
>>>>> at
>>>>> org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>>>>> at
>>>>> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>>>>> ... 17 more
>>>>>
>>>>> The closest that I
>>>>> found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no
>>>>> Which talks about class loader. (I tried there solution,
>>>>> but it did not help)
>>>>> I looked at the loading and I see that the pair of these 2
>>>>> classes is loaded from my uber jar, but twice.
>>>>>
>>>>> Have you guys seen this error before?
>>>>> Any suggestion?
>>>>>
>>>>> Boris Lublinsky
>>>>> FDP Architect
>>>>> boris.lublinsky@lightbend.com
>>>>> <ma...@lightbend.com>
>>>>> https://www.lightbend.com/
>>>>>
>>>>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf
>>>>>> <konstantin@ververica.com
>>>>>> <ma...@ververica.com>> wrote:
>>>>>>
>>>>>> Hi Boris,
>>>>>>
>>>>>> without looking at the entrypoint in much detail,
>>>>>> generally there should not be a race condition there:
>>>>>>
>>>>>> * if the taskmanagers can not connect to the
>>>>>> resourcemanager they will retry (per default the timeout
>>>>>> is 5 mins)
>>>>>> * if the JobManager does not get enough resources from
>>>>>> the ResourceManager it will also wait for the
>>>>>> resources/slots to provided. The timeout there is also 5
>>>>>> minutes, I think.
>>>>>>
>>>>>> So, this should actually be pretty robust as long as the
>>>>>> Taskmanager containers can reach the Jobmanager eventually.
>>>>>>
>>>>>> Could you provide the Taskmanager/JobManager logs for
>>>>>> such a failure case?
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Konstantin
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky
>>>>>> <boris.lublinsky@lightbend.com
>>>>>> <ma...@lightbend.com>> wrote:
>>>>>>
>>>>>> Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker
>>>>>>
>>>>>> I have created an entry point, which looks like follows:
>>>>>>
>>>>>> #!/bin/sh
>>>>>> ################################################################################
>>>>>> # from
>>>>>> https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh
>>>>>> # and
>>>>>> https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh
>>>>>> ################################################################################
>>>>>> # If unspecified, the hostname of the container is
>>>>>> taken as the JobManager address JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>>>>>
>>>>>> drop_privs_cmd() {
>>>>>> if [ $(id -u) != 0]; then # Don't need to drop privs if EUID != 0 return
>>>>>> elif [ -x /sbin/su-exec]; then # Alpine echo su-exec flink
>>>>>> else # Others echo gosu flink
>>>>>> fi }
>>>>>>
>>>>>> JOB_MANAGER="jobmanager" TASK_MANAGER="taskmanager" CMD="$1" shift if [ "${CMD}" = "help"]; then echo "Usage: $(basename $0)
>>>>>> (${JOB_MANAGER}|${TASK_MANAGER}|help)" exit 0 elif [
>>>>>> "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" =
>>>>>> "${TASK_MANAGER}"]; then if [ "${CMD}" = "${TASK_MANAGER}"]; then TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>>>>>
>>>>>> sed -i -e "s/jobmanager.rpc.address:
>>>>>> localhost/jobmanager.rpc.address:
>>>>>> ${JOB_MANAGER_RPC_ADDRESS}/g"
>>>>>> "$FLINK_HOME/conf/flink-conf.yaml" sed -i -e "s/taskmanager.numberOfTaskSlots:
>>>>>> 1/taskmanager.numberOfTaskSlots:
>>>>>> $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g"
>>>>>> "$FLINK_HOME/conf/flink-conf.yaml" echo
>>>>>> "blob.server.port: 6124" >>
>>>>>> "$FLINK_HOME/conf/flink-conf.yaml" echo
>>>>>> "query.server.port: 6125" >>
>>>>>> "$FLINK_HOME/conf/flink-conf.yaml" echo "Starting
>>>>>> Task Manager" echo "config file: " && grep '^[^\n#]'
>>>>>> "$FLINK_HOME/conf/flink-conf.yaml" exec
>>>>>> $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>>>>> else sed -i -e "s/jobmanager.rpc.address:
>>>>>> localhost/jobmanager.rpc.address:
>>>>>> ${JOB_MANAGER_RPC_ADDRESS}/g"
>>>>>> "$FLINK_HOME/conf/flink-conf.yaml" echo
>>>>>> "blob.server.port: 6124" >>
>>>>>> "$FLINK_HOME/conf/flink-conf.yaml" echo
>>>>>> "query.server.port: 6125" >>
>>>>>> "$FLINK_HOME/conf/flink-conf.yaml" echo "config file:
>>>>>> " && grep '^[^\n#]'
>>>>>> "$FLINK_HOME/conf/flink-conf.yaml" if [ -z "$1"]; then exec $(drop_privs_cmd)
>>>>>> "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@" else exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@" fi fi fi exec "$@"
>>>>>>
>>>>>> It does work for all the cases, except running
>>>>>> standalone job.
>>>>>> The problem, the way I understand it, is a racing
>>>>>> condition.
>>>>>> In kubernetes it takes several attempts for establish
>>>>>> connection between Job and Task manager,
>>>>>> while standalone-job.sh
>>>>>> tries to start a job immediately once the cluster is
>>>>>> created (before connection is established).
>>>>>> Is there a better option to implement it starting a
>>>>>> job on container startup?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Konstantin Knauf| Solutions Architect
>>>>>> +49 160 91394525
>>>>
>>>> --------------------------
>>>> Ken Krugler
>>>> +1 530-210-6378
>>>> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
>>>> Custom big data solutions & training
>>>> Flink, Solr, Hadoop, Cascading & Cassandra
>>>
>>>
>>>
>>> --
>>> Konstantin Knauf| Solutions Architect
>>> +49 160 91394525
>>> <https://www.ververica.com/>
>>>
>>> Follow us @VervericaData
>>> --
>>> Join Flink Forward <https://flink-forward.org/>- The Apache
>>> FlinkConference
>>> Stream Processing | Event Driven | Real Time
>>> --
>>> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>>> --
>>> Data Artisans GmbHRegistered at Amtsgericht Charlottenburg: HRB
>>> 158244 BManaging Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>>
>
Re: Starting Flink cluster and running a job
Posted by Boris Lublinsky <bo...@lightbend.com>.
Gordon, I double checked it several times. Here is the list
Columns are
[info] - Jar-Size including dependencies
[info] - Jar-Size
[info] - Number of transitive dependencies
[info] - Number of direct dependencies
[info] - ModuleID
[info] Done updating.
[info] TotSize JarSize #TDe #Dep Module
[info] 27.988 MB ------- MB 33 2 lightbend:fdp-flink-taxiride_2.11:2.0.0
[info] 19.632 MB ------- MB 24 6 lightbend:support_2.11:2.0.0
[info] 17.675 MB 0.000 MB 16 9 org.typelevel:cats_2.11:0.9.0
[info] 17.389 MB 1.182 MB 13 9 org.typelevel:cats-laws_2.11:0.9.0
[info] 13.115 MB 0.286 MB 7 4 org.typelevel:cats-free_2.11:0.9.0
[info] 12.830 MB 0.000 MB 7 4 org.typelevel:cats-jvm_2.11:0.9.0
[info] 12.830 MB 3.403 MB 6 4 org.typelevel:cats-core_2.11:0.9.0
[info] 8.397 MB 0.068 MB 8 5 org.apache.flink:flink-connector-kafka_2.11:1.7.1
[info] 8.175 MB 1.889 MB 4 4 org.apache.kafka:kafka-clients:2.1.0
[info] 7.973 MB 0.212 MB 7 4 org.typelevel:cats-kernel-laws_2.11:0.9.0
[info] 4.834 MB 0.007 MB 4 2 org.typelevel:cats-macros_2.11:0.9.0
[info] 4.746 MB 0.082 MB 2 2 com.typesafe.scala-logging:scala-logging_2.11:3.9.0
[info] 4.658 MB 0.035 MB 1 1 org.typelevel:machinist_2.11:0.6.1
[info] 4.623 MB 4.623 MB 0 0 org.scala-lang:scala-reflect:2.11.12
[info] 4.592 MB 4.592 MB 0 0 org.typelevel:cats-kernel_2.11:0.9.0
[info] 3.714 MB 3.714 MB 0 0 com.github.luben:zstd-jni:1.3.5-4
[info] 3.152 MB 0.043 MB 2 1 org.typelevel:discipline_2.11:0.7.2
[info] 3.109 MB 3.094 MB 1 1 org.scalacheck:scalacheck_2.11:1.13.4
[info] 2.019 MB 2.019 MB 0 0 org.xerial.snappy:snappy-java:1.1.7.2
[info] 0.803 MB 0.290 MB 2 2 ch.qos.logback:logback-classic:1.2.3
[info] 0.641 MB 0.641 MB 0 0 joda-time:joda-time:2.10.1
[info] 0.512 MB 0.512 MB 0 0 org.lz4:lz4-java:1.5.0
[info] 0.472 MB 0.472 MB 0 0 ch.qos.logback:logback-core:1.2.3
[info] 0.286 MB 0.286 MB 0 0 com.typesafe:config:1.3.3
[info] 0.195 MB 0.114 MB 3 3 org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
[info] 0.170 MB 0.167 MB 1 1 com.github.mpilquist:simulacrum_2.11:0.10.0
[info] 0.145 MB 0.145 MB 0 0 org.joda:joda-convert:2.1.2
[info] 0.041 MB 0.041 MB 0 0 org.slf4j:slf4j-api:1.7.25
[info] 0.033 MB 0.033 MB 0 0 com.google.code.findbugs:jsr305:1.3.9
[info] 0.016 MB 0.002 MB 2 1 org.typelevel:catalysts-platform_2.11:0.0.5
[info] 0.015 MB 0.012 MB 1 1 org.typelevel:catalysts-macros_2.11:0.0.5
[info] 0.015 MB 0.015 MB 0 0 org.scala-sbt:test-interface:1.0
[info] 0.007 MB 0.007 MB 0 0 org.apache.flink:force-shading:1.7.1
[info] 0.003 MB 0.003 MB 0 0 org.typelevel:macro-compat_2.11:1.1.1
And here is another view
[info] lightbend:fdp-flink-taxiride_2.11:2.0.0 [S]
[info] +-lightbend:support_2.11:2.0.0 [S]
[info] | +-ch.qos.logback:logback-classic:1.2.3
[info] | | +-ch.qos.logback:logback-core:1.2.3
[info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
[info] | | +-org.slf4j:slf4j-api:1.7.25
[info] | |
[info] | +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
[info] | | +-org.slf4j:slf4j-api:1.7.25
[info] | |
[info] | +-com.typesafe:config:1.3.3
[info] | +-joda-time:joda-time:2.10.1
[info] | +-org.joda:joda-convert:2.1.2
[info] | +-org.typelevel:cats_2.11:0.9.0 [S]
[info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | |
[info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | | |
[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | | |
[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | |
[info] | +-org.typelevel:cats-free_2.11:0.9.0 [S]
[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | | |
[info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | | | |
[info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | | | |
[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | | |
[info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | | |
[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | | |
[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | |
[info] | +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | | |
[info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | | | |
[info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | | | |
[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | | |
[info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | | |
[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | | |
[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | |
[info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
[info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
[info] | | | +-org.scala-sbt:test-interface:1.0
[info] | | |
[info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
[info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
[info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
[info] | | +-org.scala-sbt:test-interface:1.0
[info] | |
[info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | +-org.typelevel:cats-laws_2.11:0.9.0 [S]
[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
[info] | | | +-org.scala-sbt:test-interface:1.0
[info] | | |
[info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
[info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.typelevel:cats-core_2.11:0.9.0 [S]
[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | | |
[info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | | | |
[info] | | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | | | |
[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | | |
[info] | | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
[info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
[info] | | | | +-org.scala-sbt:test-interface:1.0
[info] | | | |
[info] | | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
[info] | | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | | |
[info] | | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | | | +-org.typelevel:discipline_2.11:0.7.2 [S]
[info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
[info] | | | +-org.scala-sbt:test-interface:1.0
[info] | | |
[info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | | |
[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | | |
[info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
[info] | | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
[info] | | | +-org.scala-sbt:test-interface:1.0
[info] | | |
[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | |
[info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | |
[info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] |
[info] +-org.apache.flink:flink-connector-kafka_2.11:1.7.1
[info] +-com.google.code.findbugs:jsr305:1.3.9
[info] +-org.apache.flink:flink-connector-kafka-base_2.11:1.7.1
[info] | +-com.google.code.findbugs:jsr305:1.3.9
[info] | +-org.apache.flink:force-shading:1.7.1
[info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
[info] | +-org.slf4j:slf4j-api:1.7.25
[info] |
[info] +-org.apache.flink:force-shading:1.7.1
[info] +-org.apache.kafka:kafka-clients:2.1.0
[info] | +-com.github.luben:zstd-jni:1.3.5-4
[info] | +-org.lz4:lz4-java:1.5.0
[info] | +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
[info] | +-org.slf4j:slf4j-api:1.7.25
[info] | +-org.xerial.snappy:snappy-java:1.1.7.2
[info] |
[info] +-org.slf4j:slf4j-api:1.7.15 (evicted by: 1.7.25)
[info] +-org.slf4j:slf4j-api:1.7.25
[info]
[info] Done updating.
[info] lightbend:support_2.11:2.0.0 [S]
[info] +-ch.qos.logback:logback-classic:1.2.3
[info] | +-ch.qos.logback:logback-core:1.2.3
[info] | +-org.slf4j:slf4j-api:1.7.25
[info] |
[info] +-com.typesafe.scala-logging:scala-logging_2.11:3.9.0 [S]
[info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | +-org.slf4j:slf4j-api:1.7.25
[info] |
[info] +-com.typesafe:config:1.3.3
[info] +-joda-time:joda-time:2.10.1
[info] +-org.joda:joda-convert:2.1.2
[info] +-org.typelevel:cats_2.11:0.9.0 [S]
[info] +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] |
[info] +-org.typelevel:cats-core_2.11:0.9.0 [S]
[info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | |
[info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | |
[info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] |
[info] +-org.typelevel:cats-free_2.11:0.9.0 [S]
[info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | |
[info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | | |
[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | | |
[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | |
[info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | |
[info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] |
[info] +-org.typelevel:cats-jvm_2.11:0.9.0 [S]
[info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | |
[info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | | |
[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | | |
[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | |
[info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | |
[info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] |
[info] +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
[info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
[info] | | +-org.scala-sbt:test-interface:1.0
[info] | |
[info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
[info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | |
[info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
[info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
[info] | +-org.scala-sbt:test-interface:1.0
[info] |
[info] +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] +-org.typelevel:cats-laws_2.11:0.9.0 [S]
[info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | |
[info] | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
[info] | | +-org.scala-sbt:test-interface:1.0
[info] | |
[info] | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
[info] | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | |
[info] | +-org.typelevel:cats-core_2.11:0.9.0 [S]
[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | | |
[info] | | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | | |
[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | |
[info] | +-org.typelevel:cats-kernel-laws_2.11:0.9.0 [S]
[info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
[info] | | | +-org.scala-sbt:test-interface:1.0
[info] | | |
[info] | | +-org.typelevel:catalysts-platform_2.11:0.0.5 [S]
[info] | | | +-org.typelevel:catalysts-macros_2.11:0.0.5 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | | +-org.typelevel:discipline_2.11:0.7.2 [S]
[info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
[info] | | +-org.scala-sbt:test-interface:1.0
[info] | |
[info] | +-org.typelevel:cats-kernel_2.11:0.9.0 [S]
[info] | +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | | |
[info] | | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] | |
[info] | +-org.typelevel:discipline_2.11:0.7.2 [S]
[info] | | +-org.scalacheck:scalacheck_2.11:1.13.4 [S]
[info] | | +-org.scala-sbt:test-interface:1.0
[info] | |
[info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] |
[info] +-org.typelevel:cats-macros_2.11:0.9.0 [S]
[info] | +-com.github.mpilquist:simulacrum_2.11:0.10.0 [S]
[info] | | +-org.typelevel:macro-compat_2.11:1.1.1 [S]
[info] | |
[info] | +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] | +-org.scala-lang:scala-reflect:2.11.12 [S]
[info] |
[info] +-org.typelevel:machinist_2.11:0.6.1 [S]
[info] +-org.scala-lang:scala-reflect:2.11.12 [S]
[info]
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
> On Feb 22, 2019, at 12:33 AM, Tzu-Li (Gordon) Tai <go...@ververica.com> wrote:
>
> Hi,
>
> I haven't seen this problem for Flink 1.6.x / 1.7.x, so it shouldn't be a reoccurrence of FLINK-8741.
> I've double checked the used classloaders in the Kafka connector, they seem to be correct.
>
> The fact that it works correctly in IntelliJ, but not when packaged, suggests that there could be some conflicting dependencies in the packaged jar.
>
> Could you check the actual resolved dependency tree of the project, and see if there are multiple versions of some dependency related to Kafka being pulled in? For Maven for example, that would be "mvn dependency:tree".
>
> Cheers,
> Gordon
>
> On Thu, Feb 21, 2019 at 10:49 PM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
> The relevant dependencies are
> val flinkScala = "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
> val flinkStreamingScala = "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
> val flinkKafka = "org.apache.flink" %% "flink-connector-kafka" % flinkVersion exclude("org.slf4j", "slf4j-log4j12")
> I am using SBT
> I tried both connector-kafka and connector-kaka-11 - same result
>
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com <ma...@lightbend.com>
> https://www.lightbend.com/ <https://www.lightbend.com/>
>> On Feb 21, 2019, at 1:38 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>
>> Hi Boris,
>>
>> can you the relevant parts (dependencies) of your pom.xml? Did you also try without fixing the Kafka version, i.e. running with the Kafka client version provided by the Kafka connector of Flink? Gordon (cc) dealt with FLINK-8741.
>>
>> @Gordon: have you seen this issue with 1.6/1.7 before?
>>
>> Cheers,
>>
>> Konstantin
>>
>> On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>> I found some more details on this
>> The same error for the same application was reported about a year ago http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E>
>> And was due to https://issues.apache.org/jira/browse/FLINK-8741 <https://issues.apache.org/jira/browse/FLINK-8741>
>>
>> It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with both latest kaffka-connector
>> And Kafka-connector-011
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>> wrote:
>>>
>>> Hi Boris,
>>>
>>> I haven’t seen this exact error, but I have seen similar errors caused by multiple versions of jars on the classpath.
>>>
>>> When I’ve run into this particular "XXX is not an instance of YYY" problem, it often seems to be caused by a jar that I should have marked as provided in my pom.
>>>
>>> Though I’m typically running on a YARN cluster, not w/K8s, so maybe this doesn’t apply.
>>>
>>> — Ken
>>>
>>> PS - I assume you’ve been reading https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html>
>>>
>>>
>>>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>
>>>> Konstantin,
>>>> After experimenting with this for a while, I got to the root cause of the problem
>>>> I am running a version of a Taxi ride travel prediction as my sample.
>>>> It works fine in Intellij,
>>>> But when I am trying to put it in the docker (standard Debian 1.7 image)
>>>> It fails with a following error
>>>>
>>>>
>>>> The program finished with the following exception:
>>>>
>>>> org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>>>> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>>>> at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>>>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>>>> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>>>> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>>>> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>>>> at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>>>> at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>>>> at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>>>> ... 19 more
>>>> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>>>> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>>>> at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>>>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.serialization.ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer
>>>> at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>>>> ... 17 more
>>>>
>>>> The closest that I found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no <https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no>
>>>> Which talks about class loader. (I tried there solution, but it did not help)
>>>> I looked at the loading and I see that the pair of these 2 classes is loaded from my uber jar, but twice.
>>>>
>>>> Have you guys seen this error before?
>>>> Any suggestion?
>>>>
>>>> Boris Lublinsky
>>>> FDP Architect
>>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>>
>>>>> Hi Boris,
>>>>>
>>>>> without looking at the entrypoint in much detail, generally there should not be a race condition there:
>>>>>
>>>>> * if the taskmanagers can not connect to the resourcemanager they will retry (per default the timeout is 5 mins)
>>>>> * if the JobManager does not get enough resources from the ResourceManager it will also wait for the resources/slots to provided. The timeout there is also 5 minutes, I think.
>>>>>
>>>>> So, this should actually be pretty robust as long as the Taskmanager containers can reach the Jobmanager eventually.
>>>>>
>>>>> Could you provide the Taskmanager/JobManager logs for such a failure case?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Konstantin
>>>>>
>>>>>
>>>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>>> Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker <https://github.com/apache/flink/tree/release-1.7/flink-container/docker>
>>>>> I have created an entry point, which looks like follows:
>>>>> #!/bin/sh
>>>>>
>>>>> ################################################################################
>>>>> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh>
>>>>> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh <https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh>
>>>>> ################################################################################
>>>>>
>>>>> # If unspecified, the hostname of the container is taken as the JobManager address
>>>>> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>>>>
>>>>> drop_privs_cmd() {
>>>>> if [ $(id -u) != 0 ]; then
>>>>> # Don't need to drop privs if EUID != 0
>>>>> return
>>>>> elif [ -x /sbin/su-exec ]; then
>>>>> # Alpine
>>>>> echo su-exec flink
>>>>> else
>>>>> # Others
>>>>> echo gosu flink
>>>>> fi
>>>>> }
>>>>>
>>>>> JOB_MANAGER="jobmanager"
>>>>> TASK_MANAGER="taskmanager"
>>>>>
>>>>> CMD="$1"
>>>>> shift
>>>>>
>>>>> if [ "${CMD}" = "help" ]; then
>>>>> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
>>>>> exit 0
>>>>> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
>>>>> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>>>>
>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>
>>>>> echo "Starting Task Manager"
>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>>>> else
>>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>>
>>>>> if [ -z "$1" ]; then
>>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
>>>>> else
>>>>> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
>>>>> fi
>>>>> fi
>>>>> fi
>>>>>
>>>>> exec "$@"
>>>>> It does work for all the cases, except running standalone job.
>>>>> The problem, the way I understand it, is a racing condition.
>>>>> In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh
>>>>> tries to start a job immediately once the cluster is created (before connection is established).
>>>>> Is there a better option to implement it starting a job on container startup?
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Konstantin Knauf | Solutions Architect
>>>>> +49 160 91394525
>>>
>>> --------------------------
>>> Ken Krugler
>>> +1 530-210-6378
>>> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
>>> Custom big data solutions & training
>>> Flink, Solr, Hadoop, Cascading & Cassandra
>>
>>
>>
>> --
>> Konstantin Knauf | Solutions Architect
>> +49 160 91394525
>> <https://www.ververica.com/>
>> Follow us @VervericaData
>> --
>> Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference
>> Stream Processing | Event Driven | Real Time
>> --
>> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
>> --
>> Data Artisans GmbH
>> Registered at Amtsgericht Charlottenburg: HRB 158244 B
>> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
>
Re: Starting Flink cluster and running a job
Posted by Boris Lublinsky <bo...@lightbend.com>.
The relevant dependencies are
val flinkScala = "org.apache.flink" %% "flink-scala" % flinkVersion % "provided"
val flinkStreamingScala = "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"
val flinkKafka = "org.apache.flink" %% "flink-connector-kafka" % flinkVersion exclude("org.slf4j", "slf4j-log4j12")
I am using SBT
I tried both connector-kafka and connector-kaka-11 - same result
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
> On Feb 21, 2019, at 1:38 AM, Konstantin Knauf <ko...@ververica.com> wrote:
>
> Hi Boris,
>
> can you the relevant parts (dependencies) of your pom.xml? Did you also try without fixing the Kafka version, i.e. running with the Kafka client version provided by the Kafka connector of Flink? Gordon (cc) dealt with FLINK-8741.
>
> @Gordon: have you seen this issue with 1.6/1.7 before?
>
> Cheers,
>
> Konstantin
>
> On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
> I found some more details on this
> The same error for the same application was reported about a year ago http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E>
> And was due to https://issues.apache.org/jira/browse/FLINK-8741 <https://issues.apache.org/jira/browse/FLINK-8741>
>
> It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with both latest kaffka-connector
> And Kafka-connector-011
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com <ma...@lightbend.com>
> https://www.lightbend.com/ <https://www.lightbend.com/>
>> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kkrugler_lists@transpac.com <ma...@transpac.com>> wrote:
>>
>> Hi Boris,
>>
>> I haven’t seen this exact error, but I have seen similar errors caused by multiple versions of jars on the classpath.
>>
>> When I’ve run into this particular "XXX is not an instance of YYY" problem, it often seems to be caused by a jar that I should have marked as provided in my pom.
>>
>> Though I’m typically running on a YARN cluster, not w/K8s, so maybe this doesn’t apply.
>>
>> — Ken
>>
>> PS - I assume you’ve been reading https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html>
>>
>>
>>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>
>>> Konstantin,
>>> After experimenting with this for a while, I got to the root cause of the problem
>>> I am running a version of a Taxi ride travel prediction as my sample.
>>> It works fine in Intellij,
>>> But when I am trying to put it in the docker (standard Debian 1.7 image)
>>> It fails with a following error
>>>
>>>
>>> The program finished with the following exception:
>>>
>>> org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>>> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>>> at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>>> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>>> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>>> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>>> at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>>> at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>>> at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>>> ... 19 more
>>> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>>> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>>> at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>>> at java.lang.Thread.run(Thread.java:748)
>>> Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.serialization.ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer
>>> at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>>> ... 17 more
>>>
>>> The closest that I found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no <https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no>
>>> Which talks about class loader. (I tried there solution, but it did not help)
>>> I looked at the loading and I see that the pair of these 2 classes is loaded from my uber jar, but twice.
>>>
>>> Have you guys seen this error before?
>>> Any suggestion?
>>>
>>> Boris Lublinsky
>>> FDP Architect
>>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>>
>>>> Hi Boris,
>>>>
>>>> without looking at the entrypoint in much detail, generally there should not be a race condition there:
>>>>
>>>> * if the taskmanagers can not connect to the resourcemanager they will retry (per default the timeout is 5 mins)
>>>> * if the JobManager does not get enough resources from the ResourceManager it will also wait for the resources/slots to provided. The timeout there is also 5 minutes, I think.
>>>>
>>>> So, this should actually be pretty robust as long as the Taskmanager containers can reach the Jobmanager eventually.
>>>>
>>>> Could you provide the Taskmanager/JobManager logs for such a failure case?
>>>>
>>>> Cheers,
>>>>
>>>> Konstantin
>>>>
>>>>
>>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>>> Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker <https://github.com/apache/flink/tree/release-1.7/flink-container/docker>
>>>> I have created an entry point, which looks like follows:
>>>> #!/bin/sh
>>>>
>>>> ################################################################################
>>>> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh>
>>>> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh <https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh>
>>>> ################################################################################
>>>>
>>>> # If unspecified, the hostname of the container is taken as the JobManager address
>>>> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>>>
>>>> drop_privs_cmd() {
>>>> if [ $(id -u) != 0 ]; then
>>>> # Don't need to drop privs if EUID != 0
>>>> return
>>>> elif [ -x /sbin/su-exec ]; then
>>>> # Alpine
>>>> echo su-exec flink
>>>> else
>>>> # Others
>>>> echo gosu flink
>>>> fi
>>>> }
>>>>
>>>> JOB_MANAGER="jobmanager"
>>>> TASK_MANAGER="taskmanager"
>>>>
>>>> CMD="$1"
>>>> shift
>>>>
>>>> if [ "${CMD}" = "help" ]; then
>>>> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
>>>> exit 0
>>>> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
>>>> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
>>>> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>>>
>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>>
>>>> echo "Starting Task Manager"
>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>>> else
>>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>>
>>>> if [ -z "$1" ]; then
>>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
>>>> else
>>>> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
>>>> fi
>>>> fi
>>>> fi
>>>>
>>>> exec "$@"
>>>> It does work for all the cases, except running standalone job.
>>>> The problem, the way I understand it, is a racing condition.
>>>> In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh
>>>> tries to start a job immediately once the cluster is created (before connection is established).
>>>> Is there a better option to implement it starting a job on container startup?
>>>>
>>>>
>>>>
>>>> --
>>>> Konstantin Knauf | Solutions Architect
>>>> +49 160 91394525
>>
>> --------------------------
>> Ken Krugler
>> +1 530-210-6378
>> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
>> Custom big data solutions & training
>> Flink, Solr, Hadoop, Cascading & Cassandra
>
>
>
> --
> Konstantin Knauf | Solutions Architect
> +49 160 91394525
> <https://www.ververica.com/>
> Follow us @VervericaData
> --
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference
> Stream Processing | Event Driven | Real Time
> --
> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> --
> Data Artisans GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
Re: Starting Flink cluster and running a job
Posted by Konstantin Knauf <ko...@ververica.com>.
Hi Boris,
can you the relevant parts (dependencies) of your pom.xml? Did you also try
without fixing the Kafka version, i.e. running with the Kafka client
version provided by the Kafka connector of Flink? Gordon (cc) dealt with
FLINK-8741.
@Gordon: have you seen this issue with 1.6/1.7 before?
Cheers,
Konstantin
On Thu, Feb 21, 2019 at 2:19 AM Boris Lublinsky <
boris.lublinsky@lightbend.com> wrote:
> I found some more details on this
> The same error for the same application was reported about a year ago
> http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E
> And was due to https://issues.apache.org/jira/browse/FLINK-8741
>
> It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with both
> latest kaffka-connector
> And Kafka-connector-011
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kk...@transpac.com>
> wrote:
>
> Hi Boris,
>
> I haven’t seen this exact error, but I have seen similar errors caused by
> multiple versions of jars on the classpath.
>
> When I’ve run into this particular "XXX is not an instance of YYY"
> problem, it often seems to be caused by a jar that I should have marked as
> provided in my pom.
>
> Though I’m typically running on a YARN cluster, not w/K8s, so maybe this
> doesn’t apply.
>
> — Ken
>
> PS - I assume you’ve been reading
> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html
>
>
> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <
> boris.lublinsky@lightbend.com> wrote:
>
> Konstantin,
> After experimenting with this for a while, I got to the root cause of the
> problem
> I am running a version of a Taxi ride travel prediction as my sample.
> It works fine in Intellij,
> But when I am trying to put it in the docker (standard Debian 1.7 image)
> It fails with a following error
>
>
> The program finished with the following exception:
>
> org.apache.flink.client.program.ProgramInvocationException: Job failed.
> (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
> at
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
> at
> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
> at
> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
> at
> com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
> at
> com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
> at
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
> at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
> at
> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
> at
> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
> ... 19 more
> Caused by: org.apache.kafka.common.KafkaException: Failed to construct
> kafka producer
> at
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
> at
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
> at
> org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
> at
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
> at
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
> at
> org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.kafka.common.KafkaException:
> org.apache.kafka.common.serialization.ByteArraySerializer is not an
> instance of org.apache.kafka.common.serialization.Serializer
> at
> org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
> at
> org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
> ... 17 more
>
> The closest that I found
> https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no
> Which talks about class loader. (I tried there solution, but it did not
> help)
> I looked at the loading and I see that the pair of these 2 classes is
> loaded from my uber jar, but twice.
>
> Have you guys seen this error before?
> Any suggestion?
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com
> https://www.lightbend.com/
>
> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <ko...@ververica.com>
> wrote:
>
> Hi Boris,
>
> without looking at the entrypoint in much detail, generally there should
> not be a race condition there:
>
> * if the taskmanagers can not connect to the resourcemanager they will
> retry (per default the timeout is 5 mins)
> * if the JobManager does not get enough resources from the ResourceManager
> it will also wait for the resources/slots to provided. The timeout there is
> also 5 minutes, I think.
>
> So, this should actually be pretty robust as long as the Taskmanager
> containers can reach the Jobmanager eventually.
>
> Could you provide the Taskmanager/JobManager logs for such a failure case?
>
> Cheers,
>
> Konstantin
>
>
> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <
> boris.lublinsky@lightbend.com> wrote:
>
>> Following
>> https://github.com/apache/flink/tree/release-1.7/flink-container/docker
>> I have created an entry point, which looks like follows:
>>
>> #!/bin/sh
>>
>> ################################################################################
>> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh
>> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh
>> ################################################################################
>>
>> # If unspecified, the hostname of the container is taken as the JobManager address
>> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>
>> drop_privs_cmd() {
>> if [ $(id -u) != 0 ]; then
>> # Don't need to drop privs if EUID != 0
>> return
>> elif [ -x /sbin/su-exec ]; then
>> # Alpine
>> echo su-exec flink
>> else
>> # Others
>> echo gosu flink
>> fi
>> }
>>
>> JOB_MANAGER="jobmanager"
>> TASK_MANAGER="taskmanager"
>>
>> CMD="$1"
>> shift
>>
>> if [ "${CMD}" = "help" ]; then
>> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
>> exit 0
>> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
>> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
>> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>
>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>
>> echo "Starting Task Manager"
>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>> else
>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>
>> if [ -z "$1" ]; then
>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
>> else
>> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
>> fi
>> fi
>> fi
>>
>> exec "$@"
>>
>> It does work for all the cases, except running standalone job.
>> The problem, the way I understand it, is a racing condition.
>> In kubernetes it takes several attempts for establish connection between
>> Job and Task manager, while standalone-job.sh
>> tries to start a job immediately once the cluster is created (before
>> connection is established).
>> Is there a better option to implement it starting a job on container
>> startup?
>>
>>
>
>
> --
> Konstantin Knauf | Solutions Architect
> +49 160 91394525
>
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassandra
>
>
>
--
Konstantin Knauf | Solutions Architect
+49 160 91394525
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
Re: Starting Flink cluster and running a job
Posted by Boris Lublinsky <bo...@lightbend.com>.
I found some more details on this
The same error for the same application was reported about a year ago http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E <http://mail-archives.apache.org/mod_mbox/flink-user/201802.mbox/%3CCAE7GCT4pF74LwyY=tiVzhQUQ50TkJJaWfhaw+5phCSx+VOshFQ@mail.gmail.com%3E>
And was due to https://issues.apache.org/jira/browse/FLINK-8741 <https://issues.apache.org/jira/browse/FLINK-8741>
It looks like the same issue is back in 1.7.1 and 1.6.3. I tried with both latest kaffka-connector
And Kafka-connector-011
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kk...@transpac.com> wrote:
>
> Hi Boris,
>
> I haven’t seen this exact error, but I have seen similar errors caused by multiple versions of jars on the classpath.
>
> When I’ve run into this particular "XXX is not an instance of YYY" problem, it often seems to be caused by a jar that I should have marked as provided in my pom.
>
> Though I’m typically running on a YARN cluster, not w/K8s, so maybe this doesn’t apply.
>
> — Ken
>
> PS - I assume you’ve been reading https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html>
>
>
>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>
>> Konstantin,
>> After experimenting with this for a while, I got to the root cause of the problem
>> I am running a version of a Taxi ride travel prediction as my sample.
>> It works fine in Intellij,
>> But when I am trying to put it in the docker (standard Debian 1.7 image)
>> It fails with a following error
>>
>>
>> The program finished with the following exception:
>>
>> org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>> at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>> at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>> at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>> at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>> ... 19 more
>> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>> at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.serialization.ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer
>> at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>> ... 17 more
>>
>> The closest that I found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no <https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no>
>> Which talks about class loader. (I tried there solution, but it did not help)
>> I looked at the loading and I see that the pair of these 2 classes is loaded from my uber jar, but twice.
>>
>> Have you guys seen this error before?
>> Any suggestion?
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>
>>> Hi Boris,
>>>
>>> without looking at the entrypoint in much detail, generally there should not be a race condition there:
>>>
>>> * if the taskmanagers can not connect to the resourcemanager they will retry (per default the timeout is 5 mins)
>>> * if the JobManager does not get enough resources from the ResourceManager it will also wait for the resources/slots to provided. The timeout there is also 5 minutes, I think.
>>>
>>> So, this should actually be pretty robust as long as the Taskmanager containers can reach the Jobmanager eventually.
>>>
>>> Could you provide the Taskmanager/JobManager logs for such a failure case?
>>>
>>> Cheers,
>>>
>>> Konstantin
>>>
>>>
>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>> Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker <https://github.com/apache/flink/tree/release-1.7/flink-container/docker>
>>> I have created an entry point, which looks like follows:
>>> #!/bin/sh
>>>
>>> ################################################################################
>>> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh>
>>> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh <https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh>
>>> ################################################################################
>>>
>>> # If unspecified, the hostname of the container is taken as the JobManager address
>>> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>>
>>> drop_privs_cmd() {
>>> if [ $(id -u) != 0 ]; then
>>> # Don't need to drop privs if EUID != 0
>>> return
>>> elif [ -x /sbin/su-exec ]; then
>>> # Alpine
>>> echo su-exec flink
>>> else
>>> # Others
>>> echo gosu flink
>>> fi
>>> }
>>>
>>> JOB_MANAGER="jobmanager"
>>> TASK_MANAGER="taskmanager"
>>>
>>> CMD="$1"
>>> shift
>>>
>>> if [ "${CMD}" = "help" ]; then
>>> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
>>> exit 0
>>> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
>>> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
>>> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>>
>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>
>>> echo "Starting Task Manager"
>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>> else
>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>
>>> if [ -z "$1" ]; then
>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
>>> else
>>> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
>>> fi
>>> fi
>>> fi
>>>
>>> exec "$@"
>>> It does work for all the cases, except running standalone job.
>>> The problem, the way I understand it, is a racing condition.
>>> In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh
>>> tries to start a job immediately once the cluster is created (before connection is established).
>>> Is there a better option to implement it starting a job on container startup?
>>>
>>>
>>>
>>> --
>>> Konstantin Knauf | Solutions Architect
>>> +49 160 91394525
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassandra
Re: Starting Flink cluster and running a job
Posted by Boris Lublinsky <bo...@lightbend.com>.
Thanks Ken,
That was my first instinct as well, but..
To run on the cluster I am building an uber jar for which I am fixing Kafka clients jar version
I am also fixing version of Kafka
So I do not know where another version can get from
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
> On Feb 19, 2019, at 7:02 PM, Ken Krugler <kk...@transpac.com> wrote:
>
> Hi Boris,
>
> I haven’t seen this exact error, but I have seen similar errors caused by multiple versions of jars on the classpath.
>
> When I’ve run into this particular "XXX is not an instance of YYY" problem, it often seems to be caused by a jar that I should have marked as provided in my pom.
>
> Though I’m typically running on a YARN cluster, not w/K8s, so maybe this doesn’t apply.
>
> — Ken
>
> PS - I assume you’ve been reading https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html>
>
>
>> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>
>> Konstantin,
>> After experimenting with this for a while, I got to the root cause of the problem
>> I am running a version of a Taxi ride travel prediction as my sample.
>> It works fine in Intellij,
>> But when I am trying to put it in the docker (standard Debian 1.7 image)
>> It fails with a following error
>>
>>
>> The program finished with the following exception:
>>
>> org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
>> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
>> at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
>> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
>> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
>> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
>> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
>> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
>> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
>> at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
>> at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>> at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
>> ... 19 more
>> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
>> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
>> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
>> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
>> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
>> at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
>> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
>> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
>> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.serialization.ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer
>> at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
>> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
>> ... 17 more
>>
>> The closest that I found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no <https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no>
>> Which talks about class loader. (I tried there solution, but it did not help)
>> I looked at the loading and I see that the pair of these 2 classes is loaded from my uber jar, but twice.
>>
>> Have you guys seen this error before?
>> Any suggestion?
>>
>> Boris Lublinsky
>> FDP Architect
>> boris.lublinsky@lightbend.com <ma...@lightbend.com>
>> https://www.lightbend.com/ <https://www.lightbend.com/>
>>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>>
>>> Hi Boris,
>>>
>>> without looking at the entrypoint in much detail, generally there should not be a race condition there:
>>>
>>> * if the taskmanagers can not connect to the resourcemanager they will retry (per default the timeout is 5 mins)
>>> * if the JobManager does not get enough resources from the ResourceManager it will also wait for the resources/slots to provided. The timeout there is also 5 minutes, I think.
>>>
>>> So, this should actually be pretty robust as long as the Taskmanager containers can reach the Jobmanager eventually.
>>>
>>> Could you provide the Taskmanager/JobManager logs for such a failure case?
>>>
>>> Cheers,
>>>
>>> Konstantin
>>>
>>>
>>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>>> Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker <https://github.com/apache/flink/tree/release-1.7/flink-container/docker>
>>> I have created an entry point, which looks like follows:
>>> #!/bin/sh
>>>
>>> ################################################################################
>>> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh>
>>> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh <https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh>
>>> ################################################################################
>>>
>>> # If unspecified, the hostname of the container is taken as the JobManager address
>>> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>>
>>> drop_privs_cmd() {
>>> if [ $(id -u) != 0 ]; then
>>> # Don't need to drop privs if EUID != 0
>>> return
>>> elif [ -x /sbin/su-exec ]; then
>>> # Alpine
>>> echo su-exec flink
>>> else
>>> # Others
>>> echo gosu flink
>>> fi
>>> }
>>>
>>> JOB_MANAGER="jobmanager"
>>> TASK_MANAGER="taskmanager"
>>>
>>> CMD="$1"
>>> shift
>>>
>>> if [ "${CMD}" = "help" ]; then
>>> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
>>> exit 0
>>> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
>>> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
>>> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>>
>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>>
>>> echo "Starting Task Manager"
>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>>> else
>>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>>
>>> if [ -z "$1" ]; then
>>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
>>> else
>>> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
>>> fi
>>> fi
>>> fi
>>>
>>> exec "$@"
>>> It does work for all the cases, except running standalone job.
>>> The problem, the way I understand it, is a racing condition.
>>> In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh
>>> tries to start a job immediately once the cluster is created (before connection is established).
>>> Is there a better option to implement it starting a job on container startup?
>>>
>>>
>>>
>>> --
>>> Konstantin Knauf | Solutions Architect
>>> +49 160 91394525
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com <http://www.scaleunlimited.com/>
> Custom big data solutions & training
> Flink, Solr, Hadoop, Cascading & Cassandra
Re: Starting Flink cluster and running a job
Posted by Ken Krugler <kk...@transpac.com>.
Hi Boris,
I haven’t seen this exact error, but I have seen similar errors caused by multiple versions of jars on the classpath.
When I’ve run into this particular "XXX is not an instance of YYY" problem, it often seems to be caused by a jar that I should have marked as provided in my pom.
Though I’m typically running on a YARN cluster, not w/K8s, so maybe this doesn’t apply.
— Ken
PS - I assume you’ve been reading https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html <https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/debugging_classloading.html>
> On Feb 19, 2019, at 4:34 PM, Boris Lublinsky <bo...@lightbend.com> wrote:
>
> Konstantin,
> After experimenting with this for a while, I got to the root cause of the problem
> I am running a version of a Taxi ride travel prediction as my sample.
> It works fine in Intellij,
> But when I am trying to put it in the docker (standard Debian 1.7 image)
> It fails with a following error
>
>
> The program finished with the following exception:
>
> org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
> at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
> at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
> at com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
> at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
> at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
> at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
> ... 19 more
> Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
> at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
> at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
> at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
> at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
> at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
> at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
> at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.serialization.ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer
> at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
> at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
> ... 17 more
>
> The closest that I found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no <https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no>
> Which talks about class loader. (I tried there solution, but it did not help)
> I looked at the loading and I see that the pair of these 2 classes is loaded from my uber jar, but twice.
>
> Have you guys seen this error before?
> Any suggestion?
>
> Boris Lublinsky
> FDP Architect
> boris.lublinsky@lightbend.com <ma...@lightbend.com>
> https://www.lightbend.com/
>
>> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <konstantin@ververica.com <ma...@ververica.com>> wrote:
>>
>> Hi Boris,
>>
>> without looking at the entrypoint in much detail, generally there should not be a race condition there:
>>
>> * if the taskmanagers can not connect to the resourcemanager they will retry (per default the timeout is 5 mins)
>> * if the JobManager does not get enough resources from the ResourceManager it will also wait for the resources/slots to provided. The timeout there is also 5 minutes, I think.
>>
>> So, this should actually be pretty robust as long as the Taskmanager containers can reach the Jobmanager eventually.
>>
>> Could you provide the Taskmanager/JobManager logs for such a failure case?
>>
>> Cheers,
>>
>> Konstantin
>>
>>
>> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
>> Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker <https://github.com/apache/flink/tree/release-1.7/flink-container/docker>
>> I have created an entry point, which looks like follows:
>> #!/bin/sh
>>
>> ################################################################################
>> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh>
>> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh <https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh>
>> ################################################################################
>>
>> # If unspecified, the hostname of the container is taken as the JobManager address
>> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>>
>> drop_privs_cmd() {
>> if [ $(id -u) != 0 ]; then
>> # Don't need to drop privs if EUID != 0
>> return
>> elif [ -x /sbin/su-exec ]; then
>> # Alpine
>> echo su-exec flink
>> else
>> # Others
>> echo gosu flink
>> fi
>> }
>>
>> JOB_MANAGER="jobmanager"
>> TASK_MANAGER="taskmanager"
>>
>> CMD="$1"
>> shift
>>
>> if [ "${CMD}" = "help" ]; then
>> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
>> exit 0
>> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
>> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
>> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>>
>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>>
>> echo "Starting Task Manager"
>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
>> else
>> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
>> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
>> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>>
>> if [ -z "$1" ]; then
>> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
>> else
>> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
>> fi
>> fi
>> fi
>>
>> exec "$@"
>> It does work for all the cases, except running standalone job.
>> The problem, the way I understand it, is a racing condition.
>> In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh
>> tries to start a job immediately once the cluster is created (before connection is established).
>> Is there a better option to implement it starting a job on container startup?
>>
>>
>>
>> --
>> Konstantin Knauf | Solutions Architect
>> +49 160 91394525
--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
Custom big data solutions & training
Flink, Solr, Hadoop, Cascading & Cassandra
Re: Starting Flink cluster and running a job
Posted by Boris Lublinsky <bo...@lightbend.com>.
Konstantin,
After experimenting with this for a while, I got to the root cause of the problem
I am running a version of a Taxi ride travel prediction as my sample.
It works fine in Intellij,
But when I am trying to put it in the docker (standard Debian 1.7 image)
It fails with a following error
The program finished with the following exception:
org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID: 9340e7669e7344ab827fef4ddb5ba73d)
at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:268)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:487)
at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:66)
at org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.scala:654)
at com.lightbend.fdp.sample.flink.app.TravelTimePrediction$.main(TravelTimePrediction.scala:89)
at com.lightbend.fdp.sample.flink.app.TravelTimePrediction.main(TravelTimePrediction.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:427)
at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:265)
... 19 more
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka producer
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:416)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:288)
at org.apache.flink.streaming.connectors.kafka.internal.FlinkKafkaProducer.<init>(FlinkKafkaProducer.java:116)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initProducer(FlinkKafkaProducer011.java:944)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initNonTransactionalProducer(FlinkKafkaProducer011.java:940)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:696)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.beginTransaction(FlinkKafkaProducer011.java:94)
at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.beginTransactionInternal(TwoPhaseCommitSinkFunction.java:384)
at org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.initializeState(TwoPhaseCommitSinkFunction.java:375)
at org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer011.initializeState(FlinkKafkaProducer011.java:847)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.tryRestoreFunction(StreamingFunctionUtils.java:178)
at org.apache.flink.streaming.util.functions.StreamingFunctionUtils.restoreFunctionState(StreamingFunctionUtils.java:160)
at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:96)
at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:278)
at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:738)
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:289)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:704)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.serialization.ByteArraySerializer is not an instance of org.apache.kafka.common.serialization.Serializer
at org.apache.kafka.common.config.AbstractConfig.getConfiguredInstance(AbstractConfig.java:248)
at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:327)
... 17 more
The closest that I found https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no <https://stackoverflow.com/questions/37363119/kafka-producer-org-apache-kafka-common-serialization-stringserializer-could-no>
Which talks about class loader. (I tried there solution, but it did not help)
I looked at the loading and I see that the pair of these 2 classes is loaded from my uber jar, but twice.
Have you guys seen this error before?
Any suggestion?
Boris Lublinsky
FDP Architect
boris.lublinsky@lightbend.com
https://www.lightbend.com/
> On Feb 19, 2019, at 4:50 AM, Konstantin Knauf <ko...@ververica.com> wrote:
>
> Hi Boris,
>
> without looking at the entrypoint in much detail, generally there should not be a race condition there:
>
> * if the taskmanagers can not connect to the resourcemanager they will retry (per default the timeout is 5 mins)
> * if the JobManager does not get enough resources from the ResourceManager it will also wait for the resources/slots to provided. The timeout there is also 5 minutes, I think.
>
> So, this should actually be pretty robust as long as the Taskmanager containers can reach the Jobmanager eventually.
>
> Could you provide the Taskmanager/JobManager logs for such a failure case?
>
> Cheers,
>
> Konstantin
>
>
> On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <boris.lublinsky@lightbend.com <ma...@lightbend.com>> wrote:
> Following https://github.com/apache/flink/tree/release-1.7/flink-container/docker <https://github.com/apache/flink/tree/release-1.7/flink-container/docker>
> I have created an entry point, which looks like follows:
> #!/bin/sh
>
> ################################################################################
> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh <https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh>
> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh <https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh>
> ################################################################################
>
> # If unspecified, the hostname of the container is taken as the JobManager address
> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>
> drop_privs_cmd() {
> if [ $(id -u) != 0 ]; then
> # Don't need to drop privs if EUID != 0
> return
> elif [ -x /sbin/su-exec ]; then
> # Alpine
> echo su-exec flink
> else
> # Others
> echo gosu flink
> fi
> }
>
> JOB_MANAGER="jobmanager"
> TASK_MANAGER="taskmanager"
>
> CMD="$1"
> shift
>
> if [ "${CMD}" = "help" ]; then
> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
> exit 0
> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>
> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>
> echo "Starting Task Manager"
> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
> else
> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>
> if [ -z "$1" ]; then
> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
> else
> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
> fi
> fi
> fi
>
> exec "$@"
> It does work for all the cases, except running standalone job.
> The problem, the way I understand it, is a racing condition.
> In kubernetes it takes several attempts for establish connection between Job and Task manager, while standalone-job.sh
> tries to start a job immediately once the cluster is created (before connection is established).
> Is there a better option to implement it starting a job on container startup?
>
>
>
> --
> Konstantin Knauf | Solutions Architect
> +49 160 91394525
>
> <https://www.ververica.com/>
> Follow us @VervericaData
> --
> Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference
> Stream Processing | Event Driven | Real Time
> --
> Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> --
> Data Artisans GmbH
> Registered at Amtsgericht Charlottenburg: HRB 158244 B
> Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
Re: Starting Flink cluster and running a job
Posted by Konstantin Knauf <ko...@ververica.com>.
Hi Boris,
without looking at the entrypoint in much detail, generally there should
not be a race condition there:
* if the taskmanagers can not connect to the resourcemanager they will
retry (per default the timeout is 5 mins)
* if the JobManager does not get enough resources from the ResourceManager
it will also wait for the resources/slots to provided. The timeout there is
also 5 minutes, I think.
So, this should actually be pretty robust as long as the Taskmanager
containers can reach the Jobmanager eventually.
Could you provide the Taskmanager/JobManager logs for such a failure case?
Cheers,
Konstantin
On Mon, Feb 18, 2019 at 1:07 AM Boris Lublinsky <
boris.lublinsky@lightbend.com> wrote:
> Following
> https://github.com/apache/flink/tree/release-1.7/flink-container/docker
> I have created an entry point, which looks like follows:
>
> #!/bin/sh
>
> ################################################################################
> # from https://github.com/apache/flink/blob/release-1.7/flink-container/docker/docker-entrypoint.sh
> # and https://github.com/docker-flink/docker-flink/blob/63b19a904fa8bfd1322f1d59fdb226c82b9186c7/1.7/scala_2.11-alpine/docker-entrypoint.sh
> ################################################################################
>
> # If unspecified, the hostname of the container is taken as the JobManager address
> JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
>
> drop_privs_cmd() {
> if [ $(id -u) != 0 ]; then
> # Don't need to drop privs if EUID != 0
> return
> elif [ -x /sbin/su-exec ]; then
> # Alpine
> echo su-exec flink
> else
> # Others
> echo gosu flink
> fi
> }
>
> JOB_MANAGER="jobmanager"
> TASK_MANAGER="taskmanager"
>
> CMD="$1"
> shift
>
> if [ "${CMD}" = "help" ]; then
> echo "Usage: $(basename $0) (${JOB_MANAGER}|${TASK_MANAGER}|help)"
> exit 0
> elif [ "${CMD}" = "${JOB_MANAGER}" -o "${CMD}" = "${TASK_MANAGER}" ]; then
> if [ "${CMD}" = "${TASK_MANAGER}" ]; then
> TASK_MANAGER_NUMBER_OF_TASK_SLOTS=${TASK_MANAGER_NUMBER_OF_TASK_SLOTS:-$(grep -c ^processor /proc/cpuinfo)}
>
> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
> sed -i -e "s/taskmanager.numberOfTaskSlots: 1/taskmanager.numberOfTaskSlots: $TASK_MANAGER_NUMBER_OF_TASK_SLOTS/g" "$FLINK_HOME/conf/flink-conf.yaml"
> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
>
> echo "Starting Task Manager"
> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
> exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground
> else
> sed -i -e "s/jobmanager.rpc.address: localhost/jobmanager.rpc.address: ${JOB_MANAGER_RPC_ADDRESS}/g" "$FLINK_HOME/conf/flink-conf.yaml"
> echo "blob.server.port: 6124" >> "$FLINK_HOME/conf/flink-conf.yaml"
> echo "query.server.port: 6125" >> "$FLINK_HOME/conf/flink-conf.yaml"
> echo "config file: " && grep '^[^\n#]' "$FLINK_HOME/conf/flink-conf.yaml"
>
> if [ -z "$1" ]; then
> exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "$@"
> else
> exec $FLINK_HOME/bin/standalone-job.sh start-foreground "$@"
> fi
> fi
> fi
>
> exec "$@"
>
> It does work for all the cases, except running standalone job.
> The problem, the way I understand it, is a racing condition.
> In kubernetes it takes several attempts for establish connection between
> Job and Task manager, while standalone-job.sh
> tries to start a job immediately once the cluster is created (before
> connection is established).
> Is there a better option to implement it starting a job on container
> startup?
>
>
--
Konstantin Knauf | Solutions Architect
+49 160 91394525
<https://www.ververica.com/>
Follow us @VervericaData
--
Join Flink Forward <https://flink-forward.org/> - The Apache Flink
Conference
Stream Processing | Event Driven | Real Time
--
Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen