Invalid typesafe configuration when submitting job

[Dmitry Mikhailov <d....@cleverdata.ru>]

Hi

I'm struggling with updating a flink application. While developing, I was using a dev application configuration (application.conf). While updating application.conf, and rerunning job in a production mode, I caught an exception that stated, that i'm having a dev application configuration. Looks that application.conf was somehow cached in a cluster.
Application configuration includes section of dynamic list of RichMapFunction`s.
These map steps are assembled into a pipeline before starting a job.
Dev configuration have some map steps, that are not enabled in the production configuration.
For example dev pipeline looks like: source -> dmpkit-cp-base -> dmpkit-cp-attribute-mapper -> dmpkit-cp-tm -> sink
Production pipeline lacks first mapper: source -> dmpkit-cp-attribute-mapper -> dmpkit-cp-tm -> sink

Modifying application.conf from dev to prod and rerunning job results in having dev configuration enabled, which causes errors.
Job is submitted with the parameters:
sudo -Eu dmpkit /usr/lib/flink/bin/flink run \
-m yarn-cluster \
--detached \
--yarnname cleverdata-dmpkit-customerjourney-http-job \
--yarnship /tmp/tmp.uYNTPFfAtm-cleverdata-dmpkit-customerjourney-http-job \
--class ru.cleverdata.dmpkit.customerjourney.http.job.impl.ApplicationBootstrap \
-C file:///tmp/tmp.uYNTPFfAtm-cleverdata-dmpkit-customerjourney-http-job/jackson-databind-2.9.5.jar \
.... (other dependencies jars) \
-C file:///tmp/tmp.uYNTPFfAtm-cleverdata-dmpkit-customerjourney-http-job/etc.zip \
/tmp/tmp.uYNTPFfAtm-cleverdata-dmpkit-customerjourney-http-job/cleverdata-dmpkit-customerjourney-http-job-impl_2.11-2.18.0-93.jar

application.conf is zipped into etc.zip archive and shipped in both --yarnship /tmp/... directory and -C file:///.../etc.zip file.
Inspection of all deployed jars and configurations shows none dev configurations. Started application shows that dev configuration loaded. App crashes.

However, some tracks of dev configuration can be found in the hdfs directory of a deployed job:
some tmp file states, that dev pipeline map functions enabled

hdfs dfs -cat /user/dmpkit/.flink/application_1586040264218_59021/application_1586040264218_590217815615186225961213.tmp | strings | less
contains some mentions of dmpkit-cp-base, which is dev-only map step.
I have no idea how this map function got into pipeline, when, as I said earlier, it is not enabled in configuration and non of jars and\or *.conf files contains this configuration.
I'm stuck with searching where this invalid configuration could be cached.

Cheers,
Dmitry.

Re: Invalid typesafe configuration when submitting job

[Dmitry Mikhailov <d....@cleverdata.ru>]

Hi,

I think this issue can be closed, because a probable reason have been found. It seems that https://github.com/apache/flink/blob/release-1.6.1-rc1/flink-yarn/src/main/java/org/apache/flink/yarn/entrypoint/YarnJobClusterEntrypoint.java (https://link.getmailspring.com/link/E79C8B22-D332-47DA-AB68-66C76978B44F@getmailspring.com/0?redirect=https%3A%2F%2Fgithub.com%2Fapache%2Fflink%2Fblob%2Frelease-1.6.1-rc1%2Fflink-yarn%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fyarn%2Fentrypoint%2FYarnJobClusterEntrypoint.java&recipient=dXNlckBmbGluay5hcGFjaGUub3Jn) tracks current working directory location and assembles configuration with respect to various .conf files in this directory. Restarting job from directory, that doesn't contain any *.conf files results in proper application functioning.
Cheers,
Dmitry.

On Jun 10 2020, at 12:35 pm, Dmitry Mikhailov <d....@cleverdata.ru> wrote:
> Hi
>
> I'm struggling with updating a flink application. While developing, I was using a dev application configuration (application.conf). While updating application.conf, and rerunning job in a production mode, I caught an exception that stated, that i'm having a dev application configuration. Looks that application.conf was somehow cached in a cluster.
> Application configuration includes section of dynamic list of RichMapFunction`s.
> These map steps are assembled into a pipeline before starting a job.
> Dev configuration have some map steps, that are not enabled in the production configuration.
> For example dev pipeline looks like: source -> dmpkit-cp-base -> dmpkit-cp-attribute-mapper -> dmpkit-cp-tm -> sink
> Production pipeline lacks first mapper: source -> dmpkit-cp-attribute-mapper -> dmpkit-cp-tm -> sink
>
> Modifying application.conf from dev to prod and rerunning job results in having dev configuration enabled, which causes errors.
> Job is submitted with the parameters:
> sudo -Eu dmpkit /usr/lib/flink/bin/flink run \
> -m yarn-cluster \
> --detached \
> --yarnname cleverdata-dmpkit-customerjourney-http-job \
> --yarnship /tmp/tmp.uYNTPFfAtm-cleverdata-dmpkit-customerjourney-http-job \
> --class ru.cleverdata.dmpkit.customerjourney.http.job.impl.ApplicationBootstrap \
> -C file:///tmp/tmp.uYNTPFfAtm-cleverdata-dmpkit-customerjourney-http-job/jackson-databind-2.9.5.jar \
> .... (other dependencies jars) \
> -C file:///tmp/tmp.uYNTPFfAtm-cleverdata-dmpkit-customerjourney-http-job/etc.zip \
> /tmp/tmp.uYNTPFfAtm-cleverdata-dmpkit-customerjourney-http-job/cleverdata-dmpkit-customerjourney-http-job-impl_2.11-2.18.0-93.jar
>
> application.conf is zipped into etc.zip archive and shipped in both --yarnship /tmp/... directory and -C file:///.../etc.zip file.
> Inspection of all deployed jars and configurations shows none dev configurations. Started application shows that dev configuration loaded. App crashes.
>
> However, some tracks of dev configuration can be found in the hdfs directory of a deployed job:
> some tmp file states, that dev pipeline map functions enabled
>
> hdfs dfs -cat /user/dmpkit/.flink/application_1586040264218_59021/application_1586040264218_590217815615186225961213.tmp | strings | less
> contains some mentions of dmpkit-cp-base, which is dev-only map step.
> I have no idea how this map function got into pipeline, when, as I said earlier, it is not enabled in configuration and non of jars and\or *.conf files contains this configuration.
> I'm stuck with searching where this invalid configuration could be cached.
>
> Cheers,
> Dmitry.