Re: Flink 'Job Cluster' mode Ui Access

[Jatin Banger <ba...@gmail.com>]

Hi,

Recently i upgraded flink version to 1.8.3
For Session cluster it shows the version correctly.
But for job cluster. I get this in the logs

*Starting StandaloneJobClusterEntryPoint (Version: <unknown>, Rev:6322618,
Date:04.09.2019 @ 22:07:41 CST)*

And my Classpath has these jars:

*Classpath:
/opt/flink/lib/flink-metrics-prometheus-1.8.3.jar:/opt/flink/lib/flink-shaded-hadoop-2-uber-2.6.5-7.0.jar:/opt/flink/lib/job.jar:/opt/flink/lib/log4j-over-slf4j-1.7.28.jar:/opt/flink/lib/logback-classic-1.2.3.jar:/opt/flink/lib/logback-core-1.2.3.jar:/opt/flink/lib/flink-dist_2.11-1.8.3.jar::/opt/hdfs:*

Do you have any idea what could have caused this?

Best Regards,
Jatin

On Fri, Dec 13, 2019 at 6:18 PM Chesnay Schepler <ch...@apache.org> wrote:

> Thank you for the logs.
>
> Flink can indeed find the WebUI files in the distribution, which is a bit
> odd.
> Since there are no static files serve in this case, the
> StaticFileServerHandler is never set up in the first place (hence why we
> didn't find any log statements).
>
> What I also found in the logs (and, looking back, in one of your earlier
> replies) was this: Version: <unknown>, Rev:ceba8af, Date:11.02.2019 @
> 22:17:09 CST
>
> This tells us a few things.
> a) You are not using 1.8.1, but 1.7.2 (based on the revision)
> b) You are not using an official release, since the build-date differs
> from the official releases
>
> I tried one of the official 1.7.2 releases, and the WebUI is shown both
> when using:
> (after copying the wordcount example into /lib)
> ./bin/standalone-job.sh start-foreground -j
> org.apache.flink.examples.java.wordcount.WordCount
> ./bin/standalone-job.sh start -j
> org.apache.flink.examples.java.wordcount.WordCount
>
> Right now I don't know what else to look for; there are some discrepancies
> as to what your environment is vs what you described, and as such I can
> only recommend to carefully evaluate what you have actually running and
> possibly try again with an official release.
>
> Regards,
> Chesnay
>
> On 13/12/2019 09:58, Jatin Banger wrote:
>
> Sure, here it is.
> Job Manager Logs with logging level as DEBUG
>
> On Wed, Dec 11, 2019 at 3:14 PM Chesnay Schepler <ch...@apache.org>
> wrote:
>
>> Would it be possible for you to provide us with full debug log file?
>>
>> On 10/12/2019 18:07, Jatin Banger wrote:
>>
>> Yes, I did.
>>
>> On Tue, Dec 10, 2019 at 3:47 PM Arvid Heise <ar...@ververica.com> wrote:
>>
>>> Hi Jatin,
>>>
>>> just to be sure. Did you increase the log level to debug [1] before
>>> checking for *StaticFileServerHandler*?
>>>
>>> Best,
>>>
>>> Arvid
>>>
>>> [1]
>>> https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/logging.html#configuring-log4j
>>>
>>> On Mon, Dec 9, 2019 at 7:54 AM Jatin Banger <ba...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have checked the logs with this keyword  *StaticFileServerHandler   *in
>>>> it, But there were no logs coming for "Flink Job Cluster".
>>>> Then i checked for Flink Session Cluster, i was able to find the logs
>>>> for the *StaticFileServerHandler *keyword.
>>>>
>>>> Can i raise this as bug ?
>>>>
>>>> Best Regards,
>>>> Jatin
>>>>
>>>>
>>>> On Thu, Dec 5, 2019 at 8:59 PM Chesnay Schepler <ch...@apache.org>
>>>> wrote:
>>>>
>>>>> Ok, it's good to know that the WebUI files are there.
>>>>>
>>>>> Please enable DEBUG logging and try again, searching for messages from
>>>>> the StaticFileServerHandler.
>>>>>
>>>>> This handler logs every file that is requested (which effectively
>>>>> happens when the WebUI is being served); let's see what is actually being
>>>>> requested.
>>>>>
>>>>> On 05/12/2019 05:57, Jatin Banger wrote:
>>>>>
>>>>> I have tried that already using
>>>>> '$FLINK_HOME/bin/jobmanager.sh" start-foreground
>>>>> Ui comes fine with this one.
>>>>> Which means web/index.html is present.
>>>>>
>>>>>
>>>>> On Wed, Dec 4, 2019 at 9:01 PM Chesnay Schepler <ch...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> hmm...this is quite odd.
>>>>>>
>>>>>> Let's try to narrow things down a bit.
>>>>>>
>>>>>> Could you try starting a local cluster (using the same distribution)
>>>>>> and checking whether the UI is accessible?
>>>>>>
>>>>>> Could you also check whether the flink-dist.jar in /lib contains
>>>>>> web/index.html?
>>>>>> On 04/12/2019 06:02, Jatin Banger wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am using flink binary directly.
>>>>>>
>>>>>> I am using this command to deploy the script.
>>>>>>
>>>>>>                         "$FLINK_HOME/bin/standalone-job.sh"
>>>>>> start-foreground --job-classname ${ARGS_FOR_JOB}
>>>>>> where ARGS_FOR_JOB contain job class name and all other necessary
>>>>>> details needed by the job.
>>>>>>
>>>>>> Best regards,
>>>>>> Jatin
>>>>>>
>>>>>>
>>>>>> On Fri, Nov 29, 2019 at 4:18 PM Chesnay Schepler <ch...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>>> To clarify, you ran "mvn package -pl flink-dist -am" to build Fink?
>>>>>>>
>>>>>>> If so, could you run that again and provide us with the maven output?
>>>>>>>
>>>>>>> On 29/11/2019 11:23, Jatin Banger wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> @vino yang <ya...@gmail.com>  I am using flink 1.8.1
>>>>>>>
>>>>>>> I am using the following procedure for the deployment:
>>>>>>>
>>>>>>> https://github.com/apache/flink/blob/master/flink-container/docker/README.md
>>>>>>>
>>>>>>> And i tried accessing the path you mentioned:
>>>>>>>
>>>>>>> # curl <pod-Ip>:4081/#/overview
>>>>>>> {"errors":["Not found."]}
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Jatin
>>>>>>>
>>>>>>> On Thu, Nov 28, 2019 at 10:21 PM Chesnay Schepler <
>>>>>>> chesnay@apache.org> wrote:
>>>>>>>
>>>>>>>> Could you try accessing <pid-Ip>:/#/overview ?
>>>>>>>>
>>>>>>>> The REST API is obviously accessible, and hence the WebUI should be
>>>>>>>> too.
>>>>>>>>
>>>>>>>> How did you setup the session cluster? Are you using some custom
>>>>>>>> Flink build or something, which potentially excluded flink-runtime-web from
>>>>>>>> the classpath?
>>>>>>>>
>>>>>>>> On 28/11/2019 10:02, Jatin Banger wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I checked the log file there is no error.
>>>>>>>> And I checked the pods internal ports by using rest api.
>>>>>>>>
>>>>>>>> # curl <pod-Ip>: 4081
>>>>>>>> {"errors":["Not found."]}
>>>>>>>> 4081 is the Ui port
>>>>>>>>
>>>>>>>> # curl <pod-Ip>:4081/config
>>>>>>>> {"refresh-interval":3000,"timezone-name":"Coordinated Universal
>>>>>>>> Time","timezone-offset":0,"flink-version":"<unknown>","flink-revision":"ceba8af
>>>>>>>> @ 11.02.2019 @ 22:17:09 CST"}
>>>>>>>>
>>>>>>>> # curl <pod-Ip>:4081/jobs
>>>>>>>> {"jobs":[{"id":"___job_Id_____","status":"RUNNING"}]}
>>>>>>>>
>>>>>>>> Which shows the state of the job as running.
>>>>>>>>
>>>>>>>> What else can we do ?
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Jatin
>>>>>>>>
>>>>>>>> On Thu, Nov 28, 2019 at 1:28 PM vino yang <ya...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Jatin,
>>>>>>>>>
>>>>>>>>> Flink web UI does not depend on any deployment mode.
>>>>>>>>>
>>>>>>>>> You should check if there are error logs in the log file and the
>>>>>>>>> job status is running state.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Vino
>>>>>>>>>
>>>>>>>>> Jatin Banger <ba...@gmail.com> 于2019年11月28日周四 下午3:43写道：
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> It seems there is Web Ui for Flink Session cluster, But for Flink
>>>>>>>>>> Job Cluster it is Showing
>>>>>>>>>>
>>>>>>>>>> {"errors":["Not found."]}
>>>>>>>>>>
>>>>>>>>>> Is it the expected behavior for Flink Job Cluster Mode ?
>>>>>>>>>>
>>>>>>>>>> Best Regards,
>>>>>>>>>> Jatin
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>
>