You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@zeppelin.apache.org by Som Lima <so...@gmail.com> on 2020/05/04 00:47:31 UTC

Re: Job manager URI rpc address:port

Here are the instructions . Go to getting started link.

It took me hours first time and I had many errors because I had spark
installed earlier on same machine.

Then when I switched flink remote from client to server and server to
client
it only took me five minutes because this time  it was clean install.

On Sun, 19 Apr 2020, 14:58 Jeff Zhang, <zj...@gmail.com> wrote:

> Hi Som,
>
> You can take a look at flink on zeppelin, in zeppelin you can connect to a
> remote flink cluster via a few configuration, and you don't need to worry
> about the jars. Flink interpreter will ship necessary jars for you. Here's
> a list of tutorials.
>
> 1) Get started https://link.medium.com/oppqD6dIg5
> <https://t.co/PTouUYYTrv?amp=1> 2) Batch https://
> link.medium.com/3qumbwRIg5 <https://t.co/Yo9QAY0Joj?amp=1> 3) Streaming
> https://link.medium.com/RBHa2lTIg5 <https://t.co/sUapN40tvI?amp=1> 4)
> Advanced usage https://link.medium.com/CAekyoXIg5
> <https://t.co/MXolULmafZ?amp=1>
>
>
> Zahid Rahman <za...@gmail.com> 于2020年4月19日周日 下午7:27写道：
>
>> Hi Tison,
>>
>> I think I may have found what I want in example 22.
>>
>> https://www.programcreek.com/java-api-examples/?api=org.apache.flink.configuration.Configuration
>>
>> I need to create Configuration object first as shown .
>>
>> Also I think  flink-conf.yaml file may contain configuration for client
>> rather than  server. So before starting is irrelevant.
>> I am going to play around and see but if the Configuration class allows
>> me to set configuration programmatically and overrides the yaml file then
>> that would be great.
>>
>>
>>
>> On Sun, 19 Apr 2020, 11:35 Som Lima, <so...@gmail.com> wrote:
>>
>>> Thanks.
>>> flink-conf.yaml does allow me to do what I need to do without making any
>>> changes to client source code.
>>>
>>> But
>>> RemoteStreamEnvironment constructor  expects a jar file as the third
>>> parameter also.
>>>
>>> RemoteStreamEnvironment
>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.7/api/java/org/apache/flink/streaming/api/environment/RemoteStreamEnvironment.html#RemoteStreamEnvironment-java.lang.String-int-java.lang.String...->
>>> (String
>>> <http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true> host,
>>> int port, String
>>> <http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true>
>>> ... jarFiles)
>>> Creates a new RemoteStreamEnvironment that points to the master
>>> (JobManager) described by the given host name and port.
>>>
>>> On Sun, 19 Apr 2020, 11:02 tison, <wa...@gmail.com> wrote:
>>>
>>>> You can change flink-conf.yaml "jobmanager.address" or
>>>> "jobmanager.port" options before run the program or take a look at
>>>> RemoteStreamEnvironment which enables configuring host and port.
>>>>
>>>> Best,
>>>> tison.
>>>>
>>>>
>>>> Som Lima <so...@gmail.com> 于2020年4月19日周日 下午5:58写道：
>>>>
>>>>> Hi,
>>>>>
>>>>> After running
>>>>>
>>>>> $ ./bin/start-cluster.sh
>>>>>
>>>>> The following line of code defaults jobmanager  to localhost:6123
>>>>>
>>>>> final  ExecutionEnvironment env =
>>>>> Environment.getExecutionEnvironment();
>>>>>
>>>>> which is same on spark.
>>>>>
>>>>> val spark =
>>>>> SparkSession.builder.master(local[*]).appname("anapp").getOrCreate
>>>>>
>>>>> However if I wish to run the servers on a different physical computer.
>>>>> Then in Spark I can do it this way using the spark URI in my IDE.
>>>>>
>>>>> Conf =
>>>>> SparkConf().setMaster("spark://<hostip>:<port>").setAppName("anapp")
>>>>>
>>>>> Can you please tell me the equivalent change to make so I can run my
>>>>> servers and my IDE from different physical computers.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: Job manager URI rpc address:port

Posted by Zahid Rahman <za...@gmail.com>.

You say in your case it helped.

You just didn't follow the setup instructions! That's it


Jeff didn't explain anything he gave you config settings you asked for
based on your guess work of random Ports.

It turned out you didn't follow setup instructions. I directed you to.

Unlike you I worked through my problems.
I knew Jeff didn't have access to my machine so I didn't bother him .
I worked through it.

I had come across same error and after I followed the remaining step it was
solved.
Go and look on wiki  on the subject of 0.0.0.0
The description of error message verifies wiki also.

That is why I referred you to the aforementioned instruction.
But you wanted to give the impression you are so smart and clever and you
wouldn't accept the fact you didn't follow the instructions.


How can by making the required value change it helped.

You are just trying to sound smart when you don't know jack about networks.



On Mon, 4 May 2020, 02:04 Ravi Pullareddy, <ra...@minlog.com.au>
wrote:

> Hi Som
>
>
>
> Looks like you are getting confused with this issue. It is very different
> to what Jeff has explained there. Setting 0.0.0.0 is not the solution for
> everything. There was no issue with accessing Zeppelin from Windows as WSL
> is linux kernel running within Windows system. However, the issue arose
> when the interpreter server started. The interpreter creates a run time
> server when a job is submitted. The run time server gets the IpAddress of
> the host server by using Java.net.IpAddress and thereby was getting wrong
> IpAddress because of two network addresses on the machine.
>
>
>
> Let us say if a machine has multiple network cards and you want Zeppelin
> to be bound to a specific IpAddress then 0.0.0.0 is not the solution. One
> has to get a clear understanding of what we are trying to achieve. In any
> case from Jeff explanation, I got to understand that Zeppelin spins up
> runtime interpreter server that may not line up with Zeppelin server and
> that is the issue. Although in my instance changing it to 0.0.0.0 fixed the
> issue in other instances it might not be of help.
>
>
>
> Thanks
>
> Ravi Pullareddy
>
>
>
> *From:* Som Lima <so...@gmail.com>
> *Sent:* Monday, 4 May 2020 10:48 AM
> *To:* users@zeppelin.apache.org
> *Subject:* Re: Job manager URI rpc address:port
>
>
>
> Here are the instructions . Go to getting started link.
>
>
>
> It took me hours first time and I had many errors because I had spark
> installed earlier on same machine.
>
>
>
> Then when I switched flink remote from client to server and server to
> client
>
> it only took me five minutes because this time  it was clean install.
>
> On Sun, 19 Apr 2020, 14:58 Jeff Zhang, <zj...@gmail.com> wrote:
>
> Hi Som,
>
>
>
> You can take a look at flink on zeppelin, in zeppelin you can connect to a
> remote flink cluster via a few configuration, and you don't need to worry
> about the jars. Flink interpreter will ship necessary jars for you. Here's
> a list of tutorials.
>
>
>
> 1) Get started https://link.medium.com/oppqD6dIg5
> <https://t.co/PTouUYYTrv?amp=1> 2) Batch
> https://link.medium.com/3qumbwRIg5 <https://t.co/Yo9QAY0Joj?amp=1> 3)
> Streaming https://link.medium.com/RBHa2lTIg5
> <https://t.co/sUapN40tvI?amp=1> 4) Advanced usage
> https://link.medium.com/CAekyoXIg5 <https://t.co/MXolULmafZ?amp=1>
>
>
>
>
>
> Zahid Rahman <za...@gmail.com> 于2020年4月19日周日 下午7:27写道：
>
> Hi Tison,
>
>
>
> I think I may have found what I want in example 22.
>
>
> https://www.programcreek.com/java-api-examples/?api=org.apache.flink.configuration.Configuration
>
>
>
> I need to create Configuration object first as shown .
>
>
>
> Also I think  flink-conf.yaml file may contain configuration for client
> rather than  server. So before starting is irrelevant.
>
> I am going to play around and see but if the Configuration class allows me
> to set configuration programmatically and overrides the yaml file then that
> would be great.
>
>
>
>
>
>
>
> On Sun, 19 Apr 2020, 11:35 Som Lima, <so...@gmail.com> wrote:
>
> Thanks.
>
> flink-conf.yaml does allow me to do what I need to do without making any
> changes to client source code.
>
>
>
> But
>
> RemoteStreamEnvironment constructor  expects a jar file as the third
> parameter also.
>
>
>
> *RemoteStreamEnvironment
> <https://ci.apache.org/projects/flink/flink-docs-release-1.7/api/java/org/apache/flink/streaming/api/environment/RemoteStreamEnvironment.html#RemoteStreamEnvironment-java.lang.String-int-java.lang.String...->*
> (*String*
> <http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true> host,
> int port, *String*
> <http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true>
> ... jarFiles)
>
> Creates a new RemoteStreamEnvironment that points to the master
> (JobManager) described by the given host name and port.
>
>
>
> On Sun, 19 Apr 2020, 11:02 tison, <wa...@gmail.com> wrote:
>
> You can change flink-conf.yaml "jobmanager.address" or "jobmanager.port"
> options before run the program or take a look at RemoteStreamEnvironment
> which enables configuring host and port.
>
>
>
> Best,
>
> tison.
>
>
>
>
>
> Som Lima <so...@gmail.com> 于2020年4月19日周日 下午5:58写道：
>
> Hi,
>
>
>
> After running
>
> $ ./bin/start-cluster.sh
>
> The following line of code defaults jobmanager  to localhost:6123
>
>
>
> final  ExecutionEnvironment env = Environment.getExecutionEnvironment();
>
>
>
> which is same on spark.
>
>
>
> val spark =
> SparkSession.builder.master(local[*]).appname("anapp").getOrCreate
>
>
>
> However if I wish to run the servers on a different physical computer.
>
> Then in Spark I can do it this way using the spark URI in my IDE.
>
>
>
> Conf =
> SparkConf().setMaster("spark://<hostip>:<port>").setAppName("anapp")
>
>
>
> Can you please tell me the equivalent change to make so I can run my
> servers and my IDE from different physical computers.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Best Regards
>
> Jeff Zhang
>
>

RE: Job manager URI rpc address:port

Posted by Ravi Pullareddy <ra...@minlog.com.au>.

Zahid,

1. If there was no issue as you claim, Jeff would not have requested me
to raise a Jira ticket for this. Before you express your opinion, please go
and see ZEPPELIN-4794 <https://issues.apache.org/jira/browse/ZEPPELIN-4794>
and read the comments trail where Jeff is trying to help me out and is
finding a solution. The issue I have come across has been reported by Jeff
himself via issue ZEPPELIN-4216
<https://issues.apache.org/jira/browse/ZEPPELIN-4216> dated 1st July 2019.

3. The objective of users group is try and help to find solutions and
not to be argumentative… let us leave at that.

*From:* Zahid Rahman <za...@gmail.com>
*Sent:* Monday, 4 May 2020 1:34 PM
*To:* users@zeppelin.apache.org
*Subject:* Re: Job manager URI rpc address:port

I believe you owe me an apology I dont want your thanks.

Where is Jeff's advice. LIAR

I only see him giving you config properties based on your cock and bull
theories.

There're 2 properties you can set to control the port range:

zeppelin.server.rpc.portRange

zeppelin.interpreter.rpc.portRange

On Mon, 4 May 2020, 04:05 Ravi Pullareddy, <ra...@minlog.com.au>
wrote:

Guys

Thanks for your unsolicited advice.

Bye

*From:* Som Lima <so...@gmail.com>
*Sent:* Monday, 4 May 2020 12:46 PM
*To:* users@zeppelin.apache.org
*Subject:* Re: Job manager URI rpc address:port

Jeff did not know which flavour of linux I was using when he gave the
installation instructions.

That in itself should tell you your OS theory was silly.

On Mon, 4 May 2020, 02:04 Ravi Pullareddy, <ra...@minlog.com.au>
wrote:

Hi Som

Looks like you are getting confused with this issue. It is very different
to what Jeff has explained there. Setting 0.0.0.0 is not the solution for
everything. There was no issue with accessing Zeppelin from Windows as WSL
is linux kernel running within Windows system. However, the issue arose
when the interpreter server started. The interpreter creates a run time
server when a job is submitted. The run time server gets the IpAddress of
the host server by using Java.net.IpAddress and thereby was getting wrong
IpAddress because of two network addresses on the machine.

Let us say if a machine has multiple network cards and you want Zeppelin to
be bound to a specific IpAddress then 0.0.0.0 is not the solution. One has
to get a clear understanding of what we are trying to achieve. In any case
from Jeff explanation, I got to understand that Zeppelin spins up runtime
interpreter server that may not line up with Zeppelin server and that is
the issue. Although in my instance changing it to 0.0.0.0 fixed the issue
in other instances it might not be of help.

Thanks

Ravi Pullareddy

*From:* Som Lima <so...@gmail.com>
*Sent:* Monday, 4 May 2020 10:48 AM
*To:* users@zeppelin.apache.org
*Subject:* Re: Job manager URI rpc address:port

Here are the instructions . Go to getting started link.

It took me hours first time and I had many errors because I had spark
installed earlier on same machine.

Then when I switched flink remote from client to server and server to
client

it only took me five minutes because this time it was clean install.

On Sun, 19 Apr 2020, 14:58 Jeff Zhang, <zj...@gmail.com> wrote:

Hi Som,

You can take a look at flink on zeppelin, in zeppelin you can connect to a
remote flink cluster via a few configuration, and you don't need to worry
about the jars. Flink interpreter will ship necessary jars for you. Here's
a list of tutorials.

1) Get started https://link.medium.com/oppqD6dIg5
<https://t.co/PTouUYYTrv?amp=1> 2) Batch https://link.medium.com/3qumbwRIg5
<https://t.co/Yo9QAY0Joj?amp=1> 3) Streaming
https://link.medium.com/RBHa2lTIg5 <https://t.co/sUapN40tvI?amp=1> 4)
Advanced usage https://link.medium.com/CAekyoXIg5
<https://t.co/MXolULmafZ?amp=1>

Zahid Rahman <za...@gmail.com> 于2020年4月19日周日 下午7:27写道：

Hi Tison,

I think I may have found what I want in example 22.

https://www.programcreek.com/java-api-examples/?api=org.apache.flink.configuration.Configuration

I need to create Configuration object first as shown .

Also I think flink-conf.yaml file may contain configuration for client
rather than server. So before starting is irrelevant.

I am going to play around and see but if the Configuration class allows me
to set configuration programmatically and overrides the yaml file then that
would be great.

On Sun, 19 Apr 2020, 11:35 Som Lima, <so...@gmail.com> wrote:

Thanks.

flink-conf.yaml does allow me to do what I need to do without making any
changes to client source code.

But

RemoteStreamEnvironment constructor expects a jar file as the third
parameter also.

*RemoteStreamEnvironment
<https://ci.apache.org/projects/flink/flink-docs-release-1.7/api/java/org/apache/flink/streaming/api/environment/RemoteStreamEnvironment.html#RemoteStreamEnvironment-java.lang.String-int-java.lang.String...->*
(*String*
<http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true>
host,
int port, *String*
<http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true>
... jarFiles)

Creates a new RemoteStreamEnvironment that points to the master
(JobManager) described by the given host name and port.

On Sun, 19 Apr 2020, 11:02 tison, <wa...@gmail.com> wrote:

You can change flink-conf.yaml "jobmanager.address" or "jobmanager.port"
options before run the program or take a look at RemoteStreamEnvironment
which enables configuring host and port.

Best,

tison.

Som Lima <so...@gmail.com> 于2020年4月19日周日 下午5:58写道：

Hi,

After running

$ ./bin/start-cluster.sh

The following line of code defaults jobmanager to localhost:6123

final ExecutionEnvironment env = Environment.getExecutionEnvironment();

which is same on spark.

val spark =
SparkSession.builder.master(local[*]).appname("anapp").getOrCreate

However if I wish to run the servers on a different physical computer.

Then in Spark I can do it this way using the spark URI in my IDE.

Conf = SparkConf().setMaster("spark://<hostip>:<port>").setAppName("anapp")

Can you please tell me the equivalent change to make so I can run my
servers and my IDE from different physical computers.

Best Regards

Jeff Zhang

Re: Job manager URI rpc address:port

Posted by Zahid Rahman <za...@gmail.com>.

I believe  you owe me an apology  I dont want your thanks.

Where is Jeff's advice. LIAR

I only see him giving you config properties based on your cock and bull
theories.

There're 2 properties you can set to control the port range:

zeppelin.server.rpc.portRange

zeppelin.interpreter.rpc.portRange




On Mon, 4 May 2020, 04:05 Ravi Pullareddy, <ra...@minlog.com.au>
wrote:

> Guys
>
>
>
> Thanks for your unsolicited advice.
>
>
>
> Bye
>
>
>
> *From:* Som Lima <so...@gmail.com>
> *Sent:* Monday, 4 May 2020 12:46 PM
> *To:* users@zeppelin.apache.org
> *Subject:* Re: Job manager URI rpc address:port
>
>
>
> Jeff did not know which flavour of linux I was using when he gave the
> installation instructions.
>
>
>
> That in itself should tell you your OS theory was silly.
>
>
>
> On Mon, 4 May 2020, 02:04 Ravi Pullareddy, <ra...@minlog.com.au>
> wrote:
>
> Hi Som
>
>
>
> Looks like you are getting confused with this issue. It is very different
> to what Jeff has explained there. Setting 0.0.0.0 is not the solution for
> everything. There was no issue with accessing Zeppelin from Windows as WSL
> is linux kernel running within Windows system. However, the issue arose
> when the interpreter server started. The interpreter creates a run time
> server when a job is submitted. The run time server gets the IpAddress of
> the host server by using Java.net.IpAddress and thereby was getting wrong
> IpAddress because of two network addresses on the machine.
>
>
>
> Let us say if a machine has multiple network cards and you want Zeppelin
> to be bound to a specific IpAddress then 0.0.0.0 is not the solution. One
> has to get a clear understanding of what we are trying to achieve. In any
> case from Jeff explanation, I got to understand that Zeppelin spins up
> runtime interpreter server that may not line up with Zeppelin server and
> that is the issue. Although in my instance changing it to 0.0.0.0 fixed the
> issue in other instances it might not be of help.
>
>
>
> Thanks
>
> Ravi Pullareddy
>
>
>
> *From:* Som Lima <so...@gmail.com>
> *Sent:* Monday, 4 May 2020 10:48 AM
> *To:* users@zeppelin.apache.org
> *Subject:* Re: Job manager URI rpc address:port
>
>
>
> Here are the instructions . Go to getting started link.
>
>
>
> It took me hours first time and I had many errors because I had spark
> installed earlier on same machine.
>
>
>
> Then when I switched flink remote from client to server and server to
> client
>
> it only took me five minutes because this time  it was clean install.
>
> On Sun, 19 Apr 2020, 14:58 Jeff Zhang, <zj...@gmail.com> wrote:
>
> Hi Som,
>
>
>
> You can take a look at flink on zeppelin, in zeppelin you can connect to a
> remote flink cluster via a few configuration, and you don't need to worry
> about the jars. Flink interpreter will ship necessary jars for you. Here's
> a list of tutorials.
>
>
>
> 1) Get started https://link.medium.com/oppqD6dIg5
> <https://t.co/PTouUYYTrv?amp=1> 2) Batch
> https://link.medium.com/3qumbwRIg5 <https://t.co/Yo9QAY0Joj?amp=1> 3)
> Streaming https://link.medium.com/RBHa2lTIg5
> <https://t.co/sUapN40tvI?amp=1> 4) Advanced usage
> https://link.medium.com/CAekyoXIg5 <https://t.co/MXolULmafZ?amp=1>
>
>
>
>
>
> Zahid Rahman <za...@gmail.com> 于2020年4月19日周日 下午7:27写道：
>
> Hi Tison,
>
>
>
> I think I may have found what I want in example 22.
>
>
> https://www.programcreek.com/java-api-examples/?api=org.apache.flink.configuration.Configuration
>
>
>
> I need to create Configuration object first as shown .
>
>
>
> Also I think  flink-conf.yaml file may contain configuration for client
> rather than  server. So before starting is irrelevant.
>
> I am going to play around and see but if the Configuration class allows me
> to set configuration programmatically and overrides the yaml file then that
> would be great.
>
>
>
>
>
>
>
> On Sun, 19 Apr 2020, 11:35 Som Lima, <so...@gmail.com> wrote:
>
> Thanks.
>
> flink-conf.yaml does allow me to do what I need to do without making any
> changes to client source code.
>
>
>
> But
>
> RemoteStreamEnvironment constructor  expects a jar file as the third
> parameter also.
>
>
>
> *RemoteStreamEnvironment
> <https://ci.apache.org/projects/flink/flink-docs-release-1.7/api/java/org/apache/flink/streaming/api/environment/RemoteStreamEnvironment.html#RemoteStreamEnvironment-java.lang.String-int-java.lang.String...->*
> (*String*
> <http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true> host,
> int port, *String*
> <http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true>
> ... jarFiles)
>
> Creates a new RemoteStreamEnvironment that points to the master
> (JobManager) described by the given host name and port.
>
>
>
> On Sun, 19 Apr 2020, 11:02 tison, <wa...@gmail.com> wrote:
>
> You can change flink-conf.yaml "jobmanager.address" or "jobmanager.port"
> options before run the program or take a look at RemoteStreamEnvironment
> which enables configuring host and port.
>
>
>
> Best,
>
> tison.
>
>
>
>
>
> Som Lima <so...@gmail.com> 于2020年4月19日周日 下午5:58写道：
>
> Hi,
>
>
>
> After running
>
> $ ./bin/start-cluster.sh
>
> The following line of code defaults jobmanager  to localhost:6123
>
>
>
> final  ExecutionEnvironment env = Environment.getExecutionEnvironment();
>
>
>
> which is same on spark.
>
>
>
> val spark =
> SparkSession.builder.master(local[*]).appname("anapp").getOrCreate
>
>
>
> However if I wish to run the servers on a different physical computer.
>
> Then in Spark I can do it this way using the spark URI in my IDE.
>
>
>
> Conf =
> SparkConf().setMaster("spark://<hostip>:<port>").setAppName("anapp")
>
>
>
> Can you please tell me the equivalent change to make so I can run my
> servers and my IDE from different physical computers.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Best Regards
>
> Jeff Zhang
>
>

RE: Job manager URI rpc address:port

Posted by Ravi Pullareddy <ra...@minlog.com.au>.

Guys

Thanks for your unsolicited advice.

Bye

*From:* Som Lima <so...@gmail.com>
*Sent:* Monday, 4 May 2020 12:46 PM
*To:* users@zeppelin.apache.org
*Subject:* Re: Job manager URI rpc address:port

Jeff did not know which flavour of linux I was using when he gave the
installation instructions.

That in itself should tell you your OS theory was silly.

On Mon, 4 May 2020, 02:04 Ravi Pullareddy, <ra...@minlog.com.au>
wrote:

Hi Som

Thanks

Ravi Pullareddy

*From:* Som Lima <so...@gmail.com>
*Sent:* Monday, 4 May 2020 10:48 AM
*To:* users@zeppelin.apache.org
*Subject:* Re: Job manager URI rpc address:port

Here are the instructions . Go to getting started link.

It took me hours first time and I had many errors because I had spark
installed earlier on same machine.

Then when I switched flink remote from client to server and server to
client

it only took me five minutes because this time it was clean install.

On Sun, 19 Apr 2020, 14:58 Jeff Zhang, <zj...@gmail.com> wrote:

Hi Som,

Zahid Rahman <za...@gmail.com> 于2020年4月19日周日 下午7:27写道：

Hi Tison,

I think I may have found what I want in example 22.

https://www.programcreek.com/java-api-examples/?api=org.apache.flink.configuration.Configuration

I need to create Configuration object first as shown .

Also I think flink-conf.yaml file may contain configuration for client
rather than server. So before starting is irrelevant.

I am going to play around and see but if the Configuration class allows me
to set configuration programmatically and overrides the yaml file then that
would be great.

On Sun, 19 Apr 2020, 11:35 Som Lima, <so...@gmail.com> wrote:

Thanks.

flink-conf.yaml does allow me to do what I need to do without making any
changes to client source code.

But

RemoteStreamEnvironment constructor expects a jar file as the third
parameter also.

Creates a new RemoteStreamEnvironment that points to the master
(JobManager) described by the given host name and port.

On Sun, 19 Apr 2020, 11:02 tison, <wa...@gmail.com> wrote:

You can change flink-conf.yaml "jobmanager.address" or "jobmanager.port"
options before run the program or take a look at RemoteStreamEnvironment
which enables configuring host and port.

Best,

tison.

Som Lima <so...@gmail.com> 于2020年4月19日周日 下午5:58写道：

Hi,

After running

$ ./bin/start-cluster.sh

The following line of code defaults jobmanager to localhost:6123

final ExecutionEnvironment env = Environment.getExecutionEnvironment();

which is same on spark.

val spark =
SparkSession.builder.master(local[*]).appname("anapp").getOrCreate

However if I wish to run the servers on a different physical computer.

Then in Spark I can do it this way using the spark URI in my IDE.

Conf = SparkConf().setMaster("spark://<hostip>:<port>").setAppName("anapp")

Can you please tell me the equivalent change to make so I can run my
servers and my IDE from different physical computers.

Best Regards

Jeff Zhang

Re: Job manager URI rpc address:port

Posted by Som Lima <so...@gmail.com>.

Jeff did not know which flavour of linux I was using when he gave the
installation instructions.

That in itself should tell you your OS theory was silly.

On Mon, 4 May 2020, 02:04 Ravi Pullareddy, <ra...@minlog.com.au>
wrote:

> Hi Som
>
>
>
> Looks like you are getting confused with this issue. It is very different
> to what Jeff has explained there. Setting 0.0.0.0 is not the solution for
> everything. There was no issue with accessing Zeppelin from Windows as WSL
> is linux kernel running within Windows system. However, the issue arose
> when the interpreter server started. The interpreter creates a run time
> server when a job is submitted. The run time server gets the IpAddress of
> the host server by using Java.net.IpAddress and thereby was getting wrong
> IpAddress because of two network addresses on the machine.
>
>
>
> Let us say if a machine has multiple network cards and you want Zeppelin
> to be bound to a specific IpAddress then 0.0.0.0 is not the solution. One
> has to get a clear understanding of what we are trying to achieve. In any
> case from Jeff explanation, I got to understand that Zeppelin spins up
> runtime interpreter server that may not line up with Zeppelin server and
> that is the issue. Although in my instance changing it to 0.0.0.0 fixed the
> issue in other instances it might not be of help.
>
>
>
> Thanks
>
> Ravi Pullareddy
>
>
>
> *From:* Som Lima <so...@gmail.com>
> *Sent:* Monday, 4 May 2020 10:48 AM
> *To:* users@zeppelin.apache.org
> *Subject:* Re: Job manager URI rpc address:port
>
>
>
> Here are the instructions . Go to getting started link.
>
>
>
> It took me hours first time and I had many errors because I had spark
> installed earlier on same machine.
>
>
>
> Then when I switched flink remote from client to server and server to
> client
>
> it only took me five minutes because this time  it was clean install.
>
> On Sun, 19 Apr 2020, 14:58 Jeff Zhang, <zj...@gmail.com> wrote:
>
> Hi Som,
>
>
>
> You can take a look at flink on zeppelin, in zeppelin you can connect to a
> remote flink cluster via a few configuration, and you don't need to worry
> about the jars. Flink interpreter will ship necessary jars for you. Here's
> a list of tutorials.
>
>
>
> 1) Get started https://link.medium.com/oppqD6dIg5
> <https://t.co/PTouUYYTrv?amp=1> 2) Batch
> https://link.medium.com/3qumbwRIg5 <https://t.co/Yo9QAY0Joj?amp=1> 3)
> Streaming https://link.medium.com/RBHa2lTIg5
> <https://t.co/sUapN40tvI?amp=1> 4) Advanced usage
> https://link.medium.com/CAekyoXIg5 <https://t.co/MXolULmafZ?amp=1>
>
>
>
>
>
> Zahid Rahman <za...@gmail.com> 于2020年4月19日周日 下午7:27写道：
>
> Hi Tison,
>
>
>
> I think I may have found what I want in example 22.
>
>
> https://www.programcreek.com/java-api-examples/?api=org.apache.flink.configuration.Configuration
>
>
>
> I need to create Configuration object first as shown .
>
>
>
> Also I think  flink-conf.yaml file may contain configuration for client
> rather than  server. So before starting is irrelevant.
>
> I am going to play around and see but if the Configuration class allows me
> to set configuration programmatically and overrides the yaml file then that
> would be great.
>
>
>
>
>
>
>
> On Sun, 19 Apr 2020, 11:35 Som Lima, <so...@gmail.com> wrote:
>
> Thanks.
>
> flink-conf.yaml does allow me to do what I need to do without making any
> changes to client source code.
>
>
>
> But
>
> RemoteStreamEnvironment constructor  expects a jar file as the third
> parameter also.
>
>
>
> *RemoteStreamEnvironment
> <https://ci.apache.org/projects/flink/flink-docs-release-1.7/api/java/org/apache/flink/streaming/api/environment/RemoteStreamEnvironment.html#RemoteStreamEnvironment-java.lang.String-int-java.lang.String...->*
> (*String*
> <http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true> host,
> int port, *String*
> <http://docs.oracle.com/javase/7/docs/api/java/lang/String.html?is-external=true>
> ... jarFiles)
>
> Creates a new RemoteStreamEnvironment that points to the master
> (JobManager) described by the given host name and port.
>
>
>
> On Sun, 19 Apr 2020, 11:02 tison, <wa...@gmail.com> wrote:
>
> You can change flink-conf.yaml "jobmanager.address" or "jobmanager.port"
> options before run the program or take a look at RemoteStreamEnvironment
> which enables configuring host and port.
>
>
>
> Best,
>
> tison.
>
>
>
>
>
> Som Lima <so...@gmail.com> 于2020年4月19日周日 下午5:58写道：
>
> Hi,
>
>
>
> After running
>
> $ ./bin/start-cluster.sh
>
> The following line of code defaults jobmanager  to localhost:6123
>
>
>
> final  ExecutionEnvironment env = Environment.getExecutionEnvironment();
>
>
>
> which is same on spark.
>
>
>
> val spark =
> SparkSession.builder.master(local[*]).appname("anapp").getOrCreate
>
>
>
> However if I wish to run the servers on a different physical computer.
>
> Then in Spark I can do it this way using the spark URI in my IDE.
>
>
>
> Conf =
> SparkConf().setMaster("spark://<hostip>:<port>").setAppName("anapp")
>
>
>
> Can you please tell me the equivalent change to make so I can run my
> servers and my IDE from different physical computers.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
>
> Best Regards
>
> Jeff Zhang
>
>

RE: Job manager URI rpc address:port

Posted by Ravi Pullareddy <ra...@minlog.com.au>.

Hi Som

Thanks

Ravi Pullareddy

*From:* Som Lima <so...@gmail.com>
*Sent:* Monday, 4 May 2020 10:48 AM
*To:* users@zeppelin.apache.org
*Subject:* Re: Job manager URI rpc address:port

Here are the instructions . Go to getting started link.

It took me hours first time and I had many errors because I had spark
installed earlier on same machine.

Then when I switched flink remote from client to server and server to
client

it only took me five minutes because this time it was clean install.

On Sun, 19 Apr 2020, 14:58 Jeff Zhang, <zj...@gmail.com> wrote:

Hi Som,

Zahid Rahman <za...@gmail.com> 于2020年4月19日周日 下午7:27写道：

Hi Tison,

I think I may have found what I want in example 22.

https://www.programcreek.com/java-api-examples/?api=org.apache.flink.configuration.Configuration

I need to create Configuration object first as shown .

Also I think flink-conf.yaml file may contain configuration for client
rather than server. So before starting is irrelevant.

I am going to play around and see but if the Configuration class allows me
to set configuration programmatically and overrides the yaml file then that
would be great.

On Sun, 19 Apr 2020, 11:35 Som Lima, <so...@gmail.com> wrote:

Thanks.

flink-conf.yaml does allow me to do what I need to do without making any
changes to client source code.

But

RemoteStreamEnvironment constructor expects a jar file as the third
parameter also.

Creates a new RemoteStreamEnvironment that points to the master
(JobManager) described by the given host name and port.

On Sun, 19 Apr 2020, 11:02 tison, <wa...@gmail.com> wrote:

You can change flink-conf.yaml "jobmanager.address" or "jobmanager.port"
options before run the program or take a look at RemoteStreamEnvironment
which enables configuring host and port.

Best,

tison.

Som Lima <so...@gmail.com> 于2020年4月19日周日 下午5:58写道：

Hi,

After running

$ ./bin/start-cluster.sh

The following line of code defaults jobmanager to localhost:6123

final ExecutionEnvironment env = Environment.getExecutionEnvironment();

which is same on spark.

val spark =
SparkSession.builder.master(local[*]).appname("anapp").getOrCreate

However if I wish to run the servers on a different physical computer.

Then in Spark I can do it this way using the spark URI in my IDE.

Conf = SparkConf().setMaster("spark://<hostip>:<port>").setAppName("anapp")

Can you please tell me the equivalent change to make so I can run my
servers and my IDE from different physical computers.

Best Regards

Jeff Zhang