You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by John Omernik <jo...@omernik.com> on 2016/02/10 17:54:12 UTC

Ports Drill Uses

Using https://drill.apache.org/docs/ports-used-by-drill/ as  start, I was
wondering if it is possible in the drill-overide.conf to set these ports to
be a specific port?

Do all ports on each drillbit have to be same from bit to bit in a specific
drill cluster?  I.e. if I set the ports, that will be the ports used on a
bits for that cluster? I suppose at least one port must be the same on each
bit per drill cluster, otherwise they wouldn't know how to find. Either
way, is it possible to set these to be different values?

Thanks!

John

Re: Ports Drill Uses

Posted by John Omernik <jo...@omernik.com>.
Ah ok,

so by setting drill.exec.rpc.bit.server.port to 20002, the "data" port did
that + 1


In summary
drill.exec -
http.port: 20000
rpc.user.server.port: 20001
rpc:bit:server.port: 20002

resulted in

web:20000
user: 20001
control: 20002
data: 20003





On Fri, Feb 12, 2016 at 11:37 AM, John Omernik <jo...@omernik.com> wrote:

> I found drill.exec.rpc.bit.server.port  in the over-example, I will set
> that to 20002  now to find the "Data" port
>
>
>
> On Fri, Feb 12, 2016 at 11:34 AM, John Omernik <jo...@omernik.com> wrote:
>
>> Jacques -
>>
>> I don't the behavior works how I interpreted your explanation
>>
>> I set drill.exec.http.port to be 20000, and
>> drill.exec.rpc.user.server.port to be 20001
>>
>> I expected
>>
>> web:20000
>> user:20001
>> control:20002
>> data:20003
>>
>> based on your summary.
>>
>> What I got was
>>
>> web:20000
>> user:20001
>> control:31011
>> data:31012
>>
>>
>> Could there be two more settings, drill.exec.rpc.control.server.port and
>> drill.exec.rpc.data.server.port?
>>
>>
>>
>>
>> On Thu, Feb 11, 2016 at 2:25 PM, Bob Rumsby <br...@maprtech.com> wrote:
>>
>>> Bridget is on vacation but will pick this up when she gets back next
>>> week.
>>> Kristine left the company in January.
>>>
>>> Bob
>>>
>>> On Thu, Feb 11, 2016 at 12:24 PM, John Omernik <jo...@omernik.com> wrote:
>>>
>>> > Awesome.  I would say that the page can stay on the Ports page, and
>>> just
>>> > removing the UDP thing, and then documenting those two settings and
>>> > examples of what that means (the start port) I can try to come up with
>>> > something, Is this something Kristine would want to jump on just based
>>> on
>>> > this thread or should I formalize something and let her handle it from
>>> > there?
>>> >
>>> > On Thu, Feb 11, 2016 at 1:42 PM, Jacques Nadeau <ja...@dremio.com>
>>> > wrote:
>>> >
>>> > > >
>>> > > >
>>> > > > Awesome.   Is there a doc page on this?
>>> > >
>>> > >
>>> > > There should be. Sounds like a great contribution :)
>>> > >
>>> > >
>>> > > > 1. "Starts" at, that indicates if I set
>>> drill.exec.rpc.user.server.port
>>> > > to
>>> > > > say 50000, then instead of 31010, 31011, and 31012 (per the doc
>>> page)
>>> > > then
>>> > > > it would use 50000, 50001, 50002?
>>> > > >
>>> > >
>>> > > yep
>>> > >
>>> > >
>>> > > > 2.  What about the UDP Port it uses (46655 UDP Used for JGroups and
>>> > > > Infinispan. Needed for multi-node installation of Apache Drill.)?
>>> > > >
>>> > >
>>> > > We no longer use JGroups or Infinispan. No UDP needed.
>>> > >
>>> > >
>>> > > > 3.  So this can be on per node basis, in that when it registers
>>> with
>>> > the
>>> > > > cluster defined in drill-override.conf, it registers it's ports so
>>> the
>>> > > > other nodes are aware of the ports without any other sort of
>>> config?
>>> > > (I.e.
>>> > > > I can have bit one start on port 31010 by default, and then have
>>> bit
>>> > two
>>> > > > start at 50000, etc)
>>> > > >
>>> > >
>>> > > Correct. Each node registers its information in zookeeper.
>>> > >
>>> > >
>>> > > > 4. Are there any odd issues that people have seen by using
>>> non-standard
>>> > > > ports or is it built to handle that fairly easily?
>>> > > >
>>> > >
>>> > > I don't think it has been used a lot beyond unit tests where we want
>>> to
>>> > run
>>> > > multiple Drillbits on the same node. The mechanism is fairly simple
>>> so I
>>> > > would expect major issues.
>>> > >
>>> >
>>>
>>
>>
>

Re: Ports Drill Uses

Posted by John Omernik <jo...@omernik.com>.
I found drill.exec.rpc.bit.server.port  in the over-example, I will set
that to 20002  now to find the "Data" port



On Fri, Feb 12, 2016 at 11:34 AM, John Omernik <jo...@omernik.com> wrote:

> Jacques -
>
> I don't the behavior works how I interpreted your explanation
>
> I set drill.exec.http.port to be 20000, and
> drill.exec.rpc.user.server.port to be 20001
>
> I expected
>
> web:20000
> user:20001
> control:20002
> data:20003
>
> based on your summary.
>
> What I got was
>
> web:20000
> user:20001
> control:31011
> data:31012
>
>
> Could there be two more settings, drill.exec.rpc.control.server.port and
> drill.exec.rpc.data.server.port?
>
>
>
>
> On Thu, Feb 11, 2016 at 2:25 PM, Bob Rumsby <br...@maprtech.com> wrote:
>
>> Bridget is on vacation but will pick this up when she gets back next week.
>> Kristine left the company in January.
>>
>> Bob
>>
>> On Thu, Feb 11, 2016 at 12:24 PM, John Omernik <jo...@omernik.com> wrote:
>>
>> > Awesome.  I would say that the page can stay on the Ports page, and just
>> > removing the UDP thing, and then documenting those two settings and
>> > examples of what that means (the start port) I can try to come up with
>> > something, Is this something Kristine would want to jump on just based
>> on
>> > this thread or should I formalize something and let her handle it from
>> > there?
>> >
>> > On Thu, Feb 11, 2016 at 1:42 PM, Jacques Nadeau <ja...@dremio.com>
>> > wrote:
>> >
>> > > >
>> > > >
>> > > > Awesome.   Is there a doc page on this?
>> > >
>> > >
>> > > There should be. Sounds like a great contribution :)
>> > >
>> > >
>> > > > 1. "Starts" at, that indicates if I set
>> drill.exec.rpc.user.server.port
>> > > to
>> > > > say 50000, then instead of 31010, 31011, and 31012 (per the doc
>> page)
>> > > then
>> > > > it would use 50000, 50001, 50002?
>> > > >
>> > >
>> > > yep
>> > >
>> > >
>> > > > 2.  What about the UDP Port it uses (46655 UDP Used for JGroups and
>> > > > Infinispan. Needed for multi-node installation of Apache Drill.)?
>> > > >
>> > >
>> > > We no longer use JGroups or Infinispan. No UDP needed.
>> > >
>> > >
>> > > > 3.  So this can be on per node basis, in that when it registers with
>> > the
>> > > > cluster defined in drill-override.conf, it registers it's ports so
>> the
>> > > > other nodes are aware of the ports without any other sort of config?
>> > > (I.e.
>> > > > I can have bit one start on port 31010 by default, and then have bit
>> > two
>> > > > start at 50000, etc)
>> > > >
>> > >
>> > > Correct. Each node registers its information in zookeeper.
>> > >
>> > >
>> > > > 4. Are there any odd issues that people have seen by using
>> non-standard
>> > > > ports or is it built to handle that fairly easily?
>> > > >
>> > >
>> > > I don't think it has been used a lot beyond unit tests where we want
>> to
>> > run
>> > > multiple Drillbits on the same node. The mechanism is fairly simple
>> so I
>> > > would expect major issues.
>> > >
>> >
>>
>
>

Re: Ports Drill Uses

Posted by John Omernik <jo...@omernik.com>.
Jacques -

I don't the behavior works how I interpreted your explanation

I set drill.exec.http.port to be 20000, and drill.exec.rpc.user.server.port
to be 20001

I expected

web:20000
user:20001
control:20002
data:20003

based on your summary.

What I got was

web:20000
user:20001
control:31011
data:31012


Could there be two more settings, drill.exec.rpc.control.server.port and
drill.exec.rpc.data.server.port?




On Thu, Feb 11, 2016 at 2:25 PM, Bob Rumsby <br...@maprtech.com> wrote:

> Bridget is on vacation but will pick this up when she gets back next week.
> Kristine left the company in January.
>
> Bob
>
> On Thu, Feb 11, 2016 at 12:24 PM, John Omernik <jo...@omernik.com> wrote:
>
> > Awesome.  I would say that the page can stay on the Ports page, and just
> > removing the UDP thing, and then documenting those two settings and
> > examples of what that means (the start port) I can try to come up with
> > something, Is this something Kristine would want to jump on just based on
> > this thread or should I formalize something and let her handle it from
> > there?
> >
> > On Thu, Feb 11, 2016 at 1:42 PM, Jacques Nadeau <ja...@dremio.com>
> > wrote:
> >
> > > >
> > > >
> > > > Awesome.   Is there a doc page on this?
> > >
> > >
> > > There should be. Sounds like a great contribution :)
> > >
> > >
> > > > 1. "Starts" at, that indicates if I set
> drill.exec.rpc.user.server.port
> > > to
> > > > say 50000, then instead of 31010, 31011, and 31012 (per the doc page)
> > > then
> > > > it would use 50000, 50001, 50002?
> > > >
> > >
> > > yep
> > >
> > >
> > > > 2.  What about the UDP Port it uses (46655 UDP Used for JGroups and
> > > > Infinispan. Needed for multi-node installation of Apache Drill.)?
> > > >
> > >
> > > We no longer use JGroups or Infinispan. No UDP needed.
> > >
> > >
> > > > 3.  So this can be on per node basis, in that when it registers with
> > the
> > > > cluster defined in drill-override.conf, it registers it's ports so
> the
> > > > other nodes are aware of the ports without any other sort of config?
> > > (I.e.
> > > > I can have bit one start on port 31010 by default, and then have bit
> > two
> > > > start at 50000, etc)
> > > >
> > >
> > > Correct. Each node registers its information in zookeeper.
> > >
> > >
> > > > 4. Are there any odd issues that people have seen by using
> non-standard
> > > > ports or is it built to handle that fairly easily?
> > > >
> > >
> > > I don't think it has been used a lot beyond unit tests where we want to
> > run
> > > multiple Drillbits on the same node. The mechanism is fairly simple so
> I
> > > would expect major issues.
> > >
> >
>

Re: Ports Drill Uses

Posted by Bob Rumsby <br...@maprtech.com>.
Bridget is on vacation but will pick this up when she gets back next week.
Kristine left the company in January.

Bob

On Thu, Feb 11, 2016 at 12:24 PM, John Omernik <jo...@omernik.com> wrote:

> Awesome.  I would say that the page can stay on the Ports page, and just
> removing the UDP thing, and then documenting those two settings and
> examples of what that means (the start port) I can try to come up with
> something, Is this something Kristine would want to jump on just based on
> this thread or should I formalize something and let her handle it from
> there?
>
> On Thu, Feb 11, 2016 at 1:42 PM, Jacques Nadeau <ja...@dremio.com>
> wrote:
>
> > >
> > >
> > > Awesome.   Is there a doc page on this?
> >
> >
> > There should be. Sounds like a great contribution :)
> >
> >
> > > 1. "Starts" at, that indicates if I set drill.exec.rpc.user.server.port
> > to
> > > say 50000, then instead of 31010, 31011, and 31012 (per the doc page)
> > then
> > > it would use 50000, 50001, 50002?
> > >
> >
> > yep
> >
> >
> > > 2.  What about the UDP Port it uses (46655 UDP Used for JGroups and
> > > Infinispan. Needed for multi-node installation of Apache Drill.)?
> > >
> >
> > We no longer use JGroups or Infinispan. No UDP needed.
> >
> >
> > > 3.  So this can be on per node basis, in that when it registers with
> the
> > > cluster defined in drill-override.conf, it registers it's ports so the
> > > other nodes are aware of the ports without any other sort of config?
> > (I.e.
> > > I can have bit one start on port 31010 by default, and then have bit
> two
> > > start at 50000, etc)
> > >
> >
> > Correct. Each node registers its information in zookeeper.
> >
> >
> > > 4. Are there any odd issues that people have seen by using non-standard
> > > ports or is it built to handle that fairly easily?
> > >
> >
> > I don't think it has been used a lot beyond unit tests where we want to
> run
> > multiple Drillbits on the same node. The mechanism is fairly simple so I
> > would expect major issues.
> >
>

Re: Ports Drill Uses

Posted by John Omernik <jo...@omernik.com>.
Awesome.  I would say that the page can stay on the Ports page, and just
removing the UDP thing, and then documenting those two settings and
examples of what that means (the start port) I can try to come up with
something, Is this something Kristine would want to jump on just based on
this thread or should I formalize something and let her handle it from
there?

On Thu, Feb 11, 2016 at 1:42 PM, Jacques Nadeau <ja...@dremio.com> wrote:

> >
> >
> > Awesome.   Is there a doc page on this?
>
>
> There should be. Sounds like a great contribution :)
>
>
> > 1. "Starts" at, that indicates if I set drill.exec.rpc.user.server.port
> to
> > say 50000, then instead of 31010, 31011, and 31012 (per the doc page)
> then
> > it would use 50000, 50001, 50002?
> >
>
> yep
>
>
> > 2.  What about the UDP Port it uses (46655 UDP Used for JGroups and
> > Infinispan. Needed for multi-node installation of Apache Drill.)?
> >
>
> We no longer use JGroups or Infinispan. No UDP needed.
>
>
> > 3.  So this can be on per node basis, in that when it registers with the
> > cluster defined in drill-override.conf, it registers it's ports so the
> > other nodes are aware of the ports without any other sort of config?
> (I.e.
> > I can have bit one start on port 31010 by default, and then have bit two
> > start at 50000, etc)
> >
>
> Correct. Each node registers its information in zookeeper.
>
>
> > 4. Are there any odd issues that people have seen by using non-standard
> > ports or is it built to handle that fairly easily?
> >
>
> I don't think it has been used a lot beyond unit tests where we want to run
> multiple Drillbits on the same node. The mechanism is fairly simple so I
> would expect major issues.
>

Re: Ports Drill Uses

Posted by Jacques Nadeau <ja...@dremio.com>.
>
>
> Awesome.   Is there a doc page on this?


There should be. Sounds like a great contribution :)


> 1. "Starts" at, that indicates if I set drill.exec.rpc.user.server.port to
> say 50000, then instead of 31010, 31011, and 31012 (per the doc page) then
> it would use 50000, 50001, 50002?
>

yep


> 2.  What about the UDP Port it uses (46655 UDP Used for JGroups and
> Infinispan. Needed for multi-node installation of Apache Drill.)?
>

We no longer use JGroups or Infinispan. No UDP needed.


> 3.  So this can be on per node basis, in that when it registers with the
> cluster defined in drill-override.conf, it registers it's ports so the
> other nodes are aware of the ports without any other sort of config? (I.e.
> I can have bit one start on port 31010 by default, and then have bit two
> start at 50000, etc)
>

Correct. Each node registers its information in zookeeper.


> 4. Are there any odd issues that people have seen by using non-standard
> ports or is it built to handle that fairly easily?
>

I don't think it has been used a lot beyond unit tests where we want to run
multiple Drillbits on the same node. The mechanism is fairly simple so I
would expect major issues.

Re: Ports Drill Uses

Posted by John Omernik <jo...@omernik.com>.
Awesome.   Is there a doc page on this? Some questions I have below, if we
don't have a doc page on this, could be helpful.

1. "Starts" at, that indicates if I set drill.exec.rpc.user.server.port to
say 50000, then instead of 31010, 31011, and 31012 (per the doc page) then
it would use 50000, 50001, 50002?
2.  What about the UDP Port it uses (46655 UDP Used for JGroups and
Infinispan. Needed for multi-node installation of Apache Drill.)?
3.  So this can be on per node basis, in that when it registers with the
cluster defined in drill-override.conf, it registers it's ports so the
other nodes are aware of the ports without any other sort of config? (I.e.
I can have bit one start on port 31010 by default, and then have bit two
start at 50000, etc)
4. Are there any odd issues that people have seen by using non-standard
ports or is it built to handle that fairly easily?


Thanks for the answers!

John


On Thu, Feb 11, 2016 at 1:37 AM, Jacques Nadeau <ja...@dremio.com> wrote:

> You can change the ports that Drill starts at from 31010 to something else
> by setting:
>
> drill.exec.rpc.user.server.port
>
> You can change the HTTP port by setting:
>
> drill.exec.http.port
>
> Both of these are configured in drill-override.conf on a per node basis.
> You can also disable the web server on a particular node by setting
> drill.exec.http.enabled to false.
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Wed, Feb 10, 2016 at 8:54 AM, John Omernik <jo...@omernik.com> wrote:
>
> > Using https://drill.apache.org/docs/ports-used-by-drill/ as  start, I
> was
> > wondering if it is possible in the drill-overide.conf to set these ports
> to
> > be a specific port?
> >
> > Do all ports on each drillbit have to be same from bit to bit in a
> specific
> > drill cluster?  I.e. if I set the ports, that will be the ports used on a
> > bits for that cluster? I suppose at least one port must be the same on
> each
> > bit per drill cluster, otherwise they wouldn't know how to find. Either
> > way, is it possible to set these to be different values?
> >
> > Thanks!
> >
> > John
> >
>

Re: Ports Drill Uses

Posted by Jacques Nadeau <ja...@dremio.com>.
You can change the ports that Drill starts at from 31010 to something else
by setting:

drill.exec.rpc.user.server.port

You can change the HTTP port by setting:

drill.exec.http.port

Both of these are configured in drill-override.conf on a per node basis.
You can also disable the web server on a particular node by setting
drill.exec.http.enabled to false.

--
Jacques Nadeau
CTO and Co-Founder, Dremio

On Wed, Feb 10, 2016 at 8:54 AM, John Omernik <jo...@omernik.com> wrote:

> Using https://drill.apache.org/docs/ports-used-by-drill/ as  start, I was
> wondering if it is possible in the drill-overide.conf to set these ports to
> be a specific port?
>
> Do all ports on each drillbit have to be same from bit to bit in a specific
> drill cluster?  I.e. if I set the ports, that will be the ports used on a
> bits for that cluster? I suppose at least one port must be the same on each
> bit per drill cluster, otherwise they wouldn't know how to find. Either
> way, is it possible to set these to be different values?
>
> Thanks!
>
> John
>