You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Divya Gehlot <di...@gmail.com> on 2018/07/04 04:00:41 UTC
need understanding of drill affinity configurations
Hi,
I would like to understand below configurations :
work: {
max.width.per.endpoint: 5,
global.max.width: 100,
affinity.factor: 1.2,
executor.threads: 4
},
As I am getting below error at time in some of the Drill bits :
2018-07-03 04:08:45,725 [pool-245-thread-13] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
host <DNSAddress> Skipping affinity to that host.
2018-07-03 04:08:45,731 [pool-245-thread-11] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
host <DNSAddress> Skipping affinity to that host.
2018-07-03 04:08:45,732 [pool-245-thread-12] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
host <DNSAddress> Skipping affinity to that host.
2018-07-03 04:08:45,732 [pool-245-thread-14] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
host <DNSAddress> Skipping affinity to that host.
2018-07-03 04:08:45,733 [pool-245-thread-3] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
host <DNSAddress> Skipping affinity to that host.
2018-07-03 04:08:45,733 [pool-245-thread-16] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
host <DNSAddress> Skipping affinity to that host.
2018-07-03 04:08:45,735 [pool-245-thread-15] INFO
o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
host <DNSAddress> Skipping affinity to that host.
Is there any relationship between affinity configurations and Drill logs ?
Appreciate if anybody can help me understand !
Thanks,
Divya
Re: need understanding of drill affinity configurations
Posted by Padma Penumarthy <pe...@gmail.com>.
Did you find anything in the logs which might indicate why the drillbit is
down or not responding ?
Try jstack on the drillbit process to see what it is busy with when they
are shown as
down in web UI.
Thanks
Padma
On Tue, Jul 3, 2018 at 11:22 PM, Divya Gehlot <di...@gmail.com>
wrote:
> Thanks Padma !
> The issue is strangely some of the drillbits goes down but when I check
> the processes, drillbit process would be running.
> But when check in drillbit web UI it doesn't display those down drillbits .
> I want to debug ,what's causing the issue as the process is still running.
> To make the drillbit run again I have to kill the process and restart the
> drillbits again on the impacted nodes and it's happening frequently .
> How can I debug what's making drillbit down on web UI whereas the process
> still running ?
>
> Thanks,
> Divya
>
> On Wed, 4 Jul 2018 at 12:52, Padma Penumarthy <pe...@gmail.com>
> wrote:
>
> > max.width.per.endpoint - how many minor fragments you can have per major
> > fragment on a single node
> >
> > global.max.width - max number of minor fragments per major fragment you
> can
> > have across all nodes
> >
> > affinity.factor - When deciding how many minor fragments to schedule on
> > each node,
> > this is the factor by which you favor nodes which have data locality vs.
> > nodes which do not.
> >
> > executor.threads - Seems like maximum number of threads we can have ? I
> am
> > not very sure about this one.
> >
> > These messages are debug messages, not errors.
> >
> > So, what is the problem you have ?
> >
> > Thanks
> > Padma
> >
> >
> >
> > On Tue, Jul 3, 2018 at 9:00 PM, Divya Gehlot <di...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I would like to understand below configurations :
> > > work: {
> > > max.width.per.endpoint: 5,
> > > global.max.width: 100,
> > > affinity.factor: 1.2,
> > > executor.threads: 4
> > > },
> > >
> > > As I am getting below error at time in some of the Drill bits :
> > > 2018-07-03 04:08:45,725 [pool-245-thread-13] INFO
> > > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running
> on
> > > host <DNSAddress> Skipping affinity to that host.
> > > 2018-07-03 04:08:45,731 [pool-245-thread-11] INFO
> > > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running
> on
> > > host <DNSAddress> Skipping affinity to that host.
> > > 2018-07-03 04:08:45,732 [pool-245-thread-12] INFO
> > > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running
> on
> > > host <DNSAddress> Skipping affinity to that host.
> > > 2018-07-03 04:08:45,732 [pool-245-thread-14] INFO
> > > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running
> on
> > > host <DNSAddress> Skipping affinity to that host.
> > > 2018-07-03 04:08:45,733 [pool-245-thread-3] INFO
> > > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running
> on
> > > host <DNSAddress> Skipping affinity to that host.
> > > 2018-07-03 04:08:45,733 [pool-245-thread-16] INFO
> > > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running
> on
> > > host <DNSAddress> Skipping affinity to that host.
> > > 2018-07-03 04:08:45,735 [pool-245-thread-15] INFO
> > > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running
> on
> > > host <DNSAddress> Skipping affinity to that host.
> > >
> > > Is there any relationship between affinity configurations and Drill
> logs
> > ?
> > > Appreciate if anybody can help me understand !
> > >
> > > Thanks,
> > > Divya
> > >
> >
>
Re: need understanding of drill affinity configurations
Posted by Divya Gehlot <di...@gmail.com>.
Thanks Padma !
The issue is strangely some of the drillbits goes down but when I check
the processes, drillbit process would be running.
But when check in drillbit web UI it doesn't display those down drillbits .
I want to debug ,what's causing the issue as the process is still running.
To make the drillbit run again I have to kill the process and restart the
drillbits again on the impacted nodes and it's happening frequently .
How can I debug what's making drillbit down on web UI whereas the process
still running ?
Thanks,
Divya
On Wed, 4 Jul 2018 at 12:52, Padma Penumarthy <pe...@gmail.com>
wrote:
> max.width.per.endpoint - how many minor fragments you can have per major
> fragment on a single node
>
> global.max.width - max number of minor fragments per major fragment you can
> have across all nodes
>
> affinity.factor - When deciding how many minor fragments to schedule on
> each node,
> this is the factor by which you favor nodes which have data locality vs.
> nodes which do not.
>
> executor.threads - Seems like maximum number of threads we can have ? I am
> not very sure about this one.
>
> These messages are debug messages, not errors.
>
> So, what is the problem you have ?
>
> Thanks
> Padma
>
>
>
> On Tue, Jul 3, 2018 at 9:00 PM, Divya Gehlot <di...@gmail.com>
> wrote:
>
> > Hi,
> > I would like to understand below configurations :
> > work: {
> > max.width.per.endpoint: 5,
> > global.max.width: 100,
> > affinity.factor: 1.2,
> > executor.threads: 4
> > },
> >
> > As I am getting below error at time in some of the Drill bits :
> > 2018-07-03 04:08:45,725 [pool-245-thread-13] INFO
> > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
> > host <DNSAddress> Skipping affinity to that host.
> > 2018-07-03 04:08:45,731 [pool-245-thread-11] INFO
> > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
> > host <DNSAddress> Skipping affinity to that host.
> > 2018-07-03 04:08:45,732 [pool-245-thread-12] INFO
> > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
> > host <DNSAddress> Skipping affinity to that host.
> > 2018-07-03 04:08:45,732 [pool-245-thread-14] INFO
> > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
> > host <DNSAddress> Skipping affinity to that host.
> > 2018-07-03 04:08:45,733 [pool-245-thread-3] INFO
> > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
> > host <DNSAddress> Skipping affinity to that host.
> > 2018-07-03 04:08:45,733 [pool-245-thread-16] INFO
> > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
> > host <DNSAddress> Skipping affinity to that host.
> > 2018-07-03 04:08:45,735 [pool-245-thread-15] INFO
> > o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
> > host <DNSAddress> Skipping affinity to that host.
> >
> > Is there any relationship between affinity configurations and Drill logs
> ?
> > Appreciate if anybody can help me understand !
> >
> > Thanks,
> > Divya
> >
>
Re: need understanding of drill affinity configurations
Posted by Padma Penumarthy <pe...@gmail.com>.
max.width.per.endpoint - how many minor fragments you can have per major
fragment on a single node
global.max.width - max number of minor fragments per major fragment you can
have across all nodes
affinity.factor - When deciding how many minor fragments to schedule on
each node,
this is the factor by which you favor nodes which have data locality vs.
nodes which do not.
executor.threads - Seems like maximum number of threads we can have ? I am
not very sure about this one.
These messages are debug messages, not errors.
So, what is the problem you have ?
Thanks
Padma
On Tue, Jul 3, 2018 at 9:00 PM, Divya Gehlot <di...@gmail.com>
wrote:
> Hi,
> I would like to understand below configurations :
> work: {
> max.width.per.endpoint: 5,
> global.max.width: 100,
> affinity.factor: 1.2,
> executor.threads: 4
> },
>
> As I am getting below error at time in some of the Drill bits :
> 2018-07-03 04:08:45,725 [pool-245-thread-13] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
> host <DNSAddress> Skipping affinity to that host.
> 2018-07-03 04:08:45,731 [pool-245-thread-11] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
> host <DNSAddress> Skipping affinity to that host.
> 2018-07-03 04:08:45,732 [pool-245-thread-12] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
> host <DNSAddress> Skipping affinity to that host.
> 2018-07-03 04:08:45,732 [pool-245-thread-14] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
> host <DNSAddress> Skipping affinity to that host.
> 2018-07-03 04:08:45,733 [pool-245-thread-3] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
> host <DNSAddress> Skipping affinity to that host.
> 2018-07-03 04:08:45,733 [pool-245-thread-16] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
> host <DNSAddress> Skipping affinity to that host.
> 2018-07-03 04:08:45,735 [pool-245-thread-15] INFO
> o.a.d.e.s.schedule.BlockMapBuilder - Failure finding Drillbit running on
> host <DNSAddress> Skipping affinity to that host.
>
> Is there any relationship between affinity configurations and Drill logs ?
> Appreciate if anybody can help me understand !
>
> Thanks,
> Divya
>