You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "Mehran Dashti [ BR - PD ]" <m_...@behinrahkar.com> on 2018/08/16 13:02:22 UTC
distributed drill on local file system
Hi,
I wanted to know if it is possible or possible by minimal effort to have distributed drills that work on local file system of their own?
We do not want to have HDFS as file system?
Thank you in advance.
Best Regards,
[LOGO1]
Mehran Dashti
Product Leader
09125902452
Re: distributed drill on local file system
Posted by 王亮 <wa...@gmail.com>.
I also meet this problem when querying apache2 log in different machines.
How I can test this feature?
Abhishek Girish <ag...@apache.org> 于2018年8月17日周五 上午2:47写道：
> I'd also like to add that with the DFS storage plugin configured with local
> file system, one will have to make sure all nodes with Drillbits have the
> same files (under the same directory structure). If a given query on a
> dataset spawns a distributed plan (multiple fragments), then it may fail if
> one of the fragments on a remote Drillbit cannot find the file being
> referenced by the foreman. Also, this is not something that's been well
> tried out and documented, so there may be some surprises.
>
> -Abhishek
>
> On Thu, Aug 16, 2018 at 9:44 AM Vitalii Diravka <vitalii.diravka@gmail.com
> >
> wrote:
>
> > Hi Mehran,
> >
> > This is a question for user mailing list.
> >
> > Looks like there are no issues with it, you can run Drill in distributed
> > mode on Windows, Linux or MacOS based machines.
> > It necessary to specify *zk.connect* for Zookeeper hostname and port
> number
> > in *drill-override.conf* file and to run *>bin/drillbit.sh start *[1].
> > But a Hadoop cluster is recommended for this purpose [2], therefore not
> > sure which issues can arise with this system.
> >
> > [1]
> >
> >
> https://drill.apache.org/docs/starting-drill-in-distributed-mode/#drillbit.sh-command-syntax
> > [2] https://drill.apache.org/docs/distributed-mode-prerequisites/
> >
> > Kind regards
> > Vitalii
> >
> >
> > On Thu, Aug 16, 2018 at 7:11 PM Mehran Dashti [ BR - PD ] <
> > m_dashti@behinrahkar.com> wrote:
> >
> > > Hi,
> > >
> > > I wanted to know if it is possible or possible by minimal effort to
> have
> > > distributed drills that work on local file system of their own?
> > >
> > > We do not want to have HDFS as file system?
> > >
> > >
> > >
> > > Thank you in advance.
> > >
> > >
> > >
> > >
> > >
> > > *Best Regards,*
> > >
> > >
> > >
> > > * [image: LOGO1]*
> > >
> > > *Mehran Dashti*
> > >
> > > *Product Leader*
> > >
> > > *09125902452*
> > >
> > >
> > >
> >
>
Re: distributed drill on local file system
Posted by Abhishek Girish <ag...@apache.org>.
I'd also like to add that with the DFS storage plugin configured with local
file system, one will have to make sure all nodes with Drillbits have the
same files (under the same directory structure). If a given query on a
dataset spawns a distributed plan (multiple fragments), then it may fail if
one of the fragments on a remote Drillbit cannot find the file being
referenced by the foreman. Also, this is not something that's been well
tried out and documented, so there may be some surprises.
-Abhishek
On Thu, Aug 16, 2018 at 9:44 AM Vitalii Diravka <vi...@gmail.com>
wrote:
> Hi Mehran,
>
> This is a question for user mailing list.
>
> Looks like there are no issues with it, you can run Drill in distributed
> mode on Windows, Linux or MacOS based machines.
> It necessary to specify *zk.connect* for Zookeeper hostname and port number
> in *drill-override.conf* file and to run *>bin/drillbit.sh start *[1].
> But a Hadoop cluster is recommended for this purpose [2], therefore not
> sure which issues can arise with this system.
>
> [1]
>
> https://drill.apache.org/docs/starting-drill-in-distributed-mode/#drillbit.sh-command-syntax
> [2] https://drill.apache.org/docs/distributed-mode-prerequisites/
>
> Kind regards
> Vitalii
>
>
> On Thu, Aug 16, 2018 at 7:11 PM Mehran Dashti [ BR - PD ] <
> m_dashti@behinrahkar.com> wrote:
>
> > Hi,
> >
> > I wanted to know if it is possible or possible by minimal effort to have
> > distributed drills that work on local file system of their own?
> >
> > We do not want to have HDFS as file system?
> >
> >
> >
> > Thank you in advance.
> >
> >
> >
> >
> >
> > *Best Regards,*
> >
> >
> >
> > * [image: LOGO1]*
> >
> > *Mehran Dashti*
> >
> > *Product Leader*
> >
> > *09125902452*
> >
> >
> >
>
Re: distributed drill on local file system
Posted by Vitalii Diravka <vi...@gmail.com>.
Hi Mehran,
This is a question for user mailing list.
Looks like there are no issues with it, you can run Drill in distributed
mode on Windows, Linux or MacOS based machines.
It necessary to specify *zk.connect* for Zookeeper hostname and port number
in *drill-override.conf* file and to run *>bin/drillbit.sh start *[1].
But a Hadoop cluster is recommended for this purpose [2], therefore not
sure which issues can arise with this system.
[1]
https://drill.apache.org/docs/starting-drill-in-distributed-mode/#drillbit.sh-command-syntax
[2] https://drill.apache.org/docs/distributed-mode-prerequisites/
Kind regards
Vitalii
On Thu, Aug 16, 2018 at 7:11 PM Mehran Dashti [ BR - PD ] <
m_dashti@behinrahkar.com> wrote:
> Hi,
>
> I wanted to know if it is possible or possible by minimal effort to have
> distributed drills that work on local file system of their own?
>
> We do not want to have HDFS as file system?
>
>
>
> Thank you in advance.
>
>
>
>
>
> *Best Regards,*
>
>
>
> * [image: LOGO1]*
>
> *Mehran Dashti*
>
> *Product Leader*
>
> *09125902452*
>
>
>
Re: distributed drill on local file system
Posted by Vitalii Diravka <vi...@gmail.com>.
Hi Mehran,
This is a question for user mailing list.
Looks like there are no issues with it, you can run Drill in distributed
mode on Windows, Linux or MacOS based machines.
It necessary to specify *zk.connect* for Zookeeper hostname and port number
in *drill-override.conf* file and to run *>bin/drillbit.sh start *[1].
But a Hadoop cluster is recommended for this purpose [2], therefore not
sure which issues can arise with this system.
[1]
https://drill.apache.org/docs/starting-drill-in-distributed-mode/#drillbit.sh-command-syntax
[2] https://drill.apache.org/docs/distributed-mode-prerequisites/
Kind regards
Vitalii
On Thu, Aug 16, 2018 at 7:11 PM Mehran Dashti [ BR - PD ] <
m_dashti@behinrahkar.com> wrote:
> Hi,
>
> I wanted to know if it is possible or possible by minimal effort to have
> distributed drills that work on local file system of their own?
>
> We do not want to have HDFS as file system?
>
>
>
> Thank you in advance.
>
>
>
>
>
> *Best Regards,*
>
>
>
> * [image: LOGO1]*
>
> *Mehran Dashti*
>
> *Product Leader*
>
> *09125902452*
>
>
>