You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by ganesh <g4...@gmail.com> on 2015/10/17 06:50:15 UTC
APACHE DRILL : Connect with Hadoop File System // hdfs
Hello,
I am new to Apache Drill and need some help. I am able to connect to Local
File systema nd Hive and query the files/ tables respectively.
But I want to access file on Hadoop. For that I created a new Storage
Plugin, (copied from default dfs) as "hadoop" and changed connection as:
"connection": "file:///", *ß**----------- "connection" :
"hdfs://192.168.108.133:8020/ <http://192.168.108.133:8020/>",*
Now when I am checking data from DRILL Explorer, its giving connection
error.
I tried to run query after selecting:
*use hadoop;*
0: jdbc:drill:zk=local> select * from hadoop.`/user/hduser/drill_1.csv`;
Oct 16, 2015 9:35:15 PM
org.apache.calcite.sql.validate.SqlValidatorException <init>
SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table
'hadoop./user/hduser/drill_1.csv' not found
Oct 16, 2015 9:35:15 PM org.apache.calcite.runtime.CalciteException <init>
SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1,
column 15 to line 1, column 20: Table 'hadoop./user/hduser/drill_1.csv' not
found
Error: PARSE ERROR: From line 1, column 15 to line 1, column 20: Table
'hadoop./user/hduser/drill_1.csv' not found
[Error Id: b97de3f1-d190-48ac-b2f1-a1fb23493fbf on ubuntu:31010]
(state=,code=0)
0: jdbc:drill:zk=local>
Unable to get much help for this on Internet.
I changed port number to:
*50070* ------------> namenode which I can successfully access as:
http://localhost:50070/dfshealth.html#tab-overview
*54310* -----------------> /home/hduser/hadoop/etc/hadoop/core-site.xml
None of these working. Please help
--
*Name: Ganesh Semalty*
*Location: Gurgaon,Haryana(India)*
*Email Id: g4ganeshsemalty@gmail.com <g4...@gmail.com>*
P
*Please consider the environment before printing this e-mail - SAVE TREE.*
Re: APACHE DRILL : Connect with Hadoop File System // hdfs
Posted by Abhishek Girish <ab...@gmail.com>.
My bad. I realized that once i switched modes, content of my dfs plugin
changed to point to local FS. So Ganesh, please ignore my comments.
On Sun, Oct 18, 2015 at 4:30 PM, Abhishek Girish <ag...@mapr.com> wrote:
> I brought down a drillbit which was previously in distributed mode and
> restarted it in embedded mode. A simple query to access a file failed with
> FileNotFound error - this had worked previously. Doing a "show files"
> inside the dfs storage plugin listed only files from the local file system.
> I assumed this is similar to the case Ganesh experienced. So thought I'll
> ask him to try it out.
>
> On Sunday, October 18, 2015, Jacques Nadeau <ja...@dremio.com> wrote:
>
>> Abhishek, there is no reason to ask Ganesh to move to using distributed
>> mode. That has nothing to do with the error he is experiencing.
>>
>> Ganesh,
>>
>> I'm currently running Drill on HDFS 2.7.1 and use the following path
>> successfully:
>>
>> hdfs://<ip address>:54310/
>>
>>
>>
>> --
>> Jacques Nadeau
>> CTO and Co-Founder, Dremio
>>
>> On Sat, Oct 17, 2015 at 12:31 PM, Abhishek Girish <
>> abhishek.girish@gmail.com
>> > wrote:
>>
>> > It looks like you started Drill in embedded mode and tried querying a
>> > distributed file system. Can you connect using distributed mode [1] and
>> > connect to the file system [2]. Also I'd expect that port 8020 would
>> work.
>> >
>> > [1] https://drill.apache.org/docs/starting-drill-in-distributed-mode/
>> > [2] https://drill.apache.org/docs/file-system-storage-plugin/
>> >
>> > On Sat, Oct 17, 2015 at 6:13 AM, Geercken, Uwe <
>> Uwe.Geercken@swissport.com
>> > >
>> > wrote:
>> >
>> > > Ganesh,
>> > >
>> > > I think I tried it - cannot verify it right now - on port 9000 and
>> that
>> > > did work.
>> > >
>> > > Give it a try and let me know. I will see if I find time to check
>> during
>> > > the day.
>> > >
>> > > Uwe
>> > > ________________________________________
>> > > From: ganesh [g4ganeshsemalty@gmail.com]
>> > > Sent: Saturday, October 17, 2015 6:50 AM
>> > > To: user-subscribe@drill.apache.org; user@drill.apache.org
>> > > Subject: APACHE DRILL : Connect with Hadoop File System // hdfs
>> > >
>> > > Hello,
>> > >
>> > > I am new to Apache Drill and need some help. I am able to connect to
>> > Local
>> > > File systema nd Hive and query the files/ tables respectively.
>> > >
>> > > But I want to access file on Hadoop. For that I created a new Storage
>> > > Plugin, (copied from default dfs) as "hadoop" and changed connection
>> as:
>> > >
>> > > "connection": "file:///", *ß**----------- "connection" :
>> > > "hdfs://192.168.108.133:8020/ <http://192.168.108.133:8020/>",*
>> > >
>> > >
>> > >
>> > > Now when I am checking data from DRILL Explorer, its giving connection
>> > > error.
>> > > I tried to run query after selecting:
>> > > *use hadoop;*
>> > > 0: jdbc:drill:zk=local> select * from
>> hadoop.`/user/hduser/drill_1.csv`;
>> > > Oct 16, 2015 9:35:15 PM
>> > > org.apache.calcite.sql.validate.SqlValidatorException <init>
>> > > SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table
>> > > 'hadoop./user/hduser/drill_1.csv' not found
>> > > Oct 16, 2015 9:35:15 PM org.apache.calcite.runtime.CalciteException
>> > <init>
>> > > SEVERE: org.apache.calcite.runtime.CalciteContextException: From line
>> 1,
>> > > column 15 to line 1, column 20: Table
>> 'hadoop./user/hduser/drill_1.csv'
>> > not
>> > > found
>> > > Error: PARSE ERROR: From line 1, column 15 to line 1, column 20: Table
>> > > 'hadoop./user/hduser/drill_1.csv' not found
>> > >
>> > >
>> > > [Error Id: b97de3f1-d190-48ac-b2f1-a1fb23493fbf on ubuntu:31010]
>> > > (state=,code=0)
>> > > 0: jdbc:drill:zk=local>
>> > >
>> > >
>> > > Unable to get much help for this on Internet.
>> > > I changed port number to:
>> > > *50070* ------------> namenode which I can successfully access as:
>> > > http://localhost:50070/dfshealth.html#tab-overview
>> > >
>> > > *54310* ----------------->
>> /home/hduser/hadoop/etc/hadoop/core-site.xml
>> > >
>> > >
>> > > None of these working. Please help
>> > >
>> > >
>> > >
>> > > --
>> > > *Name: Ganesh Semalty*
>> > > *Location: Gurgaon,Haryana(India)*
>> > > *Email Id: g4ganeshsemalty@gmail.com <g4...@gmail.com>*
>> > >
>> > >
>> > > P
>> > >
>> > > *Please consider the environment before printing this e-mail - SAVE
>> > TREE.*
>> > >
>> >
>>
>
Re: APACHE DRILL : Connect with Hadoop File System // hdfs
Posted by Abhishek Girish <ag...@mapr.com>.
I brought down a drillbit which was previously in distributed mode and
restarted it in embedded mode. A simple query to access a file failed with
FileNotFound error - this had worked previously. Doing a "show files"
inside the dfs storage plugin listed only files from the local file system.
I assumed this is similar to the case Ganesh experienced. So thought I'll
ask him to try it out.
On Sunday, October 18, 2015, Jacques Nadeau <ja...@dremio.com> wrote:
> Abhishek, there is no reason to ask Ganesh to move to using distributed
> mode. That has nothing to do with the error he is experiencing.
>
> Ganesh,
>
> I'm currently running Drill on HDFS 2.7.1 and use the following path
> successfully:
>
> hdfs://<ip address>:54310/
>
>
>
> --
> Jacques Nadeau
> CTO and Co-Founder, Dremio
>
> On Sat, Oct 17, 2015 at 12:31 PM, Abhishek Girish <
> abhishek.girish@gmail.com <javascript:;>
> > wrote:
>
> > It looks like you started Drill in embedded mode and tried querying a
> > distributed file system. Can you connect using distributed mode [1] and
> > connect to the file system [2]. Also I'd expect that port 8020 would
> work.
> >
> > [1] https://drill.apache.org/docs/starting-drill-in-distributed-mode/
> > [2] https://drill.apache.org/docs/file-system-storage-plugin/
> >
> > On Sat, Oct 17, 2015 at 6:13 AM, Geercken, Uwe <
> Uwe.Geercken@swissport.com <javascript:;>
> > >
> > wrote:
> >
> > > Ganesh,
> > >
> > > I think I tried it - cannot verify it right now - on port 9000 and that
> > > did work.
> > >
> > > Give it a try and let me know. I will see if I find time to check
> during
> > > the day.
> > >
> > > Uwe
> > > ________________________________________
> > > From: ganesh [g4ganeshsemalty@gmail.com <javascript:;>]
> > > Sent: Saturday, October 17, 2015 6:50 AM
> > > To: user-subscribe@drill.apache.org <javascript:;>;
> user@drill.apache.org <javascript:;>
> > > Subject: APACHE DRILL : Connect with Hadoop File System // hdfs
> > >
> > > Hello,
> > >
> > > I am new to Apache Drill and need some help. I am able to connect to
> > Local
> > > File systema nd Hive and query the files/ tables respectively.
> > >
> > > But I want to access file on Hadoop. For that I created a new Storage
> > > Plugin, (copied from default dfs) as "hadoop" and changed connection
> as:
> > >
> > > "connection": "file:///", *ß**----------- "connection" :
> > > "hdfs://192.168.108.133:8020/ <http://192.168.108.133:8020/>",*
> > >
> > >
> > >
> > > Now when I am checking data from DRILL Explorer, its giving connection
> > > error.
> > > I tried to run query after selecting:
> > > *use hadoop;*
> > > 0: jdbc:drill:zk=local> select * from
> hadoop.`/user/hduser/drill_1.csv`;
> > > Oct 16, 2015 9:35:15 PM
> > > org.apache.calcite.sql.validate.SqlValidatorException <init>
> > > SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table
> > > 'hadoop./user/hduser/drill_1.csv' not found
> > > Oct 16, 2015 9:35:15 PM org.apache.calcite.runtime.CalciteException
> > <init>
> > > SEVERE: org.apache.calcite.runtime.CalciteContextException: From line
> 1,
> > > column 15 to line 1, column 20: Table 'hadoop./user/hduser/drill_1.csv'
> > not
> > > found
> > > Error: PARSE ERROR: From line 1, column 15 to line 1, column 20: Table
> > > 'hadoop./user/hduser/drill_1.csv' not found
> > >
> > >
> > > [Error Id: b97de3f1-d190-48ac-b2f1-a1fb23493fbf on ubuntu:31010]
> > > (state=,code=0)
> > > 0: jdbc:drill:zk=local>
> > >
> > >
> > > Unable to get much help for this on Internet.
> > > I changed port number to:
> > > *50070* ------------> namenode which I can successfully access as:
> > > http://localhost:50070/dfshealth.html#tab-overview
> > >
> > > *54310* ----------------->
> /home/hduser/hadoop/etc/hadoop/core-site.xml
> > >
> > >
> > > None of these working. Please help
> > >
> > >
> > >
> > > --
> > > *Name: Ganesh Semalty*
> > > *Location: Gurgaon,Haryana(India)*
> > > *Email Id: g4ganeshsemalty@gmail.com <javascript:;> <
> g4ganeshsemalty@gmail.com <javascript:;>>*
> > >
> > >
> > > P
> > >
> > > *Please consider the environment before printing this e-mail - SAVE
> > TREE.*
> > >
> >
>
Re: APACHE DRILL : Connect with Hadoop File System // hdfs
Posted by Jacques Nadeau <ja...@dremio.com>.
Abhishek, there is no reason to ask Ganesh to move to using distributed
mode. That has nothing to do with the error he is experiencing.
Ganesh,
I'm currently running Drill on HDFS 2.7.1 and use the following path
successfully:
hdfs://<ip address>:54310/
--
Jacques Nadeau
CTO and Co-Founder, Dremio
On Sat, Oct 17, 2015 at 12:31 PM, Abhishek Girish <abhishek.girish@gmail.com
> wrote:
> It looks like you started Drill in embedded mode and tried querying a
> distributed file system. Can you connect using distributed mode [1] and
> connect to the file system [2]. Also I'd expect that port 8020 would work.
>
> [1] https://drill.apache.org/docs/starting-drill-in-distributed-mode/
> [2] https://drill.apache.org/docs/file-system-storage-plugin/
>
> On Sat, Oct 17, 2015 at 6:13 AM, Geercken, Uwe <Uwe.Geercken@swissport.com
> >
> wrote:
>
> > Ganesh,
> >
> > I think I tried it - cannot verify it right now - on port 9000 and that
> > did work.
> >
> > Give it a try and let me know. I will see if I find time to check during
> > the day.
> >
> > Uwe
> > ________________________________________
> > From: ganesh [g4ganeshsemalty@gmail.com]
> > Sent: Saturday, October 17, 2015 6:50 AM
> > To: user-subscribe@drill.apache.org; user@drill.apache.org
> > Subject: APACHE DRILL : Connect with Hadoop File System // hdfs
> >
> > Hello,
> >
> > I am new to Apache Drill and need some help. I am able to connect to
> Local
> > File systema nd Hive and query the files/ tables respectively.
> >
> > But I want to access file on Hadoop. For that I created a new Storage
> > Plugin, (copied from default dfs) as "hadoop" and changed connection as:
> >
> > "connection": "file:///", *ß**----------- "connection" :
> > "hdfs://192.168.108.133:8020/ <http://192.168.108.133:8020/>",*
> >
> >
> >
> > Now when I am checking data from DRILL Explorer, its giving connection
> > error.
> > I tried to run query after selecting:
> > *use hadoop;*
> > 0: jdbc:drill:zk=local> select * from hadoop.`/user/hduser/drill_1.csv`;
> > Oct 16, 2015 9:35:15 PM
> > org.apache.calcite.sql.validate.SqlValidatorException <init>
> > SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table
> > 'hadoop./user/hduser/drill_1.csv' not found
> > Oct 16, 2015 9:35:15 PM org.apache.calcite.runtime.CalciteException
> <init>
> > SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1,
> > column 15 to line 1, column 20: Table 'hadoop./user/hduser/drill_1.csv'
> not
> > found
> > Error: PARSE ERROR: From line 1, column 15 to line 1, column 20: Table
> > 'hadoop./user/hduser/drill_1.csv' not found
> >
> >
> > [Error Id: b97de3f1-d190-48ac-b2f1-a1fb23493fbf on ubuntu:31010]
> > (state=,code=0)
> > 0: jdbc:drill:zk=local>
> >
> >
> > Unable to get much help for this on Internet.
> > I changed port number to:
> > *50070* ------------> namenode which I can successfully access as:
> > http://localhost:50070/dfshealth.html#tab-overview
> >
> > *54310* -----------------> /home/hduser/hadoop/etc/hadoop/core-site.xml
> >
> >
> > None of these working. Please help
> >
> >
> >
> > --
> > *Name: Ganesh Semalty*
> > *Location: Gurgaon,Haryana(India)*
> > *Email Id: g4ganeshsemalty@gmail.com <g4...@gmail.com>*
> >
> >
> > P
> >
> > *Please consider the environment before printing this e-mail - SAVE
> TREE.*
> >
>
Re: APACHE DRILL : Connect with Hadoop File System // hdfs
Posted by Abhishek Girish <ab...@gmail.com>.
It looks like you started Drill in embedded mode and tried querying a
distributed file system. Can you connect using distributed mode [1] and
connect to the file system [2]. Also I'd expect that port 8020 would work.
[1] https://drill.apache.org/docs/starting-drill-in-distributed-mode/
[2] https://drill.apache.org/docs/file-system-storage-plugin/
On Sat, Oct 17, 2015 at 6:13 AM, Geercken, Uwe <Uw...@swissport.com>
wrote:
> Ganesh,
>
> I think I tried it - cannot verify it right now - on port 9000 and that
> did work.
>
> Give it a try and let me know. I will see if I find time to check during
> the day.
>
> Uwe
> ________________________________________
> From: ganesh [g4ganeshsemalty@gmail.com]
> Sent: Saturday, October 17, 2015 6:50 AM
> To: user-subscribe@drill.apache.org; user@drill.apache.org
> Subject: APACHE DRILL : Connect with Hadoop File System // hdfs
>
> Hello,
>
> I am new to Apache Drill and need some help. I am able to connect to Local
> File systema nd Hive and query the files/ tables respectively.
>
> But I want to access file on Hadoop. For that I created a new Storage
> Plugin, (copied from default dfs) as "hadoop" and changed connection as:
>
> "connection": "file:///", *ß**----------- "connection" :
> "hdfs://192.168.108.133:8020/ <http://192.168.108.133:8020/>",*
>
>
>
> Now when I am checking data from DRILL Explorer, its giving connection
> error.
> I tried to run query after selecting:
> *use hadoop;*
> 0: jdbc:drill:zk=local> select * from hadoop.`/user/hduser/drill_1.csv`;
> Oct 16, 2015 9:35:15 PM
> org.apache.calcite.sql.validate.SqlValidatorException <init>
> SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table
> 'hadoop./user/hduser/drill_1.csv' not found
> Oct 16, 2015 9:35:15 PM org.apache.calcite.runtime.CalciteException <init>
> SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1,
> column 15 to line 1, column 20: Table 'hadoop./user/hduser/drill_1.csv' not
> found
> Error: PARSE ERROR: From line 1, column 15 to line 1, column 20: Table
> 'hadoop./user/hduser/drill_1.csv' not found
>
>
> [Error Id: b97de3f1-d190-48ac-b2f1-a1fb23493fbf on ubuntu:31010]
> (state=,code=0)
> 0: jdbc:drill:zk=local>
>
>
> Unable to get much help for this on Internet.
> I changed port number to:
> *50070* ------------> namenode which I can successfully access as:
> http://localhost:50070/dfshealth.html#tab-overview
>
> *54310* -----------------> /home/hduser/hadoop/etc/hadoop/core-site.xml
>
>
> None of these working. Please help
>
>
>
> --
> *Name: Ganesh Semalty*
> *Location: Gurgaon,Haryana(India)*
> *Email Id: g4ganeshsemalty@gmail.com <g4...@gmail.com>*
>
>
> P
>
> *Please consider the environment before printing this e-mail - SAVE TREE.*
>
RE: APACHE DRILL : Connect with Hadoop File System // hdfs
Posted by "Geercken, Uwe" <Uw...@swissport.com>.
Ganesh,
I think I tried it - cannot verify it right now - on port 9000 and that did work.
Give it a try and let me know. I will see if I find time to check during the day.
Uwe
________________________________________
From: ganesh [g4ganeshsemalty@gmail.com]
Sent: Saturday, October 17, 2015 6:50 AM
To: user-subscribe@drill.apache.org; user@drill.apache.org
Subject: APACHE DRILL : Connect with Hadoop File System // hdfs
Hello,
I am new to Apache Drill and need some help. I am able to connect to Local
File systema nd Hive and query the files/ tables respectively.
But I want to access file on Hadoop. For that I created a new Storage
Plugin, (copied from default dfs) as "hadoop" and changed connection as:
"connection": "file:///", *ß**----------- "connection" :
"hdfs://192.168.108.133:8020/ <http://192.168.108.133:8020/>",*
Now when I am checking data from DRILL Explorer, its giving connection
error.
I tried to run query after selecting:
*use hadoop;*
0: jdbc:drill:zk=local> select * from hadoop.`/user/hduser/drill_1.csv`;
Oct 16, 2015 9:35:15 PM
org.apache.calcite.sql.validate.SqlValidatorException <init>
SEVERE: org.apache.calcite.sql.validate.SqlValidatorException: Table
'hadoop./user/hduser/drill_1.csv' not found
Oct 16, 2015 9:35:15 PM org.apache.calcite.runtime.CalciteException <init>
SEVERE: org.apache.calcite.runtime.CalciteContextException: From line 1,
column 15 to line 1, column 20: Table 'hadoop./user/hduser/drill_1.csv' not
found
Error: PARSE ERROR: From line 1, column 15 to line 1, column 20: Table
'hadoop./user/hduser/drill_1.csv' not found
[Error Id: b97de3f1-d190-48ac-b2f1-a1fb23493fbf on ubuntu:31010]
(state=,code=0)
0: jdbc:drill:zk=local>
Unable to get much help for this on Internet.
I changed port number to:
*50070* ------------> namenode which I can successfully access as:
http://localhost:50070/dfshealth.html#tab-overview
*54310* -----------------> /home/hduser/hadoop/etc/hadoop/core-site.xml
None of these working. Please help
--
*Name: Ganesh Semalty*
*Location: Gurgaon,Haryana(India)*
*Email Id: g4ganeshsemalty@gmail.com <g4...@gmail.com>*
P
*Please consider the environment before printing this e-mail - SAVE TREE.*