You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Alan Miller <Al...@synopsys.com> on 2015/05/21 23:04:08 UTC
[newbie]: how to query HDFS
First off, this is my first attempt at drill,
(BTW: congratulations on the release ;-)
so perhaps I misunderstood something
I want to query my parquet files on HDFS.
I setup the 1.0 release on a machine (node1)
that already had CDH5 and a working Zookeeper.
With the hdfs storage plugin config below I can query a parquet file
on the local machine just fine.
E.g.:
0: jdbc:drill:drillbit=localhost> select a,b,c FROM hdfs.`/hdfs/path/test.par` limit 5;
## drill-override.conf
drill.exec: {
cluster-id: "mydrillcluster",
zk.connect: "node1:2181"
}
## storage plugin config
{
"type": "file",
"enabled": true,
"connection": "hdfs://127.0.0.1:8020/",
"workspaces": null,
"formats": {
"parquet": {
"type": "parquet"
}
}
}
Can I query a remote HDFS, by simply pointing the storage plugin config?
After changing the IP address in the connection parameter above, I get this error.
0: jdbc:drill:drillbit=localhost> select a,b,c FROM hdfs.`/tmp/test.par` limit 5;
Error: PARSE ERROR: From line 1, column 38 to line 1, column 41: Table 'hdfs./tmp/test.par' not found
[Error Id: 4156f66c-3dac-4e87-b7f8-f0bdc19d57d7 on node1.company.com:31010] (state=,code=0)
.....
Caused by: org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: From line 1, column 38 to line 1, column 41: Table 'hdfs./tmp/test.par' not found
But the namenode:port/path is correct because this workds from node1:
[alan@node1 drill]$ hdfs dfs -fs hdfs://10.10.10.10:8020/ -ls /tmp/test.par
-rw-r--r-- 1 alan supergroup 4947359 2015-05-21 13:55 /tmp/test.par
Alan
RE: [newbie]: how to query HDFS
Posted by Alan Miller <Al...@synopsys.com>.
Sorry, I meant I can not query a remote CDH 4 cluster.
-----Original Message-----
From: Alan Miller [mailto:Alan.Miller@synopsys.com]
Sent: Friday, May 22, 2015 8:58 AM
To: user@drill.apache.org
Subject: RE: [newbie]: how to query HDFS
Thanks, yes it works now! The backtick was my problem.
I can query a remote CDH5 HDFS from node1 (not part of the cluster) but I can't query a remote CDH5 HDFS. But that's not an issue for me.
Thanks for the quick responses.
0: jdbc:drill:drillbit=localhost> select job_number,submit_time from hdfs_cdh5.root.`/tmp/test.par` limit 5;
+-------------+--------------+
| job_number | submit_time |
+-------------+--------------+
| 1700692497 | 1430313605 |
| 1786732386 | 1430319046 |
| 290092620 | 1430315818 |
| 1556821944 | 1430321161 |
| 157926450 | 1430313079 |
+-------------+--------------+
Alan
RE: [newbie]: how to query HDFS
Posted by Alan Miller <Al...@synopsys.com>.
Thanks, yes it works now! The backtick was my problem.
I can query a remote CDH5 HDFS from node1 (not part of the cluster) but I
can't query a remote CDH5 HDFS. But that's not an issue for me.
Thanks for the quick responses.
0: jdbc:drill:drillbit=localhost> select job_number,submit_time from hdfs_cdh5.root.`/tmp/test.par` limit 5;
+-------------+--------------+
| job_number | submit_time |
+-------------+--------------+
| 1700692497 | 1430313605 |
| 1786732386 | 1430319046 |
| 290092620 | 1430315818 |
| 1556821944 | 1430321161 |
| 157926450 | 1430313079 |
+-------------+--------------+
Alan
Re: [newbie]: how to query HDFS
Posted by Ted Dunning <te...@gmail.com>.
I should have added that there is nothing wrong with the dual plugin
approach on MapR. Works fine and it is up to you as a matter of personal
choice which is better.
On Fri, May 22, 2015 at 4:46 PM, Ted Dunning <te...@gmail.com> wrote:
>
> As a special case, with MapR, you can access all clusters in an
> administrative group by making sure that you have /mapr/<cluster-name> at
> the beginning of your path names. THis means that you can simply use
> different workspaces, or a workspace with a path consisting only of /mapr
> and still access files and tables on multiple clusters.
>
> The dual plugin approach would be required for HDFS based clusters.
>
>
>
> On Fri, May 22, 2015 at 9:49 AM, Abhishek Girish <ag...@mapr.com> wrote:
>
>> I tried out Tomar's steps on MapR and it was pretty straight-forward.
>>
>> I have drill installed on one cluster. The only change I made was to add a
>> new storage plug-in "dfs2" (duplicating the default dfs). I edited the
>> connection string and changed "maprfs:///" to "maprfs://<IP>". And when i
>> connected to drill via sqlline (no changes here), I was able to access the
>> remote file system by simply using the the full path of the file, prefixed
>> with "dfs2.".
>>
>> In case of HDFS, I'm assuming the steps required must be similar, except
>> the connection string (hdfs://<IP>:<port>)
>>
>> Also, as Andrews mentioned, there looks to be a typo in your query - that
>> could very well be the issue.
>>
>> Thanks,
>> Abhishek
>>
>> On Fri, May 22, 2015 at 8:27 AM, Andries Engelbrecht <
>> aengelbrecht@maprtech.com> wrote:
>>
>> > In step2 you need to have a back tick ` at the end of par and not a
>> single
>> > quote ‘ .
>> >
>> > It has been mentioned that it may not work to just point to the name
>> node
>> > on a remote cluster. I have not tried it, but suspect their may be
>> various
>> > issues with the HDFS plug in and how you are trying to use it.
>> >
>> > Perhaps if you can explain why you are trying to do this, there may be
>> > other alternatives to explore.
>> >
>> > What Hadoop distro are you using?
>> >
>> > —Andries
>> >
>> >
>> > On May 22, 2015, at 8:17 AM, Alan Miller <Al...@synopsys.com>
>> wrote:
>> >
>> > > Thanks Shiran,
>> > >
>> > > I tried that but get the same error (see below).
>> > >
>> > > Also, strangely I couldn't create the hdfs plugin in one step by using
>> > the same
>> > > config as the "dfs" plugin and changing the connection string. The UI
>> > says Invalid JSON...
>> > > I had to create the hdfs plugin in 2 steps. First using the same
>> config
>> > as the dfs plugin.
>> > > Then updated the hdfs config, by changing the connection string
>> > >
>> > > After adding the hdfs plugin with the same config as dfs (but
>> different
>> > connection
>> > > string ("connection": "hdfs://10.10.10.10:9000/",) I tried this
>> > >
>> > > 1. Copied the file from node1 to remote HDFS
>> > > [alan@node1 drill]$ hdfs dfs -fs hdfs://10.10.10.10:9000/
>> > -copyFromLocal ~/test.par /tmp
>> > > [alan@node1 drill]$ hdfs dfs -fs hdfs://10.10.10.10:9000/ -ls
>> > /tmp/test.par
>> > > -rw-r--r-- 1 alan supergroup 4947359 2015-05-22 08:09
>> > /tmp/test.par
>> > >
>> > > 2. From drill on node1
>> > > [alan@node1 drill]$ /opt/drill/bin/drill-localhost
>> > > apache drill 1.0.0
>> > > "json ain't no thang"
>> > > 0: jdbc:drill:drillbit=localhost> use hdfs;
>> > > +-------+-----------------------------------+
>> > > | ok | summary |
>> > > +-------+-----------------------------------+
>> > > | true | Default schema changed to [hdfs] |
>> > > +-------+-----------------------------------+
>> > > 1 row selected (0.422 seconds)
>> > > 0: jdbc:drill:drillbit=localhost> select * from
>> > hdfs.root.`/tmp/test.par' limit 5;
>> > > Error: PARSE ERROR: Lexical error at line 1, column 55.
>> Encountered:
>> > <EOF> after : "`/tmp/test.par\' limit 5"
>> > >
>> > > [Error Id: 1f793d84-62be-4145-bfcf-2ec3da9cb021 on
>> > node1.mycompany.com:31010] (state=,code=0)
>> > > 0: jdbc:drill:drillbit=localhost>
>> > > 0: jdbc:drill:drillbit=localhost> !quit
>> > > Closing:
>> > org.apache.drill.jdbc.DrillJdbc41Factory$DrillJdbc41Connection
>> > >
>> > > Alan
>> >
>> >
>>
>>
>> --
>>
>> Abhishek Girish
>>
>> Senior Software Engineer
>>
>> (408) 476-9209
>>
>> <http://www.mapr.com/>
>>
>
>
Re: [newbie]: how to query HDFS
Posted by Ted Dunning <te...@gmail.com>.
As a special case, with MapR, you can access all clusters in an
administrative group by making sure that you have /mapr/<cluster-name> at
the beginning of your path names. THis means that you can simply use
different workspaces, or a workspace with a path consisting only of /mapr
and still access files and tables on multiple clusters.
The dual plugin approach would be required for HDFS based clusters.
On Fri, May 22, 2015 at 9:49 AM, Abhishek Girish <ag...@mapr.com> wrote:
> I tried out Tomar's steps on MapR and it was pretty straight-forward.
>
> I have drill installed on one cluster. The only change I made was to add a
> new storage plug-in "dfs2" (duplicating the default dfs). I edited the
> connection string and changed "maprfs:///" to "maprfs://<IP>". And when i
> connected to drill via sqlline (no changes here), I was able to access the
> remote file system by simply using the the full path of the file, prefixed
> with "dfs2.".
>
> In case of HDFS, I'm assuming the steps required must be similar, except
> the connection string (hdfs://<IP>:<port>)
>
> Also, as Andrews mentioned, there looks to be a typo in your query - that
> could very well be the issue.
>
> Thanks,
> Abhishek
>
> On Fri, May 22, 2015 at 8:27 AM, Andries Engelbrecht <
> aengelbrecht@maprtech.com> wrote:
>
> > In step2 you need to have a back tick ` at the end of par and not a
> single
> > quote ‘ .
> >
> > It has been mentioned that it may not work to just point to the name node
> > on a remote cluster. I have not tried it, but suspect their may be
> various
> > issues with the HDFS plug in and how you are trying to use it.
> >
> > Perhaps if you can explain why you are trying to do this, there may be
> > other alternatives to explore.
> >
> > What Hadoop distro are you using?
> >
> > —Andries
> >
> >
> > On May 22, 2015, at 8:17 AM, Alan Miller <Al...@synopsys.com>
> wrote:
> >
> > > Thanks Shiran,
> > >
> > > I tried that but get the same error (see below).
> > >
> > > Also, strangely I couldn't create the hdfs plugin in one step by using
> > the same
> > > config as the "dfs" plugin and changing the connection string. The UI
> > says Invalid JSON...
> > > I had to create the hdfs plugin in 2 steps. First using the same config
> > as the dfs plugin.
> > > Then updated the hdfs config, by changing the connection string
> > >
> > > After adding the hdfs plugin with the same config as dfs (but different
> > connection
> > > string ("connection": "hdfs://10.10.10.10:9000/",) I tried this
> > >
> > > 1. Copied the file from node1 to remote HDFS
> > > [alan@node1 drill]$ hdfs dfs -fs hdfs://10.10.10.10:9000/
> > -copyFromLocal ~/test.par /tmp
> > > [alan@node1 drill]$ hdfs dfs -fs hdfs://10.10.10.10:9000/ -ls
> > /tmp/test.par
> > > -rw-r--r-- 1 alan supergroup 4947359 2015-05-22 08:09
> > /tmp/test.par
> > >
> > > 2. From drill on node1
> > > [alan@node1 drill]$ /opt/drill/bin/drill-localhost
> > > apache drill 1.0.0
> > > "json ain't no thang"
> > > 0: jdbc:drill:drillbit=localhost> use hdfs;
> > > +-------+-----------------------------------+
> > > | ok | summary |
> > > +-------+-----------------------------------+
> > > | true | Default schema changed to [hdfs] |
> > > +-------+-----------------------------------+
> > > 1 row selected (0.422 seconds)
> > > 0: jdbc:drill:drillbit=localhost> select * from
> > hdfs.root.`/tmp/test.par' limit 5;
> > > Error: PARSE ERROR: Lexical error at line 1, column 55.
> Encountered:
> > <EOF> after : "`/tmp/test.par\' limit 5"
> > >
> > > [Error Id: 1f793d84-62be-4145-bfcf-2ec3da9cb021 on
> > node1.mycompany.com:31010] (state=,code=0)
> > > 0: jdbc:drill:drillbit=localhost>
> > > 0: jdbc:drill:drillbit=localhost> !quit
> > > Closing:
> > org.apache.drill.jdbc.DrillJdbc41Factory$DrillJdbc41Connection
> > >
> > > Alan
> >
> >
>
>
> --
>
> Abhishek Girish
>
> Senior Software Engineer
>
> (408) 476-9209
>
> <http://www.mapr.com/>
>
Re: [newbie]: how to query HDFS
Posted by Abhishek Girish <ag...@mapr.com>.
I tried out Tomar's steps on MapR and it was pretty straight-forward.
I have drill installed on one cluster. The only change I made was to add a
new storage plug-in "dfs2" (duplicating the default dfs). I edited the
connection string and changed "maprfs:///" to "maprfs://<IP>". And when i
connected to drill via sqlline (no changes here), I was able to access the
remote file system by simply using the the full path of the file, prefixed
with "dfs2.".
In case of HDFS, I'm assuming the steps required must be similar, except
the connection string (hdfs://<IP>:<port>)
Also, as Andrews mentioned, there looks to be a typo in your query - that
could very well be the issue.
Thanks,
Abhishek
On Fri, May 22, 2015 at 8:27 AM, Andries Engelbrecht <
aengelbrecht@maprtech.com> wrote:
> In step2 you need to have a back tick ` at the end of par and not a single
> quote ‘ .
>
> It has been mentioned that it may not work to just point to the name node
> on a remote cluster. I have not tried it, but suspect their may be various
> issues with the HDFS plug in and how you are trying to use it.
>
> Perhaps if you can explain why you are trying to do this, there may be
> other alternatives to explore.
>
> What Hadoop distro are you using?
>
> —Andries
>
>
> On May 22, 2015, at 8:17 AM, Alan Miller <Al...@synopsys.com> wrote:
>
> > Thanks Shiran,
> >
> > I tried that but get the same error (see below).
> >
> > Also, strangely I couldn't create the hdfs plugin in one step by using
> the same
> > config as the "dfs" plugin and changing the connection string. The UI
> says Invalid JSON...
> > I had to create the hdfs plugin in 2 steps. First using the same config
> as the dfs plugin.
> > Then updated the hdfs config, by changing the connection string
> >
> > After adding the hdfs plugin with the same config as dfs (but different
> connection
> > string ("connection": "hdfs://10.10.10.10:9000/",) I tried this
> >
> > 1. Copied the file from node1 to remote HDFS
> > [alan@node1 drill]$ hdfs dfs -fs hdfs://10.10.10.10:9000/
> -copyFromLocal ~/test.par /tmp
> > [alan@node1 drill]$ hdfs dfs -fs hdfs://10.10.10.10:9000/ -ls
> /tmp/test.par
> > -rw-r--r-- 1 alan supergroup 4947359 2015-05-22 08:09
> /tmp/test.par
> >
> > 2. From drill on node1
> > [alan@node1 drill]$ /opt/drill/bin/drill-localhost
> > apache drill 1.0.0
> > "json ain't no thang"
> > 0: jdbc:drill:drillbit=localhost> use hdfs;
> > +-------+-----------------------------------+
> > | ok | summary |
> > +-------+-----------------------------------+
> > | true | Default schema changed to [hdfs] |
> > +-------+-----------------------------------+
> > 1 row selected (0.422 seconds)
> > 0: jdbc:drill:drillbit=localhost> select * from
> hdfs.root.`/tmp/test.par' limit 5;
> > Error: PARSE ERROR: Lexical error at line 1, column 55. Encountered:
> <EOF> after : "`/tmp/test.par\' limit 5"
> >
> > [Error Id: 1f793d84-62be-4145-bfcf-2ec3da9cb021 on
> node1.mycompany.com:31010] (state=,code=0)
> > 0: jdbc:drill:drillbit=localhost>
> > 0: jdbc:drill:drillbit=localhost> !quit
> > Closing:
> org.apache.drill.jdbc.DrillJdbc41Factory$DrillJdbc41Connection
> >
> > Alan
>
>
--
Abhishek Girish
Senior Software Engineer
(408) 476-9209
<http://www.mapr.com/>
Re: [newbie]: how to query HDFS
Posted by Andries Engelbrecht <ae...@maprtech.com>.
In step2 you need to have a back tick ` at the end of par and not a single quote ‘ .
It has been mentioned that it may not work to just point to the name node on a remote cluster. I have not tried it, but suspect their may be various issues with the HDFS plug in and how you are trying to use it.
Perhaps if you can explain why you are trying to do this, there may be other alternatives to explore.
What Hadoop distro are you using?
—Andries
On May 22, 2015, at 8:17 AM, Alan Miller <Al...@synopsys.com> wrote:
> Thanks Shiran,
>
> I tried that but get the same error (see below).
>
> Also, strangely I couldn't create the hdfs plugin in one step by using the same
> config as the "dfs" plugin and changing the connection string. The UI says Invalid JSON...
> I had to create the hdfs plugin in 2 steps. First using the same config as the dfs plugin.
> Then updated the hdfs config, by changing the connection string
>
> After adding the hdfs plugin with the same config as dfs (but different connection
> string ("connection": "hdfs://10.10.10.10:9000/",) I tried this
>
> 1. Copied the file from node1 to remote HDFS
> [alan@node1 drill]$ hdfs dfs -fs hdfs://10.10.10.10:9000/ -copyFromLocal ~/test.par /tmp
> [alan@node1 drill]$ hdfs dfs -fs hdfs://10.10.10.10:9000/ -ls /tmp/test.par
> -rw-r--r-- 1 alan supergroup 4947359 2015-05-22 08:09 /tmp/test.par
>
> 2. From drill on node1
> [alan@node1 drill]$ /opt/drill/bin/drill-localhost
> apache drill 1.0.0
> "json ain't no thang"
> 0: jdbc:drill:drillbit=localhost> use hdfs;
> +-------+-----------------------------------+
> | ok | summary |
> +-------+-----------------------------------+
> | true | Default schema changed to [hdfs] |
> +-------+-----------------------------------+
> 1 row selected (0.422 seconds)
> 0: jdbc:drill:drillbit=localhost> select * from hdfs.root.`/tmp/test.par' limit 5;
> Error: PARSE ERROR: Lexical error at line 1, column 55. Encountered: <EOF> after : "`/tmp/test.par\' limit 5"
>
> [Error Id: 1f793d84-62be-4145-bfcf-2ec3da9cb021 on node1.mycompany.com:31010] (state=,code=0)
> 0: jdbc:drill:drillbit=localhost>
> 0: jdbc:drill:drillbit=localhost> !quit
> Closing: org.apache.drill.jdbc.DrillJdbc41Factory$DrillJdbc41Connection
>
> Alan
RE: [newbie]: how to query HDFS
Posted by Alan Miller <Al...@synopsys.com>.
Thanks Shiran,
I tried that but get the same error (see below).
Also, strangely I couldn't create the hdfs plugin in one step by using the same
config as the "dfs" plugin and changing the connection string. The UI says Invalid JSON...
I had to create the hdfs plugin in 2 steps. First using the same config as the dfs plugin.
Then updated the hdfs config, by changing the connection string
After adding the hdfs plugin with the same config as dfs (but different connection
string ("connection": "hdfs://10.10.10.10:9000/",) I tried this
1. Copied the file from node1 to remote HDFS
[alan@node1 drill]$ hdfs dfs -fs hdfs://10.10.10.10:9000/ -copyFromLocal ~/test.par /tmp
[alan@node1 drill]$ hdfs dfs -fs hdfs://10.10.10.10:9000/ -ls /tmp/test.par
-rw-r--r-- 1 alan supergroup 4947359 2015-05-22 08:09 /tmp/test.par
2. From drill on node1
[alan@node1 drill]$ /opt/drill/bin/drill-localhost
apache drill 1.0.0
"json ain't no thang"
0: jdbc:drill:drillbit=localhost> use hdfs;
+-------+-----------------------------------+
| ok | summary |
+-------+-----------------------------------+
| true | Default schema changed to [hdfs] |
+-------+-----------------------------------+
1 row selected (0.422 seconds)
0: jdbc:drill:drillbit=localhost> select * from hdfs.root.`/tmp/test.par' limit 5;
Error: PARSE ERROR: Lexical error at line 1, column 55. Encountered: <EOF> after : "`/tmp/test.par\' limit 5"
[Error Id: 1f793d84-62be-4145-bfcf-2ec3da9cb021 on node1.mycompany.com:31010] (state=,code=0)
0: jdbc:drill:drillbit=localhost>
0: jdbc:drill:drillbit=localhost> !quit
Closing: org.apache.drill.jdbc.DrillJdbc41Factory$DrillJdbc41Connection
Alan
Re: [newbie]: how to query HDFS
Posted by Tomer Shiran <ts...@gmail.com>.
You don't need a drillbit on the cluster. It will be faster (data locality
etc.) but you can just run a drillbit on your client and access any remote
cluster (or even join data from multiple clusters).
It looks like you've created a new storage plugin. I would recommend
copying the entire JSON configuration string from the "dfs" plugin to your
new plugin (named "hdfs" in this example) and changing just the connection
string. That way you'll have all the format plugins registered properly and
the default workspaces. Your connection string actually seems OK.
Then try using the full path to the file/directory as in:
hdfs.root.`/path/to/file` (assuming hdfs is the name of the storage plugin
you created).
Does that help?
On Thu, May 21, 2015 at 5:23 PM, Abhishek Girish <ab...@gmail.com>
wrote:
> Hi Alan,
>
> What you are attempting to do wouldn't work. Without a drillbit running on
> the remote cluster, there is no way I see we can access that file system
> from Drill.
>
> If you'd like to connect to a remote cluster (cluster B), the options I see
> are
>
> (1) Install Drill on cluster B and use a local client :) It shouldn't be
> that hard!
>
> (2) Install Drill on cluster B and a client on cluster A. If you are
> connecting via Sqlline, provide IP of a node on cluster B using the
> "drillbit" parameter (sqlline -u jdbc:drill:drillbit=<IP>)
> I tried this with MapR and it worked for me.
>
> -Abhishek
>
> On Thu, May 21, 2015 at 4:17 PM, Alan Miller <Al...@synopsys.com>
> wrote:
>
> > I tried that initially, but since it didn't work I tried to simplify it
> as
> > much as possible.
> >
> > Are saying it "should" work?. I mean all I need to do is point the
> > connection parameter
> > to a different namenode, right?
> >
> > <http://www.mapr.com/>
> >
>
Re: [newbie]: how to query HDFS
Posted by Abhishek Girish <ab...@gmail.com>.
Hi Alan,
What you are attempting to do wouldn't work. Without a drillbit running on
the remote cluster, there is no way I see we can access that file system
from Drill.
If you'd like to connect to a remote cluster (cluster B), the options I see
are
(1) Install Drill on cluster B and use a local client :) It shouldn't be
that hard!
(2) Install Drill on cluster B and a client on cluster A. If you are
connecting via Sqlline, provide IP of a node on cluster B using the
"drillbit" parameter (sqlline -u jdbc:drill:drillbit=<IP>)
I tried this with MapR and it worked for me.
-Abhishek
On Thu, May 21, 2015 at 4:17 PM, Alan Miller <Al...@synopsys.com>
wrote:
> I tried that initially, but since it didn't work I tried to simplify it as
> much as possible.
>
> Are saying it "should" work?. I mean all I need to do is point the
> connection parameter
> to a different namenode, right?
>
> <http://www.mapr.com/>
>
RE: [newbie]: how to query HDFS
Posted by Alan Miller <Al...@synopsys.com>.
I tried that initially, but since it didn't work I tried to simplify it as much as possible.
Are saying it "should" work?. I mean all I need to do is point the connection parameter
to a different namenode, right?
Re: [newbie]: how to query HDFS
Posted by Andries Engelbrecht <ae...@maprtech.com>.
Alan,
I don't think the path is correct in your query, it is best to set up workspaces in the HDFS plugin
http://drill.apache.org/docs/file-system-storage-plugin/
See if that works.
--Andries
> On May 21, 2015, at 2:04 PM, Alan Miller <Al...@synopsys.com> wrote:
>
> First off, this is my first attempt at drill,
> (BTW: congratulations on the release ;-)
> so perhaps I misunderstood something
>
> I want to query my parquet files on HDFS.
>
> I setup the 1.0 release on a machine (node1)
> that already had CDH5 and a working Zookeeper.
> With the hdfs storage plugin config below I can query a parquet file
> on the local machine just fine.
> E.g.:
> 0: jdbc:drill:drillbit=localhost> select a,b,c FROM hdfs.`/hdfs/path/test.par` limit 5;
>
> ## drill-override.conf
> drill.exec: {
> cluster-id: "mydrillcluster",
> zk.connect: "node1:2181"
> }
> ## storage plugin config
> {
> "type": "file",
> "enabled": true,
> "connection": "hdfs://127.0.0.1:8020/",
> "workspaces": null,
> "formats": {
> "parquet": {
> "type": "parquet"
> }
> }
> }
>
> Can I query a remote HDFS, by simply pointing the storage plugin config?
> After changing the IP address in the connection parameter above, I get this error.
>
> 0: jdbc:drill:drillbit=localhost> select a,b,c FROM hdfs.`/tmp/test.par` limit 5;
>
> Error: PARSE ERROR: From line 1, column 38 to line 1, column 41: Table 'hdfs./tmp/test.par' not found
> [Error Id: 4156f66c-3dac-4e87-b7f8-f0bdc19d57d7 on node1.company.com:31010] (state=,code=0)
> .....
> Caused by: org.apache.drill.common.exceptions.UserRemoteException: PARSE ERROR: From line 1, column 38 to line 1, column 41: Table 'hdfs./tmp/test.par' not found
>
> But the namenode:port/path is correct because this workds from node1:
>
> [alan@node1 drill]$ hdfs dfs -fs hdfs://10.10.10.10:8020/ -ls /tmp/test.par
> -rw-r--r-- 1 alan supergroup 4947359 2015-05-21 13:55 /tmp/test.par
>
>
> Alan