Querying multiple TSV or CSV files at once?

[Minnow Noir <mi...@gmail.com>]

I'm trying to do ad-hoc exploration/analysis over multiple files without
having to concatenate them.  New files show up on a regular basis, and
creating large, redundant concatenated files seems inelegant for data
exploration.  I've tried the obvious (... from dfs.`/dir/*.csv` but that
only returns lines from the first file it finds, and then an error for the
next file.

Is there any current way to do this?


Thanks

Re: Querying multiple TSV or CSV files at once?

[Minnow Noir <mi...@gmail.com>]

"Failure while running fragment."

Here's an example doing it the way you suggest.  (Last 100 lines of
sqlline.log are also attached.)  It sounds like you think the multi-file
query should actually work?

Thanks




0: jdbc:drill:zk=local> select * from dfs.`/data` limit 5;
+------------+
|  columns   |
+------------+

 <first 5 rows of data from first file>

Query failed: Query failed: Failure while running fragment., refCnt: 0,
decrement: 1 [ 29674d0f-bec3-4175-b1df-3afb68c6e76f on
sandbox.hortonworks.com:31010 ]
[ 29674d0f-bec3-4175-b1df-3afb68c6e76f on sandbox.hortonworks.com:31010 ]


java.lang.RuntimeException: java.sql.SQLException: Failure while executing
query.
        at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
        at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
        at sqlline.SqlLine.print(SqlLine.java:1809)
        at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
        at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
        at sqlline.SqlLine.dispatch(SqlLine.java:889)
        at sqlline.SqlLine.begin(SqlLine.java:763)
        at sqlline.SqlLine.start(SqlLine.java:498)
        at sqlline.SqlLine.main(SqlLine.java:460)

On Sun, Feb 8, 2015 at 3:21 PM, Neeraja Rentachintala <
nrentachintala@maprtech.com> wrote:

> What is the error that you are seeing?
> Can you simply point it to the directory (without *.csv) to see if it
> helps.
>
>
> On Sun, Feb 8, 2015 at 10:33 AM, Minnow Noir <mi...@gmail.com> wrote:
>
> > I'm trying to do ad-hoc exploration/analysis over multiple files without
> > having to concatenate them.  New files show up on a regular basis, and
> > creating large, redundant concatenated files seems inelegant for data
> > exploration.  I've tried the obvious (... from dfs.`/dir/*.csv` but that
> > only returns lines from the first file it finds, and then an error for
> the
> > next file.
> >
> > Is there any current way to do this?
> >
> >
> > Thanks
> >
>

Re: Querying multiple TSV or CSV files at once?

[Neeraja Rentachintala <nr...@maprtech.com>]

What is the error that you are seeing?
Can you simply point it to the directory (without *.csv) to see if it helps.


On Sun, Feb 8, 2015 at 10:33 AM, Minnow Noir <mi...@gmail.com> wrote:

> I'm trying to do ad-hoc exploration/analysis over multiple files without
> having to concatenate them.  New files show up on a regular basis, and
> creating large, redundant concatenated files seems inelegant for data
> exploration.  I've tried the obvious (... from dfs.`/dir/*.csv` but that
> only returns lines from the first file it finds, and then an error for the
> next file.
>
> Is there any current way to do this?
>
>
> Thanks
>