You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Peter Marron <Pe...@trilliumsoftware.com> on 2013/05/28 16:59:25 UTC

Accessing Table Properies from InputFormat

Hi,

Hive 0.10.0 over Hadoop 1.0.4.

Further to my filtering questions of before.
I would like to be able to access the table properties from inside my custom InputFormat.
I've done searches and there seem to be some other people who have had a similar problem.
The closest I can see to a solution is to use
                MapredWork mrwork = Utilities.getMapRedWork(configuration);
but this fails for me with the error below.
I'm not truly surprised because I and trying to make sure that my query
runs without a map/reduce and some of the e-mails suggest that in this case:

"...no mapred job is
run, so this trick doesn't work (and instead, the Configuration object
can be used, since it's local)."

Any pointers would be very much appreciated.

Z

java.lang.IllegalArgumentException: Can not create a Path from an empty string
                at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
                at org.apache.hadoop.fs.Path.<init>(Path.java:90)
                at org.apache.hadoop.hive.ql.exec.Utilities.getHiveJobID(Utilities.java:382)
                at org.apache.hadoop.hive.ql.exec.Utilities.getMapRedWork(Utilities.java:205)
                at SimpleInputFormat.getThingy(SimpleInputFormat.java:134)
                at SimpleInputFormat.getSplits(SimpleInputFormat.java:90)
                at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:373)
                at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:486)
                at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:466)
                at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
                at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1387)
                at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:270)
                at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
                at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
                at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
                at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Failed with exception java.io.IOException:java.lang.RuntimeException: java.lang.IllegalArgumentException: Can not create a Path from an empty string

Peter Marron
Trillium Software UK Limited

Tel : +44 (0) 118 940 7609
Fax : +44 (0) 118 940 7699
E: Peter.Marron@TrilliumSoftware.com<ma...@TrilliumSoftware.com>

Re: Accessing Table Properies from InputFormat

Posted by Edward Capriolo <ed...@gmail.com>.
ORC, Parquet, and "new ones" are ... new. They do not constitute a huge
portion of the user base if they constitute any at all.

I do see a case for what you are describing, currently there are input
formats that do properties via the configuration to the task. Also I feel
like some of the confusion you are describing centers around someone
building code with the goal of only one input format in mind or one use
case. So when I see "OCR" "could benefit" and "refactor" in the same email
alarms go off. They may be false alarms but if the feature does not
immediately benefit two input formats, its creating fragmentation and I am
not behind it.









On Tue, May 28, 2013 at 1:31 PM, Owen O'Malley <om...@apache.org> wrote:

> On Tue, May 28, 2013 at 9:27 AM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> The question we are diving into is how much of hive is going to be
>> designed around edge cases? Hive really was not made for columnar formats,
>> or self describing data-types. For the most part it handles them fairly
>> well.
>>
>
> I don't view columnar or self describing data-types as an edge case. I
> think in a couple years, the various columnar stores (ORC, Parquet, or new
> ones) and text will be the primary formats. Given the performance advantage
> of binary formats, text should only be used for staging tables.
>
>
>> I am not sure what I believe about refactoring all of hive's guts. How
>> much refactoring and complexity are we going to add to support special
>> cases? I do not think we can justify sweeping API changes for the sake of
>> one new input format, or something that can be done in some other way.
>>
>
> The problem is actually, much bigger. We have a wide range of nested
> abstractions for input/output that all interact in various ways.
>
> org.apache.hadoop.mapred.InputFormat
> org.apache.hadoop.hive.ql.io.HiveInputFormat
> org.apache.hadoop.hive.ql.meta.HiveStorageHandler
> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
> org.apache.hadoop.hive.serde2.SerDe
>
> I would suggest that there is a lot of confusion about the current state
> of what is allowed and what will break things. Furthermore, because
> critical functionality like accessing table properties, partition
> properties, columnar projection, and predicate pushdown has been added
> incrementally, it isn't clear at all how to users what is available and how
> to take advantage of them.
>
> -- Owen
>
>

Re: Accessing Table Properies from InputFormat

Posted by Owen O'Malley <om...@apache.org>.
On Tue, May 28, 2013 at 9:27 AM, Edward Capriolo <ed...@gmail.com>wrote:

> The question we are diving into is how much of hive is going to be
> designed around edge cases? Hive really was not made for columnar formats,
> or self describing data-types. For the most part it handles them fairly
> well.
>

I don't view columnar or self describing data-types as an edge case. I
think in a couple years, the various columnar stores (ORC, Parquet, or new
ones) and text will be the primary formats. Given the performance advantage
of binary formats, text should only be used for staging tables.


> I am not sure what I believe about refactoring all of hive's guts. How
> much refactoring and complexity are we going to add to support special
> cases? I do not think we can justify sweeping API changes for the sake of
> one new input format, or something that can be done in some other way.
>

The problem is actually, much bigger. We have a wide range of nested
abstractions for input/output that all interact in various ways.

org.apache.hadoop.mapred.InputFormat
org.apache.hadoop.hive.ql.io.HiveInputFormat
org.apache.hadoop.hive.ql.meta.HiveStorageHandler
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
org.apache.hadoop.hive.serde2.SerDe

I would suggest that there is a lot of confusion about the current state of
what is allowed and what will break things. Furthermore, because critical
functionality like accessing table properties, partition properties,
columnar projection, and predicate pushdown has been added incrementally,
it isn't clear at all how to users what is available and how to take
advantage of them.

-- Owen

Re: Accessing Table Properies from InputFormat

Posted by Edward Capriolo <ed...@gmail.com>.
Right, hive discarding the key is rather annoying. I have a series of
key+value input formats key only input format etc. Having hive return both
the key and the value, would be a breaking change, but not be very
difficult.

The question we are diving into is how much of hive is going to be designed
around edge cases? Hive really was not made for columnar formats, or self
describing data-types. For the most part it handles them fairly well.

I am not sure what I believe about refactoring all of hive's guts. How much
refactoring and complexity are we going to add to support special cases? I
do not think we can justify sweeping API changes for the sake of one new
input format, or something that can be done in some other way.







On Tue, May 28, 2013 at 12:10 PM, Owen O'Malley <om...@apache.org> wrote:

>
>
>
> On Tue, May 28, 2013 at 8:45 AM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> That does not really make sense. Your breaking the layered approache.
>> InputFormats read/write data, serdes interpret data based on the table
>> definition. its like asking "Why can't my input format run assembly code?"
>>
>
> The current model of:
>
> SerDe
> Input/OutputFormat
> FileSystem
>
> does well for text formats, but otherwise limits the input/output formats
> to doing binary data. That creates problems if the Input/OutputFormat has
> an integrated serialization mechanism. For example, ORC requires its SerDe
> and the OrcSerde just passes along the values through serialize and
> deserialize.
>
> Also note that other formats like SequenceFile are restricted because the
> SerDe is placed above the FileFormat. Hive's SequenceFile input format
> discards the key and requires the value to be Text or BytesWritable. That
> covers many cases, but certainly not all. On the other hand, if it was
> Hive's SequenceFile InputFormat that was creating the ObjectInspector, it
> could actually handle more complex types and let Hive usefully read a wider
> range of SequenceFiles.
>
> I would propose that it would be better to push SerDes down into the
> Input/OutputFormats that can be parameterized by the serialization. Using
> them for TextInput/OutputFormat and HBaseTableInput/OutputFormat makes a
> lot of sense, but in general that isn't true.
>
> -- Owen
>

Re: Accessing Table Properies from InputFormat

Posted by Owen O'Malley <om...@apache.org>.
On Tue, May 28, 2013 at 8:45 AM, Edward Capriolo <ed...@gmail.com>wrote:

> That does not really make sense. Your breaking the layered approache.
> InputFormats read/write data, serdes interpret data based on the table
> definition. its like asking "Why can't my input format run assembly code?"
>

The current model of:

SerDe
Input/OutputFormat
FileSystem

does well for text formats, but otherwise limits the input/output formats
to doing binary data. That creates problems if the Input/OutputFormat has
an integrated serialization mechanism. For example, ORC requires its SerDe
and the OrcSerde just passes along the values through serialize and
deserialize.

Also note that other formats like SequenceFile are restricted because the
SerDe is placed above the FileFormat. Hive's SequenceFile input format
discards the key and requires the value to be Text or BytesWritable. That
covers many cases, but certainly not all. On the other hand, if it was
Hive's SequenceFile InputFormat that was creating the ObjectInspector, it
could actually handle more complex types and let Hive usefully read a wider
range of SequenceFiles.

I would propose that it would be better to push SerDes down into the
Input/OutputFormats that can be parameterized by the serialization. Using
them for TextInput/OutputFormat and HBaseTableInput/OutputFormat makes a
lot of sense, but in general that isn't true.

-- Owen

RE: Accessing Table Properies from InputFormat

Posted by Peter Marron <Pe...@trilliumsoftware.com>.
Hi,

I am a newbie and I don't want to break any layered abstractions.

I am in the situation where I want to be able to examine
the predicate in the query and if it's a filter that I recognize
then I would like to use it to cut down on the number of records
processed. In particular I would like to make sure that in such a case the records
aren't even read, so they don't need to be filtered. At the moment
I can see that by providing my own InputFormat I can arrange that
the splits that I return are filtered down to the subset that I want.
However this means that the InputFormat needs to know something
about the table to be able to parse the predicate and see if it matches
the filtering criteria. But I learn that the InputFormat doesn't have access
to the table properties. So I have a problem.

OK, the serde has access to the table properties but it's in no position
to be able to perform the filtering. By the time it sees a record it's too late.

Similarly by the time the recordReader is invoked the record has been read.

I would use a facility like indexing, but I want this to work when the query
does not perform a Map/Reduce and my understanding is that Hive will
not invoke an indexes if there is no Map/Reduce. So indexing is a non-starter.
Also there are cases where creating an index seems massive overkill for
what I am trying to achieve.

So where is the Hive hook that allows me to do what I would like to do?
Which of the layers allows me to examine the table properties and
the predicate and to (pre-)filter the records returned?

Or are you saying that what I am trying to do doesn't make sense?

Z


From: Edward Capriolo [mailto:edlinuxguru@gmail.com]
Sent: 28 May 2013 16:45
To: user@hive.apache.org
Cc: Peter Marron
Subject: Re: Accessing Table Properies from InputFormat

That does not really make sense. Your breaking the layered approache. InputFormats read/write data, serdes interpret data based on the table definition. its like asking "Why can't my input format run assembly code?"

On Tue, May 28, 2013 at 11:42 AM, Owen O'Malley <om...@apache.org>> wrote:


On Tue, May 28, 2013 at 7:59 AM, Peter Marron <Pe...@trilliumsoftware.com>> wrote:
Hi,

Hive 0.10.0 over Hadoop 1.0.4.

Further to my filtering questions of before.
I would like to be able to access the table properties from inside my custom InputFormat.
I've done searches and there seem to be some other people who have had a similar problem.
The closest I can see to a solution is to use
                MapredWork mrwork = Utilities.getMapRedWork(configuration);
but this fails for me with the error below.
I'm not truly surprised because I and trying to make sure that my query
runs without a map/reduce and some of the e-mails suggest that in this case:

"...no mapred job is
run, so this trick doesn't work (and instead, the Configuration object
can be used, since it's local)."

Any pointers would be very much appreciated.

Yeah, as you discovered, that only works in the MapReduce case and breaks on cases like "select count(*)" that don't run in MapReduce.

I haven't tried it, but it looks like the best you can do with the current interface is to implement a SerDe which is passed the table properties in initialize. In terms of passing it to the InputFormat, I'd try a thread local variable. It looks like the getRecordReader is called soon after the serde.initialize although I didn't do a very deep search of the code.

-- Owen

Re: Accessing Table Properies from InputFormat

Posted by Edward Capriolo <ed...@gmail.com>.
That does not really make sense. Your breaking the layered approache.
InputFormats read/write data, serdes interpret data based on the table
definition. its like asking "Why can't my input format run assembly code?"


On Tue, May 28, 2013 at 11:42 AM, Owen O'Malley <om...@apache.org> wrote:

>
>
>
> On Tue, May 28, 2013 at 7:59 AM, Peter Marron <
> Peter.Marron@trilliumsoftware.com> wrote:
>
>>  Hi,****
>>
>> ** **
>>
>> Hive 0.10.0 over Hadoop 1.0.4.****
>>
>> ** **
>>
>> Further to my filtering questions of before.****
>>
>> I would like to be able to access the table properties from inside my
>> custom InputFormat.****
>>
>> I’ve done searches and there seem to be some other people who have had a
>> similar problem.****
>>
>> The closest I can see to a solution is to use ****
>>
>>                 MapredWork mrwork =
>> Utilities.getMapRedWork(configuration);****
>>
>> but this fails for me with the error below.****
>>
>> I’m not truly surprised because I and trying to make sure that my query**
>> **
>>
>> runs without a map/reduce and some of the e-mails suggest that in this
>> case:****
>>
>> ** **
>>
>> “…no mapred job is
>> run, so this trick doesn't work (and instead, the Configuration object
>> can be used, since it's local).”****
>>
>> ** **
>>
>> Any pointers would be very much appreciated.
>>
>
> Yeah, as you discovered, that only works in the MapReduce case and breaks
> on cases like "select count(*)" that don't run in MapReduce.
>
> I haven't tried it, but it looks like the best you can do with the current
> interface is to implement a SerDe which is passed the table properties in
> initialize. In terms of passing it to the InputFormat, I'd try a thread
> local variable. It looks like the getRecordReader is called soon after the
> serde.initialize although I didn't do a very deep search of the code.
>
> -- Owen
>
>
>

Re: Accessing Table Properies from InputFormat

Posted by Owen O'Malley <om...@apache.org>.
On Tue, May 28, 2013 at 7:59 AM, Peter Marron <
Peter.Marron@trilliumsoftware.com> wrote:

>  Hi,****
>
> ** **
>
> Hive 0.10.0 over Hadoop 1.0.4.****
>
> ** **
>
> Further to my filtering questions of before.****
>
> I would like to be able to access the table properties from inside my
> custom InputFormat.****
>
> I’ve done searches and there seem to be some other people who have had a
> similar problem.****
>
> The closest I can see to a solution is to use ****
>
>                 MapredWork mrwork = Utilities.getMapRedWork(configuration);
> ****
>
> but this fails for me with the error below.****
>
> I’m not truly surprised because I and trying to make sure that my query***
> *
>
> runs without a map/reduce and some of the e-mails suggest that in this
> case:****
>
> ** **
>
> “…no mapred job is
> run, so this trick doesn't work (and instead, the Configuration object
> can be used, since it's local).”****
>
> ** **
>
> Any pointers would be very much appreciated.
>

Yeah, as you discovered, that only works in the MapReduce case and breaks
on cases like "select count(*)" that don't run in MapReduce.

I haven't tried it, but it looks like the best you can do with the current
interface is to implement a SerDe which is passed the table properties in
initialize. In terms of passing it to the InputFormat, I'd try a thread
local variable. It looks like the getRecordReader is called soon after the
serde.initialize although I didn't do a very deep search of the code.

-- Owen