You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@orc.apache.org by "Mcbride, Neil" <ne...@cwc.com> on 2015/11/10 14:11:14 UTC
Tez / Orc / S3
Hi there!
ORC sounds perfect for our use case (lots of Hive queries of the type
'WHERE subscriber = '12345678'). However, I can't seem to see any
performance gains over the use of standard LZO compression when the files
are stored on S3. I raised a case to AWS and was told they also weren't
seeing any benefits. The engineer I spoke to used to work for Hortonworks
and said he'd had great experiences with ORC.
Before I write it off completely for us, I wondered if you were aware of
any particular set up that is required to make better use of ORC on S3?
A typical setup is:
CREATE EXTERNAL TABLE test
180_columns_wide
) STORED AS ORC
LOCATION 's3://bucketname/orc/'
TBLPROPERTIES ('orc.compress'='SNAPPY')
;
select * from test
where subscriber_id = '12345678'
We are using Tez on EMR 4.1 (which uses Hive 1.0, I believe).
--
Regards
Neil
Re: Tez / Orc / S3
Posted by "Mcbride, Neil" <ne...@cwc.com>.
"This FS impl should have that fix."
No joy - I get the same message.
On 11 November 2015 at 14:07, Gopal Vijayaraghavan <go...@apache.org>
wrote:
>
> > Caused by: java.lang.ArithmeticException: / by zero
> > at
> >org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.run(OrcInpu
> >tFormat.java:830)
>
> (currentOffset / blockSize != stripe.getOffset() / blockSize)
>
>
> Looks like this was fixed only recently in S3A impl.
>
> https://issues.apache.org/jira/browse/HADOOP-11584
>
>
> For backwards compat, ORC should probably do a Math.min(blockSize, 1)
> internally.
>
> >> Rajesh has got a backwards port of the seek improvements as a new S3A FS
> >> impl, for drop-in replacement for those who can't wait for the next
> >> release to try it out.
> >>
> >> https://github.com/rajeshbalamohan/hadoop-aws
>
>
> This FS impl should have that fix.
>
> Cheers,
> Gopal
>
>
>
--
Regards
Neil
Re: Tez / Orc / S3
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> Caused by: java.lang.ArithmeticException: / by zero
> at
>org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.run(OrcInpu
>tFormat.java:830)
(currentOffset / blockSize != stripe.getOffset() / blockSize)
Looks like this was fixed only recently in S3A impl.
https://issues.apache.org/jira/browse/HADOOP-11584
For backwards compat, ORC should probably do a Math.min(blockSize, 1)
internally.
>> Rajesh has got a backwards port of the seek improvements as a new S3A FS
>> impl, for drop-in replacement for those who can't wait for the next
>> release to try it out.
>>
>> https://github.com/rajeshbalamohan/hadoop-aws
This FS impl should have that fix.
Cheers,
Gopal
Re: Tez / Orc / S3
Posted by "Mcbride, Neil" <ne...@cwc.com>.
Hi,
I tried the following:
CREATE EXTERNAL TABLE ocs_test (
cols...
) STORED AS ORC
LOCATION 's3a://<bucketname>/orc/'
TBLPROPERTIES ('orc.compress'='SNAPPY')
;
select subscriber_id from ocs_test limit 100
java.io.IOException: java.lang.RuntimeException: serious problem
I did also try the .jar file supplied. Here's the full log output - divide
by zero error:
INFO : Session is already open
INFO : Tez session was closed. Reopening...
INFO : Session re-established.
INFO :
INFO : Status: Running (Executing on YARN cluster with App id
application_1447235341036_0003)
INFO : Map 1: -/-
INFO : Map 1: -/-
ERROR : Status: Failed
ERROR : Vertex failed, vertexName=Map 1,
vertexId=vertex_1447235341036_0003_1_00, diagnostics=[Vertex
vertex_1447235341036_0003_1_00 [Map 1] killed/failed due
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: comverse.ocs_test initializer
failed, vertex=vertex_1447235341036_0003_1_00 [Map 1],
java.lang.RuntimeException: serious problem
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$Context.waitForTasks(OrcInputFormat.java:478)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:949)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:974)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:298)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplitsInternal(HiveInputFormat.java:412)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:330)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:129)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:245)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:239)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:239)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ArithmeticException: / by zero
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.run(OrcInputFormat.java:830)
... 3 more
]
ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1
killedVertices:0
On 10 November 2015 at 20:38, Gopal Vijayaraghavan <go...@apache.org>
wrote:
>
> > Tez is not officially supported on EMR yet but they provide bootstraps
> >for Tez 0.7. I've been using it for a month (via Hue 3.7.1) and it's
> >absolutely fine.
>
> Point me to it (off-list?), I didn't see one at emr-bootstrap-actions.
>
> > I've recreated my table as s3a and am currently inserting 670m records
> >into it. Will feedback but it sounds like I will need to wait for AWS to
> >move to Hive 2.0 to get the full benefits.
>
> Rajesh has got a backwards port of the seek improvements as a new S3A FS
> impl, for drop-in replacement for those who can't wait for the next
> release to try it out.
>
> https://github.com/rajeshbalamohan/hadoop-aws
>
>
> Cheers,
> Gopal
>
>
>
--
Regards
Neil
Re: Tez / Orc / S3
Posted by Gopal Vijayaraghavan <go...@apache.org>.
> Tez is not officially supported on EMR yet but they provide bootstraps
>for Tez 0.7. I've been using it for a month (via Hue 3.7.1) and it's
>absolutely fine.
Point me to it (off-list?), I didn't see one at emr-bootstrap-actions.
> I've recreated my table as s3a and am currently inserting 670m records
>into it. Will feedback but it sounds like I will need to wait for AWS to
>move to Hive 2.0 to get the full benefits.
Rajesh has got a backwards port of the seek improvements as a new S3A FS
impl, for drop-in replacement for those who can't wait for the next
release to try it out.
https://github.com/rajeshbalamohan/hadoop-aws
Cheers,
Gopal
Re: Tez / Orc / S3
Posted by "Mcbride, Neil" <ne...@cwc.com>.
Thank you Alan, Prasanth and Gopal.
Yes, select * was a bad example, but if you watch users it is always the
first thing they do ;-) Generally though it's revenue by area, minutes by
product, etc. which is perfect for columnar data.
Tez is not officially supported on EMR yet but they provide bootstraps for
Tez 0.7. I've been using it for a month (via Hue 3.7.1) and it's absolutely
fine.
I've recreated my table as s3a and am currently inserting 670m records into
it. Will feedback but it sounds like I will need to wait for AWS to move to
Hive 2.0 to get the full benefits.
Regards
Neil
On 10 November 2015 at 19:24, Gopal Vijayaraghavan <go...@apache.org>
wrote:
> Hi,
>
> >
> >http://mail-archives.apache.org/mod_mbox/orc-user/201509.mbox/%3c560AB8D2
> .
> >7070809@darose.net%3e
> ...
> > ORC does a lot of seeks inside its files in order to only load the data
> >you need. S3 doesn't handle seeks well, so ORC does not give you the
> >same improvements that you would see using it on HDFS directly.
>
> ORC changed the way it generates seeks recently in hive-2.0, to get
> connection re-use working (HIVE-11945).
>
> The S3A drivers still need to be fixed to handle seeks via HTTP range
> requests (HADOOP-12444), but the EMR drivers are better at it I think.
>
> > select * from test where subscriber_id = '12345678'
>
> Are the filter columns strings?
>
>
> I think the version you're running doesn't have bloom filter indexes,
> which is somewhat necessary for strings (since a uniformly distributed 1
> byte prefix effectively ruins regular index lookups).
>
>
> You can work around that issue by laying out data in order to get a tight
> grouping
>
> insert overwrite table test as select * from test sort by subscriber_id;
> -- sort by, not order by
>
> Also "select *" is a corner case, since it doesn't get you any benefit of
> the columnar layout, since all columns are being read.
>
> > We are using Tez on EMR 4.1 (which uses Hive 1.0, I believe).
>
>
> Wow, I did not know this. I will try this.
>
> Cheers,
> Gopal
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
--
Regards
Neil
Re: Tez / Orc / S3
Posted by Gopal Vijayaraghavan <go...@apache.org>.
Hi,
>
>http://mail-archives.apache.org/mod_mbox/orc-user/201509.mbox/%3c560AB8D2.
>7070809@darose.net%3e
...
> ORC does a lot of seeks inside its files in order to only load the data
>you need. S3 doesn't handle seeks well, so ORC does not give you the
>same improvements that you would see using it on HDFS directly.
ORC changed the way it generates seeks recently in hive-2.0, to get
connection re-use working (HIVE-11945).
The S3A drivers still need to be fixed to handle seeks via HTTP range
requests (HADOOP-12444), but the EMR drivers are better at it I think.
> select * from test where subscriber_id = '12345678'
Are the filter columns strings?
I think the version you're running doesn't have bloom filter indexes,
which is somewhat necessary for strings (since a uniformly distributed 1
byte prefix effectively ruins regular index lookups).
You can work around that issue by laying out data in order to get a tight
grouping
insert overwrite table test as select * from test sort by subscriber_id;
-- sort by, not order by
Also "select *" is a corner case, since it doesn't get you any benefit of
the columnar layout, since all columns are being read.
> We are using Tez on EMR 4.1 (which uses Hive 1.0, I believe).
Wow, I did not know this. I will try this.
Cheers,
Gopal
Re: Tez / Orc / S3
Posted by Prasanth J <j....@gmail.com>.
Oops.. missed the link
http://mail-archives.apache.org/mod_mbox/orc-user/201509.mbox/%3c560AB8D2.7070809@darose.net%3e
Thanks
Prasanth
> On Nov 10, 2015, at 1:10 PM, Prasanth J <j....@gmail.com> wrote:
>
> Please read this similar thread for more context on why S3 is slow. You should be using newer s3a implementation which got huge performance gains over old s3n.
>
> Thanks
> Prasanth
>
>
>> On Nov 10, 2015, at 1:05 PM, Alan Gates <alanfgates@gmail.com <ma...@gmail.com>> wrote:
>>
>> ORC does a lot of seeks inside its files in order to only load the data you need. S3 doesn't handle seeks well, so ORC does not give you the same improvements that you would see using it on HDFS directly.
>>
>> Alan.
>>
>>> Mcbride, Neil <ma...@cwc.com> November 10, 2015 at 5:11
>>> Hi there!
>>>
>>> ORC sounds perfect for our use case (lots of Hive queries of the type 'WHERE subscriber = '12345678'). However, I can't seem to see any performance gains over the use of standard LZO compression when the files are stored on S3. I raised a case to AWS and was told they also weren't seeing any benefits. The engineer I spoke to used to work for Hortonworks and said he'd had great experiences with ORC.
>>>
>>> Before I write it off completely for us, I wondered if you were aware of any particular set up that is required to make better use of ORC on S3?
>>>
>>> A typical setup is:
>>>
>>> CREATE EXTERNAL TABLE test
>>> 180_columns_wide
>>> ) STORED AS ORC
>>> LOCATION 's3://bucketname/orc/' <s3://bucketname/orc/'>
>>> TBLPROPERTIES ('orc.compress'='SNAPPY')
>>> ;
>>>
>>> select * from test
>>> where subscriber_id = '12345678'
>>>
>>> We are using Tez on EMR 4.1 (which uses Hive 1.0, I believe).
>>>
>>> --
>>> Regards
>>> Neil
>
Re: Tez / Orc / S3
Posted by Prasanth J <j....@gmail.com>.
Please read this similar thread for more context on why S3 is slow. You should be using newer s3a implementation which got huge performance gains over old s3n.
Thanks
Prasanth
> On Nov 10, 2015, at 1:05 PM, Alan Gates <al...@gmail.com> wrote:
>
> ORC does a lot of seeks inside its files in order to only load the data you need. S3 doesn't handle seeks well, so ORC does not give you the same improvements that you would see using it on HDFS directly.
>
> Alan.
>
>> Mcbride, Neil <ma...@cwc.com> November 10, 2015 at 5:11
>> Hi there!
>>
>> ORC sounds perfect for our use case (lots of Hive queries of the type 'WHERE subscriber = '12345678'). However, I can't seem to see any performance gains over the use of standard LZO compression when the files are stored on S3. I raised a case to AWS and was told they also weren't seeing any benefits. The engineer I spoke to used to work for Hortonworks and said he'd had great experiences with ORC.
>>
>> Before I write it off completely for us, I wondered if you were aware of any particular set up that is required to make better use of ORC on S3?
>>
>> A typical setup is:
>>
>> CREATE EXTERNAL TABLE test
>> 180_columns_wide
>> ) STORED AS ORC
>> LOCATION 's3://bucketname/orc/'
>> TBLPROPERTIES ('orc.compress'='SNAPPY')
>> ;
>>
>> select * from test
>> where subscriber_id = '12345678'
>>
>> We are using Tez on EMR 4.1 (which uses Hive 1.0, I believe).
>>
>> --
>> Regards
>> Neil
Re: Tez / Orc / S3
Posted by Alan Gates <al...@gmail.com>.
ORC does a lot of seeks inside its files in order to only load the data
you need. S3 doesn't handle seeks well, so ORC does not give you the
same improvements that you would see using it on HDFS directly.
Alan.
> Mcbride, Neil <ma...@cwc.com>
> November 10, 2015 at 5:11
> Hi there!
>
> ORC sounds perfect for our use case (lots of Hive queries of the type
> 'WHERE subscriber = '12345678'). However, I can't seem to see any
> performance gains over the use of standard LZO compression when the
> files are stored on S3. I raised a case to AWS and was told they also
> weren't seeing any benefits. The engineer I spoke to used to work for
> Hortonworks and said he'd had great experiences with ORC.
>
> Before I write it off completely for us, I wondered if you were aware
> of any particular set up that is required to make better use of ORC on S3?
>
> A typical setup is:
>
> CREATE EXTERNAL TABLE test
> 180_columns_wide
> ) STORED AS ORC
> LOCATION 's3://bucketname/orc/'
> TBLPROPERTIES ('orc.compress'='SNAPPY')
> ;
>
> select * from test
> where subscriber_id = '12345678'
>
> We are using Tez on EMR 4.1 (which uses Hive 1.0, I believe).
>
> --
> Regards
> Neil