You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@tajo.apache.org by Jinho Kim <jh...@apache.org> on 2014/03/05 05:27:24 UTC
Review Request 18761: TAJO-653: RCFileAppender throws IOException
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18761/
-----------------------------------------------------------
Review request for Tajo.
Bugs: TAJO-653
https://issues.apache.org/jira/browse/TAJO-653
Repository: tajo
Description
-------
I ran a bellow query. This query throws IOException in worker
{code}
tajo> create table orders_rc (
o_orderkey INT8,
o_custkey INT8,
o_orderstatus TEXT,
o_totalprice FLOAT8,
o_orderdate TEXT,
o_orderpriority TEXT,
o_clerk TEXT,
o_shippriority INT4,
o_comment TEXT)
using rcfile WITH (compression.codec=org.apache.hadoop.io.compress.GzipCodec)
tajo> insert overwrite into orders_rc select * from orders;
{code}
{code}
2014-03-05 10:29:18,763 INFO zlib.ZlibFactory (ZlibFactory.java:<clinit>(48)) - Successfully loaded & initialized native-zlib library
2014-03-05 10:29:18,765 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:18,936 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:18,954 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:18,963 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:18,975 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:18,979 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:18,993 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:18,998 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:19,014 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:19,028 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:19,039 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:19,080 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:19,094 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
2014-03-05 10:29:22,484 ERROR worker.Task (Task.java:run(383)) - java.io.IOException: write beyond end of stream
at org.apache.hadoop.io.compress.CompressorStream.write(CompressorStream.java:66)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.apache.tajo.storage.rcfile.NonSyncByteArrayOutputStream.writeTo(NonSyncByteArrayOutputStream.java:142)
at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.WriteKeyBuffer(RCFile.java:947)
at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.writeKey(RCFile.java:1046)
at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.flushRecords(RCFile.java:1005)
at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.append(RCFile.java:894)
at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.addTuple(RCFile.java:851)
at org.apache.tajo.engine.planner.physical.StoreTableExec.next(StoreTableExec.java:77)
at org.apache.tajo.worker.Task.run(Task.java:374)
at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:391)
at java.lang.Thread.run(Thread.java:744)
{code}
Diffs
-----
tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java afbcaa86dc3b69f12d4694e6e3f75b2c71a16afc
Diff: https://reviews.apache.org/r/18761/diff/
Testing
-------
Thanks,
Jinho Kim
Re: Review Request 18761: TAJO-653: RCFileAppender throws IOException
Posted by Jinho Kim <jh...@apache.org>.
> On March 5, 2014, 4:36 a.m., Hyunsik Choi wrote:
> > +1
> >
> > This patch seems to very useful for 0.8.0 too. Could you commit it to both branch-0.8.0 and master?
Thank you for the fast review.
I'll commit it to both branch-0.8.0 and master
- Jinho
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18761/#review36206
-----------------------------------------------------------
On March 5, 2014, 4:27 a.m., Jinho Kim wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18761/
> -----------------------------------------------------------
>
> (Updated March 5, 2014, 4:27 a.m.)
>
>
> Review request for Tajo.
>
>
> Bugs: TAJO-653
> https://issues.apache.org/jira/browse/TAJO-653
>
>
> Repository: tajo
>
>
> Description
> -------
>
> I ran a bellow query. This query throws IOException in worker
>
> {code}
> tajo> create table orders_rc (
> o_orderkey INT8,
> o_custkey INT8,
> o_orderstatus TEXT,
> o_totalprice FLOAT8,
> o_orderdate TEXT,
> o_orderpriority TEXT,
> o_clerk TEXT,
> o_shippriority INT4,
> o_comment TEXT)
> using rcfile WITH (compression.codec=org.apache.hadoop.io.compress.GzipCodec)
>
> tajo> insert overwrite into orders_rc select * from orders;
> {code}
>
>
> {code}
> 2014-03-05 10:29:18,763 INFO zlib.ZlibFactory (ZlibFactory.java:<clinit>(48)) - Successfully loaded & initialized native-zlib library
> 2014-03-05 10:29:18,765 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:18,936 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:18,954 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:18,963 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:18,975 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:18,979 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:18,993 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:18,998 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:19,014 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:19,028 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:19,039 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:19,080 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:19,094 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:22,484 ERROR worker.Task (Task.java:run(383)) - java.io.IOException: write beyond end of stream
> at org.apache.hadoop.io.compress.CompressorStream.write(CompressorStream.java:66)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at org.apache.tajo.storage.rcfile.NonSyncByteArrayOutputStream.writeTo(NonSyncByteArrayOutputStream.java:142)
> at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.WriteKeyBuffer(RCFile.java:947)
> at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.writeKey(RCFile.java:1046)
> at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.flushRecords(RCFile.java:1005)
> at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.append(RCFile.java:894)
> at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.addTuple(RCFile.java:851)
> at org.apache.tajo.engine.planner.physical.StoreTableExec.next(StoreTableExec.java:77)
> at org.apache.tajo.worker.Task.run(Task.java:374)
> at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:391)
> at java.lang.Thread.run(Thread.java:744)
> {code}
>
>
> Diffs
> -----
>
> tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java afbcaa86dc3b69f12d4694e6e3f75b2c71a16afc
>
> Diff: https://reviews.apache.org/r/18761/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jinho Kim
>
>
Re: Review Request 18761: TAJO-653: RCFileAppender throws IOException
Posted by Hyunsik Choi <hy...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18761/#review36206
-----------------------------------------------------------
Ship it!
+1
This patch seems to very useful for 0.8.0 too. Could you commit it to both branch-0.8.0 and master?
- Hyunsik Choi
On March 5, 2014, 1:27 p.m., Jinho Kim wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/18761/
> -----------------------------------------------------------
>
> (Updated March 5, 2014, 1:27 p.m.)
>
>
> Review request for Tajo.
>
>
> Bugs: TAJO-653
> https://issues.apache.org/jira/browse/TAJO-653
>
>
> Repository: tajo
>
>
> Description
> -------
>
> I ran a bellow query. This query throws IOException in worker
>
> {code}
> tajo> create table orders_rc (
> o_orderkey INT8,
> o_custkey INT8,
> o_orderstatus TEXT,
> o_totalprice FLOAT8,
> o_orderdate TEXT,
> o_orderpriority TEXT,
> o_clerk TEXT,
> o_shippriority INT4,
> o_comment TEXT)
> using rcfile WITH (compression.codec=org.apache.hadoop.io.compress.GzipCodec)
>
> tajo> insert overwrite into orders_rc select * from orders;
> {code}
>
>
> {code}
> 2014-03-05 10:29:18,763 INFO zlib.ZlibFactory (ZlibFactory.java:<clinit>(48)) - Successfully loaded & initialized native-zlib library
> 2014-03-05 10:29:18,765 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:18,936 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:18,954 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:18,963 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:18,975 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:18,979 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:18,993 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:18,998 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:19,014 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:19,028 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:19,039 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:19,080 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:19,094 INFO compress.CodecPool (CodecPool.java:getCompressor(108)) - Got brand-new compressor [.gz]
> 2014-03-05 10:29:22,484 ERROR worker.Task (Task.java:run(383)) - java.io.IOException: write beyond end of stream
> at org.apache.hadoop.io.compress.CompressorStream.write(CompressorStream.java:66)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at org.apache.tajo.storage.rcfile.NonSyncByteArrayOutputStream.writeTo(NonSyncByteArrayOutputStream.java:142)
> at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.WriteKeyBuffer(RCFile.java:947)
> at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.writeKey(RCFile.java:1046)
> at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.flushRecords(RCFile.java:1005)
> at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.append(RCFile.java:894)
> at org.apache.tajo.storage.rcfile.RCFile$RCFileAppender.addTuple(RCFile.java:851)
> at org.apache.tajo.engine.planner.physical.StoreTableExec.next(StoreTableExec.java:77)
> at org.apache.tajo.worker.Task.run(Task.java:374)
> at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:391)
> at java.lang.Thread.run(Thread.java:744)
> {code}
>
>
> Diffs
> -----
>
> tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java afbcaa86dc3b69f12d4694e6e3f75b2c71a16afc
>
> Diff: https://reviews.apache.org/r/18761/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Jinho Kim
>
>