You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by jinhwan choi <ch...@gmail.com> on 2019/04/15 11:09:40 UTC

[ISSUE] HADOOP3.1 & Erasure encode => BlockMissingException

Hi Drill Team



I upgrade my cluster HADOOP 3.1。

(apache drill is ver1.5)

And,  I enabled Hadoop 3.X erasure encode feature on this cluster. (and
made some new parquet file)

But after upgrade,  I faced this error.

--------------------------------------------------------------

SYSTEM ERROR: BlockMissingException: Could not obtain block:
BP-124827589-172.23.55.207-1483001524978:blk_-9223372036854744080_1954863
file=/drill/data/log_568803_R_521_2019040510190000.parquet





Please, refer to logs for more information.



[Error Id: 940a3b08-0629-400e-8cfb-fd297e1730c7 on tst-nss-hdn06:31010]

--------------------------------------------------------------



Can anyone please let us know the solution for this or if there are any
alternative.



Regards,



Choi

Re: [ISSUE] HADOOP3.1 & Erasure encode => BlockMissingException

Posted by Kunal Khatua <ku...@apache.org>.
There are multiple possibilities for this. 

1. Apache Drill currently is supported for Hadoop 2.7.4 and not yet for Hadoop 3.x.
2. The Parquet file is of a later version. Drill supports Parquet 1.10.0 
3. Your Apache Drill version is quoted as being 1.5 . I'm guessing that is a typo and you meant 1.15

From the error message (since it is a SYSTEM error), it is most likely #1. I'm not sure erasure coding can play a role, since the FS should be abstracting the erasure coding and the APIs  for reading blocks from the file would (probably) be unchanged between the Hadoop 2.x and 3.x.

~ Kunal

On 4/15/2019 11:45:22 AM, jinhwan choi <ch...@gmail.com> wrote:
Hi Drill Team



I upgrade my cluster HADOOP 3.1。

(apache drill is ver1.5)

And, I enabled Hadoop 3.X erasure encode feature on this cluster. (and
made some new parquet file)

But after upgrade, I faced this error.

--------------------------------------------------------------

SYSTEM ERROR: BlockMissingException: Could not obtain block:
BP-124827589-172.23.55.207-1483001524978:blk_-9223372036854744080_1954863
file=/drill/data/log_568803_R_521_2019040510190000.parquet





Please, refer to logs for more information.



[Error Id: 940a3b08-0629-400e-8cfb-fd297e1730c7 on tst-nss-hdn06:31010]

--------------------------------------------------------------



Can anyone please let us know the solution for this or if there are any
alternative.



Regards,



Choi