You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Adam Silberstein <ad...@trifacta.com> on 2013/06/14 18:43:05 UTC
webhdfs read error after successful pig job
Hi,
I'm having some trouble with webhdfs read after running a Pig job that completed successfully.
Here are some details:
-I am using Hadoop CDH-4.1.3 and the compatible Pig that goes with this (0.10.0 I think)
-The Pig job writes out about 10 files. I'm programmatically attempting to read each of these with webhdfs soon after pig notifies me the job is complete. The reads often all succeed. And even in the failure case, most of the reads still succeed, but one may fail.
-I wondered if I was facing a race condition where Pig was reporting success before the file was truly ready to read. However, when I run WebHDFS read with curl even hours later, the request hangs. In contrast, I can run 'cat' from the DFS command line and the file is output correctly.
-I ran fsck over the problem file and it report back totally normal.
-I looked at the namenode to see why my curl request hangs. I get this error:
ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:ubuntu (auth:SIMPLE) cause:java.io.IOException: Could not reach the block containing the data. Please try again
(I'm guessing the permissions aren't really the important thing here, the underlying cause of not reaching the block seems more reasonable).
-I have a 4 node cluster with replication set to 1.
If anyone has seen this, has diagnostic tips, or best of all, a solution, please let me know!
Thanks,
Adam
Re: webhdfs read error after successful pig job
Posted by Ed Serrano <ed...@gmail.com>.
You might want to investigate if your issue is aways on the same node.
On Fri, Jun 14, 2013 at 11:43 AM, Adam Silberstein <ad...@trifacta.com>wrote:
> Hi,
> I'm having some trouble with webhdfs read after running a Pig job that
> completed successfully.
>
> Here are some details:
>
> -I am using Hadoop CDH-4.1.3 and the compatible Pig that goes with this
> (0.10.0 I think)
>
> -The Pig job writes out about 10 files. I'm programmatically attempting
> to read each of these with webhdfs soon after pig notifies me the job is
> complete. The reads often all succeed. And even in the failure case, most
> of the reads still succeed, but one may fail.
>
> -I wondered if I was facing a race condition where Pig was reporting
> success before the file was truly ready to read. However, when I run
> WebHDFS read with curl even hours later, the request hangs. In contrast, I
> can run 'cat' from the DFS command line and the file is output correctly.
>
> -I ran fsck over the problem file and it report back totally normal.
>
> -I looked at the namenode to see why my curl request hangs. I get this
> error:
> ERROR org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:ubuntu (auth:SIMPLE)
> cause:java.io.IOException: Could not reach the block containing the data.
> Please try again
> (I'm guessing the permissions aren't really the important thing here, the
> underlying cause of not reaching the block seems more reasonable).
>
> -I have a 4 node cluster with replication set to 1.
>
>
> If anyone has seen this, has diagnostic tips, or best of all, a solution,
> please let me know!
>
> Thanks,
> Adam
>
>
>
--
-------------------------------------
*Ed Serrano*
Mobile: 972-897-5443
Re: webhdfs read error after successful pig job
Posted by Ed Serrano <ed...@gmail.com>.
You might want to investigate if your issue is aways on the same node.
On Fri, Jun 14, 2013 at 11:43 AM, Adam Silberstein <ad...@trifacta.com>wrote:
> Hi,
> I'm having some trouble with webhdfs read after running a Pig job that
> completed successfully.
>
> Here are some details:
>
> -I am using Hadoop CDH-4.1.3 and the compatible Pig that goes with this
> (0.10.0 I think)
>
> -The Pig job writes out about 10 files. I'm programmatically attempting
> to read each of these with webhdfs soon after pig notifies me the job is
> complete. The reads often all succeed. And even in the failure case, most
> of the reads still succeed, but one may fail.
>
> -I wondered if I was facing a race condition where Pig was reporting
> success before the file was truly ready to read. However, when I run
> WebHDFS read with curl even hours later, the request hangs. In contrast, I
> can run 'cat' from the DFS command line and the file is output correctly.
>
> -I ran fsck over the problem file and it report back totally normal.
>
> -I looked at the namenode to see why my curl request hangs. I get this
> error:
> ERROR org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:ubuntu (auth:SIMPLE)
> cause:java.io.IOException: Could not reach the block containing the data.
> Please try again
> (I'm guessing the permissions aren't really the important thing here, the
> underlying cause of not reaching the block seems more reasonable).
>
> -I have a 4 node cluster with replication set to 1.
>
>
> If anyone has seen this, has diagnostic tips, or best of all, a solution,
> please let me know!
>
> Thanks,
> Adam
>
>
>
--
-------------------------------------
*Ed Serrano*
Mobile: 972-897-5443
Re: webhdfs read error after successful pig job
Posted by Ed Serrano <ed...@gmail.com>.
You might want to investigate if your issue is aways on the same node.
On Fri, Jun 14, 2013 at 11:43 AM, Adam Silberstein <ad...@trifacta.com>wrote:
> Hi,
> I'm having some trouble with webhdfs read after running a Pig job that
> completed successfully.
>
> Here are some details:
>
> -I am using Hadoop CDH-4.1.3 and the compatible Pig that goes with this
> (0.10.0 I think)
>
> -The Pig job writes out about 10 files. I'm programmatically attempting
> to read each of these with webhdfs soon after pig notifies me the job is
> complete. The reads often all succeed. And even in the failure case, most
> of the reads still succeed, but one may fail.
>
> -I wondered if I was facing a race condition where Pig was reporting
> success before the file was truly ready to read. However, when I run
> WebHDFS read with curl even hours later, the request hangs. In contrast, I
> can run 'cat' from the DFS command line and the file is output correctly.
>
> -I ran fsck over the problem file and it report back totally normal.
>
> -I looked at the namenode to see why my curl request hangs. I get this
> error:
> ERROR org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:ubuntu (auth:SIMPLE)
> cause:java.io.IOException: Could not reach the block containing the data.
> Please try again
> (I'm guessing the permissions aren't really the important thing here, the
> underlying cause of not reaching the block seems more reasonable).
>
> -I have a 4 node cluster with replication set to 1.
>
>
> If anyone has seen this, has diagnostic tips, or best of all, a solution,
> please let me know!
>
> Thanks,
> Adam
>
>
>
--
-------------------------------------
*Ed Serrano*
Mobile: 972-897-5443
Re: webhdfs read error after successful pig job
Posted by Ed Serrano <ed...@gmail.com>.
You might want to investigate if your issue is aways on the same node.
On Fri, Jun 14, 2013 at 11:43 AM, Adam Silberstein <ad...@trifacta.com>wrote:
> Hi,
> I'm having some trouble with webhdfs read after running a Pig job that
> completed successfully.
>
> Here are some details:
>
> -I am using Hadoop CDH-4.1.3 and the compatible Pig that goes with this
> (0.10.0 I think)
>
> -The Pig job writes out about 10 files. I'm programmatically attempting
> to read each of these with webhdfs soon after pig notifies me the job is
> complete. The reads often all succeed. And even in the failure case, most
> of the reads still succeed, but one may fail.
>
> -I wondered if I was facing a race condition where Pig was reporting
> success before the file was truly ready to read. However, when I run
> WebHDFS read with curl even hours later, the request hangs. In contrast, I
> can run 'cat' from the DFS command line and the file is output correctly.
>
> -I ran fsck over the problem file and it report back totally normal.
>
> -I looked at the namenode to see why my curl request hangs. I get this
> error:
> ERROR org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:ubuntu (auth:SIMPLE)
> cause:java.io.IOException: Could not reach the block containing the data.
> Please try again
> (I'm guessing the permissions aren't really the important thing here, the
> underlying cause of not reaching the block seems more reasonable).
>
> -I have a 4 node cluster with replication set to 1.
>
>
> If anyone has seen this, has diagnostic tips, or best of all, a solution,
> please let me know!
>
> Thanks,
> Adam
>
>
>
--
-------------------------------------
*Ed Serrano*
Mobile: 972-897-5443