hadoop fs -tail

[Sukhendu Chakraborty <su...@gmail.com>]

Hi,

Is there a way to get around the 1KB limitation of the hadoop fs -tail
command (http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html#tail)?
In my application some of the records can have length greater than 1KB
before the newline and I would like to get the full records as part of
'tail' (not truncated versions).

Thanks,
-Sukhendu

Re: hadoop fs -tail

[Harsh J <ha...@cloudera.com>]

We should be able to extend the Tail command to allow a size
specifier. Could you file a JIRA for this?

Although, I don't think the tail prints newlines at every gap unless
there's a newline in the data itself, so your "listener" can perhaps
wait for that?

On Tue, Jul 17, 2012 at 5:19 AM, Sukhendu Chakraborty
<su...@gmail.com> wrote:
> Hi,
>
> Is there a way to get around the 1KB limitation of the hadoop fs -tail
> command (http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html#tail)?
> In my application some of the records can have length greater than 1KB
> before the newline and I would like to get the full records as part of
> 'tail' (not truncated versions).
>
> Thanks,
> -Sukhendu



-- 
Harsh J

Re: hadoop fs -tail

[Minh Duc Nguyen <md...@gmail.com>]

Sukhendu, I'm not sure how bad performance-wise this would be (depending on
the size of the file you're reading), but you could try something like this:

hadoop fs -text <your file> | tail -n <number of lines>

HTH,
Minh

On Mon, Jul 16, 2012 at 7:49 PM, Sukhendu Chakraborty <
sukhendu.chakraborty@gmail.com> wrote:

> Hi,
>
> Is there a way to get around the 1KB limitation of the hadoop fs -tail
> command (http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html#tail
> )?
> In my application some of the records can have length greater than 1KB
> before the newline and I would like to get the full records as part of
> 'tail' (not truncated versions).
>
> Thanks,
> -Sukhendu
>