You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by John Lilley <jo...@redpoint.net> on 2014/02/09 15:27:00 UTC
RE: HDFS read stats
Thanks! I would have never found that.
john
From: Ted Yu [mailto:yuzhihong@gmail.com]
Sent: Monday, January 27, 2014 4:57 PM
To: common-user@hadoop.apache.org
Subject: Re: HDFS read stats
FSDataInputStream has this javadoc:
/** Utility that wraps a {@link FSInputStream} in a {@link DataInputStream}
You can utilize this method:
@InterfaceAudience.LimitedPrivate({"HDFS"})
public InputStream getWrappedStream() {
return in;
And cast the return value to DFSInputStream
Cheers
On Mon, Jan 27, 2014 at 11:07 AM, John Lilley <jo...@redpoint.net>> wrote:
Ummm... so if I've called FileSystem.open() with an hdfs:// path, and it returns an FSDataInputStream, how do I get from there to the DFSInputStream that you say has the interface I want?
Thanks
John
From: Ted Yu [mailto:yuzhihong@gmail.com<ma...@gmail.com>]
Sent: Sunday, January 26, 2014 6:16 PM
To: common-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HDFS read stats
Please take a look at DFSInputStream#ReadStatistics which contains four metrics including local bytes read.
You can obtain ReadStatistics through getReadStatistics()
Cheers
On Sun, Jan 26, 2014 at 4:00 PM, John Lilley <jo...@redpoint.net>> wrote:
Ted,
Thanks for link! I says 2.1.0 beta fix, and I can find FileSystem$Statistics class in 2.2.0 but it only seems to talk about read/write ops and bytes, not the local-vs-remote bytes. What am I missing?
John
From: Ted Yu [mailto:yuzhihong@gmail.com<ma...@gmail.com>]
Sent: Sunday, January 26, 2014 10:26 AM
To: common-user@hadoop.apache.org<ma...@hadoop.apache.org>
Subject: Re: HDFS read stats
Please take a look at https://issues.apache.org/jira/browse/HDFS-4698
Cheers
On Sun, Jan 26, 2014 at 9:11 AM, John Lilley <jo...@redpoint.net>> wrote:
Is there a way to monitor the proportion of HDFS read data that is satisfied by local nodes vs going across the network?
Thanks
John
Re: HDFS read stats
Posted by Ted Yu <yu...@gmail.com>.
You're welcome.
On Feb 9, 2014, at 6:27 AM, John Lilley <jo...@redpoint.net> wrote:
> Thanks! I would have never found that.
> john
>
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Monday, January 27, 2014 4:57 PM
> To: common-user@hadoop.apache.org
> Subject: Re: HDFS read stats
>
> FSDataInputStream has this javadoc:
> /** Utility that wraps a {@link FSInputStream} in a {@link DataInputStream}
>
> You can utilize this method:
> @InterfaceAudience.LimitedPrivate({"HDFS"})
> public InputStream getWrappedStream() {
> return in;
> And cast the return value to DFSInputStream
>
> Cheers
>
>
> On Mon, Jan 27, 2014 at 11:07 AM, John Lilley <jo...@redpoint.net> wrote:
> Ummm… so if I’ve called FileSystem.open() with an hdfs:// path, and it returns an FSDataInputStream, how do I get from there to the DFSInputStream that you say has the interface I want?
> Thanks
> John
>
>
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Sunday, January 26, 2014 6:16 PM
>
> To: common-user@hadoop.apache.org
> Subject: Re: HDFS read stats
>
> Please take a look at DFSInputStream#ReadStatistics which contains four metrics including local bytes read.
> You can obtain ReadStatistics through getReadStatistics()
>
> Cheers
>
>
> On Sun, Jan 26, 2014 at 4:00 PM, John Lilley <jo...@redpoint.net> wrote:
> Ted,
> Thanks for link! I says 2.1.0 beta fix, and I can find FileSystem$Statistics class in 2.2.0 but it only seems to talk about read/write ops and bytes, not the local-vs-remote bytes. What am I missing?
> John
>
>
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Sunday, January 26, 2014 10:26 AM
> To: common-user@hadoop.apache.org
> Subject: Re: HDFS read stats
>
> Please take a look at https://issues.apache.org/jira/browse/HDFS-4698
>
> Cheers
>
>
> On Sun, Jan 26, 2014 at 9:11 AM, John Lilley <jo...@redpoint.net> wrote:
> Is there a way to monitor the proportion of HDFS read data that is satisfied by local nodes vs going across the network?
> Thanks
> John
>
>
>
>
>
Re: HDFS read stats
Posted by Ted Yu <yu...@gmail.com>.
You're welcome.
On Feb 9, 2014, at 6:27 AM, John Lilley <jo...@redpoint.net> wrote:
> Thanks! I would have never found that.
> john
>
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Monday, January 27, 2014 4:57 PM
> To: common-user@hadoop.apache.org
> Subject: Re: HDFS read stats
>
> FSDataInputStream has this javadoc:
> /** Utility that wraps a {@link FSInputStream} in a {@link DataInputStream}
>
> You can utilize this method:
> @InterfaceAudience.LimitedPrivate({"HDFS"})
> public InputStream getWrappedStream() {
> return in;
> And cast the return value to DFSInputStream
>
> Cheers
>
>
> On Mon, Jan 27, 2014 at 11:07 AM, John Lilley <jo...@redpoint.net> wrote:
> Ummm… so if I’ve called FileSystem.open() with an hdfs:// path, and it returns an FSDataInputStream, how do I get from there to the DFSInputStream that you say has the interface I want?
> Thanks
> John
>
>
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Sunday, January 26, 2014 6:16 PM
>
> To: common-user@hadoop.apache.org
> Subject: Re: HDFS read stats
>
> Please take a look at DFSInputStream#ReadStatistics which contains four metrics including local bytes read.
> You can obtain ReadStatistics through getReadStatistics()
>
> Cheers
>
>
> On Sun, Jan 26, 2014 at 4:00 PM, John Lilley <jo...@redpoint.net> wrote:
> Ted,
> Thanks for link! I says 2.1.0 beta fix, and I can find FileSystem$Statistics class in 2.2.0 but it only seems to talk about read/write ops and bytes, not the local-vs-remote bytes. What am I missing?
> John
>
>
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Sunday, January 26, 2014 10:26 AM
> To: common-user@hadoop.apache.org
> Subject: Re: HDFS read stats
>
> Please take a look at https://issues.apache.org/jira/browse/HDFS-4698
>
> Cheers
>
>
> On Sun, Jan 26, 2014 at 9:11 AM, John Lilley <jo...@redpoint.net> wrote:
> Is there a way to monitor the proportion of HDFS read data that is satisfied by local nodes vs going across the network?
> Thanks
> John
>
>
>
>
>
Re: HDFS read stats
Posted by Ted Yu <yu...@gmail.com>.
You're welcome.
On Feb 9, 2014, at 6:27 AM, John Lilley <jo...@redpoint.net> wrote:
> Thanks! I would have never found that.
> john
>
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Monday, January 27, 2014 4:57 PM
> To: common-user@hadoop.apache.org
> Subject: Re: HDFS read stats
>
> FSDataInputStream has this javadoc:
> /** Utility that wraps a {@link FSInputStream} in a {@link DataInputStream}
>
> You can utilize this method:
> @InterfaceAudience.LimitedPrivate({"HDFS"})
> public InputStream getWrappedStream() {
> return in;
> And cast the return value to DFSInputStream
>
> Cheers
>
>
> On Mon, Jan 27, 2014 at 11:07 AM, John Lilley <jo...@redpoint.net> wrote:
> Ummm… so if I’ve called FileSystem.open() with an hdfs:// path, and it returns an FSDataInputStream, how do I get from there to the DFSInputStream that you say has the interface I want?
> Thanks
> John
>
>
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Sunday, January 26, 2014 6:16 PM
>
> To: common-user@hadoop.apache.org
> Subject: Re: HDFS read stats
>
> Please take a look at DFSInputStream#ReadStatistics which contains four metrics including local bytes read.
> You can obtain ReadStatistics through getReadStatistics()
>
> Cheers
>
>
> On Sun, Jan 26, 2014 at 4:00 PM, John Lilley <jo...@redpoint.net> wrote:
> Ted,
> Thanks for link! I says 2.1.0 beta fix, and I can find FileSystem$Statistics class in 2.2.0 but it only seems to talk about read/write ops and bytes, not the local-vs-remote bytes. What am I missing?
> John
>
>
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Sunday, January 26, 2014 10:26 AM
> To: common-user@hadoop.apache.org
> Subject: Re: HDFS read stats
>
> Please take a look at https://issues.apache.org/jira/browse/HDFS-4698
>
> Cheers
>
>
> On Sun, Jan 26, 2014 at 9:11 AM, John Lilley <jo...@redpoint.net> wrote:
> Is there a way to monitor the proportion of HDFS read data that is satisfied by local nodes vs going across the network?
> Thanks
> John
>
>
>
>
>
Re: HDFS read stats
Posted by Ted Yu <yu...@gmail.com>.
You're welcome.
On Feb 9, 2014, at 6:27 AM, John Lilley <jo...@redpoint.net> wrote:
> Thanks! I would have never found that.
> john
>
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Monday, January 27, 2014 4:57 PM
> To: common-user@hadoop.apache.org
> Subject: Re: HDFS read stats
>
> FSDataInputStream has this javadoc:
> /** Utility that wraps a {@link FSInputStream} in a {@link DataInputStream}
>
> You can utilize this method:
> @InterfaceAudience.LimitedPrivate({"HDFS"})
> public InputStream getWrappedStream() {
> return in;
> And cast the return value to DFSInputStream
>
> Cheers
>
>
> On Mon, Jan 27, 2014 at 11:07 AM, John Lilley <jo...@redpoint.net> wrote:
> Ummm… so if I’ve called FileSystem.open() with an hdfs:// path, and it returns an FSDataInputStream, how do I get from there to the DFSInputStream that you say has the interface I want?
> Thanks
> John
>
>
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Sunday, January 26, 2014 6:16 PM
>
> To: common-user@hadoop.apache.org
> Subject: Re: HDFS read stats
>
> Please take a look at DFSInputStream#ReadStatistics which contains four metrics including local bytes read.
> You can obtain ReadStatistics through getReadStatistics()
>
> Cheers
>
>
> On Sun, Jan 26, 2014 at 4:00 PM, John Lilley <jo...@redpoint.net> wrote:
> Ted,
> Thanks for link! I says 2.1.0 beta fix, and I can find FileSystem$Statistics class in 2.2.0 but it only seems to talk about read/write ops and bytes, not the local-vs-remote bytes. What am I missing?
> John
>
>
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Sunday, January 26, 2014 10:26 AM
> To: common-user@hadoop.apache.org
> Subject: Re: HDFS read stats
>
> Please take a look at https://issues.apache.org/jira/browse/HDFS-4698
>
> Cheers
>
>
> On Sun, Jan 26, 2014 at 9:11 AM, John Lilley <jo...@redpoint.net> wrote:
> Is there a way to monitor the proportion of HDFS read data that is satisfied by local nodes vs going across the network?
> Thanks
> John
>
>
>
>
>