You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "binlijin (JIRA)" <ji...@apache.org> on 2012/11/13 03:53:12 UTC

[jira] [Created] (HDFS-4184) Add new interface for Client to provide more information

binlijin created HDFS-4184:
------------------------------

             Summary: Add new interface for Client to provide more information
                 Key: HDFS-4184
                 URL: https://issues.apache.org/jira/browse/HDFS-4184
             Project: Hadoop HDFS
          Issue Type: New Feature
            Reporter: binlijin


When hbase read or write hlog we can use dfs.datanode.drop.cache.behind.reads、dfs.datanode.drop.cache.behind.writes, when hbase read hfile during compaction we can use readahead and so on... 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-4184) Add new interface for Client to provide more information

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack resolved HDFS-4184.
-------------------------

    Resolution: Invalid

Resolving invalid as not enough detail.

The JIRA subject and description do not seem to match.  As per Ted in previous issue, please add more detail when you create issue so we can know better to what you refer.  Meantime I'm closing this.  Open a new one when better specification (this seems to require a particular version of hadoop, etc.).

Thanks Binlijin.
                
> Add new interface for Client to provide more information
> --------------------------------------------------------
>
>                 Key: HDFS-4184
>                 URL: https://issues.apache.org/jira/browse/HDFS-4184
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: binlijin
>
> When hbase read or write hlog we can use dfs.datanode.drop.cache.behind.reads、dfs.datanode.drop.cache.behind.writes, when hbase read hfile during compaction we can use readahead and so on... 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-4184) Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate

Posted by "binlijin (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

binlijin updated HDFS-4184:
---------------------------

    Description: 
HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.
{code}
When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads.
and so on... 
{code}

  was:
HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.
When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads.
and so on... 

        Summary: Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate  (was: Add new interface for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate)
    
> Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-4184
>                 URL: https://issues.apache.org/jira/browse/HDFS-4184
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: binlijin
>
> HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.
> {code}
> When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
> When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
> When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads.
> and so on... 
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-4184) Add new interface for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate

Posted by "binlijin (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

binlijin updated HDFS-4184:
---------------------------

    Description: 
HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.
When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads.
and so on... 

  was:When hbase read or write hlog we can use dfs.datanode.drop.cache.behind.reads、dfs.datanode.drop.cache.behind.writes, when hbase read hfile during compaction we can use readahead and so on... 

        Summary: Add new interface for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate  (was: Add new interface for Client to provide more information)
    
> Add new interface for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-4184
>                 URL: https://issues.apache.org/jira/browse/HDFS-4184
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: binlijin
>
> HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.
> When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
> When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
> When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads.
> and so on... 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4184) Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate

Posted by "binlijin (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495914#comment-13495914 ] 

binlijin commented on HDFS-4184:
--------------------------------

Thanks stack，i can't reopened it just now
                
> Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-4184
>                 URL: https://issues.apache.org/jira/browse/HDFS-4184
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: binlijin
>
> HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.
> {code}
> When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
> When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
> When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads.
> and so on... 
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HDFS-4184) Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack reopened HDFS-4184:
-------------------------


Here, I reopened it for you (in case you can't)
                
> Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-4184
>                 URL: https://issues.apache.org/jira/browse/HDFS-4184
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: binlijin
>
> HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.
> {code}
> When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
> When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
> When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads.
> and so on... 
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-4184) Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate

Posted by "binlijin (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

binlijin updated HDFS-4184:
---------------------------

    Description: 
HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.
{code}
When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads， and also set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
and so on... 
{code}
Current we can only set these feature global in datanode，we should set these feature per session.

  was:
HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.
{code}
When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads， and also set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
and so on... 
{code}

    
> Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-4184
>                 URL: https://issues.apache.org/jira/browse/HDFS-4184
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: binlijin
>
> HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.
> {code}
> When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
> When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
> When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads， and also set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
> and so on... 
> {code}
> Current we can only set these feature global in datanode，we should set these feature per session.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (HDFS-4184) Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HDFS-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13495911#comment-13495911 ] 

stack commented on HDFS-4184:
-----------------------------

[~aoxiang] Want to reopen this issue now you've filled in detail?
                
> Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-4184
>                 URL: https://issues.apache.org/jira/browse/HDFS-4184
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: binlijin
>
> HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.
> {code}
> When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
> When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
> When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads.
> and so on... 
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (HDFS-4184) Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate

Posted by "binlijin (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HDFS-4184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

binlijin updated HDFS-4184:
---------------------------

    Description: 
HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.
{code}
When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads， and also set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
and so on... 
{code}

  was:
HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.
{code}
When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads.
and so on... 
{code}

    
> Add the ability for Client to provide more hint information for DataNode to manage the OS buffer cache more accurate
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-4184
>                 URL: https://issues.apache.org/jira/browse/HDFS-4184
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: binlijin
>
> HDFS now has the ability to use posix_fadvise and sync_data_range syscalls to manage the OS buffer cache.
> {code}
> When hbase read hlog the data we can set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
> When hbase write hlog we can set dfs.datanode.drop.cache.behind.writes to true to drop data out of the buffer cache after writing
> When hbase read hfile during compaction we can set dfs.datanode.readahead.bytes to a non-zero value to trigger readahead for sequential reads， and also set dfs.datanode.drop.cache.behind.reads to true to drop data out of the buffer cache when performing sequential reads.
> and so on... 
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira