You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2006/07/06 12:38:30 UTC

[jira] Created: (HADOOP-347) Implement HDFS content browsing interface

Implement HDFS content browsing interface
-----------------------------------------

         Key: HADOOP-347
         URL: http://issues.apache.org/jira/browse/HADOOP-347
     Project: Hadoop
        Type: New Feature

  Components: dfs  
    Versions: 0.1.0, 0.2.0, 0.1.1, 0.3.0, 0.4.0, 0.2.1, 0.3.1, 0.3.2    
    Reporter: Devaraj Das
 Assigned to: Devaraj Das 
     Fix For: 0.4.1


Implement HDFS content browsing interface over HTTP. Clients would connect to the NameNode and this would send a redirect to a random DataNode. The DataNode, via dfs client, would proxy to namenode for metadata browsing and to other datanodes for content. One can also view the local blocks on any DataNode. Head, Tail will be provided as shorthands for viewing the first block and the last block of a file. 
For full file viewing, the data displayed per HTTP request will be a block with a PREV/NEXT link. The block size for viewing can be a configurable parameter (the user sets it via the web browser) to the HTTP server (e.g., 256 KB can be the default block size for viewing files).

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

[jira] Commented: (HADOOP-347) Implement HDFS content browsing interface

Posted by "eric baldeschwieler (JIRA)" <ji...@apache.org>.

    [ http://issues.apache.org/jira/browse/HADOOP-347?page=comments#action_12419573 ] 

eric baldeschwieler commented on HADOOP-347:
--------------------------------------------

Sounds good.

Could you provide a little more detail on the UI on the name node?  I image a link on the front page to a page that redirects to a random data node.  That the plan?

Also you probably want to redirect the reader to a node containing the first block of the file when you get the files contents.

The block viewing seems like a good option, we should also provide the ability to simply fetch the full file.  Maybe two links on the directory browsing API could provide this option?  Machine clients would be frustrated by the block structure.

Let's be sure to support tail (-f in the future).  The is very often the desired operation.  A third link on the directory browsing page would do nicely.

I like the blocked view idea.  Perhaps all of this can simply be GET arguments appended to the end of the URL?  IE "HDFS://x.com/myPath?sz=128k&off=512k"  Negative offsets could be used for tail.

So you imagine a badge at the top of the block viewed page with <prev next> links?  Might just add some block size options there too and tail.  Keep the whole API stateless (all info in the URL).


> Implement HDFS content browsing interface
> -----------------------------------------
>
>          Key: HADOOP-347
>          URL: http://issues.apache.org/jira/browse/HADOOP-347
>      Project: Hadoop
>         Type: New Feature

>   Components: dfs
>     Versions: 0.1.0, 0.2.0, 0.1.1, 0.3.0, 0.4.0, 0.2.1, 0.3.1, 0.3.2
>     Reporter: Devaraj Das
>     Assignee: Devaraj Das
>      Fix For: 0.4.1

>
> Implement HDFS content browsing interface over HTTP. Clients would connect to the NameNode and this would send a redirect to a random DataNode. The DataNode, via dfs client, would proxy to namenode for metadata browsing and to other datanodes for content. One can also view the local blocks on any DataNode. Head, Tail will be provided as shorthands for viewing the first block and the last block of a file. 
> For full file viewing, the data displayed per HTTP request will be a block with a PREV/NEXT link. The block size for viewing can be a configurable parameter (the user sets it via the web browser) to the HTTP server (e.g., 256 KB can be the default block size for viewing files).

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

[jira] Commented: (HADOOP-347) Implement HDFS content browsing interface

Posted by "Yoram Arnon (JIRA)" <ji...@apache.org>.

    [ http://issues.apache.org/jira/browse/HADOOP-347?page=comments#action_12420885 ] 

Yoram Arnon commented on HADOOP-347:
------------------------------------

a typical scenario is for a user to click on a generic link from some front page to browse the dfs, without specifying a file. The namenode will redirect to a datanode, the user browses the directory hierarchy, then views files.
it's thus impossible to direct the client to the datanode that contains the first block of "the file" - it's not known when the client first connects.

one way to go about this, is to have the datanode do redirection too when actual contents are requested. In that case we don't need to stop with the first block. Since GET requests are stateless, the client can be redirected every time he needs a new dfs block.
I suggest pushing this all out for a later version for simplicity. Have the client connect to a random node and keep him there, just load balancing clients.

> Implement HDFS content browsing interface
> -----------------------------------------
>
>          Key: HADOOP-347
>          URL: http://issues.apache.org/jira/browse/HADOOP-347
>      Project: Hadoop
>         Type: New Feature

>   Components: dfs
>     Versions: 0.1.0, 0.2.0, 0.1.1, 0.3.0, 0.4.0, 0.2.1, 0.3.1, 0.3.2
>     Reporter: Devaraj Das
>     Assignee: Devaraj Das
>      Fix For: 0.5.0

>
> Implement HDFS content browsing interface over HTTP. Clients would connect to the NameNode and this would send a redirect to a random DataNode. The DataNode, via dfs client, would proxy to namenode for metadata browsing and to other datanodes for content. One can also view the local blocks on any DataNode. Head, Tail will be provided as shorthands for viewing the first block and the last block of a file. 
> For full file viewing, the data displayed per HTTP request will be a block with a PREV/NEXT link. The block size for viewing can be a configurable parameter (the user sets it via the web browser) to the HTTP server (e.g., 256 KB can be the default block size for viewing files).

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira