[jira] [Commented] (HDFS-7036) HDFS-6776 fix requires to upgrade insecure cluster, which means quite some user pain

["Yongjun Zhang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154356#comment-14154356 ] 

Yongjun Zhang commented on HDFS-7036:
-------------------------------------

Hi [~wheat9],

Although I still think the right solution is to fix webhdfs because the root cause is that webhdfs is broken,  since we really had an disagreement here, I looked into how to fix in distcp you proposed.  Below is what I found. I would appreciate if you can tell if my analysis below makes sense to you.

As a refresher, below is the interesting part of the stack trace (reported in HDFS-6776) relevant to the discussion here.

{code}
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:440)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1018)
	at org.apache.hadoop.fs.Globber.listStatus(Globber.java:69)
	at org.apache.hadoop.fs.Globber.glob(Globber.java:217)
	at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623)
	at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
	at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
	at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
	at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154)
{code}

To fix in fistcp, we should look into the boundary portion between distcp and file system in the above stack trace:

{code
	at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
	at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81)
	at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342)
{code}

The boundary code in {{GlobbedCopyListing.doBuildListing}} is below 
{code}
  for (Path p : options.getSourcePaths()) {
      FileSystem fs = p.getFileSystem(getConf());
      FileStatus[] inputs = fs.globStatus(p); <================this is where the exception is thrown

      if(inputs != null && inputs.length > 0) {
        for (FileStatus onePath: inputs) {
          globbedPaths.add(onePath.getPath());
        }
      } else {
        throw new InvalidInputException(p + " doesn't exist");        
      }
    }
{code}

I assume your proposed solution is to catch the exception here and do a hacked file listing using the {{p}} in the above code. However, this hack won't work for *wildcard* case:, e,g,:

{code}
 hadoop  distcp webhdfs://<insecureCluster>/path/* webhdfs://<secureCluster>
{code}

We won't be able to find what the wildcard will expand to, which is supposed to be got by {{FileStatus[] inputs = fs.globStatus(p);}}. Unless we hack into the file system.

Did I misunderstand your approach? I'd appreciate if you could comment. Thanks in advance.

BTW, You asked what applications other than distcp could apply in your last comment, I answered you with the "hadoop fs -lsr" example. Would you please confirm whether it addressed your  question? Do you think we should fix it too? Thanks.



> HDFS-6776 fix requires to upgrade insecure cluster, which means quite some user pain
> ------------------------------------------------------------------------------------
>
>                 Key: HDFS-7036
>                 URL: https://issues.apache.org/jira/browse/HDFS-7036
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: webhdfs
>    Affects Versions: 2.5.1
>            Reporter: Yongjun Zhang
>            Assignee: Yongjun Zhang
>         Attachments: HDFS-7036.001.patch
>
>
> Issuing command
> {code}
>  hadoop fs -lsr webhdfs://<insecureCluster>
> {code}
> at a secure cluster side fails with message "Failed to get the token ...", similar symptom as reported in HDFS-6776.
> If the fix of HDFS-6776 is applied to only the secure cluster, doing 
> {code}
> distcp webhdfs://<insecureCluster> <secureCluster>
> {code}
> would fail same way.
> Basically running any application in secure cluster to access insecure cluster via webhdfs would fail the same way, if the HDFS-6776 fix is not applied to the insecure cluster.
> This could be quite some user pain. Filing this jira for a solution to make user's life easier.
> One proposed solution was to add a msg-parsing mechanism in webhdfs, which is a bit hacky. The other proposed solution is to do the same kind of hack at application side, which means the same hack need to be applied in each application.
> Thanks [~daryn], [~wheat9], [~jingzhao], [~tucu00] and [~atm] for the discussion in HDFS-6776.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)