[GitHub] [hadoop] xiaoyuyao commented on issue #627: HDDS-1299. Support TokenIssuer interface for running jobs with OzoneFileSystem.

[GitBox <gi...@apache.org>]

xiaoyuyao commented on issue #627: HDDS-1299. Support TokenIssuer interface for running jobs with OzoneFileSystem.
URL: https://github.com/apache/hadoop/pull/627#issuecomment-476270295
 
 
   Thanks Steve for the heads up. Ozone DT via HCFS requires Kerberos setup for various component to test e2e. There is no easy way to test inside unit test as OM/SCM/DN all login with different principles.
   We have follow up JIRA to add system test.
   
   Before committing the changes, I spend a few days to get the manual steps with various testing documented here:
   %your_hadoop_rep%/hadoop-ozone/dist/src/main/compose/ozonesecure-mr/README.md
   
   This will be the basis of system tests being added, where we may use different Hadoop-versions like 3.2 and 2.7 to test the combabilities.
   The test will be very similar to the one below just slightly different security setup.
   %your_hadoop_rep%/hadoop/hadoop-ozone/dist/src/main/smoketest/security/ozone-secure.robot
   
   
   Bq. Without tests, when people like myself make an incompatible change which breaks Ozone's DT support, we're just going to shrug and say "well, you should have written the tests"
   Agree, test will be added. I have a WIP patch for DT system tests but there is a refactor of Ozone smoke test in HDDS-1255<https://github.com/apache/hadoop/pull/632>.  I will post my test after that.
   
   HDDS-1299 based on HADOOP-14445 are all Hadoop-3 only changes, it brings Ozone DT support. Before that, Ozone does not support DT.  There is a pending HADOOP-14445 patch for Hadoop-2.
   We could build support for Hadoop-2 after that in a separate ozone branch.
   
   Thanks,
   Xiaoyu
   
   From: Steve Loughran <no...@github.com>
   Reply-To: apache/hadoop <re...@reply.github.com>
   Date: Monday, March 25, 2019 at 8:58 AM
   To: apache/hadoop <ha...@noreply.github.com>
   Cc: Xiaoyu Yao <xy...@apache.org>, State change <st...@noreply.github.com>
   Subject: Re: [apache/hadoop] HDDS-1299. Support TokenIssuer interface for running jobs with OzoneFileSystem. (#627)
   
   
   Given this has broken things, can I ask a process question here, specifically regarding Yetus's v-1
   
   The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
   
   There are lots of ways to test for DTs being ussued: The hdfs fs command, the hadoop dtuils, the actualyh M
   
     *   Marshalling/unmarshalling of tokens: org.apache.hadoop.fs.s3a.auth.delegation.TestS3ADelegationTokenSupport
     *   org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationTokens
     *   org.apache.hadoop.fs.s3a.auth.delegation.ITestRoleDelegationInFileystem
     *   org.apache.hadoop.fs.azurebfs.extensions.ITestAbfsDelegationTokens
   
   You can even verify that MR cluster launch collects the tokens: org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob
   
   Please. write tests. Apart from the MR job one its straightforward, and I've just given you all the clases to start with.
   
   Without tests, when people like myself make an incompatible change which breaks Ozone's DT support, we're just going to shrug and say "well, you should have written the tests"
   
   —
   You are receiving this because you modified the open/close state.
   Reply to this email directly, view it on GitHub<https://github.com/apache/hadoop/pull/627#issuecomment-476256841>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AGto0K4e1tFGFPgU-8npgy5vjk8phZnUks5vaO-3gaJpZM4b9Vx2>.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org