[jira] [Commented] (FLINK-9029) Getting write permission from HDFS after updating flink-1.40 to flink-1.4.2

["Stephan Ewen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-9029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16406901#comment-16406901 ] 

Stephan Ewen commented on FLINK-9029:
-------------------------------------

Hmm, interesting. And security is OFF as you mentioned?

Can you try if it works if you completely remove the Hadoop uber jar from the lib folder, and instead export the hadoop classpath as in https://ci.apache.org/projects/flink/flink-docs-master/ops/deployment/hadoop.html

There was one fix to the Kerberos functionality between 1.4.0 and 1.4.2, which is https://github.com/apache/flink/commit/20146652baff31c3824f7ae31506b8c481cdf56c

The puzzling thing is that this particular commit does not modify the Hadoop Uber Jar.

[~tzulitai] Do you have any thoughts why that could happen?

> Getting write permission from HDFS after updating flink-1.40 to flink-1.4.2
> ---------------------------------------------------------------------------
>
>                 Key: FLINK-9029
>                 URL: https://issues.apache.org/jira/browse/FLINK-9029
>             Project: Flink
>          Issue Type: Bug
>    Affects Versions: 1.4.1, 1.4.2
>         Environment: * Flink-1.4.2 (Flink-1.4.1)
>  * Hadoop 2.6.0-cdh5.13.0 with 4 nodes in service and {{Security is off.}}
>  * Ubuntu 16.04.3 LTS
>  * Java 8
>            Reporter: Mohammad Abareghi
>            Priority: Major
>
> *Environment*
>  * Flink-1.4.2
>  * Hadoop 2.6.0-cdh5.13.0 with 4 nodes in service and {{Security is off.}}
>  * Ubuntu 16.04.3 LTS
>  * Java 8
> *Description*
> I have a Java job in flink-1.4.0 which writes to HDFS in a specific path. After updating to flink-1.4.2 I'm getting the following error from Hadoop complaining that the user doesn't have write permission to the given path:
>  
> *Description*
> I have a Java job in flink-1.4.0 which writes to HDFS in a specific path. After updating to flink-1.4.2 I'm getting the following error from Hadoop complaining that the user doesn't have write permission to the given path:
> {code:java}
> WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:xng (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Permission denied: user=user1, access=WRITE, inode="/user":hdfs:hadoop:drwxr-xr-x
> {code}
> *NOTE*:
>  * If I run the same job on flink-1.4.0, Error disappears regardless of what version of flink (1.4.0 or 1.4.2) dependencies I have for job
>  * Also if I run the job main method from my IDE and pass the same parameters, I don't get above error.
> *NOTE*:
> It seems the problem somehow is in {{flink-1.4.2/lib/flink-shaded-hadoop2-uber-1.4.2.jar}}. If I replace that with {{flink-1.4.0/lib/flink-shaded-hadoop2-uber-1.4.0.jar}}, restart the cluster and run my job (flink topology) then the error doesn't appear.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)