You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2020/02/04 15:41:00 UTC
[jira] [Commented] (HADOOP-16837) Spark-SQL test running on
Windows: hadoop chgrp warnings
[ https://issues.apache.org/jira/browse/HADOOP-16837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029941#comment-17029941 ]
Steve Loughran commented on HADOOP-16837:
-----------------------------------------
Well, looks like the code you are pointing out has not changed for a long time, but as this goes near winutil it'll be something very few people will be able and willing to test.
You can submit a patch against the trunk to turn that check off if Shell.WINDOWS is true, and then AD users will get whatever outcome winutils has...we could backport to 3.2.x
The other strategy is HADOOP-13223, "winutils.exe is a bug nexus and should be killed with an axe.". Spark users are about the only people going near this, and there is no justifiable reason for requiring the library in this world. Different if you're running yarn and want to lock down staging directories -but nobody does this.
Someone did a PR for that last week but it was against the wrong branch and unreviewable https://github.com/apache/hadoop/pull/1812
If you want to get involved in that and get rid of winutils entirely, it would be the best outcome for everyone.
BTW: no, not going near branch-2 with that change, especially as only one remaining release is going to happen there. Time to upgrade
> Spark-SQL test running on Windows: hadoop chgrp warnings
> ---------------------------------------------------------
>
> Key: HADOOP-16837
> URL: https://issues.apache.org/jira/browse/HADOOP-16837
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs
> Affects Versions: 2.6.5
> Environment: Windows 10
> Winutils 2.7.1: [https://github.com/steveloughran/winutils/tree/master/hadoop-2.7.1]
> Oracle JavaSE 8
> SparkSQL 2.4.4 / Hadoop 2.6.5
> Using: -Dhive.exec.scratchdir=C:\Users\OSUser\hadoop\tmp\hive
> Set: winutils chmod -R 777 \Users\OSUser\hadoop\tmp\hive
> Reporter: Guram Savinov
> Priority: Major
> Labels: filesystem, unit-test, win10
> Attachments: HadoopGroupTest.java
>
>
> Running SparkSQL local embedded unit tests on Win10, using winutils.
> Got warnings about 'hadoop chgrp'.
> See environment info.
> {code:bash}
> -chgrp: 'TEST\Domain users' does not match expected pattern for group
> Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
> -chgrp: 'TEST\Domain users' does not match expected pattern for group
> Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
> -chgrp: 'TEST\Domain users' does not match expected pattern for group
> Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
> {code}
> Related info on SO: [https://stackoverflow.com/questions/48605907/error-in-pyspark-when-insert-data-in-hive]
> hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShellPermissions.java:210
> The problem is: backslash character isn't included to allowedChars, see attached HadoopGroupTest.java
> Original issue in Spark: https://issues.apache.org/jira/browse/SPARK-30701
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org