You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Kirti Dhar Upadhyay K via user <us...@flink.apache.org> on 2024/03/06 10:09:54 UTC
SecurityManager in Flink
Hi Team,
I am using Flink File Source with Local File System.
I am facing an issue, if source directory does not has read permission, it is returning the list of files as null instead of throwing permission exception (refer the highlighted line below), resulting in NPE.
final FileStatus[] containedFiles = fs.listStatus(fileStatus.getPath());
for (FileStatus containedStatus : containedFiles) {
addSplitsForPath(containedStatus, fs, target);
}
Debugging the issue found that, SecurityManager is coming as null while listing the files, hence skipping the permissions on directory.
What is the way to set SecurityManager in Flink?
Regards,
Kirti Dhar
RE: SecurityManager in Flink
Posted by Kirti Dhar Upadhyay K via user <us...@flink.apache.org>.
Hi Gabor,
The issue is that, read permission is not getting checked when Flink FileSource is listing the files under given source directory.
This is happening as Security Manager is coming as null.
public String[] list() {
SecurityManager security = System.getSecurityManager(); -> Here Security Manager is coming as Null.
if (security != null) {
security.checkRead(path);
}
if (isInvalid()) {
return null;
}
return fs.list(this);
}
While debugging it, found a method in Flink Security manager like below, hence I suspected towards it and queried to know the role of Flink Security manager.
public static void setFromConfiguration(Configuration configuration) {
final FlinkSecurityManager flinkSecurityManager =
FlinkSecurityManager.fromConfiguration(configuration);
if (flinkSecurityManager != null) {
try {
System.setSecurityManager(flinkSecurityManager);
} catch (Exception e) {
…
…
Regards,
Kirti Dhar
From: Gabor Somogyi <ga...@gmail.com>
Sent: Wednesday, March 6, 2024 7:17 PM
To: Kirti Dhar Upadhyay K <ki...@ericsson.com>
Cc: User@flink.apache.org
Subject: Re: SecurityManager in Flink
Hi Kirti,
Not sure what is the exact issue here but I'm not convinced that having FlinkSecurityManager is going to solve it.
Here is the condition however:
* cluster.intercept-user-system-exit != DISABLED (this must be changed)
* cluster.processes.halt-on-fatal-error == false (this is good by default)
Here is a gist what Flink's SecurityManager does:
/**
* {@code FlinkSecurityManager} to control certain behaviors that can be captured by Java system
* security manager. It can be used to control unexpected user behaviors that potentially impact
* cluster availability, for example, it can warn or prevent user code from terminating JVM by
* System.exit or halt by logging or throwing an exception. This does not necessarily prevent
* malicious users who try to tweak security manager on their own, but more for being dependable
* against user mistakes by gracefully handling them informing users rather than causing silent
* unavailability.
*/
G
On Wed, Mar 6, 2024 at 11:10 AM Kirti Dhar Upadhyay K via user <us...@flink.apache.org>> wrote:
Hi Team,
I am using Flink File Source with Local File System.
I am facing an issue, if source directory does not has read permission, it is returning the list of files as null instead of throwing permission exception (refer the highlighted line below), resulting in NPE.
final FileStatus[] containedFiles = fs.listStatus(fileStatus.getPath());
for (FileStatus containedStatus : containedFiles) {
addSplitsForPath(containedStatus, fs, target);
}
Debugging the issue found that, SecurityManager is coming as null while listing the files, hence skipping the permissions on directory.
What is the way to set SecurityManager in Flink?
Regards,
Kirti Dhar
Re: SecurityManager in Flink
Posted by Gabor Somogyi <ga...@gmail.com>.
Hi Kirti,
Not sure what is the exact issue here but I'm not convinced that
having FlinkSecurityManager is going to solve it.
Here is the condition however:
* cluster.intercept-user-system-exit != DISABLED (this must be changed)
* cluster.processes.halt-on-fatal-error == false (this is good by default)
Here is a gist what Flink's SecurityManager does:
/**
* {@code FlinkSecurityManager} to control certain behaviors that can be
captured by Java system
* security manager. It can be used to control unexpected user behaviors
that potentially impact
* cluster availability, for example, it can warn or prevent user code from
terminating JVM by
* System.exit or halt by logging or throwing an exception. This does not
necessarily prevent
* malicious users who try to tweak security manager on their own, but more
for being dependable
* against user mistakes by gracefully handling them informing users rather
than causing silent
* unavailability.
*/
G
On Wed, Mar 6, 2024 at 11:10 AM Kirti Dhar Upadhyay K via user <
user@flink.apache.org> wrote:
> Hi Team,
>
>
>
> I am using Flink File Source with Local File System.
>
> I am facing an issue, if source directory does not has read permission, it
> is returning the list of files as null instead of throwing permission
> exception (refer the highlighted line below), resulting in NPE.
>
>
>
> final FileStatus[] containedFiles = fs.listStatus(fileStatus.getPath());
> for (FileStatus containedStatus : containedFiles) {
> addSplitsForPath(containedStatus, fs, target);
> }
>
> Debugging the issue found that, SecurityManager is coming as null while
> listing the files, hence skipping the permissions on directory.
>
> What is the way to set SecurityManager in Flink?
>
>
>
> Regards,
>
> Kirti Dhar
>
>
>
RE: SecurityManager in Flink
Posted by Kirti Dhar Upadhyay K via user <us...@flink.apache.org>.
Hi Yanfei,
I am facing this issue on jdk1.8/11.
Thanks for giving pointer, I will try to set Security manager and check the behaviour.
Regards,
Kirti Dhar
-----Original Message-----
From: Yanfei Lei <fr...@gmail.com>
Sent: Wednesday, March 6, 2024 4:37 PM
To: Kirti Dhar Upadhyay K <ki...@ericsson.com>
Cc: User@flink.apache.org
Subject: Re: SecurityManager in Flink
Hi Kirti Dhar,
What is your java version? I guess this problem may be related to FLINK-33309[1]. Maybe you can try adding "-Djava.security.manager" to the java options.
[1] https://issues.apache.org/jira/browse/FLINK-33309
Kirti Dhar Upadhyay K via user <us...@flink.apache.org> 于2024年3月6日周三 18:10写道：
>
> Hi Team,
>
>
>
> I am using Flink File Source with Local File System.
>
> I am facing an issue, if source directory does not has read permission, it is returning the list of files as null instead of throwing permission exception (refer the highlighted line below), resulting in NPE.
>
>
>
> final FileStatus[] containedFiles =
> fs.listStatus(fileStatus.getPath());
> for (FileStatus containedStatus : containedFiles) {
> addSplitsForPath(containedStatus, fs, target); }
>
> Debugging the issue found that, SecurityManager is coming as null while listing the files, hence skipping the permissions on directory.
>
> What is the way to set SecurityManager in Flink?
>
>
>
> Regards,
>
> Kirti Dhar
>
>
--
Best,
Yanfei
Re: SecurityManager in Flink
Posted by Yanfei Lei <fr...@gmail.com>.
Hi Kirti Dhar,
What is your java version? I guess this problem may be related to
FLINK-33309[1]. Maybe you can try adding "-Djava.security.manager" to
the java options.
[1] https://issues.apache.org/jira/browse/FLINK-33309
Kirti Dhar Upadhyay K via user <us...@flink.apache.org> 于2024年3月6日周三 18:10写道：
>
> Hi Team,
>
>
>
> I am using Flink File Source with Local File System.
>
> I am facing an issue, if source directory does not has read permission, it is returning the list of files as null instead of throwing permission exception (refer the highlighted line below), resulting in NPE.
>
>
>
> final FileStatus[] containedFiles = fs.listStatus(fileStatus.getPath());
> for (FileStatus containedStatus : containedFiles) {
> addSplitsForPath(containedStatus, fs, target);
> }
>
> Debugging the issue found that, SecurityManager is coming as null while listing the files, hence skipping the permissions on directory.
>
> What is the way to set SecurityManager in Flink?
>
>
>
> Regards,
>
> Kirti Dhar
>
>
--
Best,
Yanfei
RE: SecurityManager in Flink
Posted by Kirti Dhar Upadhyay K via user <us...@flink.apache.org>.
Hi Hang,
You got it right. The problem is exactly at the same line where you pointed [1].
I have used below solution as of now.
```
If(!Files.isReadable(Paths.get(fileStatus.getPath().getPath()))) {
throw new FlinkRuntimeException("Cannot list files under " + fileStatus.getPath());
}
final FileStatus[] containedFiles = fs.listStatus(fileStatus.getPath());
for (FileStatus containedStatus : containedFiles) {
addSplitsForPath(containedStatus, fs, target);
}
```
Although, if you go inside localf.list(), it checks automatically for the read permission using Security Manager. This check is getting skipped as Security Manager is coming as null.
Hence I suspected towards Security Manager.
[1] https://github.com/apache/flink/blob/9b1375520b6b351df7551d85fcecd920e553cc3a/flink-core/src/main/java/org/apache/flink/core/fs/local/LocalFileSystem.java#L161C32-L161C38<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-722ab8a60d77d5b6&q=1&e=634cbd0d-6962-4ee2-bb8d-7f771a0d428c&u=https%3A%2F%2Fgithub.com%2Fapache%2Fflink%2Fblob%2F9b1375520b6b351df7551d85fcecd920e553cc3a%2Fflink-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcore%2Ffs%2Flocal%2FLocalFileSystem.java%23L161C32-L161C38>
Regards,
Kirti Dhar
From: Hang Ruan <ru...@gmail.com>
Sent: Wednesday, March 6, 2024 6:46 PM
To: Kirti Dhar Upadhyay K <ki...@ericsson.com>
Cc: User@flink.apache.org
Subject: Re: SecurityManager in Flink
Hi, Kirti.
Could you please provide the stack trace of this NPE? I check the code and I think maybe the problem lies in LocalFileSystem#listStatus.
The code in line 161[1] may return null, which will let LocalFileSystem#listStatus return null. Then the `containedFiles` is null and the NPE occurs.
I think we should add code to handle this situation as follows.
```
final FileStatus[] containedFiles = fs.listStatus(fileStatus.getPath());
if (containedFiles == null) {
throw new FlinkRuntimeException("Cannot list files under " + fileStatus.getPath());
}
for (FileStatus containedStatus : containedFiles) {
addSplitsForPath(containedStatus, fs, target);
}
```
Best,
Hang
[1] https://github.com/apache/flink/blob/9b1375520b6b351df7551d85fcecd920e553cc3a/flink-core/src/main/java/org/apache/flink/core/fs/local/LocalFileSystem.java#L161C32-L161C38<https://protect2.fireeye.com/v1/url?k=31323334-501d5122-313273af-454445555731-722ab8a60d77d5b6&q=1&e=634cbd0d-6962-4ee2-bb8d-7f771a0d428c&u=https%3A%2F%2Fgithub.com%2Fapache%2Fflink%2Fblob%2F9b1375520b6b351df7551d85fcecd920e553cc3a%2Fflink-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fflink%2Fcore%2Ffs%2Flocal%2FLocalFileSystem.java%23L161C32-L161C38>
Kirti Dhar Upadhyay K via user <us...@flink.apache.org>> 于2024年3月6日周三 18:10写道：
Hi Team,
I am using Flink File Source with Local File System.
I am facing an issue, if source directory does not has read permission, it is returning the list of files as null instead of throwing permission exception (refer the highlighted line below), resulting in NPE.
final FileStatus[] containedFiles = fs.listStatus(fileStatus.getPath());
for (FileStatus containedStatus : containedFiles) {
addSplitsForPath(containedStatus, fs, target);
}
Debugging the issue found that, SecurityManager is coming as null while listing the files, hence skipping the permissions on directory.
What is the way to set SecurityManager in Flink?
Regards,
Kirti Dhar
Re: SecurityManager in Flink
Posted by Hang Ruan <ru...@gmail.com>.
Hi, Kirti.
Could you please provide the stack trace of this NPE? I check the code and
I think maybe the problem lies in LocalFileSystem#listStatus.
The code in line 161[1] may return null, which will let
LocalFileSystem#listStatus return null. Then the `containedFiles` is null
and the NPE occurs.
I think we should add code to handle this situation as follows.
```
final FileStatus[] containedFiles = fs.listStatus(fileStatus.getPath());
if (containedFiles == null) {
throw new FlinkRuntimeException("Cannot list files under " +
fileStatus.getPath());
}
for (FileStatus containedStatus : containedFiles) {
addSplitsForPath(containedStatus, fs, target);
}
```
Best,
Hang
[1]
https://github.com/apache/flink/blob/9b1375520b6b351df7551d85fcecd920e553cc3a/flink-core/src/main/java/org/apache/flink/core/fs/local/LocalFileSystem.java#L161C32-L161C38
Kirti Dhar Upadhyay K via user <us...@flink.apache.org> 于2024年3月6日周三 18:10写道：
> Hi Team,
>
>
>
> I am using Flink File Source with Local File System.
>
> I am facing an issue, if source directory does not has read permission, it
> is returning the list of files as null instead of throwing permission
> exception (refer the highlighted line below), resulting in NPE.
>
>
>
> final FileStatus[] containedFiles = fs.listStatus(fileStatus.getPath());
> for (FileStatus containedStatus : containedFiles) {
> addSplitsForPath(containedStatus, fs, target);
> }
>
> Debugging the issue found that, SecurityManager is coming as null while
> listing the files, hence skipping the permissions on directory.
>
> What is the way to set SecurityManager in Flink?
>
>
>
> Regards,
>
> Kirti Dhar
>
>
>