You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by mushketyk <gi...@git.apache.org> on 2017/01/25 09:27:43 UTC
[GitHub] flink pull request #3206: [FLINK-5612] Fix GlobPathFilter not-serializable e...
GitHub user mushketyk opened a pull request:
https://github.com/apache/flink/pull/3206
[FLINK-5612] Fix GlobPathFilter not-serializable exception
Thanks for contributing to Apache Flink. Before you open your pull request, please take the following check list into consideration.
If your changes take all of the items into account, feel free to open your pull request. For more information and/or questions please refer to the [How To Contribute guide](http://flink.apache.org/how-to-contribute.html).
In addition to going through the list, please provide a meaningful description of your changes.
- [x] General
- The pull request references the related JIRA issue ("[FLINK-XXX] Jira title text")
- The pull request addresses only one issue
- Each commit in the PR has a meaningful commit message (including the JIRA id)
- [x] Documentation
- Documentation has been added for new functionality
- Old documentation affected by the pull request has been updated
- JavaDoc for public methods has been added
- [x] Tests & Build
- Functionality added by the pull request is covered by tests
- `mvn clean verify` has been executed successfully locally or a Travis build has passed
Fixed GlobPathFilter serialization exception. As suggested in the JIRA I've made instantiation of PathMatcher's objects lazy to avoid their serialization.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mushketyk/flink fix-serialization
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/3206.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3206
----
commit 47ab75792340191a0bbfdc8e9a3b527f819181a3
Author: Ivan Mushketyk <iv...@gmail.com>
Date: 2017-01-25T09:24:21Z
[FLINK-5612] Fix GlobPathFilter not-serializable exception
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3206: [FLINK-5612] Fix GlobPathFilter not-serializable exceptio...
Posted by mushketyk <gi...@git.apache.org>.
Github user mushketyk commented on the issue:
https://github.com/apache/flink/pull/3206
@zentol Sure. Thank you for merging this!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3206: [FLINK-5612] Fix GlobPathFilter not-serializable exceptio...
Posted by mushketyk <gi...@git.apache.org>.
Github user mushketyk commented on the issue:
https://github.com/apache/flink/pull/3206
Hi @StephanEwen, @zentol
Thank you for your comments. I've update the PR accordingly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3206: [FLINK-5612] Fix GlobPathFilter not-serializable e...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3206#discussion_r97969620
--- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/GlobFilePathFilter.java ---
@@ -52,8 +53,13 @@
private static final long serialVersionUID = 1L;
- private final ArrayList<PathMatcher> includeMatchers;
- private final ArrayList<PathMatcher> excludeMatchers;
+ private final List<String> includePatterns;
+ private final List<String> excludePatterns;
+
+ // Path matchers are not serializable so we are delaying their
+ // creation until they are used
+ transient private ArrayList<PathMatcher> includeMatchers;
--- End diff --
Now this is nitpicky, but i think the convention is that transient comes after private.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3206: [FLINK-5612] Fix GlobPathFilter not-serializable e...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/3206#discussion_r97768800
--- Diff: flink-core/src/test/java/org/apache/flink/api/common/io/GlobFilePathFilterTest.java ---
@@ -138,4 +141,30 @@ public void doubleStarPattern() {
assertFalse(matcher.filterPath(new Path("a/b")));
assertFalse(matcher.filterPath(new Path("a/b/c")));
}
+
+ @Test(expected = NullPointerException.class)
+ public void testIncluePatternIsNull() {
+ new GlobFilePathFilter(
+ null,
+ Collections.<String>emptyList());
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testExcludePatternIsNull() {
+ new GlobFilePathFilter(
+ Collections.singletonList("**"),
+ null);
+ }
+
+ @Test
+ public void testGlobFilterSerializable() throws IOException {
--- End diff --
How about using `CommonTestUtils.createCopySerializable()` here? That you can validate also that the serialized copy behaves valid.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3206: [FLINK-5612] Fix GlobPathFilter not-serializable e...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3206#discussion_r97746438
--- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/GlobFilePathFilter.java ---
@@ -52,8 +53,13 @@
private static final long serialVersionUID = 1L;
- private final ArrayList<PathMatcher> includeMatchers;
- private final ArrayList<PathMatcher> excludeMatchers;
+ private final List<String> includePatterns;
+ private final List<String> excludePatterns;
+
+ // Path matchers are not serializable so we are delaying their
+ // creation until they are used
+ private ArrayList<PathMatcher> includeMatchers;
--- End diff --
these can be transient
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3206: [FLINK-5612] Fix GlobPathFilter not-serializable exceptio...
Posted by mushketyk <gi...@git.apache.org>.
Github user mushketyk commented on the issue:
https://github.com/apache/flink/pull/3206
Hi @StephanEwen, @zentol
Thank you for helping with this PR.
I've updated it according to your review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3206: [FLINK-5612] Fix GlobPathFilter not-serializable exceptio...
Posted by EronWright <gi...@git.apache.org>.
Github user EronWright commented on the issue:
https://github.com/apache/flink/pull/3206
Just hit this issue, was about to file an issue. Awesome to see this fixed.
It is odd to me that `GlobFilePathFilter` is marked `@Internal`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3206: [FLINK-5612] Fix GlobPathFilter not-serializable e...
Posted by mushketyk <gi...@git.apache.org>.
Github user mushketyk closed the pull request at:
https://github.com/apache/flink/pull/3206
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3206: [FLINK-5612] Fix GlobPathFilter not-serializable exceptio...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/3206
Looks good, will merge this and fix the public annotation on the fly..
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3206: [FLINK-5612] Fix GlobPathFilter not-serializable exceptio...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/3206
@mushketyk The changes have been merged, could you close this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3206: [FLINK-5612] Fix GlobPathFilter not-serializable exceptio...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/3206
@EronWright Agreed, this should be `@Public(Evolving)`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3206: [FLINK-5612] Fix GlobPathFilter not-serializable e...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3206#discussion_r97746636
--- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/GlobFilePathFilter.java ---
@@ -69,8 +75,8 @@ public GlobFilePathFilter() {
* @param excludePatterns glob patterns for files to exclude
*/
public GlobFilePathFilter(List<String> includePatterns, List<String> excludePatterns) {
- includeMatchers = buildPatterns(includePatterns);
- excludeMatchers = buildPatterns(excludePatterns);
+ this.includePatterns = Preconditions.checkNotNull(includePatterns);
+ this.excludePatterns = Preconditions.checkNotNull(excludePatterns) ;
--- End diff --
spaces before `;` should be removed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3206: [FLINK-5612] Fix GlobPathFilter not-serializable e...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3206#discussion_r97746573
--- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/GlobFilePathFilter.java ---
@@ -106,12 +112,27 @@ public boolean filterPath(Path filePath) {
return true;
}
+ public ArrayList<PathMatcher> getIncludeMatchers() {
--- End diff --
these should be private
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3206: [FLINK-5612] Fix GlobPathFilter not-serializable exceptio...
Posted by mushketyk <gi...@git.apache.org>.
Github user mushketyk commented on the issue:
https://github.com/apache/flink/pull/3206
Thank you @StephanEwen !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3206: [FLINK-5612] Fix GlobPathFilter not-serializable e...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/3206#discussion_r98027531
--- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/GlobFilePathFilter.java ---
@@ -52,8 +53,13 @@
private static final long serialVersionUID = 1L;
- private final ArrayList<PathMatcher> includeMatchers;
- private final ArrayList<PathMatcher> excludeMatchers;
+ private final List<String> includePatterns;
+ private final List<String> excludePatterns;
+
+ // Path matchers are not serializable so we are delaying their
+ // creation until they are used
+ transient private ArrayList<PathMatcher> includeMatchers;
--- End diff --
Agreed, let's get the style aligned with the rest of the project.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3206: [FLINK-5612] Fix GlobPathFilter not-serializable e...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/3206#discussion_r97802390
--- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/GlobFilePathFilter.java ---
@@ -52,8 +53,13 @@
private static final long serialVersionUID = 1L;
- private final ArrayList<PathMatcher> includeMatchers;
- private final ArrayList<PathMatcher> excludeMatchers;
+ private final List<String> includePatterns;
+ private final List<String> excludePatterns;
+
+ // Path matchers are not serializable so we are delaying their
+ // creation until they are used
+ private ArrayList<PathMatcher> includeMatchers;
--- End diff --
+1 to make them transient - this documents and enforces that
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #3206: [FLINK-5612] Fix GlobPathFilter not-serializable exceptio...
Posted by zentol <gi...@git.apache.org>.
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/3206
+1 to merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3206: [FLINK-5612] Fix GlobPathFilter not-serializable e...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/3206#discussion_r98027416
--- Diff: flink-core/src/test/java/org/apache/flink/api/common/io/GlobFilePathFilterTest.java ---
@@ -138,4 +140,27 @@ public void doubleStarPattern() {
assertFalse(matcher.filterPath(new Path("a/b")));
assertFalse(matcher.filterPath(new Path("a/b/c")));
}
+
+ @Test(expected = NullPointerException.class)
+ public void testIncluePatternIsNull() {
+ new GlobFilePathFilter(
+ null,
+ Collections.<String>emptyList());
+ }
+
+ @Test(expected = NullPointerException.class)
+ public void testExcludePatternIsNull() {
+ new GlobFilePathFilter(
+ Collections.singletonList("**"),
+ null);
+ }
+
+ @Test
+ public void testGlobFilterSerializable() throws IOException {
+ GlobFilePathFilter matcher = new GlobFilePathFilter(
+ Collections.singletonList("**"),
+ Collections.<String>emptyList());
+
+ CommonTestUtils.createCopySerializable(matcher);
--- End diff --
A doof addition here would be to check that after the serialization, the filter behaves in the same way as before.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---