You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by tillrohrmann <gi...@git.apache.org> on 2015/09/17 16:15:06 UTC
[GitHub] flink pull request: [FLINK-2690] [api-breaking] [scala api] [java ...
GitHub user tillrohrmann opened a pull request:
https://github.com/apache/flink/pull/1141
[FLINK-2690] [api-breaking] [scala api] [java api] Allows CsvInputFormat to use derived Pojos
This PR adds support for the `CsvInputFormat` to use derived Pojos. In order to also find pojo fields defined in a parent class, one has to traverse the type hierarchy. This is done by the function `findAllFields`.
While working on the `CsvInputFormat`, I noticed that the `ScalaCsvInputFormat` shared almost all code with the `CsvInputFormat`. In order to reduce duplicated code, both input formats now extend the `CommonCsvInputFormat` which contains the shared code.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tillrohrmann/flink fixCsvInputFormatPojo
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/1141.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1141
----
commit bf2e2c40f340ccfe31256744380345287c9e6f9e
Author: Till Rohrmann <tr...@apache.org>
Date: 2015-09-17T13:28:25Z
[FLINK-2690] [api-breaking] [scala api] [java api] Adds functionality to the CsvInputFormat to find fields defined in a super class of a Pojo. Refactors CsvInputFormat to share code between this format and ScalaCsvInputFormat.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2690] [api-breaking] [scala api] [java ...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/1141#discussion_r40327817
--- Diff: flink-staging/flink-language-binding/flink-language-binding-generic/src/main/java/org/apache/flink/languagebinding/api/java/common/PlanBinder.java ---
@@ -256,7 +258,20 @@ protected OperationInfo createOperationInfo(Operation operationIdentifier) throw
protected abstract INFO createOperationInfo(AbstractOperation operationIdentifier) throws IOException;
private void createCsvSource(OperationInfo info) throws IOException {
- sets.put(info.setID, env.createInput(new CsvInputFormat(new Path(info.path), info.lineDelimiter, info.fieldDelimiter, info.types), info.types).name("CsvSource"));
+ if (!(info.types instanceof CompositeType)) {
+ throw new RuntimeException("The output type of a csv source has to be a tuple or a " +
+ "pojo type. The derived type is " + info);
+ }
+
+ sets.put(
--- End diff --
Style comment: I think that is a bit much line breaking for the common Java style. I think the common Java styles don't do that quite as heavy as the Scala style.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2690] [api-breaking] [scala api] [java ...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/1141#discussion_r40329651
--- Diff: flink-staging/flink-language-binding/flink-language-binding-generic/src/main/java/org/apache/flink/languagebinding/api/java/common/PlanBinder.java ---
@@ -256,7 +258,20 @@ protected OperationInfo createOperationInfo(Operation operationIdentifier) throw
protected abstract INFO createOperationInfo(AbstractOperation operationIdentifier) throws IOException;
private void createCsvSource(OperationInfo info) throws IOException {
- sets.put(info.setID, env.createInput(new CsvInputFormat(new Path(info.path), info.lineDelimiter, info.fieldDelimiter, info.types), info.types).name("CsvSource"));
+ if (!(info.types instanceof CompositeType)) {
+ throw new RuntimeException("The output type of a csv source has to be a tuple or a " +
+ "pojo type. The derived type is " + info);
+ }
+
+ sets.put(
--- End diff --
That is true. Will change it when merging.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2690] [api-breaking] [scala api] [java ...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1141#issuecomment-142953065
Looks good to me, feel free to merge.
Minor comment: We now have a `GenericCsvInputFormat` and a `CommonCSVInputFormat`. Anything preventing us from unifying them into one class? Or are there conflicts with the deprecated Record API?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2690] [api-breaking] [scala api] [java ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/1141
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2690] [api-breaking] [scala api] [java ...
Posted by chiwanpark <gi...@git.apache.org>.
Github user chiwanpark commented on the pull request:
https://github.com/apache/flink/pull/1141#issuecomment-143192190
Looks good to merge and +1 for FLINK-2692
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2690] [api-breaking] [scala api] [java ...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/1141#issuecomment-143697315
Will merge this now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2690] [api-breaking] [scala api] [java ...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the pull request:
https://github.com/apache/flink/pull/1141#issuecomment-142956414
Yes the reason why I introduced the `CommonCSVInputFormat` instead of unifying it with the `GenericCsvInputFormat` was that we still have the Record API `CsvInputFormat` in the code base. It doesn't need the pojo type information and, thus, I thought that it does not make sense to let it inherit from a class which has these fields. Actually, I would be in favour of removing the Record API's `CsvInputFormat` if possible. Moreover, I would like to more cleanly separate the tuple type `CsvInputFormat` and the pojo type `CsvInputFormat`, because the pojo support was kind of hacked into the original `CsvInputFormat`. Also the way the input format is set up, is imho a little bit clumsy and error-prone (see https://issues.apache.org/jira/browse/FLINK-2692). But this has no high priority.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---