You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by FelixNeutatz <gi...@git.apache.org> on 2015/01/07 09:49:25 UTC
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
GitHub user FelixNeutatz opened a pull request:
https://github.com/apache/flink/pull/287
[FLINK-1271] Remove writable limitation
This pull request will remove the limitation of the Hadoop Format to use writables. This makes it possible to use Parquet.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/FelixNeutatz/incubator-flink RemoveWritableLimitation
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/287.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #287
----
commit b5f399d933f0d0697c7b17752277ad4f751eb2c2
Author: FelixNeutatz <ne...@googlemail.com>
Date: 2015-01-06T19:47:00Z
[FLINK-1271] Remove Writable limitation from Hadoop Format
commit 43be886042cb145b0a4677e7e5528ea7eb1fedb0
Author: FelixNeutatz <ne...@googlemail.com>
Date: 2015-01-06T19:58:45Z
[FLINK-1271] clean format
commit 44da293b6a872e2a97665088af6b7088ef4befe6
Author: FelixNeutatz <ne...@googlemail.com>
Date: 2015-01-06T20:09:52Z
[FLINK-1271] clean format 2
commit 8001adb5272e3e2d866025445a4aa32fa82c6329
Author: FelixNeutatz <ne...@googlemail.com>
Date: 2015-01-06T20:16:48Z
[FLINK-1271] clean3
commit 6f634c6950901f63288113786e9f4afb4c32ca97
Author: FelixNeutatz <ne...@googlemail.com>
Date: 2015-01-06T20:22:53Z
[FLINK-1271] clean 4
commit e9d3b7bd6e578aafe14935fe0d3aa7daa8a4d311
Author: FelixNeutatz <ne...@googlemail.com>
Date: 2015-01-06T20:27:43Z
[FLINK-1271] clean 5
commit 0670c4cc967700cd7ba685e3e2085950bca26aa8
Author: FelixNeutatz <ne...@googlemail.com>
Date: 2015-01-06T20:31:56Z
[FLINK-1271] clean 5
commit fefb880f496043d7fc9cac896210740dfa18f57e
Author: FelixNeutatz <ne...@googlemail.com>
Date: 2015-01-06T20:36:17Z
[FLINK-1271] clean 7
commit 0eb74d2929dd4c5659a9f05064d32fbeb9bec5c8
Author: FelixNeutatz <ne...@googlemail.com>
Date: 2015-01-06T20:54:04Z
[FLINK-1271] clean up +1
commit b50324c751a52b31d8533a2d2116191715f504b0
Author: FelixNeutatz <ne...@googlemail.com>
Date: 2015-01-06T20:58:16Z
[FLINK-1271] clean up
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by FelixNeutatz <gi...@git.apache.org>.
Github user FelixNeutatz commented on the pull request:
https://github.com/apache/flink/pull/287#issuecomment-69153105
I will try ;)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/287#issuecomment-69358588
Yes, that would be nice. The test looks good.
A bit of style cleanup and some commit squashing and this is good to merge in my opinion.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/287#issuecomment-69221721
That looks good, thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/287#issuecomment-69049055
Looks good to me.
Can we add a simple test that instantiates the input format wrapper with a MapRed input format <Void, Long> (as in the buggy case) and see that it returns the correct type?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by FelixNeutatz <gi...@git.apache.org>.
Github user FelixNeutatz commented on the pull request:
https://github.com/apache/flink/pull/287#issuecomment-69220758
I added a simple unit test: HadoopInputFormatTest.java
Did you mean this test?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/287#issuecomment-68994150
Hi,
the changes look good. Can you apply the changes also to the "mapred" input/output format? https://github.com/apache/flink/blob/master/flink-addons/flink-hadoop-compatibility/src/main/java/org/apache/flink/hadoopcompatibility/mapred/HadoopInputFormat.java#L53
Also, it would be nice if you could squash your changes into one commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/287
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by FelixNeutatz <gi...@git.apache.org>.
Github user FelixNeutatz commented on the pull request:
https://github.com/apache/flink/pull/287#issuecomment-69222057
Shall also add the same test for the mapred package?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by MohamedNadjibMAMI <gi...@git.apache.org>.
Github user MohamedNadjibMAMI commented on the pull request:
https://github.com/apache/flink/pull/287#issuecomment-94259921
"If you want you can write a blog post about using Parquet with Flink." This would a great plus for the project. Hope it goes a bit deeper and enters the official documentation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/flink/pull/287#issuecomment-69160636
1) Should be possible right? Our system supports all types. So I would do it.
2) You probably don't need a full word count for testing the instantiation and type extraction.
3) In the flink-java-examples? I'm not so sure about that because it adds a lot of new dependencies to the project. If you want you can write a blog post about using Parquet with Flink.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by hsaputra <gi...@git.apache.org>.
Github user hsaputra commented on the pull request:
https://github.com/apache/flink/pull/287#issuecomment-69120239
To clean up history of the changes, could you munge the commits into one?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by FelixNeutatz <gi...@git.apache.org>.
Github user FelixNeutatz commented on the pull request:
https://github.com/apache/flink/pull/287#issuecomment-69113149
I have three questions:
1) Shall I also remove the limitations from HadoopMapFunction.java, and HadoopReduceFunction.java, ...
2) With "add a simple test that instantiates the input format wrapper" you mean another Wordcount example with the corresponding "buggy" configuration or can you explain that in more detail?
3) What about a Parquet example?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by FelixNeutatz <gi...@git.apache.org>.
Github user FelixNeutatz commented on the pull request:
https://github.com/apache/flink/pull/287#issuecomment-69513699
I rebased some of the commits as requested
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by fhueske <gi...@git.apache.org>.
Github user fhueske commented on the pull request:
https://github.com/apache/flink/pull/287#issuecomment-70112509
LGTM
Will do some tests, clean-up and merge if everything is fine
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1271] Remove writable limitation
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/287#issuecomment-69161456
Concerning (2) - Let's not add another WordCount - simply making sure that the right type info is found should be sufficient.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---