You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by ffbin <gi...@git.apache.org> on 2015/08/03 14:58:52 UTC
[GitHub] flink pull request: [FLINK-2471]FileSinkFunction invoke performanc...
GitHub user ffbin opened a pull request:
https://github.com/apache/flink/pull/973
[FLINK-2471]FileSinkFunction invoke performance optimize
FileSinkFunction invoke funtion always call updateCondition, so we can use a variable updateTime to avoid frequent operate (System.currentTimeMillis() - lastTime).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ffbin/flink FLINK-2471
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/973.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #973
----
commit c2583cf6059053d85d8ba3cb03833abebb61744b
Author: ffbin <86...@qq.com>
Date: 2015-08-03T12:27:41Z
[FLINK-2471]FileSinkFunction invoke performance optimize
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2471]FileSinkFunction invoke performanc...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/973#issuecomment-127232829
Arithmetic (addition subtraction) in a pipeline is incredibly cheap on today's processors, to this is probably micro optimization ;-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2471]FileSinkFunction invoke performanc...
Posted by ffbin <gi...@git.apache.org>.
Github user ffbin commented on the pull request:
https://github.com/apache/flink/pull/973#issuecomment-127253382
In one millisecond, it will call updateCondition about 10000 times and it will do minus operate 10000 times.But update lastTime is less than times of call updateCondition.So i add updateTime to reduce call times of minus operate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2471]FileSinkFunction invoke performanc...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/973#issuecomment-127398383
If you want to contribute some performance improvements, how about bundling them per component. Something like "Improve File Sink" pull request, where all fixes to the FileSinkFunction go together.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2471]FileSinkFunction invoke performanc...
Posted by ffbin <gi...@git.apache.org>.
Github user ffbin commented on the pull request:
https://github.com/apache/flink/pull/973#issuecomment-127445514
Hi , i can not access https://travis-ci.org/apache/flink/builds/73886757. Open it , i can only see "Do you have a question?".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2471]Improve File Sink
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/973#issuecomment-141398402
Can we close this pull request? I think
The followup idea if adding a "TimeService" with a thread that avoid too frequent calls to `System.currentTimeMillis()` is an interesting idea...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2471]FileSinkFunction invoke performanc...
Posted by gyfora <gi...@git.apache.org>.
Github user gyfora commented on the pull request:
https://github.com/apache/flink/pull/973#issuecomment-127224133
I don't see that you avoid calling System.currentTimeMillis, you remove one addition and you also add one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2471]FileSinkFunction invoke performanc...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/973#issuecomment-127260841
Can you access this link here? https://travis-ci.org/apache/flink/builds/73886757
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2471]FileSinkFunction invoke performanc...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/973#issuecomment-127256237
The minus operation is not really evaluated in isolation. It is part of the processor instruction pipeline at that point (see superscalar execution architectures). It will most likely be computed in flight with other instructions without adding overhead, because the pipeline's throughput is at that point not limited by the ALU. Judging the cost or an isolated operation does not work any more on modern processors.
The processing time is in almost all cases dominated stalls in the pipeline doe to fetching variables from memory.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2471]FileSinkFunction invoke performanc...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/973#issuecomment-127331181
BTW: I am not saying we should not merge this. It is actually a fine fix.
I am only trying to help judge the importance of issues, to help deciding what would be important to fix.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2471]FileSinkFunction invoke performanc...
Posted by ffbin <gi...@git.apache.org>.
Github user ffbin commented on the pull request:
https://github.com/apache/flink/pull/973#issuecomment-127444552
Hi Stenphan, i can not access https://travis-ci.org/apache/flink/builds/73886757. Open it , i can only see "Do you have a question?".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2471]FileSinkFunction invoke performanc...
Posted by ffbin <gi...@git.apache.org>.
Github user ffbin commented on the pull request:
https://github.com/apache/flink/pull/973#issuecomment-127259881
hi, I can run successfully in local computer,but CI is failed,How can i see why CI is failed?
I can not see the details content ,it is "Do you have a question?".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2471]FileSinkFunction invoke performanc...
Posted by ffbin <gi...@git.apache.org>.
Github user ffbin commented on the pull request:
https://github.com/apache/flink/pull/973#issuecomment-127449876
hi, System.currentTimeMillis() is cost performance very much.I want to use a thread to call System.currentTimeMillis and update a static long variable millTime.All other module do not need call System.currentTimeMillis() and can use millTime directly.I have test it can improve performance very much.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-2471]Improve File Sink
Posted by ffbin <gi...@git.apache.org>.
Github user ffbin closed the pull request at:
https://github.com/apache/flink/pull/973
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---