You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org> on 2008/04/05 10:45:24 UTC
[jira] Created: (HADOOP-3196) get rid of excessive flushes from
PipeMapper/Reducer
get rid of excessive flushes from PipeMapper/Reducer
----------------------------------------------------
Key: HADOOP-3196
URL: https://issues.apache.org/jira/browse/HADOOP-3196
Project: Hadoop Core
Issue Type: Bug
Components: contrib/streaming
Affects Versions: 0.16.2
Reporter: Joydeep Sen Sarma
there's a flush on the buffered output streams in mapper/reducer for every row of data.
// 2/4 Hadoop to Tool
if (numExceptions_ == 0) {
if (!this.ignoreKey) {
write(key);
clientOut_.write('\t');
}
write(value);
if(!this.skipNewline) {
clientOut_.write('\n');
}
clientOut_.flush();
} else {
numRecSkipped_++;
}
tried to measure impact of removing this. number of context switches reported by vmstat shows marked decline.
with flush (10 second intervals):
r b swpd free buff cache si so bi bo in cs us sy id wa
4 2 784 23140 83352 3114648 0 0 4819 32397 1175 13220 59 11 13 17
1 2 784 129724 80704 3075696 0 0 4614 27196 1156 14797 49 11 19 21
4 0 784 24160 83440 3174880 0 0 96 36070 1337 10976 67 11 9 12
5 0 784 155872 84400 3158840 0 0 125 44084 1280 11044 68 14 10 8
2 1 784 365128 87048 2892032 0 0 119 38472 1317 11610 69 14 10 7
without flush:
5 0 784 24652 56056 3217864 0 0 310 29499 1379 7603 76 9 7 8
5 3 784 118456 54568 3209992 0 0 3249 33426 1173 6828 63 11 12 14
0 2 784 227628 54820 3198560 0 0 7840 30063 1146 8899 60 10 15 15
3 1 784 25608 55048 3313512 0 0 3251 36276 1194 7915 60 10 15 15
1 2 784 197324 49968 3194572 0 0 4714 35479 1281 8204 62 13 12 13
cs goes down by about 20-30%. but having trouble measuring overall speed improvement (too many variables due to spec. execution etc. - need better benchmark).
can't hurt.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3196) get rid of excessive flushes from
PipeMapper/Reducer
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586532#action_12586532 ]
Joydeep Sen Sarma commented on HADOOP-3196:
-------------------------------------------
understood. we can make the flush timer driven as suggested.
> get rid of excessive flushes from PipeMapper/Reducer
> ----------------------------------------------------
>
> Key: HADOOP-3196
> URL: https://issues.apache.org/jira/browse/HADOOP-3196
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.16.2
> Reporter: Joydeep Sen Sarma
>
> there's a flush on the buffered output streams in mapper/reducer for every row of data.
> // 2/4 Hadoop to Tool
> if (numExceptions_ == 0) {
> if (!this.ignoreKey) {
> write(key);
> clientOut_.write('\t');
> }
> write(value);
> if(!this.skipNewline) {
> clientOut_.write('\n');
> }
> clientOut_.flush();
> } else {
> numRecSkipped_++;
> }
> tried to measure impact of removing this. number of context switches reported by vmstat shows marked decline.
> with flush (10 second intervals):
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 4 2 784 23140 83352 3114648 0 0 4819 32397 1175 13220 59 11 13 17
> 1 2 784 129724 80704 3075696 0 0 4614 27196 1156 14797 49 11 19 21
> 4 0 784 24160 83440 3174880 0 0 96 36070 1337 10976 67 11 9 12
> 5 0 784 155872 84400 3158840 0 0 125 44084 1280 11044 68 14 10 8
> 2 1 784 365128 87048 2892032 0 0 119 38472 1317 11610 69 14 10 7
> without flush:
> 5 0 784 24652 56056 3217864 0 0 310 29499 1379 7603 76 9 7 8
> 5 3 784 118456 54568 3209992 0 0 3249 33426 1173 6828 63 11 12 14
> 0 2 784 227628 54820 3198560 0 0 7840 30063 1146 8899 60 10 15 15
> 3 1 784 25608 55048 3313512 0 0 3251 36276 1194 7915 60 10 15 15
> 1 2 784 197324 49968 3194572 0 0 4714 35479 1281 8204 62 13 12 13
> cs goes down by about 20-30%. but having trouble measuring overall speed improvement (too many variables due to spec. execution etc. - need better benchmark).
> can't hurt.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3196) get rid of excessive flushes from
PipeMapper/Reducer
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586511#action_12586511 ]
Doug Cutting commented on HADOOP-3196:
--------------------------------------
> mapper and the reducer classes are consuming data -isn't that signal enough [...]?
We currently count both consumption and generation of data as activity. Some mappers might, e.g., take lists of file names, and not consume new input lines very often. Noticing progress on output is important for some applications.
I'm not arguing that this is the only or even the ideal way to handle progress notification for streaming, but rather that it is the way that things currently work, and that simply eliminating these flushes has the potential to break applications which rely on them for prompt progress reports.
> get rid of excessive flushes from PipeMapper/Reducer
> ----------------------------------------------------
>
> Key: HADOOP-3196
> URL: https://issues.apache.org/jira/browse/HADOOP-3196
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.16.2
> Reporter: Joydeep Sen Sarma
>
> there's a flush on the buffered output streams in mapper/reducer for every row of data.
> // 2/4 Hadoop to Tool
> if (numExceptions_ == 0) {
> if (!this.ignoreKey) {
> write(key);
> clientOut_.write('\t');
> }
> write(value);
> if(!this.skipNewline) {
> clientOut_.write('\n');
> }
> clientOut_.flush();
> } else {
> numRecSkipped_++;
> }
> tried to measure impact of removing this. number of context switches reported by vmstat shows marked decline.
> with flush (10 second intervals):
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 4 2 784 23140 83352 3114648 0 0 4819 32397 1175 13220 59 11 13 17
> 1 2 784 129724 80704 3075696 0 0 4614 27196 1156 14797 49 11 19 21
> 4 0 784 24160 83440 3174880 0 0 96 36070 1337 10976 67 11 9 12
> 5 0 784 155872 84400 3158840 0 0 125 44084 1280 11044 68 14 10 8
> 2 1 784 365128 87048 2892032 0 0 119 38472 1317 11610 69 14 10 7
> without flush:
> 5 0 784 24652 56056 3217864 0 0 310 29499 1379 7603 76 9 7 8
> 5 3 784 118456 54568 3209992 0 0 3249 33426 1173 6828 63 11 12 14
> 0 2 784 227628 54820 3198560 0 0 7840 30063 1146 8899 60 10 15 15
> 3 1 784 25608 55048 3313512 0 0 3251 36276 1194 7915 60 10 15 15
> 1 2 784 197324 49968 3194572 0 0 4714 35479 1281 8204 62 13 12 13
> cs goes down by about 20-30%. but having trouble measuring overall speed improvement (too many variables due to spec. execution etc. - need better benchmark).
> can't hurt.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3196) get rid of excessive flushes from
PipeMapper/Reducer
Posted by "Rick Cox (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587267#action_12587267 ]
Rick Cox commented on HADOOP-3196:
----------------------------------
Task timeouts and progress reporting do function correctly in current releases of streaming, so, while the scenarios where these flushes are necessary seem a bit far fetched (some kind of low but non-zero rate input), that doesn't mean nobody is depending on them.
For jobs with task timeouts enabled, it seems reasonable to promise a flush at least every {{mapred.task.timeout/2}} interval.
> get rid of excessive flushes from PipeMapper/Reducer
> ----------------------------------------------------
>
> Key: HADOOP-3196
> URL: https://issues.apache.org/jira/browse/HADOOP-3196
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.16.2
> Reporter: Joydeep Sen Sarma
>
> there's a flush on the buffered output streams in mapper/reducer for every row of data.
> // 2/4 Hadoop to Tool
> if (numExceptions_ == 0) {
> if (!this.ignoreKey) {
> write(key);
> clientOut_.write('\t');
> }
> write(value);
> if(!this.skipNewline) {
> clientOut_.write('\n');
> }
> clientOut_.flush();
> } else {
> numRecSkipped_++;
> }
> tried to measure impact of removing this. number of context switches reported by vmstat shows marked decline.
> with flush (10 second intervals):
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 4 2 784 23140 83352 3114648 0 0 4819 32397 1175 13220 59 11 13 17
> 1 2 784 129724 80704 3075696 0 0 4614 27196 1156 14797 49 11 19 21
> 4 0 784 24160 83440 3174880 0 0 96 36070 1337 10976 67 11 9 12
> 5 0 784 155872 84400 3158840 0 0 125 44084 1280 11044 68 14 10 8
> 2 1 784 365128 87048 2892032 0 0 119 38472 1317 11610 69 14 10 7
> without flush:
> 5 0 784 24652 56056 3217864 0 0 310 29499 1379 7603 76 9 7 8
> 5 3 784 118456 54568 3209992 0 0 3249 33426 1173 6828 63 11 12 14
> 0 2 784 227628 54820 3198560 0 0 7840 30063 1146 8899 60 10 15 15
> 3 1 784 25608 55048 3313512 0 0 3251 36276 1194 7915 60 10 15 15
> 1 2 784 197324 49968 3194572 0 0 4714 35479 1281 8204 62 13 12 13
> cs goes down by about 20-30%. but having trouble measuring overall speed improvement (too many variables due to spec. execution etc. - need better benchmark).
> can't hurt.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3196) get rid of excessive flushes from
PipeMapper/Reducer
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586493#action_12586493 ]
Joydeep Sen Sarma commented on HADOOP-3196:
-------------------------------------------
sorry - don't understand.
the Pipe mapper and the reducer classes are consuming data -isn't that signal enough to the trackers that the task is making progress?
if the data is buffered - it seems that the worst that can happen is that script output is delayed - but this seems a common pattern anyway - since reduce scripts will frequently consume huge-amounts/all of data before emitting anything.
i have checked in a patch to our local production environment to see if we observe any weirdness ..
> get rid of excessive flushes from PipeMapper/Reducer
> ----------------------------------------------------
>
> Key: HADOOP-3196
> URL: https://issues.apache.org/jira/browse/HADOOP-3196
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.16.2
> Reporter: Joydeep Sen Sarma
>
> there's a flush on the buffered output streams in mapper/reducer for every row of data.
> // 2/4 Hadoop to Tool
> if (numExceptions_ == 0) {
> if (!this.ignoreKey) {
> write(key);
> clientOut_.write('\t');
> }
> write(value);
> if(!this.skipNewline) {
> clientOut_.write('\n');
> }
> clientOut_.flush();
> } else {
> numRecSkipped_++;
> }
> tried to measure impact of removing this. number of context switches reported by vmstat shows marked decline.
> with flush (10 second intervals):
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 4 2 784 23140 83352 3114648 0 0 4819 32397 1175 13220 59 11 13 17
> 1 2 784 129724 80704 3075696 0 0 4614 27196 1156 14797 49 11 19 21
> 4 0 784 24160 83440 3174880 0 0 96 36070 1337 10976 67 11 9 12
> 5 0 784 155872 84400 3158840 0 0 125 44084 1280 11044 68 14 10 8
> 2 1 784 365128 87048 2892032 0 0 119 38472 1317 11610 69 14 10 7
> without flush:
> 5 0 784 24652 56056 3217864 0 0 310 29499 1379 7603 76 9 7 8
> 5 3 784 118456 54568 3209992 0 0 3249 33426 1173 6828 63 11 12 14
> 0 2 784 227628 54820 3198560 0 0 7840 30063 1146 8899 60 10 15 15
> 3 1 784 25608 55048 3313512 0 0 3251 36276 1194 7915 60 10 15 15
> 1 2 784 197324 49968 3194572 0 0 4714 35479 1281 8204 62 13 12 13
> cs goes down by about 20-30%. but having trouble measuring overall speed improvement (too many variables due to spec. execution etc. - need better benchmark).
> can't hurt.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3196) get rid of excessive flushes from
PipeMapper/Reducer
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12586457#action_12586457 ]
Doug Cutting commented on HADOOP-3196:
--------------------------------------
I think the purpose of these flushes is to signal activity to the parent process, so that the streaming task is not killed for inactivity. Removing the flushes, while more efficient, changes this contract. Perhaps we need something like a timer-based flush running in a separate thread?
> get rid of excessive flushes from PipeMapper/Reducer
> ----------------------------------------------------
>
> Key: HADOOP-3196
> URL: https://issues.apache.org/jira/browse/HADOOP-3196
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.16.2
> Reporter: Joydeep Sen Sarma
>
> there's a flush on the buffered output streams in mapper/reducer for every row of data.
> // 2/4 Hadoop to Tool
> if (numExceptions_ == 0) {
> if (!this.ignoreKey) {
> write(key);
> clientOut_.write('\t');
> }
> write(value);
> if(!this.skipNewline) {
> clientOut_.write('\n');
> }
> clientOut_.flush();
> } else {
> numRecSkipped_++;
> }
> tried to measure impact of removing this. number of context switches reported by vmstat shows marked decline.
> with flush (10 second intervals):
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 4 2 784 23140 83352 3114648 0 0 4819 32397 1175 13220 59 11 13 17
> 1 2 784 129724 80704 3075696 0 0 4614 27196 1156 14797 49 11 19 21
> 4 0 784 24160 83440 3174880 0 0 96 36070 1337 10976 67 11 9 12
> 5 0 784 155872 84400 3158840 0 0 125 44084 1280 11044 68 14 10 8
> 2 1 784 365128 87048 2892032 0 0 119 38472 1317 11610 69 14 10 7
> without flush:
> 5 0 784 24652 56056 3217864 0 0 310 29499 1379 7603 76 9 7 8
> 5 3 784 118456 54568 3209992 0 0 3249 33426 1173 6828 63 11 12 14
> 0 2 784 227628 54820 3198560 0 0 7840 30063 1146 8899 60 10 15 15
> 3 1 784 25608 55048 3313512 0 0 3251 36276 1194 7915 60 10 15 15
> 1 2 784 197324 49968 3194572 0 0 4714 35479 1281 8204 62 13 12 13
> cs goes down by about 20-30%. but having trouble measuring overall speed improvement (too many variables due to spec. execution etc. - need better benchmark).
> can't hurt.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3196) get rid of excessive flushes from
PipeMapper/Reducer
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12587084#action_12587084 ]
Owen O'Malley commented on HADOOP-3196:
---------------------------------------
Actually, there were so many problems with tracking progress on streaming that it was disabled by default.
{code}
// All streaming jobs have, by default, no time-out for tasks
jobConf_.setLong("mapred.task.timeout", 0);
{code}
I think it is ok to remove the flushes. The only downside that I can see is if you have your input format doing slow low-volumne fetching, you'll increase the latency of the map.
> get rid of excessive flushes from PipeMapper/Reducer
> ----------------------------------------------------
>
> Key: HADOOP-3196
> URL: https://issues.apache.org/jira/browse/HADOOP-3196
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Affects Versions: 0.16.2
> Reporter: Joydeep Sen Sarma
>
> there's a flush on the buffered output streams in mapper/reducer for every row of data.
> // 2/4 Hadoop to Tool
> if (numExceptions_ == 0) {
> if (!this.ignoreKey) {
> write(key);
> clientOut_.write('\t');
> }
> write(value);
> if(!this.skipNewline) {
> clientOut_.write('\n');
> }
> clientOut_.flush();
> } else {
> numRecSkipped_++;
> }
> tried to measure impact of removing this. number of context switches reported by vmstat shows marked decline.
> with flush (10 second intervals):
> r b swpd free buff cache si so bi bo in cs us sy id wa
> 4 2 784 23140 83352 3114648 0 0 4819 32397 1175 13220 59 11 13 17
> 1 2 784 129724 80704 3075696 0 0 4614 27196 1156 14797 49 11 19 21
> 4 0 784 24160 83440 3174880 0 0 96 36070 1337 10976 67 11 9 12
> 5 0 784 155872 84400 3158840 0 0 125 44084 1280 11044 68 14 10 8
> 2 1 784 365128 87048 2892032 0 0 119 38472 1317 11610 69 14 10 7
> without flush:
> 5 0 784 24652 56056 3217864 0 0 310 29499 1379 7603 76 9 7 8
> 5 3 784 118456 54568 3209992 0 0 3249 33426 1173 6828 63 11 12 14
> 0 2 784 227628 54820 3198560 0 0 7840 30063 1146 8899 60 10 15 15
> 3 1 784 25608 55048 3313512 0 0 3251 36276 1194 7915 60 10 15 15
> 1 2 784 197324 49968 3194572 0 0 4714 35479 1281 8204 62 13 12 13
> cs goes down by about 20-30%. but having trouble measuring overall speed improvement (too many variables due to spec. execution etc. - need better benchmark).
> can't hurt.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.