You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Runping Qi (JIRA)" <ji...@apache.org> on 2007/05/04 23:37:15 UTC
[jira] Created: (HADOOP-1328) Hadoop Streaming needs to provide a
way for the stream plugin to update global counters
Hadoop Streaming needs to provide a way for the stream plugin to update global counters
---------------------------------------------------------------------------------------
Key: HADOOP-1328
URL: https://issues.apache.org/jira/browse/HADOOP-1328
Project: Hadoop
Issue Type: New Feature
Reporter: Runping Qi
Sometimes, the stream plugin may want to create/update its own global counters.
There is no way to do so currently.
One possible way to enable that is to use the stderr output of the stream process.
The stream process can emit the global counter update information (such as GLOCAL_COUNTER COUNTER_NAME NUM)
to the stderr of the process. The Stderr handling thread can call the global counter API whenever it encounters the global counter update information line.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Created: (HADOOP-1328) Hadoop Streaming needs to provide a way for the stream plugin to update global counters
Posted by Arkady Borkovsky <ar...@yahoo-inc.com>.
This is a really good suggestion.
In general, this a good way to let a streaming process to send
information to the Java streaming class:
stderr lines that match a predefined regular expression are treated
as commands.
For example, I use this to kill the whole job: if a tasks runs into a
deterministic error (e.g. a syntax error in the script), there is no
need to wait for N*4 tasks to fail)
However, before this implemented, hadnling of streaming stderr needs
to be improved.
I still have to save the stderr in DFS with my own hacks.
On May 4, 2007, at 2:37 PM, Runping Qi (JIRA) wrote:
> Hadoop Streaming needs to provide a way for the stream plugin to
> update global counters
> ----------------------------------------------------------------------
> -----------------
>
> Key: HADOOP-1328
> URL: https://issues.apache.org/jira/browse/
> HADOOP-1328
> Project: Hadoop
> Issue Type: New Feature
> Reporter: Runping Qi
>
>
>
> Sometimes, the stream plugin may want to create/update its own
> global counters.
> There is no way to do so currently.
> One possible way to enable that is to use the stderr output of the
> stream process.
> The stream process can emit the global counter update information
> (such as GLOCAL_COUNTER COUNTER_NAME NUM)
> to the stderr of the process. The Stderr handling thread can call
> the global counter API whenever it encounters the global counter
> update information line.
>
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
[jira] Updated: (HADOOP-1328) Hadoop Streaming needs to provide a
way for the stream plugin to update global counters
Posted by "Sameer Paranjpye (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sameer Paranjpye updated HADOOP-1328:
-------------------------------------
Component/s: contrib/streaming
Assignee: Owen O'Malley
Description:
Sometimes, the stream plugin may want to create/update its own global counters.
There is no way to do so currently.
One possible way to enable that is to use the stderr output of the stream process.
The stream process can emit the global counter update information (such as GLOCAL_COUNTER COUNTER_NAME NUM)
to the stderr of the process. The Stderr handling thread can call the global counter API whenever it encounters the global counter update information line.
was:
Sometimes, the stream plugin may want to create/update its own global counters.
There is no way to do so currently.
One possible way to enable that is to use the stderr output of the stream process.
The stream process can emit the global counter update information (such as GLOCAL_COUNTER COUNTER_NAME NUM)
to the stderr of the process. The Stderr handling thread can call the global counter API whenever it encounters the global counter update information line.
> Hadoop Streaming needs to provide a way for the stream plugin to update global counters
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-1328
> URL: https://issues.apache.org/jira/browse/HADOOP-1328
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: Runping Qi
> Assigned To: Owen O'Malley
>
> Sometimes, the stream plugin may want to create/update its own global counters.
> There is no way to do so currently.
> One possible way to enable that is to use the stderr output of the stream process.
> The stream process can emit the global counter update information (such as GLOCAL_COUNTER COUNTER_NAME NUM)
> to the stderr of the process. The Stderr handling thread can call the global counter API whenever it encounters the global counter update information line.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-1328) Hadoop Streaming needs to provide a
way for the stream plugin to update global counters
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley reassigned HADOOP-1328:
-------------------------------------
Assignee: Christopher Zimmerman (was: Owen O'Malley)
> Hadoop Streaming needs to provide a way for the stream plugin to update global counters
> ---------------------------------------------------------------------------------------
>
> Key: HADOOP-1328
> URL: https://issues.apache.org/jira/browse/HADOOP-1328
> Project: Hadoop
> Issue Type: New Feature
> Components: contrib/streaming
> Reporter: Runping Qi
> Assignee: Christopher Zimmerman
>
> Sometimes, the stream plugin may want to create/update its own global counters.
> There is no way to do so currently.
> One possible way to enable that is to use the stderr output of the stream process.
> The stream process can emit the global counter update information (such as GLOCAL_COUNTER COUNTER_NAME NUM)
> to the stderr of the process. The Stderr handling thread can call the global counter API whenever it encounters the global counter update information line.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.