You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Patrick Linehan (JIRA)" <ji...@apache.org> on 2010/11/17 22:41:13 UTC
[jira] Created: (AVRO-697) Install SIGPIPE handler in the Avro
tools main method.
Install SIGPIPE handler in the Avro tools main method.
------------------------------------------------------
Key: AVRO-697
URL: https://issues.apache.org/jira/browse/AVRO-697
Project: Avro
Issue Type: Improvement
Components: java
Reporter: Patrick Linehan
The following command should complete quickly, no matter how many records "myfile.avro" contains:
java -jar avro-tools.jar tojson myfile.avro | head
However, because the tool JAR's main method does not install a SIGPIPE handler, this command can be quite slow for large files.
Suggested fix is to install a handler which calls "System.exit(128 + signal.getNumber())" in response to SIGPIPE. Typically the signal number ("getNumber") is 13, but no need to hard code it.
Potential issues:
* Tests which invoke Main.main() will necessarily force the signal handler to be installed. This may be undesirable, as the entire test JVM will be "infected" with the handler. Workarounds exist. Does not bother me, personally, however.
* Installing signal handlers generates compile-time warnings, if I'm not mistaken. Workarounds exist. Does not bother me, personally, however.
If this suggestion sounds reasonable, I'll post a patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-697) Install SIGPIPE handler in the Avro
tools main method.
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934997#action_12934997 ]
Doug Cutting commented on AVRO-697:
-----------------------------------
So the problem is that signal handlers require implementation-specific code, is that right? If so, then the reflection code should be carefully written to address that, so that things still work if the classes change or are not present. This seems like a useful feature, and if this is the only way to implement it then I guess that's what we have to do.
> Install SIGPIPE handler in the Avro tools main method.
> ------------------------------------------------------
>
> Key: AVRO-697
> URL: https://issues.apache.org/jira/browse/AVRO-697
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Patrick Linehan
>
> The following command should complete quickly, no matter how many records "myfile.avro" contains:
> java -jar avro-tools.jar tojson myfile.avro | head
> However, because the tool JAR's main method does not install a SIGPIPE handler, this command can be quite slow for large files.
> Suggested fix is to install a handler which calls "System.exit(128 + signal.getNumber())" in response to SIGPIPE. Typically the signal number ("getNumber") is 13, but no need to hard code it.
> Potential issues:
> * Tests which invoke Main.main() will necessarily force the signal handler to be installed. This may be undesirable, as the entire test JVM will be "infected" with the handler. Workarounds exist. Does not bother me, personally, however.
> * Installing signal handlers generates compile-time warnings, if I'm not mistaken. Workarounds exist. Does not bother me, personally, however.
> If this suggestion sounds reasonable, I'll post a patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-697) Install SIGPIPE handler in the Avro
tools main method.
Posted by "Patrick Linehan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933318#action_12933318 ]
Patrick Linehan commented on AVRO-697:
--------------------------------------
Shoot. Just noticed that javac in Ant has the "-Werror" flag enabled, so I won't be able to directly access the signal-related code in sun.misc. This is actually a completely reasonable setting for javac, but makes this particular bug a bit trickier.
The workaround I've used for this previously is to use basic reflection and an InvocationHandler to bypass javac's static checks. If I coded this up, would it get accepted?
> Install SIGPIPE handler in the Avro tools main method.
> ------------------------------------------------------
>
> Key: AVRO-697
> URL: https://issues.apache.org/jira/browse/AVRO-697
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Patrick Linehan
>
> The following command should complete quickly, no matter how many records "myfile.avro" contains:
> java -jar avro-tools.jar tojson myfile.avro | head
> However, because the tool JAR's main method does not install a SIGPIPE handler, this command can be quite slow for large files.
> Suggested fix is to install a handler which calls "System.exit(128 + signal.getNumber())" in response to SIGPIPE. Typically the signal number ("getNumber") is 13, but no need to hard code it.
> Potential issues:
> * Tests which invoke Main.main() will necessarily force the signal handler to be installed. This may be undesirable, as the entire test JVM will be "infected" with the handler. Workarounds exist. Does not bother me, personally, however.
> * Installing signal handlers generates compile-time warnings, if I'm not mistaken. Workarounds exist. Does not bother me, personally, however.
> If this suggestion sounds reasonable, I'll post a patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-697) Install SIGPIPE handler in the Avro
tools main method.
Posted by "Patrick Linehan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935061#action_12935061 ]
Patrick Linehan commented on AVRO-697:
--------------------------------------
Good point, Doug. I'll ensure that the code gracefully handles the case where the "sun.misc.*" classes cannot be loaded or do not implement the expected API.
> Install SIGPIPE handler in the Avro tools main method.
> ------------------------------------------------------
>
> Key: AVRO-697
> URL: https://issues.apache.org/jira/browse/AVRO-697
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Patrick Linehan
>
> The following command should complete quickly, no matter how many records "myfile.avro" contains:
> java -jar avro-tools.jar tojson myfile.avro | head
> However, because the tool JAR's main method does not install a SIGPIPE handler, this command can be quite slow for large files.
> Suggested fix is to install a handler which calls "System.exit(128 + signal.getNumber())" in response to SIGPIPE. Typically the signal number ("getNumber") is 13, but no need to hard code it.
> Potential issues:
> * Tests which invoke Main.main() will necessarily force the signal handler to be installed. This may be undesirable, as the entire test JVM will be "infected" with the handler. Workarounds exist. Does not bother me, personally, however.
> * Installing signal handlers generates compile-time warnings, if I'm not mistaken. Workarounds exist. Does not bother me, personally, however.
> If this suggestion sounds reasonable, I'll post a patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-697) Install SIGPIPE handler in the Avro
tools main method.
Posted by "Philip Zeyliger (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933184#action_12933184 ]
Philip Zeyliger commented on AVRO-697:
--------------------------------------
Seems reasonable. Main.main() isn't called by tests, I think: the commands are abstracted away so that main() is pretty small.
> Install SIGPIPE handler in the Avro tools main method.
> ------------------------------------------------------
>
> Key: AVRO-697
> URL: https://issues.apache.org/jira/browse/AVRO-697
> Project: Avro
> Issue Type: Improvement
> Components: java
> Reporter: Patrick Linehan
>
> The following command should complete quickly, no matter how many records "myfile.avro" contains:
> java -jar avro-tools.jar tojson myfile.avro | head
> However, because the tool JAR's main method does not install a SIGPIPE handler, this command can be quite slow for large files.
> Suggested fix is to install a handler which calls "System.exit(128 + signal.getNumber())" in response to SIGPIPE. Typically the signal number ("getNumber") is 13, but no need to hard code it.
> Potential issues:
> * Tests which invoke Main.main() will necessarily force the signal handler to be installed. This may be undesirable, as the entire test JVM will be "infected" with the handler. Workarounds exist. Does not bother me, personally, however.
> * Installing signal handlers generates compile-time warnings, if I'm not mistaken. Workarounds exist. Does not bother me, personally, however.
> If this suggestion sounds reasonable, I'll post a patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.