You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Esteve Fernandez (JIRA)" <ji...@apache.org> on 2010/04/25 18:32:22 UTC
[jira] Created: (AVRO-528) Add support for Twisted
Add support for Twisted
-----------------------
Key: AVRO-528
URL: https://issues.apache.org/jira/browse/AVRO-528
Project: Avro
Issue Type: New Feature
Reporter: Esteve Fernandez
Currently, the Python port of Avro only supports synchronous blocking networking, adding Twisted-enabled client and server implementations will benefit projects that can't use synchronous I/O but want to use Avro.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-528) Add support for Twisted
Posted by "Esteve Fernandez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Esteve Fernandez updated AVRO-528:
----------------------------------
Attachment: 528-support-twisted-4.patch
This version corrects an error in the way the length attribute of the RequestStreamingProducer is calculated. It can also be set to UNKNOWN_LENGTH, but it's advised that if the body length can be calculated, it should be set.
However, if there's a way in Avro to pass an object that streams the message, instead of passing the entire message, it'd be possible to use the RequestStreamingProducer without knowing the message length beforehand.
> Add support for Twisted
> -----------------------
>
> Key: AVRO-528
> URL: https://issues.apache.org/jira/browse/AVRO-528
> Project: Avro
> Issue Type: New Feature
> Components: python
> Reporter: Esteve Fernandez
> Attachments: 528-support-twisted-2.patch, 528-support-twisted-3.patch, 528-support-twisted-4.patch, 528-twisted-support.patch
>
>
> Currently, the Python port of Avro only supports synchronous blocking networking, adding Twisted-enabled client and server implementations will benefit projects that can't use synchronous I/O but want to use Avro.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-528) Add support for Twisted
Posted by "Esteve Fernandez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865104#action_12865104 ]
Esteve Fernandez commented on AVRO-528:
---------------------------------------
Hi Jeff, sorry for the slow reply, I've been sick for the last week (I still am, though, argghh).
> Thanks for the patches! It's good to see you working with Avro. I have been traveling for the
> past several days, but I'm going to take a look at these soon.
Thank you very much for your quick response and kind words :-)
> One note: I'd love to get your thoughts on https://issues.apache.org/jira/browse/AVRO-406,
> in addition to this issue.
I barely have any experience with Hadoop, and I'm still learning the intricacies of Avro, but I'll try to contribute to the discussion.
> Hey Esteve: to be clear, I should just be looking at the most recent patch (528-support-
> twisted-4.patch), right?
Yes, I started some work on making the examples launchable by twistd (Twisted's launcher, which lets you choose any of the available reactors, among other things), but I couldn't finish it. In any case, if you could have a look at the latest patch (528-support-twisted-4.patch), that would be awesome. I can always create another ticket later and submit a patch for twistd and the examples.
Thanks a lot!
> Add support for Twisted
> -----------------------
>
> Key: AVRO-528
> URL: https://issues.apache.org/jira/browse/AVRO-528
> Project: Avro
> Issue Type: New Feature
> Components: python
> Reporter: Esteve Fernandez
> Attachments: 528-support-twisted-2.patch, 528-support-twisted-3.patch, 528-support-twisted-4.patch, 528-twisted-support.patch
>
>
> Currently, the Python port of Avro only supports synchronous blocking networking, adding Twisted-enabled client and server implementations will benefit projects that can't use synchronous I/O but want to use Avro.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-528) Add support for Twisted
Posted by "Esteve Fernandez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Esteve Fernandez updated AVRO-528:
----------------------------------
Component/s: python
> Add support for Twisted
> -----------------------
>
> Key: AVRO-528
> URL: https://issues.apache.org/jira/browse/AVRO-528
> Project: Avro
> Issue Type: New Feature
> Components: python
> Reporter: Esteve Fernandez
> Attachments: 528-support-twisted-2.patch, 528-twisted-support.patch
>
>
> Currently, the Python port of Avro only supports synchronous blocking networking, adding Twisted-enabled client and server implementations will benefit projects that can't use synchronous I/O but want to use Avro.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-528) Add support for Twisted
Posted by "Esteve Fernandez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Esteve Fernandez updated AVRO-528:
----------------------------------
Attachment: 528-support-twisted-3.patch
This patch includes a full stream-based client (both for incoming and for outgoing data) and two examples of a client and a server using the Twisted portion.
Unfortunately, the Twisted.web server doesn't support streams, we're working on it, though.
> Add support for Twisted
> -----------------------
>
> Key: AVRO-528
> URL: https://issues.apache.org/jira/browse/AVRO-528
> Project: Avro
> Issue Type: New Feature
> Components: python
> Reporter: Esteve Fernandez
> Attachments: 528-support-twisted-2.patch, 528-support-twisted-3.patch, 528-twisted-support.patch
>
>
> Currently, the Python port of Avro only supports synchronous blocking networking, adding Twisted-enabled client and server implementations will benefit projects that can't use synchronous I/O but want to use Avro.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-528) Add support for Twisted
Posted by "Jeff Hammerbacher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865075#action_12865075 ]
Jeff Hammerbacher commented on AVRO-528:
----------------------------------------
Hey Esteve: to be clear, I should just be looking at the most recent patch (528-support-twisted-4.patch), right?
> Add support for Twisted
> -----------------------
>
> Key: AVRO-528
> URL: https://issues.apache.org/jira/browse/AVRO-528
> Project: Avro
> Issue Type: New Feature
> Components: python
> Reporter: Esteve Fernandez
> Attachments: 528-support-twisted-2.patch, 528-support-twisted-3.patch, 528-support-twisted-4.patch, 528-twisted-support.patch
>
>
> Currently, the Python port of Avro only supports synchronous blocking networking, adding Twisted-enabled client and server implementations will benefit projects that can't use synchronous I/O but want to use Avro.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-528) Add support for Twisted
Posted by "Jeff Hammerbacher (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12862117#action_12862117 ]
Jeff Hammerbacher commented on AVRO-528:
----------------------------------------
Hey Esteve,
Thanks for the patches! It's good to see you working with Avro. I have been traveling for the past several days, but I'm going to take a look at these soon.
One note: I'd love to get your thoughts on https://issues.apache.org/jira/browse/AVRO-406, in addition to this issue.
Regards,
Jeff
> Add support for Twisted
> -----------------------
>
> Key: AVRO-528
> URL: https://issues.apache.org/jira/browse/AVRO-528
> Project: Avro
> Issue Type: New Feature
> Components: python
> Reporter: Esteve Fernandez
> Attachments: 528-support-twisted-2.patch, 528-support-twisted-3.patch, 528-support-twisted-4.patch, 528-twisted-support.patch
>
>
> Currently, the Python port of Avro only supports synchronous blocking networking, adding Twisted-enabled client and server implementations will benefit projects that can't use synchronous I/O but want to use Avro.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-528) Add support for Twisted
Posted by "Esteve Fernandez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Esteve Fernandez updated AVRO-528:
----------------------------------
Attachment: 528-twisted-support.patch
This patch adds support for client-side networking with Avro using HTTP as the transport. A patch which adds support for Avro servers will follow.
The changes in the patch modify the current Requestor, splitting requests in two phases:
- serializing the request and sendint it
- waiting for a response
these changes don't cause any backwards incompatibiliy, but introduce a new class (BaseRequestor) which delegates to child classes the responsibility of implementing the BaseRequestor#issue_request method
> Add support for Twisted
> -----------------------
>
> Key: AVRO-528
> URL: https://issues.apache.org/jira/browse/AVRO-528
> Project: Avro
> Issue Type: New Feature
> Reporter: Esteve Fernandez
> Attachments: 528-twisted-support.patch
>
>
> Currently, the Python port of Avro only supports synchronous blocking networking, adding Twisted-enabled client and server implementations will benefit projects that can't use synchronous I/O but want to use Avro.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-528) Add support for Twisted
Posted by "Esteve Fernandez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Esteve Fernandez updated AVRO-528:
----------------------------------
Attachment: 528-support-twisted-2.patch
This version of the patch adds a streaming producer for issuing requests (http://twistedmatrix.com/documents/current/api/twisted.web.iweb.IBodyProducer.html) and a server-side implementation of the Avro HTTP transport. The latter is implemented as a Twisted.web resource that can be placed in any segment of a URL hierarhcy, supporting multiple Avro protocols to be served by the same server.
> Add support for Twisted
> -----------------------
>
> Key: AVRO-528
> URL: https://issues.apache.org/jira/browse/AVRO-528
> Project: Avro
> Issue Type: New Feature
> Reporter: Esteve Fernandez
> Attachments: 528-support-twisted-2.patch, 528-twisted-support.patch
>
>
> Currently, the Python port of Avro only supports synchronous blocking networking, adding Twisted-enabled client and server implementations will benefit projects that can't use synchronous I/O but want to use Avro.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-528) Add support for Twisted
Posted by "Esteve Fernandez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Esteve Fernandez updated AVRO-528:
----------------------------------
Status: Patch Available (was: Open)
> Add support for Twisted
> -----------------------
>
> Key: AVRO-528
> URL: https://issues.apache.org/jira/browse/AVRO-528
> Project: Avro
> Issue Type: New Feature
> Components: python
> Reporter: Esteve Fernandez
> Attachments: 528-support-twisted-2.patch, 528-support-twisted-3.patch, 528-twisted-support.patch
>
>
> Currently, the Python port of Avro only supports synchronous blocking networking, adding Twisted-enabled client and server implementations will benefit projects that can't use synchronous I/O but want to use Avro.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.