You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Michael Glaesemann (JIRA)" <ji...@apache.org> on 2010/08/25 21:12:16 UTC
[jira] Created: (AVRO-627) Add PHP Avro library
Add PHP Avro library
--------------------
Key: AVRO-627
URL: https://issues.apache.org/jira/browse/AVRO-627
Project: Avro
Issue Type: New Feature
Reporter: Michael Glaesemann
I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
of using Avro at myYearbook.com. It's based on the Python and Ruby
libraries. Current functionality includes schema parsing, Avro data file and
string IO.
Known issues:
- Support for field defaults is incomplete.
- Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-627) Add PHP Avro library
Posted by "Michael Glaesemann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Glaesemann updated AVRO-627:
------------------------------------
Attachment: AVRO-627.patch.2.zip
I set up an Ubuntu VM and it looks like the failures are due to stricter PHP error checking the Ubuntu install (E_ALL) relative to the Mac OS X install (E_ALL & ~E_NOTICE). I've tightened up the code and all the tests pass for me on both Mac OS X and Ubuntu. Updated patch attached.
> Add PHP Avro library
> --------------------
>
> Key: AVRO-627
> URL: https://issues.apache.org/jira/browse/AVRO-627
> Project: Avro
> Issue Type: New Feature
> Components: php
> Reporter: Michael Glaesemann
> Assignee: Michael Glaesemann
> Fix For: 1.4.0
>
> Attachments: AVRO-627.patch.2.zip, AVRO-627.patch.zip
>
>
> I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
> of using Avro at myYearbook.com. It's based on the Python and Ruby
> libraries. Current functionality includes schema parsing, Avro data file and
> string IO.
> Known issues:
> - Support for field defaults is incomplete.
> - Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-627) Add PHP Avro library
Posted by "Michael Glaesemann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Glaesemann updated AVRO-627:
------------------------------------
Attachment: AVRO-627.patch.zip
This includes updating the root build.sh script to
* run the PHP unit tests
* generate an interop avro file for PHP
* test PHP interop, and
* build a PHP distribution package.
The distribution and interop file generation have no additional dependencies (other than PHP). The tests require PHPUnit <http://www.phpunit.de/>
> Add PHP Avro library
> --------------------
>
> Key: AVRO-627
> URL: https://issues.apache.org/jira/browse/AVRO-627
> Project: Avro
> Issue Type: New Feature
> Reporter: Michael Glaesemann
> Attachments: AVRO-627.patch.zip
>
>
> I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
> of using Avro at myYearbook.com. It's based on the Python and Ruby
> libraries. Current functionality includes schema parsing, Avro data file and
> string IO.
> Known issues:
> - Support for field defaults is incomplete.
> - Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-627) Add PHP Avro library
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902981#action_12902981 ]
Doug Cutting commented on AVRO-627:
-----------------------------------
Now tests fail with "This is not a 64-bit platform". It looks like PHP only supports 32-bit integers on 32-bit hardware like my laptop.
Might we use gmp for longs? From my reading, a PHP float is a C double, while a PHP int is a C long, which is 32- or 64-bits, depending on the architecture.
> Add PHP Avro library
> --------------------
>
> Key: AVRO-627
> URL: https://issues.apache.org/jira/browse/AVRO-627
> Project: Avro
> Issue Type: New Feature
> Components: php
> Reporter: Michael Glaesemann
> Assignee: Michael Glaesemann
> Fix For: 1.4.0
>
> Attachments: AVRO-627.patch.2.zip, AVRO-627.patch.3.zip, AVRO-627.patch.zip
>
>
> I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
> of using Avro at myYearbook.com. It's based on the Python and Ruby
> libraries. Current functionality includes schema parsing, Avro data file and
> string IO.
> Known issues:
> - Support for field defaults is incomplete.
> - Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-627) Add PHP Avro library
Posted by "Michael Glaesemann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902609#action_12902609 ]
Michael Glaesemann commented on AVRO-627:
-----------------------------------------
Thanks for trying this out so quickly.
I'll take a look at these. All the tests pass for me on Mac OS X 10.6.4, PHP 5.3.3RC3 (cli) (built: Aug 19 2010 13:58:08).
Are you running the stock Ubuntu 10.04 PHP?
> Add PHP Avro library
> --------------------
>
> Key: AVRO-627
> URL: https://issues.apache.org/jira/browse/AVRO-627
> Project: Avro
> Issue Type: New Feature
> Components: php
> Reporter: Michael Glaesemann
> Assignee: Michael Glaesemann
> Fix For: 1.4.0
>
> Attachments: AVRO-627.patch.zip
>
>
> I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
> of using Avro at myYearbook.com. It's based on the Python and Ruby
> libraries. Current functionality includes schema parsing, Avro data file and
> string IO.
> Known issues:
> - Support for field defaults is incomplete.
> - Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-627) Add PHP Avro library
Posted by "Michael Glaesemann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903039#action_12903039 ]
Michael Glaesemann commented on AVRO-627:
-----------------------------------------
That makes sense. I'll take a look.
> Add PHP Avro library
> --------------------
>
> Key: AVRO-627
> URL: https://issues.apache.org/jira/browse/AVRO-627
> Project: Avro
> Issue Type: New Feature
> Components: php
> Reporter: Michael Glaesemann
> Assignee: Michael Glaesemann
> Fix For: 1.4.0
>
> Attachments: AVRO-627.patch.2.zip, AVRO-627.patch.3.zip, AVRO-627.patch.zip
>
>
> I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
> of using Avro at myYearbook.com. It's based on the Python and Ruby
> libraries. Current functionality includes schema parsing, Avro data file and
> string IO.
> Known issues:
> - Support for field defaults is incomplete.
> - Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-627) Add PHP Avro library
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902613#action_12902613 ]
Doug Cutting commented on AVRO-627:
-----------------------------------
> Are you running the stock Ubuntu 10.04 PHP?
Yes. I didn't previously have PHP installed and got whatever 'sudo apt-get install phpunit' installed:
{code}
Setting up php5-common (5.3.2-1ubuntu4.2) ...
Setting up php5-cli (5.3.2-1ubuntu4.2) ...
Creating config file /etc/php5/cli/php.ini with new version
update-alternatives: using /usr/bin/php5 to provide /usr/bin/php (php) in auto mode.
Setting up php-pear (5.3.2-1ubuntu4.2) ...
Setting up php-benchmark (1.2.7-4) ...
Setting up phpunit (3.4.5-1) ...
{code}
> Add PHP Avro library
> --------------------
>
> Key: AVRO-627
> URL: https://issues.apache.org/jira/browse/AVRO-627
> Project: Avro
> Issue Type: New Feature
> Components: php
> Reporter: Michael Glaesemann
> Assignee: Michael Glaesemann
> Fix For: 1.4.0
>
> Attachments: AVRO-627.patch.zip
>
>
> I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
> of using Avro at myYearbook.com. It's based on the Python and Ruby
> libraries. Current functionality includes schema parsing, Avro data file and
> string IO.
> Known issues:
> - Support for field defaults is incomplete.
> - Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-627) Add PHP Avro library
Posted by "Michael Glaesemann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12904253#action_12904253 ]
Michael Glaesemann commented on AVRO-627:
-----------------------------------------
Greate! Thanks for your help, Doug!
> Add PHP Avro library
> --------------------
>
> Key: AVRO-627
> URL: https://issues.apache.org/jira/browse/AVRO-627
> Project: Avro
> Issue Type: New Feature
> Components: php
> Reporter: Michael Glaesemann
> Assignee: Michael Glaesemann
> Fix For: 1.4.0
>
> Attachments: AVRO-627.patch.2.zip, AVRO-627.patch.3.zip, AVRO-627.patch.4.zip, AVRO-627.patch.zip
>
>
> I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
> of using Avro at myYearbook.com. It's based on the Python and Ruby
> libraries. Current functionality includes schema parsing, Avro data file and
> string IO.
> Known issues:
> - Support for field defaults is incomplete.
> - Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AVRO-627) Add PHP Avro library
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting resolved AVRO-627.
-------------------------------
Hadoop Flags: [Reviewed]
Fix Version/s: 1.4.0
(was: 1.4.1)
Resolution: Fixed
I committed this. Thanks, Michael!
I made a few changes to the build.sh dist implementation, namely:
- fixed broken README copy
- added NOTICE & LICENSE
- stopped copying .svn files from examples
- moved tarball from ../../build to ../../dist
> Add PHP Avro library
> --------------------
>
> Key: AVRO-627
> URL: https://issues.apache.org/jira/browse/AVRO-627
> Project: Avro
> Issue Type: New Feature
> Components: php
> Reporter: Michael Glaesemann
> Assignee: Michael Glaesemann
> Fix For: 1.4.0
>
> Attachments: AVRO-627.patch.2.zip, AVRO-627.patch.3.zip, AVRO-627.patch.4.zip, AVRO-627.patch.zip
>
>
> I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
> of using Avro at myYearbook.com. It's based on the Python and Ruby
> libraries. Current functionality includes schema parsing, Avro data file and
> string IO.
> Known issues:
> - Support for field defaults is incomplete.
> - Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-627) Add PHP Avro library
Posted by "Michael Glaesemann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903022#action_12903022 ]
Michael Glaesemann commented on AVRO-627:
-----------------------------------------
I put the 64-bit check in deliberately. As I understand it, the {{json_decode}} function in current releases of PHP casts JSON long values to floats on 32-bit platforms. It looks like a future release will allow representing longs as strings on 32-bit via a [JSON_BIGINT_AS_STRING option|http://www.php.net/manual/en/function.json-decode.php], but it's not available now, unfortunately. So even if I did my PHP bit-twiddling using the GMP library, I wouldn't be able to get the actual value out of the JSON, unless I parsed the JSON myself.
I'll build a 32-bit VM and see if I can figure out something.
Another option would be to start from scratch and build the PHP lib using the C library as an extension, though when I first took a look at that, it didn't look feasible.
Any other ideas? I'm open to suggestions.
> Add PHP Avro library
> --------------------
>
> Key: AVRO-627
> URL: https://issues.apache.org/jira/browse/AVRO-627
> Project: Avro
> Issue Type: New Feature
> Components: php
> Reporter: Michael Glaesemann
> Assignee: Michael Glaesemann
> Fix For: 1.4.0
>
> Attachments: AVRO-627.patch.2.zip, AVRO-627.patch.3.zip, AVRO-627.patch.zip
>
>
> I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
> of using Avro at myYearbook.com. It's based on the Python and Ruby
> libraries. Current functionality includes schema parsing, Avro data file and
> string IO.
> Known issues:
> - Support for field defaults is incomplete.
> - Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-627) Add PHP Avro library
Posted by "Michael Glaesemann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Glaesemann updated AVRO-627:
------------------------------------
Attachment: AVRO-627.patch.4.zip
Updated to use GMP PHP extension on 32-bit platforms when it's available, raising an exception if it's not available. 64-bit platforms use the native PHP operations even if GMP is available. Tested on Mac OS X 64-bit, Ubuntu 64-bit (without GMP), Ubuntu 32-bit with GMP, and CentOS 5 64-bit.
Assuming things look good, any chance of this getting rolled into the 1.4.0? I know you've already cut the RC, but I thought I'd ask anyway.
> Add PHP Avro library
> --------------------
>
> Key: AVRO-627
> URL: https://issues.apache.org/jira/browse/AVRO-627
> Project: Avro
> Issue Type: New Feature
> Components: php
> Reporter: Michael Glaesemann
> Assignee: Michael Glaesemann
> Fix For: 1.4.1
>
> Attachments: AVRO-627.patch.2.zip, AVRO-627.patch.3.zip, AVRO-627.patch.4.zip, AVRO-627.patch.zip
>
>
> I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
> of using Avro at myYearbook.com. It's based on the Python and Ruby
> libraries. Current functionality includes schema parsing, Avro data file and
> string IO.
> Known issues:
> - Support for field defaults is incomplete.
> - Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-627) Add PHP Avro library
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-627:
------------------------------
Fix Version/s: 1.4.1
(was: 1.4.0)
> Add PHP Avro library
> --------------------
>
> Key: AVRO-627
> URL: https://issues.apache.org/jira/browse/AVRO-627
> Project: Avro
> Issue Type: New Feature
> Components: php
> Reporter: Michael Glaesemann
> Assignee: Michael Glaesemann
> Fix For: 1.4.1
>
> Attachments: AVRO-627.patch.2.zip, AVRO-627.patch.3.zip, AVRO-627.patch.zip
>
>
> I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
> of using Avro at myYearbook.com. It's based on the Python and Ruby
> libraries. Current functionality includes schema parsing, Avro data file and
> string IO.
> Known issues:
> - Support for field defaults is incomplete.
> - Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-627) Add PHP Avro library
Posted by "Michael Glaesemann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Glaesemann updated AVRO-627:
------------------------------------
Attachment: AVRO-627.patch.3.zip
AVRO-627.patch.3.zip adds support for defaults and supersedes all previous patches.
> Add PHP Avro library
> --------------------
>
> Key: AVRO-627
> URL: https://issues.apache.org/jira/browse/AVRO-627
> Project: Avro
> Issue Type: New Feature
> Components: php
> Reporter: Michael Glaesemann
> Assignee: Michael Glaesemann
> Fix For: 1.4.0
>
> Attachments: AVRO-627.patch.2.zip, AVRO-627.patch.3.zip, AVRO-627.patch.zip
>
>
> I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
> of using Avro at myYearbook.com. It's based on the Python and Ruby
> libraries. Current functionality includes schema parsing, Avro data file and
> string IO.
> Known issues:
> - Support for field defaults is incomplete.
> - Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AVRO-627) Add PHP Avro library
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated AVRO-627:
------------------------------
Assignee: Michael Glaesemann
Fix Version/s: 1.4.0
Component/s: php
This looks great!
I installed phpunit on Ubuntu 10.04 and was able to run tests.
We should also update the top-level README.txt, listing PHP requirements.
Tests however failed with:
{code}
There were 2 failures:
1) SchemaTest::test_parse with data set #38 (SchemaExample)
schema_string:
{"type": "array",
"items": {"type": "enum", "name": "Test", "symbols": ["A", "B"]}}
Sub-schema for AvroArraySchema not a valid Avro schema. Bad schema: Array
(
[type] => enum
[name] => Test
[symbols] => Array
(
[0] => A
[1] => B
)
)
Failed asserting that <boolean:true> is false.
/home/cutting/src/avro/trunk/lang/php/test/SchemaTest.php:459
2) SchemaTest::test_parse with data set #40 (SchemaExample)
schema_string:
{"type": "map",
"values": {"type": "enum", "name": "Test", "symbols": ["A", "B"]}}
Sub-schema for AvroMapSchema not a valid Avro schema. Bad schema: Array
(
[type] => enum
[name] => Test
[symbols] => Array
(
[0] => A
[1] => B
)
)
Failed asserting that <boolean:true> is false.
/home/cutting/src/avro/trunk/lang/php/test/SchemaTest.php:459
FAILURES!
Tests: 189, Assertions: 261, Failures: 2, Errors: 89, Incomplete: 3.
{code}
Any idea what this could be?
It'd be great to include this in 1.4.0, which I hope to get out in the next few days.
> Add PHP Avro library
> --------------------
>
> Key: AVRO-627
> URL: https://issues.apache.org/jira/browse/AVRO-627
> Project: Avro
> Issue Type: New Feature
> Components: php
> Reporter: Michael Glaesemann
> Assignee: Michael Glaesemann
> Fix For: 1.4.0
>
> Attachments: AVRO-627.patch.zip
>
>
> I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
> of using Avro at myYearbook.com. It's based on the Python and Ruby
> libraries. Current functionality includes schema parsing, Avro data file and
> string IO.
> Known issues:
> - Support for field defaults is incomplete.
> - Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AVRO-627) Add PHP Avro library
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AVRO-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903038#action_12903038 ]
Doug Cutting commented on AVRO-627:
-----------------------------------
> the json_decode function in current releases of PHP casts JSON long values to floats on 32-bit platforms.
The only longs in JSON should be default values, no? The lack of the ability to accept large default values for longs would be a bug, but probably not many would encounter it. The value of longs in datastructures could use gmp, no? Or am I missing something?
> Add PHP Avro library
> --------------------
>
> Key: AVRO-627
> URL: https://issues.apache.org/jira/browse/AVRO-627
> Project: Avro
> Issue Type: New Feature
> Components: php
> Reporter: Michael Glaesemann
> Assignee: Michael Glaesemann
> Fix For: 1.4.0
>
> Attachments: AVRO-627.patch.2.zip, AVRO-627.patch.3.zip, AVRO-627.patch.zip
>
>
> I've developed a PHP library for Avro against the Avro 1.3.3 spec as part
> of using Avro at myYearbook.com. It's based on the Python and Ruby
> libraries. Current functionality includes schema parsing, Avro data file and
> string IO.
> Known issues:
> - Support for field defaults is incomplete.
> - Lacks RPC and protocol support, JSON encoding, and deflate codec.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.