You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Roger Meier (JIRA)" <ji...@apache.org> on 2010/08/11 21:47:16 UTC
[jira] Created: (THRIFT-847) Test Framework harmonization across
all languages
Test Framework harmonization across all languages
-------------------------------------------------
Key: THRIFT-847
URL: https://issues.apache.org/jira/browse/THRIFT-847
Project: Thrift
Issue Type: Improvement
Components: Library (C#), Library (C++), Library (Cocoa), Library (Erlang), Library (Haskell), Library (Java), Library (JavaScript), Library (OCaml), Library (Perl), Library (PHP), Library (Python), Library (Ruby), Test Suite
Affects Versions: 0.3, 0.2, 0.1
Reporter: Roger Meier
Fix For: 0.4
Today each Language supported by Thrift, have its own unit test, all are using the same Thrift IDL's located at the test directory. But the behavior of these tests seems to be different from language to language... this makes it difficult to do tests and bug fixing across different Languages. e.g.
* C++ Test and JavaScript Test Server written in Java have different responses for the same services
* C# and Java Test Server have different responses for testException as C++
I propose the following steps:
* identify the language with the reference implementation (well defined return values for all test cases)
* update the ThriftTest.thrift with details about the required return values that have to be implemented
* update test implementations and move language tests into their appropriate library directory (THRIFT-35)
* a public test server that supports multiple protocols and transports could be another enhancement for testing purposes
I'm ready to help preparing patches, just tell me what you need!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (THRIFT-847) Test Framework harmonization
across all languages
Posted by Christian Lavoie <ch...@gmail.com>.
On Tue, Oct 12, 2010 at 8:40 PM, <ro...@bufferoverflow.ch> wrote:
> ThriftTest.thrift and existing TestServer and Clients are not that important
> for me, but they already contains some tests with Base Types and additional
> in deep tests with mapmap etc. It might be a starting point.
It might, but my fear was if I were to change anything in ThriftTest
(or any of the existing tests) I risked breaking bindings I have no
way to test currently.
> One thing I like to avoid is tons of .thrift files with similar
> functionality inside test...=> difficult to maintain. If it makes sense, to
> add new ones, why not.
Agreed.
> It might be worth to create a Ticket or TestSuite Wiki Page.
> So we can do brainstroming and/or define the TestSuite and the IDL files.
> e.g.
>
> - SimpleTypes.thrift (i16, i32)
> - StringTypes.thrfit (utf-8, etc.)
> - ComplexTypes.thrift
> - Exceptions.Thrift
> - ...
>
> what can we reuse from today's IDL files? which IDL files will become
> obsolete?
I'd like to get a good grip on where we want to go first, to help
decide what we can reuse.
> after defining the TestSuite in detail we can start implement that for the
> different Languages and do interoperability tests, score card, etc.
Agreed.
> another important topic might be recommendations like using that
> datatype/feature results in incompatibility with blabla
Yep; plenty of similar nice-to-haves fall out from a good test suite :)
Thanks for the feedback Roger, much appreciated!
>
> Quoting Christian Lavoie <ch...@gmail.com>:
>
>> On Tue, Oct 12, 2010 at 4:44 PM, Roger Meier (JIRA) <ji...@apache.org>
>> wrote:
>>>
>>> I had the following things in mind:
>>> # identify the language with the reference implementation (well
>>> defined return values for all test cases) based on ThriftTest.thrift,
>>> probably Java (lib/java/test/org/apache/thrift/server/ServerTestBase.java)
>>> or cpp (test/cpp/src/TestClient.cpp test/cpp/src/TestServer.cpp)
>>
>> I had part of that discussion with Bryan on IRC -- turns out neither
>> cpp nor Java are complete implementations, so there's no such thing as
>> 'a' reference (yet?). Part of my hope here is to map out the full
>> description of 'who implements what'. That's something I had in mind
>> when I said 'advanced' features and score cards.
>>
>>> # update the ThriftTest.thrift with details about the required return
>>> values that have to be implemented
>>
>> So, the problem with ThriftTest as it stands is that it's a big mess
>> of large and small things, that not all languages implement on an
>> equal basis. I'm not even sure that all languages _can_ implement it
>> all on an equal basis (see below).
>>
>> I'd like to either prune ThriftTest (or rewrite it with Simple, at
>> some point the distinction between the two will be fairly academic, I
>> suspect) down to something that is the clear minimum requirement for
>> viable bindings. The reason I went with Simple instead of ThriftTest
>> was that I could work without breaking tests for languages I can't
>> even compile on my test machines, yet.
>>
>>> # update test implementations
>>> # add some wrapper scripts to automate the cross language tests (as
>>> Christian did)
>>
>> Actually, I'm hoping to get rid of my scripts once I find time to go
>> through the autotools machinery once again; this was just the most
>> expedient way to get the discussion going (and the c++ client bug
>> reported).
>>
>>> # move language tests into their appropriate library directory
>>> (THRIFT-35)
>>
>> That's orthogonal to doing actual cross-language testing, but I agree
>> needs to happen. It's more awkward once we start doing cross-language
>> testing though.
>>
>>> # a public test server that supports multiple protocols and transports
>>> could be another enhancement for testing purposes
>>
>> That's a good idea, but entirely orthogonal. Once that's said and
>> done, it shouldn't be too hard to extend Simple as it stands to just
>> run forever.
>>
>>> Today neary every language has already Test Server and Test Clients based
>>> on ThriftTest.thrift , we just have to modify them a bit so that they have
>>> exactly the same behavior. I think it is not worth to add another
>>> SimpleTest.thrift, the target should be extending ThriftTest.thrift.
>>
>> This is where we disagree. I'm far from convinced all language are
>> even capable of implementing a 'perfect enough' copy of the reference
>> bindings. The first case I tripped over was the bit-length of integral
>> types. Java, C++ and Haskell all support precise-lengths integrals...
>> but Java only supports signed types (you can abuse them to represent
>> unsigned values if you try hard enough).
>>
>> Perl doesn't particularly care about length of numbers; Ruby (I don't
>> know ruby very well) seems to only support one size Fixnums (machine
>> size, whatever that is on your machine -- how do we handle this, on
>> some machines Fixnum maps to i32, on others to i64?); Python doesn't
>> support i16 at all, only plain integers and long integers... and I'm
>> not sure if they support byte either (python chars, IIRC are unicode
>> characters, 32 bits long).
>>
>> Unicode is another fun topic -- not all versions of Python are
>> completely unicode clean (though I doubt thrift supports python old
>> enough for that to matter). I'm fairly sure at some point someone will
>> provide bindings for a language where utf-8 support is going to be a
>> major pain (VHDL anyone?) -- JavaScript comes to mind, if memory
>> serves it treats strings as 16-bit chars, which isn't enough for all
>> of unicode.
>>
>> Then we go in complex sets -- PHP doesn't support complex objects
>> (sets, maps, etc) as map keys, but C++, Java and Haskell do no
>> problem. Does that go in ThriftTest or not? It can't, otherwise PHP
>> can't be tested. But if it's not, I can't test C++, Java and Haskell
>> interoperability in its entirety.
>>
>> It's a mess once you get in the details of it, that's why I suggested
>> the score card approach.
>>
>>>
>>>> Test Framework harmonization across all languages
>>>> -------------------------------------------------
>>>>
>>>> Key: THRIFT-847
>>>> URL: https://issues.apache.org/jira/browse/THRIFT-847
>>>> Project: Thrift
>>>> Issue Type: Improvement
>>>> Components: C# - Library, C++ - Library, Cocoa - Library,
>>>> Erlang - Library, Haskell - Library, Java - Library, JavaScript - Library,
>>>> OCaml - Library, Perl - Library, PHP - Library, Python - Library, Ruby -
>>>> Library, Test Suite
>>>> Affects Versions: 0.1, 0.2, 0.3
>>>> Reporter: Roger Meier
>>>> Assignee: Christian Lavoie
>>>> Fix For: 0.6
>>>>
>>>> Attachments: v1-WORK_IN_PROGRESS-unified_tests.tar.gz
>>>>
>>>>
>>>> Today each Language supported by Thrift, have its own unit test, all are
>>>> using the same Thrift IDL's located at the test directory. But the behavior
>>>> of these tests seems to be different from language to language... this makes
>>>> it difficult to do tests and bug fixing across different Languages. e.g.
>>>> * C++ Test and JavaScript Test Server written in Java have different
>>>> responses for the same services
>>>> * C# and Java Test Server have different responses for testException as
>>>> C++
>>>> I propose the following steps:
>>>> * identify the language with the reference implementation (well defined
>>>> return values for all test cases)
>>>> * update the ThriftTest.thrift with details about the required return
>>>> values that have to be implemented
>>>> * update test implementations and move language tests into their
>>>> appropriate library directory (THRIFT-35)
>>>> * a public test server that supports multiple protocols and transports
>>>> could be another enhancement for testing purposes
>>>> I'm ready to help preparing patches, just tell me what you need!
>>>
>>> --
>>> This message is automatically generated by JIRA.
>>> -
>>> You can reply to this email to add a comment to the issue online.
>>>
>>>
>>
>>
>>
>> --
>> Have fun, Christian
>> http://linkedin.christianlavoie.net
>>
>> "I won't let you fall apart."
>>
>>
>>
>
>
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>
--
Have fun, Christian
http://linkedin.christianlavoie.net
"I won't let you fall apart."
Re: [jira] Commented: (THRIFT-847) Test Framework harmonization
across all languages
Posted by ro...@bufferoverflow.ch.
ThriftTest.thrift and existing TestServer and Clients are not that
important for me, but they already contains some tests with Base Types
and additional in deep tests with mapmap etc. It might be a starting
point.
One thing I like to avoid is tons of .thrift files with similar
functionality inside test...=> difficult to maintain. If it makes
sense, to add new ones, why not.
It might be worth to create a Ticket or TestSuite Wiki Page.
So we can do brainstroming and/or define the TestSuite and the IDL files. e.g.
- SimpleTypes.thrift (i16, i32)
- StringTypes.thrfit (utf-8, etc.)
- ComplexTypes.thrift
- Exceptions.Thrift
- ...
what can we reuse from today's IDL files? which IDL files will become
obsolete?
after defining the TestSuite in detail we can start implement that for
the different Languages and do interoperability tests, score card, etc.
another important topic might be recommendations like using that
datatype/feature results in incompatibility with blabla
Quoting Christian Lavoie <ch...@gmail.com>:
> On Tue, Oct 12, 2010 at 4:44 PM, Roger Meier (JIRA) <ji...@apache.org> wrote:
>>
>> I had the following things in mind:
>> # identify the language with the reference implementation (well
>> defined return values for all test cases) based on
>> ThriftTest.thrift, probably Java
>> (lib/java/test/org/apache/thrift/server/ServerTestBase.java) or cpp
>> (test/cpp/src/TestClient.cpp test/cpp/src/TestServer.cpp)
>
> I had part of that discussion with Bryan on IRC -- turns out neither
> cpp nor Java are complete implementations, so there's no such thing as
> 'a' reference (yet?). Part of my hope here is to map out the full
> description of 'who implements what'. That's something I had in mind
> when I said 'advanced' features and score cards.
>
>> # update the ThriftTest.thrift with details about the required
>> return values that have to be implemented
>
> So, the problem with ThriftTest as it stands is that it's a big mess
> of large and small things, that not all languages implement on an
> equal basis. I'm not even sure that all languages _can_ implement it
> all on an equal basis (see below).
>
> I'd like to either prune ThriftTest (or rewrite it with Simple, at
> some point the distinction between the two will be fairly academic, I
> suspect) down to something that is the clear minimum requirement for
> viable bindings. The reason I went with Simple instead of ThriftTest
> was that I could work without breaking tests for languages I can't
> even compile on my test machines, yet.
>
>> # update test implementations
>> # add some wrapper scripts to automate the cross language tests
>> (as Christian did)
>
> Actually, I'm hoping to get rid of my scripts once I find time to go
> through the autotools machinery once again; this was just the most
> expedient way to get the discussion going (and the c++ client bug
> reported).
>
>> # move language tests into their appropriate library directory (THRIFT-35)
>
> That's orthogonal to doing actual cross-language testing, but I agree
> needs to happen. It's more awkward once we start doing cross-language
> testing though.
>
>> # a public test server that supports multiple protocols and
>> transports could be another enhancement for testing purposes
>
> That's a good idea, but entirely orthogonal. Once that's said and
> done, it shouldn't be too hard to extend Simple as it stands to just
> run forever.
>
>> Today neary every language has already Test Server and Test Clients
>> based on ThriftTest.thrift , we just have to modify them a bit so
>> that they have exactly the same behavior. I think it is not worth
>> to add another SimpleTest.thrift, the target should be extending
>> ThriftTest.thrift.
>
> This is where we disagree. I'm far from convinced all language are
> even capable of implementing a 'perfect enough' copy of the reference
> bindings. The first case I tripped over was the bit-length of integral
> types. Java, C++ and Haskell all support precise-lengths integrals...
> but Java only supports signed types (you can abuse them to represent
> unsigned values if you try hard enough).
>
> Perl doesn't particularly care about length of numbers; Ruby (I don't
> know ruby very well) seems to only support one size Fixnums (machine
> size, whatever that is on your machine -- how do we handle this, on
> some machines Fixnum maps to i32, on others to i64?); Python doesn't
> support i16 at all, only plain integers and long integers... and I'm
> not sure if they support byte either (python chars, IIRC are unicode
> characters, 32 bits long).
>
> Unicode is another fun topic -- not all versions of Python are
> completely unicode clean (though I doubt thrift supports python old
> enough for that to matter). I'm fairly sure at some point someone will
> provide bindings for a language where utf-8 support is going to be a
> major pain (VHDL anyone?) -- JavaScript comes to mind, if memory
> serves it treats strings as 16-bit chars, which isn't enough for all
> of unicode.
>
> Then we go in complex sets -- PHP doesn't support complex objects
> (sets, maps, etc) as map keys, but C++, Java and Haskell do no
> problem. Does that go in ThriftTest or not? It can't, otherwise PHP
> can't be tested. But if it's not, I can't test C++, Java and Haskell
> interoperability in its entirety.
>
> It's a mess once you get in the details of it, that's why I suggested
> the score card approach.
>
>>
>>> Test Framework harmonization across all languages
>>> -------------------------------------------------
>>>
>>> Key: THRIFT-847
>>> URL: https://issues.apache.org/jira/browse/THRIFT-847
>>> Project: Thrift
>>> Issue Type: Improvement
>>> Components: C# - Library, C++ - Library, Cocoa - Library,
>>> Erlang - Library, Haskell - Library, Java - Library, JavaScript -
>>> Library, OCaml - Library, Perl - Library, PHP - Library, Python -
>>> Library, Ruby - Library, Test Suite
>>> Affects Versions: 0.1, 0.2, 0.3
>>> Reporter: Roger Meier
>>> Assignee: Christian Lavoie
>>> Fix For: 0.6
>>>
>>> Attachments: v1-WORK_IN_PROGRESS-unified_tests.tar.gz
>>>
>>>
>>> Today each Language supported by Thrift, have its own unit test,
>>> all are using the same Thrift IDL's located at the test directory.
>>> But the behavior of these tests seems to be different from
>>> language to language... this makes it difficult to do tests and
>>> bug fixing across different Languages. e.g.
>>> * C++ Test and JavaScript Test Server written in Java have
>>> different responses for the same services
>>> * C# and Java Test Server have different responses for testException as C++
>>> I propose the following steps:
>>> * identify the language with the reference implementation (well
>>> defined return values for all test cases)
>>> * update the ThriftTest.thrift with details about the required
>>> return values that have to be implemented
>>> * update test implementations and move language tests into their
>>> appropriate library directory (THRIFT-35)
>>> * a public test server that supports multiple protocols and
>>> transports could be another enhancement for testing purposes
>>> I'm ready to help preparing patches, just tell me what you need!
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>
>
>
> --
> Have fun, Christian
> http://linkedin.christianlavoie.net
>
> "I won't let you fall apart."
>
>
>
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
Re: [jira] Commented: (THRIFT-847) Test Framework harmonization
across all languages
Posted by Christian Lavoie <ch...@gmail.com>.
On Tue, Oct 12, 2010 at 4:44 PM, Roger Meier (JIRA) <ji...@apache.org> wrote:
>
> I had the following things in mind:
> # identify the language with the reference implementation (well defined return values for all test cases) based on ThriftTest.thrift, probably Java (lib/java/test/org/apache/thrift/server/ServerTestBase.java) or cpp (test/cpp/src/TestClient.cpp test/cpp/src/TestServer.cpp)
I had part of that discussion with Bryan on IRC -- turns out neither
cpp nor Java are complete implementations, so there's no such thing as
'a' reference (yet?). Part of my hope here is to map out the full
description of 'who implements what'. That's something I had in mind
when I said 'advanced' features and score cards.
> # update the ThriftTest.thrift with details about the required return values that have to be implemented
So, the problem with ThriftTest as it stands is that it's a big mess
of large and small things, that not all languages implement on an
equal basis. I'm not even sure that all languages _can_ implement it
all on an equal basis (see below).
I'd like to either prune ThriftTest (or rewrite it with Simple, at
some point the distinction between the two will be fairly academic, I
suspect) down to something that is the clear minimum requirement for
viable bindings. The reason I went with Simple instead of ThriftTest
was that I could work without breaking tests for languages I can't
even compile on my test machines, yet.
> # update test implementations
> # add some wrapper scripts to automate the cross language tests (as Christian did)
Actually, I'm hoping to get rid of my scripts once I find time to go
through the autotools machinery once again; this was just the most
expedient way to get the discussion going (and the c++ client bug
reported).
> # move language tests into their appropriate library directory (THRIFT-35)
That's orthogonal to doing actual cross-language testing, but I agree
needs to happen. It's more awkward once we start doing cross-language
testing though.
> # a public test server that supports multiple protocols and transports could be another enhancement for testing purposes
That's a good idea, but entirely orthogonal. Once that's said and
done, it shouldn't be too hard to extend Simple as it stands to just
run forever.
> Today neary every language has already Test Server and Test Clients based on ThriftTest.thrift , we just have to modify them a bit so that they have exactly the same behavior. I think it is not worth to add another SimpleTest.thrift, the target should be extending ThriftTest.thrift.
This is where we disagree. I'm far from convinced all language are
even capable of implementing a 'perfect enough' copy of the reference
bindings. The first case I tripped over was the bit-length of integral
types. Java, C++ and Haskell all support precise-lengths integrals...
but Java only supports signed types (you can abuse them to represent
unsigned values if you try hard enough).
Perl doesn't particularly care about length of numbers; Ruby (I don't
know ruby very well) seems to only support one size Fixnums (machine
size, whatever that is on your machine -- how do we handle this, on
some machines Fixnum maps to i32, on others to i64?); Python doesn't
support i16 at all, only plain integers and long integers... and I'm
not sure if they support byte either (python chars, IIRC are unicode
characters, 32 bits long).
Unicode is another fun topic -- not all versions of Python are
completely unicode clean (though I doubt thrift supports python old
enough for that to matter). I'm fairly sure at some point someone will
provide bindings for a language where utf-8 support is going to be a
major pain (VHDL anyone?) -- JavaScript comes to mind, if memory
serves it treats strings as 16-bit chars, which isn't enough for all
of unicode.
Then we go in complex sets -- PHP doesn't support complex objects
(sets, maps, etc) as map keys, but C++, Java and Haskell do no
problem. Does that go in ThriftTest or not? It can't, otherwise PHP
can't be tested. But if it's not, I can't test C++, Java and Haskell
interoperability in its entirety.
It's a mess once you get in the details of it, that's why I suggested
the score card approach.
>
>> Test Framework harmonization across all languages
>> -------------------------------------------------
>>
>> Key: THRIFT-847
>> URL: https://issues.apache.org/jira/browse/THRIFT-847
>> Project: Thrift
>> Issue Type: Improvement
>> Components: C# - Library, C++ - Library, Cocoa - Library, Erlang - Library, Haskell - Library, Java - Library, JavaScript - Library, OCaml - Library, Perl - Library, PHP - Library, Python - Library, Ruby - Library, Test Suite
>> Affects Versions: 0.1, 0.2, 0.3
>> Reporter: Roger Meier
>> Assignee: Christian Lavoie
>> Fix For: 0.6
>>
>> Attachments: v1-WORK_IN_PROGRESS-unified_tests.tar.gz
>>
>>
>> Today each Language supported by Thrift, have its own unit test, all are using the same Thrift IDL's located at the test directory. But the behavior of these tests seems to be different from language to language... this makes it difficult to do tests and bug fixing across different Languages. e.g.
>> * C++ Test and JavaScript Test Server written in Java have different responses for the same services
>> * C# and Java Test Server have different responses for testException as C++
>> I propose the following steps:
>> * identify the language with the reference implementation (well defined return values for all test cases)
>> * update the ThriftTest.thrift with details about the required return values that have to be implemented
>> * update test implementations and move language tests into their appropriate library directory (THRIFT-35)
>> * a public test server that supports multiple protocols and transports could be another enhancement for testing purposes
>> I'm ready to help preparing patches, just tell me what you need!
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
--
Have fun, Christian
http://linkedin.christianlavoie.net
"I won't let you fall apart."
[jira] Commented: (THRIFT-847) Test Framework harmonization across
all languages
Posted by "Roger Meier (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920138#action_12920138 ]
Roger Meier commented on THRIFT-847:
------------------------------------
Thanks Christian for your inputs.
I had the following things in mind:
# identify the language with the reference implementation (well defined return values for all test cases) based on ThriftTest.thrift, probably Java (lib/java/test/org/apache/thrift/server/ServerTestBase.java) or cpp (test/cpp/src/TestClient.cpp test/cpp/src/TestServer.cpp)
# update the ThriftTest.thrift with details about the required return values that have to be implemented
# update test implementations
# add some wrapper scripts to automate the cross language tests (as Christian did)
# move language tests into their appropriate library directory (THRIFT-35)
# a public test server that supports multiple protocols and transports could be another enhancement for testing purposes
Today neary every language has already Test Server and Test Clients based on ThriftTest.thrift , we just have to modify them a bit so that they have exactly the same behavior. I think it is not worth to add another SimpleTest.thrift, the target should be extending ThriftTest.thrift.
> Test Framework harmonization across all languages
> -------------------------------------------------
>
> Key: THRIFT-847
> URL: https://issues.apache.org/jira/browse/THRIFT-847
> Project: Thrift
> Issue Type: Improvement
> Components: C# - Library, C++ - Library, Cocoa - Library, Erlang - Library, Haskell - Library, Java - Library, JavaScript - Library, OCaml - Library, Perl - Library, PHP - Library, Python - Library, Ruby - Library, Test Suite
> Affects Versions: 0.1, 0.2, 0.3
> Reporter: Roger Meier
> Assignee: Christian Lavoie
> Fix For: 0.6
>
> Attachments: v1-WORK_IN_PROGRESS-unified_tests.tar.gz
>
>
> Today each Language supported by Thrift, have its own unit test, all are using the same Thrift IDL's located at the test directory. But the behavior of these tests seems to be different from language to language... this makes it difficult to do tests and bug fixing across different Languages. e.g.
> * C++ Test and JavaScript Test Server written in Java have different responses for the same services
> * C# and Java Test Server have different responses for testException as C++
> I propose the following steps:
> * identify the language with the reference implementation (well defined return values for all test cases)
> * update the ThriftTest.thrift with details about the required return values that have to be implemented
> * update test implementations and move language tests into their appropriate library directory (THRIFT-35)
> * a public test server that supports multiple protocols and transports could be another enhancement for testing purposes
> I'm ready to help preparing patches, just tell me what you need!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-847) Test Framework harmonization across
all languages
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated THRIFT-847:
---------------------------------
Fix Version/s: 0.6
(was: 0.5)
> Test Framework harmonization across all languages
> -------------------------------------------------
>
> Key: THRIFT-847
> URL: https://issues.apache.org/jira/browse/THRIFT-847
> Project: Thrift
> Issue Type: Improvement
> Components: C# - Library, C++ - Library, Cocoa - Library, Erlang - Library, Haskell - Library, Java - Library, JavaScript - Library, OCaml - Library, Perl - Library, PHP - Library, Python - Library, Ruby - Library, Test Suite
> Affects Versions: 0.1, 0.2, 0.3
> Reporter: Roger Meier
> Fix For: 0.6
>
>
> Today each Language supported by Thrift, have its own unit test, all are using the same Thrift IDL's located at the test directory. But the behavior of these tests seems to be different from language to language... this makes it difficult to do tests and bug fixing across different Languages. e.g.
> * C++ Test and JavaScript Test Server written in Java have different responses for the same services
> * C# and Java Test Server have different responses for testException as C++
> I propose the following steps:
> * identify the language with the reference implementation (well defined return values for all test cases)
> * update the ThriftTest.thrift with details about the required return values that have to be implemented
> * update test implementations and move language tests into their appropriate library directory (THRIFT-35)
> * a public test server that supports multiple protocols and transports could be another enhancement for testing purposes
> I'm ready to help preparing patches, just tell me what you need!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-847) Test Framework harmonization across
all languages
Posted by "Bryan Duxbury (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Duxbury updated THRIFT-847:
---------------------------------
Fix Version/s: 0.5
(was: 0.4)
This is a really cool idea, but I don't think this is going to make it into TRUNK before the 0.4 branch gets cut.
> Test Framework harmonization across all languages
> -------------------------------------------------
>
> Key: THRIFT-847
> URL: https://issues.apache.org/jira/browse/THRIFT-847
> Project: Thrift
> Issue Type: Improvement
> Components: Library (C#), Library (C++), Library (Cocoa), Library (Erlang), Library (Haskell), Library (Java), Library (JavaScript), Library (OCaml), Library (Perl), Library (PHP), Library (Python), Library (Ruby), Test Suite
> Affects Versions: 0.1, 0.2, 0.3
> Reporter: Roger Meier
> Fix For: 0.5
>
>
> Today each Language supported by Thrift, have its own unit test, all are using the same Thrift IDL's located at the test directory. But the behavior of these tests seems to be different from language to language... this makes it difficult to do tests and bug fixing across different Languages. e.g.
> * C++ Test and JavaScript Test Server written in Java have different responses for the same services
> * C# and Java Test Server have different responses for testException as C++
> I propose the following steps:
> * identify the language with the reference implementation (well defined return values for all test cases)
> * update the ThriftTest.thrift with details about the required return values that have to be implemented
> * update test implementations and move language tests into their appropriate library directory (THRIFT-35)
> * a public test server that supports multiple protocols and transports could be another enhancement for testing purposes
> I'm ready to help preparing patches, just tell me what you need!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (THRIFT-847) Test Framework harmonization across
all languages
Posted by "Christian Lavoie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Lavoie reassigned THRIFT-847:
---------------------------------------
Assignee: Christian Lavoie
> Test Framework harmonization across all languages
> -------------------------------------------------
>
> Key: THRIFT-847
> URL: https://issues.apache.org/jira/browse/THRIFT-847
> Project: Thrift
> Issue Type: Improvement
> Components: C# - Library, C++ - Library, Cocoa - Library, Erlang - Library, Haskell - Library, Java - Library, JavaScript - Library, OCaml - Library, Perl - Library, PHP - Library, Python - Library, Ruby - Library, Test Suite
> Affects Versions: 0.1, 0.2, 0.3
> Reporter: Roger Meier
> Assignee: Christian Lavoie
> Fix For: 0.6
>
> Attachments: v1-WORK_IN_PROGRESS-unified_tests.tar.gz
>
>
> Today each Language supported by Thrift, have its own unit test, all are using the same Thrift IDL's located at the test directory. But the behavior of these tests seems to be different from language to language... this makes it difficult to do tests and bug fixing across different Languages. e.g.
> * C++ Test and JavaScript Test Server written in Java have different responses for the same services
> * C# and Java Test Server have different responses for testException as C++
> I propose the following steps:
> * identify the language with the reference implementation (well defined return values for all test cases)
> * update the ThriftTest.thrift with details about the required return values that have to be implemented
> * update test implementations and move language tests into their appropriate library directory (THRIFT-35)
> * a public test server that supports multiple protocols and transports could be another enhancement for testing purposes
> I'm ready to help preparing patches, just tell me what you need!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (THRIFT-847) Test Framework harmonization across
all languages
Posted by "Christian Lavoie (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Christian Lavoie updated THRIFT-847:
------------------------------------
Attachment: v1-WORK_IN_PROGRESS-unified_tests.tar.gz
So, here's a very rough draft of unfinished cross language tests. To run, {{cd}} into a working thrift tree, extract the tar, it'll create a {{crosstest/}} subdir with a few shell scripts:
{noformat}
$ ls
SimpleTest.cpp SimpleTest.thrift java-build.sh
SimpleTest.hs cpp-build.sh java-run.sh
SimpleTest.java hs-build.sh
{noformat}
To build the tests:
{noformat}
$ ./java-build.sh && ./cpp-build.sh && ./hs-build.sh
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
[ 1 of 14] Compiling Thrift.Transport ( ../lib/hs/src/Thrift/Transport.hs, build-hs/Thrift/Transport.o )
[ 2 of 14] Compiling Thrift.Transport.Handle ( ../lib/hs/src/Thrift/Transport/Handle.hs, build-hs/Thrift/Transport/Handle.o )
[ 3 of 14] Compiling Thrift.Protocol ( ../lib/hs/src/Thrift/Protocol.hs, build-hs/Thrift/Protocol.o )
[ 4 of 14] Compiling Thrift.Protocol.Binary ( ../lib/hs/src/Thrift/Protocol/Binary.hs, build-hs/Thrift/Protocol/Binary.o )
[ 5 of 14] Compiling Thrift ( ../lib/hs/src/Thrift.hs, build-hs/Thrift.o )
[ 6 of 14] Compiling Thrift.Server ( ../lib/hs/src/Thrift/Server.hs, build-hs/Thrift/Server.o )
[ 7 of 14] Compiling SimpleTest_Types ( gen-hs/SimpleTest_Types.hs, build-hs/SimpleTest_Types.o )
[ 8 of 14] Compiling Base_Iface ( gen-hs/Base_Iface.hs, build-hs/Base_Iface.o )
[ 9 of 14] Compiling Simple_Iface ( gen-hs/Simple_Iface.hs, build-hs/Simple_Iface.o )
[10 of 14] Compiling Base ( gen-hs/Base.hs, build-hs/Base.o )
[11 of 14] Compiling Simple ( gen-hs/Simple.hs, build-hs/Simple.o )
[12 of 14] Compiling Base_Client ( gen-hs/Base_Client.hs, build-hs/Base_Client.o )
[13 of 14] Compiling Simple_Client ( gen-hs/Simple_Client.hs, build-hs/Simple_Client.o )
[14 of 14] Compiling Main ( SimpleTest.hs, build-hs/Main.o )
Linking hs-test ...
{noformat}
And to run the tests in {{both}} mode (each language is run against itself, but through a TCP socket):
{noformat}
$ ./cpp-simple-test both && ./java-run.sh both && ./hs-test both
Running in both client-and-server mode
Running byte tests
arg: both
Starting server on port 9090 ...
Running byte numeric tests
Running i16 numeric tests
Running i32 numeric tests
Running i64 numeric tests
Running double numeric tests
SUCCESS
Runing in both client and server mode
Testing Simple.addBytes...
{noformat}
You can also run each pair of executables with one in {{server}} and one in {{client}} mode:
{noformat}
$ ./java-run.sh server & sleep 5 && ./cpp-simple-test client && ./hs-test client && sleep 1 && kill $(jobs -p)
[1] 12018
arg: server
Starting server on port 9090 ...
Running in client-only mode
Running byte tests
Runing in client-only mode
Testing Simple.addBytes...
{noformat}
On my Mac machine, I can get all pairs (and all three tests in {{both}} modes) to run with 0.5.0, but NOT with HEAD; the cpp client seems to be broken since dreiss' facebook patches; just guessing here, but the error message is telling:
{noformat}
$ ./hs-test server & sleep 3 && ./cpp-simple-test client
[1] 22775
Runing in server-only mode
Running in client-only mode
Running byte tests
terminate called after throwing an instance of 'apache::thrift::transport::TTransportException'
what(): Base TTransport cannot write.
Abort trap
{noformat}
So, here's my plan, and my RFC:
# Finish the simplest possible tests, and implement a handful of clients (I suspect I'll get perl, haskell, c++ and java done) and a handful of servers (haskell, c++ and java; most likely)
# Get this hooked to the autotools build system and {{make check}}
# Define a set of {{base}} tests: you aren't a proper thrift bindings unless you pass those
# Define different set of {{advanced}} tests: you match specific features exactly as the reference languages do (== java, cpp, I think). In particular, I introduced a bug where Haskell consider {{byte}} to be unsigned, and java/cpp don't. Under this definition, Haskell would fail the advanced tests that check for numerical type implementation details
# Create some sort of score card for each language -- to see how closely they match the reference implementation behaviour
Please comment on the approach, I'd like some feedback before I spend too much time implementing tests nobody will like.
> Test Framework harmonization across all languages
> -------------------------------------------------
>
> Key: THRIFT-847
> URL: https://issues.apache.org/jira/browse/THRIFT-847
> Project: Thrift
> Issue Type: Improvement
> Components: C# - Library, C++ - Library, Cocoa - Library, Erlang - Library, Haskell - Library, Java - Library, JavaScript - Library, OCaml - Library, Perl - Library, PHP - Library, Python - Library, Ruby - Library, Test Suite
> Affects Versions: 0.1, 0.2, 0.3
> Reporter: Roger Meier
> Fix For: 0.6
>
> Attachments: v1-WORK_IN_PROGRESS-unified_tests.tar.gz
>
>
> Today each Language supported by Thrift, have its own unit test, all are using the same Thrift IDL's located at the test directory. But the behavior of these tests seems to be different from language to language... this makes it difficult to do tests and bug fixing across different Languages. e.g.
> * C++ Test and JavaScript Test Server written in Java have different responses for the same services
> * C# and Java Test Server have different responses for testException as C++
> I propose the following steps:
> * identify the language with the reference implementation (well defined return values for all test cases)
> * update the ThriftTest.thrift with details about the required return values that have to be implemented
> * update test implementations and move language tests into their appropriate library directory (THRIFT-35)
> * a public test server that supports multiple protocols and transports could be another enhancement for testing purposes
> I'm ready to help preparing patches, just tell me what you need!
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.