You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@thrift.apache.org by "Louis-Philippe Gauthier (JIRA)" <ji...@apache.org> on 2011/09/05 19:30:09 UTC
[jira] [Created] (THRIFT-1321) Map serialization is broken in the
Erlang library
Map serialization is broken in the Erlang library
-------------------------------------------------
Key: THRIFT-1321
URL: https://issues.apache.org/jira/browse/THRIFT-1321
Project: Thrift
Issue Type: Bug
Components: Erlang - Library
Affects Versions: 0.7
Reporter: Louis-Philippe Gauthier
Attachments: bug.diff
{quote}
diff --git a/lib/erl/src/thrift_protocol.erl b/lib/erl/src/thrift_protocol.erl
index 193b07a..3a97a86 100644
--- a/lib/erl/src/thrift_protocol.erl
+++ b/lib/erl/src/thrift_protocol.erl
@@ -358,7 +358,7 @@ write(Proto0, {{map, KeyType, ValType}, Data}) ->
vtype = term_to_typeid(ValType),
size = dict:size(Data)
}),
- Proto2 = dict:fold(fun(KeyData, ValData, ProtoS0) ->
+ Proto2 = dict:fold(fun(KeyData, [ValData], ProtoS0) ->
{ProtoS1, ok} = write(ProtoS0, {KeyType, KeyData}),
{ProtoS2, ok} = write(ProtoS1, {ValType, ValData}),
ProtoS2
{quote}
dict:fold/3 always return ValData as a list and therefore breaks the pattern matching.
{quote}
Dict = dict:new(),
Dict2 = dict:append("key1", "value1", Dict),
Dict3 = dict:append("key2", [1,2,3], Dict2),
Dict4 = dict:append("key3", <<"value3">>, Dict3),
dict:fold(fun(Key, Value, AccIn) -> io:format("~p : ~p~n", [Key, Value]) end, [], Dict4).
"key1" : ["value1"]
"key2" : [[1,2,3]]
"key3" : [<<"value3">>]
{quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1321) Map serialization is broken in the
Erlang library
Posted by "Anthony Molinaro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107614#comment-13107614 ]
Anthony Molinaro commented on THRIFT-1321:
------------------------------------------
Okay, I see what the issue is here. You are using dict:append/3 which turns an entry in the dictionary into a list of entries. This doesn't actually work with thrift (or it probably does in some odd way). Instead you should be using dict:store/3 or dict:from_list/1 when creating the maps. See if I were to make the change you are suggesting it would actually break serialization completely for the normal cases where you are creating a map. Using append you are not
creating a map (unless you are creating a map<string,list> or something like that at which point I think it might make sense.
Using your first example you should be doing something like
{noformat}
Dict = dict:new(),
Dict2 = dict:store("key1", "value1", Dict),
Dict3 = dict:store("key2", [1,2,3], Dict2),
Dict4 = dict:store("key3", <<"value3">>, Dict3),
dict:fold(fun(Key, Value, AccIn) -> io:format("~p : ~p~n", [Key, Value]) end, [], Dict4).
"key1" : "value1"
"key2" : [1,2,3]
"key3" : <<"value3">>
{noformat}
> Map serialization is broken in the Erlang library
> -------------------------------------------------
>
> Key: THRIFT-1321
> URL: https://issues.apache.org/jira/browse/THRIFT-1321
> Project: Thrift
> Issue Type: Bug
> Components: Erlang - Library
> Affects Versions: 0.7
> Reporter: Louis-Philippe Gauthier
> Attachments: bug.diff
>
>
> dict:fold/3 always return ValData as a list and therefore breaks the pattern matching (guards).
> {quote}
> Dict = dict:new(),
> Dict2 = dict:append("key1", "value1", Dict),
> Dict3 = dict:append("key2", [1,2,3], Dict2),
> Dict4 = dict:append("key3", <<"value3">>, Dict3),
> dict:fold(fun(Key, Value, AccIn) -> io:format("~p : ~p~n", [Key, Value]) end, [], Dict4).
> "key1" : ["value1"]
> "key2" : [[1,2,3]]
> "key3" : [<<"value3">>]
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1321) Map serialization is broken in the
Erlang library
Posted by "Anthony Molinaro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107349#comment-13107349 ]
Anthony Molinaro commented on THRIFT-1321:
------------------------------------------
I see how that could be the case, but I'd like to reproduce this error with a simple case, starting with a thrift definition and doing serialize/deserialize (like in the snippet I included before). I used binary values in the snippet and things seemed to work, so I'm still not certain how this manifests itself.
Do you have a test which shows the bug happening? Otherwise I can try to figure out a way to reproduce it sometime soon.
> Map serialization is broken in the Erlang library
> -------------------------------------------------
>
> Key: THRIFT-1321
> URL: https://issues.apache.org/jira/browse/THRIFT-1321
> Project: Thrift
> Issue Type: Bug
> Components: Erlang - Library
> Affects Versions: 0.7
> Reporter: Louis-Philippe Gauthier
> Attachments: bug.diff
>
>
> dict:fold/3 always return ValData as a list and therefore breaks the pattern matching (guards).
> {quote}
> Dict = dict:new(),
> Dict2 = dict:append("key1", "value1", Dict),
> Dict3 = dict:append("key2", [1,2,3], Dict2),
> Dict4 = dict:append("key3", <<"value3">>, Dict3),
> dict:fold(fun(Key, Value, AccIn) -> io:format("~p : ~p~n", [Key, Value]) end, [], Dict4).
> "key1" : ["value1"]
> "key2" : [[1,2,3]]
> "key3" : [<<"value3">>]
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (THRIFT-1321) Map serialization is broken in the
Erlang library
Posted by "Louis-Philippe Gauthier (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Louis-Philippe Gauthier updated THRIFT-1321:
--------------------------------------------
Description:
dict:fold/3 always return ValData as a list and therefore breaks the pattern matching (guards).
{quote}
Dict = dict:new(),
Dict2 = dict:append("key1", "value1", Dict),
Dict3 = dict:append("key2", [1,2,3], Dict2),
Dict4 = dict:append("key3", <<"value3">>, Dict3),
dict:fold(fun(Key, Value, AccIn) -> io:format("~p : ~p~n", [Key, Value]) end, [], Dict4).
"key1" : ["value1"]
"key2" : [[1,2,3]]
"key3" : [<<"value3">>]
{quote}
was:
dict:fold/3 always return ValData as a list and therefore breaks the pattern matching.
{quote}
Dict = dict:new(),
Dict2 = dict:append("key1", "value1", Dict),
Dict3 = dict:append("key2", [1,2,3], Dict2),
Dict4 = dict:append("key3", <<"value3">>, Dict3),
dict:fold(fun(Key, Value, AccIn) -> io:format("~p : ~p~n", [Key, Value]) end, [], Dict4).
"key1" : ["value1"]
"key2" : [[1,2,3]]
"key3" : [<<"value3">>]
{quote}
> Map serialization is broken in the Erlang library
> -------------------------------------------------
>
> Key: THRIFT-1321
> URL: https://issues.apache.org/jira/browse/THRIFT-1321
> Project: Thrift
> Issue Type: Bug
> Components: Erlang - Library
> Affects Versions: 0.7
> Reporter: Louis-Philippe Gauthier
> Attachments: bug.diff
>
>
> dict:fold/3 always return ValData as a list and therefore breaks the pattern matching (guards).
> {quote}
> Dict = dict:new(),
> Dict2 = dict:append("key1", "value1", Dict),
> Dict3 = dict:append("key2", [1,2,3], Dict2),
> Dict4 = dict:append("key3", <<"value3">>, Dict3),
> dict:fold(fun(Key, Value, AccIn) -> io:format("~p : ~p~n", [Key, Value]) end, [], Dict4).
> "key1" : ["value1"]
> "key2" : [[1,2,3]]
> "key3" : [<<"value3">>]
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1321) Map serialization is broken in the
Erlang library
Posted by "Louis-Philippe Gauthier (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104523#comment-13104523 ]
Louis-Philippe Gauthier commented on THRIFT-1321:
-------------------------------------------------
It's broken if you use binary values...
Proto2 = dict:fold(fun(KeyData, ValData, ProtoS0) ->
{ProtoS1, ok} = write(ProtoS0, {KeyType, KeyData}),
{ProtoS2, ok} = write(ProtoS1, {ValType, ValData}),
ProtoS2
end,
Proto1,
Data),
dict:fold/3 always return ValData as a list...
And therefore the the guard (in thrift_binary_protocol) always match the first one...
write(This0, {string, Str}) when is_list(Str) ->
{This1, ok} = write(This0, {i32, length(Str)}),
{This2, ok} = write(This1, list_to_binary(Str)),
{This2, ok};
write(This0, {string, Bin}) when is_binary(Bin) ->
{This1, ok} = write(This0, {i32, size(Bin)}),
{This2, ok} = write(This1, Bin),
{This2, ok};
> Map serialization is broken in the Erlang library
> -------------------------------------------------
>
> Key: THRIFT-1321
> URL: https://issues.apache.org/jira/browse/THRIFT-1321
> Project: Thrift
> Issue Type: Bug
> Components: Erlang - Library
> Affects Versions: 0.7
> Reporter: Louis-Philippe Gauthier
> Attachments: bug.diff
>
>
> dict:fold/3 always return ValData as a list and therefore breaks the pattern matching (guards).
> {quote}
> Dict = dict:new(),
> Dict2 = dict:append("key1", "value1", Dict),
> Dict3 = dict:append("key2", [1,2,3], Dict2),
> Dict4 = dict:append("key3", <<"value3">>, Dict3),
> dict:fold(fun(Key, Value, AccIn) -> io:format("~p : ~p~n", [Key, Value]) end, [], Dict4).
> "key1" : ["value1"]
> "key2" : [[1,2,3]]
> "key3" : [<<"value3">>]
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (THRIFT-1321) Map serialization is broken in the
Erlang library
Posted by "Anthony Molinaro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Anthony Molinaro resolved THRIFT-1321.
--------------------------------------
Resolution: Not A Problem
Not actually a bug, see my last comment.
> Map serialization is broken in the Erlang library
> -------------------------------------------------
>
> Key: THRIFT-1321
> URL: https://issues.apache.org/jira/browse/THRIFT-1321
> Project: Thrift
> Issue Type: Bug
> Components: Erlang - Library
> Affects Versions: 0.7
> Reporter: Louis-Philippe Gauthier
> Attachments: bug.diff
>
>
> dict:fold/3 always return ValData as a list and therefore breaks the pattern matching (guards).
> {quote}
> Dict = dict:new(),
> Dict2 = dict:append("key1", "value1", Dict),
> Dict3 = dict:append("key2", [1,2,3], Dict2),
> Dict4 = dict:append("key3", <<"value3">>, Dict3),
> dict:fold(fun(Key, Value, AccIn) -> io:format("~p : ~p~n", [Key, Value]) end, [], Dict4).
> "key1" : ["value1"]
> "key2" : [[1,2,3]]
> "key3" : [<<"value3">>]
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (THRIFT-1321) Map serialization is broken in the
Erlang library
Posted by "Louis-Philippe Gauthier (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Louis-Philippe Gauthier updated THRIFT-1321:
--------------------------------------------
Description:
dict:fold/3 always return ValData as a list and therefore breaks the pattern matching.
{quote}
Dict = dict:new(),
Dict2 = dict:append("key1", "value1", Dict),
Dict3 = dict:append("key2", [1,2,3], Dict2),
Dict4 = dict:append("key3", <<"value3">>, Dict3),
dict:fold(fun(Key, Value, AccIn) -> io:format("~p : ~p~n", [Key, Value]) end, [], Dict4).
"key1" : ["value1"]
"key2" : [[1,2,3]]
"key3" : [<<"value3">>]
{quote}
was:
{quote}
diff --git a/lib/erl/src/thrift_protocol.erl b/lib/erl/src/thrift_protocol.erl
index 193b07a..3a97a86 100644
--- a/lib/erl/src/thrift_protocol.erl
+++ b/lib/erl/src/thrift_protocol.erl
@@ -358,7 +358,7 @@ write(Proto0, {{map, KeyType, ValType}, Data}) ->
vtype = term_to_typeid(ValType),
size = dict:size(Data)
}),
- Proto2 = dict:fold(fun(KeyData, ValData, ProtoS0) ->
+ Proto2 = dict:fold(fun(KeyData, [ValData], ProtoS0) ->
{ProtoS1, ok} = write(ProtoS0, {KeyType, KeyData}),
{ProtoS2, ok} = write(ProtoS1, {ValType, ValData}),
ProtoS2
{quote}
dict:fold/3 always return ValData as a list and therefore breaks the pattern matching.
{quote}
Dict = dict:new(),
Dict2 = dict:append("key1", "value1", Dict),
Dict3 = dict:append("key2", [1,2,3], Dict2),
Dict4 = dict:append("key3", <<"value3">>, Dict3),
dict:fold(fun(Key, Value, AccIn) -> io:format("~p : ~p~n", [Key, Value]) end, [], Dict4).
"key1" : ["value1"]
"key2" : [[1,2,3]]
"key3" : [<<"value3">>]
{quote}
> Map serialization is broken in the Erlang library
> -------------------------------------------------
>
> Key: THRIFT-1321
> URL: https://issues.apache.org/jira/browse/THRIFT-1321
> Project: Thrift
> Issue Type: Bug
> Components: Erlang - Library
> Affects Versions: 0.7
> Reporter: Louis-Philippe Gauthier
> Attachments: bug.diff
>
>
> dict:fold/3 always return ValData as a list and therefore breaks the pattern matching.
> {quote}
> Dict = dict:new(),
> Dict2 = dict:append("key1", "value1", Dict),
> Dict3 = dict:append("key2", [1,2,3], Dict2),
> Dict4 = dict:append("key3", <<"value3">>, Dict3),
> dict:fold(fun(Key, Value, AccIn) -> io:format("~p : ~p~n", [Key, Value]) end, [], Dict4).
> "key1" : ["value1"]
> "key2" : [[1,2,3]]
> "key3" : [<<"value3">>]
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (THRIFT-1321) Map serialization is broken in the
Erlang library
Posted by "Anthony Molinaro (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097743#comment-13097743 ]
Anthony Molinaro commented on THRIFT-1321:
------------------------------------------
I'm not seeing how this is a bug, I put together this to test
{noformat}
struct A { 1: map<string, string> x }
{noformat}
{noformat}
-module(test_thrift_1321).
-include("thrift1321_types.hrl").
-ifdef(TEST).
-include_lib("eunit/include/eunit.hrl").
broken_map_serialization_test() ->
S = #a { x = dict:from_list ([{<<"foo">>,<<"bar">>},{<<"baz">>,<<"bob">>}]) },
{ok, T0} = thrift_memory_buffer:new (),
{ok, P0} = thrift_binary_protocol:new (T0),
{P1, ok} = thrift_protocol:write (P0,
{{struct, element(2, thrift1321_types:struct_info(a))}, S}),
{_, {ok, R}} = thrift_protocol:read (P1,
{struct, element(2, thrift1321_types:struct_info(a))}, a),
?assertEqual (S, R).
-endif.
{noformat}
And the map serializes and deserializes fine. Do you have a test case of this not working? I pretty extensively use map to talk erlang to java and java to erlang so if this were a bug I would expect to have seen it before, but then again I only really use map<string,string> so maybe that case works where others fail? Anyway, if you have an example that would be best, thanks.
> Map serialization is broken in the Erlang library
> -------------------------------------------------
>
> Key: THRIFT-1321
> URL: https://issues.apache.org/jira/browse/THRIFT-1321
> Project: Thrift
> Issue Type: Bug
> Components: Erlang - Library
> Affects Versions: 0.7
> Reporter: Louis-Philippe Gauthier
> Attachments: bug.diff
>
>
> dict:fold/3 always return ValData as a list and therefore breaks the pattern matching (guards).
> {quote}
> Dict = dict:new(),
> Dict2 = dict:append("key1", "value1", Dict),
> Dict3 = dict:append("key2", [1,2,3], Dict2),
> Dict4 = dict:append("key3", <<"value3">>, Dict3),
> dict:fold(fun(Key, Value, AccIn) -> io:format("~p : ~p~n", [Key, Value]) end, [], Dict4).
> "key1" : ["value1"]
> "key2" : [[1,2,3]]
> "key3" : [<<"value3">>]
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (THRIFT-1321) Map serialization is broken in the
Erlang library
Posted by "Louis-Philippe Gauthier (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/THRIFT-1321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Louis-Philippe Gauthier updated THRIFT-1321:
--------------------------------------------
Attachment: bug.diff
> Map serialization is broken in the Erlang library
> -------------------------------------------------
>
> Key: THRIFT-1321
> URL: https://issues.apache.org/jira/browse/THRIFT-1321
> Project: Thrift
> Issue Type: Bug
> Components: Erlang - Library
> Affects Versions: 0.7
> Reporter: Louis-Philippe Gauthier
> Attachments: bug.diff
>
>
> dict:fold/3 always return ValData as a list and therefore breaks the pattern matching.
> {quote}
> Dict = dict:new(),
> Dict2 = dict:append("key1", "value1", Dict),
> Dict3 = dict:append("key2", [1,2,3], Dict2),
> Dict4 = dict:append("key3", <<"value3">>, Dict3),
> dict:fold(fun(Key, Value, AccIn) -> io:format("~p : ~p~n", [Key, Value]) end, [], Dict4).
> "key1" : ["value1"]
> "key2" : [[1,2,3]]
> "key3" : [<<"value3">>]
> {quote}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira