You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@thrift.apache.org by Stuart Reynolds <st...@gmail.com> on 2014/12/05 00:35:21 UTC
Can separate generated serialization code from generated datatype definitions?
I'm starting to use thrift for an existing Java/Scala project to
provide RPC access to a set of services. One notable feature of Thrift
(compared to other serialization frameworks) is that Thrift appears be
responsible, not only for serialization, but also generating the
(server) data structures. In many cases, this makes it difficult to
use with existing methods of serialization, such as hibernate, that
might employ annotation of types. :-(
While there are workarounds - these are a LOT of work for an existing
project and in some cases leads to additionally marshaling from
Thrift's MyObject type to another same-but-different version of the
same type -- its precisely this boilerplate that what I'd like to
avoid by employing automated code generation.
So, is it possible to separate the Thrift-generated serialization code
from the POJO class definition (leaving my to edit those myself)?
I suspect not because its looks like things are required to inherit
from TBase, though I'm not sure how extensive this dependency is.
(Am also looking at Swift, but am concerned about the overheards due
to reflection).
- Stuart
Re: Can separate generated serialization code from generated datatype definitions?
Posted by Stuart Reynolds <st...@stureynolds.com>.
Thanks for the advice about Hibernate.
On Fri, Dec 5, 2014 at 2:03 PM, Matt Chambers <mv...@me.com> wrote:
>
> Unless something has changed recently, Hibernate is not a serialization framework, its just an ORM.
>
Strictly speaking, unless your database is in the same process,
serialization is a requirement of every ORM.
I don't know of ANY way to get an object out of running process (into
a file, a database, or another process) without serialization. I'm not
being pedantic here -- it seems like a strict general requirement. We
can split semantic hairs, but Hibernate, Jackson, and many other
frameworks use annotations for getting business objects in out out a
Java process.
> @Override
> public ProjectT mapRow(ResultSet rs, int rowNum)
> throws SQLException {
> ProjectT project = new ProjectT();
> project.setId(rs.getInt("pk_project"));
> project.setName(rs.getString("name"));
> project.setActive(rs.getBoolean("active"));
> return project;
> }
... entails a lot of error-prone busywork that would be better automated.
My point is, its a common necessity to serialize/unserialize objects
during communication with MANY different processes, each ultimately
serialized over various protocols --- we deal with with: cassandra,
REST, JSON RPC, Thrift RPC, sql, redis ...., all variously dealing
with the serialization of business objects. I'd rather not have to
visit 5 pieces of code when adding, removing or renaming an object
field. Part of the great benefit of thrift (beyond formalizing an IPC
procotol) is eliminating exactly this kind of manual and error-prone
data marshaling. However, this is (unnecessarily) at the expense of
removing automated data-marshaling benefits for each of the other many
external processes a develop might deal with. It unnecessary because a
cleaner design can separate automated serialization code from
developer controllable class definitions.
Anyhows.... it looks like thrift2swift was exactly the answer I was
looking for, and is working pretty intrusively:
https://github.com/facebook/swift
Generates IDLs from class definitions under my control and plays
nicely with other system ==> huge win for code simplicity and
productivity.
- Stuart
Re: Can separate generated serialization code from generated datatype
definitions?
Posted by Matt Chambers <mv...@me.com>.
I only say it always gets in the way because as a project grows people tend to have to work around it more and more (just my experience) and then it makes the DAOs seem inconsistent. Some use hibernate, some don't, some are 1/2 and 1/2. Just using the JdbcTemplate keeps it consistent, and a bit faster as well.
Unless something has changed recently, Hibernate is not a serialization framework, its just an ORM.
On Dec 5, 2014, at 4:55 PM, Stuart Reynolds <st...@stureynolds.com> wrote:
> Maybe Hibernate's as POS, maybe not. My point is Hibernate's not the
> other serialization framework people want to use in the same project
> as Thrift.
>
> On Fri, Dec 5, 2014 at 10:27 AM, Matt Chambers <mv...@me.com> wrote:
>>
>> Hibernate is always in the way.
>>
>> Usually I just slap an HTTP Servlet up to serve my thrift services, then people can use the thrift jquery client on top of my existing service instance that is already being served out by another thrift server.
>>
>> public class ThriftServlet extends TServlet implements HttpRequestHandler {
>>
>> @Autowired
>> public ThriftServlet(PlowRpcService.Iface service) {
>> super(new PlowRpcService.Processor<PlowRpcService.Iface>(service), new TJSONProtocol.Factory());
>> }
>>
>> @Override
>> public void handleRequest(HttpServletRequest req, HttpServletResponse rsp)
>> throws ServletException, IOException {
>> doPost(req, rsp);
>> }
>> }
>>
>> -Matt
>>
>> On Dec 5, 2014, at 12:35 PM, Stuart Reynolds <st...@stureynolds.com> wrote:
>>
>>> Thanks -- I know about RowMapper (and many other workarounds).
>>> Hibernate is one of many serialization frameworks I work with (I'm
>>> happy with it, for the most part, and go around it when it gets in the
>>> way).
>>>
>>> We also have a REST interface, and would like JSON or XML
>>> serialization of our business object. Here, again, I'd like to
>>> serialize these objects as plain JSON (not thrift JSON which adds
>>> considerable requirements on a REST client). Allowing Thrift exclusive
>>> ownership of the object definitions is a hindrance.
>>>
>>> - Stuart
>>>
>>>
>>> On Fri, Dec 5, 2014 at 8:25 AM, Matt Chambers <mv...@me.com> wrote:
>>>> Correction
>>>>
>>>> public List<ProjectT> getAll() {
>>>> return jdbc.query("SELECT * FROM project", MAPPER);
>>>> }
>>>>
>>>> On Dec 5, 2014, at 11:21 AM, Matt Chambers <mv...@me.com> wrote:
>>>>
>>>>> public List<ProjectT> getAll() {
>>>>> return jdbc.query("SELECT * FROM project");
>>>>> }
>>>>
>>
Re: Can separate generated serialization code from generated datatype definitions?
Posted by Stuart Reynolds <st...@stureynolds.com>.
Maybe Hibernate's as POS, maybe not. My point is Hibernate's not the
other serialization framework people want to use in the same project
as Thrift.
On Fri, Dec 5, 2014 at 10:27 AM, Matt Chambers <mv...@me.com> wrote:
>
> Hibernate is always in the way.
>
> Usually I just slap an HTTP Servlet up to serve my thrift services, then people can use the thrift jquery client on top of my existing service instance that is already being served out by another thrift server.
>
> public class ThriftServlet extends TServlet implements HttpRequestHandler {
>
> @Autowired
> public ThriftServlet(PlowRpcService.Iface service) {
> super(new PlowRpcService.Processor<PlowRpcService.Iface>(service), new TJSONProtocol.Factory());
> }
>
> @Override
> public void handleRequest(HttpServletRequest req, HttpServletResponse rsp)
> throws ServletException, IOException {
> doPost(req, rsp);
> }
> }
>
> -Matt
>
> On Dec 5, 2014, at 12:35 PM, Stuart Reynolds <st...@stureynolds.com> wrote:
>
>> Thanks -- I know about RowMapper (and many other workarounds).
>> Hibernate is one of many serialization frameworks I work with (I'm
>> happy with it, for the most part, and go around it when it gets in the
>> way).
>>
>> We also have a REST interface, and would like JSON or XML
>> serialization of our business object. Here, again, I'd like to
>> serialize these objects as plain JSON (not thrift JSON which adds
>> considerable requirements on a REST client). Allowing Thrift exclusive
>> ownership of the object definitions is a hindrance.
>>
>> - Stuart
>>
>>
>> On Fri, Dec 5, 2014 at 8:25 AM, Matt Chambers <mv...@me.com> wrote:
>>> Correction
>>>
>>> public List<ProjectT> getAll() {
>>> return jdbc.query("SELECT * FROM project", MAPPER);
>>> }
>>>
>>> On Dec 5, 2014, at 11:21 AM, Matt Chambers <mv...@me.com> wrote:
>>>
>>>> public List<ProjectT> getAll() {
>>>> return jdbc.query("SELECT * FROM project");
>>>> }
>>>
>
Re: Can separate generated serialization code from generated datatype
definitions?
Posted by Matt Chambers <mv...@me.com>.
Hibernate is always in the way.
Usually I just slap an HTTP Servlet up to serve my thrift services, then people can use the thrift jquery client on top of my existing service instance that is already being served out by another thrift server.
public class ThriftServlet extends TServlet implements HttpRequestHandler {
@Autowired
public ThriftServlet(PlowRpcService.Iface service) {
super(new PlowRpcService.Processor<PlowRpcService.Iface>(service), new TJSONProtocol.Factory());
}
@Override
public void handleRequest(HttpServletRequest req, HttpServletResponse rsp)
throws ServletException, IOException {
doPost(req, rsp);
}
}
-Matt
On Dec 5, 2014, at 12:35 PM, Stuart Reynolds <st...@stureynolds.com> wrote:
> Thanks -- I know about RowMapper (and many other workarounds).
> Hibernate is one of many serialization frameworks I work with (I'm
> happy with it, for the most part, and go around it when it gets in the
> way).
>
> We also have a REST interface, and would like JSON or XML
> serialization of our business object. Here, again, I'd like to
> serialize these objects as plain JSON (not thrift JSON which adds
> considerable requirements on a REST client). Allowing Thrift exclusive
> ownership of the object definitions is a hindrance.
>
> - Stuart
>
>
> On Fri, Dec 5, 2014 at 8:25 AM, Matt Chambers <mv...@me.com> wrote:
>> Correction
>>
>> public List<ProjectT> getAll() {
>> return jdbc.query("SELECT * FROM project", MAPPER);
>> }
>>
>> On Dec 5, 2014, at 11:21 AM, Matt Chambers <mv...@me.com> wrote:
>>
>>> public List<ProjectT> getAll() {
>>> return jdbc.query("SELECT * FROM project");
>>> }
>>
Re: Can separate generated serialization code from generated datatype definitions?
Posted by Stuart Reynolds <st...@stureynolds.com>.
Thanks -- I know about RowMapper (and many other workarounds).
Hibernate is one of many serialization frameworks I work with (I'm
happy with it, for the most part, and go around it when it gets in the
way).
We also have a REST interface, and would like JSON or XML
serialization of our business object. Here, again, I'd like to
serialize these objects as plain JSON (not thrift JSON which adds
considerable requirements on a REST client). Allowing Thrift exclusive
ownership of the object definitions is a hindrance.
- Stuart
On Fri, Dec 5, 2014 at 8:25 AM, Matt Chambers <mv...@me.com> wrote:
> Correction
>
> public List<ProjectT> getAll() {
> return jdbc.query("SELECT * FROM project", MAPPER);
> }
>
> On Dec 5, 2014, at 11:21 AM, Matt Chambers <mv...@me.com> wrote:
>
>> public List<ProjectT> getAll() {
>> return jdbc.query("SELECT * FROM project");
>> }
>
Re: Can separate generated serialization code from generated datatype
definitions?
Posted by Matt Chambers <mv...@me.com>.
Correction
public List<ProjectT> getAll() {
return jdbc.query("SELECT * FROM project", MAPPER);
}
On Dec 5, 2014, at 11:21 AM, Matt Chambers <mv...@me.com> wrote:
> public List<ProjectT> getAll() {
> return jdbc.query("SELECT * FROM project");
> }
Re: Can separate generated serialization code from generated datatype
definitions?
Posted by Matt Chambers <mv...@me.com>.
You should just ditch Hibernate and if your using Spring just use JDBC template and RowMapper.
Lets say you had a ProjectT struct like?:
struct ProjectT {
1:i32 id,
2:string name,
3:bool active
}
First, in your DAO class you want to accept a DataSource and create the template.
private JdbcTemplate jdbc;
@Autowired
public void setDataSource(DataSource dataSource) {
this.jdbc = new JdbcTemplate(dataSource);
}
Then in your DAO, make a single static RowMapper which will convert a result set into a single object, then you pass that into certain JDBC template methods. (see below)
private static final RowMapper<ProjectT> MAPPER = new RowMapper<ProjectT>() {
@Override
public ProjectT mapRow(ResultSet rs, int rowNum)
throws SQLException {
ProjectT project = new ProjectT();
project.setId(rs.getInt("pk_project"));
project.setName(rs.getString("name"));
project.setActive(rs.getBoolean("active"));
return project;
}
};
Now you can add methods that your service will call into.
# SELECT * used for brevity in these examples.
# Get one of them.
public ProjectT get(int id) {
return jdbc.queryForObject("SELECT * FROM project WHERE pk_project=?", MAPPER, id)
}
# Get all of them.
public List<ProjectT> getAll() {
return jdbc.query("SELECT * FROM project");
}
public List<String> getAllNames() {
return jdbc.queryForList("SELECT name FROM project", String.class);
}
You can use some Aspect Oriented techniques I posed about in the past to easily convert Spring data exceptions to your thrift exception types as well. Once you get setup with this is just a 1 liner to call into the DAO from your service, then a 1 liner to do the query, and a single Aspect class that handles exception conversion. You'll be surprised how fast you can whip out performant services with this method, and its easy to bring in memcache or Google loading caches as needed.
-Matt
On Dec 4, 2014, at 6:35 PM, Stuart Reynolds <st...@gmail.com> wrote:
> I'm starting to use thrift for an existing Java/Scala project to
> provide RPC access to a set of services. One notable feature of Thrift
> (compared to other serialization frameworks) is that Thrift appears be
> responsible, not only for serialization, but also generating the
> (server) data structures. In many cases, this makes it difficult to
> use with existing methods of serialization, such as hibernate, that
> might employ annotation of types. :-(
>
> While there are workarounds - these are a LOT of work for an existing
> project and in some cases leads to additionally marshaling from
> Thrift's MyObject type to another same-but-different version of the
> same type -- its precisely this boilerplate that what I'd like to
> avoid by employing automated code generation.
>
> So, is it possible to separate the Thrift-generated serialization code
> from the POJO class definition (leaving my to edit those myself)?
>
> I suspect not because its looks like things are required to inherit
> from TBase, though I'm not sure how extensive this dependency is.
>
> (Am also looking at Swift, but am concerned about the overheards due
> to reflection).
>
> - Stuart