You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@bigtop.apache.org by Konstantin Boudnik <co...@apache.org> on 2018/10/07 18:07:30 UTC
Bringing AVRO as a dependecy...
Guys,
I want to bounce a question off the team: what would be the best way to bring
Avro dependency into the stack? I don't think it deserves to be a separate
component per se - after all it serves a niche purpose and doesn't stand on
its own.
However, it makes sense to carry around in case an app needs such a
serialization. I've been thinking of perhaps of packing it up as a Hadoop
or HBase dependency, but it doesn't seem like the most straight forward idea,
to be honest. Or may be we should introduce a separate helper package with
serializers and such?
What would be a preferred way from your stand point?
--
With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622
Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.
Re: Bringing AVRO as a dependecy...
Posted by Konstantin Boudnik <co...@apache.org>.
Thanks guys, something along these lines were forming in the back of
my head. Good to see sketchy thinking is sensible.
The use-case is pretty straight-forward: let the applications on top
of the stack to read/write Avro format to HBase (or else) without a
need to bundle the serde dependency in each and every case.
--
With regards,
Konstantin (Cos) Boudnik
2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622
Disclaimer: Opinions expressed in this email are those of the author,
and do not necessarily represent the views of any company the author
might be affiliated with at the moment of writing.
On Mon, Oct 8, 2018 at 11:39 AM, Roman Shaposhnik <ro...@shaposhnik.org> wrote:
> On Sun, Oct 7, 2018 at 11:37 PM Konstantin Boudnik <co...@apache.org> wrote:
>>
>> Guys,
>>
>> I want to bounce a question off the team: what would be the best way to bring
>> Avro dependency into the stack? I don't think it deserves to be a separate
>> component per se - after all it serves a niche purpose and doesn't stand on
>> its own.
>>
>> However, it makes sense to carry around in case an app needs such a
>> serialization. I've been thinking of perhaps of packing it up as a Hadoop
>> or HBase dependency, but it doesn't seem like the most straight forward idea,
>> to be honest. Or may be we should introduce a separate helper package with
>> serializers and such?
>>
>> What would be a preferred way from your stand point?
>
> The way I see this is two-fold:
> 1. it could be useful to have a standalone pacakge with all the jar
> dependencies
> so that any component of Bigtop stack to use that instead of
> whatever comes through
> Maven dependencies. This will be similar to how we standardize on
> zookeper deps
> and such.
>
> 2. it could be useful to provide binary scripts in that package that
> would automate/facilitate
> some of these kinds of steps:
> http://avro.apache.org/docs/current/gettingstartedjava.html#Compiling+the+schema
> http://avro.apache.org/docs/current/gettingstartedpython.html
> http://avro.apache.org/docs/current/idl.html
> and anything like that.
>
> Do you have any other usecases in mind, Cos?
>
> Thanks,
> Roman.
Re: Bringing AVRO as a dependecy...
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Sun, Oct 7, 2018 at 11:37 PM Konstantin Boudnik <co...@apache.org> wrote:
>
> Guys,
>
> I want to bounce a question off the team: what would be the best way to bring
> Avro dependency into the stack? I don't think it deserves to be a separate
> component per se - after all it serves a niche purpose and doesn't stand on
> its own.
>
> However, it makes sense to carry around in case an app needs such a
> serialization. I've been thinking of perhaps of packing it up as a Hadoop
> or HBase dependency, but it doesn't seem like the most straight forward idea,
> to be honest. Or may be we should introduce a separate helper package with
> serializers and such?
>
> What would be a preferred way from your stand point?
The way I see this is two-fold:
1. it could be useful to have a standalone pacakge with all the jar
dependencies
so that any component of Bigtop stack to use that instead of
whatever comes through
Maven dependencies. This will be similar to how we standardize on
zookeper deps
and such.
2. it could be useful to provide binary scripts in that package that
would automate/facilitate
some of these kinds of steps:
http://avro.apache.org/docs/current/gettingstartedjava.html#Compiling+the+schema
http://avro.apache.org/docs/current/gettingstartedpython.html
http://avro.apache.org/docs/current/idl.html
and anything like that.
Do you have any other usecases in mind, Cos?
Thanks,
Roman.
Re: Bringing AVRO as a dependecy...
Posted by Evans Ye <ev...@apache.org>.
A serde package seems to be a right choice for me.
But I'm gonna ask in another angle: What's your use case? If we have a
specific use case to support, we have better chance to keep it on track
from release to release.
Konstantin Boudnik <co...@apache.org> 於 2018年10月8日 週一 上午2:07寫道：
> Guys,
>
> I want to bounce a question off the team: what would be the best way to
> bring
> Avro dependency into the stack? I don't think it deserves to be a separate
> component per se - after all it serves a niche purpose and doesn't stand on
> its own.
>
> However, it makes sense to carry around in case an app needs such a
> serialization. I've been thinking of perhaps of packing it up as a Hadoop
> or HBase dependency, but it doesn't seem like the most straight forward
> idea,
> to be honest. Or may be we should introduce a separate helper package with
> serializers and such?
>
> What would be a preferred way from your stand point?
> --
> With regards,
> Konstantin (Cos) Boudnik
> 2CAC 8312 4870 D885 8616 6115 220F 6980 1F27 E622
>
> Disclaimer: Opinions expressed in this email are those of the author,
> and do not necessarily represent the views of any company the author
> might be affiliated with at the moment of writing.
>