You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by ZhangJian He <sh...@gmail.com> on 2021/11/15 14:51:35 UTC
[DISCUSS] Add Pulsar io Pulsar connector
Dear all
My team are suggesting some of our customers use pulsar instead of kafka
for their needs.
Before, my team used a pulsar-io-kafka-connector, now my team wants to use
a pulsar-io-to-pulsar-connector server for these customers.
And I notice now we don't have a pulsar-io-pulsar-connector.
Should I develop a connector?
And should the connector be maintained in the pulsar main repo ?
IMO, if we dicided to develop a pulsar-io-connector, it's more reasonable
to maintain it in the pulsar main repo. (At least, the
pulsar-io-kafka-connector is in main repo)
Looking forward to your opinions.
Thanks
ZhangJian He
Re: [DISCUSS] Add Pulsar io Pulsar connector
Posted by Lan Liang <li...@163.com>.
Actually, we really have same case in production.
We have not use pulsar function or pulsar io, just implemented it from ourself code.
Best Regards,
Lan Liang
On 11/18/2021 04:22，Neng Lu<nl...@apache.org> wrote：
@sijie the case here might be tricky. They may want to move data across
pulsar clusters operated by different org or teams.
Remember we previously added the ability to send messages to external
pulsar clusters for pulsar function but got reverted. I think this is the
case they are trying to tackle.
On Wed, Nov 17, 2021 at 10:29 AM Sijie Guo <gu...@gmail.com> wrote:
I don't think you need a separate connector.
An identity function should be able to do the job for you.
- Sijie
On Mon, Nov 15, 2021 at 3:34 PM Neng Lu <nl...@apache.org> wrote:
Just did a quick search, it's interesting we don't have a pulsar
connector
to move data among pulsar clusters.
I guess people usually write their own pulsar client to move data around.
On Mon, Nov 15, 2021 at 3:11 PM ZhangJian He <sh...@gmail.com> wrote:
Yes, move data across different pulsar clusters which belongs to
different
company or organization
Thanks
ZhangJian He
Neng Lu <nl...@apache.org> 于2021年11月16日周二 上午2:50写道：
Hi,
What's your new connector used for in the customer use cases?
A `pulsar-io-kafak-connector` is used for moving data between kafka
and
pulsar.
But in your case, a `pulsar-io-pulsar-connector`, do you mean you
want
to
move data across different pulsar clusters?
On Mon, Nov 15, 2021 at 6:51 AM ZhangJian He <sh...@gmail.com>
wrote:
Dear all
My team are suggesting some of our customers use pulsar instead of
kafka
for their needs.
Before, my team used a pulsar-io-kafka-connector, now my team wants
to
use
a pulsar-io-to-pulsar-connector server for these customers.
And I notice now we don't have a pulsar-io-pulsar-connector.
Should I develop a connector?
And should the connector be maintained in the pulsar main repo ?
IMO, if we dicided to develop a pulsar-io-connector, it's more
reasonable
to maintain it in the pulsar main repo. (At least, the
pulsar-io-kafka-connector is in main repo)
Looking forward to your opinions.
Thanks
ZhangJian He
Re: [DISCUSS] Add Pulsar io Pulsar connector
Posted by Neng Lu <nl...@apache.org>.
@sijie the case here might be tricky. They may want to move data across
pulsar clusters operated by different org or teams.
Remember we previously added the ability to send messages to external
pulsar clusters for pulsar function but got reverted. I think this is the
case they are trying to tackle.
On Wed, Nov 17, 2021 at 10:29 AM Sijie Guo <gu...@gmail.com> wrote:
> I don't think you need a separate connector.
>
> An identity function should be able to do the job for you.
>
> - Sijie
>
> On Mon, Nov 15, 2021 at 3:34 PM Neng Lu <nl...@apache.org> wrote:
>
> > Just did a quick search, it's interesting we don't have a pulsar
> connector
> > to move data among pulsar clusters.
> > I guess people usually write their own pulsar client to move data around.
> >
> >
> > On Mon, Nov 15, 2021 at 3:11 PM ZhangJian He <sh...@gmail.com> wrote:
> >
> > > Yes, move data across different pulsar clusters which belongs to
> > different
> > > company or organization
> > >
> > > Thanks
> > > ZhangJian He
> > >
> > > Neng Lu <nl...@apache.org> 于2021年11月16日周二 上午2:50写道：
> > >
> > > > Hi,
> > > >
> > > > What's your new connector used for in the customer use cases?
> > > > A `pulsar-io-kafak-connector` is used for moving data between kafka
> and
> > > > pulsar.
> > > > But in your case, a `pulsar-io-pulsar-connector`, do you mean you
> want
> > to
> > > > move data across different pulsar clusters?
> > > >
> > > >
> > > > On Mon, Nov 15, 2021 at 6:51 AM ZhangJian He <sh...@gmail.com>
> > wrote:
> > > >
> > > > > Dear all
> > > > >
> > > > > My team are suggesting some of our customers use pulsar instead of
> > > kafka
> > > > > for their needs.
> > > > > Before, my team used a pulsar-io-kafka-connector, now my team wants
> > to
> > > > use
> > > > > a pulsar-io-to-pulsar-connector server for these customers.
> > > > >
> > > > > And I notice now we don't have a pulsar-io-pulsar-connector.
> > > > >
> > > > > Should I develop a connector?
> > > > > And should the connector be maintained in the pulsar main repo ?
> > > > >
> > > > > IMO, if we dicided to develop a pulsar-io-connector, it's more
> > > reasonable
> > > > > to maintain it in the pulsar main repo. (At least, the
> > > > > pulsar-io-kafka-connector is in main repo)
> > > > >
> > > > > Looking forward to your opinions.
> > > > >
> > > > >
> > > > > Thanks
> > > > > ZhangJian He
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Add Pulsar io Pulsar connector
Posted by Sijie Guo <gu...@gmail.com>.
I don't think you need a separate connector.
An identity function should be able to do the job for you.
- Sijie
On Mon, Nov 15, 2021 at 3:34 PM Neng Lu <nl...@apache.org> wrote:
> Just did a quick search, it's interesting we don't have a pulsar connector
> to move data among pulsar clusters.
> I guess people usually write their own pulsar client to move data around.
>
>
> On Mon, Nov 15, 2021 at 3:11 PM ZhangJian He <sh...@gmail.com> wrote:
>
> > Yes, move data across different pulsar clusters which belongs to
> different
> > company or organization
> >
> > Thanks
> > ZhangJian He
> >
> > Neng Lu <nl...@apache.org> 于2021年11月16日周二 上午2:50写道：
> >
> > > Hi,
> > >
> > > What's your new connector used for in the customer use cases?
> > > A `pulsar-io-kafak-connector` is used for moving data between kafka and
> > > pulsar.
> > > But in your case, a `pulsar-io-pulsar-connector`, do you mean you want
> to
> > > move data across different pulsar clusters?
> > >
> > >
> > > On Mon, Nov 15, 2021 at 6:51 AM ZhangJian He <sh...@gmail.com>
> wrote:
> > >
> > > > Dear all
> > > >
> > > > My team are suggesting some of our customers use pulsar instead of
> > kafka
> > > > for their needs.
> > > > Before, my team used a pulsar-io-kafka-connector, now my team wants
> to
> > > use
> > > > a pulsar-io-to-pulsar-connector server for these customers.
> > > >
> > > > And I notice now we don't have a pulsar-io-pulsar-connector.
> > > >
> > > > Should I develop a connector?
> > > > And should the connector be maintained in the pulsar main repo ?
> > > >
> > > > IMO, if we dicided to develop a pulsar-io-connector, it's more
> > reasonable
> > > > to maintain it in the pulsar main repo. (At least, the
> > > > pulsar-io-kafka-connector is in main repo)
> > > >
> > > > Looking forward to your opinions.
> > > >
> > > >
> > > > Thanks
> > > > ZhangJian He
> > > >
> > >
> >
>
Re: [DISCUSS] Add Pulsar io Pulsar connector
Posted by Neng Lu <nl...@apache.org>.
Just did a quick search, it's interesting we don't have a pulsar connector
to move data among pulsar clusters.
I guess people usually write their own pulsar client to move data around.
On Mon, Nov 15, 2021 at 3:11 PM ZhangJian He <sh...@gmail.com> wrote:
> Yes, move data across different pulsar clusters which belongs to different
> company or organization
>
> Thanks
> ZhangJian He
>
> Neng Lu <nl...@apache.org> 于2021年11月16日周二 上午2:50写道：
>
> > Hi,
> >
> > What's your new connector used for in the customer use cases?
> > A `pulsar-io-kafak-connector` is used for moving data between kafka and
> > pulsar.
> > But in your case, a `pulsar-io-pulsar-connector`, do you mean you want to
> > move data across different pulsar clusters?
> >
> >
> > On Mon, Nov 15, 2021 at 6:51 AM ZhangJian He <sh...@gmail.com> wrote:
> >
> > > Dear all
> > >
> > > My team are suggesting some of our customers use pulsar instead of
> kafka
> > > for their needs.
> > > Before, my team used a pulsar-io-kafka-connector, now my team wants to
> > use
> > > a pulsar-io-to-pulsar-connector server for these customers.
> > >
> > > And I notice now we don't have a pulsar-io-pulsar-connector.
> > >
> > > Should I develop a connector?
> > > And should the connector be maintained in the pulsar main repo ?
> > >
> > > IMO, if we dicided to develop a pulsar-io-connector, it's more
> reasonable
> > > to maintain it in the pulsar main repo. (At least, the
> > > pulsar-io-kafka-connector is in main repo)
> > >
> > > Looking forward to your opinions.
> > >
> > >
> > > Thanks
> > > ZhangJian He
> > >
> >
>
Re: [DISCUSS] Add Pulsar io Pulsar connector
Posted by ZhangJian He <sh...@gmail.com>.
Yes, move data across different pulsar clusters which belongs to different
company or organization
Thanks
ZhangJian He
Neng Lu <nl...@apache.org> 于2021年11月16日周二 上午2:50写道：
> Hi,
>
> What's your new connector used for in the customer use cases?
> A `pulsar-io-kafak-connector` is used for moving data between kafka and
> pulsar.
> But in your case, a `pulsar-io-pulsar-connector`, do you mean you want to
> move data across different pulsar clusters?
>
>
> On Mon, Nov 15, 2021 at 6:51 AM ZhangJian He <sh...@gmail.com> wrote:
>
> > Dear all
> >
> > My team are suggesting some of our customers use pulsar instead of kafka
> > for their needs.
> > Before, my team used a pulsar-io-kafka-connector, now my team wants to
> use
> > a pulsar-io-to-pulsar-connector server for these customers.
> >
> > And I notice now we don't have a pulsar-io-pulsar-connector.
> >
> > Should I develop a connector?
> > And should the connector be maintained in the pulsar main repo ?
> >
> > IMO, if we dicided to develop a pulsar-io-connector, it's more reasonable
> > to maintain it in the pulsar main repo. (At least, the
> > pulsar-io-kafka-connector is in main repo)
> >
> > Looking forward to your opinions.
> >
> >
> > Thanks
> > ZhangJian He
> >
>
Re: [DISCUSS] Add Pulsar io Pulsar connector
Posted by Neng Lu <nl...@apache.org>.
Hi,
What's your new connector used for in the customer use cases?
A `pulsar-io-kafak-connector` is used for moving data between kafka and
pulsar.
But in your case, a `pulsar-io-pulsar-connector`, do you mean you want to
move data across different pulsar clusters?
On Mon, Nov 15, 2021 at 6:51 AM ZhangJian He <sh...@gmail.com> wrote:
> Dear all
>
> My team are suggesting some of our customers use pulsar instead of kafka
> for their needs.
> Before, my team used a pulsar-io-kafka-connector, now my team wants to use
> a pulsar-io-to-pulsar-connector server for these customers.
>
> And I notice now we don't have a pulsar-io-pulsar-connector.
>
> Should I develop a connector?
> And should the connector be maintained in the pulsar main repo ?
>
> IMO, if we dicided to develop a pulsar-io-connector, it's more reasonable
> to maintain it in the pulsar main repo. (At least, the
> pulsar-io-kafka-connector is in main repo)
>
> Looking forward to your opinions.
>
>
> Thanks
> ZhangJian He
>