You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jack Lund <ja...@braintreepayments.com> on 2016/05/31 15:43:19 UTC
Resetting the Offset of a Kafka Sink Connector
I'm trying to use the Connector API to write data to a backing store (HDFS
for now, but probably something like S3 later) for potential replay back
into Kafka later. However, I can't seem to find how to reset the offsets
for the SinkConnector.
I've found the rewind() function on the WorkerSinkTask:
https://github.com/apache/kafka/blob/c9485b78a6e43747daf1314ae9532839fb7bc810/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L396
However, WorkerSinkTask doesn't seem to provide any access to the
WorkerSinkTaskContext, even though the latter has accessors to reset the
offsets with. Without those, I'm not sure what the purpose of the rewind
method is, since it doesn't seem to be possible to set the offsets at all.
Is this by design?
Thanks.
-Jack Lund
Braintree Payment Systems
Re: Resetting the Offset of a Kafka Sink Connector
Posted by Jack Lund <ja...@braintreepayments.com>.
Ah, cool. Thanks!
On Sat, Jun 4, 2016 at 7:42 PM Ewen Cheslack-Postava <ew...@confluent.io>
wrote:
> Connectors don't perform any data copying and don't rewind offsets --
> that's the job of Tasks. In your SinkTask implementation you have access to
> the SinkTaskContext via its context field.
>
> -Ewen
>
> On Tue, May 31, 2016 at 9:47 AM, Jack Lund <
> jack.lund@braintreepayments.com>
> wrote:
>
> > Yes, the one that the SinkConnector uses is the WorkerSinkTaskContext,
> but,
> > unfortunately, it creates it and uses it internally, but doesn't expose
> any
> > accessors for it, nor does the constructor allow me to pass one in for it
> > to use.
> >
> > -Jack
> >
> > On Tue, May 31, 2016 at 11:34 AM Dean Arnold <re...@gmail.com>
> > wrote:
> >
> > > Have you tried either of the SinkTaskContext.offset() methods ?
> > >
> > >
> > >
> >
> https://kafka.apache.org/0100/javadoc/org/apache/kafka/connect/sink/SinkTaskContext.html
> > >
> > > On Tue, May 31, 2016 at 8:43 AM, Jack Lund <
> > > jack.lund@braintreepayments.com>
> > > wrote:
> > >
> > > > I'm trying to use the Connector API to write data to a backing store
> > > (HDFS
> > > > for now, but probably something like S3 later) for potential replay
> > back
> > > > into Kafka later. However, I can't seem to find how to reset the
> > offsets
> > > > for the SinkConnector.
> > > >
> > > > I've found the rewind() function on the WorkerSinkTask:
> > > >
> > > >
> > >
> >
> https://github.com/apache/kafka/blob/c9485b78a6e43747daf1314ae9532839fb7bc810/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L396
> > > >
> > > > However, WorkerSinkTask doesn't seem to provide any access to the
> > > > WorkerSinkTaskContext, even though the latter has accessors to reset
> > the
> > > > offsets with. Without those, I'm not sure what the purpose of the
> > rewind
> > > > method is, since it doesn't seem to be possible to set the offsets at
> > > all.
> > > >
> > > > Is this by design?
> > > >
> > > > Thanks.
> > > >
> > > > -Jack Lund
> > > > Braintree Payment Systems
> > > >
> > >
> >
>
>
>
> --
> Thanks,
> Ewen
>
Re: Resetting the Offset of a Kafka Sink Connector
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
Connectors don't perform any data copying and don't rewind offsets --
that's the job of Tasks. In your SinkTask implementation you have access to
the SinkTaskContext via its context field.
-Ewen
On Tue, May 31, 2016 at 9:47 AM, Jack Lund <ja...@braintreepayments.com>
wrote:
> Yes, the one that the SinkConnector uses is the WorkerSinkTaskContext, but,
> unfortunately, it creates it and uses it internally, but doesn't expose any
> accessors for it, nor does the constructor allow me to pass one in for it
> to use.
>
> -Jack
>
> On Tue, May 31, 2016 at 11:34 AM Dean Arnold <re...@gmail.com>
> wrote:
>
> > Have you tried either of the SinkTaskContext.offset() methods ?
> >
> >
> >
> https://kafka.apache.org/0100/javadoc/org/apache/kafka/connect/sink/SinkTaskContext.html
> >
> > On Tue, May 31, 2016 at 8:43 AM, Jack Lund <
> > jack.lund@braintreepayments.com>
> > wrote:
> >
> > > I'm trying to use the Connector API to write data to a backing store
> > (HDFS
> > > for now, but probably something like S3 later) for potential replay
> back
> > > into Kafka later. However, I can't seem to find how to reset the
> offsets
> > > for the SinkConnector.
> > >
> > > I've found the rewind() function on the WorkerSinkTask:
> > >
> > >
> >
> https://github.com/apache/kafka/blob/c9485b78a6e43747daf1314ae9532839fb7bc810/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L396
> > >
> > > However, WorkerSinkTask doesn't seem to provide any access to the
> > > WorkerSinkTaskContext, even though the latter has accessors to reset
> the
> > > offsets with. Without those, I'm not sure what the purpose of the
> rewind
> > > method is, since it doesn't seem to be possible to set the offsets at
> > all.
> > >
> > > Is this by design?
> > >
> > > Thanks.
> > >
> > > -Jack Lund
> > > Braintree Payment Systems
> > >
> >
>
--
Thanks,
Ewen
Re: Resetting the Offset of a Kafka Sink Connector
Posted by Jack Lund <ja...@braintreepayments.com>.
Yes, the one that the SinkConnector uses is the WorkerSinkTaskContext, but,
unfortunately, it creates it and uses it internally, but doesn't expose any
accessors for it, nor does the constructor allow me to pass one in for it
to use.
-Jack
On Tue, May 31, 2016 at 11:34 AM Dean Arnold <re...@gmail.com> wrote:
> Have you tried either of the SinkTaskContext.offset() methods ?
>
>
> https://kafka.apache.org/0100/javadoc/org/apache/kafka/connect/sink/SinkTaskContext.html
>
> On Tue, May 31, 2016 at 8:43 AM, Jack Lund <
> jack.lund@braintreepayments.com>
> wrote:
>
> > I'm trying to use the Connector API to write data to a backing store
> (HDFS
> > for now, but probably something like S3 later) for potential replay back
> > into Kafka later. However, I can't seem to find how to reset the offsets
> > for the SinkConnector.
> >
> > I've found the rewind() function on the WorkerSinkTask:
> >
> >
> https://github.com/apache/kafka/blob/c9485b78a6e43747daf1314ae9532839fb7bc810/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L396
> >
> > However, WorkerSinkTask doesn't seem to provide any access to the
> > WorkerSinkTaskContext, even though the latter has accessors to reset the
> > offsets with. Without those, I'm not sure what the purpose of the rewind
> > method is, since it doesn't seem to be possible to set the offsets at
> all.
> >
> > Is this by design?
> >
> > Thanks.
> >
> > -Jack Lund
> > Braintree Payment Systems
> >
>
Re: Resetting the Offset of a Kafka Sink Connector
Posted by Dean Arnold <re...@gmail.com>.
Have you tried either of the SinkTaskContext.offset() methods ?
https://kafka.apache.org/0100/javadoc/org/apache/kafka/connect/sink/SinkTaskContext.html
On Tue, May 31, 2016 at 8:43 AM, Jack Lund <ja...@braintreepayments.com>
wrote:
> I'm trying to use the Connector API to write data to a backing store (HDFS
> for now, but probably something like S3 later) for potential replay back
> into Kafka later. However, I can't seem to find how to reset the offsets
> for the SinkConnector.
>
> I've found the rewind() function on the WorkerSinkTask:
>
> https://github.com/apache/kafka/blob/c9485b78a6e43747daf1314ae9532839fb7bc810/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L396
>
> However, WorkerSinkTask doesn't seem to provide any access to the
> WorkerSinkTaskContext, even though the latter has accessors to reset the
> offsets with. Without those, I'm not sure what the purpose of the rewind
> method is, since it doesn't seem to be possible to set the offsets at all.
>
> Is this by design?
>
> Thanks.
>
> -Jack Lund
> Braintree Payment Systems
>