You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Tommy Becker <to...@tivo.com> on 2015/01/08 22:21:32 UTC
Checkpoint tool question
I'm debugging a new stream task and need to reprocess the input streams repeatedly, and consequently I'm trying to use the checkpoint tool to reset the offsets. The tool, when run without --new-offsets emits messages with the following pattern:
Current checkpoint for taskname Partition 6: tasknames.Partition 0.systems.kafka.streams.mytopic.partitions.0 = 323165
But putting the following in a reset-offsets.properties file doesn't work:
systems.kafka.streams.mytopic.partitions.0 = 0
Instead the tool emits this:
Warning: ignoring unrecognised property: systems.kafka.streams.mytopic.partitions.0 = 0
I had some success using this at one point, but it's been a while and I think it was under 0.7.0. Am I missing something?
-Tommy Becker
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
Re: Checkpoint tool question
Posted by Chris Riccomini <cr...@linkedin.com.INVALID>.
Hey Tommy,
This looks like a bug. I've opened this:
https://issues.apache.org/jira/browse/SAMZA-504
Try escaping the space in the property key like this:
tasknames.Partition\ 8.systems.kafka.streams.mytopic.partitions.8=0
Cheers,
Chris
On 1/8/15 5:50 PM, "Tommy Becker" <to...@Tivo.com> wrote:
>Hi Chris,
>Yes, I've tried that as well, and the existing offsets are simply spit
>out again. I do see the following:
>
>Warning: ignoring unrecognised property: tasknames.Partition =
>8.systems.kafka.streams.mytopic.partitions.8=0
>
>which is puzzling because the line for partition 8 in the file looks like
>this:
>
>tasknames.Partition 8.systems.kafka.streams.mytopic.partitions.8=0
>________________________________________
>From: Chris Riccomini [criccomini@linkedin.com.INVALID]
>Sent: Thursday, January 08, 2015 6:35 PM
>To: dev@samza.incubator.apache.org
>Subject: Re: Checkpoint tool question
>
>Hey Tommy,
>
>Have you tried doing:
>
>tasknames.Partition 0.systems.kafka.streams.mytopic.partitions.0=0
>
>
>Cheers,
>Chris
>
>On 1/8/15 1:21 PM, "Tommy Becker" <to...@tivo.com> wrote:
>
>>I'm debugging a new stream task and need to reprocess the input streams
>>repeatedly, and consequently I'm trying to use the checkpoint tool to
>>reset the offsets. The tool, when run without --new-offsets emits
>>messages with the following pattern:
>>
>>Current checkpoint for taskname Partition 6: tasknames.Partition
>>0.systems.kafka.streams.mytopic.partitions.0 = 323165
>>
>>But putting the following in a reset-offsets.properties file doesn't
>>work:
>>
>>systems.kafka.streams.mytopic.partitions.0 = 0
>>
>>Instead the tool emits this:
>>
>>Warning: ignoring unrecognised property:
>>systems.kafka.streams.mytopic.partitions.0 = 0
>>
>>I had some success using this at one point, but it's been a while and I
>>think it was under 0.7.0. Am I missing something?
>>
>>
>>-Tommy Becker
>>
>>
>>________________________________
>>
>>This email and any attachments may contain confidential and privileged
>>material for the sole use of the intended recipient. Any review, copying,
>>or distribution of this email (or any attachments) by others is
>>prohibited. If you are not the intended recipient, please contact the
>>sender immediately and permanently delete this email and any attachments.
>>No employee or agent of TiVo Inc. is authorized to conclude any binding
>>agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
>>Inc. may only be made by a signed written agreement.
>
>________________________________
>
>This email and any attachments may contain confidential and privileged
>material for the sole use of the intended recipient. Any review, copying,
>or distribution of this email (or any attachments) by others is
>prohibited. If you are not the intended recipient, please contact the
>sender immediately and permanently delete this email and any attachments.
>No employee or agent of TiVo Inc. is authorized to conclude any binding
>agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
>Inc. may only be made by a signed written agreement.
RE: Checkpoint tool question
Posted by Tommy Becker <to...@Tivo.com>.
Hi Chris,
Yes, I've tried that as well, and the existing offsets are simply spit out again. I do see the following:
Warning: ignoring unrecognised property: tasknames.Partition = 8.systems.kafka.streams.mytopic.partitions.8=0
which is puzzling because the line for partition 8 in the file looks like this:
tasknames.Partition 8.systems.kafka.streams.mytopic.partitions.8=0
________________________________________
From: Chris Riccomini [criccomini@linkedin.com.INVALID]
Sent: Thursday, January 08, 2015 6:35 PM
To: dev@samza.incubator.apache.org
Subject: Re: Checkpoint tool question
Hey Tommy,
Have you tried doing:
tasknames.Partition 0.systems.kafka.streams.mytopic.partitions.0=0
Cheers,
Chris
On 1/8/15 1:21 PM, "Tommy Becker" <to...@tivo.com> wrote:
>I'm debugging a new stream task and need to reprocess the input streams
>repeatedly, and consequently I'm trying to use the checkpoint tool to
>reset the offsets. The tool, when run without --new-offsets emits
>messages with the following pattern:
>
>Current checkpoint for taskname Partition 6: tasknames.Partition
>0.systems.kafka.streams.mytopic.partitions.0 = 323165
>
>But putting the following in a reset-offsets.properties file doesn't work:
>
>systems.kafka.streams.mytopic.partitions.0 = 0
>
>Instead the tool emits this:
>
>Warning: ignoring unrecognised property:
>systems.kafka.streams.mytopic.partitions.0 = 0
>
>I had some success using this at one point, but it's been a while and I
>think it was under 0.7.0. Am I missing something?
>
>
>-Tommy Becker
>
>
>________________________________
>
>This email and any attachments may contain confidential and privileged
>material for the sole use of the intended recipient. Any review, copying,
>or distribution of this email (or any attachments) by others is
>prohibited. If you are not the intended recipient, please contact the
>sender immediately and permanently delete this email and any attachments.
>No employee or agent of TiVo Inc. is authorized to conclude any binding
>agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
>Inc. may only be made by a signed written agreement.
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
Re: Checkpoint tool question
Posted by Chris Riccomini <cr...@linkedin.com.INVALID>.
Hey Tommy,
Have you tried doing:
tasknames.Partition 0.systems.kafka.streams.mytopic.partitions.0=0
Cheers,
Chris
On 1/8/15 1:21 PM, "Tommy Becker" <to...@tivo.com> wrote:
>I'm debugging a new stream task and need to reprocess the input streams
>repeatedly, and consequently I'm trying to use the checkpoint tool to
>reset the offsets. The tool, when run without --new-offsets emits
>messages with the following pattern:
>
>Current checkpoint for taskname Partition 6: tasknames.Partition
>0.systems.kafka.streams.mytopic.partitions.0 = 323165
>
>But putting the following in a reset-offsets.properties file doesn't work:
>
>systems.kafka.streams.mytopic.partitions.0 = 0
>
>Instead the tool emits this:
>
>Warning: ignoring unrecognised property:
>systems.kafka.streams.mytopic.partitions.0 = 0
>
>I had some success using this at one point, but it's been a while and I
>think it was under 0.7.0. Am I missing something?
>
>
>-Tommy Becker
>
>
>________________________________
>
>This email and any attachments may contain confidential and privileged
>material for the sole use of the intended recipient. Any review, copying,
>or distribution of this email (or any attachments) by others is
>prohibited. If you are not the intended recipient, please contact the
>sender immediately and permanently delete this email and any attachments.
>No employee or agent of TiVo Inc. is authorized to conclude any binding
>agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
>Inc. may only be made by a signed written agreement.
RE: Checkpoint tool question
Posted by Tommy Becker <to...@Tivo.com>.
Hi Navina,
I'm running the following:
bin/checkpoint-tool.sh --config-path config/myjob.properties --new-offsets /path/to/reset-offsets.properties
The job.name property is set in myjob.properties (this is the same config I use to run the job).
-Tommy
________________________________________
From: Navina Ramesh [nramesh@linkedin.com.INVALID]
Sent: Thursday, January 08, 2015 5:33 PM
To: dev@samza.incubator.apache.org
Subject: Re: Checkpoint tool question
Hi Tommy,
Can you please provide the complete command that you are trying to use ? I
suspect you are not providing the task name.
Navina
On 1/8/15, 1:21 PM, "Tommy Becker" <to...@tivo.com> wrote:
>I'm debugging a new stream task and need to reprocess the input streams
>repeatedly, and consequently I'm trying to use the checkpoint tool to
>reset the offsets. The tool, when run without --new-offsets emits
>messages with the following pattern:
>
>Current checkpoint for taskname Partition 6: tasknames.Partition
>0.systems.kafka.streams.mytopic.partitions.0 = 323165
>
>But putting the following in a reset-offsets.properties file doesn't work:
>
>systems.kafka.streams.mytopic.partitions.0 = 0
>
>Instead the tool emits this:
>
>Warning: ignoring unrecognised property:
>systems.kafka.streams.mytopic.partitions.0 = 0
>
>I had some success using this at one point, but it's been a while and I
>think it was under 0.7.0. Am I missing something?
>
>
>-Tommy Becker
>
>
>________________________________
>
>This email and any attachments may contain confidential and privileged
>material for the sole use of the intended recipient. Any review, copying,
>or distribution of this email (or any attachments) by others is
>prohibited. If you are not the intended recipient, please contact the
>sender immediately and permanently delete this email and any attachments.
>No employee or agent of TiVo Inc. is authorized to conclude any binding
>agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
>Inc. may only be made by a signed written agreement.
________________________________
This email and any attachments may contain confidential and privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments) by others is prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete this email and any attachments. No employee or agent of TiVo Inc. is authorized to conclude any binding agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo Inc. may only be made by a signed written agreement.
Re: Checkpoint tool question
Posted by Navina Ramesh <nr...@linkedin.com.INVALID>.
Hi Tommy,
Can you please provide the complete command that you are trying to use ? I
suspect you are not providing the task name.
Navina
On 1/8/15, 1:21 PM, "Tommy Becker" <to...@tivo.com> wrote:
>I'm debugging a new stream task and need to reprocess the input streams
>repeatedly, and consequently I'm trying to use the checkpoint tool to
>reset the offsets. The tool, when run without --new-offsets emits
>messages with the following pattern:
>
>Current checkpoint for taskname Partition 6: tasknames.Partition
>0.systems.kafka.streams.mytopic.partitions.0 = 323165
>
>But putting the following in a reset-offsets.properties file doesn't work:
>
>systems.kafka.streams.mytopic.partitions.0 = 0
>
>Instead the tool emits this:
>
>Warning: ignoring unrecognised property:
>systems.kafka.streams.mytopic.partitions.0 = 0
>
>I had some success using this at one point, but it's been a while and I
>think it was under 0.7.0. Am I missing something?
>
>
>-Tommy Becker
>
>
>________________________________
>
>This email and any attachments may contain confidential and privileged
>material for the sole use of the intended recipient. Any review, copying,
>or distribution of this email (or any attachments) by others is
>prohibited. If you are not the intended recipient, please contact the
>sender immediately and permanently delete this email and any attachments.
>No employee or agent of TiVo Inc. is authorized to conclude any binding
>agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
>Inc. may only be made by a signed written agreement.