You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2012/09/10 02:10:07 UTC
[jira] [Created] (KAFKA-501) getOfffset Api needs to return
different latest offset to regular and follower consumers
Jun Rao created KAFKA-501:
-----------------------------
Summary: getOfffset Api needs to return different latest offset to regular and follower consumers
Key: KAFKA-501
URL: https://issues.apache.org/jira/browse/KAFKA-501
Project: Kafka
Issue Type: Bug
Reporter: Jun Rao
Fix For: 0.8
For follower consumers, getOffset should return logEndOffset as the latest offset. For regular consumers, getOffset should return highWatermark as the latest offset.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-501) getOfffset Api needs to return
different latest offset to regular and follower consumers
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy resolved KAFKA-501.
------------------------------
Resolution: Fixed
> getOfffset Api needs to return different latest offset to regular and follower consumers
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-501
> URL: https://issues.apache.org/jira/browse/KAFKA-501
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Rao
> Assignee: Joel Koshy
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: KAFKA-501-v1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For follower consumers, getOffset should return logEndOffset as the latest offset. For regular consumers, getOffset should return highWatermark as the latest offset.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-501) getOfffset Api needs to return
different latest offset to regular and follower consumers
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464790#comment-13464790 ]
Jun Rao commented on KAFKA-501:
-------------------------------
Actually, for #2, you are right, offsets are returned in descending order. So this is fine.
If the rest of the comments are addressed, the patch can be checked in without another round of review.
> getOfffset Api needs to return different latest offset to regular and follower consumers
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-501
> URL: https://issues.apache.org/jira/browse/KAFKA-501
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Rao
> Assignee: Joel Koshy
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: KAFKA-501-v1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For follower consumers, getOffset should return logEndOffset as the latest offset. For regular consumers, getOffset should return highWatermark as the latest offset.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (KAFKA-501) getOfffset Api needs to return
different latest offset to regular and follower consumers
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy reassigned KAFKA-501:
--------------------------------
Assignee: Joel Koshy
> getOfffset Api needs to return different latest offset to regular and follower consumers
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-501
> URL: https://issues.apache.org/jira/browse/KAFKA-501
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Rao
> Assignee: Joel Koshy
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For follower consumers, getOffset should return logEndOffset as the latest offset. For regular consumers, getOffset should return highWatermark as the latest offset.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-501) getOfffset Api needs to return
different latest offset to regular and follower consumers
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464453#comment-13464453 ]
Jun Rao commented on KAFKA-501:
-------------------------------
Thanks for patch v1. Looks good overall. Some minor comments:
1. javaapi.OffsetRequest: No need to expose replicaId in the constructor since all java clients are non-follower.
2. Log.getOffsetBefore, KafkaApis: So far, offsets are returned in ascending order.
3. removed unused imports: LogManager, LogOffsetTest
4. UpdateOffsetsInZK: Not directly related to this jira, but this class should probably be moved to tools package.
5. ZookeeperConsumerConnector: Do we need to import scala.Some?
Also, could you rebase?
> getOfffset Api needs to return different latest offset to regular and follower consumers
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-501
> URL: https://issues.apache.org/jira/browse/KAFKA-501
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Rao
> Assignee: Joel Koshy
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: KAFKA-501-v1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For follower consumers, getOffset should return logEndOffset as the latest offset. For regular consumers, getOffset should return highWatermark as the latest offset.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-501) getOfffset Api needs to return
different latest offset to regular and follower consumers
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-501:
-----------------------------
Priority: Blocker (was: Major)
> getOfffset Api needs to return different latest offset to regular and follower consumers
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-501
> URL: https://issues.apache.org/jira/browse/KAFKA-501
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Rao
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For follower consumers, getOffset should return logEndOffset as the latest offset. For regular consumers, getOffset should return highWatermark as the latest offset.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-501) getOfffset Api needs to return
different latest offset to regular and follower consumers
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13465004#comment-13465004 ]
Joel Koshy commented on KAFKA-501:
----------------------------------
Thanks for the review. Addressed the above comments and committed to 0.8.
> getOfffset Api needs to return different latest offset to regular and follower consumers
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-501
> URL: https://issues.apache.org/jira/browse/KAFKA-501
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Rao
> Assignee: Joel Koshy
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: KAFKA-501-v1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For follower consumers, getOffset should return logEndOffset as the latest offset. For regular consumers, getOffset should return highWatermark as the latest offset.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-501) getOfffset Api needs to return
different latest offset to regular and follower consumers
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466630#comment-13466630 ]
Jun Rao commented on KAFKA-501:
-------------------------------
Just realized that we need to patch KafkaETLContext to use getMetaData api to figure out the leader so that it can make the getOffsetBefore call.
> getOfffset Api needs to return different latest offset to regular and follower consumers
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-501
> URL: https://issues.apache.org/jira/browse/KAFKA-501
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Rao
> Assignee: Joel Koshy
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: KAFKA-501-v1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For follower consumers, getOffset should return logEndOffset as the latest offset. For regular consumers, getOffset should return highWatermark as the latest offset.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-501) getOfffset Api needs to return
different latest offset to regular and follower consumers
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy updated KAFKA-501:
-----------------------------
Attachment: KAFKA-501-v1.patch
Overview of changes:
1 - For non-followers, collapse any offsets > hw into a single entry == hw.
2 - I think the returned offsets are reverse sorted, but I did that again
anyway, for the above collapsing to work correctly. I could just as well
use Seq.span.
3 - Made the offsetrequest batched, using a map. So this is very similar to
what was done for KAFKA-391. So I also needed to provide javaapi versions
of the OffsetRequest and OffsetResponse.
4 - One side-effect of batching was I was forced to clean up the layering
issue. i.e., I prevent log from taking offsetrequest.
5 - Switched from Array to Seq in the scala version (and got rid of the
equals).
6 - For consistency, made simple-consumer's getOffsetsBefore take a request
and return a response.
Other points:
7 - I'm not very sure we should be providing the OffsetRequest directly to
clients - i.e., maybe we should only have a request builder (like we have
for FetchRequest). This would make it easier to use from the Java side and
also we can hide the replicaId option - maybe it shouldn't be exposed to
clients (especially on the Java-side)
8 - Code in getOffsetsBefore can be cleaned up quite a bit I think, but I left
it as is for now.
9 - I was thinking of nesting PartitionOffsetRequestInfo inside the
OffsetRequest object - that would make the code clearer, but only if users
always fully qualify the reference.
10 - Previously getOffsetsBefore could through an exception on Error. Now, the
client will need to explicitly check the errorCode. We may need to go
through all usages and assess whether we want to throw an exception or
not.
> getOfffset Api needs to return different latest offset to regular and follower consumers
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-501
> URL: https://issues.apache.org/jira/browse/KAFKA-501
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Rao
> Assignee: Joel Koshy
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: KAFKA-501-v1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For follower consumers, getOffset should return logEndOffset as the latest offset. For regular consumers, getOffset should return highWatermark as the latest offset.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-501) getOfffset Api needs to return
different latest offset to regular and follower consumers
Posted by "Joel Koshy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joel Koshy closed KAFKA-501.
----------------------------
> getOfffset Api needs to return different latest offset to regular and follower consumers
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-501
> URL: https://issues.apache.org/jira/browse/KAFKA-501
> Project: Kafka
> Issue Type: Bug
> Reporter: Jun Rao
> Assignee: Joel Koshy
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
> Attachments: KAFKA-501-v1.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> For follower consumers, getOffset should return logEndOffset as the latest offset. For regular consumers, getOffset should return highWatermark as the latest offset.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira