[jira] [Commented] (CAMEL-25807) camel-github2 - the commit consumer silently skips the older commits when more than 100 commits are pushed between two polls

["Claus Ibsen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CAMEL-25807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18129808#comment-18129808 ] 

Claus Ibsen commented on CAMEL-25807:
-------------------------------------

Merged via https://github.com/apache/camel/pull/28199

_Claude Code on behalf of davsclaus_

> camel-github2 - the commit consumer silently skips the older commits when more than 100 commits are pushed between two polls
> ----------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CAMEL-25807
>                 URL: https://issues.apache.org/jira/browse/CAMEL-25807
>             Project: Camel
>          Issue Type: Bug
>          Components: camel-github
>            Reporter: shashank
>            Assignee: shashank
>            Priority: Major
>             Fix For: 4.24.0
>
>
> {{CommitConsumer.poll}} (main 7aae27315946, lines 104-165) lists the commits of the branch, newest first, and stops after 100 ("Stop after 100 commits to avoid fetching too many", lines 113-117). It then clips the list at the last routed commit ({{lastSha}}) and routes the commits before it, oldest first; {{onRouting}} moves {{lastSha}} to each routed commit, so it ends at the newest one.
> When more than 100 commits were pushed to the branch between two polls, {{lastSha}} is not among the 100 newest: the poll routes those 100 (the ones not in the remembered set), {{lastSha}} becomes the newest commit, and the next poll stops at it. The older new commits are never routed, and nothing is logged. This happens with a large merge or push, and at the first poll after GitHub was unreachable (or the polls failed, for example on the rate limit) while more than 100 commits were pushed.
> *Fix*: when there is a last routed commit, the poll reads the listing up to it, at most 1000 commits (the number of commits the consumer already remembers in its set of routed shas). When the last commit is not found (it is no longer on the branch, after a force push) or there is none ({{startingSha=beginning}}), only the 100 newest commits are considered, as before. Upgrade note in the 4.24 guide.
> *Tests*: new {{GitHub2CommitConsumerManyNewCommitsTest}} (mocked repository listing, as in {{GitHub2ConsumerPositionTest}}). {{everyNewCommitIsRouted}} (150 commits pushed after {{startingSha=c0}}) fails on main: the routed list is {{[c51, c52, ..., c150]}} instead of {{[c1, ..., c150]}}; passes with the fix. {{withoutTheLastCommitOnTheBranchOnlyTheNewest100AreRouted}} passes on main and with the fix (the force push case keeps its behaviour). camel-github2 unit tests: 16 run, 0 failures.
> *How it was found*: the same Z3 model as the camel-jira newIssues consumer (a cursor that moves to the newest item of a poll whose answer is cut to the newest M): an item is lost as soon as M+1 items arrive between two polls, none when the poll reads up to the cursor; then the JUnit test above. A variant of CAMEL-25806 (camel-jira newIssues).
> Related: CAMEL-21256 ({{startingSha=last}} with a large history). When there is no {{lastSha}}, or it is not in the listing (a force push), the poll still reads at most the newest 100 commits as before, so that case is unchanged.
> *Affected*: main, camel-4.23.x, camel-4.22.x and camel-4.18.x (the same cut at 100; read, not run). camel-4.14.x has no camel-github2.
> _Filed with Claude Code on behalf of allthingssecurity._



--
This message was sent by Atlassian Jira
(v8.20.10#820010)