You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "KaranShishodia (via GitHub)" <gi...@apache.org> on 2025/12/19 06:48:04 UTC
[I] [Feature] Batched Summarization for Long-Term Memory [flink-agents]
KaranShishodia opened a new issue, #387:
URL: https://github.com/apache/flink-agents/issues/387
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar.
### Description
The current design feeds all items into the LLM at once, which risks exceeding the context window. A practical solution is to introduce batched summarization with the following workflow:
- Chunking Input Data
- Split the memory items into fixed-size batches (configurable, e.g., 50–100 items per batch).
- Each batch is summarized independently, producing a concise representation.
- Hierarchical Summarization
- After batch-level summaries are generated, feed those summaries into a second summarization step.
- This produces a global summary that captures the overall context without overwhelming the LLM.
- Configurable Parameters
- Allow users to configure:
- Batch size (number of items per summarization call).
- - Summarization depth (single-pass vs. hierarchical).
- Retention policy (e.g., keep both batch summaries and global summary for traceability).
- Implementation Sketch (Pseudo-Java/Python)
List<String> items = getMemoryItems();
int batchSize = 50;
List<String> batchSummaries = new ArrayList<>();
for (int i = 0; i < items.size(); i += batchSize) {
List<String> batch = items.subList(i, Math.min(i + batchSize, items.size()));
String summary = llm.summarize(batch);
batchSummaries.add(summary);
}
// Hierarchical summarization
String globalSummary = llm.summarize(batchSummaries);
storeSummary(globalSummary);
- Advantages
- Prevents context overflow by respecting LLM limits.
- Scales to very large memory sets.
- Maintains fidelity by layering summaries instead of discarding details
Next Steps
- Add a runtime configuration option for batch size and summarization depth.
- Implement a summarization operator in the Flink runtime that can be reused across agents.
- Provide benchmarks comparing single-pass vs. batched summarization to validate efficiency
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Batched Summarization for Long-Term Memory [flink-agents]
Posted by "xintongsong (via GitHub)" <gi...@apache.org>.
xintongsong commented on issue #387:
URL: https://github.com/apache/flink-agents/issues/387#issuecomment-3981820304
Thanks @addu390, replied in the other thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Batched Summarization for Long-Term Memory [flink-agents]
Posted by "KaranShishodia (via GitHub)" <gi...@apache.org>.
KaranShishodia commented on issue #387:
URL: https://github.com/apache/flink-agents/issues/387#issuecomment-3673812060
public class BatchedSummarizationOperator {
private final int batchSize;
private final LLMClient llmClient;
public BatchedSummarizationOperator(int batchSize, LLMClient llmClient) {
this.batchSize = batchSize;
this.llmClient = llmClient;
}
public String summarize(List<String> memoryItems) {
List<String> batchSummaries = new ArrayList<>();
for (int i = 0; i < memoryItems.size(); i += batchSize) {
List<String> batch = memoryItems.subList(i, Math.min(i + batchSize, memoryItems.size()));
String summary = llmClient.summarize(batch);
batchSummaries.add(summary);
}
// Hierarchical summarization
return llmClient.summarize(batchSummaries);
}
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Batched Summarization for Long-Term Memory [flink-agents]
Posted by "xintongsong (via GitHub)" <gi...@apache.org>.
xintongsong closed issue #387: [Feature] Batched Summarization for Long-Term Memory
URL: https://github.com/apache/flink-agents/issues/387
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Batched Summarization for Long-Term Memory [flink-agents]
Posted by "KaranShishodia (via GitHub)" <gi...@apache.org>.
KaranShishodia commented on issue #387:
URL: https://github.com/apache/flink-agents/issues/387#issuecomment-3673808498
The current design feeds all items into the LLM at once, which risks exceeding the context window. A practical solution is to introduce batched summarization with the following workflow:
Chunking Input Data
Split the memory items into fixed-size batches (configurable, e.g., 50–100 items per batch).
Each batch is summarized independently, producing a concise representation.
Hierarchical Summarization
After batch-level summaries are generated, feed those summaries into a second summarization step.
This produces a global summary that captures the overall context without overwhelming the LLM.
Configurable Parameters
Allow users to configure:
Batch size (number of items per summarization call).
Summarization depth (single-pass vs. hierarchical).
Retention policy (e.g., keep both batch summaries and global summary for traceability).
Implementation Sketch (Pseudo-Java/Python)
List items = getMemoryItems();
int batchSize = 50;
List batchSummaries = new ArrayList<>();
for (int i = 0; i < items.size(); i += batchSize) {
List batch = items.subList(i, Math.min(i + batchSize, items.size()));
String summary = llm.summarize(batch);
batchSummaries.add(summary);
}
// Hierarchical summarization
String globalSummary = llm.summarize(batchSummaries);
storeSummary(globalSummary);
Advantages
Prevents context overflow by respecting LLM limits.
Scales to very large memory sets.
Maintains fidelity by layering summaries instead of discarding details
Next Steps
Add a runtime configuration option for batch size and summarization depth.
Implement a summarization operator in the Flink runtime that can be reused across agents.
Provide benchmarks comparing single-pass vs. batched summarization to validate efficiency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Batched Summarization for Long-Term Memory [flink-agents]
Posted by "addu390 (via GitHub)" <gi...@apache.org>.
addu390 commented on issue #387:
URL: https://github.com/apache/flink-agents/issues/387#issuecomment-3978820857
@xintongsong The issue is a duplicate of #384, I just dropped a question there. I can see how it can be super simple with a `batchSize`, the hierarchical batching like the above is reasonable too, but also see that being barely enough and needing more advanced batching strategies.
Do you have an opinion on how to go about it? If not, I'm also down to come up with a few ideas!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Batched Summarization for Long-Term Memory [flink-agents]
Posted by "xintongsong (via GitHub)" <gi...@apache.org>.
xintongsong commented on issue #387:
URL: https://github.com/apache/flink-agents/issues/387#issuecomment-3673927287
@KaranShishodia
Thanks for your proposal. We also realized that the current one-pass full compaction won't work when it comes to large memory amount. The reason we didn't address this issue in the first PR is just to avoid complicating the PR too much. And I really appreciate it that you'd like to contribute on this.
To move ahead, I'd suggest to create a detailed design doc for your proposal in [Discussions](https://github.com/apache/flink-agents/discussions). This feature could be big and complex, thus deserves detailed design discussions. Github Discussions is where we keep the design docs.
Does it sound good to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Batched Summarization for Long-Term Memory [flink-agents]
Posted by "KaranShishodia (via GitHub)" <gi...@apache.org>.
KaranShishodia commented on issue #387:
URL: https://github.com/apache/flink-agents/issues/387#issuecomment-3673809415
The current design feeds all items into the LLM at once, which risks exceeding the context window. A practical solution is to introduce batched summarization with the following workflow:
Chunking Input Data
Split the memory items into fixed-size batches (configurable, e.g., 50–100 items per batch).
Each batch is summarized independently, producing a concise representation.
Hierarchical Summarization
After batch-level summaries are generated, feed those summaries into a second summarization step.
This produces a global summary that captures the overall context without overwhelming the LLM.
Configurable Parameters
Allow users to configure:
Batch size (number of items per summarization call).
Summarization depth (single-pass vs. hierarchical).
Retention policy (e.g., keep both batch summaries and global summary for traceability).
Implementation Sketch (Pseudo-Java/Python)
List items = getMemoryItems();
int batchSize = 50;
List batchSummaries = new ArrayList<>();
for (int i = 0; i < items.size(); i += batchSize) {
List batch = items.subList(i, Math.min(i + batchSize, items.size()));
String summary = llm.summarize(batch);
batchSummaries.add(summary);
}
// Hierarchical summarization
String globalSummary = llm.summarize(batchSummaries);
storeSummary(globalSummary);
Advantages
Prevents context overflow by respecting LLM limits.
Scales to very large memory sets.
Maintains fidelity by layering summaries instead of discarding details
Next Steps
Add a runtime configuration option for batch size and summarization depth.
Implement a summarization operator in the Flink runtime that can be reused across agents.
Provide benchmarks comparing single-pass vs. batched summarization to validate efficiency
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature] Batched Summarization for Long-Term Memory [flink-agents]
Posted by "KaranShishodia (via GitHub)" <gi...@apache.org>.
KaranShishodia commented on issue #387:
URL: https://github.com/apache/flink-agents/issues/387#issuecomment-3673810433
This approach balances scalability with accuracy, and aligns with Flink’s streaming philosophy by treating summarization as a staged pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org