You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openserverless.apache.org by "sciabarracom (via GitHub)" <gi...@apache.org> on 2024/11/23 11:18:51 UTC
[I] A streaming proxy for LLM answers [openserverless]
sciabarracom opened a new issue, #79:
URL: https://github.com/apache/openserverless/issues/79
## Why you need it?
LLm applications require a way to stream the results as they are generated
## How it could be?
We need a stereaming proxy. It should work this way.
Since actions while produce input cannot communicate back with the frontend, they should write the temporary content in redis.
The proxy is mounted on `/stream` of each user with an ingress, and when it receive a request `/stream/<key>` it will stream the content as it arrives in the key `<user>:<key>`
## Other related information
We should explore the best way to execute the request.
We should be able to execute a request liket `how can I become rich` and forget.
Normally openwhisk returns an activation id, and the result should be written in redis using this id for the streaming . So an asyncrhonous request should be more or less:
1. webaction invoking an asyncrhonous request and returning the activation id
2. action talking to an LLM and writing the result to redis in the key corresponding to the activtion id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@openserverless.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] A streaming proxy for LLM answers [openserverless]
Posted by "sciabarracom (via GitHub)" <gi...@apache.org>.
sciabarracom closed issue #79: A streaming proxy for LLM answers
URL: https://github.com/apache/openserverless/issues/79
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@openserverless.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] A streaming proxy for LLM answers [openserverless]
Posted by "francescotimperi (via GitHub)" <gi...@apache.org>.
francescotimperi commented on issue #79:
URL: https://github.com/apache/openserverless/issues/79#issuecomment-2496200649
An idea could be to use Redis/Valkey Streams. (an example [here](https://redis.io/learn/howtos/solutions/streams/streaming-llm-output) )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@openserverless.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] A streaming proxy for LLM answers [openserverless]
Posted by "sciabarracom (via GitHub)" <gi...@apache.org>.
sciabarracom commented on issue #79:
URL: https://github.com/apache/openserverless/issues/79#issuecomment-2517210217
please check https://github.com/apache/openserverless/issues/82 as this spec is outdated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@openserverless.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org