You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Sofia’s World <mm...@gmail.com> on 2025/04/01 04:51:00 UTC
Re: calling an LLM from a Beam function
Hi Radek
uhm, how does that differ from creating something like this?
am i missing something?
Kind regards
Marco
class LLMProcessor(beam.DoFn):
def __init__(self, key):
"""Initiate the OAI API client."""
self.client = OpenAI(
api_key=key
)
def process(self, element: str):
logging.info(f'Input elements:{element}')
return self.client.do_some_processing(..)
On Thu, Mar 27, 2025 at 8:48 AM Radek Stankiewicz via user <
user@beam.apache.org> wrote:
> hi Sofia,
>
> here you have nice example
> https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/custom_remote_inference.ipynb
>
> where CloudVisionModelHandler is custom code that can invoke any client
> library.
> you can pass the key as one of the constructors to CloudVisionModelHandler
> or you can load it from any preferred secret managers e.g.
>
> def load_model(self):
> """Initiate the OAI API client."""
> client = OpenAI(
>
> api_key=client.access_secret_version(name="OPENAI_API_KEY").data.decode('UTF-8')
> )
> return client
>
> def run_inference(self, batch, oai_client, inference):
> response = oai_client.responses.create( # your LLM magic goes here.
> [..]
>
> using env variables won't work here as like you've noticed, beam is
> running on multiple machines and you can't set env variables there.
>
>
>
>
>
> On Thu, Mar 27, 2025 at 9:22 AM Sofia’s World <mm...@gmail.com> wrote:
>
>> Hello
>> presumably is possible to kick off a beam process that invokes an LLM>?
>> but the only issue i have is how/where do i store the OpenAI key for
>> example
>>
>> in my c current colab/pc setup i have the key configured in my
>> environment.. but Beam will run
>> on multiple machines... how do i configure the OPENAI_KEY for example?
>>
>> thanks and regards
>> Marco
>>
>
Re: calling an LLM from a Beam function
Posted by Sofia’s World <mm...@gmail.com>.
Great thanks Radek! works like a charm!
kind regards
Marco
On Tue, Apr 1, 2025 at 9:34 AM Sofia’s World <mm...@gmail.com> wrote:
> I see .I should have read code better..I did not see run inference is part
> of beam rather than the specific example.thanks
>
> On Tue, 1 Apr 2025, 09:02 Radek Stankiewicz, <ra...@google.com> wrote:
>
>> hey Marco,
>> In your case, as your model inference is remote and has a custom handler,
>> the only difference is that RunInference transform is adding batching
>> before invoking the handler.
>> Today this handler is pretty simple but I would imagine that this
>> RunInference in the future may introduce certain optimizations, like async
>> calls which today you would have to develop yourself in parDo.
>>
>> Radek
>>
>>
>> On Tue, Apr 1, 2025 at 6:51 AM Sofia’s World <mm...@gmail.com> wrote:
>>
>>> Hi Radek
>>> uhm, how does that differ from creating something like this?
>>> am i missing something?
>>>
>>> Kind regards
>>> Marco
>>>
>>>
>>> class LLMProcessor(beam.DoFn):
>>>
>>> def __init__(self, key):
>>> """Initiate the OAI API client."""
>>> self.client = OpenAI(
>>> api_key=key
>>> )
>>>
>>> def process(self, element: str):
>>> logging.info(f'Input elements:{element}')
>>>
>>> return self.client.do_some_processing(..)
>>>
>>>
>>> On Thu, Mar 27, 2025 at 8:48 AM Radek Stankiewicz via user <
>>> user@beam.apache.org> wrote:
>>>
>>>> hi Sofia,
>>>>
>>>> here you have nice example
>>>> https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/custom_remote_inference.ipynb
>>>>
>>>> where CloudVisionModelHandler is custom code that can invoke any client
>>>> library.
>>>> you can pass the key as one of the constructors
>>>> to CloudVisionModelHandler or you can load it from any preferred secret
>>>> managers e.g.
>>>>
>>>> def load_model(self):
>>>> """Initiate the OAI API client."""
>>>> client = OpenAI(
>>>>
>>>> api_key=client.access_secret_version(name="OPENAI_API_KEY").data.decode('UTF-8')
>>>> )
>>>> return client
>>>>
>>>> def run_inference(self, batch, oai_client, inference):
>>>> response = oai_client.responses.create( # your LLM magic goes here.
>>>> [..]
>>>>
>>>> using env variables won't work here as like you've noticed, beam is
>>>> running on multiple machines and you can't set env variables there.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Mar 27, 2025 at 9:22 AM Sofia’s World <mm...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello
>>>>> presumably is possible to kick off a beam process that invokes an LLM>?
>>>>> but the only issue i have is how/where do i store the OpenAI key for
>>>>> example
>>>>>
>>>>> in my c current colab/pc setup i have the key configured in my
>>>>> environment.. but Beam will run
>>>>> on multiple machines... how do i configure the OPENAI_KEY for example?
>>>>>
>>>>> thanks and regards
>>>>> Marco
>>>>>
>>>>
Re: calling an LLM from a Beam function
Posted by Sofia’s World <mm...@gmail.com>.
I see .I should have read code better..I did not see run inference is part
of beam rather than the specific example.thanks
On Tue, 1 Apr 2025, 09:02 Radek Stankiewicz, <ra...@google.com> wrote:
> hey Marco,
> In your case, as your model inference is remote and has a custom handler,
> the only difference is that RunInference transform is adding batching
> before invoking the handler.
> Today this handler is pretty simple but I would imagine that this
> RunInference in the future may introduce certain optimizations, like async
> calls which today you would have to develop yourself in parDo.
>
> Radek
>
>
> On Tue, Apr 1, 2025 at 6:51 AM Sofia’s World <mm...@gmail.com> wrote:
>
>> Hi Radek
>> uhm, how does that differ from creating something like this?
>> am i missing something?
>>
>> Kind regards
>> Marco
>>
>>
>> class LLMProcessor(beam.DoFn):
>>
>> def __init__(self, key):
>> """Initiate the OAI API client."""
>> self.client = OpenAI(
>> api_key=key
>> )
>>
>> def process(self, element: str):
>> logging.info(f'Input elements:{element}')
>>
>> return self.client.do_some_processing(..)
>>
>>
>> On Thu, Mar 27, 2025 at 8:48 AM Radek Stankiewicz via user <
>> user@beam.apache.org> wrote:
>>
>>> hi Sofia,
>>>
>>> here you have nice example
>>> https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/custom_remote_inference.ipynb
>>>
>>> where CloudVisionModelHandler is custom code that can invoke any client
>>> library.
>>> you can pass the key as one of the constructors
>>> to CloudVisionModelHandler or you can load it from any preferred secret
>>> managers e.g.
>>>
>>> def load_model(self):
>>> """Initiate the OAI API client."""
>>> client = OpenAI(
>>>
>>> api_key=client.access_secret_version(name="OPENAI_API_KEY").data.decode('UTF-8')
>>> )
>>> return client
>>>
>>> def run_inference(self, batch, oai_client, inference):
>>> response = oai_client.responses.create( # your LLM magic goes here.
>>> [..]
>>>
>>> using env variables won't work here as like you've noticed, beam is
>>> running on multiple machines and you can't set env variables there.
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Mar 27, 2025 at 9:22 AM Sofia’s World <mm...@gmail.com>
>>> wrote:
>>>
>>>> Hello
>>>> presumably is possible to kick off a beam process that invokes an LLM>?
>>>> but the only issue i have is how/where do i store the OpenAI key for
>>>> example
>>>>
>>>> in my c current colab/pc setup i have the key configured in my
>>>> environment.. but Beam will run
>>>> on multiple machines... how do i configure the OPENAI_KEY for example?
>>>>
>>>> thanks and regards
>>>> Marco
>>>>
>>>
Re: calling an LLM from a Beam function
Posted by Radek Stankiewicz via user <us...@beam.apache.org>.
hey Marco,
In your case, as your model inference is remote and has a custom handler,
the only difference is that RunInference transform is adding batching
before invoking the handler.
Today this handler is pretty simple but I would imagine that this
RunInference in the future may introduce certain optimizations, like async
calls which today you would have to develop yourself in parDo.
Radek
On Tue, Apr 1, 2025 at 6:51 AM Sofia’s World <mm...@gmail.com> wrote:
> Hi Radek
> uhm, how does that differ from creating something like this?
> am i missing something?
>
> Kind regards
> Marco
>
>
> class LLMProcessor(beam.DoFn):
>
> def __init__(self, key):
> """Initiate the OAI API client."""
> self.client = OpenAI(
> api_key=key
> )
>
> def process(self, element: str):
> logging.info(f'Input elements:{element}')
>
> return self.client.do_some_processing(..)
>
>
> On Thu, Mar 27, 2025 at 8:48 AM Radek Stankiewicz via user <
> user@beam.apache.org> wrote:
>
>> hi Sofia,
>>
>> here you have nice example
>> https://github.com/apache/beam/blob/master/examples/notebooks/beam-ml/custom_remote_inference.ipynb
>>
>> where CloudVisionModelHandler is custom code that can invoke any client
>> library.
>> you can pass the key as one of the constructors
>> to CloudVisionModelHandler or you can load it from any preferred secret
>> managers e.g.
>>
>> def load_model(self):
>> """Initiate the OAI API client."""
>> client = OpenAI(
>>
>> api_key=client.access_secret_version(name="OPENAI_API_KEY").data.decode('UTF-8')
>> )
>> return client
>>
>> def run_inference(self, batch, oai_client, inference):
>> response = oai_client.responses.create( # your LLM magic goes here.
>> [..]
>>
>> using env variables won't work here as like you've noticed, beam is
>> running on multiple machines and you can't set env variables there.
>>
>>
>>
>>
>>
>> On Thu, Mar 27, 2025 at 9:22 AM Sofia’s World <mm...@gmail.com>
>> wrote:
>>
>>> Hello
>>> presumably is possible to kick off a beam process that invokes an LLM>?
>>> but the only issue i have is how/where do i store the OpenAI key for
>>> example
>>>
>>> in my c current colab/pc setup i have the key configured in my
>>> environment.. but Beam will run
>>> on multiple machines... how do i configure the OPENAI_KEY for example?
>>>
>>> thanks and regards
>>> Marco
>>>
>>