Integrating LLMs as PTransform in beam pipeline

[Ganesh <ga...@gmail.com>]

Hi All,

I've been working on a custom Ptransform that integrates large language
models as a PTransform in an Apache beam pipeline using Langchain.

The aim of this integration is to leverage the capabilities of LLMs for
data processing and transformations of unstructured data.
It treats language models as PTransform block in a data pipeline. The
transform takes a prompt that has instructions on what to do with the input
element, The LLM applies the instruction on the element and the model's
output is returned as a Pcollection.

It uses langchian to interact with the models as it provides a common
interface for interacting with multiple model providers.
I'm excited to share langchian-beam Ptransform with the beam community.
Would like to hear your thoughts and feedback.

Repository link - https://github.com/Ganeshsivakumar/langchain-beam


Thanks,
Ganesh.

Re: Integrating LLMs as PTransform in beam pipeline

[Reza Ardeshir Rokni <ra...@gmail.com>]

Hi,

Thank you for contributing! You may also want to look at working with
RunInference, for example:

https://beam.apache.org/releases/pydoc/current/apache_beam.ml.inference.vllm_inference.html#module-apache_beam.ml.inference.vllm_inference

Cheers

Reza

On Sun, 1 Dec 2024 at 14:55, LDesire <tw...@icloud.com> wrote:

> Hi Ganesh,
> I'm very excited to see this new PTransform for Beam's Java SDK.
> Thank you for sharing this valuable contribution.
>
> Best regards
>
>
>
> 2024. 12. 1. 오후 11:46, Ganesh <ga...@gmail.com> 작성:
>
> Hi All,
>
> I've been working on a custom Ptransform that integrates large language
> models as a PTransform in an Apache beam pipeline using Langchain.
>
> The aim of this integration is to leverage the capabilities of LLMs for
> data processing and transformations of unstructured data.
> It treats language models as PTransform block in a data pipeline. The
> transform takes a prompt that has instructions on what to do with the input
> element, The LLM applies the instruction on the element and the model's
> output is returned as a Pcollection.
>
> It uses langchian to interact with the models as it provides a common
> interface for interacting with multiple model providers.
> I'm excited to share langchian-beam Ptransform with the beam community.
> Would like to hear your thoughts and feedback.
>
> Repository link - https://github.com/Ganeshsivakumar/langchain-beam
>
>
> Thanks,
> Ganesh.
>
>
>

Re: Integrating LLMs as PTransform in beam pipeline

[LDesire <tw...@icloud.com>]

Hi Ganesh, 
I'm very excited to see this new PTransform for Beam's Java SDK. 
Thank you for sharing this valuable contribution. 

Best regards



> 2024. 12. 1. 오후 11:46, Ganesh <ga...@gmail.com> 작성:
> 
> Hi All, 
> 
> I've been working on a custom Ptransform that integrates large language models as a PTransform in an Apache beam pipeline using Langchain.
> 
> The aim of this integration is to leverage the capabilities of LLMs for data processing and transformations of unstructured data. 
> It treats language models as PTransform block in a data pipeline. The transform takes a prompt that has instructions on what to do with the input element, The LLM applies the instruction on the element and the model's output is returned as a Pcollection. 
> 
> It uses langchian to interact with the models as it provides a common interface for interacting with multiple model providers. 
> I'm excited to share langchian-beam Ptransform with the beam community. Would like to hear your thoughts and feedback. 
> 
> Repository link - https://github.com/Ganeshsivakumar/langchain-beam
> 
> 
> Thanks, 
> Ganesh.