Langchain-Beam : Integrating LLMs and embedding models into ETL pipelines

In this talk, I will introduce Langchain-Beam, its an open source library that integrates Langchain with Apache Beam. This library provides transforms to integrate LLMs and embedding models into Apache Beam ETL pipelines.

Talk will cover:

1. Overview of Apache Beam programming model.

2. How to use Langchain-Beam's transforms as a stage in a Beam pipeline and use the models to process input data within the pipeline

3. Run example pipeline on GCP dataflow

Repository: https://github.com/Ganeshsivakumar/langchain-beam

Docs: https://ganeshsivakumar.github.io/langchain-beam/docs/intro/

Example Langchain-Beam pipelines: https://github.com/Ganeshsivakumar/langchain-beam/tree/main/example/langchain-beam-example/src/main/java/com/langchainbeam/example