FOSS-Powered Observability: Building an Open Source Logging Pipeline with Fluent Bit, Fluentd, Kafka, and OpenSearch on Kubernetes

OpenSearch is a powerful open-source search and analytics engine, but running it effectively on Kubernetes requires a well-structured approach. In this session, I will explore how to deploy, configure, and operate OpenSearch in a Kubernetes-native way. I will dive into best practices for maintaining OpenSearch clusters, optimizing performance, and handling scaling challenges. I will also introduce the openseaarch-operator opensource project.

Additionally, I will cover how to ship Kubernetes logs to OpenSearch, enabling centralized log management for a single cluster and also across multiple clusters. I will discuss scalable log ingestion strategies involving off the shelf as well as custom made tools. The session will also include practical demonstrations on building insightful dashboards for observability and setting up alerts to proactively monitor system health. I will also touch upon related opensource projects like fluentbit, fluentd, kafka and others while going through different architecture iterations

This talk will provide a comprehensive guide to managing OpenSearch effectively within Kubernetes.

• Deploying opensearch for log and event management

• Operating opensearch on kubernetes

• Creating a single cluster and multi-cluster, multi-cloud architecture for your organization's logging needs

• Log ingestion using tools like fluentbit and fluentd

• How opensource tools are all you need for the most resilient and highly available needs