Lightning Talk Intermediate

AI Documentation Observability: Measuring Trust in the Age of AI

Screening

Session Description

AI is changing how developers use documentation. Instead of reading documentation pages, many now ask AI assistants and agents for answers. While AI makes information easier to access, it also raises an important question: How can we trust the answers?

This talk introduces the idea of AI Documentation Observability—applying observability principles to documentation used by AI systems. Rather than treating documentation as static text, we can measure how it is retrieved, referenced, and used to generate answers.

We’ll explore practical ways to monitor documentation quality by tracking source attribution, confidence, hallucinations, user feedback, and continuous evaluation. Using real-world examples and open source tools such as Langfuse / MLFLow and OpenTelemetry, we’ll see how teams can identify problems, improve documentation quality, and build greater trust in AI-assisted responses.

Key Takeaways

Attendees will leave with practical ideas for making documentation not only easier for AI to consume, but also easier to verify, monitor, and continuously improve.

References

https://drive.google.com/file/d/1vBWwlfmyFUdcaHVmgpuxTu_BXGmU5E7e/view?usp=sharing

Session Categories

Engineering practice - productivity, debugging

Which track are you applying for?

Documentation & Technical Writing

Speakers

Shivraj Patil Principal Software Engineer | Red Hat

Shivraj Patil is a Principal Support Engineer at Red Hat with over a decade of experience helping customers solve complex enterprise challenges across Linux, OpenShift, and cloud-native technologies. Alongside his technical role, he has led global support teams, mentored engineers, and driven operational excellence through automation and developer enablement.
Recently transitioning from people leadership back into a principal engineering role, Shivraj has been exploring how AI is transforming software development. His current work focuses on building open-source, AI-assisted developer tools, self-hosted platforms, and Android applications that demonstrate how modern engineers can rapidly turn ideas into production-ready software.
Outside of work, he is an active contributor to the open-source community, volunteers with DevConf.IN, and enjoys building side projects, motorcycle touring, and photography.

Veerkumar Patil Principal Software Quality Engineer | Red Hat

VeerkKumar Patil is a Principal Software Quality Engineer at Red Hat with extensive experience in quality engineering, test automation, and developer tooling. As a QE Lead, he specializes in building scalable test automation frameworks, AI-powered quality engineering workflows, and modern software delivery practices.

His expertise spans Agentic AI, AI evaluations, harness engineering, Cypress, Playwright, Selenium, API testing, containers, and DevOps. Passionate about applying AI to solve real engineering problems, he focuses on improving developer productivity and building intelligent engineering workflows that enhance software quality throughout the development lifecycle.

An advocate for practical automation and open-source technologies, he enjoys experimenting with emerging AI capabilities and sharing real-world lessons that help engineering teams build and deliver software with greater confidence.