Most shared knowledge around MLOps is built on an unspoken assumption: models are massive, deployed in the cloud, and generalized for all users, perhaps with a bit of LoRA-based fine-tuning for personalization. But what happens when your architecture doesn't fit that mold?
When you ask a model something... it answers. you don't know why and the company that built it doesn't fully know either. the only people who can audit it are the ones inside the building that built it. seven companies are racing to build AGI. they've agreed on one thing: you don't get to look inside.
It’s that ‘the company that built it doesn't fully know either’ idea which sticks out, and I’d argue not just at the deeper interpretability level, but also at a higher level engineering layer where the ability to track your models across lifecycles isn't fully solved. 'what data was this model trained on?', 'did someone update the model being served in this usecase?', 'how can we verify the model that ends up being served is the right one?' and a many other questions which we've solved for first-hand as we go.
This talk aims to shed a light on different, and less common kinds of machine learning and what these standard solutions end up looking like in those cases. And by looking at the solutions we build in these different usecases, and looking at what they have in common we’re able to break it down into fundamental challenges all such systems must solve for after tailoring them to their particular scenario. It draws upon the speaker’s experience simultaneously working across and maintaining model training -> deployment lifecycles across multiple contexts: from training one neural network to serve to millions vs training a personalised one for them all, and how the processes and tools that emerge to manage and help keep track work out.
As more AI startups emerge and the lines between research ML and production engineering blur, relying on literature biased toward centralized LLMs turns model maintenance into something like an 'accident-prone area'. This talk intends to share the perspective from working on the edges (no pun intended), building our solutions from scratch and comparing notes with the mainstream solutions to arrive at the common fundamentals one must always solve for. By looking at the solutions required in these less common, high-cardinality use cases, we can strip away the mainstream biases and isolate the fundamental challenges every ML system must solve.
This talk plans to give a framework to classify training lifecycles using 4 axes and what kind of workflow/tool fits each best.
