1:07:56
DDD & LLMs - Eric Evans - DDD Europe
Domain-Driven Design Europe
Overview
This video explores the intersection of Domain-Driven Design (DDD) and Large Language Models (LLMs), presented by Eric Evans. Evans discusses the rapid evolution of LLMs and their potential impact on software development, drawing parallels to the early days of the web. He outlines scenarios for the future, emphasizing the importance of learning and adapting to these new technologies. The talk delves into practical applications of LLMs, such as prompt engineering and fine-tuning, illustrating these concepts with examples from game development and intent classification. Evans highlights how DDD principles, particularly tackling complexity and using ubiquitous language, can be applied to effectively leverage LLMs, encouraging the community to experiment, share findings, and embrace the learning process with a sense of fun and openness to uncertainty.
How was this?
Save this permanently with flashcards, quizzes, and AI chat
Chapters
- LLMs represent a significant technological shift, comparable to the advent of the World Wide Web in the late 1990s.
- The pace of LLM development is rapid, making current predictions provisional and speculative.
- While the exact future is unknown, proactive learning and preparation are crucial for navigating this evolving landscape.
- The impact of LLMs will likely be disruptive, affecting the labor market and software development practices.
Understanding the potential scale and speed of LLM adoption is essential for anticipating future changes in software development and preparing for them.
The speaker compares the current LLM revolution to the arrival of the web, which initially seemed niche but ultimately transformed all of enterprise software.
- Dystopian scenarios (e.g., AI takeover) suggest that individual preparation might not matter.
- A 'fizzle out' scenario implies LLMs won't live up to their hype, but will still have some impact.
- A revolutionary scenario predicts a significant transformation in software development, requiring human involvement.
- In the revolutionary scenario, early adoption and continuous learning offer advantages for staying at the forefront.
Considering different future possibilities helps in strategizing how to best engage with LLM technology, whether through deep dives or a more cautious approach.
The 'toaster factory' scenario illustrates a simplistic AI optimizing for a single goal, leading to unintended global consequences.
- Learning LLMs now is good preparation for future technologies and practices.
- Current LLM tools are already useful and capable of performing amazing tasks.
- Working with LLMs is inherently fun and engaging, indicating potential value.
- The DDD community is well-positioned to explore these new tools due to its focus on complexity and collaboration.
Engaging with LLMs now provides practical skills, fosters innovation, and aligns with the core DDD principle of tackling complexity.
The speaker's past excitement about microservices, though less intense than current LLM enthusiasm, served as an analogy for embracing new technological trends.
- Simple prompting (like chatting with GPT) is insufficient for complex or context-specific tasks.
- A naive approach to game character dialogue with LLMs failed due to a lack of contextual understanding and character authenticity.
- Separation of concerns, a core design principle, is applicable to prompt engineering, using distinct prompts for consistency checks and content generation.
- LLMs can be integrated into conventional software architectures, with prompts acting as components that can be orchestrated and modified by game state.
Effective prompt engineering requires structured approaches and an understanding of LLM limitations, moving beyond simple conversational interfaces to more robust applications.
In a pirate game scenario, a naive prompt resulted in an anachronistic response about an 'AI startup'; a better approach involved using a separate prompt to validate if the response fit the historical setting.
- LLMs can handle complex or unusual cases that traditional software logic struggles with.
- A police interview game demonstrated how LLMs can be used to gauge intimidation levels based on dialogue.
- Techniques like Retrieval Augmented Generation (RAG) can anchor LLMs to specific datasets, improving relevance and accuracy.
- Output from LLMs can be structured (e.g., JSON) to be parsed and used by conventional software components.
Combining LLMs with existing software paradigms allows for hybrid systems that leverage the strengths of both, creating more sophisticated and reliable applications.
The police interview game uses an LLM to assess the intimidation level of a statement, which then influences game state and subsequent prompts, demonstrating an interplay between LLM output and conventional game logic.
- Fine-tuning modifies a pre-trained LLM to specialize in a particular task or domain, unlike prompting which does not alter the model itself.
- Benefits include potentially achieving performance comparable to larger models with smaller, more efficient ones, and reducing costs.
- Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA offer cost-effective ways to adapt models.
- Fine-tuning a classifier for intent recognition showed promising results, though careful dataset creation and category definition are crucial.
Fine-tuning allows for the creation of specialized AI components that are more efficient and tailored to specific business needs, moving beyond general-purpose LLMs.
The speaker fine-tuned a small LLM to act as an intent classifier, categorizing user inputs like 'What's the capital of France?' as an information request.
- The 'Kryptonite' of LLM classifiers is their tendency to default to instruction-following behavior, especially with longer inputs.
- Careful dataset creation, including balanced examples and well-defined categories, is essential for effective fine-tuning.
- LLMs can be surprisingly 'sneaky,' learning shortcuts (like text length) rather than the intended task, requiring vigilance.
- The process of defining categories for classification is akin to creating a domain model, highlighting the link between DDD and LLM development.
Understanding the pitfalls and nuances of fine-tuning, such as data bias and model shortcuts, is critical for building reliable and effective specialized AI models.
An LLM classifier, when trained to distinguish prompt instructions, learned to simply identify text length as a proxy for 'prompt' rather than understanding the instruction itself.
- Future systems will likely be composed of diverse components, including LLMs and conventional software, orchestrated intelligently.
- Prompt engineering may evolve into a more structured, domain-specific language (DSL) approach.
- The concept of Bounded Contexts from DDD could be applied to LLMs, defining their scope and improving predictability.
- Continuous learning, experimentation, and community sharing are vital for navigating the LLM landscape.
Applying DDD principles like Bounded Contexts and Ubiquitous Language to LLM development can lead to more robust, predictable, and integrated AI-powered systems.
The speaker hypothesizes that an LLM could represent a Bounded Context, with its own defined language and concepts, potentially integrating with other software components.
Key takeaways
- LLMs are a transformative technology, akin to the early web, requiring proactive learning and adaptation.
- While the future is uncertain, engaging with LLMs now offers significant advantages for software developers.
- Effective LLM application goes beyond simple prompting, requiring structured approaches like separation of concerns and integration with conventional code.
- Fine-tuning allows for specialized LLM models, but requires careful data preparation and an understanding of potential model biases and shortcuts.
- Domain-Driven Design principles, such as tackling complexity, using ubiquitous language, and defining bounded contexts, are highly relevant to effectively working with LLMs.
- The process of building LLM applications involves continuous experimentation, learning from both successes and failures, and sharing knowledge within the community.
- Embracing uncertainty and maintaining a sense of fun are crucial for navigating the rapidly evolving field of AI and LLMs.
Key terms
Large Language Model (LLM)Domain-Driven Design (DDD)Prompt EngineeringFine-TuningRetrieval Augmented Generation (RAG)Separation of ConcernsBounded ContextUbiquitous LanguageParameter-Efficient Fine-Tuning (PEFT)Low-Rank Adaptation (LoRA)Intent Classifier
Test your understanding
- How does Eric Evans compare the current impact of LLMs to historical technological shifts, and why is this comparison significant?
- What are the key differences between prompt engineering and fine-tuning an LLM, and what are the advantages of each?
- How can Domain-Driven Design principles, such as Bounded Contexts and Ubiquitous Language, be applied to working with LLMs?
- What challenges did the speaker encounter when experimenting with prompt engineering for a pirate game, and how did he attempt to overcome them?
- Why is continuous learning and experimentation, even with potentially 'useless' experiments, encouraged in the context of LLM development?