36:12
FlyRank Internship ML: What ML Actually Solves Here (week 2) w/ Mirza Asceric
FlyRank AI
Overview
This video explains how machine learning (ML) is applied to solve real-world problems, particularly within the context of SEO and content management at FlyRank. It differentiates AI, ML, and Large Language Models (LLMs), emphasizing that ML is used for pattern recognition and prediction from data, not just for immediate information retrieval. The video details the process of framing ML problems, distinguishing between supervised and unsupervised learning, and introduces practical techniques for using AI tools effectively. It also highlights the critical importance of data quality, problem definition, and generalization in building successful ML models, using concrete examples from FlyRank's operations.
How was this?
Save this permanently with flashcards, quizzes, and AI chat
Chapters
- Machine learning is not always necessary; simpler tools like dashboards or rules suffice for immediate data needs.
- The core skill is knowing when to apply ML versus other methods for problem-solving.
- This week's focus is on transforming search questions into specific ML tasks, defining predictions, and establishing evaluation metrics.
- The internship involves defining problems, choosing ML approaches, and hands-on implementation with real data.
Understanding when and why to use ML is crucial for efficient problem-solving, preventing the over-application of complex tools when simpler solutions are adequate.
A simple rule correctly identified the right pages 24% of the time, while a real ML model was three times better, illustrating the potential of ML when applied correctly.
- Artificial Intelligence (AI) is a broad term for systems that act intelligently, including rule-based systems.
- Machine Learning (ML) is a subset of AI where programs learn patterns from data to make predictions about future behavior.
- Large Language Models (LLMs) like ChatGPT are a specific type of ML model, often used off-the-shelf by prompting.
- The internship focuses on building custom ML models using proprietary search data, rather than just using pre-trained LLMs.
Clarifying these distinctions helps learners understand the specific capabilities and limitations of each technology, guiding them to choose the right tool for the job.
A system with hand-written rules can be considered AI, while ML requires learning from examples, and LLMs are a specific, often pre-trained, implementation within ML.
- Dashboards are for understanding current data states.
- Rules are for situations where the logic is known and can be explicitly defined (e.g., a temperature threshold).
- Machine learning is required for predicting future states or identifying complex patterns that cannot be easily codified into rules.
- FlyRank uses hand-written rules for immediate content flagging (e.g., 'needs CTR fix'), but ML is needed to find the underlying patterns causing poor performance.
This distinction helps learners identify whether a problem requires a simple rule-based solution or the predictive power of machine learning.
A rule can determine if it's currently cold (e.g., below 20°C), but predicting next week's weather requires ML and historical data analysis.
- Supervised learning involves models trained on data where each example has a known correct answer or label.
- Unsupervised learning involves models that find patterns and group similar data points without pre-existing labels.
- FlyRank's 'refresh scoring,' 'ranking signals,' and 'CTR opportunities' lanes are examples of supervised learning.
- FlyRank's 'archetype clustering' lane is an example of unsupervised learning, where the model groups similar keywords.
Knowing the difference between supervised and unsupervised learning helps in selecting the appropriate ML approach based on the nature of the available data and the problem to be solved.
Supervised learning predicts a 'click trend' (falling or rising) for new pages based on historical data with known trends; unsupervised learning groups keywords by traffic and conversions into categories like 'navigational,' 'commercial,' or 'informational'.
- Overfitting occurs when a model memorizes the training data instead of learning generalizable patterns.
- An overfit model performs well on seen data but poorly on new, unseen data.
- This can happen with small datasets or when models are too complex for the data.
- Generalization is the goal: the model should perform well on data it has never encountered before.
Avoiding overfitting ensures that ML models are robust and reliable in real-world applications, providing accurate predictions on new data, not just data they were trained on.
A student memorizing answers from last year's exam will fail a different exam the next year; similarly, an ML model that memorizes specific training examples will fail when presented with new data.
- Effective ML starts with clearly defining the problem: what are we solving, and what does success look like?
- Key elements include defining the goal, the decision to be improved, the ML task type (e.g., classification, scoring), the target variable, and the evaluation metric.
- A proxy target is used when the ideal target is not directly measurable (e.g., predicting 'worth a refresh slot' using a measurable stand-in).
- The metric 'precision at fifty' measures how many of the top 50 recommended items actually meet the criteria.
A structured problem-framing process ensures that the ML model is aligned with business objectives and that its performance is measured meaningfully.
For FlyRank's 'refresh scoring' lane, the goal is to earn more from existing content. The decision is which pages get one of the 50 monthly refresh slots. The task is scoring, the target is 'worth a refresh slot' (using a proxy), and the metric is precision at fifty.
- Use AI tools to ask clarifying questions, identify knowledge gaps, and gather context before implementation.
- Engage in objective discussions with AI, asking it to defend its choices and checking them against data dictionaries.
- Always seek a second opinion from the same or a different AI model to validate findings and analysis.
- AI can help understand datasets, identify data quality issues (like missing values or NULLs), and suggest appropriate handling methods.
Strategic use of AI tools can significantly enhance the efficiency and effectiveness of the ML development process, from initial understanding to final validation.
Asking an AI to explain 'overfitting like a junior' and then following up with specific questions about terms like 'depth' helps clarify complex concepts and tailor them to the specific problem.
- ML models are highly dependent on data quality; imperfect data is the norm.
- Learners must understand the data they have, its limitations (e.g., missing values, NULLs), and how to handle them through cleaning and normalization.
- The ultimate goal of ML is generalization – performing well on unseen data.
- The comprehensive MD file in the repository serves as a golden nugget for understanding ML concepts, framing problems, and validating models.
Recognizing and addressing data quality issues is paramount, as even the best model will fail if trained on or applied to flawed data.
Handling NULL values in boolean columns by filtering with 'IS TRUE' instead of 'equals false' is a critical data cleaning step that affects analysis outcomes.
Key takeaways
- Machine learning is a tool to be used strategically when simpler methods are insufficient for prediction or complex pattern identification.
- Clearly defining the problem, objective, and success metrics is more critical than the specific ML algorithm chosen.
- Understanding the difference between AI, ML, and LLMs helps in selecting the appropriate technology for a given task.
- Supervised learning requires labeled data for training, while unsupervised learning finds inherent structures in unlabeled data.
- Generalization is the primary goal of ML; models must perform well on new, unseen data, not just the data they were trained on.
- Data quality is foundational; rigorous data cleaning, understanding its limitations, and proper handling of missing values are essential for reliable ML outcomes.
- AI tools can be powerful assistants for learning, problem framing, and validation, but critical thinking and decision-making remain the human's responsibility.
- The provided MD file is an invaluable resource for navigating ML concepts, problem framing, and best practices throughout the internship.
Key terms
Machine Learning (ML)Artificial Intelligence (AI)Large Language Models (LLMs)Supervised LearningUnsupervised LearningOverfittingGeneralizationClick-Through Rate (CTR)ImpressionsProxy TargetPrecision at FiftyData NormalizationNULL values
Test your understanding
- When is machine learning the appropriate tool for a problem, and when might a simpler approach suffice?
- How does machine learning differ from broader artificial intelligence and specific LLMs like ChatGPT?
- What is the fundamental difference between supervised and unsupervised machine learning, and how do these apply to the internship lanes?
- Why is generalization the ultimate goal in machine learning, and what are the risks associated with overfitting?
- Describe the key steps involved in framing an ML problem, from defining the goal to establishing evaluation metrics.