27:52
Rosa Gutiérrez - Solid Queue internals, externals and all the things in between - Rails World 2024
Ruby on Rails
Overview
This video details the development and internals of Solid Queue, a new background job processing system designed to be the default in Rails 8. The speaker, Rosa Gutiérrez, explains the motivations behind creating Solid Queue, highlighting the limitations of existing solutions like Resque and Delayed Job. The talk covers the architectural decisions, database interactions, performance optimizations, and unique features such as concurrency control and batch operations. It emphasizes how Solid Queue was built through a process of "promises-driven development" and extensive real-world testing within the Hey.com platform, leading to its robust and battle-tested 1.0 release.
How was this?
Save this permanently with flashcards, quizzes, and AI chat
Chapters
- Solid Queue was announced at Rails World 2023, marking a significant development in Rails background job processing.
- The project was initially motivated by a need to overcome limitations in existing gems like Resque, which could lead to job loss if workers died.
- The speaker employed 'promises-driven development,' using a conference announcement to boost motivation and ensure project delivery.
- Several internal and forked gems were used to address specific needs before Solid Queue's unified approach.
Understanding the historical context and initial problems that led to Solid Queue's creation helps appreciate its design choices and the value it brings to the Rails ecosystem.
The speaker mentions a video of David Heinemeier Hansson announcing Solid Queue at Rails World 2023, which served as a catalyst for focused development.
- Existing solutions often risked job loss due to worker crashes (e.g., Resque).
- The need for running multiple workers in separate processes was met by gems like Resque Pool.
- Scheduled and delayed jobs required dedicated solutions like Resque Scheduler.
- Sequential job execution was necessary to prevent application logic conflicts when jobs needed to run in a specific order.
- Scheduling millions of jobs far in advance caused Redis memory exhaustion, necessitating a database-backed approach for scheduled jobs.
This chapter outlines the specific pain points with prior job systems, demonstrating the complex requirements that Solid Queue was built to solve comprehensively.
The problem of Redis memory exhaustion when scheduling millions of jobs 30-90 days in advance, leading to the idea of using a database for future scheduled jobs.
- Solid Queue leverages relational databases, inspired by Solid Cache, for job storage, simplifying integration with Active Record.
- It aims for a simpler, more understandable codebase compared to some complex existing backends.
- A key challenge in database-backed job systems is worker contention; Solid Queue uses `SELECT FOR UPDATE SKIP LOCKED` to efficiently claim jobs without blocking other workers.
- To handle worker failures, a process registry with heartbeats tracks active workers, allowing claimed jobs to be released if a worker dies.
This section explains the core technical innovations, particularly how Solid Queue efficiently manages job claiming and worker failures using database features and a process registry.
The use of `SELECT FOR UPDATE SKIP LOCKED` in PostgreSQL and MySQL 8+ to allow multiple workers to fetch jobs concurrently without blocking each other.
- The system separates jobs into different tables (ready, scheduled, claimed) to keep the polling table small and queries fast.
- Dispatcher agents move scheduled jobs to the ready table, and workers move jobs from ready to claimed.
- Polling queries are highly optimized using specific indexes on the `ready_executions` table.
- Queries are designed to handle single queues or wildcard selections efficiently, minimizing examined rows and database load.
- The system achieves very low latency for polling queries, examining an average of less than one row per poll.
Efficient polling is crucial for background job systems. This chapter details the database schema and indexing strategies that make Solid Queue performant at scale.
The use of two specific indexes on the `ready_executions` table: one for querying by a single queue name and another for wildcard queries, ensuring fast retrieval.
- Instead of strictly sequential jobs, Solid Queue implements 'concurrency controls' using semaphores.
- Limit checking is moved from polling to the enqueuing process to avoid touching optimized polling queries.
- A 'blocked execution' state is introduced for jobs that cannot immediately acquire a semaphore.
- This approach can increase write operations during enqueuing, especially for concurrency-controlled scheduled jobs.
- To handle the increased database load, Solid Queue recommends or defaults to a separate database for the job system.
This chapter addresses the complex requirement of managing job execution order and preventing overlaps, introducing a novel approach that balances application logic with system performance.
Using a semaphore concept where a job must 'gain' permission before being enqueued, ensuring that only a limited number of similar jobs run concurrently.
- Solid Queue 1.0 was released after extensive testing and running in production at Hey.com, processing millions of jobs daily.
- The system offers features like batch operations and a management dashboard (Mission Control).
- Migration from Resque to Solid Queue provided performance benefits, especially for batch enqueuing, without negatively impacting global response times.
- The project encourages community contributions, particularly for improving support and identifying issues with PostgreSQL and SQLite, as the primary development experience is with MySQL.
- Solid Queue aims to be a robust, well-tested default for Rails, built from real-world needs and scaled challenges.
This concluding section highlights Solid Queue's maturity, real-world performance gains, and the collaborative effort required to make it a reliable default for the Rails community.
Hey.com runs Solid Queue processing 20 million jobs per day on 74 VMs, demonstrating its capability to handle large-scale operations.
Key takeaways
- Solid Queue offers a robust, database-backed alternative to traditional background job processors like Resque, mitigating risks of job loss.
- Efficient job claiming is achieved through `SELECT FOR UPDATE SKIP LOCKED`, preventing worker contention and improving throughput.
- A process registry with heartbeats ensures that jobs claimed by failed workers are automatically released, maintaining system availability.
- Performance is optimized through careful database schema design, indexing, and highly efficient polling queries that minimize database load.
- Concurrency control, implemented via semaphores, allows for managing job execution limits at enqueue time, rather than during polling.
- Solid Queue is designed for scalability, recommending a separate database to handle the potential load from high-throughput job processing.
- The 1.0 release is battle-tested, having been developed and run in production at scale before public release, ensuring reliability.
Key terms
Solid QueueBackground JobsResqueDelayed JobWorkerPollingSELECT FOR UPDATE SKIP LOCKEDProcess RegistryHeartbeatSemaphoreConcurrency ControlBatch OperationsMission Control
Test your understanding
- What problem does `SELECT FOR UPDATE SKIP LOCKED` solve in Solid Queue's job claiming mechanism?
- How does Solid Queue ensure that jobs are not lost if a worker process crashes?
- Why is it important to keep the polling table small in a database-backed job system, and how does Solid Queue achieve this?
- What is the difference between Solid Queue's approach to sequential jobs and traditional methods, and why was this change made?
- What are the benefits of running Solid Queue on a separate database, and when might this be necessary?