1:03:03
✅ System Design 13: Design Social Media like Facebook | Instagram | Feeds Generation | HLD | LLD
InterviewWithBunny
Overview
This video explains how to design a social media platform like Facebook or Instagram, focusing on feed generation. It covers gathering functional and non-functional requirements, identifying core entities, designing APIs, and performing high-level and low-level system design. Key aspects include handling user authentication, content posting, feed generation using fan-out models (push and pull), and optimizing for scale and availability using microservices, Kafka, and caching strategies.
How was this?
Save this permanently with flashcards, quizzes, and AI chat
Chapters
- Core functional requirements include user signup/login, creating/sharing posts (text, image, video), following other users, liking/commenting on posts, and viewing a personalized feed.
- Out-of-scope features are stories, uploads, and messaging/chat functionalities.
- Non-functional requirements focus on a massive scale (500 million daily active users) and prioritizing high availability over strong consistency, leading to an eventually consistent system.
- Acceptable latency for uploading a post is around 500 milliseconds.
Clearly defining requirements is crucial for scope management and ensuring the system meets user needs and performance expectations.
Users can follow each other (like Instagram), but friend requests (like Facebook) are out of scope for simplicity.
- Key entities identified are User, Post, Follower/Friend, Like/Comment, and Feed.
- API endpoints are designed for user onboarding (register, login, get/update metadata), post management (create, get, edit, delete), feed retrieval (paginated), and user interactions (like, unlike, comment, follow, unfollow).
- API design often follows a one-to-many relationship with functional requirements.
- Feed retrieval APIs use pagination to handle potentially millions of posts from numerous followed users.
Structuring the system around core entities and defining clear APIs provides a blueprint for how different parts of the application will interact.
A `GET /feed` API endpoint is designed to retrieve a paginated list of posts relevant to the logged-in user.
- A microservices architecture is necessary due to the large scale.
- Key services include User Service (onboarding, authentication), Content Service (handling posts, storing metadata in Post DB and media in Blob Storage/S3), Follower Service (managing relationships), Engagement Service (likes, comments), and Feed Service (generating user feeds).
- A Load Balancer and API Gateway are essential for routing traffic and handling authentication/authorization.
- The Feed Service initially relies on fetching follower lists and then querying the Post DB, which is inefficient at scale.
This layer outlines the major components of the system and their basic responsibilities, setting the stage for detailed design.
The Content Service persists textual posts in a 'Post' database and media files (images/videos) in an S3 bucket.
- User Service uses a relational database (like PostgreSQL) for storing user metadata (ID, username, email, password) and handles authentication by returning JWT tokens.
- Content Service is enhanced with Kafka for buffering and processing uploads. A Moderator Service sanitizes content, directing valid posts to 'filtered_post' and invalid ones to 'blog_post' topics.
- A PostConsumer Service ingests filtered posts into a Cassandra database (chosen for write efficiency) for metadata and potentially S3 for media.
- Video uploading follows a chunking and encoding process similar to OTT platforms.
Detailed design choices for services like content handling, especially incorporating Kafka and Cassandra, address scalability and content moderation challenges.
Uploaded content first goes into a Kafka topic ('raw_post'), then is processed by a Moderator Service, and finally ingested into a Cassandra database if deemed valid.
- Follower Service manages user relationships, storing follower/following IDs and status in a PostgreSQL database. Graph databases are suggested for complex relationship queries (e.g., friend-of-a-friend suggestions).
- Engagement Service uses Kafka to buffer high volumes of likes and comments before they are processed and stored.
- Comments are stored in a Cassandra database, while likes are stored in PostgreSQL, with aggregated like counts potentially updated in the comment data.
- A separate service might be needed to aggregate likes from the PostgreSQL table and update the comment database.
Implementing robust follower and engagement tracking, especially using Kafka for buffering, ensures data integrity and handles high interaction loads.
Likes and comments are first sent to a Kafka broker, then consumed by the Engagement Service and stored in appropriate databases (Cassandra for comments, PostgreSQL for likes).
- Directly joining follower and post data at runtime is infeasible for large-scale feeds; pre-computation is required.
- The 'fan-out' model is used: Push model for users with few followers, Pull model for celebrities with many followers.
- A Feed Service interacts with a Feed Cache (e.g., Redis) for fast read access to pre-computed feeds.
- A Fan-out Service, triggered by new posts, updates the feeds of followers (push model) by writing to Kafka, which is then consumed to update the Feed DB and Feed Cache.
- For celebrities (pull model), feeds are generated on demand or via a backfilling process.
The fan-out model and caching are critical for efficiently generating personalized feeds at scale, providing a smooth user experience.
When a user posts, a Fan-out Service identifies their followers and pushes the post information through Kafka to update the feeds of those followers in the Feed Cache.
- Feed Cache has limitations; a TTL and limited number of posts (e.g., 50-100) per user are implemented.
- A Backfilling process is initiated when a user exhausts their cached feed, making API calls to generate more content.
- Post Materialization service pre-computes and stores recent user activities (e.g., last 100 posts) in a Redis cache.
- The Backfilling service uses the Follower DB and Redis cache to quickly retrieve recent posts from followed users, then writes new feed data back to Kafka for the Fan-out Service to update the cache.
- A Follower Cache can further optimize by storing only the most interacted-with users, reducing the scope of feed generation.
Strategies like backfilling and caching recent activities ensure infinite scrolling and a responsive feed, even when initial caches are depleted.
When a user scrolls past their pre-loaded 100 posts, a Backfilling service is triggered to fetch more recent posts from their followed users via Redis and update the feed.
Key takeaways
- Social media feed generation is complex and requires pre-computation using fan-out models rather than real-time joins.
- Prioritizing availability over strong consistency is essential for large-scale social platforms.
- Microservices architecture, Kafka for asynchronous processing, and caching (Redis) are fundamental for scalability and performance.
- Choosing the right database (e.g., Cassandra for write-heavy loads, PostgreSQL for relational data) is critical for different system components.
- Content moderation is a necessary step in the content pipeline, using services to filter invalid or malicious posts.
- The fan-out model has variations (push vs. pull) optimized for users with few versus many followers.
- Backfilling and caching strategies are key to achieving an 'infinite scroll' experience in feeds.
Key terms
Functional RequirementsNon-Functional RequirementsMicroservices ArchitectureAPI GatewayLoad BalancerKafkaCassandraPostgreSQLRedisBlob Storage (S3)Fan-out Model (Push/Pull)Feed CacheBackfillingPost MaterializationContent ModerationEventually ConsistentJWT Token
Test your understanding
- Why is prioritizing availability over strong consistency important for a social media platform like Facebook or Instagram?
- How does the fan-out model address the challenge of generating personalized feeds at a large scale?
- What is the role of Kafka in the content upload and feed generation pipelines, and why is it used?
- Explain the difference between the push and pull models within the fan-out strategy and when each would be applied.
- How do caching mechanisms (like Feed Cache and Redis for recent activities) contribute to the performance and user experience of the social media feed?