More CPUs = Better? I tested 4 Programming Languages to Find Out

Arjay McCandless

7 chapters7 takeaways12 key terms5 questions

Overview

This video explores how adding more CPU cores affects the performance of applications built with different programming languages. The creator tested Python, Java, Go, and Rust by building identical applications and running them on servers with varying numbers of virtual CPUs (vCPUs). The goal was to measure how many simulated users each configuration could support before performance degraded, revealing that while more cores generally increase capacity significantly, the actual gains and efficiency depend heavily on the language's concurrency model and how effectively it utilizes the available processing power. The experiment also highlights that database and other system processes can become bottlenecks, limiting the benefits of more CPU cores for the application itself.

How was this?

Save this permanently with flashcards, quizzes, and AI chat

Chapters

The experiment tests how adding CPU cores (vCPUs) impacts application capacity across Python, Java, Go, and Rust.
The same application, 'Conduit' (a Medium clone), was deployed in each language with identical data and API endpoints.
Performance was measured by the maximum number of simulated users a server could handle while meeting strict response time and error rate requirements.
The test accounts for potential CPU steal time and acknowledges that vCPUs may not equate to dedicated physical cores.
RAM and database performance scale with the server size, and all components (Nginx, backend, Postgres) share available CPU.

Understanding the experimental setup is crucial for interpreting the results, as it defines the scope and limitations of the performance comparisons.

A server with 8 vCPUs doesn't guarantee 8x performance because of shared resources and how software utilizes concurrency.

Python uses multiple worker processes to leverage multiple cores, as the Global Interpreter Lock (GIL) prevents true multi-threaded execution of Python code.
Each worker process has its own interpreter and resources, allowing parallel request handling.
The optimal configuration found was one worker process per CPU core, yielding a ~7x increase in user capacity from 1 to 8 cores.
Adding more workers than cores did not improve performance and increased memory usage.
Python's application layer consumed a larger share of CPU compared to other languages, with Postgres using less.

This demonstrates how languages with limitations like Python's GIL can still scale effectively by employing process-based concurrency, but with specific trade-offs.

On an 8-core server, 8 Python worker processes were optimal, while 17 workers consumed more memory without a performance boost.

Java utilizes virtual threads (Project Loom) to manage concurrency efficiently, allowing many lightweight threads to run on fewer OS threads.
This model enables effective use of multiple cores by allowing threads to yield when waiting for I/O, such as database operations.
Java achieved a ~7.2x increase in user capacity from 1 to 8 cores, similar to Python's scaling.
Java applications consumed significantly more memory than other stacks, even without exhausting the available RAM.
The framework (Spring Boot) was not optimized or compiled to native code for this test.

Java's approach shows how modern language features can abstract away low-level threading complexities, leading to high concurrency and scalability.

On an 8-core server, Java supported 21,000 users, demonstrating strong performance despite high memory usage.

Go uses Goroutines, lightweight concurrent tasks managed by the Go scheduler, which effectively distributes work across multiple OS threads within a single process.
This allows a single Go application to natively utilize multiple CPU cores without needing separate processes.
Go achieved a ~7x increase in user capacity from 1 to 8 cores, with performance scaling well, especially from 4 to 8 cores.
The Go application itself had very low memory overhead compared to Java.
The standard library was used, minimizing framework-specific overhead.

Go's built-in concurrency model with Goroutines offers a highly efficient way to utilize multi-core processors within a single process, leading to excellent performance and low resource consumption.

An 8-core Go server handled 22,000 users, showing strong capacity scaling and efficient resource use.

Rust, using Actix and Tokio's multi-threaded runtime, employs asynchronous tasks managed by worker threads, typically one per core.
This allows tasks to yield when waiting for I/O, enabling other tasks to run on the same core and effectively utilizing multiple cores within a single process.
Rust achieved approximately a 6.5x increase in user capacity from 1 to 8 cores, performing comparably to Go and Java.
Expensive operations like password hashing were offloaded to a separate blocking pool to avoid hindering async task execution.
While performing well, Rust did not significantly outperform Go or Java in this specific test.

Rust's asynchronous programming model, combined with its efficient runtime, provides robust concurrency and performance, making it a strong contender for high-load applications.

On a 4-core server, Rust achieved its highest recorded score, demonstrating strong performance before reaching the 8-core configuration.

In most tested stacks (Java, Go, Rust), the PostgreSQL database consumed over half of the CPU resources, indicating it was a primary bottleneck.
The application code itself used around 27-30% of CPU, and Nginx about 11-12%.
Python showed a different profile, with its application layer using more CPU (55%) and Postgres less (31%).
Performance scaling wasn't perfectly linear across core counts; the CPU speed per core slightly decreased on larger machines (2, 4 cores) before recovering on the 8-core machine.
This non-linear scaling is partly due to variations in the underlying virtualized hardware performance and potential coordination overhead in multi-core systems.

Identifying bottlenecks is critical for performance optimization; in this case, the database often limited the application's ability to fully utilize additional CPU cores.

The 4-core machine's CPUs were about 20% slower per core than the 1-core machine, explaining why the 4-core results were not a perfect 4x improvement.

Adding cores significantly increased total user capacity (6.5x to 8x), but the cost per user did not necessarily decrease.
For example, Go's 8-core server offered more total capacity but was less cost-efficient per user than the 4-core server.
The experiment confirmed that CPU was the limiting factor for this specific workload.
Effective utilization of multiple cores depends heavily on the programming language's concurrency model and how well it distributes work.
While languages like Go and Rust offer efficient native concurrency, Python and Java can also scale well with appropriate strategies (worker processes, virtual threads).

Understanding the relationship between performance gains, cost, and the underlying technology is essential for making informed infrastructure and development decisions.

The 8-core Go server handled 229 users per dollar, while the $112 8-core server handled only 196 users per dollar, showing diminishing returns in cost-efficiency.

Key takeaways

1Adding more CPU cores generally increases application capacity, but the gains are not always linear or proportional to the number of cores added.
2The programming language's concurrency model (e.g., processes, threads, goroutines, async tasks) is a primary determinant of how effectively it can utilize multiple CPU cores.
3In many web application scenarios, the database can become a significant bottleneck, limiting the performance benefits gained from scaling the application server's CPU.
4While languages like Go and Rust offer efficient, built-in concurrency, Python and Java can achieve high scalability through techniques like multi-processing or virtual threads.
5The performance and cost-efficiency of scaling up servers are influenced by factors beyond just the application code, including underlying hardware variations and system overhead.
6Optimizing for concurrency and understanding potential bottlenecks are crucial for maximizing application performance and resource utilization.
7Even with identical application logic, different programming languages and their runtimes can exhibit vastly different performance characteristics under load.

Key terms

vCPU (virtual CPU)Global Interpreter Lock (GIL)Worker ProcessesVirtual ThreadsGoroutinesAsync TasksTokio RuntimeCPU Steal TimeConcurrency ModelBottleneckUser CapacityRequest Per Second

Test your understanding

1How does Python's Global Interpreter Lock (GIL) influence its strategy for utilizing multiple CPU cores, and what is the trade-off?
2Explain the role of virtual threads in Java's performance scaling and how they differ from traditional OS threads.
3What are Goroutines in Go, and how do they enable a single Go process to effectively use multiple CPU cores?
4Why might the performance gains from adding CPU cores not be perfectly linear, considering factors beyond the application code?
5How does the database's performance act as a potential bottleneck for application scalability, even when more CPU cores are available to the application server?