Course Outline (Part 17)

A poorly optimized database might feel fast when you only have 100 users, but once your application grows to 100,000 users, those tiny inefficiencies multiply until the entire server crashes under the load.

Performance optimization in MongoDB generally boils down to three things:

  1. Writing smart, efficient queries.
  2. Using Indexes correctly.
  3. Ensuring the server has enough RAM to keep the “working set” in memory.

In this module, we will learn how to identify and fix bottlenecks.

1. Query Optimization

Before you blame the server hardware, you must ensure your application is asking the database for data in the most efficient way possible.

Rule 1: Use Projection to Reduce Network Traffic

If you have a document with 50 fields, and you only need to display the user’s name and avatar on the screen, do not fetch the entire document.

// BAD: Sends massive amounts of unnecessary data over the network
db.users.find({ status: "active" })

// GOOD: Only sends exactly what the application needs
db.users.find({ status: "active" }, { name: 1, avatar: 1, _id: 0 })

Sending less data means less CPU work for the database to serialize the BSON, and less network bandwidth consumed between the database and your web server.

Rule 2: Put $match First in Aggregations

As discussed in Module 9, if you are using an Aggregation Pipeline, your $match stage should almost always be the very first stage. If you wait until stage 3 to filter out 90% of your documents, you have forced stages 1 and 2 to do massive amounts of useless computation.

Rule 3: Avoid Negative Operators ($ne, $nin)

Indexes are designed to find things that are there. They are terrible at finding things that aren’t there. Queries using $ne (Not Equal) or $nin (Not In) will often force the database to scan large portions of the index or the collection. Try to rethink your logic to use positive operators ($in, $eq) whenever possible.

2. Index Optimization (The ESR Rule)

We learned how to create indexes in Module 8. But how do you know which fields to put in a Compound Index, and in what order?

MongoDB engineers created a golden rule for Compound Indexes called the ESR Rule (Equality, Sort, Range).

If you have a query that filters by category, sorts by price, and looks for items within a certain date range, your compound index must follow this exact order:

  1. (E)quality: Fields that are filtered by exact matches (e.g., { category: "Electronics" }) must come first in the index.
  2. (S)ort: Fields that determine the order of the results (e.g., .sort({ price: 1 })) must come second.
  3. (R)ange: Fields that use range operators like $gt, $lt, or $regex (e.g., { createdAt: { $gt: ISODate("2026-01-01") } }) must come last.

The Golden Index:

db.products.createIndex({ category: 1, price: 1, createdAt: 1 })

3. The explain() Command (Deep Dive)

The only way to prove that your queries are optimized is to ask MongoDB exactly how it plans to execute them. You do this by appending .explain("executionStats") to your query.

db.products.find({ category: "Electronics" }).explain("executionStats")

When you run this, MongoDB returns a large JSON document detailing the performance. Here are the three most critical metrics you must look at:

  1. winningPlan.stage:
    • You want this to say IXSCAN (Index Scan).
    • If it says COLLSCAN (Collection Scan), your query is looking at every document in the collection, and you urgently need an index!
  2. totalKeysExamined vs totalDocsExamined:
    • This ratio should ideally be 1:1. If you examined 50 keys in the index, you should only examine 50 documents.
    • If totalDocsExamined is 100,000, but the query only returned 5 results, your index is highly inefficient.
  3. executionTimeMillis:
    • How long the query took to run. You want this as close to 0 as possible.

4. Memory Management (The Working Set)

MongoDB relies heavily on your computer’s RAM.

When you query data, MongoDB reads it from the slow hard drive and puts it into fast RAM (specifically, the WiredTiger storage engine cache).

The portion of your data that is accessed most frequently by your users is called the Working Set. This usually includes all of your Indexes, plus the most recently accessed documents.

The Golden Hardware Rule: Your server’s RAM must be larger than your Working Set.

If your Working Set exceeds your available RAM, MongoDB has to constantly swap data back and forth from the slow hard drive (a process called “Page Faulting”). When this happens, your database performance will instantly fall off a cliff, and queries that took 2 milliseconds will suddenly take 5 seconds.

If you are experiencing severe performance degradation and high disk I/O, the solution is almost always to upgrade your server to have more RAM.

5. Performance Monitoring

You shouldn’t wait for users to complain about slow loading times before you optimize. You should actively monitor your database.

  • MongoDB Atlas Metrics: If you use Atlas, the main dashboard provides a “Slow Queries” log and real-time CPU/RAM charts.
  • The Database Profiler: If you manage your own servers, you can enable the internal database profiler. It automatically records any query that takes longer than a specified threshold (e.g., 100ms) into a special system.profile collection.

To turn on the profiler to log all queries slower than 100ms:

db.setProfilingLevel(1, { slowms: 100 })

You can then query db.system.profile.find() just like a normal collection to find your worst-performing queries and fix them!

Summary

Performance optimization is an ongoing process.

  • Optimize Queries: Use Projections, avoid negative operators, and filter early in aggregations.
  • Optimize Indexes: Follow the ESR (Equality, Sort, Range) rule for compound indexes.
  • Verify: Always use .explain("executionStats") to prove your index works.
  • Monitor RAM: Ensure your Working Set fits entirely in memory to avoid catastrophic disk swapping.

In Module 17, we will dive into a feature that guarantees data integrity across multiple operations: Transactions!

Discussion

Loading comments...