Client-side Data Persistence with IndexedDB

by
Tags: , ,
Category:

A Deep Dive into IndexedDB

In a previous article, I compared client-side storage solutions: localStorage, sessionStorage, cookies, and touched briefly on IndexedDB. In the vast ecosystem of web storage solutions, IndexedDB stands out as a powerful, low-level API for client-side storage of significant amounts of structured data. While cookies, localStorage, and sessionStorage are suited for storing smaller data sets, IndexedDB is designed to manage larger volumes of data, including complex data types that shouldn’t be stored as strings like files and blobs. Let’s dive deeper into the intricacies of IndexedDB and understand why it’s a compelling choice for robust web applications.

What is IndexedDB?

IndexedDB operates as a transactional database system similar to SQL-based RDBMSs. But instead of traditional tables, IndexedDB utilizes “object stores” to house data. Essentially, it acts like a large local repository filled with JavaScript objects, neatly indexed for easy access. When you query this system, it uses indexes to generate a cursor. This cursor allows you to traverse the resulting dataset. With this in mind, IndexedDB shares many characteristics of a NoSQL database, more specifically the category of document-based NoSQL databases like MongoDB.
Key Features of IndexedDB

Object-Oriented: IndexedDB stores data in the form of objects, not rows of tables. This aligns well with the object-centric paradigm of JavaScript.

Key-Value Store: Every object in the database has a key that’s used for retrieval. This key can be auto generated or set during the storage process.

Indices: For efficient querying, IndexedDB allows you to create indices on object properties. For instance, if you store a user object that may include a unique field such as an email, you could create an index on the email property for faster lookups similar to SQL-like databases.

Transactions: Actions in IndexedDB are wrapped in transactions. This ensures database integrity, even if a particular operation fails.

Versioning: Whenever you update the database structure, such as adding a new object store, you need to upgrade the database version. This feature ensures forward compatibility.

Why Use IndexedDB?

Large Storage Capacity: While localStorage and sessionStorage might offer around 5-10MB, IndexedDB can store much more. Some browsers allow it to use up to 60% of the available free disk space.

Complex Data Types: Beyond simple key-value pairs, IndexedDB can store arrays, objects, and even binary data types like Blobs and ArrayBuffers. A full list of all the types that IndexedDB supports are types that are supported by the structured clone algorithm.

Performance: For large-scale applications, IndexedDB, with its indexing capabilities, provides a performant way to retrieve data without scanning the entire database.

Offline Access: In Progressive Web Applications (PWAs) where offline functionality is crucial, IndexedDB can store data locally, enabling applications to function without an active internet connection.

Shared Storage: Since all tabs in a browser share the same IndexedDB instance for a specific domain, any data written by one tab is immediately available to all other tabs.

Working with IndexedDB

Interacting with IndexedDB involves a series of asynchronous operations. Here’s a basic workflow:

Open a Database: Before any operations, you must open a database. If the database doesn’t exist, it’s created automatically.

  let openRequest = indexedDB.open('myDatabase', 1);

Define Object Stores: Once a database is opened or created, you can define object stores.

  openRequest.onupgradeneeded = function(e) {
      let db = e.target.result;
      let store = db.createObjectStore('users', { keyPath: 'id' });
  };

CRUD Operations: With the database set, you can perform CRUD (Create, Read, Update, Delete) operations.

Create: Add data to an object store.

    let transaction = db.transaction('users', 'readwrite');
    let users = transaction.objectStore('users');
    users.add({ id: 1, name: 'Lazlo' });

Read: Retrieve data using a key or an index.

    let request = users.get(1);
    request.onsuccess = function() {
        console.log(request.result); // Logs { id: 1, name: 'Lazlo' }
    };

Update: Modify existing data in an object store. For updating, you can use the put method which will update if the record exists or add if it doesn’t.

let transaction = db.transaction('users', 'readwrite');
let users = transaction.objectStore('users');
let updatedUser = { id: 1, name: 'Nandor' }; // We've changed the name from 'Lazlo' to 'Nandor'
users.put(updatedUser);

Delete: Remove data from an object store using a key.

let transaction = db.transaction('users', 'readwrite');
let users = transaction.objectStore('users');
users.delete(1); // This will delete the record with id: 1

These basic CRUD operations form the foundation of interacting with IndexedDB. The API also offers more advanced features like cursors for iterating over multiple records, creating indexes for efficient searches, and handling transactions to ensure data integrity.

Advanced Features of IndexedDB

Cursors: Cursors allow you to iterate over multiple records, offering a way to process or fetch ranges of data.

let transaction = db.transaction('users', 'readonly');
let users = transaction.objectStore('users');

let cursorRequest = users.openCursor();

cursorRequest.onsuccess = function(e) {
    let cursor = e.target.result;
    if (cursor) {
        console.log(cursor.key, cursor.value); // Logs each key and value
        cursor.continue(); // Move to the next record
    }
};

Indexes: Indexes provide efficient data lookups without scanning the entire object store. First, you need to create an index during the database setup phase.

// Create an index
let objectStore = db.createObjectStore('users', { keyPath: 'id' });
objectStore.createIndex('name', 'name', { unique: false });

// Query an index
let transaction = db.transaction('users', 'readonly');
let users = transaction.objectStore('users');
let nameIndex = users.index('name');

// Get all records where name is 'Lazlo'
let request = nameIndex.getAll('Lazlo');
request.onsuccess = function() {
    let matchingUsers = request.result;
    console.log(matchingUsers); // Array of users with the name 'Lazlo'
};

Transactions: Transactions ensure that a series of database operations are completed successfully and fully. If something goes wrong in one of the operations, the entire transaction can be aborted, and none of the operations take effect.

let transaction = db.transaction('users', 'readwrite');

transaction.oncomplete = function() {
    console.log('Transaction completed.');
};

transaction.onerror = function() {
    console.error('Transaction error:', transaction.error);
};

let users = transaction.objectStore('users');
users.add({ id: 2, name: 'Nandor' });
users.add({ id: 3, name: 'Nadja' });
users.add({ id: 4, name: 'Colin' });

// Assume we have a mistake here; id 2 already exists
users.add({ id: 2, name: 'Guillermo' });

// Due to the error in the above line (adding a user with an existing id), 
// none of the adds in this transaction will take effect.

By utilizing these advanced features, developers can craft intricate and efficient operations in IndexedDB, thereby ensuring a more fluid and responsive user experience in web applications.

Challenges and Considerations

While IndexedDB offers a robust solution for client-side storage, it’s essential to be aware of its nuances and potential challenges.

Complexity in Simplicity: IndexedDB is a low-level API, which inherently makes it more detailed and complex than simpler options like localStorage.

Consistent Browser Support: Most modern browsers support IndexedDB. However, slight variations in their implementations can occasionally pop up. It’s a good practice to test your application across different browsers to ensure consistent performance. Can I use is a great resource for researching browser support for various web APIs.

Variable Storage Limits: While IndexedDB provides generous storage, there’s no fixed cap. However, be cautious about relying entirely on it, especially since browsers might remove data under certain storage constraints.
The Importance of Backups: Never rely solely on IndexedDB. It’s crucial to have backup mechanisms, whether server-side or on the cloud, to safeguard your data.

Simplifying IndexedDB with Helpful Packages

IndexedDB is often considered a low-level API, making it somewhat challenging to integrate directly in today’s web development. Modern web apps often leverage ES6 features, especially promises, for a smoother development experience. Fortunately, there are packages that transform the basic IndexedDB API into a more developer-friendly interface.

While there are numerous options available, in this article, we’ll focus on the idb package, which stands out due to its popularity, boasting around 6 million downloads weekly.

I’ve set up two demonstrations to highlight the difference between using the raw IndexedDB API and the enhanced idb package. Both demos showcase CRUD operations on a user database:

Upon inspection, you’ll notice the idb version is much more in line with contemporary API practices. It capitalizes on the async/await pattern for handling promises, in contrast to the raw API where you’re tasked with managing success and failure cases individually for each operation.

IndexedDB in the wild

I examined three of my frequently used applications to see if they use IndexedDB. Here’s how you can check too (using Google Chrome as an example).

  1. Open the website in Chrome.
  2. Go to Developer Tools: Click on the Overflow Menu, select More Tools, and then Developer Tools. Alternatively, use Ctrl+Shift+I or Cmd+Option+I shortcuts.
  3. In Developer Tools, click the “Application” tab which will show a left sidebar. There will be a section for IndexedDB under the Storage heading. If IndexedDB is being used then there will be databases that can be viewed.

Examples of Apps Using IndexedDB:

  • Gmail appears to be storing emails for offline reading and searching.
  • Slack looks to be storing recent messages with a timestamp to possibly reduce constant server checks or reduce the amount of redundant data sent to a client if they already have the latest messages.
  • Notion seems to save user actions in a “transactions” store. Notion pages are typically made up of movable and editable content blocks so they probably save these actions and then send a bulk update to sync up with the main database.

Conclusion: IndexedDB’s Place in the Modern Web Landscape

IndexedDB has carved out its niche as a formidable solution for client-side storage, especially when it comes to handling larger, structured data sets. It’s a powerful tool for developers looking to craft web applications that are both performance-oriented and resilient in the face of connectivity issues. However, being a low-level API, it presents a steeper learning curve and there are nuances with various browser implementations and storage limitations.