NeDB for Node

Dr. Greg Bernstein

Updated October 27th, 2019

Data Persistence

References

Why?

  • Part of a servers job is to be a data repository

  • Examples: web pages, templates, CSS files, images, user data

Why a Database?

  • Why not just files in the file system?
  • Databases bring
    • Organization, fast lookups
    • Good for lots of “little” data, rather than a file per item

Types of Databases

  • Relational: Traditional high performance databases used in business and elsewhere. Requires highly structured data. Open source examples: PostgreSQL, SQLite
  • Document Oriented: Less structured and more flexible than Relational. Easy to get started with. Examples: CouchDB, MongoDB, etc…
  • Graph Databases Wikipedia.

  • And more…

CRUD?

Basic Database Operations

  • Create: create “things” and insert them into the database

  • Read: lookup/search for “things” in the database and read them

  • Update: update “things” already in the database

  • Delete: delete “things” from the database

Example Document Database

JavaScript Document Database

NeDB at NPM

“Embedded persistent or in memory database for Node.js, nw.js, Electron and browsers, 100% JavaScript, no binary dependency. API is a subset of MongoDB’s and it’s plenty fast.”

Installation

npm install nedb --save

NEDBExamples.zip archive of examples from these slides.

Database Creation and Storage

DataStore Concept

  • In NeDB a db (datastore) is like a JavaScript array and it can hold arbitrary JavaScript objects (called documents)
  • This is similar to a MongoDB collection
  • NeDB (like MongoDB) provides many methods for insert, update, find, and remove objects from these stores.

  • You can use multiple datastores in an application to hold different types of data.

File Based Persistence

Can set up NeDB for in-memory or file based use, we want file based. Use:

const DataStore = require('nedb');
db = new DataStore({filename: __dirname + '/tempDB', autoload: true});

Where /tempDB is the name I gave to this particular data file.

Document Creation

Document _id

  • Every document (object) in a datastore needs an _id property
  • It must be a string
  • It must be unique (NeDB can generate these for us)

Inserting Documents

Single Insert

const DataStore = require('nedb');
db = new DataStore({filename: __dirname + '/tempDB', autoload: true});
let blog1 = {_id: "0", title: "Python Snake or language",
            content: `This will become the blog content.
            but for right now it is a placeholder`
};
db.insert(blog1);

Inserting Documents

Batch Insert: simpleDBInit.js

let blog1 = {_id: "0", title: "Python Snake or language",
            content: `This will become the blog content.
            but for right now it is a placeholder`
};
let blog2 = {_id: "1", title: "C++ Closer to the Metal",
            content: `So powerful, but so error prone.
            Difficult to master.`
};
let blog3 = {_id: "2", title: "JavaScript Browsers Friend",
            content: `So powerful, no type checking.
            Keeps getting better.`
};
db.insert([blog1, blog2, blog3], function(err, newDocs) {
    if(err) {
        console.log("Something went wrong when writing");
        console.log(err);
    } else {
        console.log("Added " + newDocs.length + " docs");
    }
});

Inserting Documents

Let NeDB assign _ids (tasDBInit.js):

const DataStore = require('nedb');
const db = new DataStore({filename: __dirname + '/tassieDB', autoload: true});

const mammals = require('./Narawntapu.json');
// We let NeDB create _id property for us.

db.insert(mammals, function(err, newDocs) {
    if(err) {
        console.log("Something went wrong when writing");
        console.log(err);
    } else {
        console.log("Added " + newDocs.length + " mammals");
    }
});

Note on Example

Node makes it easy to import a JSON file into a variable. Part of the Narawntapu.json file. Data from Tasmania Parks and Wildlife Service

[{"sciName": "Ornithorhynchus anatinus", "comName": "Platypus"},
{"sciName": "Tachyglossus aculeatus setosus" , "comName": "Echidna"},
{"sciName": "Antechinus minimus minimus", "comName": "Swamp Antechinus"},
{"sciName": "Dasyurus maculatus maculatus", "comName": "Spotted-tailed Quoll"},
{"sciName": "Dasyurus viverrinus", "comName": "Eastern Quoll"},
{"sciName": "Sarcophilus harrisii", "comName": "Tasmanian Devil"},

A Peek at the file format 1

Right after initial inserts blogDB:

{"_id":"0","title":"Python Snake or language","content":"This will become the
blog content.\n\t\t\tbut for right now it is a placeholder"}
{"_id":"1","title":"C++ Closer to the Metal","content":"So powerful, but so
error prone.\n\t\t\tDifficult to master."}
{"_id":"2","title":"JavaScript Browsers Friend","content":"So powerful, no
type checking.\n\t\t\tKeeps getting better."}

A Peek at the file format 2

NEDB assigned ids

{"sciName":"Potorous tridactylus apicalis",
"comName":"Potoroo","_id":"5D7Ow144oVfJZVS4"}
{"sciName":"Macropus rufogriseus rufogriseus",
"comName":"Bennetts Wallaby","_id":"8VFOq6jETFLLps6n"}
{"sciName":"Cercartetus nanus nanus",
"comName":"Eastern Pygmy Possum","_id":"9EdTSg2FrqWjVuU0"}
{"sciName":"Sarcophilus harrisii",
"comName":"Tasmanian Devil","_id":"9VhfxCg11W54Yx6n"}
{"sciName":"Dasyurus maculatus maculatus"
,"comName":"Spotted-tailed Quoll","_id":"CYXazYuO44GTg24M"}

Searching the Database

Overview

From Finding Documents

  • Use find to look for multiple documents matching you query, or findOne to look for one specific document.

  • You can select documents based on field equality or use comparison operators ($lt, $lte, $gt, $gte, $in, $nin, $ne). You can also use logical operators $or, $and, $not and $where.

  • You can use regular expressions in two ways: in basic querying in place of a string, or with the $regex operator.

Finding Documents 1

Getting all the documents in a store simpleDBfinds.js

db = new DataStore({filename: __dirname + '/blogDB', autoload: true});
// Get all the documents in the database, the {} does this...
db.find({}, function(err, docs) {
    if (err) {
        console.log("something is wrong");
    } else {
        console.log("We found " + docs.length + " documents");
        console.log(docs);
    }
});

Finding Documents 2

Lots of ways to find things in the datastore (findAnimals.js), example with string matching via regex:

const DataStore = require('nedb');
const db = new DataStore({filename: __dirname + '/tassieDB', autoload: true});
// Get a list of all the types of Possum that live in the park:
// Using a JavaScript regular expression
db.find({"comName": /Possum/}, function(err, docs) {
    if (err) {
        console.log("something is wrong");
    } else {
        console.log("We found " + docs.length + " Types of Possums");
        console.log(docs);
    }
});

JavaScript RegEx

JavaScript RegEx: Just a bit

  • RegEx literal var myReg = /stuff between slashes/;

  • ^, $: Match begining and end of a line

  • *,+: Matches preceding expression 0 (1) or more times

  • .: Matches any single character

JavaScript RegEx: Hands on

mystring = "This is a message to CS351: Rule the web!"
mystring.match(/a mes/)
mystring.match(/a bmes/)
mystring.match(/^This/)
mystring.match(/^message/)

Query Operators

From findAnimals.js:

// Kangaroos or Wallabys
// Use $or operator
db.find({$or: [{"comName": /Kangaroo/}, {"comName": /Wallaby/}]}, function(err, docs) {
    if (err) {
        console.log("something is wrong");
    } else {
        console.log("We found " + docs.length + " Kangaroo like thing");
        console.log(docs);
    }
});

Sorting and Pagination

From sortPagAnimals.js:

const DataStore = require('nedb');
const db = new DataStore({filename: __dirname + '/tassieDB', autoload: true});
// Sort by common name, limit to the first 5
db.find({}).sort({"comName": 1}).limit(5).exec(function(err, docs) {
    if (err) {
        console.log("something is wrong");
    } else {
        console.log("First 5 Sorted by Common name");
        console.log(docs);
    }
});

Projections

Restricting the document fields returned. From sortPagAnimals.js:

// The {"sciName": 1} argument to find restricts the fields
// that are returned
db.find({}, {"sciName": 1}).sort({"sciName": 1}).limit(5).exec(function(err, docs) {
    if (err) {
        console.log("something is wrong");
    } else {
        console.log("First 5 sorted by Scientific name");
        console.log(docs);
    }
});

Counting

Use when you don’t need details back countAnimals.js

// Count all
db.count({}, function(err, count) {
    if (err) {
        console.log("something is wrong");
    } else {
        console.log(`counted ${count} mammals`)
    }
});
// count Devils
db.count({"comName": /Devil/}, function(err, count) {
    if (err) {
        console.log("something is wrong");
    } else {
        console.log(`We have ${count} type(s) of Devils`);
    }
});

Changing and Deletion

Updating

Lots of support for updating documents (updateAnimals.js). Here we add and set a new field:

// update Devil
db.update({"comName": /Devil/}, {$set: {"status": "Endangered"}}, function(err, doc) {
    if (err) {
        console.log("something is wrong");
    } else {
        console.log(`Updated Tas Devil`);
        console.log(doc);
    }
});
db.find({"comName": /Devil/}, function(err, doc){
    if (err) {
        console.log("something is wrong");
    } else {
        console.log(`Updated Tas Devil`);
        console.log(doc);
    }
})

Updating

Lots of support for updating documents (updateAnimals.js). Here we remove a field:

db.update({"comName": /Devil/}, {$unset: {"status": "Endangered"}}, function(err, doc) {
    if (err) {
        console.log("something is wrong");
    } else {
        console.log(`Updated Tas Devil`);
        console.log(doc);
    }
});
db.find({"comName": /Devil/}, function(err, doc){
    if (err) {
        console.log("something is wrong");
    } else {
        console.log(`Updated Tas Devil`);
        console.log(doc);
    }
})

Removing Documents

Clearing out everything

db = new DataStore({filename: __dirname + '/blogDB', autoload: true});
// Clear the database
db.remove({}, { multi: true },
    function (err, numRemoved) {
        console.log("removed " + numRemoved);
});

Indexing

Indexing

From Indexing:

NeDB supports indexing. It gives a very nice speed boost and can be used to enforce a unique constraint on a field. You can index any field, including fields in nested documents using the dot notation.

NEDB and Promises

NEDB-Promises

  • Like many Node.js libraries NEDB provides for asynchronous operation via callbacks. However we have learned that JavaScript Promises can help keep our code more readable and straight forward.

  • nedb-promises provides a nice Promise “wrapper” for NEDB.

nedb-promises Usage 1

  • npm install --save nedb-promises
  • Initialization:
const Datastore = require('nedb-promises')
let datastore = Datastore.create('/path/to/db.db')

nedb-promises Usage 2

From nedb-promises

datastore.find({ field: true })
  .then(...)
  .catch(...)

datastore.findOne({ field: true })
  .then(...)
  .catch(...)

datastore.insert({ doc: 'yourdoc' })
  .then(...)
  .catch(...)

Example of nedb-promises

Purely fictitious but the animals are real!

  • Two databases: tassieDB animals of Tasmania, usersDB a database of recent Tasmania national park visitors (fictitious).

  • Park visitors can register animal sightings via a web page and we will add the sighting information to both databases.

  • See file sightings1.js in the NEDBExamples.zip archive.

Setting up Databases

Getting access to existing databases and some sighting data.

const Datastore = require('nedb-promises')
let tassieDB = Datastore.create(__dirname + '/tassieDB');
let visitorsDB = Datastore.create(__dirname + '/usersDB');
// Who is this and what did they see?
let sighting1 = {email: "columbic1841@live.com",
                sciName: "Macropus rufogriseus rufogriseus"};
// Who is this and what did they see?
let sighting2 = {email: "madonna1803@gmail.com",
                sciName: "Macropus giganteus tasmaniensis"};

Serial Database Operations

If ordering was important we can do something like this:

async function recordSighting(s) {
    try {
        let visitor = await visitorsDB.findOne({email: s.email});
        let animal = await tassieDB.findOne({sciName: s.sciName});
        console.log(`${visitor.firstName} ${visitor.lastName} saw a ${animal.comName}`);
        // Now update
        let up1 = await visitorsDB.update({email: s.email},
                                          {$push: {sightings: s.sciName}});
        let up2 = await tassieDB.update({sciName: s.sciName},
                                        {$push: {sightings: s.email}});
        console.log(`updated ${up1} visitor, and ${up2} animal(s)`);
    } catch (e) {
        console.log(`error: ${e}`);
    }
}

Result of All Serial Operations

Logging output:

Calvin Kemp saw a Bennetts Wallaby

User database update:

{"firstName":"Calvin","lastName":"Kemp","email":"columbic1841@live.com",
"role":"customer","passHash":"$2a$13$50zNNJmOwnFxfFEVn9TssuHUHlay1pu4rzJ70BmC7q9JT3WSyOkxi",
"_id":"FLSCM7vJ1g78iTKy","sightings":["Macropus rufogriseus rufogriseus"]}

Some Parallel DB Operations

Lookups and updates are done in parallel, but lookups come first then updates.

async function parallelRecordSighting(s) {
    try {
        let ps = [visitorsDB.findOne({email: s.email}),
                  tassieDB.findOne({sciName: s.sciName})];
        let [visitor, animal] = await Promise.all(ps);
        console.log(`${visitor.firstName} ${visitor.lastName} saw a ${animal.comName}`);
        // Now update
        ps = [visitorsDB.update({email: s.email}, {$push: {sightings: s.sciName}}),
              tassieDB.update({sciName: s.sciName}, {$push: {sightings: s.email}})];
        let [up1, up2] = await Promise.all(ps);
        console.log(`updated ${up1} visitor, and ${up2} animal(s)`);
    } catch (e) {
        console.log(`error: ${e}`);
    }
}

Result of Some Parallel Operations

Logging output:

Scarlet Cantu saw a Forester Kangaroo

Animal database update:

{"sciName":"Macropus giganteus tasmaniensis",
"comName":"Forester Kangaroo","_id":"YYz79ksRvD0QY7oD",
"sightings":["madonna1803@gmail.com"]}