References

• Wikipedia NoSQL, Document-oriented Database

• What is MongoDB

• NeDB, nedb-promises We’ll use the promise version.

Why?

• Part of a servers job is to be a data repository

• Examples: web pages, templates, CSS files, images, user data

Why a Database?

• Why not just files in the file system?
• Databases bring
  • Organization, fast lookups
  • Good for lots of “little” data, rather than a file per item

Types of Databases

• Relational: Traditional high performance databases used in business and elsewhere. Requires highly structured data. Open source examples: PostgreSQL, SQLite

• Document Oriented: Less structured and more flexible than Relational. Easy to get started with. Examples: CouchDB, MongoDB, etc…

• Graph Databases Wikipedia.

• And more…

CRUD?

Basic Database Operations

• Create: create “things” and insert them into the database

• Read: lookup/search for “things” in the database and read them

• Update: update “things” already in the database

• Delete: delete “things” from the database

JavaScript Document Database

NeDB at NPM

“Embedded persistent or in memory database for Node.js, nw.js, Electron and browsers, 100% JavaScript, no binary dependency. API is a subset of MongoDB’s and it’s plenty fast.”

Installation

npm install nedb-promises --save

NEDBExamples.zip archive of examples from these slides.

MongoDB

From [MongoDB

MongoDB and Node.js

From NPM: mongodb official adapter

DataStore Concept

• In NeDB a db (datastore) is like a JavaScript array and it can hold arbitrary JavaScript objects (called documents)

• This is similar to a MongoDB collection

• NeDB (like MongoDB) provides many methods for insert, update, find, and remove objects from these stores.

• You can use multiple datastores in an application to hold different types of data.

File Based Persistence

Can set up NeDB for in-memory or file based use, we want file based. Use:

import DataStore from "nedb-promises";
const db = DataStore.create("./tempDB");

Where /tempDB is the name I gave to this particular data file.

Document _id

• Every document (object) in a datastore needs an _id property
• It must be a string
• It must be unique (NeDB can generate these for us)

Inserting Documents

Single Insert

import DataStore from "nedb-promises";
const db = DataStore.create("./blogDB");
let blog1 = {_id: "0", title: "Python Snake or language",
    content: `This will become the blog content. but for right now it is a placeholder`
};
db.insert(blog1); // Returns a promise

Inserting Documents 1

Batch Insert: simpleDBInit.mjs

import DataStore from "nedb-promises";
const db = DataStore.create("./blogDB");
// In these examples we set the _id property

let blog1 = {
  _id: "0",
  title: "Python Snake or language",
  content: `This will become the blog content. but for right now it is a placeholder`,
};
let blog2 = {
  _id: "1",
  title: "C++ Closer to the Metal",
  content: `So powerful, but so error prone. Difficult to master.`,
};
let blog3 = {
  _id: "2",
  title: "JavaScript Browsers Friend",
  content: `So powerful, no type checking. Keeps getting better.`,
};

db.insert([blog1, blog2, blog3])
  .then(function (newDocs) {
    console.log("Added " + newDocs.length + " docs");
  })
  .catch(function (err) {
    console.log("Something went wrong when writing");
    console.log(err);
  });

Inserting Documents 2

Let NeDB assign _ids (tasDBInit.mjs):

import { readFile } from "fs/promises";
import DataStore from "nedb-promises";
const db = DataStore.create("./tassieDB");

// Get sample data from JSON file
const mammals = JSON.parse(
  await readFile(new URL("./Narawntapu.json", import.meta.url))
);

async function cleanAndInsert() {
  // Clear out any existing entries if they exist
  let numRemoved = await db.remove({}, { multi: true });
  console.log("clearing database, removed " + numRemoved);

  // We let NeDB create _id property for us.
  let newDocs = db.insert(mammals);
  console.log("Added " + newDocs.length + " mammals");
}

cleanAndInsert();

Note on Example

Part of the Narawntapu.json file. Data from Tasmania Parks and Wildlife Service

[{"sciName": "Ornithorhynchus anatinus", "comName": "Platypus"},
{"sciName": "Tachyglossus aculeatus setosus" , "comName": "Echidna"},
{"sciName": "Antechinus minimus minimus", "comName": "Swamp Antechinus"},
{"sciName": "Dasyurus maculatus maculatus", "comName": "Spotted-tailed Quoll"},
{"sciName": "Dasyurus viverrinus", "comName": "Eastern Quoll"},
{"sciName": "Sarcophilus harrisii", "comName": "Tasmanian Devil"},

A Peek at the file format 1

Right after initial inserts blogDB:

{"_id":"0","title":"Python Snake or language","content":"This will become the
blog content.\n\t\t\tbut for right now it is a placeholder"}
{"_id":"1","title":"C++ Closer to the Metal","content":"So powerful, but so
error prone.\n\t\t\tDifficult to master."}
{"_id":"2","title":"JavaScript Browsers Friend","content":"So powerful, no
type checking.\n\t\t\tKeeps getting better."}

A Peek at the file format 2

NEDB assigned ids

{"sciName":"Potorous tridactylus apicalis",
"comName":"Potoroo","_id":"5D7Ow144oVfJZVS4"}
{"sciName":"Macropus rufogriseus rufogriseus",
"comName":"Bennetts Wallaby","_id":"8VFOq6jETFLLps6n"}
{"sciName":"Cercartetus nanus nanus",
"comName":"Eastern Pygmy Possum","_id":"9EdTSg2FrqWjVuU0"}
{"sciName":"Sarcophilus harrisii",
"comName":"Tasmanian Devil","_id":"9VhfxCg11W54Yx6n"}
{"sciName":"Dasyurus maculatus maculatus"
,"comName":"Spotted-tailed Quoll","_id":"CYXazYuO44GTg24M"}

Overview

From Finding Documents

• Use find to look for multiple documents matching you query, or findOne to look for one specific document.

• You can select documents based on field equality or use comparison operators ($lt, $lte, $gt, $gte, $in, $nin, $ne). You can also use logical operators $or, $and, $not and $where.

• You can use regular expressions in two ways: in basic querying in place of a string, or with the $regex operator.

Finding Documents 1

Getting all the documents in a store simpleDBfinds.js

import DataStore from "nedb-promises";
const db = DataStore.create("./blogDB");
// Get all the documents in the database
db.find({}).then(function (docs) {
  console.log("We found " + docs.length + " documents");
  console.log(docs);
});

Finding Documents 2

Lots of ways to find things in the datastore (findAnimals.mjs), example with string matching via regex:

import DataStore from "nedb-promises";
const db = DataStore.create("./tassieDB");

async function findThings() {
  let docs = await db.find({});
  console.log("We found " + docs.length + " Types of mammals");
  console.log(docs);

  // Get a list of all the types of Possums that live in the park:
  // Using a JavaScript regular expression
  docs = await db.find({ comName: /Possum/ });
  console.log("We found " + docs.length + " Types of Possums");
  console.log(docs);

  // Get a list off all types of Bandicoot
  // FYI https://en.wikipedia.org/wiki/Eastern_barred_bandicoot
  docs = await db.find({ comName: /Bandicoot/ });
  console.log("We found " + docs.length + " Types of Bandicoot");
  console.log(docs);

  // Kangaroos or Wallabys
  // Use $or operator
  docs = await db.find({
    $or: [{ comName: /Kangaroo/ }, { comName: /Wallaby/ }],
  });
  console.log("We found " + docs.length + " Kangaroo like thing");
  console.log(docs);
}

findThings();

JavaScript RegEx

• MDN JavaScript Regular Expressions Guide

• "Are used with the exec and test methods of RegExp, and with the match, replace, search, and split methods of String.

JavaScript RegEx: Just a bit

• RegEx literal var myReg = /stuff between slashes/;

• ^, $: Match begining and end of a line

• *,+: Matches preceding expression 0 (1) or more times

• .: Matches any single character

JavaScript RegEx: Hands on

mystring = "This is a message to CS351: Rule the web!"
mystring.match(/a mes/)
mystring.match(/a bmes/)
mystring.match(/^This/)
mystring.match(/^message/)

Query Operators

From findAnimals.mjs:

  // Kangaroos or Wallabys
  // Use $or operator
  docs = await db.find({
    $or: [{ comName: /Kangaroo/ }, { comName: /Wallaby/ }],
  });
  console.log("We found " + docs.length + " Kangaroo like thing");
  console.log(docs);

Sorting and Pagination

From sortPagAnimals.mjs:

import DataStore from "nedb-promises";
const db = DataStore.create("./tassieDB");

async function lookThemUp() {
  // Sort by common name, limit to the first 5
  let docs = await db.find({}).sort({ comName: 1 }).limit(5).exec();
  console.log("First 5 Sorted by Common name");
  console.log(docs);

  // The {"sciName": 1} argument to find restricts the fields that are returned
  docs = await db.find({}, { sciName: 1 }).sort({ sciName: 1 }).limit(5).exec();
  console.log("First 5 sorted by Scientific name");
  console.log(docs);
}

lookThemUp();

Projections

Restricting the document fields returned. From sortPagAnimals.mjs:

  // The {"sciName": 1} argument to find restricts the fields that are returned
  docs = await db.find({}, { sciName: 1 }).sort({ sciName: 1 }).limit(5).exec();
  console.log("First 5 sorted by Scientific name");
  console.log(docs);

Counting

Use when you don’t need details back countAnimals.mjs

import DataStore from "nedb-promises";
const db = DataStore.create("./tassieDB");

// Count all
db.count({}).then(function (count) {
  console.log(`We counted ${count} mammals`);
});

// count Devil
db.count({ comName: /Devil/ }).then(function (count) {
  console.log(`We have ${count} type(s) of Devils`);
});

Updating

Lots of support for updating documents (updateAnimals.mjs). Here we add and set, then remove a new field:

// Shows Updating
import DataStore from "nedb-promises";
const db = DataStore.create("./tassieDB");

async function updateThem() {
  // update Devil
  let doc = await db.update(
    { comName: /Devil/ },
    { $set: { status: "Endangered" } }
  );
  console.log(`Updated Tas Devil`);
  console.log(doc);

  doc = await db.find({ comName: /Devil/ });
  console.log(`Updated Tas Devil`);
  console.log(doc);

  doc = await db.update(
    { comName: /Devil/ },
    { $unset: { status: "Endangered" } }
  );
  console.log(`Updated Tas Devil`);
  console.log(doc);

  doc = await db.find({ comName: /Devil/ });
  console.log(`Updated Tas Devil`);
  console.log(doc);
}

updateThem();

Removing Documents TODO: STOPPED HERE

Clearing out everything from tasDBInit.mjs

async function cleanAndInsert() {
  // Clear out any existing entries if they exist
  let numRemoved = await db.remove({}, { multi: true });
  console.log("clearing database, removed " + numRemoved);

  // We let NeDB create _id property for us.
  let newDocs = db.insert(mammals);
  console.log("Added " + newDocs.length + " mammals");
}

cleanAndInsert();

Indexing for Speed

From Indexing:

NeDB supports indexing. It gives a very nice speed boost and can be used to enforce a unique constraint on a field. You can index any field, including fields in nested documents using the dot notation.

Animal Sighting Example

Purely fictitious but the animals are real!

• Two databases: tassieDB animals of Tasmania, usersDB a database of recent Tasmania national park visitors (fictitious).

• Park visitors can register animal sightings via a web page and we will add the sighting information to both databases.

• See file sightings.mjs in the NEDBExamples.zip archive.

Setting up Databases

Getting access to existing databases and some sighting data. From sightings.mjs

import DataStore from "nedb-promises";
const visitorsDB = DataStore.create("./usersDB");
const tassieDB = DataStore.create("./tassieDB");

let sighting1 = {email: "columbic1841@live.com", 
                sciName: "Macropus rufogriseus rufogriseus"};
let sighting2 = {email: "madonna1803@gmail.com", 
                sciName: "Macropus giganteus tasmaniensis"};

Serial Database Operations

If ordering was important we can do something like this (From sightings.mjs):

async function recordSighting(s) {
  try {
    let visitor = await visitorsDB.findOne({ email: s.email });
    let animal = await tassieDB.findOne({ sciName: s.sciName });
    console.log(
      `${visitor.firstName} ${visitor.lastName} saw a ${animal.comName}`
    );
    // Now update
    let up1 = await visitorsDB.update(
      { email: s.email },
      { $push: { sightings: s.sciName } }
    );
    let up2 = await tassieDB.update(
      { sciName: s.sciName },
      { $push: { sightings: s.email } }
    );
    console.log(`updated ${up1} visitor, and ${up2} animal(s)`);
  } catch (e) {
    console.log(`error: ${e}`);
  }
}

Result of All Serial Operations

Logging output:

Calvin Kemp saw a Bennetts Wallaby

User database update:

{"firstName":"Calvin","lastName":"Kemp","email":"columbic1841@live.com",
"role":"customer","passHash":"$2a$13$50zNNJmOwnFxfFEVn9TssuHUHlay1pu4rzJ70BmC7q9JT3WSyOkxi",
"_id":"FLSCM7vJ1g78iTKy","sightings":["Macropus rufogriseus rufogriseus"]}

Some Parallel DB Operations

Lookups and updates are done in parallel, but lookups come first then updates.

async function parallelRecordSighting(s) {
  try {
    let ps = [
      visitorsDB.findOne({ email: s.email }),
      tassieDB.findOne({ sciName: s.sciName }),
    ];
    let [visitor, animal] = await Promise.all(ps);
    console.log(
      `${visitor.firstName} ${visitor.lastName} saw a ${animal.comName}`
    );
    // Now update
    ps = [
      visitorsDB.update(
        { email: s.email },
        { $push: { sightings: s.sciName } }
      ),
      tassieDB.update(
        { sciName: s.sciName },
        { $push: { sightings: s.email } }
      ),
    ];
    let [up1, up2] = await Promise.all(ps);
    console.log(`updated ${up1} visitor, and ${up2} animal(s)`);
  } catch (e) {
    console.log(`error: ${e}`);
  }
}

Result of Some Parallel Operations

Logging output:

Scarlet Cantu saw a Forester Kangaroo

Animal database update:

{"sciName":"Macropus giganteus tasmaniensis",
"comName":"Forester Kangaroo","_id":"YYz79ksRvD0QY7oD",
"sightings":["madonna1803@gmail.com"]}