Static Site Generators

Dr. Greg Bernstein

Last update: September 22, 2019

Static Sites

Static Web Pages

Wikipedia static web page

A static web page is a web page that is delivered to the user exactly as stored, in contrast to dynamic web pages which are generated by a web application.

Why Static Websites?

Read: 6 Reasons Why You Should Go for a Static Website

  1. Security
  2. Reliability
  3. Speed
  4. Hosting and Price
  5. Scalability
  6. Technology Advancements

Managing Static Websites

  1. Creating multiple pages with consistent look and feel
  2. Linking multiple pages
  3. Updating multiple pages
  4. Checking/Testing multiple pages

Static Site Generators

  • Software to help create and maintain static sites
  • Huge range of functionality from very specific blog or documentation generators to general purpose libraries/platforms
  • Main languages used: JavaScript, Python, Ruby, and Go
  • For up to date list of popular frameworks see StaticGen

Metalsmith Site Generator

References

“An extremely simple, pluggable static site generator.”

Installation

  • Metalsmith and its plugins are NPM packages
  • Prepare a minimal package.json file
{"name": "my-first-metal-site",  "version": "0.0.1"}
  • Install with npm install --save metalsmith

Basic Concepts I

From Metalsmith:

The task of a static site generator is to produce static build files that can be deployed to a web server. These files are built from source files.

Basic Concepts II

From Metalsmith basic processing flow:

  1. from a source directory read the source files and extract their information
  2. manipulate the information
  3. write the manipulated information to files into a destination directory

Basic Concepts III: plugins

From Metalsmith architecture/plugins:

Metalsmith is built on this reasoning. It takes the information from the source files from a source directory and it writes the manipulated information to files into a destination directory. All manipulations, however, it exclusively leaves to plugins.

Basic Concepts III: plugins

From Metalsmith architecture/plugins:

Manipulations can be anything: translating templates, transpiling code, replacing variables, wrapping layouts around content, grouping files, moving files and so on. This is why we say »Everything is a Plugin«. And of course, several manipulations can be applied one after another. Obviously, in this case the sequence matters.

Basic Flow Example

Directory Structure

├── README.md
├── build.js
├── package-lock.json
├── package.json
└── src
    ├── MyTest.txt
    ├── anotherTest.txt
    ├── strange.dat1
    ├── test1.txt
    └── weird.dat1

Build.js File

var Metalsmith  = require('metalsmith');
Metalsmith(__dirname)         // __dirname defined by node.js:
  .source('./src')            // source directory
  .destination('./build')     // destination directory
  .clean(true)                // clean destination before
  .build(function(err) {      // build process
    if (err) throw err;       // error handling is required
  });

Running it: node build.js

├── README.md
├── build
│   ├── MyTest.txt
│   ├── anotherTest.txt
│   ├── strange.dat1
│   ├── test1.txt
│   └── weird.dat1
├── build.js
├── package-lock.json
├── package.json
└── src
    ├── MyTest.txt
    ├── anotherTest.txt
    ├── strange.dat1
    ├── test1.txt
    └── weird.dat1

What Happened?

  • Metalsmith copied all the files from the src directory into the build directory
  • Amazing!
  • Can you think of other ways to do this without Metalsmith?
  • Professor are you feeling okay?

Yet more functionality

We can ignore files:

var Metalsmith  = require('metalsmith');
Metalsmith(__dirname)         // __dirname defined by node.js:
  .source('./src')            // source directory
  .destination('./build')     // destination directory
  .clean(true)                // clean destination before
  .ignore("*.dat1")           // ignore this file type
  .build(function(err) {      // build process
    if (err) throw err;       // error handling is required
  });

Running it: node build.js

├── README.md
├── build
│   ├── MyTest.txt
│   ├── anotherTest.txt
│   ├── test1.txt
├── build.js
├── package-lock.json
├── package.json
└── src
    ├── MyTest.txt
    ├── anotherTest.txt
    ├── strange.dat1
    ├── test1.txt
    └── weird.dat1

Making Our Own Plugins

A Closer Look

What kind of code is this?

console.log("Starting Processing!");
Metalsmith(__dirname)         // __dirname defined by node.js:
  .source('./src')            // source directory
  .destination('./build')     // destination directory
  .clean(true)                // clean destination before
  .ignore("*.dat1")           // Use to ignore files and directories
  .build(function(err, files) { // build process
    if (err) {
        throw err;          // error handling is required
    } else {
        console.log(`Finished Processing: ${Object.keys(files)}`);
  }});

JavaScript Method Chaining

Some examples with standard JS built-in objects:

var myString = "Website Development. Code the Web!"
myString.toLowerCase().includes("web")

var myDate = new Date()
myDate.toISOString().toLowerCase().split("t")

With our Own Object

To provide for chaining return a reference to the object, below we use this to help us:

myThing = {
  count: 0,
  speak(x) {
    console.log(`Hi ${x},`);
    this.count++;
    return this;
  },
  yell(x) {
    console.log(`HELLO ${x.toUpperCase()}!`)
    this.count++;
    return this;
  }  
}

// Try chaining...
myThing.speak("web dev").yell("web dev");

Metalsmith without Chaining

const Metalsmith  = require('metalsmith');
var ms = Metalsmith(__dirname);
// Set up Options:
ms.metadata({                 // add any variable you want
    hello: "World",           // use them in layout, other plugins
    myCourse: "Website Development",
  });
ms.source('./src');          // source directory
ms.destination('./build');     // destination directory
ms.clean(true);                // clean destination before
ms.ignore("*.dat1");           // Use to ignore files and directories
// Run it!
ms.build(function(err, files) {
    if (err) {
        throw err;     
    } else {
        console.log(`Finished Processing: ${Object.keys(files)}`);
}});

Creating and Using a Plugin

myPluginHello.js: A plugin that really doesn’t do anything

function plugin() {
  return function(files, metalsmith, done){
    setImmediate(done); // For asynchronous operation, schedules done callback
    console.log("Hello class from myPluginHello.js!");
  };
}

module.exports = plugin; // How we "export" from a module in Node.js

Using the plugin

See the Metalsmith use method below

const Metalsmith  = require('metalsmith');
// Here is how we "import" from another file:
const plugHello = require('./myPluginHello.js');

Metalsmith(__dirname)
  .source('./src')            // source directory
  .destination('./build')     // destination directory
  .use(plugHello())           // Use our Hello plugin note the ()!
  .build(function(err, files) {      // build process
    if (err) {
        throw err;          // error handling is required
    } else {
        console.log(`Finished Processing: ${Object.keys(files)}`);
  }});

Processing Files

Our plugin function is given files, metalsmith, and done objects. Let’s look at these with a plugin!

// myPlugin.js
function infoPlugin() {
  return function(files, metalsmith, done){
    setImmediate(done); // For asynchronous operation, schedules done callback
    console.log(JSON.stringify(files, replacer4Buffer, 2));
    Object.keys(files).forEach(function(file){
      let data = files[file];
      // This is where you would really do your
      // file processing. The contents is a Node.js *Buffer*.
      //console.log(data.contents.toString());
    });
  };
}

// Helps JSON.stringify deal with Node.js *Buffer*
function replacer4Buffer(key, value) {
   if (key === 'contents') {
       return this[key].toString();
   } else {
       return value;
   }
}

module.exports = infoPlugin; // How we "export" from a module in Node.js

Running two plugins

The plugins are run in the order they are called from use

const Metalsmith  = require('metalsmith');
const pluginInfo = require('./myPlugin.js');
const plugHello = require('./myPluginHello.js');

console.log("Starting Processing!");
Metalsmith(__dirname)         // __dirname defined by node.js:
  .source('./src')            // source directory
  .destination('./build')     // destination directory
  .use(plugHello())           // Use our Hello plugin
  .use(pluginInfo())          // Then our Info plugin
  .build(function(err, files) {      // build process
    if (err) {
        throw err;          // error handling is required
    } else {
        console.log(`Finished Processing: ${Object.keys(files)}`);
  }});

Partial Output

All the files in src get put in the files JavaScript object indexed by their file name. This includes their content!

What about metalsmith

Of biggest interest is the metadata

function goodbyePlugin() {
  return function(files, metalsmith, done){
    setImmediate(done);
    console.log("Hello class from myPluginGoodbye.js!");
    // Uncomment below to see everything in metalsmith var
    //console.log(JSON.stringify(metalsmith, null, 2));
    console.log("Look at the global metadata object:")
    console.log(metalsmith.metadata());

  };
}

module.exports = goodbyePlugin;

Setting Metadata

Use the metadata function to set any global metadata

const Metalsmith  = require('metalsmith');
const pluginInfo = require('./myPlugin.js');
const plugHello = require('./myPluginHello.js');
const plugGoodbye = require('./myPluginGoodbye.js');

console.log("Starting Processing!");
Metalsmith(__dirname)         // __dirname defined by node.js:
  .metadata({                 // add any variable you want
    hello: "World",           // use them in layout, other plugins
    myCourse: "Website Development",
  })
  .source('./src')            // source directory
  .destination('./build')     // destination directory
  .clean(true)                // clean destination before
  .ignore("*.dat1")           // Use to ignore files and directories
  .use(plugHello())           // Use our Hello plugin
  .use(plugGoodbye())
  .use(pluginInfo())
  .build(function(err, files) {      // build process
    if (err) {
        throw err;          // error handling is required
    } else {
        console.log(`Finished Processing: ${Object.keys(files)}`);
  }});

Partial Output

Global metadata available to all plugins

Processing Markdown

Look for a Plugin

Go to NPM and search:

Plugin Options

Install metalsmith-markdown

  • npm install --save metalsmith-markdown
  • Create some Markdown files in our src directory
  • Modify our build.js file to include Markdown processing

build.js File

const Metalsmith  = require('metalsmith');
const markdown = require('metalsmith-markdown');
const infoPlugin = require('./myPlugin.js'); // Only for understanding

Metalsmith(__dirname)
  .source('./src')
  .destination('./build')
  .clean(true)
  .ignore("*.dat1")           
  .use(infoPlugin())          // Look at files before processing
  .use(markdown())
  .use(infoPlugin())         // Look at files after processing
  .build(function(err, files) {
    if (err) throw err;
    else {
        console.log("Output files:");
        console.log(Object.keys(files));
    }
  });

Running it

node build.js result:

├── README.md
├── build
│   ├── MeMe.html
│   ├── MyRestaurant.html
│   ├── MyTest.txt
│   ├── anotherTest.txt
│   └── test1.txt
├── build.js
├── package-lock.json
├── package.json
└── src
    ├── MeMe.md
    ├── MyRestaurant.md
    ├── MyTest.txt
    ├── anotherTest.txt
    ├── strange.dat1
    ├── test1.txt
    └── weird.dat1

Results

Snooping with the infoPlugin:

Issues

  • The HTML files are not complete. No <body>, <html>, or <head> elements.
  • No links/navigation between pages.
  • No styling and/or JavaScript functionality in pages.

Layouts and Templates

Using a Template Engine

Installation

We need metalsmith-layouts and jstransformer-nunjucks

  • npm install --save metalsmith-layouts
  • npm install --save jstransformer-nunjucks
  • Need a directory to hold our “layouts” and at least one template.

Directory Structure

├── README.md
├── build
│   ├── MeMe.html
│   ├── MyRestaurant.html
│   ├── MyTest.txt
│   ├── anotherTest.txt
│   └── test1.txt
├── build.js
├── layouts
│   └── base.njk
├── node_modules (not shown!!!)
├── package-lock.json
├── package.json
└── src
    ├── MeMe.md
    ├── MyRestaurant.md
    ├── MyTest.txt
    ├── anotherTest.txt
    ├── strange.dat1
    ├── test1.txt
    └── weird.dat1

Template File

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <title>All the same</title>
  </head>
<body>
  <main>
{{contents | safe}} <!-- A Nunjucks filter that stops HTML escaping -->
  </main>
</body>
</html>

Updated build.js

var Metalsmith  = require('metalsmith');
var markdown = require('metalsmith-markdown');
var layouts = require('metalsmith-layouts');

Metalsmith(__dirname)         // __dirname defined by node.js:
                              // name of current working directory
  .metadata({                 // add any variable you want
                              // use them in layout, other plugins
    author: "Dr. B",
    myClass: "Web Systems",
  })
  .source('./src')            // source directory
  .destination('./build')     // destination directory
  .clean(true)                // clean destination before
  .ignore("*.dat1")           // Use to ignore files and directories
  .use(markdown())
  .use(layouts({
    default: "base.njk",
    directory: "layouts"
}))
  .build(function(err) {      // build process
    if (err) throw err;       // error handling is required
  });

Two things to note here:

  1. The use of layouts and its configuration
  2. The metadata call with values available to all plugins

Example Rendered File

MeMe.html

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <title>All the same</title>
  </head>
<body>
  <main>
<h1 id="all-about-me">All About Me</h1>
<p>This is a test file for <em>Markdown</em> processing.</p>
<p>Bullet list:</p>
<ul>
<li>Why would anyone follow someone on twitter?</li>
<li>How can anyone say anything of importance in so little space?</li>
<li>Repeating something many times <em>doesn&#39;t</em> make it true.</li>
</ul>
<p><strong>Emphasis</strong></p>
<ol>
<li>Schools need to teach practical &quot;critical thinking&quot;</li>
<li>Otherwise students will learn about con-men, salesmen, and politicians the <strong><em>hard way</em></strong>.</li>
<li>Do you really think that most financial institutions want you to be <em>financially literate</em>?</li>
<li>Avoid <strong>most</strong> debt.</li>
</ol>
 <!-- A Nunjucks filter that stops HTML escaping -->
  </main>
</body>
</html>

Issues

  • All our HTML files have the same title
  • No description meta data for the page
  • Do we need a separate template for each page to add meta data?

Attaching Metadata to Files

YAML

YAML (YAML Ain’t Markup Language) is a human-readable data serialization language. It is commonly used for configuration files,

YAML Front Matter

  • You can define variables including arrays and objects via YAML
  • This is typically done at the beginning of Markdown files or even HTML files that will under go further processing.
---
title: The Family Restaurant
description: The family restaurant was Bernstein's Fish Grotto. I worked there and was motivated to never go into the restaurant business!

---

Updated Template File

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <title>{{title}}</title>
      <meta name="author" content="{{author}}" >
      <meta name="description" content="{{description}}" >
  </head>
<body>
  <main>
{{contents | safe}} <!-- A Nunjucks filter that stops HTML escaping -->
  </main>
</body>
</html>

Example Rendered File (metadata)

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <title>All About Me</title>
      <meta name="author" content="Dr. B" >
      <meta name="description" content="A rant to test out Markdown processing." >
  </head>
<body>
  <main>
<h1 id="all-about-me">All About Me</h1>
<p>This is a test file for <em>Markdown</em> processing.</p>
<p>Bullet list:</p>
<ul>
<li>Why would anyone follow someone on twitter?</li>
<li>How can anyone say anything of importance in so little space?</li>
<li>Repeating something many times <em>doesn&#39;t</em> make it true.</li>
</ul>
<p><strong>Emphasis</strong></p>
<ol>
<li>Schools need to teach practical &quot;critical thinking&quot;</li>
<li>Otherwise students will learn about con-men, salesmen, and politicians the <strong><em>hard way</em></strong>.</li>
<li>Do you really think that most financial institutions want you to be <em>financially literate</em>?</li>
<li>Avoid <strong>most</strong> debt.</li>
</ol>
 <!-- A Nunjucks filter that stops HTML escaping -->
  </main>
</body>
</html>

How?

Metalsmith uses the gray-matter library

More Metalsmith Processing

Add a Handcrafted HTML Page

  • To the src directory I added the HomeWork2.html and hw2.css files.
  • After running node build.js we get the following:
├── build
│   ├── HomeWork2.html
│   ├── MeMe.html
│   ├── MyRestaurant.html
│   ├── MyTest.txt
│   ├── anotherTest.txt
│   ├── hw2.css
│   └── test1.txt

Great! But…

contents of hw2.css:

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <title></title>
      <meta name="author" content="Dr. B" >
      <meta name="description" content="" >
  </head>
<body>
  <main>
body {background-color:#92d0e0;}
main {background-color: white;
  max-width: 800px;
margin-left: auto;
  margin-right: auto;
  margin-top: 2em;
  margin-bottom: 2em;

padding: 1em;
border: solid #00000080 7px;}
header {text-align: center;
font-family: sans-serif;}
ol {padding-left: 1em}
li {padding-left: 0.5em}
      code{white-space: pre-wrap;}
      span.smallcaps{font-variant: small-caps;}
      span.underline{text-decoration: underline;}
      div.column{display: inline-block; vertical-align: top; width: 50%;}
   <!-- A Nunjucks filter that stops HTML escaping -->
  </main>
</body>
</html>

Not so great…

contents of MyTest.txt

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <title></title>
      <meta name="author" content="Dr. B" >
      <meta name="description" content="" >
  </head>
<body>
  <main>
This is regular text.
I'm putting some *markdown* in here to see if it gets
**processed**. <!-- A Nunjucks filter that stops HTML escaping -->
  </main>
</body>
</html>

What Happened?

  • The markdown plugin only processed the *.md files
  • The layouts plugin processed everything!
  • Metalsmith sends all files (except those ignored) to all plugins, it is up to the plugin to decide which to process and may need help!

Controlling layouts

  • The pattern parameter takes a multimatch field to filter out files.
  • This is a popular technique. A quick look at the NPM showed around 1 million downloads a week for multimatch and over 9 million for its faster replacement micromatch.
  • Added pattern: ["*", "!*.txt", "!*.css", "!HomeWork2.html"] to stop processing of the text, css and our custom HTML file.

What Does Metalsmith Do?

  • Every file in the source directory is transformed into a JavaScript Object.
  • That Object contains a member called contents that holds the contents of the file.
  • YAML “front matter” in the files is also added to this object as members.

Example Markdown File

From Metalsmith home page my-file.md:

---
title: A Catchy Title
draft: false
---

An unfinished article...

As a JS Object

The corresponding JS Object from the Metalsmith home page:

{
  'relative_to_sourcepath/my-file.md': {
    title: 'A Catchy Title',
    draft: false,
    contents: 'An unfinished article...',
    mode: '0664',
    stats: {
      /* keys with information on the file */
    }    
  }
}

All the files as a JS Object

The JS files object for all files from the Metalsmith home page:

files = {
  "relative_to_sourcepath/file1.md": {
    title: 'A Catchy Title',
    draft: false,
    contents: 'An unfinished article...',
    mode: '0664',
    stats: {
      /* keys with information on the file */
    }    
  },
  "relative_to_sourcepath/file2.md": {
    title: 'An Even Better Title',
    draft: false,
    contents: 'One more unfinished article...',
    mode: '0664',
    stats: {
      /* keys with information on the file */
    }    
  }
}

Example Plugin

From Metalsmith home page:

/**
 * Metalsmith plugin to hide drafts from the output.
 */
function plugin() {
  return function(files, metalsmith, done){
    setImmediate(done);
    Object.keys(files).forEach(function(file){
      var data = files[file];
      if (data.draft) delete files[file];
    });
  };
}

Putting Metalsmith to Work

Some Example Plugins

I have used these

Interesting looking

Adding Navigation

  • How would you add navigation to all (or a subset of pages)?
  • Only want to define this “menu” once to keep things consistent and avoid broken links.
  • How about defining it in a JSON file?

JSON File for Navigation

[{"title": "Markdown Test", "url": "MeMe.html"},
 {"title": "My Restaurant", "url": "MyRestaurant.html"},
 {"title": "A Homework Set", "url": "HomeWork2.html"}
]

Update build.js

var Metalsmith  = require('metalsmith');
var markdown = require('metalsmith-markdown');
var layouts = require('metalsmith-layouts');
var nav = require("./nav.json");

Metalsmith(__dirname)         // __dirname defined by node.js:
                              // name of current working directory
  .metadata({                 // add any variable you want
                              // use them in layout, other plugins
    author: "Dr. B",
    links: nav,               // Add navigation information
  })
  .source('./src')            // source directory
  .destination('./build')     // destination directory
  .clean(true)                // clean destination before
  .ignore("*.dat1")           // Use to ignore files and directories
  .use(markdown())
  .use(layouts({
    default: "base.njk",
    directory: "layouts",
    pattern: ["*", "!*.txt", "!*.css", "!HomeWork2.html"]
}))
  .build(function(err) {      // build process
    if (err) throw err;       // error handling is required
  });

Update base.njk

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
    <title>{{title}}</title>
      <meta name="author" content="{{author}}" >
      <meta name="description" content="{{description}}" >
      <link href="simple.css" rel="stylesheet">
  </head>
<body>
  <main>
      {% include "nav.njk" %} <!-- This file builds the navigation-->
{{contents | safe}} <!-- A Nunjucks filter that stops HTML escaping -->
  </main>
</body>
</html>

Create the nav.njk file

<nav>
<ul>
    {% for item in links %}
    <li><a href={{item.url}}>{{item.title}}</a></li>
    {% endfor %}
</ul>
</nav>

A few touches

  • Add some CSS to style things and pick some ugly colors simple.css
  • Run node build.js
  • Take a look in the build directory

Rendered with Navigation