In Review Node.js App Structure Guide

Create a Node.js web application structure that can be used as a standard guide or starter template scalable to any size.

Category: Node.js Web Applications Last Updated: 09/26/19

Github: https://github.com/TechandStartup/node-app-structure

Context:

Goal: Provide curated standardized building blocks that software developers can trust to use out of the box, and can easily customize to meet the requirements of their projects.
Skill level: Applies to anyone who is or wants to be a web developer. Applies to all skill levels, from brand new programmers to experienced Node.js web devs.
Simple Sites: Non programmers who want to build a simple website for their small business or organization would be better served using Wordpress. It doesn't require any programming, and you can have a basic site up within hours. It is too slow at high volume however.
This is a Guide: Not meant to be a tutorial per se, but rather a guide that can be used as a starting point or template for building a web app.
Opinionated yet flexible: For each decision point a recommended approach is taken after weighing the pros and cons of the options. But because these are just guides you can modify whatever you like thus providing the best of both worlds.
Evaluation criteria: Each approach option is weighed with the following criteria in mind: performance, popularity, maturity, clarity, simplicity, consistency, intuitiveness, beginner friendliness, conventions, best practices, rolling your own favored over installing packages, favoring the core web languages (JavaScript, JSON, HTML, CSS) over DSLs (domain specific languages) or other programming languages, node package documentation.
Up-to-date: Each guide is reviewed when new major or minor versions of dependency packages are released and updates are made where appropriate.

        

Setup the app

Express Docs: Getting Started | Express Generator | Express Guide | API Package.json: Docs Prerequisite: Make sure you have a recent version of Node.js (LTS - Latest Stable Version) and npm (comes with Node.js) installed in your local environment. Generate the app using express with flags to make ejs the templating engine. The --git flag just adds a .gitignore file. npx express-generator node-app-structure --view=ejs --git cd node-app-structure Install the libraries added to your package.json file by express-generator. npm install

Create the base file structure

Create the folders and files needed for a standard app. We'll add an articles collection to the file structure later. Use the below UNIX commands to create the folders and files we need in addition to the ones generated by the express generator. mkdir creates a directory, touch creates a file, mv moves a file, and rm removes a file. mkdir controllers touch controllers/pagesController.js mkdir models mkdir views/pages touch views/pages/about.ejs mv views/index.ejs views/pages/home.ejs mv views/error.ejs views/pages/error.ejs mkdir views/layouts touch views/layouts/header.ejs touch views/layouts/footer.ejs rm routes/users.js This file structure was inspired by the structure of a Ruby on Rails app. • Express uses the MVC (Model-View-Controller) pattern. However, the express generator combines the controller actions with the routes. Instead we have a directory for controllers and a separate controller file for each collection. • Start with a controller for general pages like the home page, an about page, and the error page. • Put all the routes in one file routes/index.js. • In the views directory add a view folder that matches the controller. Rename the landing page from index.js to home.js and move it to the pages directory. Also, move the error page there. • In addition, add a folder for layouts and put header and footer files there. These are "partials" that get inserted into other ejs files. • Add a model folder for when we add collections.

Routes and Controller

• Change the app.js file to use only one router removing the usersRouter and renaming indexRouter to just router. • Reflect the change in location of the errors.ejs file. • Change the variable declarations from var to const. // app.js const createError = require('http-errors'); const express = require('express'); const path = require('path'); const cookieParser = require('cookie-parser'); const logger = require('morgan'); const router = require('./routes/index'); const app = express(); // view engine setup app.set('views', path.join(__dirname, 'views')); app.set('view engine', 'ejs'); app.use(logger('dev')); app.use(express.json()); app.use(express.urlencoded({ extended: false })); app.use(cookieParser()); app.use(express.static(path.join(__dirname, 'public'))); app.use('/', router); // catch 404 and forward to error handler app.use(function(req, res, next) { next(createError(404)); }); // error handler app.use(function(err, req, res, next) { // set locals, only providing error in development res.locals.message = err.message; res.locals.error = req.app.get('env') === 'development' ? err : {}; // render the error page res.status(err.status || 500); res.render('pages/error'); }); module.exports = app; // routes/index.js const express = require('express'); const router = express.Router(); const pagesController = require('../controllers/pagesController'); // Pages routes router.get('/', pagesController.home); router.get('/about', pagesController.about); module.exports = router;

Add controller

// controllers/pagesController.js // GET / exports.home = (req, res) => { res.render('pages/home', { title: 'Node App Structure' }); }; // GET /about exports.about = (req, res) => { res.render('pages/about', { title: 'About' }); };

Add a View Layout

// views/layouts/header.ejs <!DOCTYPE html> <html> <head> <title><%= typeof title === 'undefined' ? 'Node-App-Structure' : title %></title> <link rel='stylesheet' href='/stylesheets/style.css' /> </head> <body> <nav> <a href="/">Home</a> <a href="/about">About</a> </nav> // views/layouts/footer.ejs </body> </html> // views/pages/home.ejs <% include ../layouts/header %> <h1><%= title %></h1> <hr> <p>Welcome to the <%= title %></p> <% include ../layouts/footer %> // views/pages/about.ejs <% include ../layouts/header %> <h1><%= title %></h1> <hr> <p>About Info goes here.</p> <% include ../layouts/footer %> // views/pages/error.ejs <% include ../layouts/header %> <h1><%= message %></h1> <h2><%= error.status %></h2> <!-- Error stack is for debugging. Don't show in production --> <pre><%= error.stack %></pre> <% include ../layouts/footer %>

Run the app

Run the start script from the package.json file. npm start View the app in the browser on the default port 3000: http://localhost:3000 Use hot-reloading with nodemon The Nodemon package gives you hot reloading. It reloads your app to the server whenever you save a file. Install the nodemon package globally so you can use it with all your Node.js apps. npm install -g nodemon Run the app with: nodemon

Setup Rationale:

Why Express? • It is by far the most popular MVC framework for Node.js. • It is light weight. That adds two benefits: Easy to modify to your needs, and does not do a lot of opaque "magic" behind the scenes so you can see mostly what's going on in your code. The downside of this is it takes noteably longer to build an app prototype than it does with a more full service framework like Ruby on Rails. It also means Node/Express developers use many different solutions to solve the same problems. Why use Express-generator? • It is made by the Express team so is most likely to be kept up-to-date. Why use EJS templating as opposed to the Jade/Pug default, Moustache, Handlebars, or others? • EJS has the least abstraction from HTML and JavaScript. It simply combines HTML syntax and JavaScript syntax on the same page. Similar to PHP and Ruby on Rails' ERB (which it was modeled after). Why this file structure? • The express generator's structure that combines controllers with routes makes viewing the route structure messy. The larger the app the messier it gets. Much clearer to have all the routes on one line each and all in one file. And explicitly create controller files for each collection. • We use the Ruby on Rails file structure as inspiration (i.e., as a precedent) because it is mature, a lot of developers are familiar with it, and it is layed out logically. Why use const instead of var? • Const and Let are new ways to declare variables since ECMAScript 2015 (aka ES6) was released. • Var creates a global variable, which pollutes the global namespace if it doesn't need to be global. Const or let are block scoped and therefore preferrable. • The Google JavaScript style guide recommends always declaring variables with const if you don't intend to change their value. This is open to debate but it sounds reasonable to us and seems to be a popular practice. Why not add Babel? • Babel transpiles all ES6 and later JavaScript to ES5. • Recent versions of Node.js recognize nearly all ES6 syntax. The only ES6 syntax that we would be using here that is not yet available are Import and Export. Instead we use the older require and module.exports syntax. We view this as such a minor difference that we don't feel the need to add a transpiling layer to our code. Node.js has already added experimental usage of Import/Export so we can expect those to be incorporated for use in the Node.js LTS version at some point in the future. • Reference: Node.js ES6 docs | List of supported syntax Why rename index.ejs to home.ejs • This is a minor point. Using index as the name of the default page is a historical precedent but calling it home is much more intuitive.

Connect a Database

Mongoose.js Docs: Getting Started | Guides | API Dotenv Docs: Readme We are using a MongoDB database. You can either use a cloud version like Atlas, or install it locally on your machine. We recommend installing and using a local version when working on practice apps. For production apps if you are using a cloud version then you should be testing the cloud version in development as well. Install the mongoose and dotenv packages locally. npm install mongoose dotenv • Mongoose.js is the most popular ODM (Object-Document-Mapper) for MongoDB. • Dotenv is used for managing Environmental Variables in a .env file. This meshes well with hosting platforms like Heroku. Create a database called my_local_db (or whatever name you prefer). If using a local version open a new terminal window and run the mongo database (from any directory): mongod Then in a separate window (any directory) run the Mongo DB Shell. mongo And create the database. use my_local_db Create the .env file. touch .env Assign the database url to a constant. // .env MONGODB_URI=mongodb://localhost:27017/my_local_db Connect to the database on application start. // app.js ... const mongoose = require('mongoose'); require('dotenv').config(); ... // Connect to the MongoDB database mongoose.connect( process.env.MONGODB_URI, { useNewUrlParser: true, useFindAndModify: false, useCreateIndex: true, useUnifiedTopology: true } ); const db = mongoose.connection; db.on('error', console.error.bind(console, 'connection error:')); db.once('open', () => { console.log('Connected to the Database.') }); ... • The various options added to the mongoose.connect method relate to changes in mongoose. You'll get warnings or errors without them. • We're logging messages to the terminal console on database connection and connection error.

Database Setup Rationale:

Why use MongoDB? • You can either use a SQL database like Postgres or MySQL, or a NoSQL database like MongoDB. Both are popular and reasonable options. We are using MongoDB just because it is easier for someone not familiar with databases to get started with since it uses JSON syntax (which is essentially JavaScript syntax) instead of the SQL query language. • Mongoose.js, the most popular MongoDB ORM package, has notably more downloads than sequelize, the most popular SQL ORM package: npmtrends.com. Why not use the GraphQL query language? • GraphQL is a great relatively new database query language. We are using REST for our base case application though, because it is very mature. For someone new to web development, they should have a solid understanding of the REST architecture. • For a small app, adding a GraphQL layer can be seen as overkill. The performance benefits are realized at scale.

Add a Collection

Blog apps are common and many non-blog apps include blogs. So we'll use that for our collection, calling it articles. Use the below Unix commands to add the controller, model, view directory and view files. touch controllers/articlesController.js touch models/article.js mkdir views/articles touch views/articles/list.ejs touch views/articles/details.ejs touch views/articles/create.ejs touch views/articles/update.ejs touch views/articles/delete.ejs And two files that apply to all collections: touch views/layouts/form-errors.ejs touch controllers/controllerMethods.js

Add the model

Mongoose Docs: Validation // models/article.js const mongoose = require('mongoose'); const Schema = mongoose.Schema; const articleSchema = new Schema({ title: { type: String, validate: { validator: function(value) { return /^[\w'",.!? ]+$/.test(value); }, message: props => `${props.value} should only contain letters, numbers, spaces, and '",!?. characters.` }, required: [true, "Title is required"], minlength: [2, "Title must be at least 2 characters."], maxlength: [100, "Title must be no more than 100 characters"] }, content: { type: String, required: [true, "Content is required"], }, published: { type: Boolean, default: false, } }, {timestamps: true}); module.exports = mongoose.model('Article', articleSchema); • Articles have three fields: title, content, and published. • The timestamps option will automatically add createdAt and updatedAt fields and insert a timestamp value when a document is created or updated. • The only required property for each field is the type. • Mongoose provides built-in validators that it automatically runs before saving to a document to the database. If a validation fails the document is not created and an error object is returned. • You can add custom error messages to the validator property. • Use the "required" validator for title and content. • Set minimum and maximum character length validators. • Add a custom validator to title to limit what characters can be used. • Add a default value of false to the published boolean field.

Add the RESTful routes

// routes/index.js const express = require('express'); const router = express.Router(); const pagesController = require('../controllers/pagesController'); const articlesController = require('../controllers/articlesController'); // Pages routes router.get('/', pagesController.home); router.get('/about', pagesController.about); // Articles routes router.get('/articles', articlesController.list); router.get('/articles/create', articlesController.createView); router.post('/articles/create', articlesController.create); router.get('/articles/:id', articlesController.details); router.get('/articles/:id/update', articlesController.updateView); router.post('/articles/:id/update', articlesController.update); router.get('/articles/:id/delete', articlesController.deleteView); router.post('/articles/:id/delete', articlesController.delete); module.exports = router;

Controller

Add a shared method for handling form errors // controllers/controllerMethods.js exports.formErrors = (err) => { const errors = []; for (var property in err.errors) { errors.push({msg: err.errors[property].message}); } return errors; } • This creates an errors array and pushes each error message into the array.
Add the Controller actions
// controllers/articlesController.js const Article = require('../models/article'); const createError = require('http-errors'); const { formErrors } = require('./controllerMethods'); // GET /articles exports.list = (req, res, next) => { Article.find() .sort({'title': 'asc'}) .limit(50) .exec((err, articles) => { if (err) { next(err); } else { res.render('articles/list', { title: 'Articles', articles: articles }); } }); }; // GET /articles/:id exports.details = (req, res, next) => { Article.findById(req.params.id, (err, article) => { // if id not found mongoose throws CastError. if (err || !article) { next(createError(404)); } else { res.render('articles/details', { title: 'Article', article: article }); } }); }; // GET /article/create exports.createView = (req, res, next) => { res.render('articles/create', { title: 'Create Article' }); }; // POST /article/create exports.create = async (req, res, next) => { try { const newArticle = await Article.create(req.body); res.redirect(`/articles/${newArticle._id}`); } catch (err) { if (err.name == 'ValidationError') { const errors = formErrors(err); res.render('articles/create', { article: req.body, errors: errors }); } else { console.error(err); res.status(500).send(err); } } }; // GET /articles/:id/update exports.updateView = (req, res, next) => { Article.findById(req.params.id, (err, article) => { // if id not found throws CastError. if (err || !article) { next(createError(404)); } else { res.render('articles/update', { title: 'Update Article', article: article }); } }); }; // POST /articles/:id/update exports.update = async (req, res, next) => { const article = { title: req.body.title, content: req.body.content, published: req.body.published, _id: req.params.id }; try { const updatedArticle = await Article.findByIdAndUpdate( // get article id from the URL parameters. req.params.id, article, // Options: new: true to return the updated article. RunValidators default is false for updates. {new: true, runValidators: true} ); res.redirect(`/articles/${article._id}`); } catch (err) { if (err.name == 'ValidationError') { const errors = formErrors(err); res.render('articles/update', { article: article, errors: errors }); } else { res.status(500).send(err); } } }; // GET /articles/:id/delete exports.deleteView = (req, res, next) => { Article.findById(req.params.id, (err, article) => { // if id not found throws CastError. if (err || !article) { next(createError(404)); } else { res.render('articles/delete', { title: 'Delete Account', article: article }); } }); }; // POST articles/:id/delete exports.delete = (req, res, next) => { Article.findByIdAndRemove(req.body.id, (err) => { if (err) { next(err); } else { res.redirect('/'); } }) };

Views

Add a link in the navbar to articles. // views/layouts/headers.ejs <a href="/articles">Articles</a> Add the form-errors partial template. // views/layouts/form-errors.ejs <% if(typeof errors !== 'undefined') { %> <ul> <li><strong>Correct Any Errors Below And Resubmit:</strong></li> <% for (var error of errors) { %> <li><%= error.msg %></li> <% } %> </ul> <% } %> Add the list view. // views/articles/list.ejs <% include ../layouts/header %> <h1>Articles</h1> <ul> <% articles.forEach(function(article) { %> <li> <%= !article.published ? '(Draft) ' : '' %> <a href="/articles/<%= article.id %>"><%= article.title %></a> </li> <% }); %> </ul> <a href='/articles/create'>Create new article</a> <% include ../layouts/footer %> Add the details view. // views/articles/details.ejs <% include ../layouts/header %> <h1> <%= !article.published ? '(Draft) ' : '' %> <%= article.title %> </h1> <small><%= article.updatedAt.toLocaleString("en-US", {month: 'short', day: 'numeric', year: 'numeric'}); %></small> <hr> <div> <%= article.content %> </div> <hr> <a href="/articles/<%= article.id %>/update" class='btn btn-info float-right'>Update Article</a> <% include ../layouts/footer %> Add the create form. // views/articles/create.ejs <% include ../layouts/header %> <h1>Create Article</h1> <% include ../layouts/form-errors %> <form method="POST" action="/articles/create"> <div> <label for="title">Title</label><br> <input type="text" name="title" value="<%= typeof article === 'undefined' ? '' : article.title %>" maxlength="200" autofocus> </div> <div> <label for="content">Content</label><br> <textarea name="content"> <%= typeof article === 'undefined' ? '' : article.content %> </textarea> </div> <div> <input type="checkbox" name="published" id="published" value=true <%= typeof article !== 'undefined' && article.published === 'true' ? 'checked' : '' %>> <label for="published">Publish</label> </div> <div> <button type="submit">Submit</button> <a href="/articles">Cancel</a> </div> </form> <% include ../layouts/footer %> Add the update form view. // views/articles/update.ejs <% include ../layouts/header %> <h1>Update Article</h1> <% include ../layouts/form-errors %> <form method="POST" action="/articles/<%= article._id %>/update"> <div> <label for="title">Title</label><br> <input type="text" name="title" value="<%= typeof article === 'undefined' ? '' : article.title %>" maxlength="200" required> </div> <div> <label for="content">Content</label><br> <textarea name="content" rows="10" required><%= typeof article === 'undefined' ? '' : article.content %></textarea> </div> <div> <input type="checkbox" name="published" id="published" value=true <%= typeof article !== 'undefined' && article.published ? 'checked' : '' %> > <label for="published">Publish</label> </div> <div> <button type="submit">Submit</button> <a href="/articles/<%= article._id %>">Cancel</a> </div> </form> <hr> <h3>Delete Article</h3> <a href="/articles/<%= article._id %>/delete">Delete</a> <% include ../layouts/footer %> Add the delete form view. // views/articles/delete.ejs <% include ../layouts/header %> <h1>Delete Article: <%= article.title %></h1> <hr> <p> Are you sure you want to delete this article? <form method='POST' action='/articles/<%= article._id %>/delete'> <input type="hidden" name="id" value="<%= article._id %>"> <button type='submit'>Yes - Delete Article</button> <a href="/articles/<%= article.id %>/update">No - Cancel</a> </form> </p> <% include ../layouts/footer %>

Collection structure rationale

Why these route, view, and controller action names? • Since we are performing CRUD (Create-Read-Update-Delete) operations we are mirroring the CRUD names. For Read we are reading both as a list (an array of objects from the database to be precise) and a single document (an object from the database). List and Details seem to be the most intuitive names for these routes/actions/views. • For the routes that write to the database, use the same URL as the URL that gets the corresponding form. Just use a different HTTP method (GET or POST). • We are using POST for all write requests because HTML forms only recognize POST and GET methods. If you use the PUT, PATCH, or DELETE methods in an HTML form they will be treated as GET requests.

Collection structure Open Items

Validation: Use Mongoose validator, validator or express-validator, other validation libraries? Currently we are using Mongoose's built in validators because: • Mongoose comes with a validator that has built-in checks and allows you to build custom validators. • If we can do our validation with a library that is already installed then no need to add the overhead of another library. However: • A seemingly simple validation for uniqueness turns out to be very complicated because Mongoose treats new and update validations differently. So uniqueness validation must either be done in the controller separate from the other validations, or by using a plugin called mongoose-unique-validator. • Validator and Express-validator also offer sanitization, converting <, >, &, ', ", and / into HTML entities. If that is important then use one of those libraries. Express-validator is built on top of validator. For apps using an SQL database, the sequelize ORM package uses validator as a dependency.