In Review Node.js API Structure Guide

Create a Node.js API structure that can be used as a standard guide or starter template scalable to any size.

Category: Node.js Web Applications Last Updated: 09/26/19

Github: https://github.com/TechandStartup/node-api-structure

Context:

Goal: Provide curated standardized building blocks that software developers can trust to use out of the box, and can easily customize to meet the requirements of their projects.
Skill level: Applies to anyone who is or wants to be a web developer. Applies to all skill levels, from brand new programmers to experienced Node.js web devs.
Simple Sites: Non programmers who want to build a simple website for their small business or organization would be better served using Wordpress. It doesn't require any programming, and you can have a basic site up within hours. It is too slow at high volume however.
This is a Guide: Not meant to be a tutorial per se, but rather a guide that can be used as a starting point or template for building a web app.
Opinionated yet flexible: For each decision point a recommended approach is taken after weighing the pros and cons of the options. But because these are just guides you can modify whatever you like thus providing the best of both worlds.
Evaluation criteria: Each approach option is weighed with the following criteria in mind: performance, popularity, maturity, clarity, simplicity, consistency, intuitiveness, beginner friendliness, conventions, best practices, rolling your own favored over installing packages, favoring the core web languages (JavaScript, JSON, HTML, CSS) over DSLs (domain specific languages) or other programming languages, node package documentation.
Up-to-date: Each guide is reviewed when new major or minor versions of dependency packages are released and updates are made where appropriate.

        

Setup the app

Express Docs: Getting Started | Express Guide | API Package.json: Docs Prerequisite: Make sure you have a recent version of Node.js (LTS - Latest Stable Version) and npm (comes with Node.js) installed in your local environment. From the command line, go to the directory where you want to place the app. Then make a directory for the project and cd into it: mkdir node-api-structure cd node-api-structure Instantiate a Node.js app. npm init or to skip the setup questions and use defaults: npm init --yes Install the express web framework. npm install express Change the package.json file to use "server.js" as it's main file instead of index.js. And add a script to run the server in production. Your file should look something like: // package.json { "name": "node-api-structure", "version": "1.0.0", "description": "Web API server built with Node.js, Express, and MongoDB", "main": "server.js", "scripts": { "start": "node server.js", "test": "echo \"Error: no test specified\" && exit 1" }, "keywords": [], "author": "", "license": "ISC" }

Create the base file structure

Create a skeleton app folder and file structure using the below UNIX commands. This is a classic MVC (Model-View-Controller) pattern. Mkdir creates a directory, touch creates a file. touch server.js mkdir routes touch routes/index.js mkdir controllers mkdir models touch .gitignore > "node_modules/" Add the node_modules folder to the .gitignore file. No need to put that code on github. Also add the .env file which will hold our environmental variables which may be secret and not for sharing on github. // .gitignore node_modules/ .env

Server

• Create the server with express. • Return the traditional Hello World courtesy of Express' getting started guide. // server.js const express = require('express'); const app = express(); const port = 3000; app.get('/', (req, res) => res.send('Hello World!')); app.listen(port, () => console.log(`Example app listening on port ${port}!`));

Run the app

Run the start script from the package.json file. npm start View the app in the browser on the default port 3000: http://localhost:3000 View your server side console logs in your terminal. Use hot-reloading with nodemon The Nodemon package gives you hot reloading. It reloads your app to the server whenever you save a file. Install the nodemon package globally so you can use it with all your Node.js apps. npm install -g nodemon Run the app with: nodemon

Setup Rationale:

Why Express? • It is by far the most popular MVC framework for Node.js. • It is light weight. That adds two benefits: Easy to modify to your needs, and does not do a lot of opaque "magic" behind the scenes so you can see mostly what's going on in your code. The downside of this is it takes noteably longer to build an app prototype than it does with a more full service framework like Ruby on Rails. It also means Node/Express developers use many different solutions to solve the same problems. Why not use Express-generator? • Express-generator is meant for a full application including the views, not an API. Why use the MVC pattern? • MVC is the de facto standard for making web applications. Express was designed with MVC in mind but doesn't require it. We'll go with convention and use MVC. Why name the main file server instead of index or app? The default main file created with npm init is index. And the default main file when generating a full app with express-generator is app. App makes sense for a full application including views. For only the back-end server portion server is more of a convention and seems more intuitive. In fact, the nodemon command looks for a file named server by default. Why not add Babel? • Babel transpiles all ES6 and later JavaScript to ES5. • Recent versions of Node.js recognize nearly all ES6 syntax. The only ES6 syntax that we would be using here that is not yet available are Import and Export. Instead we use the older require and module.exports syntax. We view this as such a minor difference that we don't feel the need to add a transpiling layer to our code. Node.js has already added experimental usage of Import/Export so we can expect those to be incorporated for use in the Node.js LTS version at some point in the future. • Reference: Node.js ES6 docs | List of supported syntax

Connect a Database

Mongoose.js Docs: Getting Started | Guides | API Dotenv Docs: Readme Cors Docs: Readme Cross-Origin Resource Sharing: MDN Docs We are using a MongoDB database. You can either use a cloud version like Atlas, or install it locally on your machine. We recommend installing and using a local version when working on practice apps. For production apps if you are using a cloud version then you should be testing the cloud version in development as well.

Install the mongoose, dotenv, and maybe cors packages locally. • Mongoose.js is the most popular ORM (Object-Relational-Mapper) for MongoDB. • Dotenv is used for managing Environmental Variables in a .env file. This meshes well with hosting platforms like Heroku. • Cors will allow your API to accept requests from origins other than itself. By default Express will block cross-origin HTTP requests for security reasons. For a single-page-app with a React, Vue or Angular client you shouldn't need to allow cross-origin HTTP requests so you won't need this package. If your API is meant to be accessed by mobile apps or other websites then you do need this package. The Postman API testing tool works without cors. npm install mongoose dotenv or npm install mongoose dotenv cors Create a database called my_local_db (or whatever name you prefer). If using a local version open a new terminal window and run the mongo database (from any directory): mongod Then in a separate window (any directory) run the Mongo DB Shell. mongo And create the database. use my_local_db Create the .env file. touch .env Assign the database url to a constant. // .env MONGODB_URI=mongodb://localhost:27017/my_local_db Connect to the database on application start.

Server file

// server.js const express = require('express'); const mongoose = require('mongoose'); // const cors = require('cors'); require('dotenv').config(); const router = require('./routes/index'); const app = express(); const port = 3000; // Connect to the MongoDB database mongoose.connect( process.env.MONGODB_URI, { useNewUrlParser: true, useFindAndModify: false, useCreateIndex: true, useUnifiedTopology: true } ); const db = mongoose.connection; db.on('error', console.error.bind(console, 'connection error:')); db.once('open', () => { console.log('Connected to the Database.') }); // app.use(cors()) app.use(express.urlencoded({ extended: true })); app.use(express.json()); app.use('/', router); app.listen(port, () => console.log(`Server listening on port ${port}!`)); • The various options added to the mongoose.connect method relate to changes in mongoose. You'll get warnings or errors without them. • We're logging messages to the terminal console on database connection and connection error. • Uncomment the cors require and use statements if you need to allow CORS. Add a temporary route for testing. Use res.sent to return a message or res.json return a JSON object. // routes/index.js const express = require ('express'); const router = express.Router(); router.get('/', function(req, res) { // res.send('Hello World2!'); res.json({"message": "Hello World3!"}); }); module.exports = router;

Database Setup Rationale:

Why use MongoDB? • You can either use a SQL database like Postgres or MySQL, or a NoSQL database like MongoDB. Both are popular and reasonable options. We are using MongoDB just because it is easier for someone not familiar with databases to get started with since it uses JSON syntax (which is essentially JavaScript syntax) instead of the SQL query language. • Mongoose.js, the most popular MongoDB ORM package, has notably more downloads than sequelize, the most popular SQL ORM package: npmtrends.com. Why not use the GraphQL query language? • GraphQL is a great relatively new database query language. We are using REST for our base case application though, because it is very mature. For someone new to web development, they should have a solid understanding of the REST architecture. • For a small app, adding a GraphQL layer can be seen as overkill. The performance benefits are realized at scale.

Add a Collection

Blog apps are common and many non-blog apps include blogs. So we'll use that for our collection, calling it articles. Use the below Unix commands to add the controller and model files, plus a file for methods applying to all controllers. touch controllers/articlesController.js touch models/article.js touch controllers/controllerMethods.js

Add the model

Mongoose Docs: Validation // models/article.js const mongoose = require('mongoose'); const Schema = mongoose.Schema; const articleSchema = new Schema({ title: { type: String, validate: { validator: function(value) { return /^[\w'",.!? ]+$/.test(value); }, message: props => `${props.value} should only contain letters, numbers, spaces, and '",!?. characters.` }, required: [true, "Title is required"], minlength: [2, "Title must be at least 2 characters."], maxlength: [100, "Title must be no more than 100 characters"] }, content: { type: String, required: [true, "Content is required"], }, published: { type: Boolean, default: false, } }, {timestamps: true}); module.exports = mongoose.model('Article', articleSchema); • Articles have three fields: title, content, and published. • The timestamps option will automatically add createdAt and updatedAt fields and insert a timestamp value when a document is created or updated. • The only required property for each field is the type. • Mongoose provides built-in validators that it automatically runs before saving to a document to the database. If a validation fails the document is not created and an error object is returned. • You can add custom error messages to the validator property. • Use the "required" validator for title and content. • Set minimum and maximum character length validators. • Add a custom validator to title to limit what characters can be used. • Add a default value of false to the published boolean field.

Add the RESTful routes

// routes/index.js const express = require('express'); const router = express.Router(); const articlesController = require('../controllers/articlesController'); // Articles routes router.get('/articles', articlesController.list); router.post('/articles/create', articlesController.create); router.get('/articles/:id', articlesController.details); router.post('/articles/:id/update', articlesController.update); router.post('/articles/:id/delete', articlesController.delete); module.exports = router;

Controller

Add a shared method for handling form errors // controllers/controllerMethods.js exports.formErrors = (err) => { const errors = []; for (var property in err.errors) { errors.push({msg: err.errors[property].message}); } return errors; } • This creates an errors array and pushes each error message into the array.
Add the Controller actions
// controllers/articlesController.js const Article = require('../models/article'); const { formErrors } = require('./controllerMethods'); // GET /api/articles exports.list = (req, res, next) => { Article.find() .sort({'title': 'asc'}) .limit(50) .exec((err, articles) => { if (err) { res.send(err); } else { res.json(articles); } }); }; // GET /articles/:id exports.details = async (req, res, next) => { try { const article = await Article.findById(req.params.id); res.json(article); } catch (err) { if (err.name === 'CastError' && err.kind === 'ObjectId') { return res.status(404).send('Not Found') } res.send(err); } }; // POST /articles/create exports.create = async (req, res, next) => { try { const article = await Article.create(req.body); console.log('New Article Created!', article); res.json({message: "Article successfully created", article }); } catch (err) { if (err.name == 'ValidationError') { const errors = formErrors(err); console.error(errors); res.status(422).json({ article: req.body, errors: errors }); } else { console.error(err); res.status(500).send(err); } } }; // PATCH /articles/:id/update exports.update = async (req, res, next) => { try { const article = await Article.findByIdAndUpdate( // get article id from the URL parameters. req.params.id, req.body, // Options: new: true to return the updated article. RunValidators default is false for updates. {new: true, runValidators: true} ); res.json({ message: 'Article updated', article }); } catch (err) { if (err.name == 'ValidationError') { const errors = formErrors(err); res.status(422).json({ article: req.body, errors: errors }); } else { res.status(500).send(err); } } }; // DELETE /api/articles/:id/delete exports.delete = (req, res, next) => { Article.findByIdAndRemove(req.params.id, (err, article) => { if (err) { res.status(500).send(err); } else if (!article) { res.status(422).send("Article not found."); } else { res.json({ message: "Article deleted", article }); } }) };

Collection structure rationale

Why these route, view, and controller action names? • Since we are performing CRUD (Create-Read-Update-Delete) operations we are mirroring the CRUD names. For Read we are reading both as a list (an array of objects from the database to be precise) and a single document (an object from the database). List and Details seem to be the most intuitive names for these routes/actions. Why use the PATCH and DELETE HTTP methods instead of POST? • HTML forms only recognize the HTTP POST and GET methods. But mobile apps and single-page-apps with front-end libraries like React, Vue, or Angular can make requests using any valid HTTP method. So for clarity and intuitiveness we use PATCH and DELETE for update and delete requests. • Update requests can be made with either PUT or PATCH requests. PUT is meant to replace an existing document and PATCH is meant to modify only specific fields in a document. So we believe PATCH more accurately represents the update operation. We are not replacing the createdAt and _id fields. Why use Mongoose validator and not express-validator or other validation libraries? • Mongoose comes with a validator that has built-in checks and allows you to build custom validators. • If we can do our validation with a library that is already installed then no need to add the overhead of another library. Why not add an "api" namespace? If your API is meant to be accessed by third-party websites then adding an "api" namespace to the URL is a good idea. If this is the back end for a mobile app or for an integrated single-page-app then an api namespace is not necessary. Why not use separate router files based on route namespaces for each collection instead of putting them all in one routes/index.js file? • For clarity. Having all your routes in one file makes it easier to see your overall route structure. • There may be times when you don't want the namespace in your route. For example you may want to use /signup instead of /user/signup for your user create url.