A sensible architecture for complex Node.js projects
NB: you need a browser with
JavaScript enabled
to watch this presentation!
tripu
name: cover # <br /> A sensible architecture <br /> <br /> for complex .logo[] projects .footnote[[Commit conference](https://2018.commit-conf.com/) <br /> Copyright © 2018 [tripu](https://tripu.info/) <br /> <br /> .nb[(Press `→` or swipe left)]] --- name: about class: middle .footer[*A sensible architecture for complex Node.js projects*] ## Expected audience <br /> <br /> and caveats --- name: agenda .footer[*A sensible architecture for complex Node.js projects*] ## Agenda .twoCols[ .column[ * [`$ whoami`](#me) * **[The challenges](#challenges)** * **[Organisation of code](#organisation)** * **[Automatic builds](#builds)** * [Updates](#updates) * [Vulnerabilities](#vulns) * [Linting & hinting](#lint) ] .column[ * [Code coverage](#coverage) * [Continuous integration](#ci) * [Validate markup & CSS](#check) * [Documentation](#doc) * [Other tasks](#other) * [More resources](#resources) * **[Questions, comments, debate](#qa)** ] ] --- name: me .footer[*A sensible architecture for complex Node.js projects*] ## `$ whoami` .twoCols[ .column[ * Software engineer, Granada University * 14 years of work experience * Web developer at the [W3C](https://www.w3.org/) ] .column[ * Many years with JS; ~4 with Node.js * “tripu” everywhere on the Interwebs * `[https://tripu.info](https://tripu.info/)` ] ] .center[.portrait[]] --- name: challenges .footer[*A sensible architecture for complex Node.js projects*] ## The challenges ◼◻ .twoCols[ .column[ * A large project (large codebase) * Several “pieces”, varied in nature .nb[(background processing, scheduled tasks, API, mail handling, web UI…)] * Those “pieces” should be ready to be: * published (to npm) independently * deployed alone (*microservices*) * managed as separate processes * With shared libraries .nb[(persistence layer, utilities…)] * With “aspects” .nb[(authorisation, logging…)] * With (some) shared configuration ] ] --- .footer[*A sensible architecture for complex Node.js projects*] ## The challenges ◼◼ .twoCols[ .column[ * A large project (large codebase) * Several “pieces”, varied in nature .nb[(background processing, scheduled tasks, API, mail handling, web UI…)] * Those “pieces” should be ready to be: * published (to npm) independently * deployed alone (*microservices*) * managed as separate processes * With shared libraries .nb[(persistence layer, utilities…)] * With “aspects” .nb[(authorisation, logging…)] * With (some) shared configuration ] .column[ * Following good engineering practices: * One-step builds * Coherent releases of all “pieces” * Generated documentation * Static analysis of code * Consistent coding style * Automated tests * Valid markup & CSS * Always latest npm packages * No unused npm packages * Handling of vulnerabilities * Parseable logs * Static front-end asset generation * Continuous integration/deployment ⋮ ] ] --- name: organisation .footer[*A sensible architecture for complex Node.js projects*] ## Organisation of code ◼◻◻◻◻ The temptations: * Monolithic project (single repo, single `package.json`) * Independent projects (several repos, several `package.json`) --- .footer[*A sensible architecture for complex Node.js projects*] ## Organisation of code ◼◼◻◻◻ * [*Manyrepos* vs. *monorepos*](https://medium.com/@maoberlehner/monorepos-in-the-wild-33c6eb246cb9) * *Monorepo*: *“single repository holding the code of multiple projects which may or may not be related in some way”* * Examples: * Google (~86 TB, *Piper*) * Facebook (*Mercurial*) * Twitter (several GB, *Git*) * Handy tools: * npm: [Lerna](https://github.com/lerna/lerna) * Yarn: [Workspaces](https://yarnpkg.com/blog/2017/08/02/introducing-workspaces/) --- .footer[*A sensible architecture for complex Node.js projects*] ## Organisation of code ◼◼◼◻◻ Contents of the single Git repo for *Project Foo*: .twoCols[ .column[ ```yml .gitignore .eslintrc.json .jsdoc.json .jshintrc .mocha.opts config.json build.sh doc/ index.html ⋮ logs/ ⋮ 2018-11-21.log 2018-11-22.log 2018-11-23.log ``` ] .column[ ```yml foo-api/ package.json package-lock.json known-vulns.txt server.js lib/ doc/ index.html ⋮ test/ test.js node_modules/ foo-auth/ foo-db/ foo-logger/ foo-ui/ foo-utils/ ⋮ ``` ] ] --- .footer[*A sensible architecture for complex Node.js projects*] ## Organisation of code ◼◼◼◼◻ `foo-ui/package.json`: ```json "name": "foo-ui", ⋮ "dependencies": { "foo-auth": "file:../foo-auth/", "foo-db": "file:../foo-db/", "foo-logger": "file:../foo-logger/", "foo-utils": "file:../foo-utils/", ⋮ } ``` `foo-streaming/package.json`: ```json "name": "foo-streaming", ⋮ "dependencies": { "foo-db": "file:../foo-db/", ⋮ } ``` --- .footer[*A sensible architecture for complex Node.js projects*] ## Organisation of code ◼◼◼◼◼ `foo-ui/server.js`: ```js const logger = require('foo-logger'); // Use it ``` `foo-streaming/server.js`: ```js const db = require('foo-db'), auth = require('foo-auth'); // Use them ``` --- name: builds .footer[*A sensible architecture for complex Node.js projects*] ## Automatic builds ◼◻◻◻ ```bash #!/bin/bash SELF=$(basename $0) if [[ $# -gt 1 ]]; then echo "$SELF: wrong syntax; try “--help”" exit 1 elif [[ $# -eq 1 ]]; then if [[ $1 == "-h" || $1 == "--help" ]]; then echo 'Build all Foo modules at once' echo "Syntax: “$SELF [options]”" echo ' “--scratch”, “-s”' echo ' “--clean”, “-c”' echo ' “--help”, “-h” ' exit 0 elif [[ $1 == "-c" || $1 == "--clean" ]]; then ONLYCLEAN=true elif [[ $1 == "-s" || $1 == "--scratch" ]]; then SCRATCH=true else echo "$SELF: wrong syntax; try “--help”" exit 1 fi fi ``` --- .footer[*A sensible architecture for complex Node.js projects*] ## Automatic builds ◼◼◻◻ ```bash $ ./build.sh run build.sh: wrong syntax; try “--help” $ ./build.sh -h Build all Foo modules at once Syntax: “build.sh [options]” “--scratch”, “-s” “--clean”, “-c” “--help”, “-h” ``` --- .footer[*A sensible architecture for complex Node.js projects*] ## Automatic builds ◼◼◼◻ ```bash for i in `ls -d foo-*/`; do echo "--------------------------------------- Checking “$i”" cd $i rm package-lock.json if [[ $SCRATCH == true || $ONLYCLEAN == true ]]; then rm -rf node_modules/ doc/ fi if [[ $ONLYCLEAN != true ]]; then npm install npm run build || break fi cd ../ done ``` --- .footer[*A sensible architecture for complex Node.js projects*] ## Automatic builds ◼◼◼◼ ```bash if [[ $SCRATCH == true ]]; then rm -rf doc/ fi if [[ $ONLYCLEAN != true ]]; then echo "--------------------------------------- Generating aggregated documentation" npx jsdoc --configure .jsdoc.json \ foo-api/server.js foo-api/test/ \ foo-auth/server.js foo-auth/test/ \ foo-db/server.js foo-db/test/ \ foo-logger/server.js foo-logger/test/ \ foo-ui/server.js foo-ui/test/ foo-ui/lib/ foo-ui/static/js/ \ foo-utils/server.js foo-utils/test/ ⋮ fi ``` --- name: updates .footer[*A sensible architecture for complex Node.js projects*] ## Updates * Automatic on GH: [Greenkeeper](https://greenkeeper.io/) * Semi-automatic using tools: * [`npm-check`](https://www.npmjs.com/package/npm-check) ```yml $ npx npm-check -i minami -i eslint-plugin-node ❤️ Your modules look amazing. Keep up the great work. ❤️ ``` * [`updtr`](https://www.npmjs.com/package/updtr) ```yml $ npx updtr -s exact -t 'npm run build' Running updtr with custom configuration: ● test command: npm run build ● save: exact Found 2 updates. ● popper.js 1.14.5 success ● eslint-plugin-node 8.0.0 success 2 successful updates. Finished after 99.0s. ``` * Manual: `$ sed -i 's/"mocha":\s\+"5\.0\.9"/"mocha":\ "5.2.0"/g' */package.json` --- name: vulns .footer[*A sensible architecture for complex Node.js projects*] ## Vulnerabilities: ◼◻ Before: *Node Security Platform* — now deprecated ✗ With `nsp`, there was a file for exceptions `.nsprc`: ```json { "exceptions": [ "https://nodesecurity.io/advisories/577", "https://nodesecurity.io/advisories/581" ] } ``` Problems: * No fine-grained control of exceptions * This masks vulnerabilities already solved --- .footer[*A sensible architecture for complex Node.js projects*] ## Vulnerabilities: ◼◼ Today: `npm audit` ✓ `npm audit` doesn't have a mechanism for exceptions — but here's a solution `known-vulnerabilities.txt`: ```yml https://nodesecurity.io/advisories/577 https://nodesecurity.io/advisories/577 https://nodesecurity.io/advisories/581 ``` `package.json`: ```json "scripts": { "audit": "npm audit | grep -oP https://nodesecurity.io/advisories/\\d+ | diff known-vulns.txt -", ⋮ "build": "npm run audit && npm run lint && ⋯ && npm run test && npm run jsdoc" } ``` --- name: lint .footer[*A sensible architecture for complex Node.js projects*] ## Linting & hinting ◼◻ `package.json`: ```json "devDependencies": { "eslint": "5.9.0", "eslint-plugin-node": "8.0.0", "jshint": "2.9.6" }, "scripts": { "lint": "eslint server.js utils.js tests.js", "hint": "jshint server.js utils.js tests.js", "build": "npm run audit && npm run lint && npm run hint && ⋯ ", } ``` --- .footer[*A sensible architecture for complex Node.js projects*] ## Linting & hinting ◼◼ `.eslintrc.json`: ```json { "env": { "node": true, "browser": true, "mocha": true }, "plugins": ["node"], "extends": ["eslint:recommended", "plugin:node/recommended"], "rules": { "node/no-unpublished-require": "off" }, "overrides": [ ⋯ ] } ``` `.jshintrc`: ```json { "esversion": 6, "strict": "global", "undef": true, "unused": true } ``` --- name: coverage .footer[*A sensible architecture for complex Node.js projects*] ## Code coverage ◼◻◻ With [`nyc`](https://www.npmjs.com/package/nyc) and [Coveralls](https://coveralls.io/) `package.json`: ```json "devDependencies": { "coveralls": "3.0.2", "mocha": "5.2.0", "nyc": "13.1.0" }, "scripts": { "test": "nyc -r html -r text mocha \"tests.js\" --timeout 2000 -R spec --colors", "build": "npm run audit && ⋯ && npm test && npm run browserify && npm run jsdoc", "coveralls": "nyc report --reporter=text-lcov | coveralls" }, "nyc": { "include": [ "server.js", "utils.js" ] } ``` --- .footer[*A sensible architecture for complex Node.js projects*] ## Code coverage ◼◼◻ ```json 7 passing (5s) -----------|----------|----------|----------|----------|-------------------| File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s | -----------|----------|----------|----------|----------|-------------------| All files | 69.77 | 59.62 | 66.67 | 70.4 | | server.js | 83.95 | 66.67 | 100 | 84.62 |... 45,147,152,154 | utils.js | 45.83 | 52.83 | 20 | 46.81 |... 53,55,57,59,61 | -----------|----------|----------|----------|----------|-------------------| ``` --- .footer[*A sensible architecture for complex Node.js projects*] ## Code coverage ◼◼◼ .center[.small[]] --- name: ci .footer[*A sensible architecture for complex Node.js projects*] ## Continuous integration * [Travis CI](https://travis-ci.com/) * [CircleCI](https://circleci.com/) * GitLab CI/CD (pipelines, jobs, schedules) * [GitHub Actions](https://github.com/features/actions) (?) --- name: check .footer[*A sensible architecture for complex Node.js projects*] ## Validate markup & CSS [`https://github.com/validator/validator`](https://github.com/validator/validator) `package.json`: ```json "scripts": { "validate": "npm run launch-server && \ java -jar ../../validator/build/dist/vnu.jar \ http://localhost:3000/register \ http://localhost:3000/login \ http://localhost:3000/sitemap \ http://localhost:3000/films/the-matrix \ ⋮ static/about.html static/terms-and-conditions.html ⋮ " ⋮ } ``` --- name: doc .footer[*A sensible architecture for complex Node.js projects*] ## Documentation With [`jsdoc-to-markdown`](https://www.npmjs.com/package/jsdoc-to-markdown), Handlebars templates, and some post-processing `package.json`: ```json "devDependencies": { "jsdoc-to-markdown": "4.0.1", }, "scripts": { "doc": "jsdoc2md *.js lib/*.js | sed 's/^### $//' > .github/README.md", }, "jsdoc2md": { "template": "template.hbs", "heading-depth": "3", } ``` `template.hbs`: ```json ⋮ ## Documentation {{>main}} ``` --- name: other .footer[*A sensible architecture for complex Node.js projects*] ## Other tasks **Testing**: [`mocha`](https://www.npmjs.com/package/mocha) + assertion libraries **Logging** (neglected “aspect”): [`winston`](https://www.npmjs.com/package/winston) **Generation of client-side assets**: ```json "devDependencies": { "sass": "1.15.1" }, "scripts": { "compile": "npx sass --no-source-map -s expanded foo-bs.scss static/foo-bs.css && npx sass --no-source-map -s compressed foo-bs.scss static/foo-bs.min.css", "build": "npm run audit && ⋯ && npm run compile && npm run test && ⋯ " } ``` **Process management** in production: [PM2](https://pm2.io/doc/en/runtime/overview/) --- name: resources .footer[*A sensible architecture for complex Node.js projects*] ## More resources * W3C: * [Node.js best practices and recommended tools](https://w3c.github.io/nodejs/doc/) * [Resources about JavaScript and Node.js](https://www.w3.org/wiki/Resources_about_JavaScript_and_Node.js) * [Public JS repos on GH](https://github.com/w3c?language=js) * [W3C Developers](https://www.w3.org/developers/) (validators, checkers, MOOCs, events…) * For demos, PoCs, tests, quick experiments: * [CodeSandbox](https://codesandbox.io/) * [RunKit](https://runkit.com/home) --- name: qa class: middle .footer[*A sensible architecture for complex Node.js projects*] ## Questions, <br /> <br /> comments, <br /> <br /> debate