The new npm CLI: a year in review; or, what you may have missed!
First published just under a year ago, npm@5 has probably seen the fastest rate in major changes of any prior npm version. Even if you’ve been following us closely, you probably still haven’t been able to keep up with everything that’s been going on with npm@5.
With [email protected] getting tagged as latest this week, we thought it would be a good idea to summarize all the changes that have happened since [email protected] was first tagged — and what you can expect to get in one upgrade when you first install [email protected]!
This one’s first because it’s the most noticeable change for anyone doing the initial switch. npm is now between 4x and 17x faster than npm@4, especially in CI settings. This means developers can iterate on their changes faster, share their changes faster, and spend less time waiting between pushing a new build and continuing deployment. These sorts of speed differences are not just about making existing things fast, either—they’re about opening doors to entire new opportunities and allowing our users to scale projects beyond what they could do before.
We’ll continue to improve performance—both the CLI and registry teams are dedicated to getting you all that code as fast as your hardware allows it.
package-lock.json and automatic conflict resolution
Along with the speed change, you might have noticed a new file in your git repositories. package-lock.json is a so-called “lock file” that saves information about your node_modules/ tree since you last edited your dependencies. Even though it’s a generated file, it’s meant to be committed into git and has a number of benefits, including increased reproducibility across teams, reduced network overhead when installing, and making it easier to debug issues with dependencies.
Because this was such a significant change to the way npm works, initial iterations of package-lock.json were a bit of a bumpy ride, with unexpected changes and platform differences. As of npm@6, your lock file will be stable not just between you and your coworkers on similar platforms, but across operating systems! You can even use npm install --package-lock-only to generate one of these without installing into node_modules/.
Some folks faced issues when using package-lock.json that led to inscrutable git conflicts. As of [email protected] (and, by extension, npm@6), npm will now automatically resolve package-lock conflicts when you run npm install during a conflicted state. To make this experience even smoother, we’ve released npm-merge-driver, which lets you do all the rebasing and merging and other git backflips. You can get started with the merge driver by doing $ npx npm-merge-driver install -g one time. After that, all your future merge and rebase conflicts in any git repo will be resolved in the background.
Those of you who used package-lock.json throughout the npm@5 release line will see a big git diff next time you do an npm install with npm@6: the requires field is changing its format slightly, and we’re adding a new from field to locked git dependences (if you use those) but you should not see this happen again if you and your team are all on npm@6 or later.
What’s that npx thing we mentioned? Why, it’s the npm package runner! [email protected] introduced a new workflow tool included with npm itself. npx solves a lot of specific problems, but its overall purpose is making the experience of working with npm-based CLI tools easy, smooth, and seamless!
npx can actually do a lot of different things, but the biggest benefits are:
You can run project-local binaries with it. That means you don’t need to install things like grunt-cli, gulp, bower, and tsc globally—you can install them as devDependencies and use npx to run the local versions without hassle (for example, $ npm i -D standard && npx standard).
You can do one-off, temporary installs of command line utilities that you run rarely, such as generators: $ npx create-react-app.
You can easily try different versions of these tools with a single install if you need to compare them: $ npx standard@8 && npx standard@10.
You can even install the shell auto-fallback to not even have to write $ npx ... in many cases.
npx has already made a huge difference in the daily workflow of many of our users. Have you tried it yet?
Thanks to the new package-lock.json feature, we were able to add a cool new command in [email protected]: npm ci! This added a second installer to the npm CLI that takes advantage of projects with package-lock.json files in order to rapidly extract to node_modules/. It skips a lot of the interactivity and human-oriented checks that npm install does, and even makes it so that any inconsistencies trigger errors, rather than having the npm install command fix them for you. npm ci, as the name implies, is meant primarily for use in Continuous Integration/Continuous Deployment situations where you want fast installs that give you early warnings about potential errors. npm ci is about 2–3x faster than a regular npm install.
Finally, as with npm it, there’s now an npm cit for doing an npm ci install + npm test in a single command.
Two-factor authentication and token management
Last October, npm introduced two-factor authentication for a number of account-related actions, including publishing and dist-tag management. Roughly 9 billion weekly downloads—about 43% of downloads—now involve packages that were published by 2fa-protected accounts. That’s amazing news for the overall security of the ecosystem.
As part of that release, we also included the ability to create “read-only” tokens, filter tokens by CIDR, and manually manage your active tokens through the CLI and the website.
Brand new cache and offline installs
npm@5 introduced a completely new system cache for package downloads, replacing the older, buggier, and slower implementation. It was built on top of a standalone library, cacache, and was designed to work closely with the new package-lock.json format by enabling content-addressable storage of package data. This allows it to do much faster lookups of tarball data. It also allows npm to verify your package data on every single install, so you can always trust that what comes out of the registry or the cache is exactly the data you expect it to be.
This process also upgraded our shasum infrastructure to support Subresource Integrity, with new tarballs being published with sha512, instead of the now-insecure sha1 algorithm. This method also makes it much easier to upgrade our infrastructure once faster and better cryptographic hashing algorithms become available in Node.js.
Finally, the new cache architecture enabled a much-awaited npm feature: seamless offline installation! If you don’t have network access or can’t reach the registry for some reason, npm will automatically install from your cache without the need for any extra flags or preparation. Just keep a warm cache and go ahead and do your work on a plane or while you’re disconnected from the network to preserve bandwidth. If you’re on a functional but slow connection, you can even use the new --prefer-offline flag to opt into stale data in exchange for minimal network hits.
npm ci and lock files aren’t the only exciting feature for devops folks! As part of npm@5, Isaac completely rewrote node-tar, which is a huge part of the recent performance boost. It also gave us the ability to pack tarballs with pinned file timestamps.
npm can now generate and publish tarballs built at different times on different platforms with identical shasums, enabling reproducible builds!
Keep in mind that you still need to make sure your build run-scripts generate reproducible build artifacts for this to work!
npm publish and npm pack improvements
npm now prints a summary of tarball details whenever you package up an npm project. This summary includes information about the files that you’re including, along with their file sizes (which can help make sure you’re publishing only what you intend), the name and version of the package, and the shasum and integrity for the generated tarball.
In addition, both npm publish and npm pack now support --dry-run and --json flags. With these, you can test that you’re getting what you expect before you actually publish your project. If you’re trying to get reproducible builds working, you can use --dry-run to check that your scripts are generating your published files consistently.
Because npm publish and npm pack both accept any package specifier, you can even use them to test other people’s packages to make sure, for example, that the tarball published to npm actually matches the tagged git version: $ npm pack --dry-run github:zkat/figgy-pudding && npm pack --dry-run figgy-pudding.
You can even hook into the new prepack, pack, and postpack lifecycle scripts along with the new prepare lifecycle to automate this detection as desired.
npm has also greatly improved its interaction with git dependencies, adding two major features to git deps:
First, git-based packages with prepare scripts defined will have their devDependencies installed and the prepare script executed before the package tarball for the dependency is packaged and installed. This means that if you want to work off a fork but you want a build step for your library, you can still have that build process! You don’t need to check in built/minified .js directly into git anymore—npm will generate them for you.
Second, npm added semver-range support for git dependencies. Just like you can do $ npm install pkg@^1.2, you can now do npm install github:usr/pkg#semver:^1.2. Semver support will work off branch and tag names, so as long as you keep those up, you’ll be able to use semver with them.
These features should help smooth out the experience for users that are heavily reliant on git dependencies.
npm hooks have been available for a couple of years, but the only way to manage them was through a separate CLI. As of npm@6, you can manage hooks directly with npm itself.
This feature allows you to add webhooks to npm packages, scopes, or users, and have the npm registry post to an external endpoint when related events happen. Hooks can be a great way to integrate npm into your release flow or to get better analytics. Check them out!
file: dependency symlinking
Previously, directory dependencies (that is, what would install if you did $ npm install ../some-pkg), would go through a two-step process: first, they would be packaged into a tarball, as if for publishing, and then that tarball would be installed as a dependency. However, updating that dependency’s code in the future required that you do a full install of it again. That caused some friction for users wanting to work based off local packages or who were working on monorepos.
Now, those dependencies are installed as symlinks, so any changes you make to them will be immediately visible to your package without needing to do a whole new install.
Improved output and usability improvements
npm@6 also includes a number of smaller aesthetic and usability improvements we’ve been incrementally landing in npm:
npm install saves by default
npm install now savess dependency changes by default. You no longer need to remember to add --save before your dependencies get written to package.json (and package-lock.json). This applies to other commands as well, like npm update and npm uninstall.
New npm view output format
Previously, you’d get several screenfuls of colorful JSON output any time you did $ npm view. Now you get this:
You can still access the JSON version by using --json or you can access specific packument fields with $ npm view [...], but the default view is now much easier to view and understand and it includes what we think is the most essential information about the package.
Running npm install in npm@4 would get you a massive amount of tree output. This output made sense when projects were smaller, but it eventually became unreadable for users and took a long time to calculate and print out.
Now, this is all you get when you run npm install on a big project:
We also added the by N contributors section to increase awareness of everyone putting effort into improving the npm ecosystem. You can find out how to give back by running $ npx thanks.
npm update respects latest
The npm update command was a weird creature before: unlike npm install, it would ignore the latest tag and instead install the highest semver-matching version available for that package. Sometimes, these higher versions were intended as prerelease versions and could cause unintentional installs of unstable software.
As of npm@6, $ npm update foo and $ npm install foo will have similar semantics on this front, with the exception that update will limit its install to a version matching the currently-saved semver range for that package.
install and update skip deprecated versions.
Along those lines, if you try and install a package and it has deprecated some of its versions, npm@6 will try to avoid installing the deprecated ones and find other matching versions if possible. Deprecated versions will only be installed if there’s no other way to fulfill the requested version.
Module: example Versions: 1.0.0 1.1.0 1.1.2 1.1.3 (deprecated)
There’s a lot of very exciting changes coming in 2018! For more info about what we have planned, take a look at this post.
We’re working hard this year to continue minimizing friction for JavaScript developers and make the experience of using npm as smooth, seamless, and secure as we can. npm is the largest package ecosystem in the world and continues to grow at an incredible pace—we’re glad to have the opportunity to make that possible and bring JavaScript and web development into the future it deserves!
Watch this space: follow @npmjs on Twitter for release updates and subscribe to our newsletter!
