>> Arborist Deep Dive

Quite a lot has happened in npm since our last update way back in 2019. We're overdue for a status update on npm v7.

Despite some massive distracting changes (some unfortunate, some very fortunate), development work has been proceeding steadily.

Yesterday, Edward Thomson presented a demo of a few of the features coming in npm v7 at GitHub Satellite. The Q&A session was awesome, and it was clear that a lot of you are excited for this update to the npm CLI, and have a lot of questions that unfortunately couldn't be adequately answered in such a short time.

This post started as a brief update on what we've been doing, intended to go live along with the talk, but it got long. So, instead, this is the first post, with more to come, where I take you all through some of the major changes coming in npm v7, the thinking and motivation behind them, and how they might affect you and your work.

Vision

npm v7 is based on a handful of technical and user experience principles, and each post in this series will tie back to one or more of these in some way.

Reduce noise that is not actionable. Or, in other words "make npm yell at you less". We've gone through the entire project from the data management to the presentation layers, stripping output that doesn't provide worthwhile information.

Manage your packages for you. There are a handful of places in npm where v6 and earlier versions sort of throw up their hands and make you fix stuff by yourself. But npm is a package manager! It shouldn't make you manage packages, that's its job!

Strict separation of concerns. Moving forward, the npm CLI itself is becoming strictly the user-interface layer, and we've moved out all complex tree management and registry interactions to @npmcli/arborist, pacote, and the various libnpm* modules.

Be as fast as possible while behaving correctly. The key to fast software is to do as little as possible, including doing as little waiting as possible. From the refactor to use Minipass streams to refactoring out a decade's worth of compatibility fixes for long-dead Node.js versions, npm v7 is leaner and faster because it's lazier.

If we know the right thing to do, let's not yell at you to do it. Let's just do it for you, and do it all the way the first time. That's what npm loves you means. It means taking care of you and your project like a good and faithful assistant, only troubling you to get involved when absolutely necessary.

Coming Attractions

In the coming npm v7 posts, we'll cover:

A deep dive on Arborist, the tree management and analysis utility that does a lot of heavy lifting in npm v7.

Updates and impacts from some of the major refactoring that's happened in the npm CLI dependency stack over the last year.

Automatically installing peerDependencies: why, how, and what you might want to be aware of.

Improvements to the performance, effectiveness, and UI of npm audit.

Updates to the package-lock.json format, added handling for yarn.lock files, and why we are going to keep using package-lock.json.

Support for acceptDependencies

Changes to how npm handles install scripts (RFC0021 RFC0022 and RFC0024)

Fixing the "noisy chokidar/fsevents" problem, and how this is related to npm fund.

Updates to permission handling while running as root.

Dropping the --depth config in npm update and npm outdated.

Changes to config options (especially, config options that determine what gets installed).

Improvements to the handling of "files" in package.json.

Workspaces

Overrides

Staged publishes

And, presumably, updates about the beta and eventual GA releases, because this is going to take a minute :)

RFC Process

The npm CLI team has been running an open RFC process for the last several months now, and the impact on our development process has been great.

Usually, writing the code is the easy part. The hard part is all the thinking and talking that goes into making sure we're writing code for a feature that users will benefit from, and that we've explored the edge cases well enough to be confident we're not overlooking something important.

If you're interested in getting involved with npm's design process, we want you to participate! Building something like this is a team effort, and we all benefit from getting a broad range of perspectives from our community.

Is there something that you really wish npm would do, or do differently, or stop doing? Head over to npm/rfcs and give us a proposal. Obviously the more engaged and thorough you can be, the more easy it'll be for us to get it done, but every great idea has to start somewhere, so don't feel like you have to have all the answers before you begin :)

Thanks!

If there's something you'd like to see covered around npm v7, you can post an issue on the npm/npm-v7-blog repo.

Stay tuned for more from us, and I hope these posts prove to be fun and informative!

This week npm@6 is going to be promoted to latest and so now is an excellent time to look forward. If you dig into it you’ll find that it doesn’t have much in the way of breaking changes. Later this year we’ll be releasing npm@7. First there are a few new features, these will initially ship as npm@6 minor releases:

We will be introducing package aliases as described in #5499 for use with assets, to support conflict resolution.

As an optimization, npm install will be updated to be an alias for npm ci when they would produce the same result, that is, that your package-lock is compatible with your package.json and you’re starting with no node_modules.

Combine bundling and the v8 compile cache to reduce latency when running the npm cli.

The breaking changes in npm@7 should not be disruptive but are important:

npm login and npm adduser will finally be split apart. Up till now they’ve been the same command, which is why npm login prompted for an email address.

npm init will be getting an overhaul, both with allowing easy third-party init templates and with some new one’s from us, to help you configure your CI and git environments.

Changes to how npm runs under sudo and how it handles permissions as root to make it less surprising. Look for a new RFC soon!

The npm link rewrite as described in its associated RFC.

Output from preinstall, install and postinstall scripts when running for a dependency will have their output captured and it will only be displayed if there’s an error. They will run with stdin closed. This means that install lifecycle scripts in projects installed as dependencies will not be able to be interactive. For all other lifecycle scripts, output will be captured and routed to the log and also outputted to stdout.

Allow package aliases to be used for any dependency type.

After npm@7, in the late summer and fall of 2018, our plans are somewhat more tentative, but we have a few things we’d like to get out to you all:

npx has been a huge success, but it can mean that you’re running untrusted code from the registry. We have a number of thoughts about how to improve its security, from looking for known vulnerabilities to catching typo-squatting.

Integrating the functionality of shrinkpack into npm directly. This would allow for entirely offline deploys even without a cache. It will also let you deploy with git dependencies without having git installed on your production machines.

We have two installation engines with different tradeoffs in npm install and npm ci. If one fails due to certain classes of error then we should try the other.

Workspaces, as seen in Yarn, for management of certain kinds of monorepos.

An option to run npm in low memory environments, this would disable in-memory caching of data at the cost of speed.

Begin exploring updating dependencies in your package.json from your source code… (npm init auto?)

As we work our way through the year we will likely be rewriting a number of other components in npm, either as we need them or as the community provides them.

Stand alone libraries to generate a “logical tree”, that is, one that represents the relationships between modules but not their locations on disk.

New, faster, on-disk tree readers that produce trees like the above from an existing node_modules.

Two-pass installer where resolution of the logical tree is distinct from computing the physical tree. With this we may be able to, at long last, end the pain around peerDependencies.

Many years ago, npm’s configuration was extracted as a stand alone module, but it was so tightly coupled with the rest of npm that we ultimately brought it back in. We’ve been slowly working on extracting it again, carefully decoupling it from the rest of the code base.

npm outdated could use with a rewrite as it has the hackiest of shims to make it work at all. It was originally designed to run off of read-installed output which is what npm@1 and npm@2 used to read node_modules. A version that was designed to run off of modern npm data structures would be much easier to understand.

Similarly, npm ls would benefit a great deal from being updated to work off of modern data structures.

We’ve nearly eliminated npm-registry-client (and in turn request) from the npm code base. With the advent of npm-registry-fetch (which is based off make-fetch-happen and in turn node-fetch) we can finally complete this.

Once we have support for Workspaces, npm itself could benefit from bringing a bunch of its closely related modules together into a monorepo.

For more…

Watch this blog, follow @npmjs on Twitter and subscribe to our newsletter!