One simple trick for JavaScript package maintainers to avoid breaking their user’s software and to ship stable releases
How to make use of npm’s package distribution tags to create release channels just like Google Chrome’s
Software releases are exciting for both package authors and users. The hard work of the past weeks is finally going to be manifested in this one shiny new version. All the to-do list items are crossed off, the tests are green, the release blogpost is ready. Soon it will be read and shared by thousands, Twitter is going to fill up with praise and cheering. Phew! Done! 💪🎉
The excitement of a release is an important factor to keep everyone involved in a project happy and motivated and it’s a lot of fun. While everyone should be entitled to a little celebration, there’s a recurring pattern that soon the first party poopers will appear on the issue tracker.
Help! I was so excited to try the new version, but now I get this weird error?! 😓
– Someone on the internet — probably
Ouch! With all the excitement, ambition, and pressure to finally get the new version out the building, a tiny regression snuck into the build. Might not look like the biggest deal, because the next patch release is immediately sent on the way, but depending on how many users there are this might have broken a few thousand builds already — and remember, this should be a time for celebration (or rest) instead.
Before revealing the solution for this, I want to give a recent real life example, to illustrate how relevant this problem is. Please note that I’m not trying to pick on the amazing people and their fantastic project I’ll mention now. This isn’t meant to be an attack, only constructive feedback and actionable advice.
On October 29th the author of the babel JavaScript compiler, a project that has about 20k downloads a day and thousands of projects depending on it, held the closing keynote at EmberCamp London. In this talk he released the brand new version 6.0.0 — live on stage. It took three attempts and several forced pushes, but eventually the release script ran to completion. Exciting news for the entire JavaScript ecosystem 🎉
Besides all the changes that have been made, the release blogpost highlights the intense efforts required by the maintainers (emphasis mine).
Anyone who’s ever been involved with an open source project knows that it’s a ton of work, but at the end of the day it’s incredibly gratifying to be able to contribute even the tiniest bit back to the community.
Sebastian has been up literally all night to finish all the last minute things needed to launch Babel 6 during his EmberCamp keynote. It’s currently 1am for me in San Francisco and I’m writing this blog post as quickly as possible, listening to EDM music to stay awake.
Only about 14 minutes after the initial 6.0.0 release version 6.0.2 had to be released. There is no hint in the changelog what this version addressed specifically, but in hindsight it doesn’t matter, as version 6.0.12 followed in the same night, as well as 6.0.14 and 6.0.15 in the next two days. Among the fixes is a misspelled package name. Their changelog states: “Gaps between patch versions are faulty, broken or test releases”.
This is a very good illustration of the pattern I was trying to describe above. Let’s have a look at how we can stop breaking things for our users and have a more sustainable way of addressing releases like that.
The “latest” dist-tag gets used by npm publish by default
Luckily, npm, the JavaScript package manager and registry of choice, has a simple feature that helps avoiding release chaos: Package distribution tags, or dist-tags, in short. In fact, if you ever typed npm install <package> before, you have implicitly used this feature already.
When we type npm install <package>, what we end up with isn’t necessarily the highest version of that package, but the version with the “latest” dist-tag attached to it. The “latest” dist-tag gets used by npm publish by default, which means usually we install the version that got published last.
Most of the time the last and the highest version are the same, but imagine doing a security fix for an older version, e.g. moving from 3.2.13 to 3.2.14, while the highest stable one is 4.6.12 already. If we only used npm publish, mainstream users who type npm install <package> will end up with the “latest” 3.2.14 after that, even though 4.6.12 is clearly higher.
This might sound a little confusing at first, but once we tap into maintaining multiple version tracks, it’s exactly what we need to gain full control over what users of a package install.
Now using npm publish --tag=previous for version 3.2.14 all the new or mainstream users would keep getting 4.6.12. Users who’re still using the outdated 3.2.13 version would automatically get the bug fix (assuming they’re using version ranges like ^3.0.0 or ~3.2.0). By explicitly attaching the “previous” dist-tag to version 3.2.14 we essentially created a new release channel, so everyone ends up with the version they expect. Using the tag command-line option it’s possible to create these channels with arbitrary names, for example “3.x-latest” would be a good fit, too (Tag names may not be a valid SemVer range, that’s the only exception).
- npm install <package> (or npm install <package>@latest) still results in version 4.6.12 being installed
- npm install <package>@previous results in version 3.2.14 being installed
- When there was “<package>”: “^3.0.0”, or “<package>”: “~3.2.0” in the package.json before npm install will automatically get 3.2.14
Now that we know how npm install implicitly uses dist-tags, let’s have a look at how we can use this for ✨ the future of our packages ✨
dist-tags work exactly the same for new and higher version tracks that are about to be released for the first time. If we stick with the example from above it could be version 5.0.0, the next major or breaking release. Again instead of using the bare npm publish we will make use of dist-tags to explicitly attach a different tag than “latest” to this release.
npm publish --tag=next. No mainstream user will get this new version automatically, but we can now invite everyone to try it out by running npm install <package>@next. Depending on the scope and maturity of this release we could just announce the release in IRC or Slack, but there is nothing that stops us from publishing a release blogpost and being vocal about it on Twitter or a conference stage even.
- npm install <package>@next results in version 5.0.0 being installed
- But again: npm install <package> (or npm install <package>@latest) still results in version 4.6.12 being installed
Think of this like Google Chrome, Chrome Beta, and Chrome Canary. These are all distributions of the same software, but in different states of maturity (release channels). Users know which level of stability they can expect while the development team can iterate and experiment with a very high speed and early feedback.
No doubt, the community, the people who are into this project, will give “next” a try immediately. Inside their projects, they can use existing tools and workflows. npm makes this process so straightforward for everyone, that there is no additional effort needed, only deliberate decisions and consent.
The result is: unavoidable regressions that sneak into every release will now be caught by your early adopters, with the difference that no one will install or deploy them, without making a conscious decision first.
Newcomers and regulars alike won’t be scared away by broken default versions. Once bugs are identified patches can be published in the “next” release channel the same way, for example version 5.0.1. As soon as there is enough confidence it is a single command to move the latest tag over to the highest version:
npm dist-tag add <package>@5.0.1 latest 👏
- Only now: npm install <package> (or npm install <package>@latest) results in version 5.0.1 being installed
Another effect of this technique is the reduced pressure to have everything ready and perfect on the first try. We can’t break anyone’s code. We can publish experimental, incomplete versions, or ones we aren’t 100% confident about yet. Just as we have linters, code reviews, unit and integration tests the “next” dist-tag is another layer of security which allows us to iterate faster, rather than slower.
Similarly we don’t simply push changes to master, we go through a branch and pull request based review process. In much the same way we should consider publishing versions with the “next” dist-tag first a best practice. Only after the early-adopter crowd integrated them into their projects and provided positive feedback we should promote versions to “latest”. I’d like to think of this as crowd testing. It is well-supported, it integrates into existing workflows without breaking them and can be used by everyone with very little to mostly no overhead at all. Naturally the crowd testing can only work well if there is a big, loyal crowd using your module, but no matter what: Using release channels acts as an additional security net to catch mistakes of any kind and adopting good habits early leaves you well prepared for overwhelming success that might follow later. And so that’s the simple trick that lets us ship stable releases to our users.
This release strategy is employed by npm itself and looking at their releases page you can see how well it’s working for them, even with two separate latest and next versions for their stable and LTS tracks respectively. The same concept is used for the development of the new version of Hoodie, a project I’m working on.
Looking back at the babel example from above, this could have saved quite a bit of stress, invaluable hours of sleep and broken builds — without losing any of the progress, excitement, and well-deserved celebration. Publishing version 6.0.0 of babel with the “next” dist-tag, would’ve still resulted in all the positive outcomes for the project, but without it being a broken default.
I have written a tool called semantic-release, which fully automates package publishing. It has full support for dist-tags built in, so there are no limits to automating the chores involved. There are several talks out there, where I talk about dependency and release management.
If you are struggling to keep your dependencies up-to-date and working, you should look into another tool I’m working on: Greenkeeper will drastically reduce the dull work involved. It’s free for open source.
I would love to hear your experiences with dist-tags and managing, or releasing packages, so please do not hesitate to tell me on Twitter.
Fun fact: In order to publish a valuable, well-written and understandable blogpost I invited folks to give me feedback on my first version before publishing it for everyone. What you’re reading now is a heavily improved and streamlined version. As you see, release channels do work.
Thank you Jake Champion, Lewis Cowper, Kent C. Dodds, Jan Lehnardt, Soledad Penadés, maxogden and the amazing npm CLI team for pioneering this technique.
