Today I want to share with you some pain and thoughts on current state of npm packages and node_modules folder in your application root.

Most of the times npm packages are declared as dependencies of your app in package.json and installed every time you need them. Generally there should not be much more than single npm install for each working copy of your project.

The Pain

But things get complicated when you decide to use continuous integration/deployment, where your project is being constantly built from scratch. Our TeamCity instance builds about 150 times a day on average and npm packages are installed every single time. Usually it takes about 30 seconds to Mars install all packages from local npm mirror. That is about 75 minutes every day spent on repeated installs of the same npm packages. Ouch. I think that’s a lot of time and potential bottleneck.

Moreover, npm itself is not very stable and I often end up resolving some weird unclear and hardly reproducible portability issues, cleaning caches and reinstalling different versions. Just recently there was a network-related accident with our registry mirror, at the same time several broken packages were published on npmjs.org, European mirror had some outdated packages and npm@1.4.7 somehow became extremely buggy at our environment. An entire team was unable to work for several hours. And this really makes me upset.

Alternative View

I came across an old article where Mikeal Rogers talks about the reasons for committing your node_modules into git. For the first time this may sound crazy to you, but the justifications he gives are very strong. And if you’re still not convinced, take a look at this question on StackOverflow. Just in case you haven’t noticed, all three top-voted answers recommend the same. Even npm documentation agrees with this approach!

Actually, original article was about fixing dependencies of your dependencies and now there is npm shrinkwrap to the rescue. But it still does not give you full confidence in deployment. And I decided to try and committed my node_modules folder into git. You know what?

Authors Doesn’t Care

Some npm packages are so huge that committing them becomes scary:

• bower installation takes up to 31 megabytes on your drive! OMG, is this real? Please tell me that I need all that stuff. Node.js itself is only 11 megs and npm is 10 MB! I’m gonna switch!
• grunt is 6 MB. I love you guys, but I really don’t need CoffeeScript there.
• mocha is 1.5 megs, but my TeamCity doesn’t care that it has jade and growl.
• express is 872 KB. Well, that’s great, isn’t it? But I believe you can do it even smaller.

No offense! I hope that you can see the trend now. Dear contributors, please don’t forget how much space and bandwidth is consumed by your library!

Think of it as a real life product such as notebook with pre-installed Windows or almost any modern smartphone. They often have dozens of useless builtin branded crap like audio players, image editors or Chinese keyboards which you’ll try to uninstall as quickly as possible and continue to use good old whatever-amp. But usually you are doomed to live with this bloatware forever.

Anatomy of npm Package

Let’s now talk about useful payload of your npm package. I asked this question on StackOverflow but it has not received as much attention as I would have liked.

Speaking in terms of object-oriented design, your npm package should be highly cohesive and loosely coupled. Your package should do only one thing and do it well.

So, what should you put into npm package?

README.* (if any) and package.json can’t be ignored so you just can’t skip them. Next thing is library core functionality, the stuff people install your lib for. No bells and whistles. If your package is about trimming strings, please don’t make it occasionally brew beer. No doubt, it’s great and useful, but you should put it in another brewmaster package. And one more thing… oh, there’s not. That’s all we need!

From the other side, here are the things that have no place in your package:

• Dev-only files like Gruntfile.js, .jshintrc, .travis.yml and so on. Nobody needs your code style config in their app. Also make sure that external dev-only packages are listed in devDependencies instead of regular dependencies.
• Tests. You should not include tests in your distribution package unless your library is heavily depend on platform and it’s behavior should be verified.
• Different versions of the same functionality. You better show me how to build one from source by myself.
• Hosted docs, logos examples and other assets. Beautiful logos, nodemailer! Sorry, I can’t take it with me, it adds another 200 KB to my repo.
• Other stuff that can be optional: plugins, contribs, rulesets etc.

Use your .npmignore wisely. Start with:

*

!lib/**
!bin/**
!index.js

You can also use files and directories fields in package.json. Some helpful links to read:

• npm Developers Guide
• npm package.json description

Example

As a recent example please have a look at teamcity-service-messages (don’t forget to star it!). This package has no external dependencies and installed instantly:

teamcity-service-messages
├── LICENSE.txt
├── README.md
├── index.js
├── lib
│   └── message.js
└── package.json

There is nothing extra.

Underscore.js is doing good job too:

underscore
├── LICENSE
├── README.md
├── package.json
├── underscore-min.js
└── underscore.js

Wonderful! 5 files, 76 KB only! I’m ready to commit it right away!

Conclusion

Libs developers. Please, include only vital components of your library to npm packages. Review your payload and deps now.

App devs. Please, remove node_modules from your .gitignore. Make your development and deployment a little bit simpler and faster.

What do you think about this? Have you experienced similar problems? Have you ever thought about committing your node_modules into git? How can we help each other?