Storing binary dependencies in your Version Control System is like storing packets of cereal in the deep freeze: expensive, and a little odd.  These days, we seem to do nothing but weave complex webs of code using open source or paid-for libraries.  But how do you manage those libraries?   Checking your projects' binary dependencies into version control is often the default choice.  Everybody knows that it's wasteful, but who wants to solve the problem?

At a certain level, it's pragmatic.  If you only depend on five .jar or .dll files in your project, check 'em in!  You're best off delivering some working software to the people who pay your wages than worrying about the best place to manage them.  But when do you stop?  When you have ten libraries to manage?  When you have twenty?  It's easy to start off being pragmatic and then find out one day that you can't make use of those great new features in 'commons-transmogrifier' because it'll take you two days to resolve the dependencies, all the while working out what those strange error messages are when 'commons-foo' tries to invoke methods on an odd version of 'commons-bar'.  Easy to start out being pragmatic and end up paying compound interest on technical debt.

So what's the solution?  You need a Dependency Manager.  It need not be the fancy tool that your architect wants you to use; it might be an HTTP server on an old desktop that contains all of your binary dependencies, and a build script to fetch those dependencies from the server.

It might be as simple as that.  At the other end of the scale you might have a commercial product. It doesn't matter. What does matter is that you're able to easily declare the dependencies of your codebase.  Using 'commons-agnosticator'?  Version 0.9a?  Fine. That's a lot more helpful than saying 'Bob checked in a version of 'commons-agnosticator' last summer.  If you're lucky he put the version number in the commit.  Now that you've stated what your codebase needs, your dependency manager can run along and get that.  This is particularly useful when you're dealing with a platform that doesn't make the versioning of your binary dependencies mandatory.  I'm looking in the direction of Santa Clara. Or is that Redwood City?

What about the dependencies of your dependencies?  Don't go checking in libraries into the codebase without thinking about those. Have you got all the logging and utility libraries that you need there?  Once you manage to enumerate those, then the dependency manager will mentally construct a dependency tree, resolve all the libraries that it needs, and cheerfully nip off and fetch them.  To me, that's where dependency managers start to become valuable.  Transitive dependencies (the dependencies of your dependencies) can become a bore.  Why did a developer check in a utility library last autumn?  Was it to support that SVG library that you pulled out in the winter of your discontent?  Is it required by anything else?

I once wrote a Ruby program out of sheer frustration to help resolve this.  It would attempt to disable a library in your project, and then build your project.  If everything worked fine, it would report the library as possibly unused.  The code's on GitHub, but be warned - it's probably nasty.  [http://github.com/simpsonjulian/djbt/blob/master/bflb.rb] Why the heck did I do that? Because the project I was working on had dozens and dozens of dependencies - some of which weren't used at all. With a dependency manager you start by declaring the first class libraries that you need. Their dependencies just arrive with the real dependencies of the code, and you don't need to worry about them.

So far I've been writing about third party and open source dependencies. What about the artifacts that your organization produces?  You can put them in the repository, too. With a little work you can make the process of producing and consuming binary artifacts easier.  You can configure your Continuous Integration system to publish to the repository after every successful build, or you can publish a release to the repository when you think the time is right. It’s up to you.

So what dependency managers can you use?  For Java, the daddy of them all is Maven (don't panic, you don't have to convince all the developers in your project to switch build tool- read on).  The heart of a Maven project is the POM – an XML file that declares all the facts about your project for the tool to act upon.  Included in these facts are the external dependencies that your application needs to build and run.  You don't need to tell it much else – Maven insists that your source trees are laid out in a predictable manner, so once it knows some metadata about the project, and any dependencies, it'll just crack on with building your project and running tests.

Because there are no surprises in a Maven project, it's easy to write plugins for Maven.  So adding all those useful things like code coverage and other quality measures is a matter of installing something like Sonar[http://www.sonarsource.org/] and then moving on.  There's a decent ecosystem of other plugins available for Maven.

Maven may be a step too far for some projects, however.  If you've built up a few years of code around another build tool, migrating might be a big job.  There's no need to suffer in silence though – you've got two choices.  The Maven project also supplies plugins for Apache Ant, so you can use the same POM file to describe dependencies in your project.  You'll need a step in your build to invoke the plugins and go fetch all the dependencies.  And their dependencies. And so on. And so on.

The other option is Ivy.  Ivy has been around for a few years as a standalone dependency manager.  Recently it became part of the Apache Ant project and as such has excelved integration with Ant.  It works in a similar way to Maven and can use the same repositories as Maven with a little configuration.  Ivy also uses an xml file to declare dependencies – but in the case of Ivy, only dependencies.  In Maven you'd configure loads of other things.  The format of the XML file is different to Maven's but the concepts are very similar.

If you're working on .NET, don't feel left out.  There's a project underway called NMaven[http://nmaven.codeplex.com/], which is a port of Maven to the .NET platform.  I think NMaven represents a great opportunity for .NET shops; with some good IDE support it might allow teams to get a handle on the many enterprise components that you always end up dragging into your projects.

Of course, nothing comes for free.  Implementing a dependency manager in an in-flight project might not be easy.  You'll need to educate the team about the build changes and deal with the inevitable teething problems.  You'll need to change the process by which you integrate code between teams and departments, without losing the communication that you have.  Integration boundaries often seem to reflect organizational boundaries, and you don't want to replace inter-team communication with yet another enterprise tool.

The introduction of a dependency manager can have subtle (and not to subtle) effects on the developer environment.  Changing the libraries that your IDE has come to know can confuse it and cause it to scan the filesystem looking for changes.  On one project that I worked on we had to make two copies of a library directory: one for the IDE, and one for the build.  The Ant build was the canonical build, so if the IDE had to have an injection of dependencies, the developer could run a build task to do that.

There's also the security risks of depending on code that your build tool just pulled from the Internet.  As always, there's a trade-off between security and convenience: you can get 20 libraries in under a minute, or you can take hours to download, verify md5sums, and check them into your project.  If you're in doubt, err on the side of caution and maintain your own repository of verified dependencies.

If you're suffering from DLL hell, or you're scarred by Jars, a Dependency Manager might be just the ticket for your team.  There's a few choices to make, so I'd suggest that you investigate your options carefully.ion and are still no silver bullets in our industry; but the right toolthe right attitude can make a world of difference.

Julian Simpson writes about building and deploying software at The Build Doctor [http://www.build-doctor.com].  He lives in the UK because he doesn’t like the noise of air transport.  You can follow him on Twitter at @builddoctor[http://twitter.com/builddoctor].

Trackback(0)

TrackBack URI for this entry

Comments (0)

Subscribe to this comment's feed

Write comment