Home  Articles Columns Ask Mr. Make Painless non-recursive Make
|
Painless non-recursive Make |
|
|
Written by John Graham-Cumming
|
|
Monday, 04 June 2007 |
One of the common objections to using non-recursive Make is that with
recursive Make its possible to go to anywhere in the source code tree
and type 'make'. Doing so typically builds the objects that are
defined by the Makefile at that level in the tree (and possibly below
if the Makefile recurses).
Oftentimes, non-recursive Make systems (based on include statements instead of make
invocations) do not offer this flexibility and Make must be run from
the top-level directory. Even though non-recursive Make is typically
more efficient (meaning that running from the top-level directory
should be quick), it's important to be able to give developers the same
level of functionality as a recursive Make system.
In this
article I outline a pattern for non-recursive Make systems that
supports the familiar make-anywhere style common to recursive Make
systems. Typing 'make' in a directory will build everything in that
directory and below.
A simple recursive Make
Imagine a simple project with the following sub-directories:
/src/
/src/library/
/src/executable/
/src/ is the top level directory and is where you'd type 'make' to get a full build. Inside /src/ there's the library/ directory which build a library called lib.a from source files lib1.c and lib2.c:
/src/library/lib1.c
/src/library/lib2.c
The /src/executable/ directory builds an executable file called exec from two source files (foo.c and bar.c) and the library lib.a:
/src/executable/foo.c
/src/executable/bar.c
The
classic recursive Make solution to this means putting a Makefile in
each subdirectory that contains the rules to build the objects and a
top-level Makefile that recurses into each sub-directory. Here's the
contents of /src/Makefile
SUBDIRS = library executable
.PHONY: all
all:
for dir in $(SUBDIRS); do \
$(MAKE) -C $$dir; \
done
It
enters each directory in turn and runs Make to build first the library
and then the executable. The dependency between the executable and
the library (i.e. the fact that the library needs to be built before
the exectuable) is implicit in the order in which the directories are
specified in SUBDIRS.
An improvement on that is to unwind the loop inside the rule for all, create separate rules for each sub-directory and then explicitely specify the dependency between executable and library.
Here's how to do that using phony targets for each directory:
SUBDIRS = library executable
.PHONY: $(SUBDIRS)
$(SUBDIRS):
$(MAKE) -C $@
.PHONY: all
all: $(SUBDIRS)
executable: library
That's much clearer, but it's still recursive with separate Make invocations for each sub-directory.
A flexible non-recursive Make system
When moving to non-recursive Make the ideal top-level Makefile would look like this:
SUBDIRS = library executable
include $(addsuffix /Makefile,$(SUBDIRS))
That
simply says to include the Makefile in each sub-directory. The trick
is to make that work! Before showing you how I make that actually work
here are the skeletons of the contents of the Makefiles in the library and executable sub-directories.
# /src/library/Makefile
include root.mak
include top.mak
SRCS := lib1.c lib2.c
BINARY := lib
BINARY_EXT := $(_LIBEXT)
include bottom.mak
and
# /src/executable/Makefile
include root.mak
include top.mak
SRCS := foo.c foo.c
BINARY := exec
BINARY_EXT := $(_EXEEXT)
include bottom.mak
Each of those Makefiles specifies the source files to be built (in the SRCS macro), the name of the final linked binary (in the BINARY macro) and the type of the binary (using the BINARY_EXT macro which is set from special macros _LIBEXT and _EXEEXT).
To actually make this work they both include common Makefiles root.mak, top.mak and bottom.mak which are located in the /src/ directory.
Since the .mak
included Makefiles are not in the sub-directories GNU Make needs to go
looking for them. The simplest way to do that is to use the [t]-I[/tt]
command-line option to GNU Make that adds a directory to the include
search path. To find the .mak files in the /src you can do:
make -I /src
It's
unfortunate to ask a user to add anything to the Make command-line so I
use a simple method of automatically walking up the source tree to find
the .mak files. Here's the actual Makefile for /src/library:
sp :=
sp +=
_walk = $(if $1,$(wildcard /$(subst $(sp),/,$1)/$2) $(call _walk,$(wordlist 2,$(words $1),x $1),$2))
_find = $(firstword $(call _walk,$(strip $(subst /, ,$1)),$2))
_ROOT := $(patsubst %/root.mak,%,$(call _find,$(CURDIR),root.mak))
include $(_ROOT)/root.mak
include $(_ROOT)/top.mak
SRCS := lib1.c lib2.c
BINARY := lib
BINARY_EXT := $(_LIBEXT)
include $(_ROOT)/bottom.mak
The _find function walks up a directory tree starting from the directory in $1 looking for the file named $2. The actual find is achieved by calling the _walk function which walks up the tree finding every instead of the file $2 in each of the successively shorter paths from $1.
The block of code at the start of the Makefile finds the location of root.mak (which is in the same directory as top.mak and bottom.mak (i.e. /src/) and saves that directory in _ROOT.
Subsequently, the Makefile can use $(_ROOT)/ to include the root.mak, top.mak and bottom.mak Makefiles without any need to type anything other than make.
Here's the contents of the first included Makefile (root.mak):
_push = $(eval _save$1 := $(MAKEFILE_LIST))
_pop = $(eval MAKEFILE_LIST := $(_save$1))
_INCLUDE = $(call _push,$1)$(eval include $(_ROOT)/$1/Makefile)$(call _pop,$1)
DEPENDS_ON = $(call _INCLUDE,$1)
DEPENDS_ON_NO_BUILD = $(eval _NO_RULES := T)$(call _INCLUDE,$1)$(eval _NO_RULES :=)
For
the moment we'll ignore its contents and return to what these functions
are used for when we look at dependencies between modules. The real
work begins with top.mak:
_OUTTOP ?= /tmp/out
.PHONY: all
all:
_MAKEFILES := $(filter %/Makefile,$(MAKEFILE_LIST))
_INCLUDED_FROM
:= $(patsubst $(_ROOT)/%,%,$(if $(_MAKEFILES),$(patsubst
%/Makefile,%,$(word $(words $(_MAKEFILES)),$(_MAKEFILES)))))
ifeq ($(_INCLUDED_FROM),)
_MODULE := $(patsubst $(_ROOT)/%,%,$(CURDIR))
else
_MODULE := $(_INCLUDED_FROM)
endif
_MODULE_PATH := $(_ROOT)/$(_MODULE)
_MODULE_NAME := $(subst /,_,$(_MODULE))
$(_MODULE_NAME)_OUTPUT := $(_OUTTOP)/$(_MODULE)
_OBJEXT := .o
_LIBEXT := .a
_EXEEXT :=
The _OUTTOP macro defines the top-level directory into which all binary output (object files etc.) will be placed. It's defined with a ?= so that it can be overriden on the command-line, and I've given it a default vaule of /tmp/out/.
Next top.mak sets up the default target for Make as the classic all. Here is has no dependencies, but we'll add them later on for each module that's going to be built.
After that a number of macros end up setting the _MODULE_PATH to the full path to the module directory being built. For example, when building the library module _MODULE_PATH would be /src/library.
Setting this macro is complex because determining the module directory
has to be independent of the directory from which GNU Make was executed
(so that the library can be built from the top-level, for a 'make all',
or from the individual library directory, for an individual developer
build, or even can be included as a dependency on a different module).
The _MODULE_NAME is simple the path relative to the root of the tree with / replace by _. For the simple example the two modules have _MODULE_NAME's library and exectuable. But if library had a sub-directory containing a module called sublibrary then its _MODULE_NAME would be library_sublibrary.
The _MODULE_NAME is also used to create the special macro $(_MODULE_NAME)_OUTPUT which has a computed name based on _MODULE_NAME. So, for the library module the macro library_OUTPUT is created with the full path of the directory into which library's object files should be written. The output path is based on _OUTTOP and the relative path to the module being built. In that way the /tmp/out/ tree mirrors the source tree.
Finally,
some standard definitions of extensions used on file names are set up.
Here I've shown definitions for Linux systems, but these can easily be
changed for systems such as Windows that don't use .o for an object file or .a for a library.
bottom.mak uses these macros to set up the rules that will actually build the module:
$(_MODULE_NAME)_OBJS := $(addsuffix $(_OBJEXT),$(addprefix $($(_MODULE_NAME)_OUTPUT)/,$(basename $(SRCS)))) $(DEPS)
$(_MODULE_NAME)_BINARY := $($(_MODULE_NAME)_OUTPUT)/$(BINARY)$(BINARY_EXT)
ifneq ($(_NO_RULES),T)
ifneq ($($(_MODULE_NAME)_DEFINED),T)
all: $($(_MODULE_NAME)_BINARY)
.PHONY: $(_MODULE_NAME)
$(_MODULE_NAME): $($(_MODULE_NAME)_BINARY)
_IGNORE := $(shell mkdir -p $($(_MODULE_NAME)_OUTPUT))
_CLEAN := clean-$(_MODULE_NAME)
.PHONY: clean $(_CLEAN)
clean: $(_CLEAN)
$(_CLEAN):
rm -rf $($(patsubst clean-%,%,$@)_OUTPUT)
$($(_MODULE_NAME)_OUTPUT)/%.o: $(_MODULE_PATH)/%.c
@$(COMPILE.c) -o '$@' '$<'
$($(_MODULE_NAME)_OUTPUT)/$(BINARY).a: $($(_MODULE_NAME)_OBJS)
@$(AR) r '$@' $^
@ranlib '$@'
$($(_MODULE_NAME)_OUTPUT)/$(BINARY)$(_EXEEXT): $($(_MODULE_NAME)_OBJS)
@$(LINK.cpp) $^ -o'$@'
$(_MODULE_NAME)_DEFINED := T
endif
endif
The first thing bottom.mak does is set up two macros with computed names: $(_MODULE_NAME)_OBJS (which is the list of object files in the module computed from the SRCS macro by transforming the extension) and $(_MODULE_NAME)_BINARY (which is the name of the binary file created by the module; this would typically be the library or executable being built).
The $(_MODULE_NAME)_OBJS also includes any object files that are needed by the module but not built by it by including the DEPS macro. We'll see later how this is used to define a dependency between the library and executable in the example.
Next, if rules for this module have not previously been set up (controlled by the $(_MODULE_NAME)_DEFINED macro) and have not been explicitely disabled by the _NO_RULES macro the actual rules to build the module are defined.
In this example I show rules for Linux, this is where you'd change this example for another operating system.
First, all has the current binary (from $(_MODULE_NAME)_BINARY)
added as a prerequisite so that the module gets built when a full build
is done. Then there's a rule that associates the module name with
the module binrary so that it's possible to type something like make library at the top level of the build to build just the library.
Then there's a general clean rule and a module specific clean (for the library module there's a rule to just clean its objects called clean-library). Clean is implemented as a simple rm -rf since all the output is organized in a specific directory of _OUTTOP.
After that a $(shell)
is used to setup up the directory into which the module's output will
go. Finally, specific rules associated the object files in this
module's output directory and source files in this module's source
directory.
So, with all that infrastructure in place we can finally come to the Makefile in the executable directory:
sp :=
sp +=
_walk = $(if $1,$(wildcard /$(subst $(sp),/,$1)/$2) $(call _walk,$(wordlist 2,$(words $1),x $1),$2))
_find = $(firstword $(call _walk,$(strip $(subst /, ,$1)),$2))
_ROOT := $(patsubst %/root.mak,%,$(call _find,$(CURDIR),root.mak))
include $(_ROOT)/root.mak
$(call DEPENDS_ON,library)
include $(_ROOT)/top.mak
SRCS := foo.c bar.c
BINARY := exec
BINARY_EXT := $(_EXEEXT)
DEPS := $(library_BINARY)
include $(_ROOT)/bottom.mak
It
looks very like the Makefile for the library, but there are a couple of
differences. Since the executable needs the library the DEPS
line specifies that the executable depends on the binary file created
by the library. Since each module has unique macros for objects and
binaries, it's easy to define that dependency just by referring to $(library_BINARY) which will expand to the full path to the library file created by the library module.
To actually ensure that $(library_BINARY) is defined its necessary to include the Makefile from the library directory. The root.mak file provides two functions that make this trivial: DEPENDS_ON and DEPENDS_ON_NO_BUILD.
DEPENDS_ON_NO_BUILD
just sets up the macros for the specified module so that they can be
used in the Makefile. If that function were used in the executable
Makefile then the library (lib.a) would have to exist for the
executable to build successfully. On the other hand, of DEPENDS_ON is used this ensures that the library will get built if necessary.
DEPENDS_ON_NO_BUILD
provides functionality similar to a classic recursive build which
doesn't know how to build that library, but depends on it anyway. DEPENDS_ON is more flexible since without recursion you can specify a relationship and make sure that code is built.
OK, so what can we do with this?
This
system provides a great deal of flexibility. Here are just a few
examples that illustrate that the non-recursive Make system is just as
flexible as a recursive one (and more so!):
Building everything from the top-level is a simple Make (in all these examples I do a make -n so that the commands are clearly shown):
$ cd /src
$ make -n
cc -c -o '/tmp/out/library/lib1.o' '/home/jgc/doc/nonrecursive/library/lib1.c'
cc -c -o '/tmp/out/library/lib2.o' '/home/jgc/doc/nonrecursive/library/lib2.c'
ar r '/tmp/out/library/lib.a' /tmp/out/library/lib1.o /tmp/out/library/lib2.o
ranlib '/tmp/out/library/lib.a'
cc -c -o '/tmp/out/executable/foo.o' '/home/jgc/doc/nonrecursive/executable/foo.c'
cc -c -o '/tmp/out/executable/bar.o' '/home/jgc/doc/nonrecursive/executable/bar.c'
g++ /tmp/out/executable/foo.o /tmp/out/executable/bar.o /tmp/out/library/lib.a -o'/tmp/out/executable/exec'
As is cleaning everything:
$ cd /src
$ make -n clean
rm -rf /tmp/out/library
rm -rf /tmp/out/executable
From the top-level directory its possible to ask for any individual module to be built or cleaned:
$ cd /src
$ make -n clean-library
rm -rf /tmp/out/library
$ make -n library
cc -c -o '/tmp/out/library/lib1.o' '/home/jgc/doc/nonrecursive/library/lib1.c'
cc -c -o '/tmp/out/library/lib2.o' '/home/jgc/doc/nonrecursive/library/lib2.c'
ar r '/tmp/out/library/lib.a' /tmp/out/library/lib1.o /tmp/out/library/lib2.o
ranlib '/tmp/out/library/lib.a'
And if we ask that the executable module be built the library gets built at the same time because of the dependency:
$ cd /src
$ make -n executable
cc -c -o '/tmp/out/executable/foo.o' '/home/jgc/doc/nonrecursive/executable/foo.c'
cc -c -o '/tmp/out/executable/bar.o' '/home/jgc/doc/nonrecursive/executable/bar.c'
cc -c -o '/tmp/out/library/lib1.o' '/home/jgc/doc/nonrecursive/library/lib1.c'
cc -c -o '/tmp/out/library/lib2.o' '/home/jgc/doc/nonrecursive/library/lib2.c'
ar r '/tmp/out/library/lib.a' /tmp/out/library/lib1.o /tmp/out/library/lib2.o
ranlib '/tmp/out/library/lib.a'
g++ /tmp/out/executable/foo.o /tmp/out/executable/bar.o /tmp/out/library/lib.a -o'/tmp/out/executable/exec'
OK, so much for the top-level; if we pop down into the library module we can build or clean it just as easily:
$ cd /src/library
$ make -n clean
rm -rf /tmp/out/library
$ make -n
cc -c -o '/tmp/out/library/lib1.o' '/home/jgc/doc/nonrecursive/library/lib1.c'
cc -c -o '/tmp/out/library/lib2.o' '/home/jgc/doc/nonrecursive/library/lib2.c'
ar r '/tmp/out/library/lib.a' /tmp/out/library/lib1.o /tmp/out/library/lib2.o
ranlib '/tmp/out/library/lib.a'
and, of course, doing this in the executable directory will build the library as well:
$ cd /src/executable
cc -c -o '/tmp/out/library/lib1.o' '/home/jgc/doc/nonrecursive/library/lib1.c'
cc -c -o '/tmp/out/library/lib2.o' '/home/jgc/doc/nonrecursive/library/lib2.c'
ar r '/tmp/out/library/lib.a' /tmp/out/library/lib1.o /tmp/out/library/lib2.o
ranlib '/tmp/out/library/lib.a'
cc -c -o '/tmp/out/executable/foo.o' '/home/jgc/doc/nonrecursive/executable/foo.c'
cc -c -o '/tmp/out/executable/bar.o' '/home/jgc/doc/nonrecursive/executable/bar.c'
g++ /tmp/out/executable/foo.o /tmp/out/executable/bar.o /tmp/out/library/lib.a -o'/tmp/out/executable/exec'
What sub-modules?
Suppose, that the source tree was actually:
/src/
/src/library/
/src/library/sublibrary
/src/executable/
where there's an additional sublibrary underneath the library which builds slib.a from slib1.c and slib2.c using the following Makefile:
sp :=
sp +=
_walk = $(if $1,$(wildcard /$(subst $(sp),/,$1)/$2) $(call _walk,$(wordlist 2,$(words $1),x $1),$2))
_find = $(firstword $(call _walk,$(strip $(subst /, ,$1)),$2))
_ROOT := $(patsubst %/root.mak,%,$(call _find,$(CURDIR),root.mak))
include $(_ROOT)/root.mak
include $(_ROOT)/top.mak
SRCS := slib1.c slib2.c
BINARY := slib
BINARY_EXT := $(_LIBEXT)
include $(_ROOT)/bottom.mak
To define it as a dependency of library means simply adding a DEPENDS_ON call to the Makefile in the library directory:
sp :=
sp +=
_walk = $(if $1,$(wildcard /$(subst $(sp),/,$1)/$2) $(call _walk,$(wordlist 2,$(words $1),x $1),$2))
_find = $(firstword $(call _walk,$(strip $(subst /, ,$1)),$2))
_ROOT := $(patsubst %/root.mak,%,$(call _find,$(CURDIR),root.mak))
include $(_ROOT)/root.mak
$(call DEPENDS_ON,library/sublibrary)
include $(_ROOT)/top.mak
SRCS := lib1.c lib2.c
BINARY := lib
BINARY_EXT := $(_LIBEXT)
include $(_ROOT)/bottom.mak
In this example I didn't add a DEPS line so the library doesn't depend on sublibrary at the object level we're simply declaring sublibrary as a sub-module of library that needs to be built if library is.
Going back and repeating the examples above we see that the sublibrary has been successfully included in the library build (and automatically in the executable build).
Here's the full build from the top, followed by a clean:
$ cd /src
$ make -n
cc -c -o '/tmp/out/library/sublibrary/slib1.o' '/home/jgc/doc/nonrecursive/library/sublibrary/slib1.c'
cc -c -o '/tmp/out/library/sublibrary/slib2.o' '/home/jgc/doc/nonrecursive/library/sublibrary/slib2.c'
ar r '/tmp/out/library/sublibrary/slib.a' /tmp/out/library/sublibrary/slib1.o /tmp/out/library/sublibrary/slib2.o
ranlib '/tmp/out/library/sublibrary/slib.a'
cc -c -o '/tmp/out/library/lib1.o' '/home/jgc/doc/nonrecursive/library/lib1.c'
cc -c -o '/tmp/out/library/lib2.o' '/home/jgc/doc/nonrecursive/library/lib2.c'
ar r '/tmp/out/library/lib.a' /tmp/out/library/lib1.o /tmp/out/library/lib2.o
ranlib '/tmp/out/library/lib.a'
cc -c -o '/tmp/out/executable/foo.o' '/home/jgc/doc/nonrecursive/executable/foo.c'
cc -c -o '/tmp/out/executable/bar.o' '/home/jgc/doc/nonrecursive/executable/bar.c'
g++ /tmp/out/executable/foo.o /tmp/out/executable/bar.o /tmp/out/library/lib.a -o'/tmp/out/executable/exec'
$ make -n clean
rm -rf /tmp/out/library/sublibrary
rm -rf /tmp/out/library
rm -rf /tmp/out/executable
Asking for the sublibrary to be built:
$ cd /src
$ make -n clean-library_sublibrary
rm -rf /tmp/out/library/sublibrary
$ make -n library_sublibrary
cc -c -o '/tmp/out/library/sublibrary/slib1.o' '/home/jgc/doc/nonrecursive/library/sublibrary/slib1.c'
cc -c -o '/tmp/out/library/sublibrary/slib2.o' '/home/jgc/doc/nonrecursive/library/sublibrary/slib2.c'
ar r '/tmp/out/library/sublibrary/slib.a' /tmp/out/library/sublibrary/slib1.o /tmp/out/library/sublibrary/slib2.o
ranlib '/tmp/out/library/sublibrary/slib.a'
And if we ask that the executable module be built the library gets built at the same time because of the dependency (and also the sublibrary):
$ cd /src/executable
$ make -n executable
cc -c -o '/tmp/out/library/sublibrary/slib1.o' '/home/jgc/doc/nonrecursive/library/sublibrary/slib1.c'
cc -c -o '/tmp/out/library/sublibrary/slib2.o' '/home/jgc/doc/nonrecursive/library/sublibrary/slib2.c'
ar r '/tmp/out/library/sublibrary/slib.a' /tmp/out/library/sublibrary/slib1.o /tmp/out/library/sublibrary/slib2.o
ranlib '/tmp/out/library/sublibrary/slib.a'
cc -c -o '/tmp/out/library/lib1.o' '/home/jgc/doc/nonrecursive/library/lib1.c'
cc -c -o '/tmp/out/library/lib2.o' '/home/jgc/doc/nonrecursive/library/lib2.c'
ar r '/tmp/out/library/lib.a' /tmp/out/library/lib1.o /tmp/out/library/lib2.o
ranlib '/tmp/out/library/lib.a'
cc -c -o '/tmp/out/executable/foo.o' '/home/jgc/doc/nonrecursive/executable/foo.c'
cc -c -o '/tmp/out/executable/bar.o' '/home/jgc/doc/nonrecursive/executable/bar.c'
g++ /tmp/out/executable/foo.o /tmp/out/executable/bar.o /tmp/out/library/lib.a -o'/tmp/out/executable/exec'
Conclusion
Although
nothing like as simple as recursive Make, this non-recursive system is
very flexible (allowing dependencies between individual binary files
across modules; that's not possible with recursive Make) without losing
the 'go to any directory and type Make' that engineers are familiar
with.
Let me know if you use it, or improve on it.
John Graham-Cumming is Mr Make and Chief Scientist at Electric Cloud, Inc., a Silicon Valley start-up that speeds up builds by 10-20x using cluster technology.
Trackback(0)
 |
|
Recent Issues of CM Journal
Whitepaper SpotlightStay up to date with Configuration Management and Application Lifecycle
Management technology products and services by browsing our featured white
papers below:
See all the Featured Whitepapers>>
|
Tool Spotlight
|
|
|
CollabNet
CollabNet Subversion is an enterprise-ready distribution of Subversion® that includes, in one package,...
Read More
|
|
|
AccuRev
AccuRev is a best-of-breed, process-centric software configuration
management (SCM) solution for...
Read More
|
|
|
|
|
|
|
TrackBack URI for this entry
Subscribe to this comment's feed
