|
Makefile Optimization: $(shell) and := go Together |
|
|
Written by John Graham-Cumming
|
|
Monday, 08 November 2004 |
In this article I serve up a recipe for speeding up your Makefiles with the addition of a suitably placed colon. But to get there you need to understand GNU Make's $(shell) function and the difference between = and :=.
$(shell) explained
$(shell) is GNU Make's equivalent of the back tick (`) operator in the shell. $(shell) executes a command, flattens the result (i.e. turns all line endings into spaces) and returns the resulting string.
For example, if you want to get the output of the date command into a variable called NOW you'd write
NOW = $(shell date)
Or if you want to count the number of files in the current directory and get that number into FILE_COUNT do
FILE_COUNT = $(shell ls | wc -l )
Note that since $(shell) flattens output you can get the names of all the files in the current directory into a macro with:
FILES = $(shell ls)
The newline between files is replaced with a single space making FILES a space-separated list of filenames.
You'll commonly see in Makefiles an execution of the pwd command to get the current working directory into a variable (in this case CWD):
CWD = $(shell pwd)
It's this command we'll take a look at later on when considering how to optimize an example Makefile that wastes time getting the working directory over and over again.
The difference between = and :=
99% of the time you'll see macro definitions in Makefiles that use the = form:
FOO = foo
BAR = bar
FOOBAR = $(FOO) $(BAR)
all: $(FOOBAR)
$(FOOBAR):
@echo $@ $(FOOBAR)
FOO = fooey
BAR = barney
In the example above macros FOO, BAR and FOOBAR are “recursively expanded” macros. That means that when we ask for the value of a macro any macros that it references are expanded at that point. For example, if we ask for the value of $(FOOBAR) it will get the value of $(FOO) and $(BAR) put them together with the space in between and return foo bar. Expansion through as many levels of macros as necessary is done only when the variable is used.
In the Makefile above this has the interesting side effect that FOOBAR
appears to have two different values. Running it prints out:
foo fooey barney
bar fooey barney
The value of FOOBAR is used to define the list of prerequisites to the all rule and is expanded as foo bar, the same thing happens for the next rule which defines rules for foo and bar.
But when the rules are run, the value of FOOBAR as used in the echo comes out as fooey barney. (You can verify that the value of FOOBAR was foo bar when the rules were defined by looking at the value of $@---the target being built---when the rules are run).
There are two cases to remember:
- When a rule is being defined in a Makefile, macros will evaluate to their value at that point in the Makefile.
- Macros used in rule bodies (i.e. in the commands) have their final value: whatever value the macro had at the end of the Makefile.
If we change the definition of FOOBAR to use a := instead of = running the Makefile gives a very different result.
foo foo bar
bar foo bar
Now FOOBAR has the same value everywhere. That because := forces the right-hand side of the definition to be expanded at that moment. Rather than storing $(FOO) $(BAR) as the definition of FOOBAR, GNU Make stores the expansion of $(FOO) $(BAR) which at that point is foo bar. The fact that FOO and BAR are redefined later in the Makefile is irrelevant, FOOBAR has already been expanded and set to a fixed string. GNU Make refers to variables defined in this way as “simply expanded.”
Once a macro has become simply expanded it remains that way unless it is redefined using the = operator. This means that if you append to a simply expanded macro the text being appended is expanded before the append.
For example,
FOO=foo
BAR=bar
BAZ=baz
FOOBAR := $(FOO) $(BAR)
FOOBAR += $(BAZ)
BAZ=bazzy
results in FOOBAR being foo bar baz. If = had been used instead of := then when $(BAZ) was appended it would not have been expanded and the resulting FOOBAR would have been foo baz bazzy.
Take a look at this example Makefile.
CWD = $(shell pwd)
SRC_DIR=$(CWD)/src/
OBJ_DIR=$(CWD)/obj/
OBJS = $(OBJ_DIR)foo.o $(OBJ_DIR)bar.o $(OBJ_DIR)baz.o
$(OBJ_DIR)%.o: $(SRC_DIR)%.c ; @echo Make $@ from $<
all: $(OBJS)
@echo $? $(OBJS)
It gets the current working directory into CWD, defines a source and object directory as sub-directories of the CWD, defines a set of objects (foo.o, bar.o and baz.o) to be built in the OBJ_DIR, sets up a pattern rule showing how to build a .o from a .c and finally states that by default the Makefile should build all the objects and print out a list of those that were out of date ($? is the list of prerequisites of a rule that were out of date) and a full list of objects.
You might be surprised to learn that this Makefile ends up making eight shell invocations just to get the CWD value. Imagine how many times GNU Make would make costly calls to the shell in a real Makefile with hundreds or thousands of objects!
There are so many calls to $(shell) because the Makefile uses recursively expanded macros: i.e. macros whose value is determined when the macro is used and not at definition time. OBJS references OBJ_DIR three times which references CWD each time; every time OBJS is referenced there are three calls to $(shell pwd). Any other reference to SRC_DIR or OBJ_DIR (e.g. the pattern rule definition) results in another $(shell pwd).
But there's a quick fix for this. Just change the definition of CWD to simply expanded by inserting a : to turn = into :=. Since the working directory doesn't change during the Make we can safely get it once:
CWD := $(shell pwd)
Now there's a single call out to the shell to get the working directory. In a real Makefile this could be a large time saver.
Since it can be hard to follow through a Makefile to see everywhere a macro is used there's a simple trick that will cause Make to print out the exact line at which a macro is expanded. If we insert $(warning Call to shell) in the definition of CWD so that its definition is
CWD = $(warning Call to shell)$(shell pwd)
then we get the following output when we run Make:
Makefile:8: Call to shell
Makefile:8: Call to shell
Makefile:10: Call to shell
Makefile:10: Call to shell
Makefile:10: Call to shell
Make /somedir/obj/foo.o from /somedir/src/foo.c
Make /somedir/obj/bar.o from /somedir/src/bar.c
Make /somedir/obj/baz.o from /somedir/src/baz.c
Makefile:11: Call to shell
Makefile:11: Call to shell
Makefile:11: Call to shell
/somedir/obj/foo.o /somedir/obj/bar.o /somedir/obj/baz.o /somedir/obj/foo.o
/somedir/obj/bar.o /somedir/obj/baz.o
The $(warning) doesn't change the value of CWD, but it does output a message to STDERR. From the output you can see that eight calls to the shell and which lines in the Makefile caused them.
If CWD is defined using := the $(warning) trick verifies that CWD is expanded only once:
Makefile:1: Call to shell
Make /somedir/obj/foo.o from /somedir/src/foo.c
Make /somedir/obj/bar.o from /somedir/src/bar.c
Make /somedir/obj/baz.o from /somedir/src/baz.c
/somedir/obj/foo.o /somedir/obj/bar.o /somedir/obj/baz.o /somedir/obj/foo.o
/somedir/obj/bar.o /somedir/obj/baz.o
Quick Recipe
A quick way to find out if you are using the expensive combination of = and $(shell) in your Makefile is to run the command:
grep -n \$\(shell Makefile | grep -v :=
It'll print out the line number and details of every line in Makefile that contains a $(shell) and doesn't contain a :=. Take a look at what it finds to see if your Makefile can be optimized.
John Graham-Cumming is Founder and VP of Engineering at Electric Cloud, Inc. Prior to joining Electric Cloud, John was a Venture Consultant with Accel Partners, VP of Internet Technology at Interwoven, Inc. (IWOV), VP of Engineering at Scriptics Corporation (acquired by Interwoven), and Chief Architect at Optimal Networks, Inc.
John holds BA and MA degrees in Mathematics and Computation and a Doctorate in Computer Security from Oxford University. John is the creator of the highly acclaimed open source POPFile project. He also holds two patents in network analysis and has others pending.
Trackback(0)
 |
Articles 
TrackBack URI for this entry
Subscribe to this comment's feed
