| 
Home arrow Articles arrow Columns arrow Ask Mr. Make arrow Rebuilding When CPPFLAGS Changes
Rebuilding When CPPFLAGS Changes Print
Written by John Graham-Cumming   
Tuesday, 28 February 2006


This article shows how to implement an important "missing feature" of GNU Make: the ability to rebuild targets when the commands for those targets change. GNU Make rebuilds a target when it is "out of date;" that is, it rebuilds when some of the prerequisites are newer than the target itself. But what if the target appears up to date, under the definition, but the actual commands to build the target have changed.

For example, what happens if you do a non-debug build (by typing make) and then run a debug build by typing make DEBUG=1. Unless the build has been structured so that the names of targets are dependent on whether the build is debug or non-debug, nothing happens at all.

GNU Make has no way of detecting that some targets ought to be rebuilt, because it doesn't take into account changing the commands. If, for example DEBUG=1 causes the flags passed to the compiler to change then the target ought to be rebuilt.

This article shows how, in a few lines of GNU Make code, to make that happen.

An Example

Here's an example Makefile that's used throughout this article to demonstrate the rebuilding when commands change system. To make the operation of the system very clear I've avoided using built-in GNU Make rules so this Makefile isn't as simple as it could be.

The Makefile creates two .o files: foo.o and bar.o by compiling corresponding .c files. The compilation is done using the built-in variable COMPILE.C (which will be something like normally be the name of a suitable compiler for your system, references to variables like CPPFLAGS and use of $@ and $< to compile the right thing).

There's a specific reference to $(DEBUG), it's turned into a preprocessor variable called DEBUG using the compiler's -D option. I haven't bothered showing the contents of foo.c and bar.c as they are irrelevant.

all: foo.o bar.o

foo.o: foo.c
    $(COMPILE.C) -DDEBUG=$(DEBUG) -o $@ $<

bar.o: bar.c
    $(COMPILE.C) -o $@ $<

Here's what happens if we first run make (which means that DEBUG is undefined) followed by make DEBUG=1.

$ make
g++    -c -DDEBUG= -o foo.o foo.c
g++    -c -o bar.o bar.c

Now foo.o and bar.o have been created (with DEBUG empty) and so typing make again does nothing:

$ make
make: Nothing to be done for `all'.

Typing make DEBUG=1 also does nothing, even though the object file foo.o would likely be different if it were rebuilt with DEBUG defined.

$ make DEBUG=1
make: Nothing to be done for `all'.

The signature system described below will correct that, with very little work for the Makefile maintainer.

The Example Revisited

To fix the problem described above this article introduces a helper Makefile called signature. The contents and working of signature are described below, but first let's look at how the example Makefile is modified to used the signature helper.

include signature

all: foo.o bar.o

foo.o: foo.c
    $(call do,$$(COMPILE.C) -DDEBUG=$$(DEBUG) -o $$@ $$<)

bar.o: bar.c
    $(call do,$$(COMPILE.C) -o $$@ $$<)

-include foo.sig bar.sig

Three changes have been made to the file: firstly include signature has been added at the start so that the code that handles the signature updating is included. The commands in the two rules have been wrapped with $(call do,...) and the $ signs for each command have been quoted with a second $.

Lastly for each .o file being managed by signature there's an include of a corresponding .sig file. The final line of the Makefile includes foo.sig (for foo.o) and bar.sig (for bar.o). Notice that -include is used in case the .sig file is missing.

Before seeing how this works here are some example of it in operation. First run a clean build (i.e. with no .o files present) and then rerun make to see that there's nothing to do:

$ make
g++    -c -DDEBUG= -o foo.o foo.c
g++    -c -o bar.o bar.c
$ make
make: Nothing to be done for `all'.

But now setting DEBUG to 1 on the make command-line cause foo.o to rebuild because its 'signature' (i.e. the actual commands to be run to build foo.o have changed.

$ make DEBUG=1
g++    -c -DDEBUG=1 -o foo.o foo.c

Of course, bar.o was not rebuilt because it was truly up to date (it's object was new and there were no command changes). Running make DEBUG=1 again says that there's nothing to be done, but just typing make rebuilds foo.o again because DEBUG is now undefined.

$ make DEBUG=1
make: Nothing to be done for `all'.
$ make
g++    -c -DDEBUG= -o foo.o foo.c

The signature system also works for variables that are hidden within a recursive variable. In GNU Make COMPILE.C actual expands CPPFLAGS to create the complete compiler command-line. Here's what happens if we modify CPPFLAGS on the command-line by adding a definition:

$ make CPPFLAGS+=-DFOO=foo
g++ -DFOO=foo -c -DDEBUG= -o foo.o foo.c
g++ -DFOO=foo -c -o bar.o bar.c

Both foo.o and bar.o were rebuilt because CPPFLAGS had changed (and because CPPFLAGS was part of the commands used to build those two object files).

Of course, changing a variable that isn't referenced doesn't cause anything to be updated. Here's an example starting from a clean build and then redefining SOMEVAR.

$ make
g++    -c -DDEBUG= -o foo.o foo.c
g++    -c -o bar.o bar.c
$ make SOMEVAR=42
make: Nothing to be done for `all'.

How Signature Works

To understand how this works the first place to look is inside a .sig file. The .sig files are automatically generated by signature for each rule that uses the $(call do,...) (the details of how are later on).

Here, for example, is the contents of the foo.sig file after the first clean build was run:

$(eval @ := foo.o)
$(eval % := )
$(eval < := foo.c)
$(eval ? := foo.force)
$(eval ^ := foo.c foo.force)
$(eval + := foo.c foo.force)
$(eval * := foo)

foo.o: foo.force

$(if $(call sne,$(COMPILE.C) -DDEBUG=$(DEBUG) -o $@ $<,g++    -c -DDEBUG= -o foo.o foo.c),$(shell touch foo.force))

The first seven lines capture the state of the automatic variables as defined when the foo.o rule is being processed. These are needed so that the current commands for a rule can be compared with the commands the last time the rule was run.

Next comes the line foo.o: foo.force. This says that foo.o must be rebuilt is foo.force is newer. It's this line that causes foo.o to get rebuilt when the commands change, and it's the next line that touches foo.force if the commands have changed.

The long $(if ...) statement uses the GMSL (see http://gmsl.sf.net/) sne (string not equal) to compare the current commands for foo.o (by expanding them) against their value the last time they were expanded. If the commands have changed then $(shell touch foo.force) is called.

Since the .sig files are processed when the Makefile is being parsed (they are just Makefile's themselves read using include), all the .force files will have been updated before any rules run. And so this small .sig file does all the work of forcing an object file to rebuild when the commands change.

The .sig files themselves are created by signature:

include gmsl

last_target :=

dump_var = \$$(eval $1 := $($1))

define new_rule
@echo "$(call map,dump_var,@ % < ? ^ + *)" > $S
@$(if $(wildcard $F),,touch $F)
@echo $@: $F >> $S
endef

define do
$(eval S := $*.sig)$(eval F := $*.force)$(eval C := $1)
$(if $(call sne,$@,$(last_target)),$(call new_rule),$(eval last_target := $@))
@echo "$(subst $$,\$$,$$(if $$(call sne,$1,$C),$$(shell touch $F)))" >> $S
$C
endef

signature include the GNU Make Standard Library and then defines the important do macro used to wrap the commands in a rule. When do is called it creates the appropriate .sig file containing the state of all the automatic variables.

The capture of automatic variables is done by the new_rule macro called by do which uses the GMSL map function to call another macro (dump_var) for each of @ % < ? ^ + *. new_rule also ensures that the corresponding .force file has been created.

Lastly, do writes out the complex $(if ...) statement that contains both the unexpanded and expanded versions of the commands for the current rule. And then it actually runs the commands (that's the $C) at the end.

Limitations

There are a small number of limitations to this technique. Firstly if the commands in a rule contain any side effects (e.g. they call $(shell ...) then the system may misbehave if there was an assumption that the $(shell ...) was only called once.

Secondly, it's vital that signature is included before any of the .sig files.

And lastly, if the Makefile is edited and the commands in a rule change the signature system will not notice. If you do that then it's vital to regenerate the corresponding target so that the .sig is updated.

Conclusion

Not a lot of code, for a lot of benefit. Let me know if you use it in a large project!

Lastly, reader Semih Cemiloglu wrote to say that a while back I'd promised to write an article on tracing which rules in a Makefile fire and in what order. Although some of that is covered in recent articles on the GNU Make Debugger, I'll return in April with the missing piece! Thanks for the reminder Semih.


John Graham-Cumming is Founder and VP of Engineering at Electric Cloud, Inc. Prior to joining Electric Cloud, John was a Venture Consultant with Accel Partners, VP of Internet Technology at Interwoven, Inc. (IWOV), VP of Engineering at Scriptics Corporation (acquired by Interwoven), and Chief Architect at Optimal Networks, Inc.

John holds BA and MA degrees in Mathematics and Computation and a Doctorate in Computer Security from Oxford University. John is the creator of the highly acclaimed open source POPFile project. He also holds two patents in network analysis and has others pending.

Trackback(0)
Comments (7)add
...
written by John Graham-Cumming , January 08, 2007
There's a neater way than doing $(filter-out) if you are using a recent (3.80+) version of GNU Make: use an order-only prerequisite. If you change the line

@echo $@: $F >> $S

to

@echo '$@: | $F' >> $S

then the .force file does not appear in $? or $^.

John.

report abuse
vote down
vote up
Votes: +0
...
written by John Graham-Cumming , January 08, 2007
You are correct that changing $* to $@ is the right thing to do. To be honest, I'm not sure why I use $* in the code above it would be better if $@ is used. Looks like an error in the original article. Nice find!

Then signature would be:

include gmsl

last_target :=

dump_var = $$(eval $1 := $($1))

define new_rule
@echo "$(call map,dump_var,@ % < ? ^ + *)" > $S
@$(if $(wildcard $F),,touch $F)
@echo $@: $F >> $S
endef

define do
$(eval S := $@.sig)$(eval F := $@.force)$(eval C := $1)
$(if $(call sne,$@,$(last_target)),$(call new_rule),$(eval last_target := $@))
@echo "$(subst $$,$$,$$(if $$(call sne,$1,$C),$$(shell touch $F)))" >> $S
$C
endef

report abuse
vote down
vote up
Votes: +0
...
written by Bert Wesarg , January 08, 2007
Oh,

and for the link command, i need to $(filter-out) the .force prerequisite.

Bert
report abuse
vote down
vote up
Votes: +0
...
written by Bert Wesarg , January 08, 2007
> Why did you change from double quotes to single? Just curious.
Because my shell (bash) try to expand and execute the '$(eval ...)' strings, generated in the dump_var macro.

> You shouldn't need to change from $* to $@ for pattern rules since $@ is still
> valid.

I changed Your '$*' to '$@'. See first line fo do macro:
$(eval S := $*.sig)$(eval F := $*.force)$(eval C := $1)
Maybe you version listed in this document is older than your actual code.

Bert
report abuse
vote down
vote up
Votes: +0
...
written by John Graham-Cumming , January 08, 2007
If you have a rule with more than one line you just $(call do,...) for each line:

foo.o: foo.c
$(call do,compile it)
$(call do,other stuff)
$(call do,more stuff)

do is clever enough to know that you are still inside the same rule.

Why did you change from double quotes to single? Just curious.

It's true that in a pattern rule you could decide to use $* instead of $@ but be careful because you've lost the extension so potentially you could end up with the same .sig file being used for two different targets if they share the same stem.

You shouldn't need to change from $* to $@ for pattern rules since $@ is still valid.

John.

report abuse
vote down
vote up
Votes: +0
...
written by Bert Wesarg , January 04, 2007
Part 2:
%: %.o
$(call do,$(LINK.o) $^ $(LOADLIBES) $$(LDLIBS) -o $$@)

So, I get a foo.o.sig and a foo.sig.

One problem I didn't resolved is signatures in combination with automatic dependency generation. These pattern rules are two or more liners, so I don't know how to do this with the do function?

Sincerely
Bert Wesarg

report abuse
vote down
vote up
Votes: +0
...
written by Bert Wesarg , January 04, 2007
Thank you for the article,

i tried this by myself, but on variable level (CC is saved in .cc, and so on), but failed when i use target-specific variables. Your target level is obliviously the right thing.

But I had to make some changes to your code:
(1) I changed the double quotes for the @echo lines to single quotes
(2) I changed the $* to $@ in $(eval S := $*.sig)$(eval F := $*.force), with this its now possible to use the do function in pattern rules:

%.o: %.c
$(call do,$$(COMPILE.C) -o $$@ $$
report abuse
vote down
vote up
Votes: +0
Write comment
smaller | bigger

security image
Write the displayed characters


busy
 
< Prev   Next >
[ Back ]

Video News

Whitepaper Spotlight

Stay up to date with Configuration Management and Application Lifecycle Management technology products and services by browsing our featured white papers below: See all the Featured Whitepapers>>
Tool Spotlight
CollabNet
 CollabNet Subversion is an enterprise-ready distribution of Subversion® that includes, in one package,...
Read More
AccuRev
 AccuRev is a best-of-breed, process-centric software configuration management (SCM) solution for...
Read More
IBM Rational Build Forge Express Edition
 IBM Rational Build Forge Express Edition is a flexible and robust build automation framework developed,...
Read More
Sapient ResultSpace
 ResultSpace is the Agile Application Lifecycle Management (ALM) solution that enables software development...
Read More