In the beginning…

…was the definition.

In this article I am going to lay out my definitions for some terminology that will become increasingly important as I develop my CMS model.

The terms I will be discussing are as follows.

• Stream
• Branch
• Configuration Item
• Revision
• Configuration
• Component
• Repository
• Configuration Management Database
• Record

At this point I caution the reader that these definitions are deliberately quite loose and informal. Each will be expanded, refined, rewritten and formalised as I work through articles in this blog. For now, my working definitions are as follows.

Project

A coordinated effort usually conducted by several individuals to deliver a Product. Project describes the totality of planning and activity requires to gather requirements and interpret these into Product.

Product

That which is to be delivered by a Project. Products include, but are not limited to:

• Executable software
• Documents — manuals, design documents, requirements, installation guides, administration and maintenance manuals
• Hardware — computers, network components, any other physical components required as part of the Product
• Training materials — exercise version of data or system components, trainer presenation, training the trainer material, sandbox systems for trainees
• Source code — when developing for a 3rd party. Source code may also be a deliverable in interpreted languages or when delivering web content such as HTML.
• Media — video, graphics, audio

Stream

Projects often consist of more than one piece of development. A common strategy is to manage these pieces of development as a sort of sub-project. Timescales of these Streams are overlapped to allow the project to compress its overall timescale.

Branch

An implementation technique used in development to manage simultaneous changes to common items. In software development Branches are common and used to allow two or more developers to work on the same source code simultaneously.

Configuration Item

A configuration item is an item within the configuration management system that is the focus for change management.

Revision

Each time an item is modified and submitted into version control, a new revision is created. In this way an item’s history can be traced by looking back through the sequence of revisions.

Delta

The difference between two revisions.

Configuration

A specific arrangement of item revisions.

Component

An item that is subject to version control, but is not elevated to the status of a configuration item.

Repository

A safe store for item revisions.

Configuration Management Database

A database containing information about each item revision and their relationships to one another and to records.

Record

A set of data that is subject to a process or workflow but not necessarily version control. Records normally carry information used to account for an item’s current disposition or the current state of a process or workflow.

Filed under: Plain Old Blog

Fences and Ambulances

Suppose you are in charge of a cliff edge. Your task is to maintain the views from the cliff, but keep visitors safe. You can construct a fences along the top of the cliff, to stop people falling over, or you can place ambulances at the foot of the cliff, to clear up once someone falls over.

Fences seem most appealing (even if only for humanitarian reasons), but they are expensive to construct and maintain. Furthermore, constructing a high fence will block the beautiful views across the sea and people will stop visiting the site (which is bad). So your fences can be only so high before they become an impediment and stop people using the facilities. Unfortunately, keeping the fences too low means people can still climb over them and (you guessed is) fall to their doom.

Maintaining ambulances at the foot of the cliff is also expensive. There is the possibility that people falling over the cliff will be so harmed that they do not survive the fall and even those that do survive may be out of commission for some time. Ambulances seem like a last resort, a safety net. Too few ambulances means more injured fallers die when they could have been saved, too many ambulances means increased cost without any benefit.

There is a balance to be struck. Fences to guard against the unwary and ambulances to recover the unfortunate (or plain stupid).

This balance between ambulances and fences is the line that process engineers walk. We try to design fences that are cheap to install, efficient to maintain, stop the majority of people falling over the cliff, and allow people to enjoy the facilities along the cliff edge safely. At the same time we know that there will always be those who venture beyond the fences (they cannot be secured without blocking the view and driving people away — possibly to another cliff), so we prepare a small fleet of ambulances to minimise the impact (forgive the pun) of those who fall.

Creating a process that allows facilities to be used efficiently but also provide a means of identifying and safely recovering when things go wrong is the skill of a good process engineer.

A good example of this is the combination of change and incident management. Change management is a fence, a process designed to avoid problems. Incident management is an ambulance, a process designed to minimise problems once they occur. When change management fails because either someone circumvents the process (climbs the fence) or because the process is incorrectly applied (the fence is damaged or too low) then incident management identifies the resulting problem and hopefully resolves it as quickly as possible (ambulances recovering the injured faller and saving their life).

If change management processes are too draconian they will be expensive (delaying change and making your organisation unresponsive) or simply circumvented when users perceive them to be an unacceptable impediment. If incident management is under resourced they will be incapable of recovering from problems when they occur. As change management becomes less effective the cost of incident management rises, as change management becomes more effective the need for incident management declines.

Striking the right balance is a matter of judgement and will vary from organisation to organisation, and from service to service. The more sensitive your service is to problems, the more change control is called for, the less sensitive your service is to problems, the less change control is required.

A problem in, for example, an order processing system for a mail-order company has serious consequences and suggests more change control should be applied. The cost of failure is so high that incident management costs would be unacceptable. Within the same organisation the system that produces the print catalogue is less sensitive, so change management costs would normally be lower and more incident management might be called for.

The key to good process design and implementation is balance between constructing fences and maintaining ambulances.

Filed under: CMCrossroads, Plain Old Blog, Process, Tools 'n' Tips

CMS Tool: High-level architecture

Continuing my musings about a universal configuration management tool I’ve drafted the basic architecture. This is summarised in the following diagram (after the break).

This simple architecture serves to isolate the key levels of abstraction in the CMS.

As with all multi-layer architectures the important feature is that each layer creates an abstraction then allows layers it serves to use a stable interface. The CM Data layer, for example, will need to deal directly with individual the tools we wish to interrogate. (Recall that I am currently only proposing a tool that analyses and reports the content of the underlying tools, writing a tool to also update the content of those tools adds a whole new set of challenges to the problem. Challenges I have no desire to deal with right now, but which will be served well by this architecture nonetheless.) The CM Data layer presents one, hopefully, stable interface to the CM Model, so the CM Model will not have to be changed is a tool changes its interface (changing its database schema, command line, API, or whatever method we use to access it).

CM Data

This layer presents a common interface to the CM tools that the CMS is fronting. CM Data can be divided into two categories.

• Item library — this holds the items the CM tools store; files, directories, symbolic links, etc.
• Data about items — this is data such as relationships between items, attributes of items (owner, author, date modified, and so on).

The CM Data layer will provide methods for accessing these data. Items will be held in a Repository and the data about these items will be held in a CMDatabase.
All version control tools are a Repository (they store items) and provide at least some data about those items (author, modified date, ancestor/successor relationships, contains relationships), so they provide some functions of a CMDatabase. More advanced tools (like Dimensions and Synergy/CM) provide the same Repository functions but they expand these considerably (adding in records such as change requests) and the CMDatabase functionality is also expanded. These points will be revisited and clarified as we proceed.

CM Model

The CM Model will provide an abstraction of common CM artefacts (items and their relationships, collections, baselines, branches, changes and so on). The point of this abstraction is to, once again, provide a consistent and stable model to the layers that use the CM Model. So, the code in the Business Logic and Presentation layers is written to use the CM Model rather than the various implementations of these artefacts presented by the underlying tools. This means our system can access, say, an Item without needing to know how it is represented in the tool or even which tool it is held in.

The CM Model is perhaps the most important part of the architecture to design. It needs to be sufficiently comprehensive to accommodate the various tools’ concepts but at the same time it needs to be as simple as possible and as stable as possible to preserve a stable interface to the rest of the system.

Business Logic

The Business Logic layer is where all of the rules about how the CM Model is to be interpreted and managed. The most challenging aspect of the Business Layer is the need to allow users to define their own rules without requiring them to program anything into the system (although we will surely make the tool easy to extend programatically). This layer will require a detailed set of user cases and a considerable investment of time in the early design if the final product is to be flexible enough to accommodate a wide variety of users.

The Business Logic layer’s functionality can be divided into two classes; those features of SCM that are fundamental and those that are definable. Establishing fundamental features may be a matter of diktat on my part, but never-the-less I will try to only hardcode rule when absolutely necessary, preferring instead to provide configurable rules. *scratched head* This could take some time.

Presentation

Finally there is the presentation layer. Here all of the various methods of presenting the system’s functionality are gathered. This is perhaps the simplest layer to define functionally but the most difficult to design aesthetically. Having established what the system can do in the other layers the function of the Presentation layer is to provide a method of allowing the user to request that functionality, and providing a means for the system to provide the results to the user. The challenge with user interface (UI) design is how to achieve these two functions in an easy to use and, preferably, intuitive way.

In the first instance I shall probably provide a command line interface as a priority with some form of rudimentary GUI as a secondary development. Once these are done other interfaces may be considered.

Filed under: CMCrossroads, Plain Old Blog, SCM Tool, Software Configuration Management Tagged: CM Tools

Parallel development: theory and practice

Having spent the past couple of weeks with a client working through the issues that need to be carefully considered when version controlling software, and in particular how to manage and control parallel development. I have come to three conclusions:

1. People are often more afraid of the perceived problems than the practical realities of parallel development.
2. People do not truly appreciate the problems and practical realities of version control and parallel development.
3. There is a need for more theoretical work on the topic and, perhaps more significantly, a need for more formal expression of the process and problems involved.
4. There is a quite substantial book that could be written on the subject.

The first two points are related to the last two insofar as people need a clear explanation of the discipline and, to really acquire a firm understanding of the subject, you need to commit to studying it. There is simply no easy way to explain the more complex material (and by simple I mean one side of A4 summaries demanded by people who think they need to know but who have no practical exposure to the technical nature of software development or version control — there is no ‘managers guide too..’).

As to the third point, I have expressed a concern about this before in various forums. Rather than continue to observe the problem I have decided to start doing something about it. So, over the coming weeks and months I will be expanding some ideas I’ve been mulling over for many years in a series of posts to this blog (special category set up here) and on my website. Be warned though, these articles will probably get very theoretical, often be speculative, and may raise as many new questions as they resolve existing ones. We shall see.

As for the final point, as I build up a sufficiently cogent set of material while writing on the blog (a process unlikely to be well structured), I will assemble more coherent articles on my website and who knows I might get round to pulling them together into a book one day. (Ha! Another of the many books I either intend to write or am in the process of writing but may never finish or publish — *sigh* so many things to do, so little time.) I may even pull material together and add it to the CM wiki or the ITSLM BoK. You never know, stranger things have happened.

Rather than start here, I’ll start an introductory post right now and hopefully have it up in the next day or two. In the meantime, you could check out some past posts on the subject; here and here

Filed under: Configuration Management, ITSLM, Parallel Development: theory and practice, Plain Old Blog, Software Configuration Management, Version Control

Who’s afraid of the big bad merge?

A common objection to using parallel development is the fear of the inevitable merging required to reintegrate the changes as the development proceeds. In this post I will take a look at some of the issues that arise from managing parallel development and, perhaps more importantly, provide some guidance on how to avoid the pitfalls of parallel development.

Simple Merges

The most basic merge is a very straightforward (although, as we shall see, can have hidden complexities).

In this example changes to the file are merged between the two branches resulting in a new revision on the upper branch. Since the changes are to different parts of the file the merge can be performed automatically, resulting in a file containing both sets of changes.

The most obvious limitation of these simple merges is that the file must be ‘mergeable’, which for most tools means the file must be record oriented. Text files are a collection of records, one for each line in the file. Binary files tend to be resistant to merge unless a special tool is provided (MSWord can merge two documents with reasonable accuracy).

Conflicting Merges

Sometimes a merge involves changes to the same records (the same lines in a text file) in the same file. The illustration below shows just such a merge.

The most notable feature of this merge is the line marked ‘Developer resolves conflicts’. Conflicting merges require human intervention to decide which parts of the conflicting lines should be accepted into the final merged version of the file.

Sometimes these conflict resolutions will require more than a simple decision about which line from the two changes should be adopted and the developer will need to rework both changes to make them work together. Fortunately the majority of merge conflicts are relatively straightforward providing integrations are performed frequently.

Renamed and Moved Files

While merging a file’s content is commonly supported by version control tools, tracking renamed and moved files during a merge operation is less well supported but in todays software development world, where refactoring is common, it is increasingly essential.

The problem with renamed and moved files are summarised in the following illustration.

In the top branch file2 has been modified, but in the lower branch file2 has been both modified and renamed to file3. The desired result, shown in the final version to the right, contains the changes from both file2 in the top branch and the changes in file3 in the lower branch. But most importantly, the file has also been renamed to file3 in the upper branch.

Rename/move conflicts

Merges involving renames do not necessarily run smooth. Not only are they subject to the same content merge issues as the simple file merge but also the problem when the file is renamed/moved in both branches. This situation is illustrated below.

In this case the tool has chosen to include both renamed files. This may not be what was intended, so care must be taken when dealing with renamed and moved files. Other tools will display a conflict and require user intervention to decide the correct name to assign to the file in the merged dir1.

Mitigation

Before moving on to some other issues you will encounter when using parallel development, we pause briefly to consider how to avoid, or at least mitigate, some of the issues mentioned so far.

Firstly, despite the impression the previous sections may have conjured, merging is not fraught with perils. After many years of managing parallel development on projects ranging from the simple to very complex, I can assure you that, with a little common sense and some forethought the issues can be minimised (but seldom completely eliminated).

1. Merge as early as you can, and then merge often to keep each merge as small and manageable as possible.
2. When you have input to a project’s design, ensure a sound architecture. This will minimise overlapping changes and consequently merge conflicts.
3. If you identify files that are merged often, try to decompose the file into smaller files to avoid high merge rates. (This is common with files used for configuration or files containing a lot of utility code.)
4. Keep refactoring on one branch. Renaming and moving files can cause tricky merge issues when performed on many branches.

Now, on to some issues that, while not unique to parallel development, can make your life difficult if not managed properly.

Filed under: CMCrossroads, Plain Old Blog, Software Configuration Management, Version Control Tagged: branching, dependencies, merging, parallel development