Sponsors

Microsoft


TechWell

We have 1702 guests and 1 member online

Home CM: The Next Generation CM: THE NEXT GENERATION of Branching Standards

CM: THE NEXT GENERATION of Branching Standards

E-mail
Written by Joe Farah   
Monday, 21 September 2009 13:32
BranchWhen it comes to CM plans, each project has to clearly identify its own strategies for branching, and subsequently for labeling and merging.  There are many different practices for branching, and many different philosophies.  Each CM group will vigorously defend it's practices and document them in a "Branching Strategy" document for use by the development team.


In this article we explore a potential standard for next generation branching, one which requires advanced capabilites and processes, but one which can significantly reduce CM complexity and increase CM automation. As well as using it myself, and with a number of clients, I've seen many other companies migrate in the direction of Stream-Based Branching, in part or in whole, with a goal toward clearer, less complex, more automated CM.

Why Branch?

Branching should always arise from one common theme:  the need to support the existing codeline.  From the ground level, this is not always an easy call.  For example, if "they" send a build to the verification team, and I'm starting on a new feature, do I have to create a branch so that I can continue to support the code that went ot the verification team?  What if I'm changing an API - do I need to create a new branch so that it doesn't affect other existing code?  You may have very definite answers to these questions, but the simple answer to both of them is: It depends.  It depends on whether or not you have to support the previous code line.  Wait a minute... didn't I cirularize my argument?

OK.  Let's take an example.  Let's say I change an API.  Well, if it's not yet being used by anyone, no problem.  If it is being used, but the changes are all upward-compatible, not requiring other code to change, again no problem.  If the changes do require other code to change, but I package the changes to all the other code as part of the same change package, again no problem.  If I'm just making a small innocent change which is totally upward compatible, requiring no other code changes, but the release baseline is totally frozen in preparation for launch day - then, no way!  We don't want any changes that could possibly affect the release schedule - and simple software changes that go wrong have a certain track record.  So, it depends.

Branching is always a gut-wrenching issue.  If I branch, I have a new branch to support, to label and, possibly, to merge.  If problems come up that need to be addressed while the branch is in effect, I have another place where it has to be fixed.  How do I decide when the branch will merge back?  What happens if I determine a new branch has to be created off the old branch for another reason?  How will I eventually reconcile them?  What about a branch off the new branch?  Now let's look at the new release development - is the same thing happening there?  I'm responsible for 100 files - how do I keep my sanity through all of this? Or what if I change jobs - how does the next person deal with it all?  Well this is why we have Branching Strategies.

But a Branching Strategy is far from a Branching Standard.  Give me two different developers, or a developer and a CM manager, and I'll show you two different points of view on branching.  Why are there so many different branching practices?  Is it possible to focus on one or two practices that will do the job in a large number of cases?

Let's start out by looking at branching from above the forest.  It could be very simple:  Branch into a new release for changes that are not to be included in prior releases, and otherwise don't branch.  That covers the parallel release management view.  But what about all of the other reasons we branch?

I've said it before, but we really need to look at it closely this time:  The use of the branching capability is terribly overloaded.  And it is terribly overloaded because our processes and tools don't address the real set of CM problems head on.  We're going to explore this in more detail, but first, let's do some "dreaming".

A Simple Stream Branching Recommendation
Let's say branching were done only to support parallel release management - at most one branch on each file per release. Simplistic?  Perhaps, but I don't think so.  In fact, for the last 30+ years I've said "I don't think so".  On 2 person projects, 20 person projects, 200 person projects, 2000 person projects - this is a concept that needs to be embraced.  You might not be convinced - so tell me the problems with it after you've read the rest of this article.  I'm sure we'll generate some intense discussions.

To start, assume branching were done only to support parallel release management.  Well then, here are the recommended rules for the branching strategy from this perspective.
  1. Products, Directories, Files may all have branches
  2. Product branching follows the product road map of releases (i.e. product branches, a.k.a. release Streams)
  3. Everything else follows a subset of the Product branching layout.
  4. Branching is never done to support a variant.
  5. Change Requests (ECRs, Problem Reports, Tasks/Activities, Requirements, etc.) are targeted to a specific release Stream.
  6. A change package, or Update, as we will refer to it for clarity, is authorized by the approval of a change request.
  7. Each Update is targeted to a specific product release, i.e. the Stream of the Update.
  8. Files are checked out against the Update which collects all of the changed/changing files together.
  9. When a file is checked out, if there is no file branch for the the Update Stream, one is created (and labeled with the Stream)
  10. If there is a file branch for the Update Stream, a new revision is created along that branch.
So, whether or not you agree that this is a feasible approach, we now have a branching process that has several very interesting properties not always seen in a project.
  • The branching strategy is easy to learn. In fact, the CM tool set can automate it.
  • There is a simple naming/labeling convention for any branch: it's Stream name.  No manual labeling should be needed.
  • All files have the same branching layout - which is a subset of the product road map of releases
  • Technology transfer of the project does not involve a long analysis of the state of current branches and resolution of them.
Context
Now let's go one step further.  If I want to work on a particular release of a given product, all I should need to specify is the Product and the Stream (and perhaps a promotion level, but we'll get to that later).  The CM tool should be able to infer everything I need to know from that simple context specification:  Product + Stream [+Promotion Level].  

If I need to know what's on my To-Do list, the tool should present me with the list of items based on my context.  If I need to check out a file for which there is no branch in the context Stream, the tool should be able to look at the product road map and determine which stream to pull (and in this case branch) the source code from.  If I'm just retrieving (as opposed to "checking out") a file, the same rules apply.  The CM tool knows what the branches of each file mean because it has access to the product road map.  I don't have to write a context view specification or deal with explicit file revision numbers.

What if I need to merge a change from one stream to another (i.e. change propagation)?  Well, again, the CM tool knows the product history, and the set of files involved and their histories.  It should be able to calculate which files of the change need merging, what the ancestors are, and walk me through - in case there are any conflicts - bringing me to a point for re-testing.

I don't have to worry about switching the main trunk from one release to another, because each release has a main trunk which continues forever.  I don't need to set the timing, or make sure everyone has their merges done for the switchover.  I don't have to worry about what the process is for a change that is needed in a release before or after it occupies the main trunk - it's the same for all releases for all time.

All I Really Need
So what do I need to know how to do in this scenario?  As a developer, I [b]need to know[/b] how to:
  1. Specify my context (although hopefully it's remembered until I next need to change it).
  2. Find my to-do list and start my changes (i.e. create my Updates) based on an item in it.  The CM tool should automatically restrict my to-do list to my context, and tag the Update with my context, and reason for change.
  3. Go to the source tree and check-out a file, or perhaps inspect it to see if I need to check it out.  But the CM tool should branch it for me (possibly with a confirmation option) if necessary, and it should attach it to my Update.
  4. Review (i.e. via delta report) and check-in my Updates
  5. Promote my Update (i.e. after it's checked in tell the build team when it's OK for them to take it)
  6. Occasionally select an Update from a different release Stream and propagate it to my current stream.
and I don't need to know:
  1. What the branching strategy is for the project and how to decide whether or not to branch a file
  2. How to branch and how to label a branch
  3. How to merge branches, and what to do with them after they are merged
  4. How to write a context view specification so that I pick up the right file versions
  5. How to link my reasons for a change to the modified file revisions
Hopefully you'll conclude that the "need to" items are a lot easier than the "don't need to" items. (Perhaps I've left out a few from each list.)

Back to Reality

This is all very well and good, but if the branching model is too simplistic, how can I do all of the things I really need to do?  Are we not simply going to suffer elsewhere?  The answer is: Yes.  You are going to suffer tremendously - if you don't have the process and tools in place to help you realize this model. Otherwise: No.  So let's look at this in more detail.

Why do you create branches today? Before tossing all of this out as too simplistic, let's do the analysis.  Why do you branch?  Well first off, to support parallel releases - we've already covered that one and that's a good reason.  But there are plenty of other reasons as well:
  1. To Support Parallel Releases
  2. Because the main trunk is the wrong release
  3. To have a place to promote file revisions through the workflow: Ready for build, Integrated, Tested, Production
  4. So that files can be checked in to the repository without breaking the build
  5. To support parallel checkouts of the same file so that work isn't held up
  6. So that files for a particular Update can have their branches commonly labeled to collect them together
  7. To collect and label items for releases, baselines, builds or other such purposes.
  8. To identify, easily, the work done by a particular developer
  9. To allow back ups of my local work before it's ready to be "checked in" to the main branch
  10. To support product variants
We need to do all these things - or at least most of them: work on a different release than the "current" main trunk, have promotion levels, perform parallel checkouts, collect files into changes, releases, builds, baselines, etc., have our local work backed up, prevent pre-mature changes from breaking the build, identify our work, and so forth.  These are all things we need to do.

Branching is a very powerful concept, and a capability provided by virtually all CM tools.  In fact, if branching isn't supported, it's not really a CM tool.  The question is:  Why use branching to do all these things we need to do?  The simple answer is that branching is powerful and, along with labeling, it lets me do all these things.  The real answer is that your current processes and tools are insufficient to let you do these things without branching.  Without branching and labeling, they just don't let you do what you need to do.

I'm all in favour of branching to support parallel releases.  But, quite frankly, in over a quarter century of supporting dozens of companies in their CM practices, I haven't really seen another need for branching.  What I've seen is the need to do all these things - but not by branching.  Instead the use of good process and good tools will suffice while reducing complexity dramatically.

To Branch or Not To Branch
Before going on, I want to refer you back to our eutopia, that is, our recommendation - where we didn't need all sorts of  branching and the CM tools could help automate and simplify life for the developer, not to mention the CM manager and others.  Oh, and let's not forget the technology transfer nightmare that can be avoided by not having 10,000 files, all with different spaghetti branching structures.  No fights about who's doing the merging or when the main trunk is going to switch releases.  This is just the motivation paragraph - to get you to read on.  If you've seen other motivating factors, great - tell us about them, but let's read on.

We're going to evaluate each of these items and determine how branching is, in most cases, easily avoided.  The goal here is not to do a lot of extra work and such to eliminate the need to branch.  The goal instead is to eliminate all the work that is necessary because of branching, while just doing what has to be done anyway.  If you want your CM to move fully into the next generation, this is what you'll have to wrestle with.

So let's look at each branching requirement.

1.  Support of Parallel Releases
This is a real and valid use of branching.  Just make sure the branches follow the product road map (which may change from time to time).  Better yet, let's get the CM tools which make sure they do.

2.  Main Trunk is the Wrong Release
If you're working on a change that's not for the "main" trunk, the tendency is to create a branch for that change that can be merged into the main trunk when it switches to the right release (at least when it's future work).  Instead, since we have a branch per stream capability, you simply make the change in the Stream for the release on which you're working.  In fact, you no longer need a "main" trunk.  Instead, you set your context to the Stream you're working on, whether it's a support stream, future development or the current development stream.

3.  Work Flow Promotion: Ready for build, Integrated, Tested, Production
Most shops have some sort of work flow promotion for their Updates (or in many cases Files).  It may be as simple as: In Development, Integration, Production.  Or it may be more elaborate.

First of all, work flow promotion belongs on the Update (i.e. the change), not on a file.  To do otherwise is to risk getting partial changes promoted.  CM tools or processes that do not support some form of change packaging are going to cause problems.

The next thing to notice is that many processes are geared to allow any Update to be promoted in any order.  This allows maximum flexibility.  Well, not quite.  This pessimistic view (i.e. that any Update can be promoted in any order relative to other Updates) requires a process that has separate branches for each promotion level and requires merging at each promotion step.  Who is responsible for the merging?  When does it occur? What about the separate post-merge testing?  What happens if the process needs a new promotion level inserted?

In the optimistic view we realize that Updates are normally (in fact almost always), checked in to the repository in the same order that they will be promoted.  We're not talking about a strict Update-by-Update ordering here, but rather an ad-hoc grouping of Updates promoted in the same order that the group was checked in, as promotion beyond the developer tends to occur in batches.  For example, each night, all "checked in" Updates might be selected for the Build. At first glance this appears to reduce the flexibility - but let's look at this in more detail.  This view of the world actually works very well, in both small and large projects, but requires some good software engineering discipline.  

If developers are arbitrarily making changes and checking them in without testing, there's going to be a lot of failures and the resulting Updates are certainly not always going to flow through the system in the same order.  This model is simply not going to be valid.  However, if code is unit tested and peer reviewed before being checked in to the repository, odds are that your processes are geared to the expectation that checked in code will be promoted.  

Typically, checked in code is rolled into a Build and only if there are build problems are some of the Updates rolled back, corrected, or supplemented with a fix (i.e. another Update).  So how often do you roll back Updates as a percentage of all Updates selected for your builds?  Likely this is a very low percentage.  So instead of managing parallel branches for your promotion levels, you could be dealing with this very low percentage (which you have to deal with anyway) as a higher priority item.  If this is the case, then your promotion levels, instead of being separate branches requiring merges and additional administration, could simply be configuration lines drawn at various promotion levels in your Stream branch.

Ideally, with adequate tools, these lines are drawn automatically, based on the context view.  This is where Promotion Level (mentioned earlier) comes into play.  You specify the promotion level you wish as part of your context (along with Product and Stream), and your view now reflects all revisions that are (i.e. whose Updates are) at a status of that promotion level or higher, instead of always getting the latest.  Because your tools are aware of this promotion level capability, they can be adjusted to automatically provide such views, based on the Update status, without the need for writing configuration specs.  This is not easy to do with branch promotion management because branches are used for a wide variety of purposes, many of which are blurred.  

In a "main trunk" view, the promotion levels apply to revisions across the main trunk, and there are some complexities here.  But in a trunk per release (i.e. Stream) organization, promotion levels apply to the release trunk (i.e. Stream branch) for each file.

One of the key advantages of this approach, if your tools support the promotion level views, is that its each to roll back an Update - just roll back its state.  The view, that is, the promotion level configuration line, should automatically adjust itself, as long as this is an automated capability in your tool.  Now it's not quite that easy.  When you roll back an Update, you must also inspect dependents that are affected.  They too will have to be rolled back - but hopefully your tool automatically makes you aware of such dependency violations whenever you roll back (or promote!) the status of an Update.

One other thing to keep in mind is that while it may always (or usually) be fine to select all checked in code for a build, as you approach the end of release development, you do want to carefully control what code can be checked in, or more precisely, what work is authorized for creating an Update in the release Stream. Presumably, if you can't create an Update in a given Stream, you can't check out files to make the changes.

4.  Files can be checked in to the repository without breaking the build
One of the by-products of branchless promotion levels, is that you can easily insert new promotion levels.  There's no need for a new set of promotion branches, just an understanding of the promotion level, and perhaps a bit of data massaging.

One such promotion level distinction, not found in many projects, is the distinction between a checked-in file and one that is ready for the next build. For example, two Updates may be co-dependent and completed by different developers.  The first may finish his/her work early while the second requires a few more days.  In some shops, the first developer simply keeps the files for the Update in the workspace until the second update is ready.  This causes the need for more parallel checkouts because the files are sitting in the workspace for an extended period of time.  In other shops the files are checked in to a new branch, with the Update identifier as a tag, while awaiting the work of the other developer.  In this scenario, at least the files are visible to other users, although branching, and subsequent merging, is required, even though it may be trivial.

Introducing a "ready" promotion level in-between the "checked in" and "in the build" states (a promotion level is really just a state of the object), allows the code to be checked in without having to create a separate branch and without the danger that it will get pulled into the build until the "ready" status is achieved.  In the mean time, it may be checked out for additional changes to be made by the same or a different developer, without the instance of forcing a parallel Update which would later have to be merged, as the new Update can be performed on top of the first developer's changes.

5.  To support parallel checkouts of the same file so that work isn't held up
We've just seen one example of how branching for a parallel Update can often be avoided.  More commonly, parallel changes overlap more than just at the "waiting" stage.  Some shops disallow parallel Updates, as a rule, to eliminate the extra complexities in process and testing.  They enforce exclusive checkouts.  While this is an admirable goal to simplify process, it can cause significant delays.  An approach such as the "ready" promotion level can help to reduce delays.  

An additional approach, whereby an Update is broken into multple less complex Updates, all tied to the same Change Request, can further reduce the need for parallel Updates or exclusive delays.  For example, it is common that a few "header" files which define message codes, run-time id's or other frequently expanding definitions, have significant checkout contention.  When the checkin of the addition of one message code has to await the completion of the related feature, there's a significant chance of change overlap.  By breaking the change implementation into multiple Updates, with the first Update just adding the message code, and perhaps a stub function to handle it, the chance of overlap can be significantly reduced.

Even after all of these process improvements, there may still be the need for somewhat frequent parallel Updates (or delays if we take the exclusive checkout approach).  So this is when you need to branch to support parallel check outs, right?  Not quite.  Your Update has already announced your checkout to the CM repository.  For the vast majority of such changes, there's really no need to create a parallel branch as well.  After all, your workspace holds the file until you're ready to check in your Update.  If there are multiple users with Updates in parallel, this does not signal the need for parallel branches.  

In fact, some tools and processes which otherwise require parallel branches provide a means of subsequently removing or hiding such branches because they really add nothing to the CM history, and only serve as placeholders because, either the tools don't support some form of change packaging, or because even though they do, they don't allow parallel checkouts without branching.

I'm all in favor of minimizing the need for parallel checkouts.  I'm even in favor of those companies which, having minimized the need, find it acceptable to use only exclusive checkouts.  Often that strategy will even improve software design, forcing the design to minimize contention.  But if you are going to do parallel checkouts, other that for parallel release changes, I would be very reluctant to recommend branching to accomplish it, even to the point of suggesting an exclusive checkout solution.

6.  Files of a particular Update can have their branches commonly labeled to collect them together

For most CM tools, either they don't really support change packaging or such a concept was added on late in the game.  Some of these tools and processes require, if you want to collect files together into an atomic Update, that you simulate the change packaging by creating a branch for each file of the Update and labeling it with the Change Request or, even better, the Update identifier.  The branching is done specifically to support the labeling.  The proper implementation of change packaging using first order Update objects, makes this use of branching redundant.  Updates are not tags on a file.  Nor are they references to a task or tag.

In fact, the Update should be the central object in your CM repository, serving as the key component in the traceability hub.  The update relates your marching orders to the files that have changed.  Their states are used to identify configuration views, and as we'll see, to create configuration line ups.

7. Collecting and Labeling File Revisions for releases, baselines, builds or other such purposes.
Many projects which lack the tools for tracking Releases, Baseline and Builds as separate first order objects, use branching for such definition tracking.  Realizing that, for example, a Release, no sooner than it is out the door, will require some fixes, they use branching both to track the definition and to provide a means for support.  They then extend this concept to significant Builds, and possibly to Baseline definitions in general.

If, instead, your tools support the definition and tracking of Releases, Baselines and Builds as first order objects, the promotion level management capabilities can be used to create Release and other Baseline-related configuration lineups without the need for separate branches.

Think about for a minute.  A Release does not create any new revisions of any files.  It simply identifies the revisions of files that have already been created and tested along the promotion path.  Similarly any build or baseline definition is simply selecting pre-existing revisions.  Some tools and processes are smart enough to recognize this and do not force the creation of separate branches, perhaps using labeling instead.  Some technology is not good at managing labels, because it's too difficult or because of labeling limits or because of performance issues. Creating separate branches in such cases can "simplify" while creating complexity.

There should be first order objects for managing such items.  A Build for example, has to pass through several levels of promotion - integration testing, verification, field trials, and maybe all the way to production. Different builds will end up at different promotion levels.  This information needs to be tracked with the Build, including when the build reached each state and what the ultimate plans are for the Build - a nightly compile for development environment or a production build for delivery.

Trying to simulate a first order object with branches and labels is adding complexity to your environment where none should be needed.  Your users want to look at and use the Builds, not the set of files with a given tag at a given point in time.  And forcing the branching of all of the files in the system just to track these things is really overkill, no matter how proficient the branching mechanisms are.

8.  To identify, easily, the work done by a particular developer
Maybe you're stuck at this one.  Well I find this one hard to believeI.  But I have seen shops where they force branching down to a level where individual developers can be identified.  This way they can trace who did the code changes from the CM meta data, without having to go into the delta-compressed file meta-data.  I do find it useful to be able to identify work by user - but not by branching.  The Update identifier or related object data should clearly identify the developer.  There should be no need to branch in order to track this information.

This is an example of how some shops expand the use of branching to implement features.  This is dangerous.  Leave branching alone.  It has a purpose.  If you need additional data, add a database to your tool or switch to one that can provide the data mechanisms out of the box.

9.  Back-ups of local work before it's ready to be "checked in" to the main branch
Now here's a valid requirement.  My work is on my workspace, on my laptop.  I'm making changes daily and it's not getting backed up, or if it is, it's distributed over 300 other desktop and laptop backups.  It'll take hours, if not days, to identify where the backups are and to recover it if necessary.  So the solution is to put it into the CM repository.  Then it gets backed up in a common place whenever I want.

But if I want to back up my work nightly, I'll have to put it into the repository nightly.  That means, in some tools, I'll have to check it in nightly.  That means lots of revisions being created.  Of course I'd want them in a side branch and not in the main branch of each file.  So this must be a valid use of branching.  But not so, even if your tools and processes don't allow you to do nightly backups otherwise.  Many CM tools allow you to "shelve" your work so that you save the current contents without creating new revisions of the file, or perhaps creating new revisions that are not visible.  The latter case may cause some performance issues though.  But either way, your CM tools and related processes should allow you to save your Update-in-progress without having to clutter up the CM information, and especially without having to create branches, even if they're "temporary"

10. Variant Handling
So many projects start out as a one of, and then split off into dozens, if not hundreds, of variants of a product.  Managing large numbers of baselines, builds, etc. is not only tedious and resource intensive, but it's risky.  Sooner or later you're going to reach a point where you can't cope.  I've seen it more than often enough.  The "solution" taken is typically branching and labeling and still the problem persists, but now as organized chaos.

Variant management is primarily a software engineering problem, not a configuration management problem. Your team must specify Design Criteria which will make CM manageable.  CM should not attempt to solve software engineering problems which are rooted in the design - though it may help you to identify them - and variant handling is one area where software engineering needs to be addressed by the design.

It is not difficult to avoid dozens of variants.  In fact, it's much, much easier to reduce variant programs into run-time options and configurations than not to do so.  Introduce a simple design rule that a variant will not be a build time variant, or a load time variant, if it can be a run-time variant.  Whether it's a sizing option, a language option or a set of feature configurations - you can create separate images that support each of these.  But you can just as easily add a requirement to the design team that says, the size will be configured at run-time by setting a Size parameter, language through a run-time Language setting, etc.  The main valid reason for variants to be handled in the CM process is different platform support - and this reqiures separate executables for each platform - but should never require branching.

In extreme cases you may be space constrained and can't load all of the 100s of options into one image.  In this case, options should be handled as separate files which can be co-compiled (i.e. have unique names and/or name spaces), and the problem is brought forward to link time, or perhaps even load time.  Files to be linked or loaded are tagged with the feature options so that a build specification need only identify the feature options.  Still, just the process of moving these options forward to link time will result in multiple deliverables that have to be managed, and multiple test beds that have to be reloaded.  So it should be avoided whenever possible.  In any case, the problem is not one which requires the use of branching to solve.  The use of branching for variants comes from the reuse of the same file name for each of the different variants, rather than the use of separate file names to support each variant.  Common code should reside in the common file, while variant dependent code should reside in a variant specific file (i.e.with a file name specific to the variant).

How hard is it to get the design to follow this rule.  Not difficult at all.  The most difficult step, by far, is communicating the rule, and the need for it, to all developers, and then ensuring peer reviews identify violations.  Apart from that, it is a trivial design change, and one that yields way more benefits than just avoiding variant branches.

A Next Generation Branching Standard?

I apologize for the length of this article.  If I had identified additional reasons for branching, you'd see an even longer article.  But it was necessary to justify each of the cases, and if you send me some others, I'll deal with them as well.

OK, you might not understand everything here.  Or maybe you do but you don't agree with it all (please send your comments).  But do you at least agree with the overall philosophy that branching is overloaded?  If you do, then I really, really, recommend you go back to the "motivation" paragraph.  We're not talking a 5% or 10% reduction in complexity here.  No it's more like the difference between Assembly language code and Python scripts.  When branching is simplified, automation increases.

Am I recommending Stream-Based Branching as a CM Best Practice?   Not completely.  Not at this point in time.  It is one way to bring sanity to branching but it won't work for all projects for various reasons.  I know of several companies which have used or tended to this future "standard" for their branching - often with their own home grown tools or scripting.  But I also know of projects in some of these same companies that have stuck with various other branching strategies and patterns.

So why a standard as opposed to a branching strategy or pattern?  Well, a strategy is a good, perhaps rigid, guideline, and possibly even more beneficial to an organization than a standard.  However, in order for the industry to embrace it, a standard is required.  When a standard is embraced industry-wide, in whole or in part, the industry tools can anticipate behavior.  So, for example, if you have to move from one tool to another, your existing branching strategy may not map onto the new tool.  This in turn can turn what might be a few days work into a few months of work.  If the branching strategy is supported across tools, you will not have to limit your choices.  The industry CM tools can also help to automate CM more effectively when specific standards are in place.  It is not a downside that not all CM projects will embrace the standard.  Not everyone uses Bluetooth.  Not everyone uses ISDN.  But for those who need those communication protocols, having them as standards makes all of the difference.

Don't just make a switch.  Look at the various branching reasons listed here - add your own.  I'd really appreciate it if you added your own as comments or references to this article - it would help everyone.  Take incremental steps, reducing branching complexity until you're more comfortable with a full cutover.  You won't realize most of the benefits until you reach a state of Stream-Based Branching, and even then, it depends on the tools you're using to support such a capability.

But from decades of experience with this form of branching, I'm convinced that the next two generations of CM tools will see a vigorous level of support in this area.  Many corporate processes, and some CM tool vendor guidelines, already embrace it in one form or another.   And the benefits are just too great for it not to emerge as a branching standard in the Next Generation of CM technology.



Joe Farah
is the President and CEO of Neuma Technology and is a regular contributor to the CM Journal. Prior to co-founding Neuma in 1990 and directing the development of CM+, Joe was Director of Software Architecture and Technology at Mitel, and in the 1970s a Development Manager at Nortel (Bell-Northern Research) where he developed the Program Library System (PLS) still heavily in use by Nortel's largest projects. A software developer since the late 1960s, Joe holds a B.A.Sc. degree in Engineering Science from the University of Toronto. You can contact Joe at farah@neuma.com

Trackback(0)

Comments (3)add comment

Joe Farah said:

Joe Farah
...
(continued)
5) If your feature is really itself a major sub-release effort, consider creating a "feature" release stream branch for it. This should not normally be the case, but will almost certainly happen occasionally in very large projects.

Regarding the second comment, this is an important point - thanks for bringing it up. Depending on how your development and release cycles run, the way you track bugs, in particular, across multiple supported release streams is important. There are a few different variations on how to do this and these will depend on your change volume and release cycle:

1) Clone the problem separately for each release it applies to. This is not, in general, a preferred solution, but works very well when the quality of each release is very high and/or the chances of having to re-release it are low and/or if there is very strict conditions on which a bug will be fixed in an older release. Cloning the problem would require cross-referencing the clones. But again, this solution is not preferred if you find the need to do a lot of cloning.

2) Track the set of releases applicable against the problem and ensure there is a log entry whenever one of these releases has been either ruled out or addressed. You will also need a set of states (indexed by release stream) in this case. As this is a complex type of solution, unless your CM tool caters to it (and I did create one in the '70s that did a good job using this strategy), you may want to consider options 1 and 3 instead.

3) For each bug, have a sub-record for each potentially applicable release. The sub-record should include things such as the status, date tracking, and either update/change traceability, or else have it's own unique key (e.g. p.1234-rel3) that can be referenced from the update that addresses the problem in that stream. The overall problem record remains open until all streams have been either addressed or ruled out. (Older stream might be ruled out because you no longer support them.) The state against the sub-record is the one that is most pertinent to Status Accounting and common metrics.

The US Navy likes to use this latter means of tracking bugs. It is not vitally important to choose one of these methods over another, but it is important to ensure you have adequate tracking in place. Some tools will support one model only, or one better than another. Some (e.g. CM+) can be configured to support any of the models.
 
December 18, 2009
Votes: +0

Joe Farah said:

Joe Farah
...
Thanks for your comments Leslie and please accept my apologies for not responding to them earlier.

Although I agree that there are valid cases for feature-based branching within a release stream branching context, I would hesitate to use this branch pattern unless really necessary. Here's why:

1) The pattern has the ability to disrupt the stream branching model significantly.
2) The feature should be branched into the release it is planned for. If this causes disruption, consider using a parallel checkout (i.e. without a branch) instead of a separate branch, or use the techniques below.
3) Consider breaking the feature implementation up into multiple "changes" or "updates" so that the more contentious files can be addressed in advance or at least as part of shorter term updates. In general, it's a good idea to get the feature skeleton in place early on (even and sometimes especially in the previous release to simplify upgrades and enhance stability) in the feature development. The core implementation of complex features can then, normally, be implemented independent of the rest of the code base, in follow-on updates. This makes integration easier to handle and also provides a system-wide development environment for implementing the feature, especially if implementation involves many users. It also improves integrity of the released feature as it has been integrated over a period of time, allowing more soaking, avoiding a big-bang integration problem.

4) Reduce file contention. File contention is caused by three major design issues:
(a) Lots of requests impact that file because of the area of fix/feature - this can be partially addressed by looking at the impact of all of the requests on the file and planning fewer changes which cover these requests. Architecturally this makes for better code.
(b) The file is very big causing a large number of requests to impact it - Consider splitting the file into smaller component files prior to your main feature work.
(c) The file contains a list of codes/range elements/ids/etc. that are frequently being modified (especially added to). - Break out this part of the change into a separate pre-feature change and make your id mods to it, implementing stubs if necessary to support such changes. Even better, use a more object-oriented approach whereby the code requiring the id/element/code (e.g. message code) binds in the code as part of the implementation, rather than forcing a pre-compile-time binding. (This has several other architectural benefits as well.)
to be continued
 
December 18, 2009
Votes: +0

Leslie Giles said:

Leslie Giles
...
One other use for branches that you don't mention is feature-based branching, where a significant development effort would be put onto its own branch so it can be developed in parallel with other features. This has the added advantage that the target release for the feature can be chosen "late" - it can be slipped to later releases or even accelerated for an earlier release without impacting anything else.

If you are proposing a standard for branching, then you also need to consider how to manage changes that are targeted at multiple releases. For any particular change request (e.g. a bug) it should be possible to track the status of the corresponding changes in each target release. Many bug tracking/change control systems assume there is one resolution for each bug, and when a bug is "Resolved" then it is done.
 
October 13, 2009
Votes: +1

Write comment

You must be logged in to post a comment. Please register if you do not have an account yet.

busy
Last Updated on Tuesday, 22 September 2009 08:29
 
clozaril generic canada propranolol cost comparison amex accept visa cheap online buy clomid citrate low acivir pills buy lasix online india no prescription order advair diskus online no presctiption adalat india buy cytotec online mexico no prescription what is the medicine lexapro used for suprax medication coupons low altace side effects ramipril us pharmacies online that sell bupron pyridium online no prescription buy viagra super active online no prescription usa ordering antabuse india where can i buy differin online buy lexapro tablets generic ditropan available cymbalta drugs side effects femara medication assistance program where can i buy prograf online without a prescription buy amitriptyline bar ordering altace without pharmacy bactroban genuine supplier scammers purchase cymbalta 20 mg antabuse where can i buy buy betnovate scalp application online cheap prescription allopurinol without where can i buy terramycin eye ointment for my cat where can i buy xenical online purchase acivir pills 11 pharmacy acticin drugstore online canadian can i order ventolin online albendazole without transcript order valtrex without a perscription buy pariet medicine order lisinopril 5mg online prevacid australia pharmacy vermox canada online no prescription digoxin online order lanoxin cost of clonidine without insurance drugs that contain ampicillin canadian kamagra terramycin online uk wellbutrin sr canada pharmacy canada doxycycline tablets online cordarone medication side effects oral combivent canada companies only diflucan canada order ephedraxin generic licensed pharmacy vermox levlen fast canada buy lozol mg online for cafergot buy without where to buy diflucan with visa topamax drug in canada pharmacy viagra super active india sale buy diclofenac gel generic lamisil no prescription needed online without pharmacy rx assistance for accutane buy proventil online canada i want to pay some ventolin order calan online no prescription with a visa ordering lasix mexico buy cheap extendaquin discount card for strattera flagyl generic name metronidazole prescription aciphex heartburn buy cytotec mg buy bactrim antibiotic purchase evista on line in usa cheap buy alphagan online buy ephedraxin generic pharmacies salbutamol buy combivent diarex no prescription fedex ups v drugstore accutane generic name for advair diskus 600 50 buy abilify 300 mg delivery order combivent online buy fucidin h no prescription online cheap accutane cost comparison order roxithromycin cheap buy diflucan online australia order cheap buy alli uk online approved flagyl without a prescription from india overnight levitra cheap aciphex online rabeprazole delivery cheap actos buy combivent us nizagara generic avodart from canada buy betnovate lowest price alli fda approved to buy without prescription purchase viagra plus in us order elimite pills ceftin online usa non prescription tetracycline safe furosemide no prescription buy flomax with no script buy eurax hc cream aricept prescription online nexium prescription cost target esomeprazole generic actoplus metformin do you have to be 18 to buy alli pills gallbladder buy arimidex us generic cholestoplex usa celebrex online prescriptions buy order alesse silagra buy online australia buy seroquel online usa order strattera without rx accutane on line in the mexico buy albendazole zentel premarin without a prescription kamagra now network buy glucophage xr online where do i erexin in usa order albendazole from usa without prescription generic viagra soft tabs online buy lopid mg effexor xr australia companies only buy ampicillin online us no prescription cephalexin mg order how to purchase celebrex buy renagel philippines dostinex online usa online no prescription uk for lamisil lotensin from china order cozaar canada ordering doxycycline uk actos price in india lamisil tablets and alcohol colchicine with prescription order strattera cod overnight purchase actos online pharmacy 1 order aciclovir tablets online 200mg cardura drugs com hypertension nizagara canadian medications without prescription purchase orlistat 60mg capsules buy cheap deltasone where to buy yasmin online uk cheap methotrexate best place to buy nolvadex in india buy accutane 3 20mg pharmacies canadian cheap altace erythromycin composition cheap levaquin 500mg zyvox 400 mg generic silagra usa citalopram online pharmacy indian alesse online cod medication advair diskus inhalation abilify and cold medicine buy 10 mg online abilify where to buy diflucan no prescription approved amitriptyline low dose recommended allegra 180 mg fexofenadine hcl buy roaccutane no prescription fast delivery how to buy arimidex in canada generic avapro in brand name amoxil ephedraxin canadian source current price viagra soft where to buy online drugs in the canada is nexium available on line in canada medicine cymbalta australia canadian styplon over the counter glucophage where to buy avodart online usa order xenical uk buy plavix online usa cheap ventolin inhalers sale buy aciphex generic drugs buy protonix without a subscription buy alphagan online thru paypals coumadin by mail where to buy ampicillin online canada where can i buy alli diet pills in england pharmacy online buy amaryllis minerva augmentin with prescription morning sickness medicine zofran where can i buy buspar without actoplus met prescription buy cefadroxil without rx diarex medication accutane overnight mastercard over the counter medicine that acts as levitra professional valtrex no prescription buy online cialis online pharmacy canada augmentin without prescriptions buy revatio online uk no prescription quickest bupropion prescription buy acticin elimite brand name coversyl plus blood pressure medication bystolic comparison atenolol buy ortho tri cyclen online mexico no prescription canadian pharmacy cleocin gel advair diskus comments ditropan free shipping where to buy periactin online canada cheap effexor xr venlafaxine buy clomid online without rx buy clonidine online orlistat without rx ordering nolvadex on line uk buy diovan without prescription prevacid uk boots buy cialis professional low cheap price canada deltasone generic bystolic blood pressure medication amoxil no prescription buy triamterene online without rx buy alli ireland pills online protonix on line is levothroid available on line in canada order altace 5mg pharmacies in buy allopurinol glucophage prescription information ceftin drug in fr pharmacy genuine bentyl online pharmacy spastic colon buy actos pioglitazone medication atarax canada sale where diclofenac buy arjuna herb propranolol without a prescription from us lexapro online pharmacies with out prescription buy prometrium cheap ditropan mg uk nolvadex cheap mexico pharmacy order viagra professional cheap very cheap proventil plendil reviews augmentin without a prescription from australia buy indocin tablets us betnovate no rx discount skelaxin order zithromax online no prescription with a visa brand viagra online in uk pharmacies buy albendazole online online crestor with no prescription best site for atarax how to purchase phenergan purchase prograf cheap benicar online no prescription buy sarafem mg buy tinidazole online from mexico buy carafate generic vermox generic canada peco exelon company cheap lasix online non prescription where to buy doxazosin online usa buy lotensin paypal cheap 84 alli 60mg low cost avalide online without what is suprax mg used to treat buy celebrex online no prescription purchase prevacid on line in mexico ordering seroquel canada how to order valtrex online levitra plus canadian source can i order cardura online grifulvin in usa purchase generic evista nitroglycerin uk boots alesse low dose pill like when does actos google generic augmentin tablets online is there a generic for allegra sinequan buy online canada buy sominex online us no prescription buy alesse ethinyl estradiol buy cymbalta canada plavix mg no prescription paroxetine mg canada finpecia birth control online mexico buy cheap indocin synthroid pills colors non prescription cialis super active purchase buspar on line in australia cheapest pharmacy buy altace viagra from usa sarafem canada drug store india buy generic actoplus met propecia on line in the uk allies of the british in the french and indian war no prescription tenormin order prometrium 200 mg cheap us buy augmentin cash prilosec vs nexium medication order lozol online buy augmentin cheap clavulanate generic aricept drugs alzheimer fifths disease ordering acticin drugs similar to prednisolone where to buy furosemide with echeck buy 1 generic acivir pills mg buspar without a perscription prescription drug lotensin generic viagra in 2012 benicar hct canadian pharmacy how to get cafergot propranolol is the generic name for prednisone from mexico buy revatio online thru paypals ampicillin order in the us suhagra pct buy motilium birth control online australia purchase motilium on line in usa pariet usa no prescription norvasc canada companies only avodart prescription prices yagara online in usa cheap ampicillin pricelist canadian exelon pharmacy online prescription abana online cheap canadian seroquel no prescription strattera user reviews adults accutane order in the us generic allegra pictures buy motrin no prescription lisinopril dosage by weight order floxin canadian discount crestor of canada order cipro online no prescription with a echeck where to buy nolvadex from india drug amantadine generic bupron discount no prescription purchase actoplus met on line in india buy sominex uk abilify prescriptions patient their altace without a script doctor prescription abilify coupons safe to buy generic diflucan from uk cheap buy acivir pills without purchase zantac online no prescription pariet order online no prescription zyprexa discount pharmacy floxin shop deltasone drugstore buy requip xl online purchase lisinopril 20 mg cheapest coversyl buy diclofenac potassium uk is there a generic drug for actos buy estrace canada order ciprodex from canada ordering aciclovir india allopurinol over the counter buying prescription amantadine india influenza cipro dosage for ear infection silagra generic name best place to buy cafergot in australia order lamisil online no prescription with a mastercard sarafem where to buy in canada over the counter bupron sr citalopram over the counter australia diflucan pills buy generic brand viagra india cabgolin overnight vantin no rx flagyl medication uses voltaren gel prescription information cialis professional uk boots cheapest acticin canada order plavix buy cheap alli uk purchase prinivil 20 mg generic version of clomid differin without a prescription from us cheap abilify 5mg pharmacy buy periactin lowest price ortho tri cyclen pct buy xenical uk no prescription purchase plendil 10 mg abilify where can i buy pharmacies in buy allopurinol cod cheap kamagra supplier reviews medicine aciclovir decadron comprimido non prescription cialis professional buy naprosyn paypal zoloft buy online us buy chloromycetin quality order viagra jelly from zenegra without a prescription from us florinef where to get alli over the counter glaxosmithkline drug how to get clozaril purchase cialis soft on line in the uk levothroid from canada voltaren uk pharmacy levothroid overnight pharmacy usa buy atorlip-20 without prescription medication class cordarone intravenous pharmacy japan allopurinol for sale buy bupropion xl cheapest buy antivert pharmacy order trazodone uk pharmacy purchasing periactin online zovirax online canadian how to get viagra sublingual bactrim no prescription overnight delivery discount generic lipitor 20 mg low aciphex pharmacy buy roxithromycin cheap viagra soft generic uk tinidazole canadian source where to buy alli diet pills aid ampicillin online erection packs 1 with no rx lotensin overnight pharmacy purchase sinequan cheap cheap doxycycline 200 mg mircette from usa minocin online pharmacies with out prescription cephalexin online order can you buy cialis in canada non prescription orlistat buy clonidine hcl order evista online no prescription with a echeck purchase extendaquin lowest price cheap tadacip mg pill rx health antabuse products ordering dostinex usa lasix with no perscription buy extendaquin visa canada proventil tablets online what is reglan mg used to treat buy luvox mg online order orlistat on line india cheap diflucan one prescription elavil drug in fr pharmacy where do i atarax in australia indian buy accutane buy estrace online buy arimidex cheaply prescription current price risperdal buy bactrim online from canada ordering benicar buy levitra professional online canada no prescription carafate canada online no prescription buy alphagan us drugstore fast delivery buy kamagra online buy cheap online uk kamagra tablets order lasix from uk without prescription metoclopramide generic reglan can i order levitra professional online order suhagra from uk without prescription singulair online uk best place to buy ampicillin in usa celebrex with no perscription buy cephalexin generic purchase flagyl in uk buy femara lowest price cheap buy allopurinol online pharmacy generic astelin spray pharmacy purchase orlistat online no prescription proventil pharmacy price phenergan usa sale order vermox line buy omnicef tablets buspar without a prescription from mexico digoxin online order suprax canada pharmacy benicar canadian pharmacy buy flonase online canada allegra d no prescription online cheapest rythmol paypal discount wellbutrin sr mg silagra discounted allies and enemies of mexico what is lipitor mg used to treat order synthroid on line usa zetia mexico is luvox available on line in canada kamagra pills online in the canada canadian alli cheap topamax mexico pharmacy buy kamagra oral jelly online buy viagra super active pills in the india cheapest robaxin paypal zocor canada online no prescription seroquel uk no prescription viagra no prescription online uk paroxetine tablets bp over the counter drugs similar to lipitor buy zovirax online safely prevacid from canadian pharmacy order bactroban ointment online generic actoplus met 17 500 cymbalta from india pharmacies anxiety atarax 10mg low dose coumadin and aspirin vermox birth control online mexico uk cheap alesse online pharmacy 1 order online aciphex mg zofran mexican pharmacy prescription brand cialis mg india where can i get shuddha guggulu from buspar over the counter australia buy aciphex cheapest prescription buy unisom generic glucotrol xl reviews colchicine generic name buy fluoxetine without prescription motilium fast mexico kamagra online next day delivery buy cheap discount motilium diflucan fluconazole 150mg alphagan without a script robaxin australia pharmacy atarax over the counter rx hydroxyzine estrace medication information order lipitor on line india cardizem buy mexico online cytotec with no prescription clomid where to buy in canada diflucan over the counter india ordering finpecia mexico allopurinol discount coupon prescription legal to buy atarax purchase aceon online strattera no script buy cleocin clindamycin desyrel without a transcript prescription safe to buy generic abilify from uk phenergan mg canada purchase synthroid lowest price order accutane 1mg online is there a shortage of cytotec neurontin uk no prescription generic actos 30mg tablet propranolol without rx arimidex buy online us where to buy prograf online canada delivery prescription abilify online bactroban cream no prescription cheap buy cymbalta tablets canada online cephalexin with no prescription buy zanaflex online no prescription usa buy amoxil 100 lantus cheap us pharmacy actoplus met generic usa purchase ranitidine dosage of augmentin for dogs where clomid buy uk prescription generic no prescription buy celebrex strattera birth control online india buy citalopram pills in the mexico alphagan pharmacies brimonidine ophthalmic flomax without prescription medication buy micardis discount delivery antivert online buy cheap lithium batteries camera i want to pay some voltaren how to order nizagara online purchase coumadin lowest price nexium mg no prescription no prescription pharmacy buy alesse order metformin from mexico without prescription allegra for purchase without a prescription abilify over the counter sale mexitil birth control online canada indian viagra jellyfish seroquel from usa combivent online from usa where to buy diamox in uk lipitor shop glucophage canadian online purchase ditropan in uk nizagara from australia lipitor where to get order florinef 0.1 mg buy abilify tablets usa buy coreg generic buy noroxin low cheap price where to buy allopurinol 300mg cheapest propecia paypal where to buy lipitor generic cheapest ephedraxin paypal purchase cholestoplex lowest price where to buy nexium for cheap how to get cialis super active brand amoxil on line no prescription vermox cephalexin over the counter walgreens proscar medication canadian pharmacy levothroid tamoxifen discount code discount alli of canada buy neurontin medication order dilantin 30 mg purchase tamoxifen citrate erythromycin over counter cream order ortho tri cyclen lo where to buy lotensin online usa buy aricept online uk pharmacy diclofenac overnight free delivery protonix online with out prescription allegra cheap online buy lipitor in canada prices zanaflex mexico pharmacy pharmacies buy allegra online cheap anafranil prices drugs voltaren pills online in the canada nexium no prescription online buy evista mg online discount lipitor coupon pfizer zofran mexican pharmacy buy vasotec online canada no prescription where to buy wellbutrin alli discount voucher code amaryl mg uk purchase flomax canada crestor commercials buy cheap prescription biaxin online buy actos 15mg pharmacy voltaren generic india erythromycin cheap australia pharmacy best place to buy uk cheap kamagra trandate usa generic ceftin india erythromycin without a prescription from canada altace medication high blood pressure buy vpxl tablets india dreamtilt online pharmacy where to buy trazodone online without rx how much does generic lamisil cost order dilantin online buy innopran online order abilify 11 mg abilify prescriptions squibb company purchase abilify on line in india can i buy antivert over the counter cardizem generic form buy lexapro without rx citalopram generic usa cialis mg canadian rx diclofenac delivery metoclopramide in usa generic combivent canada prescription allegra canada mexico pharmacy diclofenac gel where to buy bactrim with mastercard provera where to buy in canada cheap nexium india where do i nexium in canada purchase aciclovir 200mg tablets where to buy amaryl online canada can u get high from trazodone aid buy advair diskus 500 fluticasone buy actos medication canada drugs ceftin shopping where tetracycline is there a shortage of augmentin buy mobic no prescription buy aldactone pfizer generic buy augmentin cheap generic drugs buy lisinopril online us no prescription bystolic where can i buy nizagara without a prescription from australia avodart online from usa ordering amoxil canada prednisone tablets buy frumil have no prescription buy midamor online propecia stores uk buy silagra in india delivery deltasone online buy tenormin 50 mg zovirax mexico companies only buy cialis super active online drugs similar to yagara buy paxil without prescription zoloft pills online in the india buy mexitil online no prescription actoplus met online without prescription advair diskus generic drug fluticasone alli without a prescription order zoloft 100mg differin prescription drug buy tamoxifen online from india buy differin paypal buy arimidex with paypal cheap baclofen no prescription triamterene overnight brand amoxil online pharmacies with out prescription aricept drugs side effects where to buy kamagra mg metformin prescription uk midamor no prescription needed canadian online pharmacy chloromycetin us companies only cipro mg no prescription purchase generic erection packs 2 viagra jelly generic tinidazole generic canada vasodilan drug in canada pharmacy depakote order online elavil australia pharmacy prescription xenical slimming pills erection packs 5 drugs cialis buy combivent pills in the canada oder mg tablets of revatio cheap us altace online alli noprescrition needed canada online pharmacy propecia tablets uk best place to buy zithromax in usa prevacid no prescription buy depakote er online order ortho tri cyclen on line mexico lexapro over the counter australia furosemide online order lexapro drug in mexico pharmacy antabuse tablets 200mg low dosage of accutane side effects estrace generic estradiol discount arimidex cheap dostinex 1mg order sodium generic diclofenac where to buy diflucan in uk prescription cheap cipro uk prednisone purchase doxycycline dosage for malaria prophylaxis buy hydrochlorothiazide online canada no prescription generic adalat cc 60 mg suhagra from india purchase zoloft online buy generic hytrin online with prescription cheap medrol mg buy cafergot online overnight delivery buy actos online usa cheap singulair drugs buy hytrin online order estrace cream online canadian pharmacy topamax without a prescription from australia generic actos 30mg pioglitazone hydrochloride generic actos 45 mg pioglitazone hydrochloride metformin to buy in uk buy roxithromycin with differin buy mexico order etodolac 300 mg where to buy plavix cheap order coreg lowest price cheap abana 1 quality pharmacy purchase strattera cheap generic proventil ventolin buy cleocin t clindamycin topical buy prometrium online overnight delivery fluoxetine overnight pharmacy buy tetracycline online pharmacy buy alesse online no prescription overnight mexico pharmacies online that sell periactin prednisolone online without prescription buy cheap rythmol premarin generic drug tetracycline online cheap buy colchicine without a prescription atarax mg buy canadian buy decadron online is protonix over the counter where do i viagra soft in usa buy revatio drug buy suprax online canada lisinopril generic cost liposafe without a perscription finpecia online mexico avodart pills online in the usa tamoxifen on line in the uk flagyl canada drug store lipitor generic canada release date cheap provera no prescription acticin prescriptions amaryl drug in fr pharmacy yasmin from canada uk buy accutane online cheap buy tablets generic viagra kamagra online cheap isoptin without a script abilify drug in canada pharmacy price india ampicillin cheap is augmentin generic plavix overnight pharmacy motilium without a prescription from india purchase citalopram cheap buy lotrel online from usa buy cialis soft tabs online diamox prescription diamox delivery order lipitor purchase etodolac cheap order abilify 5 mg drugs buy atacand discount buy colchicine 1mg online minocycline antibiotic reviews acne elimite online without prescription diamox birth control online canada prescription drug levaquin 750 strattera online from india lanoxin online yagara no script lasix drug in uk pharmacy order colchicine from usa without prescription safe lexapro no precription generic zocor canada tamoxifen generic purchase allegra in usa cheap roaccutane 40 mg generic arimidex canada order benicar no rx canadian pharmacy clonidine reviews children indian medrol approved cheap low cost overnight clozaril online aciclovir over counter order levlen cheap generic viagra price exelon buy online canada pyridium over counter combivent online india periactin overnight prevacid otc in canada purchase norvasc generic brand cialis usa vpxl india sale can i order fluoxetine online can you buy lanoxin over the counter amaryl buy uk buy kamagra online next day delivery jelly buy cheap pharmacy cod ampicillin purchase stromectol 3mg buy minipress online from canada purchase clonidine 2 mg robaxin medication classification desyrel pharmacies buy zanaflex overnight shipping cheap requip no prescription celebrex medication generic where to viagra in chennai noroxin delivered to your home canada buy roxithromycin without prescription cheapest female cialis paypal is 5mg of lexapro effective where can i get some flomax buy avapro lowest price buy benicar mg buy brand viagra tablets mexico ordering nolvadex usa albendazole us buy generic prescription etodolac canadian medications without prescription order diclofenac 50mg online prescription mobic information warnings cheap generic cialis free shipping buy lasix visa where to metformin propranolol online uk trial erection packs 1 without prescription buy protonix 40 mg motilium buy uk levaquin over the counter uk order clozaril from australia without prescription buy generic cialis online pharmacy purchase paxil cheap vermox pills online in the uk trial erection packs 1 overseas viagra plus delivered to your home lexapro reviews 2012 where can i get some clozaril order avodart buy robaxin online from canada where to buy propecia in singapore best place to buy differin in mexico buy cheap hyzaar no prescription clonidine to purchase without a prescription buy alphagan us online prescription coumadin blood thinner medicine purchase lithium grease order omnicef 300mg buy bactrim ds no prescription trimethoprim augmentin in usa zocor buy online canada furosemide pills online in the uk lipitor over the counter canada flagyl noprescrition needed usa online pharmacy acticin drugstore pills buy advair diskus $500 over the counter erection packs 1 altace pharmacy low cost over the counter medicine pariet buy lotrisone lotion online betamethasone us pharmacies online that sell cymbalta lisinopril generic side effects order bupropion order altace without cheap aceon 2 mg ceftin buy online mexico prescription lexapro cost buy cialis soft flagyl to buy in the uk without a prescription citalopram usa pharmacy cardizem discount inhouse pharmacy finpecia cialis no prescription canadian pharmacy ventolin online mexico order florinef cheap advair diskus uk no prescription adalat online sony tv alli over the counter diet drug orlistat buy aciclovir 200mg tablets buy anafranil online no prescription plavix medication program buy cheap provera viagra soft drugs cheap clonidine cheap 1 accutane mg isotretinoin buy robaxin for horses zyprexa 10 mg vial capoten india angiotensin losartan fucidin fast uk order prednisolone 5mg proventil generic inhaler viagra soft generic canada aciphex non generic online next day delivery topamax with prescription buy cefixime online overnight delivery over the counter bupropion hydrochloride motilium on line in the india canadian alesse prescription buy order ephedraxin generic cheap allegra fast usa buy tretinoin 0,025 no prescription buy abilify 2mg online pharmacies safe to buy generic augmentin from canada usa buy vasotec without prescription celexa pills online in the canada brand cialis medication low advair diskus available order bactroban online buy bactrim mg alli online from india levitra to buy order trial erection packs 1 ranitidine online in usa medrol drug in germany pharmacy flonase no prescription fedex ups cheap albendazole 400 mg purchase tricor 145 mg buy ventolin inhaler online no prescription carafate prescription needed cheap order zestril online no prescription with a echeck where can i get some hytrin accutane on line in the canada over the counter glucotrol allopurinol discount online doxycycline tablets online buy tretinoin online uk can you buy viagra professional topamax from canada cheap flomax 0.4 mg revatio shopping buy voltaren low cheap price order flagyl no rx canadian pharmacy order antabuse 500mg cheap uk generic compazine prescription cheap cabana us sale cephalexin india sale abilify international pharmacies accredited elavil on line avodart prescription medicine ordering cialis india buy lamisil tablets canada how to buy alli diet pill best site for pariet pharmacy buy drugs lipitor purchase depakote in australia order avodart online no prescription cheap topamax online coversyl plus medication side effects perindopril discount buy alphagan drugs can you buy ciprofloxacin buy abilify no prescription online cheap prices brand cialis drug in usa pharmacy where to buy augmentin furosemide 40 mg purchase online nexium medicine for gerd moduretic uk boots pharmacies canadian cheap allegra buy trileptal online uk safe to buy generic trazodone from uk when will zyprexa generic be available cheap voltaren gel cheap lanoxin 0.25 mg purchase lisinopril online neurontin cheap australia pharmacy flonase online purchase where can i buy lithium batteries order roaccutane 40 mg online brand cialis in canada online no prescription uk for silagra periactin online with out prescription purchase vantin lowest price how long to get seroquel out of your system purchase buspar on line in canada order cytotec online finpecia mg india where can i buy lasix online without a prescription buy cialis super active mg online where to buy prednisolone clomid tablets online ordering atorlip-20 usa cheap doxycycline hyclate buy zovirax online australia no prescription purchase styplon in canada buy cheap accutane in usa generic alphagan p available how much is differin gel without insurance order accutane 4 mg online cheap accutane 4 mg purchase tinidazole on line in mexico where can i buy aciclovir is there a generic for advair diskus generic buspar uk terramycin no script buy cheap online augmentin without alesse medication oral contraceptive levonorgestrel and ethinyl estradiol effexor xr buy online india over the counter medicine motilium ampicillin drugs interactions dosage order bactroban ointment retin medication overnight delivery zyvox order online no prescription purchase cozaar 50mg online buy seroquel generic accutane drugs interactions buy combivent online bupropion dosage depression how to get off paroxetine propecia uk boots buy cheap generic anafranil buy motilium discount discount trial erection packs 2 mg zyloprim birth control online us buy retin a 0,05 diarex where can i buy order prednisolone from india without prescription synthroid online from usa where to triamterene buy cheap generic buy antabuse online thru paypals cordarone medication side effects patient ordering valtrex uk oder mg tablets of prinivil order antabuse no prescription buy hytrin visa order minipress online no presctiption buy aciclovir online india no prescription order clozaril from usa without prescription order propecia online no presctiption buy chloromycetin eye ointment canadian diclofenac for discount aciclovir buy online india pyridium shopping discount brand viagra is there a generic for proventil inhaler cheap noroxin 400mg online buy mobic low cheap price trazodone fast usa order yasmin online no presctiption without a script next day prescription altace best place to buy remeron online valtrex without prescription pills buy capoten captopril silagra tablets online where to buy lamisil online levitra plus without a prescription from canada buy naprosyn uk buy lanoxin online cialis 5mg online australia metformin online australia buy abana generic prescription propecia online prescription metoclopramide shortage viagra without rx generic advair diskus 100 51 colchicine over the counter india where to buy unisom sleepmelts aids purchase valtrex lowest price purchase seroquel us where to buy celebrex with mastercard tinidazole non perscription countries purchase keftab lowest price purchase tinidazole on line in canada buy colchicine paypal canadian pharmacy viagra professional buy lithium chloride powder where could i buy cozaar without prescription tetracycline buy online australia order tadacip online betnovate prescription buy tadacip tablets australia buy lithium ion batteries uk medicine cephalexin is for what cheap actos online pharmacy buy cialis soft online with no prescription the chepest decadron strattera usa sale purchase avodart in canada finpecia reviews order metformin canada viagra jelly overnight delivery cheap cabana adrenergic over the counter medicine lisinopril order brand viagra online no presctiption cheap aciphex 10 canadian pharmacy ordering lasix on line buy aciphex discount amantadine no prescription store generic buy abilify 2mg lowest clonidine online order cialis super active online uk viagra soft canada no prescription cephalexin canada pharmacy order cialis jelly lowest price generic augmentin dosage acticin online low cost discount 300mg buy allopurinol mg viagra by mail order buy clomid no prescription needed nizagara buy online uk lipothin online order alli on line canada lopressor canadian online overnight pharmacy hyzaar paypal where to buy tinidazole online where do i propranolol in australia ordering actoplus met online pharmacy purchase advair diskus in canada zovirax generic mexico where do i diovan in usa order allopurinol all rights canadian buy anafranil mg buy amitriptyline tablets india low cost canadian buy actos ordering trazodone canada purchase lamisil in us buy anafranil online uk torsemide from china drugs similar to chloramphenicol how to purchase lanoxin calan lowest price buy plendil online buy periactin tablets mexico order diovan online pharmacy colchicine without prescription pills diamox drugs com diuretic how to buy nitroglycerin in canada evista no prescription needed canadian online pharmacy buy lasix without a subscription order micardis from canada without prescription where can i buy alli in canada shipping available ordering aciphex medication purchase sarafem in usa safe abilify no prescription order calan cheap actos birth control online canada buy cheap premarin very cheap cardura buy zoloft online without a prescription buy strattera pills in the india cheap coversyl plus online arimidex online without prescription lexapro generic release 2012 india pharmacies online that sell ventolin generic phenergan with codeine allopurinol drugs levaquin canada overnight pharmacy buy get bentyl cialis jelly no prescription drugs similar to proscar metoclopramide stores allegra d generic name buy amitriptyline with paypal order coreg from canada lipitor drug in germany pharmacy order endep cod prescription buy motilium visa capoten pharmacy mg buy what is fluoxetine mg used to treat robaxin overnight pharmacy order abilify 1 aripiprazole pharmacy where can i buy viagra from purchase atarax in canada revatio mg india where to buy amitriptyline online uk where cane i buy clomid in south africa buy brand viagra online no prescription canada buy online no prescription order avodart crestor interactions with other drugs xenical tablets online what is xenical mg used to treat buy differin low cheap price buy retin a 0 05 generic tablets buy albendazole online where to buy ceftin online mexico prevacid without generic for allopurinol available generic nitroglycerin ointment where to buy doxazosin pill rx health abilify products calan by mail mobic on line i want to pay some suhagra order tretinoin canada online pariet with no prescription tamoxifen for purchase without a prescription buy generic buspar online no prescription the chepest elocon best place to buy roxithromycin online xenical prescription only nolvadex noprescrition needed canada online pharmacy buy finpecia tablets mexico purchase paxil cr pharmacies buy biaxin online cheap augmentin from uk pharmacies in buy alphagan canadian how to order brand viagra online over the counter medicine vermox antabuse drug in mexico pharmacy canadian pharmacy accutane drug medicine actos type pioglitazone hydrochloride where to buy paxil online without rx generic for trazodone purchase accutane on line in mexico pill rx health order elavil products indian ampicillin canadian flonase ditropan pct buy lopid online pharmacy shipped to canada buspar medication on line overnight membership order accutane cheap diarex overnight diabetes medicine actos buy unisom online sleeping aid buy ventolin nebules 5mg online synthroid medicine price atarax mg usa florinef tablets online order depakote uk pharmacy purchase augmentin 500mg albendazole to purchase without a prescription diet pills and effexor xr buy ampicillin online uk where to buy fucidin online mexico buy minocin discount buy viagra overseas order neurontin 300 mg citalopram without prescription medication where to buy plavix online usa altace without drug prescription lisinopril discount card discount ephedraxin mg oder mg tablets of nizagara non prescription betapace alli prescription weight loss coumadin low vitamin k diet buy inderal cheap online order citalopram rx buy finpecia generic diovan reviews abilify overnight delivery medicine cefadroxil used maxalt generic equivalent order clomid without prescription buy arjuna buy zebeta low cheap price ordering brand levitra usa skelaxin generic cost buy ampicillin uk he doctor purchase cordarone online buy amitriptyline online cheap price where can i buy tretinoin hydroquinone maxi peel buy lithium generic abilify by mail please where to buy yasmin mg purchase levitra 5mg online where to buy albendazole generic abana without a script canadian pharmacy tinidazole order hyzaar 50 12.5 mg allegra over the counter 2009 sanofi aventis where to buy cephalexin online uk where to buy propecia online uk withdrawal from paxil symptoms chloramphenicol pharmacies in cash order ventolin online canada pyridium online order isoniazid rx prednisolone acetate no prescription order drugstore doxazosin online valtrex medicine for herpes where do i cafergot in uk buy brand cialis mg cheap us buy albendazole buy zenegra online safely buy alphagan us drugs order medicines baclofen oral relaxant buy metoclopramide syrup robaxin without a prescription from us buy online uk kamagra oral jelly cymbalta for purchase without a prescription cardura online no prescription pariet generic canada amantadine through generic name for viagra ordering cephalexin usa buy provera online uk nolvadex for purchase without a prescription zocor no rx where to buy tetracycline online mexico lopid mg no prescription canada pharmacies online that sell luvox tenormin with no rx purchase albendazole without prior script crestor uk pharmacy discount kamagra of canada cheapest xenical buy online buy sarafem online canada where to buy amitriptyline 10mg 75mg through prescription buy abilify where to buy prednisone online uk buy atorlip 5 paypal cordarone for discount cheap albendazole atlantic drugs medicine amoxil 250 safe to buy generic xenical from canada buy robaxin have no prescription no script zithromax mg the chepest ortho tri cyclen cheap sumycin no prescription purchase albendazole purchase procardia in usa xenical medication canada paroxetine buy online canada order diarex cod order does alli work day buy robaxin online from uk over the counter premarin where to buy diclofenac gel purchase keftab online low altace side effects drugstore buy alli pills antabuse from mexico where do i elocon in uk levaquin canada companies only buy bactrim suspension order vermox in when is actos going generic prescription nitrofurantoin online with out prescription lasix drugs dogs buy advair diskus online with no prescription cheap uk free shipping buy alesse cheap generic compazine uk quick delivery purchase citalopram in mexico canadian pharmacy buy ceftin online lotrisone cheap us pharmacy dostinex no prescription fedex ups lexapro canadian online overnight pharmacy order nexium 40mg tablets cheap zoloft no prescription tadacip uk boots order allegra online cafergot pb medication voltaren tablets 50mg purchase strattera on line in uk best place to buy phenergan online celebrex canada pharmacy purchase avodart buspar mg canada actos uk drugs buy cheap pharmacies allington maidstone order aciclovir 300mg tablets prevacid where can i buy purchase actos 300 mg celebrex australia pharmacy medicine baclofen used what is levaquin medication is used for ordering xenical india buy allegra d uk buy nitrofurantoin tablets delivery adalat online without script buy exelon online singulair over counter equivalents clomid overnight delivery medicine amitriptyline 50 mg brand amoxil online meds augmentin tablets alcohol cheap paxil generic provera canada van you buy alli in canada reglan india buy vermox online from uk best place to buy tenormin without a prescription buy levitra plus discount cleocin medication class purchase amaryllis kit purchase kamagra jelly where to buy combivent online uk online mevacor overnight delivery buy sumycin uk pharmacy buy tenormin online no prescription betapace no prescription buy carafate pills in the canada order adalat 1 pharmacy overnight purchase luvox in australia buy actoplus met tablets us order actos canada generic pioglitazone suprax no script brand cialis online purchase without a script ampicillin online pharmacy is there a generic cialis in canada orlistat from australia cheap actos generic drugstore cheap calan blanes holiday secure diflucan over the counter where liposafe zoloft prescription order metformin from usa without prescription dostinex canada drug store buy cytotec online in london viagra professional online pharmacy generic nizoral canada prevacid over counter infants valtrex mg no prescription amoxil where to buy in canada erection packs 2 drugs prescription alphagan mg canada can you buy aciclovir tablets over the counter uk genital can i buy prevacid in canada buy albendazole 400mg discount topamax of canada seroquel generic australia buy furosemide tablets pharmacies in buy alphagan online canadian buy depakote online no rx canada and femara no prescription buy actos overnight delivery proventil online with out prescription female cialis online without prescription buy cheap prilosec plavix in india cytotec cheap canada pharmacy medicine augmentin 625 can i buy clomid in canada viagra soft shopping purchase topamax online no prescription where to buy furosemide without a prescription buy bupropion xl 150 safe to buy generic fucidin from uk brand cialis online in uk buy lamisil cream 1 low alesse estradiol purchase lanoxin lowest price purchase glucophage online no prescription vpxl over the counter usa hydrochlorothiazide needed buy purchase fucidin ointment flagyl online purchase canada brand cialis tablets online altace prescription assistance program accutane delivery can you buy pariet over the counter buy erection packs 2 without a subscription buspar online uk purchase haldol cheap canada pharmacies online that sell advair ordering viagra from canada overnight pharmacy tinidazole stds bacterial infections zoloft discount no prescription buy acivir pills us canadian bid voltaren no prescription allopurinol online meds buy digoxin mexico buy propranolol paypal avapro in buy generic blood pressure ordering zantac on line canadian astelin buy cheap viagra online forum where to buy baclofen acivir pills overnight delivery aciphex on line in the canada mexico pharmacies online that sell elavil hyzaar delivery canada buy chloromycetin without prescription coupons for abilify medication orlistat from uk strattera shoppers drug mart zocor online in usa toprol order online betnovate over the counter pyridium availability in canada viagra no prescription needed canadian online pharmacy buy desyrel generic over the counter clomid uk viagra super active over the counter canada buy differin tablets uk buy wellbutrin sr uk cheap canadian aciphex online cardizem buy online australia alesse drugs com over the counter medicine that acts as antabuse buy trial erection packs 1 online no prescription canada where can i buy vermox over the counter bactrim cheap uk pharmacy discount omnicef is there a shortage of levitra cheapest antivert rx buy antabuse mg online for buy alesse pharmacy diamox drugs com remeron online pharmacies with out prescription how to purchase atorlip-5 buy toprol xl 25mg buy deltasone mg how to get medrol furosemide over the counter buy diovan hct 160 25 where to buy revatio with mastercard amitriptyline without a prescription from mexico buy phenergan with paypal buy actos prescription cheap price biaxin mg uk levitra professional.com buy prograf online with out prescription where to buy colchicine tablets usp buy levlen pill accutane buy online approved generic paxil medication reviews cheap colchicine buy bactroban cream australia paroxetine overnight shipping pharmacy how to get seroquel xr canada buy zantac without prescription buy cheap generic advair diskus fluticasone what are nexium tablets used for tinidazole online without rx aciphex medication information is there a shortage of propranolol cheapest place to buy suhagra prescription order acticin cod ceftin overnight delivery where can i buy alesse buy accutane uk prescription over the counter drugs similar to orlistat what is the generic name for levaquin buy colchicine online cialis soft fast india what is risperdal medication buy nexium tablets australia nexium oral capsule delayed release best place to buy accutane online lexapro 5mg effective seroquel on line in the mexico how to order ranitidine low advair diskus without corticosteroids cialis professional online uk buy nitroglycerin spray canada order nexium 40 mg capsule order diovan on line canada buy tamoxifen without a subscription medicine amitriptyline 5 mg effexor xr mg india estrace medication women generic cialis online reviews order chloromycetin cheap pills no prescription prednisone safe to buy generic prednisone from uk purchase orlistat 60mg order buy actos online propecia over the counter canada the chepest aciclovir buy cheap arjuna buy elavil discount buy fucidin no prescription propecia lowest price evista tablets price where to buy flagyl for dogs online pharmacy benicar order erection packs 2 from lowest voltaren gel canada pharmacy combivent over the counter usa discount prednisolone otc cafergot pills online in the uk mail order nizagara online aciclovir australia companies only diclofenac tablets spc buy acivir pills cheap tetracycline where to get current price antabuse amitriptyline drugs side effect order effexor no rx canadian pharmacy cheap oral jelly india kamagra buy female viagra pills in india buy flomaxtra online buy avalide 150 12.5 amoxil without rx order prednisolone 25mg cozaar online pharmacy amaryl drugs com dosage buy vermox paypal suprax tablets dosage order actos diabetes pioglitazone buy requip discount order viagra jellyfish from pills buy cheap cipro elavil reviews fibromyalgia online pharmacy gasex lasix pills online buy minocycline for acne buy no prescription zyvox online order depakote from australia without prescription zithromax trusted online drug stores in canada ortho tri cyclen mg no prescription atacand fast canada canadian differin gel buy reglan online no prescription advair diskus generic generic prescription usa buy cialis jelly without prescription buy metformin for weight loss buy amaryl pills in the india actos in canada lisinopril australia pharmacy elimite dosage head lice pariet pharmacy mail order buy prometrium online without prescription overnight buy lotrisone order rosuvastatin in drugs crestor purchase roxithromycin online buy viagra super active with brand viagra with no rx where to buy serpina mg where to buy mobicam where can i buy viagra online with no prescription ditropan online india valtrex generic price where could i buy mexitil without prescription best site for rosuvastatin buy cheap generic aldactone discount order motilium no rx canadian pharmacy buy levitra professional mg buy dostinex online from mexico buy orlistat where can i buy clomid as pct canada benicar hct diovan medication side effects buy metformin generic cordarone medication interactions cordarone with no rx buy protonix lowest price maxalt combination nexium prescription information buy depo provera online no prescription estrace without a prescription purchase fluoxetine in australia where to buy revatio online uk safe abana no precription canadian pharmacy zyvox elavil in canada prescription without buy carafate allegra on line purchase phenergan online