Continuous Staging: Scaling Continuous Integration to Multiple Component Teams

This month we will discuss some of the difficulties encountered when attempting continuous integration for multiple component teams working together to develop a large system. We describe the concept of a staging area to help coordinate the teams and stabilize the interdependencies between built versions of components.

Continuous Integration for Component-Based Development

In small team development, the practice of continuous integration [2] is an effective technique for keeping every one on the team coordinated with the latest results of all changes. Practicing continuous update [3] and private build [4] in one's private workspace [5] as part of a two-phased codeline commit strategy helps ensure that workspaces and work tasks stay in sync. Integration builds ensure that the team's codeline remains stable, consistent, coherent, and correct.

Any built version of a component that needs to be accessed by internal stakeholders (such as a QA/V&V group) needs to be identified by sing a label/tag). This ensures that anyone who needs to look at it, even after it is no longer the latest and greatest, can easily do so.  They can then also see which version of the component and its corresponding source code they are viewing.

In an ideal world, we could build the entire system directly from the sources in a one-step process, for everyone working on any component. Ideally, we would have the storage capacity, network bandwidth, processing power, and load distribution necessary to build the whole system.  At the very least, the ability to incrementally build the whole system every time before a developer commits their changes to the codeline.

Sometimes, for reasons of build-cycle-time, or network resource load, or schedule coordination (e.g., multiple time zones, or interdependent delivery schedules of components) this is not always feasible.  What happens when there are multiple teams and components, each with its own integration schedule or rhythm, that needs to coordinate with a larger-grained system integration strategy? There are many dependent factors to consider, including:

  • Relationships between sub teams, and their respective integration rhythms
  • Build-time dependencies between components (e.g., libraries and APIs)
  • Geographically dispersed teams and team-members and differences in time zones
  • Repository size and performance
  • System-build performance and available network resources

System Integration for Multiple Component Teams

Many large projects and systems require multiple teams of people to work together. In component-based development, it is common practice to see a system partitioned into multiple subsystems and/or components, with a team allocated to each component of the larger system (a component team [1]).  Each component-team develops a separately buildable part of the overall system, and typically develops and modifies only the source for their component.  The component team then takes ownership of the component's source code.  

  • Each component may be used or reused in one or more products of the overall delivered system (or within a product-family).
  • Some components may require delivery and distribution of their source code in order to build other components; while other components may require delivery and distribution of only binaries, or binaries and interface definitions (e.g., header files in C/C++).
  • The resulting delivered component versions can then be assembled together into the final system (subject to system and integration testing of course).
  • Each team should be responsible for ensuring that it delivers working code and executables to other internal stakeholders. When a two-phased codeline commit protocol is used, each task-level commit to the codeline is essentially stating that you have successfully compiled and tested the entire component with your changes incorporated.

If the team is large or dispersed enough to actually warrant sub teams, it may become necessary for each component-team to deliver tested, closed, sealed and signed, versioned binary libraries to the rest of the component teams. If the repository contains several components, and people build only their components to test their changes (rather than the entire system), then they are ensuring they have tested, closed, sealed and signed deliverables only for that one specific component! 


About the author

About the author

About the author

CMCrossroads is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.