Learning from Concurrent, Parallel, and Distributed Systems Design


This month we do a bit of a context switch from the world of parallel development to the world of concurrent, parallel, and distributed systems design (and then back again). The purpose is to see if any of the same patterns of concurrent, parallel, and distributed processing apply to the case of concurrent, parallel, and distributed development.

The Multiple Dimensions of Parallel Development

Multiple projects, multiple variants, multiple products, multiple teams, multiple sites, multiple customer/install bases, … multi-everything! Why does it all have to be so complicated! Each "multi"-something introduces a new dimension of complexity and scale for software development. The more "multi's" we have, the more diverse and complex the task of managing, organizing, integrating, coordinating and tracking all of the work.

Agile development methods tend to focus on simplicity and keeping things simple. When it comes to process, they also believe that it’s usually better to start small and scale-up by adding incrementally instead of starting with a large all inclusive menu and trying to pare down. As a result, most agile methods don’t have too many explicit practices to handle the cases of multiple projects, multiple variants, multiple products, multiple teams, multiple sites, and multiple customer/install bases. Instead the preference is to first try to find ways to eliminate these scenarios before trying to find practices to handle them.

This is probably sound advice as many of us are often too quick to jump to a solution before asking if the problem is one that really should be solved rather than avoided. Still, for many of us, it is nonetheless the case that these scenarios are business realities that aren’t going to go away anytime soon. So we nevertheless must seek solutions for these problems.

Agile methods and the agile community sprang from software patterns and the patterns community. So patterns are often a good first place to look for finding such solutions. However, the existing body of patterns literature devoted to CM is substantially smaller than for software design patterns. When we cant find what we’re looking for in the CM patterns literature, we may need to look elsewhere.

Core Concepts for Concurrency
If we look at the world of concurrent/parallel and distributed systems design, there are many common concepts and solutions that may also apply to the domain of parallel development. We just need to comprehend them, and how to map them from their “native” domain into our own, so we can see if they are applicable to us!

The literature on concurrent/parallel and distributed computing is fraught with technical jargon about processors, processes, and threads (among other things). Here is an oversimplified primer of some basic concurrency concepts:

  • A process is a task for a processor to execute. It includes the logical flow of operations to execute, and an area in memory to store the results of computations. Processes may be heavyweight or lightweight; Lightweight processes are called threads.
  • A heavyweight process has its own separate flow of control/execution, and its own storage area (address space).
  • A lightweight process (or thread) has its own separate flow of control but executes in a shared address space with other threads

With that in mind, we can now describe concurrent processing, parallel processing, distributed processing, and multi-threading.

  • Concurrent processing amounts to doing more than one thing (executing more than one process) at the same time with the same processor.
  • Throw in another processor, and we have parallel processing: two processors executing at the same time to perform separate (but possibly related) tasks
  • Put them on different machines in a network, and we have distributed processing
  • Multi-threading (a.k.a. multi-threaded processing) is literally multi-tasking! It is when multiple threads are active at the same time for a single process.

For a better explanation of these and other concurrency concepts, I heartily recommend chapter 5 of 


About the author

About the author

About the author

CMCrossroads is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.