Why your software development process needs an audit and why it doesn’t have to be so painful

When an old philosopher first coined the phrase ‘two heads are better than one,’ they were definitely talking about software development processes. Okay, not really, but a second set of eyes can go a long way in assessing your team’s processes and helping to solve some of your toughest challenges. Enter the software process audit. While regular team retrospectives should always be your first line of defense against inefficient workflows, an outside perspective can help solve challenges that the team is just too close to see. 

There are several formal certification programs built on software process audits and assessments. Depending on the specifics of your company, these may be a key component to your quality management program. The rigid procedures and bloated documentation that can come with these audits have always bothered me a bit in an agile environment, however. When the point is to iterate quickly to solve problems and deliver customer value, where do you fit an audit and report that can take more than three months to complete. I’m a proponent of a more lightweight review that you can implement much more quickly and cheaply. Let’s call it a Software Process Checkup.

Don’t get me wrong, many industry-standard audits and assessments can be valuable, if not required for your team. I’ll touch on some of the good reasons for these in a second. That said, more frequent and less painful software process checkups deliver incredible value to an agile team.

First: What is a software process audit?

Put most simply, a software process audit is a type of software audit or review focused on the processes of the development team. These are also called software process reviews, software quality assurance (SQA) audits, layered process audit systems, system development audits, and a host of other names. I’m going to use audit, review, and assessment interchangeably from here on out to refer to any independent assessment of a software development process.

Audits involve an independent auditor analyzing a software development process to provide suggestions for improvements. Audits may have particular target areas. The SOC 2 and SOC 3 audit processes, for example, focus mainly on how a software organization safeguards data, especially customer data. There are also more general assessments like the Capability Maturity Model Integration (CMMI-Dev) that focus on the overall process and capabilities of a development organization.

While there are some clear differences between the different types of software process audits, any audit should have a few main components:

  • Clear objectives – Before any review, both the reviewer and the stakeholder requesting the assessment should be clear on its purpose. Is the goal to resolve one particular pain point that the team keeps bumping up against? Should all or a subset of process areas be considered? How much time and effort should be spent on the review? There are no right or wrong answers to these questions, but everyone should be on the same page before the review begins.
  • Varied viewpoints – A software process audit should seek to look at the process from all angles. Auditors should seek input from developers, designers, managers, QA testers, business stakeholders, and anyone else who participates in or is affected by the software development process. An audit focused on only one group of stakeholders is sure to miss the mark.
  • Well defined process – Similarly to the objectives of the audit, everyone should be on the same page when it comes to the steps of the assessment. What significant activities will the audit include? Which stakeholders will the review involve? How long will all of this take? What are the expected deliverables of the review? Again, no right or wrong answers, but this needs to be clear up front.
  • Independence – Every team should regularly review their own processes anyway, but the purpose of an audit is to get an independent set of eyes. Stakeholders are often too close to the process to see problems or new approaches. They may also be biased towards solutions that make their lives easier without recognizing impacts on other stakeholders. An independent auditor or team of auditors will be able to compile the feedback of all parties into one cohesive set of suggestions. 
  • Actionable outcomes – I’ve seen too many software processes audits that end in a lengthy report that details process deficiencies without clear next steps to resolve them. Understanding the process and its issues is only half the battle. A proper audit must also offer clear, actionable steps that can be taken to resolve them.

What is the purpose of a software process audit?

The primary purpose of any software process audit is to help find ways to improve your software process. Beyond the basics, there are a few other outcomes you may be looking for:

  • Solve specific problems in your process – You may already have a good idea of what you need to fix. Maybe you’re continually finding bugs that made it past QA. Maybe your business stakeholders are often complaining that new features aren’t what they were looking for. Maybe your developers are stressed and turnover is high. If you know what the problem is going in, you can often use a process audit to find the root cause and solve it.
  • Ensure you’re following current best practices – Software development practices change at the speed of light. Evolving tools and techniques can often unlock new potential in your existing processes. An independent auditor can likely suggest some new ideas that you weren’t even aware of!
  • Get a “Second Set of Eyes” – Again, every good team should be evaluating themselves and finding ways to improve every day. People working in a process every day, however, may be too close to it to see the true nature of its flaws. A second set of eyes can cut through bias and “the way it’s always been done” to deliver out of the box solutions.
  • Check that you are meeting your goals efficiently – Even if you are delivering high-quality software that meets your customers’ needs, you may be doing it at a higher cost than you need to. A review of your process may find ways to achieve your goals more quickly and cheaply. 
  • Analyze large and complex systems – Development operations in large enterprises quickly grow beyond the purview of a single team’s retrospective process. No one group can completely resolve complexities arising from communications and dependencies between teams. An audit process can help to marry the viewpoints of individual teams and stakeholders to uncover organizational issues and solutions.
  • Measure improvement – More formal audit programs like SOC and CMMI include precise measurement criteria to ensure that their audits are repeatable and scalable. These metrics can be used to compare two different teams or to measure how a given group has improved between audits. Less formal processes like a Software Process Checkup can also include similar measurements. Defining and measuring these metrics does add some overhead, however. It’s important to ensure that they help to achieve the goals of that particular assessment before implementing. 
  • Get Certified – Certain formal audit processes result in the certification of an organization or team. Independent auditors can apply standards like ISO 9001 and CMMI-Dev to officially rate your processes. Some customers, especially in the government space, may require that your process be certified to a particular level to award you a contract. While formal certification may open up certain customer segments, it can be costly. If not required for your industry, you’ll want to focus on other, less onerous ways to gain the benefits listed above without the overhead of a formal audit and certification program. Which is a great segway…

So why use a Software Process Checkup?

Hopefully, by now I’ve convinced you of the value of a second set of eyes when evaluating and improving your software processes. Like any good agile practitioner, you’re probably saying: “How can I get these benefits without an expensive and lengthy audit?” The answer I’ve come to is what I call a Software Process Checkup.

By focusing on the objectives of your process review and leaving out some of the more onerous documentation and measurement requirements, you can get 80% of the benefits for 20% of the effort. I’ll explain how this works in a second, but in comparison to a formal audit and certification process, the Software Process Checkup is exponentially quicker and cheaper.

This cost and time savings means that you can afford to run these processes more frequently. This is super important in an agile environment where you should be iterating and improving often. ISO 9001 certification, for instance, can take somewhere between three and six months. By the time that’s completed, many of the recommendations may be obsolete!

A more targeted Software Process Checkup can give you actionable recommendations on your specific problem areas. These will allow you to improve the most problematic spots quickly, then to come back and review other pain points. Rinse and repeat. In this way, you can consistently iterate and improve over time.

How does a Software Process Checkup work? 

The whole basis of the Software Process Checkup is that it should be targeted. Think of the Pareto principle here. We’re looking to streamline the audit process to give you 80% of the value in just 20% of the cost. We accomplish this by focusing on the most impactful improvement areas and avoiding extraneous analysis and documentation. These are the three steps I follow in a targeted Software Process Checkup:

Step 1: Determine objectives

The most important part of the Software Process Checkup occurs at the very beginning. The auditor, the process owner, and any stakeholders who will be involved should have a shared understanding of the objectives of the checkup. This can usually be gleaned through a brief conversation around the process objectives and any major pain points.

First, agree on the basics – Don’t rely on assumptions. Before the checkup starts, make sure that everyone is clear on things like:

  • Budget for the checkup
  • Expected duration of the checkup
  • Time constraints on the involvement of any team members or stakeholders throughout the checkup
  • Expected output of the checkup – hopefully, this is a list of actionable recommendations around the specific target areas (keep reading for more on this)

Then, pick particular process objectives to focus on – Dig deep here to understand what you truly value in your process. Sure you’re trying to develop software, or solve your customers’ problems, or analyze particular data sets, but that’s not enough. Are you more focused on solving the problem at hand quickly or cheaply? Is quality your primary concern, or are you looking to get a prototype to market so you can begin iterating? Are the product requirements well known up front, or do you value agility and the ability to pivot? 

While every process should be focussed on efficiently solving a problem, remember that there are many different ways to measure success. You need to think through which metrics are most important to your team and your business. Try to determine what you’re trying to get out of this particular checkup, and remember that this may change by the next time you do this. Some focus areas may include (but are certainly not limited to):

  • Cost
  • Speed
  • Agility
  • Customer involvement/ Transparency of the process
  • Employee Satisfaction (this one is often overlooked, and turnover is likely your business’ most significant expense)
  • Security/ Protecting customer data

Remember, the idea is that the process checkup should be quick enough and cheap enough for you to run it often and iteratively improve. With this in mind, try to narrow down to just one or two most important focus areas. You can always look at the process from the perspective of a different focus area the next time you run a checkup. 

Then, determine your biggest pain points – Once you have one or two areas narrowed down, try to identify your most significant pain points in those areas. The main thrust of the process checkup will be determining the root causes of these pain points and proposing realistic solutions. 

It is critical that you look at the major pain points from the perspectives of all of the different process stakeholders. In addition to the management perspective, make sure to get the opinion of team members, key customers, internal security folks, DevOps or IT, owners of any dependent or interconnected processes, etc. Depending on the scope of the analysis, this discovery of specific pain points may be a significant activity in the checkup. 

If you have the budget, you could use a structured KJ session (more on this in a second) to help the team come together on the most critical problem areas. For a more lightweight approach, you could use a shorter kickoff meeting with representation from major stakeholder groups. Just remember, defining the most crucial problem to solve may be the biggest driver of the value of the checkup. Don’t skimp here!

Step 2: Apply lightweight analysis tools

Any auditor who runs Software Process Checkups will likely have their favorite set of tools. Ultimately the tools used in the analysis will be determined by the constraints of the review and the main checkup objectives laid out in step 1. A few of my favorite lightweight process analysis tools are:

Stakeholder interviews – These are the bread and butter of any process checkup. Sit down with people and ask them what they think. Remember, the people working in and interacting with a process every day will know it the best. Solicit feedback from management, customers, developers/designers/QA etc., internal support folks, owners of connected processes and systems, and anyone else who interacts with or is affected by the process. Budget or time constraints and availability of individual stakeholders may limit the scope of your interviews, but you should try to include as many different perspectives as possible.

Shadowing – Another great way to get a feel for the process is to watch it first hand. In the spirit of keeping costs and analysis time down, try to focus on the key components of the process and shadow representative instances. These are often agile ceremonies, code reviews, requirements gathering sessions, or any other touchpoints where stakeholders come together to make decisions. Ask about the most critical points in the process during stakeholder interviews and focus on these. 

KJ sessions – KJ sessions are my favorite way to quickly help a large group of people come to a conclusion together. The KJ is a structured methodology using post-its and voting. The idea is to get many ideas out on the table, then converge on the best ones without wasting time getting sidetracked or bickering. For more on KJ sessions, check out this great description: https://articles.uie.com/kj_technique/.

The 5 Why technique – In the 5 Why method, you continue to ask “Why?” each time you get an answer until you get to the bottom of the problem. You don’t necessarily need to ask Why exactly five times, but you should keep asking until you’ve identified the root cause. You can use this technique in stakeholder interviews or in larger group settings. Don’t let the simplicity fool you, this is a powerful tool! An example may sound something like:

Customers keep reporting issues with the payment screen. Why?

There are often timeouts that prevent the payment from processing. Why?

Code that causes timeouts is being deployed to production. Why?

QA is not catching these errors in testing. Why?

The payments don’t timeout in the test environment during QA testing. Why?

Our testing environment doesn’t match the production environment closely enough. (root cause)

Fishbone diagrams – Fishbone diagrams are similar to a 5 Why analysis, but they are a bit more directed. These use a visual diagram to help determine root causes in specific areas. Here’s an example:

Fishbone Example: Bad Coffee
A fun example of a Fishbone Diagram for Bad Coffee. CC BY-SA 3.0.

Basic Process Mapping – Creating a visual depiction can be a powerful way to communicate about and understand a process. To keep the cost down, focus on the communication aspect, rather than formatting the prettiest picture in Powerpoint or Visio. I like to start with a rough whiteboard diagram focused on just a portion of the process at a time. This allows for targeted communication around the process steps while minimizing the time spent on documentation. 

An example of a rough process map on a whiteboard. Allowing for cross-outs, non-standard notation, and abbreviations make this an efficient communication tool.

Step 3: Keep reporting focused and actionable

This step is pretty straight forward, but it’s also easy to mess up. Remember that the point of the Software Process Checkup is to get the value of an independent audit of your system without the overhead of more formal audit processes. In step 1, we defined the most critical problems that we are out to solve. The measure of your final report should be that it gives actionable suggestions to address that specific problem or problems without extraneous documentation.

One common pitfall I’ve seen is writing volumes on standard process models before defining specific suggestions for this one team. Of course, background on standard processes or techniques is often necessary to provide context for recommendations. After all, it doesn’t help to tell someone that they should be using more automated testing if they don’t understand what that means. An excellent way to achieve this context without spending too much time and money is to make liberal use of the wealth of content on various processes and techniques available online. Linking out to others’ descriptions of the basics allows the auditor to focus on connecting the dots, illustrating how solutions can be applied to this specific problem. 

Another thing to remember is that each step should be truly actionable. “Developers should be peer-reviewing each other’s code,” is not nearly as helpful as detailing how to configure the source control repository to enforce code reviews on pull requests. The team should understand not only the deficiencies in their process or even the theory behind the solution, but also what they can do today to begin fixing the issue. Don’t be afraid to use anecdotes to illustrate suggestions. Examples of how a recommendation can be applied are often easier to understand than the general steps of a solution.

One final note on the recommendations that come out of a process checkup: remember that these are just suggestions. If the auditor has done a good job, by this time they should understand the team culture and propose things that should work for this specific group. That said, no one will understand the team as well as the unit itself. If you’re sure a particular suggestion won’t work for your organization, tell the auditor! The auditor should not merely give one prescriptive solution and leave it at that. The final report should describe the theory behind a solution and offer alternatives. 

Similarly, don’t think of the report as the final deliverable of the process checkup. The written report is a starting point to begin the conversation around the problems and potential solutions. The auditor and the team should sit down and go through the suggestions to ensure that the team understands what they need to do next, and the auditor can correct any misunderstandings and update recommendations as necessary. I like to deliver draft recommendations to be reviewed and discussed with major stakeholders before finalizing any suggestions into a final written report.

Ready to take your software process to the next level with a Software Process Checkup? Check me out at JanaitisEngineering.com.

Andy is a web developer turned team management consultant and founder of TeamWins.org. He’s passionate about working with teams to solve business problems in efficient, effective, and repeatable ways. Interested in working with Andy? Reach out at JanaitisEngineering.com.