The ISO standard does require you to “establish and maintain” an internal audit program. An added benefit to doing so is to help establish “controlled conditions” in the operation of your processes. The people that do the best job of the ISO stuff are the people that have the most effective audit program.

So here are some things to think about as you go about the job of administering your internal audit program

Establishing an internal audit program is an important job

The Requirements of the Standard

The organization shall: plan, establish, implement and maintain an audit programme(s) including the frequency, methods, responsibilities, planning requirements and reporting, which shall take into consideration the importance of the processes concerned, changes affecting the organization, and the results of previous audits;

So there is a lot of work in this one paragraph. What most people do is have a high level audit plan, and then set up a schedule and plans for each individual audit. There is an “internal audit procedure” that tells how to select the auditors and define the reporting requirements.

There are a number of resources available as to how to do this.

What is baffling to me is why people make this complicated.

The High Level “Program” plan

This is where you define how often you are going to do internal audits, what gets audited, and how all of that is scheduled.

Typically companies set up their system so that all of the processes are audited annually. However, there is no requirement in the ISO standard for this to be done. It is quite possible for some of the processes to be more or less often (that’s the part where it says “taking into consideration the results of the previous audits”.) If you have one or more processes that are well controlled and never give you trouble, by all means audit them less often than annually. If you have processes that are making you crazy, audit them all the time.

So a high level audit plan may look something like this, since your “processes” are well defined:

“We will audit the contract review, production, final inspection, purchasing, and receiving processes annually. We will audit the Quality Management, Resource Management and Leadership processes in the year of our Recertification audit..”

What this does is make it clear that you are focusing your effort on the productive processes, It allows you to reduce auditing the overhead activities, and keeps you focused on where the value is being added.

Process Definition is an important aspect of the Internal Audit Program

Auditing the Individual Processes

Once the high level plan is established, the normal practice is to schedule the process audits on an annual schedule. Most companies do not like the disruption of internally auditing everything in their plant at the same time. So, it is typical for an audit schedule to be set up to audit the processes one at a time throughout the year.

There are a couple of problems with this. One is that it is the same as procrastination. You start out in January with the intention of doing one thing, and when the time comes to do it, there are always temptations to do another thing.

You also miss out on being able to simulate the audit experience. There is such a thing, among employees, as “audit performance anxiety”. In companies where there is a lot of turnover, this is especially common. This is because new employees are less likely to have the experience of being audited. Having the audit done “all at once” puts everybody in the same frame of mind.

It also puts everybody in the same boat in regard to fixing the findings, if any.

An Internal Audit Program focusing on “High Risk” activities

In the place where I am the contract quality manager, and only get a couple of days a month, I have the process audits over a 90 day period. I interview the process owners, because typically they are the ones who get interviewed during the registrar audit. I also spend time auditing the processes where there have been previous non-conformances.

When the standard says “taking into consideration changes in the organization”, it is telling me to do just that. Make sure all of the new employees are audited. Review areas of past non conformity. Review past audit findings to see if corrective actions have worked.

Who Audits the Auditor?

There is a further thing to think about. In the small business where I am the quality manager, there are about 20 employees. I come in a couple of days a month mainly to do the internal audits, and run the management reviews. I also administer the document control and corrective action system.

So the question is, who audits these activities? In the 2008 standard it explicitly stated “auditors shall not audit their own work.” However, in the 2015 standard, that wording has been changed slightly. It now says “select auditors and conduct audits to ensure objectivity and the impartiality of the audit process.”

So there is an intent that I should be audited by someone else, but not a specific requirement.

A Finding Booby Trap

Here is a “finding booby trap”. In my side gig where I am the quality manager, I am the most experienced auditor in the place, of course. At most places there is someone like me who does the internal auditing. In most places, there is also a training requirement. The standard itself does not require that auditors be either trained or experienced. So in theory, I could be audited by anybody in the place, rather than me, and it would be fine. I have trouble getting people to do this job, even though I try to be non-threatening.

In places that have imposed a training requirement this is audit bait. I check the audit to see who audited the auditor, and then ask for some training credentials or evidence for that person.

There is an annual job I do for a client who mainly does his own internal auditing. I come in as an external third party for their internal audit program, and audit the part of the standard that he is in charge of. This gets him clearly off the hook, relative to auditing his own work. He doesn’t have to pay me to do his whole system, and he has his objective evidence. I would love to do something like this in the place where I am the quality manager, but there is a budget, which is zero.

The Audit Evidence

In my earlier post on “internal audit fails” I talk about excessive reliance on the checklist. It is very tempting to find a canned checklist, check all the boxes and retain it as objective evidence that you did the audit.

Checklists are quite often used as part of an internal audit program

The good thing about the checklist is that it provides objective evidence. The auditor can’t argue with it.

The standard is not at all clear on how much evidence is needed to prove that an audit was done. The standard says ” retain documented information as evidence of the implementation of the audit programme and the audit results. ” It does not say what form this needs to take.

I do contract auditing in addition to my jobs with the registrars and my quality manager job. A couple of times, I have been called out by an external auditor who did not think my audit was done properly. In one case, I was accused of not having enough documentation that I did the audit, because I did not record lot numbers, identifications, etc. from the client I audited. In another case, I actually recorded too much.

Some auditors are insecure that way.

“Suitable Documented Information”

In theory, the “documented information” could only be a statement by me that I audited the processes as indicated in the audit plan. It does not require any detail over and above that, as long as it is suitable for “the organization” i.e. “the client.” So, this could easily have been in the form of a three-line memo. I, Jim, have audited the XYZ company, according to the attached plan. All of the processes were audited as planned. I have the following findings.” and there is a list of findings.

Whether or not this is suitable is up to the client to determine. The management review requires the client to review “results of audits.” If you determine that this is sufficient objective evidence, you should be good with the three-line audit report.

Most of the time, in reality, you will get resistance from your registrar auditor unless you are more convincing. The amount to which this is true depends entirely on the auditor. I believe it is also dependent on the registrar that the auditor works for.

I believe I have talked about “auditor overreach” repeatedly. Here is one example where an auditor can impose his or her idea of the “properly implemented internal audit” on you, the client. A very wise YouTube producer cautions you on this topic.

Report Overkill

I saw an audit report from a registrar the other day that was 35 pages long, and it had a very extensive write up of all of the evidence that the auditor looked at. The attitude toward this from the two registrars that I work for is different. One requires me to fill out a process audit form, and requires me to write down everything I looked at. My other registrar only requires me to do that with “key documents” such as the quality manual, a few corrective actions, the management review and the audit report. The completion of this documentation is time consuming. But, it satisfies their own requirements for audit documentation.

In theory, it is auditor overreach for an to impose any standard of reporting on you, the client. You have to balance “sufficiently documenting the audit” versus overkill, which is expensive.

A reasonable auditor will recognize this and adjust accordingly. An unreasonable auditor will insist on similar documentation requirements to their own requirements. That’s not in the standard, but that doesn’t mean an auditor won’t try to occasionally lay something on you out of principle.

Audit Checklist

Documenting the Findings

This will be the topic of a future blog. To make it short, for the moment, the findings should be documented in this way:

What was the requirement? (based on the ISO standard clause)

What is the reality?

What specific evidence was examined to support the auditor’s determination of reality?

That way, the auditee can understand exactly what the fix needs to be.

The main failure mode on this is that internal auditors are not always familiar with the standard.

More About Findings

I have to say something about this. In my job working for the registrar I see a lot of internal audits. One failure I see is the internal audit is not finding anything. I can’t tell you how many times this has happened, unfortunately. The checklist is beautifully filled out, there are no findings, and then I walk out onto the floor and find a dozen.

The internal audit when best done, should find about twice as many findings as I do. That way I know that whoever the auditor is was knowlegeable about the standard. I also know that the internal auditor is identifying non conformities. What then gets done about this? That is the second question.

When I audit an internal audit

After having done a few hundred audits, I, as a registrar auditor, can pretty much figure out how much time to spend on this.

Are they auditing the required processes?

The first thing I do is check what the high-level audit program is. Then, I look for the definition of processes that the client supposedly prepared for clause 4.4. If the “processes” do not appear on the internal audit program, then I ask questions. The “processes” as defined by the client’s system need to be in alignment with the internal audit in some discernible way.

If some of the processes were omitted, I don’t immediately consider the audit non-conforming. I might, however, ask questions as to how the processes were audited.

The standard does say this:

 “whether the quality management system:conforms to…the organization’s own requirements for its quality management system….(and) the requirements of this International Standard.

So I am at the very least going to try to determine whether the “requirements of the international standard” have been audited as part of some other process. I will pick a requirement of the standard (someplace where the standard says “shall”) and see if that requirement has been audited somehow.

An expert with a clipboard, quality auditing

Who did the audit?

It should be reasonably clear from the documentation who did the audit. The majority of the time this is the client’s quality manager, or some knowledgeable person within the company. If training is required for this, I look at any evidence of training. Keep in mind that there is nothing in the ISO standard that requires internal auditor training of any type. Training is self-imposed by the organization. It is relatively easy to produce a pleasant looking training certificate. It is probably more important to be knowledgeable about the process.

The Internal Audit Quality Matrix (invented by me).

Here is a little matrix, that is about my emotional state when I look at internal audit records. I ask two questions, Was the audit done “thoughtfully?” And, did the audit come up with a lot of high quality findings?

The Internal Audit Effectiveness Matrix thinks about audit thoughtfulness and the quality of findings.

Important note: Neither “thoughtfulness” nor “high quality” as it applies to findings are defined in the standard. You will never see me write a non conformance on this. I will also never say this out loud in an audit. However, I will use this as a rule of thumb to look for audit trails.

Thoughtful internal auditing

How is this defined? There is no fixed rule of thumb on this. Here are some examples: If it looks like the internal used a checklist, checked all of the boxes and did not take any notes, and looks like it was done in about 10 minutes, that audit was not done thoughtfully. It was done mechanically.

After having seen a few of these, they are relatively easy to spot. If it looks like the client downloaded some checklist off the web, checked all the boxes and handed it in, it is pretty clear.

If the client used the company’s work instructions, went line-by-line through the procedures and took notes as to what they saw, I am ready to say that some work was done on this.

The client may have hired a third-party auditor to come in and do the internal audit, and if that person did a reasonable job, that is probably fine.

Is it possible for the client to have gone through the checklist and done a thoughtful audit? Sure. I will keep an open mind. Is it possible that the client went through their requirements in detail, took a lot of notes, and still did not do a good job of auditing? Yes I am ready to say this is also possible.

There are shades of gray.

Low Quality Findings

This is also hard to define. One clue to this is whether the findings were properly documented, referencing a clause in the standard. Another clue is whether the findings applied to a value added process.

Here are some examples:

“The employee did not have three sharpened pencils in his work station”.

In this case, there may be some rule on this in the company, and they may be in violation of it (some 5S programs do call for this sort of thing) but this is not the kind of finding that is going to drive change in the business. This is a bit nitpicky and will in all probability make the auditee and their supervisor mistrust the system.

High Quality Findings in the Internal Audit Program

Here is another example:

“The required setting for the equipment was 180-220 degrees, the reality was 170 degrees. The equipment setting was not done per the requirements. There was no evidence of an authorization from the supervisor or process engineer for doing so.”

Do you see where this is much better? There is a clause in the standard 8.5.6 which states:

The organization shall retain documented information describing the results of the review of changes, the person(s) authorizing the change, and any necessary actions arising from the review.

This finding is also well documented. It clearly states what the requirement was, it states what the reality was, and it does mention the requirement in the standard.

It also addresses an equipment setting issue which may be very important to the operation of the equipment.

Is is true that a setting of 170 degrees may be harmless? Of course. That is not necessarily up to the auditor to decide. What is the case is that a deviation was made without the proper authorization. When the client does the investigation, they can determine where the failure happened. So in this case, the auditor did his or her job and this is a high quality finding.

Document Control and Calibration

Side point: I am not saying document control issues are not important. I am also not saying calibration issues are not important. What I am saying is that it is easy for the auditor to find an obsolete work instruction or tool without a calibration sticker. If the audit report consists of a few of these, the audit may be missing something. Since nothing happens in isolation chances are if the document control and calibration processes are chaotic, there is more chaos elsewhere in the system. These may be indicators of bigger problems, rather than problems in and of themselves.

This is why in 8.2.2 of the standard says:

which shall take into consideration the importance of the processes concerned

This is because the ISO system wants you set some priorities. I guess if there are “a lot” of document control issues, it means that document control is not important at this place. You then have to ask what “is” important?

Clues from the Internal Audit Program

So in the diagram above, we clearly want the situation where the internal audit is thoughtful, and they are finding high quality findings. This is a properly functioning system that adds value to the business.

The situation where the internal audit process is mechanical, but they are still managing to find high-value findings is okay. You have to wonder, though, how they are doing it. Normally, it takes some effort to find high quality findings.

The situation where the internal audit process is thoughtful but there are no findings is also illogical. A thoughtful internal audit plan that does not turn up findings may that the client’s process is mature and there actually aren’t any findings. It may also mean that the audit team is missing something. Further investigation is in order to make this determination.

Finally there could be a situation where the audit process is not thoughtful, and they aren’t finding anything. This is called “wheel spinning” and is a bit of a red flag.

The Client is “Trained” by the Registrar Audit

This is something that happens with reasonable frequency. I go into a place, where for many years, the client has had the same registrar auditor that never writes up anything. What that does is set the tone for the internal audit program, and the internal auditors never find anything either.

So I, as the fresh set of eyes, find a dozen areas of non-conformity in the first half day. What happened?

In effect, the registrar auditor has trained the client not to worry too much about areas of non conformity. “Our auditor neer writes anything up, therefore we don’t have to work too hard on our internal audit.”

The end result is a system that is adrift. The people that do the best job of this have a strong internal audit program, and also, have “good auditors” from their registrar to help develop this process.

The Worst Case Scenario

This is where I, as the third party contract auditor, go nuts.

A client needs to pump up his or her internal audit program. They hire me, at considerable expense. I find a dozen high value NC that were basically missed by the external auditor. The external auditor shows up for the audit, and doesn’t find anything.

This reflects just as badly on me as the opposite, where I go in, and don’t find anything. (This practically never happens. I almost always find something.)

So the client then wonders, does my contract auditor have a problem?

This puts me, the contract auditor in a situation: What if I go in and don’t find anything, and the auditor comes in behind me and there are a dozen issues?

As for me, wearing my hat as a contract auditor, I am going to err on the side of finding things. That way if there is an issue it is on the registrar auditor to sort out.

Unfortunately as we have said many times, the area of highest variability in this entire system is the auditor.

Following Up on Findings

This is another area where I, as an ISO auditor, quite often spend some time. I will inspect the internal audit records, find areas of non conformity that are picked up by the internal auditor, and then see if they were handled properly.

This is called “backing up the internal auditor” and something I believe in personally. If the internal auditors aren’t backed up, you will eventually end up with findings like the sharpened pencil findings above.

What this means is that I will go through the corrective action system and see what happened once the non-conformity was raised. (Click link for an article on Corrective Actions) If I find out that the client did not fix the problem even after it was pointed out by the internal auditor, I may very well write a non-conformance against management for not improving the quality system.

When to Audit the Internal Audit

When I do an audit for a registrar, I typically audit the internal audit on the first day. I quite often deliberately track down whether actions are being taken.That tells me whether the corrective action system is being followed.

There is an opposing school of thought that says to do this on the last day of the registrar audit. You can then see if you, the auditor, found the same issues that the internal auditor found. That leads you to the same conclusion, which is the degree of support that the client’s internal audit system is getting.

I can be convinced either way.

Is your Internal Audit Program Effective?

This is a story of what actually happened: I was the registrar audit in a manufacturing place. I reviewed the internal audit documentation, and found that the audit used a checklist, and it did look like it was a situation in the upper left. The checklist was checked, it looked like it had been done in an hour or two, and there were no findings.

I looked at the client’s corrective action system and found out that corrective actions were not being followed up on, and were late.

So the holistic message that was being sent to me, the auditor, was that the management was a bit indifferent to making changes or improvements.

So after a lot of discussion, I asked the quality manager the following simple question: Is your internal audit program effective?

“Yes” was the reply.

What then happened was predictable, I went out on the floor and found numerous non conformances, and a lot more conversations were had. Three were found in the same process suggesting a potential major non-conformance.

So the question is: Is it up to me, the auditor, to determine if the corrective action system is “effectively implemented and maintained?” The answer is no. It is the client’s job to do this. Where I would usually look for a determination of this is in either the management review, or possibly the internal audit closing meeting, if there is one.

How can you, the client, determine Internal Audit effectiveness?

If I were the management in one of these places, I would ask the following questions:

  1. Are we finding anything?
  2. Are the findings “high quality?”
  3. How do the findings we have compare to what the registrar auditor is finding?
  4. Are we taking our findings seriously and fixing the problems?

If the answer to any of these is “no” then you should probably rethink some aspects of the high level system.

Here is where, once every few years, a contract audit by an experienced registrar auditor may be in order. That will help reset the system.

The bottom line on all of this.

When a client establishes a new system, or when a new quality management team takes over, there is an opportunity for introspection.

You do want to do internal audits in a cost-effective and value-added way, that find meaningful opportunities for improvement, but at the same time, does not bog you down sharpening pencils.

You do want to use the ISO registration and implementation process as a way to drive change in the business. There is no one right answer to this. The answer to this is to start somewhere, and then use the registrar audits and results of corrective actions as a way to evaluate effectiveness.

Your needs may change over time as well. An internal audit program that functioned fine at one point may need to be updated. Continual improvement, even in the internal audit program, is a core principle of the ISO system.

Tags