It is clear that management needs a mechanism that provides indicators of any looming crisis, a measurement of real risk that can be acted upon. This can't include tools such as key performance indicators, traditional compliance based auditing or in process measure because , while useful, they are backward looking, recording what has already happened. They tell us little about what is likely to happen in the future.
Compliance auditing focuses on the tangible-what can be seen and touched-meaning outputs and the quality of these in terms of conformance. It operates at a transactional level. But in the real world it is not just outputs that count. More important are outcomes -what these outputs mean to customers and other stakeholders- because this is what is really delivered. It is the organization's culture and how people behave together that deliver its results. Traditional compliance auditing misses many of these usually intangible issues and therefore only gathers part of the picture.
Compliance auditing should not be disbanded, but it is clearly not enough for the 21st century where identifying the risk to outcomes is becoming vital. The practice of auditing needs to identify not just what is visible but also to understand the importance of what is currently invisible, before it is too late.
The Project Structure
Recognizing the need for change the High Performance Group has intitiated a project to define the ways in which the apparent shortcomings of auditing could be addressed. a key part of the tool kit was a behavioral assessment methodology with the aim of:
- Creating an auditing method to assess the real-world, based on what people actually do.
- Showing how system and process risks can be scored , benchmarked and managed consistently.
- Determining how significantly more evidence can be gathered and how this could be analyzed to the delivery of outcomes.
- Determining the effect of this on audit process and the role of the auditor.
The ongoing project involves the inclusion of a number of large and small organizations using the approach and providing valuable feedback. As a focus , the University of Canterbury in New Zealand has also been using the approach in a carefully controlled environment whereby students studying for a Master's in Business Administration could apply techniques as part of their studies and provide critical validation. UKAS has also reviewed the approach as part of the requirements 4 and 5 and it has been used to gather evidence for certifications.
The university runs an MBA in Quality Management in which students study management systems and business processes as living entities , culture and behavioral indicators and auditing and assessment techniques. External validation of the behavioral assessment approach took place as an integral part of this course and in the last four years 18 very different organizations have taken the assessment, including:
- Hospice/hospital/medical care providers
- A financial services company
- Educational establishment
- Food producers
- Manufacturers including those of aeronautical , mechanical components and electronic components.
There are several factors behind the rationale for using the behavioral assessment technique. Once we move away from considering transactional compliance -type issues , organizations are complex. At the compliance level it is simple; evidence exists or it doesn't and often it is invisible. There are huge number of interactions that take place between people in an organization, each affecting the other in different ways and with different levels of importance . When this complexity is understood , it is quickly realized that collecting the amount of data required , analyzing and reporting risks to outcomes and objectives is beyond the auditor; they need help.
As a result, an online assessment tool was used to:
- Gather more evidence
- Reduce intrusion for the organizations concerned.
- Ensure confidentiality and therefore be potentially more revealing
- Be more cost effective
- Consistently collate and analyze results
- Reduce the students /auditors time in data collection
This web-based approach is not a survey, but a sophisticated assessment technique that tests organizational competence based on behavioral science , with an outcome and risk focus. It places conformance as a step in the maturity towards effectiveness and efficiency.
The assessment
While it is belived that the underpinning science and assessment logic applies to any subject, the university wanted a control subject where the results could be benchmarked across the target organizations. For this reason ISO 9001 was chosen as the subject.
The online assessment was designed in UK with the behavioral indicators established as description of business-as -normal , as experienced by those who would be participating. The assessment were planned by the students by identifying the internal and external stakeholders who are impacted by or take part in the scope forming the basis of the benchmarking activity from which risks can be identified.
Organizations were invited to take part over a four week period with each auditee contributing about ten minutes of time. The assessment system collected the data for the students and analyzed its importance , automatically providing a technical report against clauses of the standard.
The data was then analyzed against the eight quality principles contained in ISO 9000 series and the other factors associated with process management.
To validate the results the students and auditors reviewed the findings determining:
- Areas of potential risk , by department, function or group , using the scoring system where 40% represents acceptable performance. Scores above 40% represented increasing levels of effectiveness in the subject and it being embedded in the whole organization management system. Scores below suggest increasing levels of risk.
- The areas that would be specifically targeted for compliance auditing. The logic was that if the behavioral assessment was itself an effective auditing technique then there is no point re-auditing the same thing twice.
Each organization was then visited by the students/auditors to:
- Validate the results of the behavioral assessment against the reality of the organization
- Look closely at the targeted areas identified
- Compare the results with those that the certification body reported if one had been recently involved.
- To gather feedback on general accuracy and use of the approach.
Research Findings
The MBA student reviewed the findings of the validation activity , which the university summarized as follows:
" The evidence collected over the late four years shows that the approach has been consistently accurate, as well as time-and cost -effective.It addressed well-documented and inherent weaknesses associated with surveys and more traditional assessment approaches. While nothing is foolproof and more traditional auditing activities should not be discarded , the approaches would, for various reasons, find extremely difficult to achieve without significant cost."
" The main strength of the approach is that it collects tactical data based on the outcome of behavior as experienced by different stakeholder groups. The ability to automatically and therefore consistently analyze the importance of each piece of tactical information is the key elements that makes the assessment tool very valuable. The findings provide a profile of potential risks leaving the user to determine which risks are the most important on which to base improvement solutions that best support the delivery of objectives."
Audit fatigue
The same audit data can be used in different ways to report against standards, outcomes and objectives as is required. Using the same data more than once cuts the costs of auditing auditor intrusion and, at the same time, it adds value for the organization.
The changing role of the auditor
Auditing is now required to act in a new paradigm . The inclusion of other audit techniques means that auditors system in terms of how open , closed and mature it is and select appropriate audit techniques.
As the audit takes place , the different techniques will yield different levels of evidence and the skills is to build this together to demonstrate conformance and the risk to outcome/objectives. Thse auditing practices require changing the role of the uditor in all areas of understanding , planning, analyzing and reporting.
Audit planning is a core competence
Audit planning should not simply be about creating an agenda or meeting schedule to ensure each clause is covered; it should be much wider. It is about understanding:
- The outcomes the system or process has been designed to achieve
- The balance of effectiveness and efficiency outcomes measures
- How to use a range of techniques that will make visible the inherent risks in the organization
- The factors that will increase or decrease risk and apply these to the audit scope
- The requirements of the specific standard or governance requirements and the range of audit techniques that may be used in assessing them
- How the use of technology and non-human based activity not only saves costs but is desirable to increase value.
Every standard organization, system or process is different therefore so must be the audit programme.
Compliance and conformance
There is sometimes confusion between compliance and conformance. Compliance is an auditing technique with weaknesses if used alone. A reliance on compliance can drive in risk and rarely achieves the outcome- driven audit reports now needed or measure of effectiveness.
The need to assess the real world and what actually happens
The culture of an organization or business process is determined by the interaction of different people. The complexity of an organization at a behavioral level means that traditional compliance- based techniques are not really effective.
This is because questions frequently condition the response , confidentiality becomes an issue, issues raised can be sensitive, there is no hard evidence to check and the large amount of data needed to assess behaviors makes consistent analysis very difficult. Consistency is often difficult at a compliance level and this is only magnified at a behavioral level.
What is needed to assess behaviors
The research shows that to assess behavior we need to collect a lot more information in order to give the level of confidence that is required by organizations and develop a 360- degree approach , involving as many of the stakeholders as possible.
Using audit evidence
Traditionally a piece of information is used as evidence that a requirement is or isn't being met. This is a one-to-one relationship. When assessing behaviors there are many sources of evidence and each has a different level of importance dependent upon how critical it is to the delivery of one or more objectives. It is a many-to- many relationship The increase in evidence makes using the auditor's judgement almost impossible. This means that an IT solution is inevitable as an aid.
Where next?
Organizations are recognising the need for change because they:
- Understand it is how people behave that drives results.
- Need to manage risk to the future delivery of objectives
- Need to balance transactional techniques (such as compliance , key performance indicators and control charts) with lead indicators
- Need to present strategic information to management using graphs, numbers and benchmarks
- Need to focus on what outputs mean to stakeholders in terms of outcomes
- Understand the desire to move on without throwing out the past.
The project has shown that our challenge as auditors is to develop strategically in the way that we plan and deliver audit programmes. We have to think more about outcomes than we have one in the past. We cannot be satisfied with just looking back and saying this or that was wrong- that's easy. It is more of a challenge to help stop the event from occuring in the first place.
It is difficult to audit a process where documented evidence isn't available and then work out if it conforms to a requirement. It is to measure the risk to achieving something. We need to embrace new tools -that is about audit management.
Walang komento:
Mag-post ng isang Komento