Linggo, Agosto 22, 2010

Does Mystery Shoppers Work? - A Study Focused On The Mind of The Mystery Shoppers by Jolito Ortizo Padilla published in Singapore Times.

Service performance and its impact on the customer experience are seen as the key to customer satisfaction worlwide.Gronroos, a professor of service and relationship marketing, divided this service performance quality into two parts- technical and functional. Technical quality is concerend with "what" is done whereas functional quality focuses on "how". The "how", although harder to measure, is vital. Service managers need to reduce variation in both process and product quality so that all customers have the same experience with what they received and in how it is delivered and so leave highly satisfied.

The main difficulties with monitoring performance are associated with the diverse nature of services, their relatively short shelf -lives, their blend of tangible elements (collecting products at a supermarket) and intangible ones (being served at the check-out) and the fact that consumption takes place simultaneously with production. One method that has increased in popularity in recent years is the use of mystery shoppers, a growth industry worth US1.5bn worldwide. There are number a number of key stakeholders in the monitoring of services by mystery shoppers, namely the organization that is under surveillance, its management and staff, the organization employing mystery shoppers and finally the mystery shoppers themselves. So is it effective? The study described in this article focuses on what the experience of mystery shopping is like for mystery shoppers.

What is Mystery Shopping?

In his book Managing Service Quality, AM Wilson defined mystery shoppers as researchers who "act as customers or potential customers to monitor the quality of processess and procedures used in the delivery of a service." The aim in using mystery shoppers is to obtain facts rather than perceptions. This approach captures behaviours that are best observed in natural settings as opposed to interviews after the event or surveys. It uses a structured approach of checklists and codes to gather and measure information about service performance in every encounters.

There are many ways to measure customer satisfaction, but it is the quality of the collected data that differentiates this technique. Mystery shopping results can be used as a diagnostic tool to identify failing or weaknesses in procedures and processes, to encourage, develop and motivates staff and to assess an organization's competitiveness through benchmarking. The service-quality loop of service standards, employee performance, training and rewards can all be linked by using mystery shopper evaluations in a positive way.

However, there are disadvantages to using mystery shoppers. For example:
- Employee may view the evaluation process as threatening-management are "spying" on them.
- It reviews processess not their outcomes- the service was excellent but the meal could be inedible.
- If used as an ongoing method of evaluation it will result in constant need for mystery shoppers. This will inevitably use of staff time and impact on training and finances.
- Staff can become complacent about the level of service they provide after the novelty of being mystery shoppers wears off.
- Memory demands placed on assessors could affect the accuracy of the information obtained.
- It offers a snapshot of the service process, that may not be representative.
- It has been suggested that mystery shopping violates the principle of informed consent, as staff are not always aware that they are being observed.
- Staff may attempt to pale " spot the mystery shopper."
- Busy staff may spend time on something that is not a genuine enquiry.

These issues can affect the reliability of mystery shoppping as a technique to measure customer service. It has been questioned whether mystery shopping techniques can even be considered as methodologically sound ways of collecting data on performance. Specifically, the method and sample sizes have been criticised by some. In order for the data to be valid and reliable, the mystery shopping study has to be well desinged with particular attention on the data gathering process, the person doing the study (the mystery shoppers) and the reporting process.

The reliability is defined as the extent to which a given procedure produces similar results on other occassions given similar conditions-it is crucial if staff are to be rewarded on the basis of the results. Objective measurement is obviously possible in determining whether an activity did or did not take place, such as whether or not a customer was addressed by name, but to minimize subjectivity shoppers justify their rating selection by providing appropriate verbatim comments on their reports.It is also advisable to build up data overtime. For example, there may not be enough staff on the shop floor during lunch times or peak holiday periods.

A mystery shoppers feedback evaluation reports the perceptions of an individual rather the representative interaction and is influenced by the behaviour and appearance of both participants. The effects of individual differences between assessors can be reflected in assessments. For example, in department stores men tend to get priority over women according to Morrison et al. Also, the style of dress chosen and gender interaction can further influence service priority. Women are also more likely to provide more accurate mystery shopper reports than men and the age of the assessor can affect mystery shopper reports, with young adults being the most reliable.

The Study

To obtain the views of mystery shoppers themselves, a questionnaire was developed based on the issues arising from previous study. Participants were recruited via advertising the questionnaire as a web link on mystery shopping boards to attract as large sample as possible. In total 85 mystery shoppers participated in the study consisting of 22 men and 63 women. Participants came of Asia and from a range of social backgrounds, ranging from 21 to 60. The length of mystery shopping experience varied, the average being two to three years. The most experienced respondents had at least ten years' experience. Here we present some of the key findings of the study.

Deception and Guilt

Analysis indicated that mystery shoppers did not view adopting the role of a customer as deception, since staff have been aware that mystery shoppers would assess them. They felt it was vital to capture a true account of the customer experience and staff performance and this could only done covertly. They felt they were acting on behalf of the consumers generally. Another uncomfortable issue identified with feelings of guilt about being dishonest about circumstances including meployment and financial details. They also felt uncomfortable in giving negative feedback , especially a staff member was being helpful but not following company policy.

With regards to giving negative feedback on a mystery shop, 27 percent of respondents reported that they felt guilty, while 73 percent said they did not. Regarding uncertainty of what to include in reports , over half of respondents admitted feeling unsure. Almost 60 percent of respondents relied on checklists to recall the information required for completing the report. The vast majority of mystery shoppers in this sample (83 percent) believed that their report was a true reflection of the customer experience.

The results of an independent analysis found that women felt guiltier in giving negative feedback in comparison to men. Women relied on using checklists to recall information and felt that their report was a true reflection of customer experiences while men found it more difficult to meet deadlines and were unsure at times what to include.

Whatever the findings of mystery shoppers exercises , the results need to be fed back to employees in a positive manner. Management must be informed of what value has been obtained from mystery shopping programmes and any changes necessary to improve customer service. Results in some organizations can be linked to bonuses, awards and prizes. However, the mystery shoppers should also receive feedback on their performance and the quality of their reports from their employing organization.

Training The Mystery Shoppers

In the study, respondents were asked to rank qualities needed to be a good mystery shopper.
Quality Ranking
Ability to follow briefs 1
Objective 2
Work to Deadline 3
Ability to think on one's feet 4
Anonymous 5
Independent 6
Critical 7
Smart 8

A statistical test test, the non-parametric Mann-Whitney analysis, indicated that there was a significant difference regarding the qualities required to be a good mystery shopper depending on the experience of the respondent. Those with over three years' experience identified " able to think on your feet" as a key quality.

The information gained from mystery shoppers, if acted on , can reduce mistakes that lead to a loss of customers and loss of competitiveness. Therefore, recruitment of the right assessor is vital. They need to be typical of the organizations' customer base or they will be easily identified. AM Wilson argues that for most scenarios the adopted persona of a mystery shopper should be "neutral rather than aggressive" and recruits should be selected on the basis of having such personality.

Intensive training should be given so that they have a thorough understanding of their assignments. Training is important regarding briefing of the scenarios that are to be used, what observations are required and the type of questions the mystery shoppers has to ask. The type of training received can also impact the reliability of the results. Scenarios of training received can also impact the reliability of the results. Scenarios should then be rehearsed to ensure the service encounter is realistic. to reduce the chances of detection by staff. Training in data collection skills is important and shoppers should receive memory testing. As a result of the study, respondents indicated that online training is the most popular method. Those who have experienced mock msytery shops allowed variations in their experience. Those who had experienced in house specialist tarining rated it highly and those who were accompanied by an experienced assessor rated it as highly effective.

Respondents were asked to rank eight aspcts of the training they received from " most useful" to "least useful." These aspects included the completion of assessment reports, submission of assessment reports, background information on clients , preparation for visits, how to handle objections, common errors, using specialist equipment and how to avoid being spotted. The most useful aspects of training received included preparations for visits, completion of reports and common errors.


Nearly two thirds of respondents reported changes in staff behaviour during their visit, indicating that staff had been alerted to their mystery shops. Respondents noticed briefing notes/posters on staff notice boards, computers and shop windows announcing that a mystery shopper visit was due and for staff to be alert. Scores from previous visits were also prominently displayed and staff had been overheard discussing upcoming mystery shopping visits.

Some shoppers perceived that staff were waiting for "trigger question." The subsequent following of the checklist would mean that the mystery shopper was easily distingushible from "normal" customer. Some staff became nervous and gave too much information, while others used gestures to warn colleagues that they were dealing with a potential mystery shopper. Other behaviours included a " floor show" when asking for a receipt -a sure way of identifying oneself for claiming expenses for the visit-in eating establishment where staff would suddenly start cleaning surfaces and managers enquire how your meal is.

Nearly 65 percent of those surveyed reported that they were not treated differently due to their gender. However, a qualitative analysis indicated that female respondent reported being treated differently in ceratin establishments such as car parks where they ere told to return with their partner or husband.Female respondents also reported being treated differently by male staff in electrical and mobile phone outlets., where they were steered toward purchasing a simpler model they wanted. Credit card sales representatives were reported as tending to approach men rather than women.

How To Improve

Corporate managers rely on mystery shopper reports for monitoring the performance of their frontline people and processes and their obvious need for the data collected to be valid and reliable. The study identified both good and bad practices in the recruitment, training and use of mystery shoppers. Clearly bad practice needs to be eliminated. Responsibility for this is with mystery shopper employment agencies. These agencies need to follow some straightforward guidelines regarding recruitment and use of mystery shoppers, including:
- Recruit the right people and match people profile with the client brief.
- Train recruits in all areas of the job including different monitoring techniques, memory and observational skills, interviewing skils, avoiding detection, understanding assignments briefs, the use of checklists and completing reports.
- Have a three month probationary period for all new recruits, monitor their performance and give feedback.
- Give all mystery shoppers periodic performance appraisal.
- Ensure that client assignment briefs are rubost and will not lead to the easy identification of the mystery shopper. Be prepared to refuse to take on a clinet if the briefs are poor quality.

Overall, the findings from this study concur that the use of mystery customers, as a covert method of measuring performances, appears useful and effective. However, many companies that offer mystery shopping services are not selective when it comes to recruitment and offer no training to new recruits. This research suggests that the validity and reliability of data so gathered may therefore be questionable.

Further research that explores attitudes towars mystery shopping from other stakeholders' perspectives would add to the understanding of the effectiveness of mystery shopping as a measurement , diagnostic and improvement tool> employees' views would give another perspective, as would comparing the recruitment and operating methods of various mystery customer agency. A survey of the views of the management of organizations who would employ mystery shoppers would make the circle complete.

1 komento: