GUIDELINES FOR RADIO AUDIENCE RESEARCH

By
PIET SMIT
Chief Technical Officer

CONTENTS

There is a general danger that users of Market Research data will misinterpret results and thereby create confusion in the minds of interested parties. It is also possible for surveys to be conducted in such a manner that the results will not provide data that can be reliably used for their intended purpose. This scenario is especially true for Radio Audience Research and therefore SAARF has been asked to produce a summary of Best Demonstrated Research Practice that can provide authoritative guidelines to the marketing industry.

The need for such a document has increased in recent times. The Independent Broadcasting Authority has granted in excess of 80 Community and Commercial Radio licenses over the past three years, thereby creating the potential for a much more fragmented Radio audience than hitherto. Naturally each station wishes to determine the size and demographic characteristics of its audience and therefore it is vital that not only the credibility of SAARF's AMPS "currency" is maintained, but that the many newcomers to the Radio industry are made aware of the pitfalls of conducting inappropriate market research. These comments are written at a time when several Community Radio station owners have commissioned research that does not meet recommended technical criteria.

The AMPS Diary samples are designed to measure national and regional audiences down to the provincial level. Most of the Community stations' footprint areas are more localised, and consequently the AMPS Diary sample is too small to measure some of them. Before the licensing of Community Radio stations, SAARF had required a minimum of 200 AMPS Diaries in the reception area of a regional radio station. In an attempt to report information for as many Community stations as possible, this criterion has for the interim been lowered to a minimum of 140 diaries. A further criterion is that at least 40 respondents should claim any listening during the seven-days for which the AMPS Diary is kept, before demographic details of the audience can be published. To further assist these stations to provide at least a total audience estimate, it was decided that, if a station has a minimum of 10 seven-day listeners, a total audience figure will be reported.

The current structure of the AMPS Diary sample is sufficient to measure 46 of the 75 community stations on one Diary sample and 54 on two combined samples. However, few of them qualify for reporting. In the Oct/Nov 96 AMPS Diary, only four Community stations qualified for full reporting (40 or more seven-day listeners) while an additional 19 qualified for reporting of only a total audience figure (more than 10 seven-day listeners).

For the stations for which the AMPS Diary sample is too small, SAARF monitors the audience levels and if the information indicates that a station could probably be reported if the sample was large enough, and if it is economically justifiable to increase the sample, it is increased to qualify over two combined Diary samples. The sample for the Oct/Nov AMPS 96 Diary was for this reason increased in some areas.

Because audience estimates are essential for programme planning and for marketing of advertising time, SAARF understands that station owners will do their own research if the station is not reported in the AMPS Diary, or if they disagree with the AMPS Diary figures. Having years of experience in doing media audience research, SAARF's aim with this document is to provide guidelines which station owners can use to plan and conduct their own research.

SAARF also invites all station owners to consult with us before finalising their research brief.

The 7-day leave-behind and self-recording diary method (such as the AMPS Diaries) is used commonly around the World to estimate radio audiences. Many countries which do not have a peoplemeter system, also use the diary method to measure television audiences. The main difference between a diary-method and any other method, is that the diary monitors actual behaviour rather than recall of past behaviour.

Respondents are requested to record their listening for each station they have listened to, by quarter hour. Because the respondent has to fill in the diary himself/herself, it is essential that diary-keepers should be literate. Not withstanding the high incidence of illiteracy, particularly in rural areas of South Africa, a diary can be used. The lack of literacy can be overcome by using either a literate household member or a neighbour to assist the Diary respondent in completing the Diary. Appendix A shows examples of AMPS Diary pages.

2.1.1 Advantages of the 7- Day diary

• As mentioned earlier, the diary does not rely on recall of past behaviour. Recall errors and the effect of memory decay is reduced.
• Because respondents provide information for a seven-day period, weekly cumulative audiences and weekly reach and frequency can be calculated.
• Separate and comparable information is available for each day of the week, ie. Monday, Tuesday, through to Sunday.

2.1.2 Disadvantage of the 7-Day Diary

• The research relies on the cooperation of respondents to enter their listening correctly and promptly. However, in practice it is known that late entries and forgetting to enter data may occur. However, because the respondent is aware that he/she has to record the time and station, the impact is minimal.

To reduce the impact of atypical events on audience levels, it is advisable to balance the fieldwork across a number of weeks. The AMPS Diaries usually cover at least six-weeks, and from 1997 onwards, the four quarterly diaries will span the entire year, excluding the July and December/January school holidays.

Telephone research is an acceptable method for obtaining radio audience information, but it has certain limitations. Next to diaries, it is the most commonly used method. Particularly in South Africa, where the penetration of telephones is low in certain sectors of the population, it cannot be used for the entire population. Telephone interviewing relies on recall and it is not advisable to expect from the respondent accurate recall for more than one day (‘yesterday'). Experience has shown that respondents tend to rather recall habitual than actual behaviour.

2.2.1 Advantages of telephone interviews

• It is cheaper than personal interviewing. However, because of the limitation that recall is not advisable for a seven day period, surveys which use telephone interviewing should use balanced samples for each day of the week. Consequently, seven interviews are required to obtain seven-day information as can be done with one diary, that is samples need to be larger.
• A telephone survey can be completed in a relatively short time. However, to reduce the impact of atypical events, interviewing should be spread across a number of weeks to obtain an average figure as opposed to one week only.

2.2.2 Disadvantages of telephone interviewing

• The reliance on telephone penetration has already been pointed out, as well as the larger sample requirements.
• It is recommended that the duration of telephone interviews should not exceed 15 minutes.
• The lack of the opportunity to use prompt material is an important factor to keep in mind when telephone interviewing is done. Together with the reliance on recall, the researcher is obliged to rely on unprompted recall or to read out all the possible stations. In certain areas it would be almost impossible because of the large number of stations. Prompted recall has proved to be the more reliable method.

This research method entails mailing questionnaires to a sample of ‘potential' respondents. It is relatively inexpensive, assuming a sufficient response rate is obtained.

2.3.1 Advantages of Postal Surveys

• Possible interviewer biases do not exist.
• The geographic distribution of the sample does not affect the cost. Mail surveys can reach a geographically dispersed sample at the same cost that would apply to a smaller area.
• As has been mentioned already, mail surveys are an inexpensive way to collect data.

2.3.2 Disadvantages of Mail surveys

• The universe is limited to respondents with a reasonable degree of literacy.
• The most important shortcoming of mail surveys is a low response rate. Furthermore, it is usually almost impossible to determine the representativeness of the response because of possible differential response rates, even if information about the universe is available. There is no way to determine whether respondents and non-respondents are similar.
• Because the questionnaire is the only means of communication, misunderstandings can easily occur. The physical appearance of the questionnaire, its format, layout and the wording of the questions are extremely important.
• One is reliant on the efficiency of the postal service.
• A number of areas have no or only a rudimentary postal service.

In most of the queries which SAARF receives regarding audience levels, reference is made to the number of competition entrants, telephone calls or letters received, or people who made financial or other contributions.

All these results are based on a self-selected sample and cannot be grossed up or generalised to the universe. Such samples are statistically known as non-probability samples and conclusions can only be made for that sample. For instance, it is valid to conclude that people who entered for a competition were listening when the competition was announced. Whether they were listening at other times and on other days cannot be deduced from the fact that they have entered for the competition. The same applies for people who phoned or wrote to the station.

Because of the invalidity of such sources as the above to estimate audience sizes, this problem is discussed at almost every international media audience research conference. Even for more qualitative purposes such as views expressed regarding the quality or liking of programmes, the results cannot be generalised. It is again the views of some listeners and others may not agree. At best, letters, phone calls and competitions can be used to assist in the design of research.

Another source of concern is that audience estimates are sometimes based on unproven, unrelated or unclear assumptions. One such example is to obtain information about the population size, then assume that a certain percentage would listen to a station.

For any research for which a sample is used to estimate certain aspects of the population, it is essential that the universe which is researched be defined and described. Before this step has been executed, it is impossible to design a sample which represents that population.

Many sources of information on the South African population exist. The most detailed source is the latest Population Census of the Central Statistical Services. The most recent Census was done during October 1996 and top-line results will be available early in 1997. However, censuses are only conducted about every five years and because of population growth, deaths and migration, the information becomes outdated quickly. Institutions such as the Development Bank of Southern Africa (DBSA), the Human Sciences Research Council (HSRC), the Institute for Future Research (University of Stellenbosch), the Bureau of Market Research (BMR of Unisa), market research agencies, and many others regularly update population estimates by using mortality, fertility and migration trends. The BMR, for instance, annually publishes a report entitled: "Population Estimates for the RSA by Magisterial District", which provides information for each district by race. All AMPS surveys use the BMR population data for universe purposes.

It is recommended that the different available sources of estimates of the population be investigated, and the best one be selected to obtain details for a specific target market. Such information would make it possible to not only define the population which forms the target market, but also describe them by means of demographic and geographic characteristics. Such information will also make it possible to design a sample with high precision.

Apart from the size of the sample, the way in which it is designed is also extremely important. In this regard, it is important that a reliable and credible source or sources of population information be used to design the sample. When detailed information is available on the universe or population that forms the target market, it is possible to design a sample which can be grossed-up to estimate the listenership of the defined population.

There are a variety of standard sampling methods available which can be used for this purpose. In statistical terms they are referred to as probability samples. A probability sample implies that each person in the universe should have a fixed probability, which can also be calculated, to be included in the sample.

Other techniques, commonly referred to as non-probability sample designs, should be avoided when the aim of the research is to estimate, or quantify the audience. Quota sampling is one such a method.

The following probability sampling methods can be considered, either separately or in tandem.

4.2.1 Random sampling

A random sample is basically a sample which is selected by chance. The principle involved is the same as when competition entries are placed in a container and winners blindly drawn.

Statistical handbooks or computer programmes can be used to generate random numbers. If a sample of 1 000 has to be drawn from a universe of 10 000, then 1 000 numbers which fall between 1 and 10 000 are drawn after each unit in the population has been numbered. The units which each of these numbers represent are then identified and they form the sample.

Although this method is statistically regarded as a probability sampling technique, it does not guarantee perfect representativeness. The representativeness of the sample can be improved by using stratification or systematic sampling, which are described below.

4.2.2 Stratification

If it is known (or expected) that listening levels will or may be different among different sub-populations, for instance in different towns, cities, suburbs, or for different demographic sub-groups, eg. males and females or different age groups, then the incidence of such variables can be controlled to coincide with their incidence in the population. For example, if the geographical distribution of the target market of a city/town is known, the sample can be designed to represent these suburbs proportionately. Or, if 40% of the population is in the age group 16-24 years, the sample can be designed in such a way that 40% of the sample also falls in this group. This procedure is known as stratification.

More than one variable can be used simultaneously or interlaced to stratify the sample, in which case it is referred to as multistage-stratification, such as in the fictitious example below.

In the above example that covers four suburbs, 10% of the universe is living in Suburb A and is in the age group 16-24. When stratification by suburb and age is used, 10% of the sample will also be 16-24 years and living in Suburb A, and so on. More variables can be added to the above matrix, in which case the number of cells will increase. The number of cells is determined by the product of the number of categories which is used for the individual variables. In the above example, 4 suburbs and 4 age groups are used and provide a matrix with 4 x 4 = 16 cells.

4.2.3 Systematic sampling

Systematic sampling is another way to ensure that the sample is spread across the entire population, and not biased towards certain sub-populations and under-representing others.

If a systematic sample of 1 000 has to be drawn from a universe of 10 000, the following procedure will be followed:

Determine the sampling interval :

If 1 000 numbers have to be drawn from 10 000, every 10th number (universe ¸ sample) has to be used to ensure that the sample is spread evenly/systematically between 1 and 10 000. The sampling interval in this case is 10.

For the above example, the sampling interval is ten, which means that every 10th number must be selected, starting from a random starting point from 1 to 10. This point is chosen as is described under random sampling.

Selecting the sample :

Say that 7 is selected as the starting point, then numbers 7, 17, 27, 37, .......through to 9 997 will be used, resulting in 1 000, the intended sample.

4.2.4 Cluster sampling

Cluster sampling is usually used for economic reasons. If any of the previous three methods are used alone, particularly if a large geographic area has to be covered, the sampling points will be spread individually across the entire area which would increase the cost. Cluster sampling can be used in such cases to reduce the cost of travel.

Cluster sampling implies that fewer than the required points be selected and that more than one respondent be selected in the vicinity of each point. Such points are referred to as clusters.

However, it should be remembered that cluster sampling decreases the precision of the sample to the universe.

4.2.5 Disproportionate sampling

When it is important to obtain a large enough sample for separate analysis of one or more small sub-populations, for instance to report separately for a community radio station with a relatively small footprint area, that area can be over-sampled. Before reporting the results, such over-sampling should be down-weighted to reflect the correct proportion in the total population. To retain statistical validity, the level of over-sampling should not exceed a ratio of 2:1.

4.2.6 Sample size

There is a general misconception that the size of the sample is determined by the size of the universe and that large populations can only be researched by using large samples. Statistically, there is no direct relationship between the size of the universe and the size of the sample required to estimate certain aspects of that universe accurately. The size of the sample is determined by the following factors:

• The number of factors that can cause variation in the results. If it is, for instance, expected that everybody in the population will respond more or less the same, a small sample will accurately estimate this response. However, if it is expected that males and females will respond differently, a larger sample is required. If it is expected that males and females as well as people of different ages will behave differently, an even larger sample will be required.
• The level of detail in which the results will be reported. If the aim of the survey is only to report the behaviour of the total population, it can be done relatively accurately by using a small sample. If, however, it is important to report details of certain sub-populations separately, eg. geographic regions, demographic sub-groups such as gender and age, a much larger sample is required.
• The level of accuracy of the results which is required. Because there is an inversely proportional correlation between the size of the sample and the accuracy of the results, a large sample should provide more accurate results than a smaller one.

Given the above, the following minimum requirements are usually set:

• Each weighting cell should preferably contain at least 20 respondents.
• The ratio of weights should not be more than 2:1 if disproportionate sampling is used.
• Each reporting cell should have at least 40 respondents.

4.2.7 Substitution

For a variety of reasons, it always happens that not all the initially selected respondents will form part of the final sample. Some respondents will refuse to participate, while others might be difficult to contact. To compensate for this and to ensure that the final sample size and structure will be the same as the selected sample, substitution can be used. However, if the level of substitution is high and if it is not controlled properly, it can bias the results. The following requirements are usually set:

• Substitutes must be selected by using a probability sampling technique, similar to the procedures which were used to select the initial sample.
• Substitution should only be allowed when it is absolutely sure that the initially selected respondent cannot be included. This implies that every attempt should be made to include persons who are difficult to contact. Usually, three call-backs are required on different days of the week and at different times before substitution is allowed.
• When reporting the results, the level of substitution should be provided and, if it is high, the possible effect on the results should be indicated.

Finally, it is recommended that a statistician be consulted.

A questionnaire which is used to collect valid information is more than just a list of questions. Therefore, it is important that attention be given to not only the formulation of every question, but also to how they are arranged to ensure a logical, simple, understandable and unbiased interview. Furthermore, it is important that the questionnaire be tested and improved in a mini-sample before it is used in the final survey. The following guidelines can be used:

• Use the language of the target group. This does not only imply that the interview should be conducted in the respondent's language of preference, but also that the level of the language should be simple and easily understandable. Researchers frequently tend to communicate using marketing language rather than common language.
• Keep the questions - and the questionnaire - as short as possible. Avoid asking ‘nice to know' questions which do not really contribute to the aim of the survey.
• Bias through ‘leading' questions must be avoided. Even a statement such as "I am doing this research on behalf of ....(name of a radio station).." can bias the results.
• Care should be taken to avoid general research problems such as over-claiming on certain questions, under-claiming on others, the probability of a rotation effect, and many others. Consultation with experienced researchers in media audience research will help to reduce such possible biases.

Data collection is the most crucial aspect of all research, because mistakes which are made when collecting the information, very often cannot be corrected later. Therefore, it is important that:

1. Interviewers be properly selected during recruitment to ensure that they have the abilities which are required to do high quality work;

2. All selected interviewers should be trained properly in the basics of scientific data collecting;

3. Interviewers should be briefed in the application of every specific questionnaire, and pilot interviews should be done before they commence with the real interviews;

4. Strict control measures should be applied to ensure that respondent errors, interviewer mistakes, misunderstanding of the questions and situation errors are limited.

5. To ensure a high level of accuracy of the results, a minimum of 10% of all interviews are usually checked back. During the back-checking, both the selection of the sampling point and of the correct respondent must be checked, as well as that the information in the questionnaire has been recorded accurately. The check-backs should include the work of all interviewers.

The following audience measurements are used in the AMPS Diary reports:

The average ¼-hour audience is an arithmetic (the ordinary method) average across more than one ¼-hour. It is calculated by adding-up the audiences of the quarter hours to be reflected, and dividing by the number of quarter hours for which the numbers were added. This measure is usually used to estimate advertising channel audiences, which can be used in determining advertising rates. This figure is used to estimate the potential audience which would be reached if an advertisement is placed in that time slot.

The net audience reflects the number of different people who listened during a specified time period. In the advertising industry this is referred to as the ‘reach' or ‘coverage'. To calculate the net reach, persons who listened during two or more quarter hours during the time under consideration, are counted only once.

Gross audience :

The gross audience of a channel or programme is the sum of the relevant quarter hour audiences, irrespective of duplication of persons. The same person is counted once for each quarter hour that he/she listened during the channel or programme. If this figure is divided by the net audience, the average duration of listening is obtained.

Cumulative audience :

When audiences are calculated across more than one day, for instance Monday to Friday, it can either be done by calculating the arithmetic average, or by calculating a cumulative audience. The cumulative audience is the net audience of the first day in the calculation, plus new listeners on the other days. It is, in other words, the net or unduplicated audience across more than one day.

The above audience measures are related. For instance, if there is no flow of audience during a time-slot, in other words if all persons who listened at the beginning still listen at the end, and no new listeners enter, the average ¼-hour audience and the net audience will be the same. In this example, the average duration of listening will be the same as the duration of the time-slot under consideration. If the net audience is twice as large as the average ¼-hour figure, then the average listener has listened for only half of the total time.

When audience estimates are quoted, it is essential that the measure used also be mentioned. The average Monday to Friday and Cumulative Monday to Friday audiences usually differ. Similarly, the average daily audience will differ from the net daily audiences; the average 7-day and cumulative 7-day audiences will differ, etc. Without knowing which measure is referred to, it is impossible to correctly interpret the data.

Because the aim of media planning basically is to select those stations which reach as many of a specific target market a given number of times (called the frequency) in a certain time period (eg. seven days), research which does not provide a reach and frequency estimate will be of little value for selling advertising time.

When a seven-day diary is used, the daily reach can be calculated for every single day, and for any combination of days up to seven days. The frequency is the number of days on which a person listened at a specific time.

When telephone interviewing is done and recall of only ‘yesterday' listening is used, the frequency of listening across more than one day cannot be calculated.

All research for which samples are used to estimate the behaviour, attitudes, etc., of the population is subject to sampling (or statistical) errors. If probability samples such as the samples which are described in Paragraph 4.2 are used, the size (or margin) of the error can be calculated. The margin of error can be calculated for different levels of accuracy, but in most research, the 95% confidence level is used. If the margin of error is calculated at the 95% confidence level, it means that if 100 similar samples are used, the error would for 95 of them be within the relevant margin, whilst 5 could fall outside this figure.

Two variables determine the size of the margin of error, namely the size of the sample and the degree of unanimity of the response. The latter refers to the ratio of the proportion of the sample who responded positively (in this instance the listeners) and those who responded negatively (non-listeners).

The formula for calculating the error is:

s = p x (100-p) x 1.96
           n

Where:

s = Standard Error
p = Penetration (% who listened)
n = Sample Size

As in many other countries, the South African Market Research Industry strives for a high standard of research, as well as to protect the interests of the different stake-holders. Most of the leading researchers are members of the Southern African Market Research Association (SAMRA). SAMRA is a professional association and has a code of conduct, to which its members subscribe. The stakeholders are the general public who provide the information, the client who pays for it, and the researcher.

Furthermore, the Association of Market Research Organisations (AMRO), is an umbrella body to which many of the large research providers belong. This body also sets standards to which its members subscribe.

Individual researchers who are members of SAMRA and research providers who are members of AMRO are obliged to the best of their ability, to ensure that the research practitioner(s) with which they are associated and the people conducting research on their behalf adhere to this Code of Conduct. More information on SAMRA can be obtained from them at :

P O Box 91879 Tel: (011) 482-1419
AUCKLAND PARK Fax: (011) 482-4609
2006

The South African Advertising Research Foundation was founded in 1974 as a non-profit industry body. SAARF was formed because of a need in the marketing and advertising communities for a comprehensive, unbiased, reliable, regular and technically excellent media audience survey. Its purpose is to provide information about the population's use of the media, products, and services so as to enable reliable targeting for advertising purposes. The data are in such a format that it is used, among others, for the buying and selling of advertising time and space in the media and for strategic editorial and programme planning.

Over the years, the All Media and Products Surveys (AMPS) have established themselves as reliable, valid and credible research vehicles. Apart from commissioning the AMPS surveys, SAARF also assists media owners, advertisers and advertising agencies in a series of other areas such as training in the use of AMPS data.

SAARF's mission is to serve its members, and other interested persons and bodies are invited to liaise with SAARF about any aspect related to media audience, market and marketing and advertising research.

SAARF will issue a press release when new AMPS Diary results are published. Such press releases will comment on radio listening levels and listening trends in general terms, not on the performance of specific stations. Station owners who wish to issue press releases or newsletters with information about their specific stations(s) are invited to consult with SAARF in this regard to ensure that the information is interpreted correctly.

The members of the SAARF Radio Research Committee, who are representative of the radio, advertising and marketing industries, have also committed themselves to assist SAARF in ensuring that the SAARF radio audience data are used correctly.

a. AMPS Diary Technical Report, South African Advertising Research Foundation (SAARF, Johannesburg, 1994.

b. ARF Guidelines Handbook, Advertising Research Foundation (ARF), New York, 1990.

c. Description of Methodology, The Arbitron Company, USA, 1992.

d. Minimum Standards for Electronic Media Ratings Research, Electronic Media Ratings Council, New York, 1994.

e. SAMRA Yearbook, Southern African Market Research Association, Johannesburg, 1996.

f. Towards Harmonization of International Radio Audience Measurement, Conference of International Broadcasters' Audience Research Service (CIBAR) in collaboration with the European Broadcasting Union, Geneva, 1995.

g. 1995 Report on Radio and Television Audience Measurement in Europe, European Society of Opinion and Marketing Research (ESOMAR), Amsterdam, 1995.

The above publications can be consulted in the SAARF Library.