Factors That Can Have an Effect on Survey Results:

A Look at the RAMS Survey

It is well-known that there are an enormous number of factors that can affect the outcome of a survey.

Some of these factors are time sensitive and may only have an influence if it coincides with the fieldwork period of the survey. Others are as a result of planned or unplanned changes to the survey such as changes in sampling, weighting, the scope of the survey, a change in contractors or a change in methodology.

Some of the most difficult ones to predict or to quantify are those that are completely extraneous to the survey such as macro economic factors on the one hand, and on the opposite side of the spectrum, changes in e.g. some of the individual media that are being measured. However it is important that they should be considered when looking at audience figures.

So let’s have a look at some of those factors that can have an effect on our radio survey RAMS and then look at some of the factors currently at play.

In the recently released RAMS survey, a number of changes will be noticed:

1.  Annual Universe Update

Firstly at this time of the year, the adult universe for the survey is updated using the population figures as prepared by demographers at the Bureau of Market Research (BMR) of UNISA.

The BMR population update is done for us on an annual basis because the census figures which forms the base of all quantitative research in the country is done only once every 10 years and those of the previous census conducted in 2001 are now outdated.

In order to stay current as far as this important aspect is concerned, results from the 2001 census plus other sources of information such as the number of babies born in the past years, deaths, emigration, the Aids pandemic, migration, etc. are taken and complex models are then used to determine the best estimate of the South African population nationally, provincially as well as down to magisterial level.

Naturally these changes are not uniform over the whole of South Africa and can therefore lead to greater changes in certain geographical areas than others. For example, it can also lead to changes in the population profile of certain areas (e.g. those areas most affected by Aids).

All of this can naturally have an effect on audiences of certain media.

2.  Annual Small Town/Village & Rural Update

It must be remembered that RAMS is a truly national survey and is conducted from the Cape to the Limpopo and from the east coast to the west.

However, due to the cost of doing research in the far-flung rural areas, data from these areas are collected throughout the year in such a way that we can update the information once or twice a year depending on funding.

Over the past 4 years or so SAARF was in the unhappy situation where these updates took place only once a year and because of this, noticeable changes could be observed after every update as a whole year’s changes went into the one update.

From 2009 we will revert to six-monthly updates which will ensure more up-to-date data as well as less noticeable stepwise changes when this happens. In future these updates will happen in about April and again in October of every year.

The RAMS update that took place in October 2009 was hopefully the last of the once a year updates.

3.  Changes in Methodology

From time to time changes in methodology are made to ensure that we stay current with developments, not only in research, but also in the markets that we operate in. In addition, the needs of our stakeholders are continuously changing and this must be taken into account as well.

However, because such changes can have the effect that trend lines are broken, SAARF tries to do methodological changes only when necessary. The previous big change in the RAMS methodology was approximately 5 years ago, when we changed from 2 releases per year to 6 and introduced flooding in large urban areas. This year we have introduced national flooding (thus not just in large urban areas but everywhere) and at the same time we have reduced the interview age for adults to 15+.

From previous experience we know that flooding leads to a slight once off decline in ratings (4% in time spent listening and incidence of listening in this October release) but has (especially in developing countries) the enormous advantage of pushing up the sample to way more diaries than the industry would have been able to afford. The current change to national flooding has led to an increase in the number of diaries by about 34% for very little extra cost.

4.  Change From Two Contractors To One.

Changes in contractors always lead to changes in the outcome of a media audience survey. Unfortunately these changes are very difficult to predict and depending on research design, not always easy to pinpoint.

Worldwide this is one of those aspects that countries battle with as frequent changes in contractors can play havoc with currency surveys which should by nature be stable and credible.

SAARF has changed back to the use of one contractor from 2009 due to the cost of having two contractors and this could have had an influence on results of the RAMS survey albeit a small one.

In the AMPS audit conducted by international media research expert Erhard Meier during 2007, he refers to the so-called “contractor effect” and says, “In my assessment, that this effect is not present with the current AMPS is likely to be the result of a great effort, which all parties involved seem to have put into providing a high quality survey.”

Remember also that the RAMS diary is a self-completion diary piggybacked on AMPS and thus interviewer effects for RAMS are minimized.

As the current contractor has been involved in the RAMS survey since its inception, and has proven through the years to be world-class, it is not expected that this change would be detrimental to the quality of the survey.

5.  Unusual Events during Fieldwork Period Such As Sport, Elections, Etc.

During the fieldwork for the 2009 releases of RAMS, numerous events happened which could have had an effect on the consumption of media and products.

In addition, seasonal effects are also sometimes visible when analysing radio consumption throughout the year.

6.  Competing Media and Other Activities

One can never look at a medium such as radio in isolation.

Other media and especially television can have a large influence on radio listening. During major sporting events for example, one can very often see an increase in television viewing and a concomitant reduction in radio listening.

Over the past year or so, there have been large increases in television viewing while time spent on radio listening has been reducing.

However, the large increase in television viewing especially in small urban/rural areas (according to both AMPS and TAMS) is not just due to special events, but is also driven by increased TV ownership.

It is also clear that the use of computers and internet as well as other electronic devices are consuming more and more time of listeners and will eventually have an effect on media consumption.

7.  Extraneous Factors

South Africa and the world are in the throes of an economic downturn that is much stronger than anticipated and many stakeholders do not quite understand the extent of this dilemma.

Over the past year or so, about 700 000 South Africans lost their jobs. Household income is under pressure for the first time in many years and the effect of this will surely impact on media and product consumption.

Unfortunately, the bad news is that this is not over yet and that it may take longer than expected for the recovery to kick in.

In a communication received from Professor Carel van Aardt of the BMR he says that according to the Quarterly Labour Force Survey conducted by Stats South Africa, nearly half a million jobs were lost from the 2^nd to the 3^rd quarter of this year.

He says that “If you add that to the nearly half a million who lost their jobs during the first two quarters of this year and the 200 000 jobs lost during 2008 we are looking at about 1.2 million job losses since the onset of the recession.

If that is true, we are in much more dire straits than many analysts estimated/predicted. The 700 000 job losses during the period June 2008 to June 2009 are in line with the recently published AMPS 2009 figures. But a further half a million jobs during the third quarter was unexpected.

Its effect on personal and household incomes, on household consumption expenditure and GDP growth in SA (with household consumption expenditure explaining about two thirds of GDP) is going to be huge.”

8. Summary

As can be seen from the above, it is not always easy to pinpoint those factors that could have had an effect on media audience movements.

Neither is it easy to decide to what extent (if any) these factors individually influenced audience movements.

In studying media audience figures the following questions must be asked:

1.  What did we change as far as editorial/programming is concerned.

2.  Did we move programmes to other time slots?

3.  Did we introduce new programmes?

4.  Did we change announcers or DJ’s?

5.  Are the audience movements statistically significant?

     (Go to http://www.saarf.co.za/calculatesignificant/Significant%20Difference%20User's%20Guide.pdf to learn more about and calculate significant differences).

6.  What percentage of my listeners falls outside of the large urban areas and could have been affected by the small town/village/rural update?

7.  Did the universe update have any significant effect in my region?

8.  What about sporting events, elections, etc. that could have affected my audience behaviour?

9.  Did TV viewing increase?

10. Were there any changes to the research that could have influenced the outcome of the survey?

11. What about extraneous factors such as the current economic downturn and accompanying job losses?

END