Contact
The Halo Works Ltd

+44(0)1276 477445
or email us

Data Analysis

Weighting

We use two types of weighting; quantity weighting and respondent weighting. In both cases an answer may not always be counted as 1, when tables are incremented.

Quantity weight refers to weighting by answer to a question. For instance, if the following question was used to weight your tables "How many pints of beer did you drink last week?" the base would not be the number of respondents, but the number of pints of beer consumed.

Respondent weighting is often known as target weighting. In respondent weighting a respondent may count as more or less than 1, when weighted tables are calculated. A respondent weight is normally applied to tables in order to gross up one particular type of respondent.

For example, if you had interviewed 40 males and 60 females and you needed to have interviewed 50 males and 50 females, it would be possible to apply a respondent weight so the males are counted as 1.25 and the females .83, instead of 1, in table calculations (providing you have an entry on your questionnaire which identifies the sex of the respondent).

Mean scores, averages and other statistics

There are two sorts of questions that will produce various statistics when used as the rows of a table: quantities, usually Integer or Weight questions. For example, salary, height or volume. For these questions when used as the rows of the table either a full list of values can be requested, or the statistics only.

Scored questions, usually rating scales to which scores (analysis values) have been assigned.

For example:

  • Like a lot (2)
  • Like a little (1)
  • Indifferent (0)
  • Dislike a little (-1)
  • Dislike a lot (-2)

When using these types of entries on your tables the following statistics may be produced:

  • Base for statistics
  • Sum of values
  • Sum of squares of values
  • Average or
  • Standard deviation
  • Standard error
For tables with all rows listed the following can also be requested
  • Error variance
  • Mean score divided by standard error
  • Medians
  • Quartiles to Deciles
  • Maximum value
  • Minimum value
  • Modal value

Significance testing, confidence limits and probabilities

Question: Is this figure significant? Reply: Compared to what?

When discussing 95% or 99% significance it is important to bear in mind that a figure from our survey (a percentage or a mean score) can only be significantly different when it is compared with some other figure from our survey, or a known value (from another source). QPSMR has many ways of marking a figure as 95% significant but what does this mean?

Let us assume we have a product rating mean score for two subsets of data. Men have a mean of 1.56 and Women have a mean of 1.34. When compared, QPSMR flags Men as having a significantly higher mean at the 95% level than Women.

What this means is that there is only a 1 in 20 chance a difference this large (0.22) would be obtained if the two subsets had been drawn from the same universe. To put this another way, we can be 95% confident there is a real difference between Men and Women when rating the product.

The use of the word "significance", however, in this context is unfortunate because it is not the same as the normal use of the word. On a rating scale of 1 to 5 we might have a mean score difference of 0.01 that is "significant" and a difference of 2.00 that is not "significant". The word "confidence" is more descriptive.

So when a figure is marked at 99% it means it is very unlikely (1 in 100) that there is no difference between the two samples being looked at. The reverse is not true; we cannot say that if a figure is not marked then there is no difference - it may just be that the sample sizes are not large enough for us to be confident.

Following many of the tests are a probability figure. This is a way of showing the confidence level of the statistic as a value between 0.0000 and 1.0000, so small figures are significant and large ones are not. A 95% significance will show as a probability of 0.0500 or less, and 99% significance as 0.0100 or less.

In general, to convert a probability to a significance level, subtract the probability from 1.0000 and use the first two digits. For example, a probability of 0.1278 becomes 0.8722 which is 87.22% significant.

What is compared with what?

Only columns that appear in the same break will be tested against each other. The majority of the tests compare subsets of the data shown as the columns on a table. As an example we will use six columns:

  • Total
  • Male
  • Female
  • Young
  • Middle
  • Old

The standard method is to test whether any particular subset (column) of the data is different from the remainder of the sample. In this way it will highlight any "interesting" columns for further investigation.

In our example, Males are tested against Females, Young against Middle and Old together, Middle against Young and Old together, and also Old against Young and Middle together. The purpose of testing in this way is to highlight columns (breakdowns) which are "different".

This method of testing includes the total sample in every test and is therefore, more likely to detect differences than other types of comparison. Because the "rest of the data" is calculated by subtracting the column being investigated from the total column, this standard method will only work on tables with a total column. Cells are marked with one asterisk (*) for 95% significance and two (**) for 99%

The second method for comparison is to label each column with an identifier, for example

Demographics^Area\North (a), Mid (b), South (c), Not stated, Sex\Male (m), Female (f)

The identifiers must be included at the end of the individual labels (preferably on a new line) enclosed within parentheses. In this method individual pairs of columns are compared.

You will notice that headers (Area and Sex) and an overheader (Demographics) have been used. In this example, each area is compared with the other two, and the appropriate letter markers placed against the cell, if significant differences are found.

For example if "Mid" is found to be different and higher to both of the other areas, with a significance of 90% (as set by selecting format 90) it will be marked with lower case letters "ac" after the value. If "Mid" was found to be different and higher to both of the other areas, with a significance of 95% (as set by selecting format 95) it will be marked with upper case letters "AC" after the value.

The "Not stated" column will not be tested because it does not have an identifier. Note Males and Females will be compared, so that if Females are different and higher the letter "m" (lower case, or upper case, depending on the significance level) will be placed next to the cell.

If Format SHG 2 (test with overheaders) is used then all five identified columns will be compared with each other

Distribution t-tests

Where a table has a list of items (for example "Likes") down the side Format SIG can be used to mark cells.

Each row of the table is treated separately and cells are marked depending on whether the percentage is different to the same row in the column it is being compared with.

You can choose to use the combined variance with SIG1, or separate variances with SIG2.

Mean or average t-tests

Where a table has rows from which a mean score or average is produced (and a total column is included) our software will automatically calculate t-tests and mark the averages as appropriate.

  • As default, a ** (2 star/asterisk) marker means that the column marked is 99% significantly different to the rest of the sample (the total column, minus that individual column).
  • A * (single star/asterisk) marker, indicates that the column marked is 95% significantly different to the rest of the sample.

You may choose whether to use the combined variance or separate variances

You may request a complete grid of t-test values, comparing every column with every other column. This grid can be reduced to testing only within each group of columns.

If a respondent rates two or more products, you could produce a banked table with the products as the breakdown and use column identifiers to compare them. A more accurate method is to subtract the score for one product from the score for the other.

For example, with scores 1 to 5 the relative score will be between –4 and +4. This relative score can then be tabulated and will give a t-test value comparing the mean score with the expected fixed value of 0.0. If the expected value for a mean is zero, and if format MSE is >1.96, this is 95% significant and >2.57 is 99%.

F-tests

An F-test can be performed on all of the columns within each group. This test is used to establish whether the group of columns (for example - Area) affects the row mean or average, without looking at all of the individual pairs of columns.

Other tests

For table rows which are assumed to be in order (for example - Small, Medium and Large) but you do not wish to attach score values to the rows, two non-parametric tests can be applied:

  • Kolmogorov-Smirnoff
  • Mann-Whitney-Wilcoxon

For tables with any distribution down the side:

  • Chi-squared
Source of this text: QPSMR Knowledge Base

 

  • "The tool has provided my company with a unique insight in to how we are viewed"

    Major outsource provider

  • "I couldn't believe the accuracy of results and the amazing insight I got"

    Global airline services business

  • "The 'External' survey should be absolute gold dust to a company"

  • "Halo should be a fundamental part of every company's toolkit"

  • "All results were presented in a very clear and unambiguous way"

  • "The Halo Works team have been fantastic to work with"

  • "The survey challenged my perceptions as to whether we were using the right measures to tell us whether we are doing a good job or not."

    Global consulting business

  • "I am and will continue to, recommending the survey to all my clients and I happily commend it to you, the reader."

    Global customer management and strategic consultancy

  • "The process has made the team stronger as each person can give their opinions anonymously."

  • "We now have a series of action points that we can measure the effectiveness of."

  • "The people at Halo were supportive and tremendously helpful."

  • "Halo is now part of our annual review."

  • "I cannot believe after twenty months, countless issues, consultants, workshops and brainstorms that we have finally solved this problem. Now thanks to the Halo Process we have made huge progress in just four weeks "

    Rydon Group CEO

  • “The Halo Works team hit every deadline, however tight, and the end results and workshop was great fun and we are still using the insights one year on.”

  • “The work The Halo Works did for us was so much better than anything else we have done in the past.”

    Construction Business

  • “If you want an excellent research agency, choose The Halo Works as they are the best there is.”

  • “The Halo Works are the only research company I recommend because they are consistently excellent.”

  • “I have always found The Halo Works to be very enthusiastic and innovative. I would not hesitate in using them again when needed.”

  • “I find them very professional with a good understanding of their and our business.”

    Airline handling business

  • “They are totally reliable and with real integrity. This alongside their excellent techniques makes them very good to have your team.”

  • “They are such a strong part of our team I sometimes forget they don't work for us.”

  • "Research and consultation is used throughout the authority to deliver an excellent service. The Halo Works have helped to make this happen, especially with the professional management of our Citizens' Panel"

    Chichester District Council

  • "We have worked closely with The Halo Works for a number of years and we have found them to be very supportive and professional in their advice and in the many consultation exercises they have helped us with".

    Strategy & Communications Manager- Rushmoor Borough Council

  • "The Halo Works have developed a good relationship with our staff and Councillors and have a sound understanding of the Council, the area and of local issues. These have proved helpful in their analysis, presentation and intepretation of our survey results".

    Rushmoor Borough Council