What is a pulse survey, and when do short surveys make sense?

What is a pulse survey?

Pulse surveys are short questionnaires sent at regular intervals. A recent interview study of how organisations use them describes them as «short, frequent questionnaires» and notes «a notable lack of research examining these practices». Among the organisations in the study, the frequency varied from weekly to every three months.

That sets the pulse survey apart from the main mapping, which measures all the psychosocial factors thoroughly and is the basis for the risk assessment. The pulse is a follow-up, not a new mapping.

What do the rules say?

The Norwegian rules do not use the term pulse survey, and they do not set how often you should measure. But they set requirements that a short survey can help you meet:

On measures, Arbeidstilsynet writes that to find out whether they have the desired effect, they must be evaluated, and that measures that do not work as planned must be adjusted, replaced or supplemented. A short survey on the topics where measures are under way is one way to evaluate. It does not give the whole answer, but it shows whether employees notice a difference.

When do short surveys make sense?

• After a measure. If you measure the same statements that led to the measure, you see whether the figures move.
• Upon changes. A reorganisation, new management or new systems can affect the risk. A short survey on the topics concerned can show whether something is happening, pending a more thorough mapping.
• To keep the work alive. A short survey between main mappings reminds everyone that the working environment is a topic all year – provided something happens with the answers.

The interview study concludes that «pulse surveys should not replace annual comprehensive risk assessment surveys; rather, they should serve as a complementary tool». It also found that pulse survey processes «were generally not integrated with psychosocial risk management practices». That link is what makes the measurement useful.

What research says about measurement error

Short surveys come at a price: the fewer the statements and the fewer who answer, the less certain the figure.

• Single items are less reliable than scales. A review of single-item measures in organisational research states that «multiple item measures have better psychometric properties than single-item measures», though «in some circumstances single-item measures can provide useful information». A classic study estimated the minimum reliability of a single overall job satisfaction item at between .45 and .69, where 1 is perfect.
• Small groups give unstable averages. How reliable a group average is depends, among other things, on how many are in the group. In a small group, one answer can move the figure a lot.
• Random variation can look like change. Regression to the mean means that a group that scored unusually low on one occasion often scores closer to the average next time – even without a measure. The effect grows with measurement error, and when only the groups that scored lowest are followed up.

A study of pulse surveys in one healthcare organisation found that they «provided reliable estimates of overall engagement, but scores remained stable across 8 months», with practically no differences between groups or over time despite high participation. It concluded that pulse surveys «may be adequate for estimating overall employee sentiment but not useful for detecting change over time or differences between groups».

That does not make short surveys worthless, but small shifts should not be read as change. Look for clear and lasting movements, and hold them up against what you know from conversations, safety inspections and incident reports.

Compare like with like

A pulse survey should be compared with the previous pulse survey, with the same statements and the same groups. If you change the statements, the groups or the answer scale, a change in the figures may be due to the measurement and not the working environment. The same applies when a short pulse survey is compared with the full main mapping. See how often to measure the working environment.

Willingness to answer falls when nothing happens

Frequent surveys can wear down the willingness to answer. A study among students found that «administering multiple surveys in one academic year can significantly suppress response rates in later surveys». A workplace is not a university, but the mechanism is easy to recognise.

What seems to matter most is whether the answers are used. In the interview study, employees’ willingness to respond «was highly dependent on their perception that their responses from pulse surveys were used meaningfully», and failure to follow up could lead to disengagement, survey fatigue and erosion of trust. An older study of employee surveys found that those who did not answer more often held negative beliefs about how the organisation handles survey data – for example, that it does not act on it.

• Say what has been done since the last survey before you send the next.
• Keep the survey short, and ask only about the topics where something is happening.
• Do not measure more often than you can follow up.

Anonymity in small surveys

Short surveys are often sent to small groups, and anonymity is then more fragile. The authors of the interview study write that calculating a group average on single items for a small number of respondents is problematic for reliability and for detecting change, and that «reporting results on small groups of as few as 3–5 people is not an acceptable practice from an ethical or a methodological point of view».

A practical starting point is to show no group with fewer than five answers, and to look at the figures for the whole organisation when groups are smaller. The Norwegian central government’s employee survey (MUST) uses the same threshold for a report, and shows the distribution across answer options only from ten answers. Read more about what an anonymous employee survey means.

Pulse survey and main mapping

A measure being carried out does not mean it works. A systematic review of organisational-level interventions at work found that «About half of the studies (19) reported significant effects», and that «Success rates were higher among more comprehensive interventions tackling material, organisational and work-time related conditions simultaneously». That is an argument for measuring whether measures are felt, not only whether they were carried out. See from results to measures.

Checklist

1. Tie the pulse survey to measures or changes, not just to the calendar.
2. Use the same statements as in the main mapping for the topics you follow up.
3. Keep it short: few topics, few statements.
4. Show no group with fewer than five answers.
5. Read small shifts with caution.
6. Say what has happened before the next survey, and involve the safety representative in the plan.