Internal communication measurement and evaluation is our Insight team’s bread and butter. For over 15 years, we’ve been assessing and advising on the state of organisational communication through one-off audits and regular measurement series.

But like many professionals, we’re starting to see changes in how we work thanks to the rise of AI. Already we’ve experienced how using AI tools can help us to organise, analyse and summarise data. It’s been particularly useful to speed up quantitative research analysis, which previously might have required hours spent poring over spreadsheets or grappling with formulae.

Indeed, we’re seeing how we might go further in the future in the wider world of research. It ranges from the intriguing, such as YouGov’s BrandVoices Index using AI-moderated interviewers (AIMIs), to the concerning, with some market research agencies reporting participants using AI-generated responses that are impairing their findings.

But could relying on AI for measurement and evaluation change the way we learn about our organisations and employees? Can we take any steps to prevent over-reliance on AI insights rather than our own expertise and experience?

Threatening organisational knowledge

 We’ve spoken to several IC professionals who are concerned that they will have less awareness of what’s going on with their employees if they rely on AI to tell them. They worry that they might end up having a poorer understanding of employees’ wants and needs if they rely on AI to filter data and tell them what’s important.

Research from leading universities such as MIT and Cornell backs these fears up, finding that use of AI leads to reduced knowledge retention and critical thinking.

We too have noticed that using AI can give you a more superficial understanding of research, versus reviewing it with your own brain.

In a recent audit, we trialled analysing focus group transcripts using both human brains and AI engines. We found that AI got about 60-70% of the way there, but tended to privilege frequency over meaning, and smooth out contradictory data to fit its own narrative.

We also got very different results when giving the same data and prompt to different AI platforms, as they prioritised different elements.

This means you can be left with neat themes that may appear right on the surface, but don’t contain the detailed insights that support interpretation and understanding.

It’s also limited to online data – what about the knowledge you gain from ‘walking the floor’ and being immersed in company culture? Where’s that in the AI mix?

And yes, you can use AI-driven sentiment analysis to review the content of your enterprise social network, but what about the things that employees don’t want to post on a public platform, and would only say in a confidential conversation?

We need to be careful that organisational insight doesn’t come to be seen as something you extract, rather than something you develop.