Copilot: A Great Tool, But Not a Magic Bullet for Knowledge Management

Why Copilot is not enough for knowledge management.

A recent post from my friend Marc Anderson was a thought-provoking article on knowledge management. Based on Viva Topics going away in February of 2025 (See Frequently asked questions about changes coming to Topics - Microsoft Topics | Microsoft Learn), he argues that Copilot, the AI-powered assistant for Microsoft 365, is not a substitute for proper knowledge management practices. Unlike some of his views on JavaScript, I agree with him on his knowledge management position. I want to share some of my thoughts on why Copilot is not enough for knowledge management, especially in the corporate environment.

How Copilot works and what are its limitations

Copilot is a tool that helps users create and edit documents, presentations, and spreadsheets by suggesting content, design, and data based on natural language prompts. It uses a Large Language Model to generate text, charts, tables, images, and formulas for a variety of topics and domains.

Large Language Models (LLMs) are trained on data up to a certain date and time, for example GPT 4 was up to approximately April 2023. So, when asking general questions it can be outdated or wrong. For example, I asked Copilot the age old question who is better Manchester United or Liverpool, part of the response included:

• As of now, Manchester United is performing well in the Premier League, sitting at the top of the table.
• Liverpool, despite the loss to United earlier in the season, remains a strong contender.

Not true! As of today 3/1/2024, Liverpool is at the top and Manchester United is in 6th place. So clearly on general topics Copilot cannot be trusted 100% - but we knew this already.

Another example, I asked Copilot for Work , “Who is our largest client?" The first time it responded with:

• Based on emails and references in documents its “Spyglass MTG” – clearly wrong
• A second attempt result in the correct answer based on a 1 line in chat between me and our data Practice lead, Bill Richard in August of last year, which at the time was his guess.

So again, in the work environment Copilot cannot be trusted 100% - we knew that also.

Going forward

Copilot is a great tool for content and data generation, but it is not a magic bullet for knowledge management. It does not replace the need for proper documentation, collaboration, and governance of the content and the data, with review by subject matter experts to sort the wheat from the chaff.

I hope that Microsoft will be creating a new product under the Copilot banner which is being used for all their AI related technology. The LLMs underneath Copilot have great ability and potential to summarize content and identify Topics covered – but we still need to effectively curate the information generated.