Internal Knowledge Assistant

Context

FieldPulse’s product knowledge lives in three places: a public help center, an internal Confluence wiki, and the heads of the people who have been there longest. Support, sales, and implementation all need it constantly, and the third source does not scale — every answered question is an interrupted colleague.

This is a Slack assistant that answers those questions. You mention it, it searches the documentation, and it replies in the thread with citations.

Problem

The bar for an internal knowledge bot is deceptively high. Nobody has to use it. If it is wrong twice in a row, people stop asking and go back to interrupting each other, and you never get them back.

So the failure mode that matters is not “cannot answer.” It is “answers confidently and wrongly.” A bot that says I don’t know, here’s who does stays useful. A bot that invents a plausible-sounding feature description poisons the well.

Approach

Version one was a low-code workflow: an n8n canvas wiring together a vector store, an embedding model, a reranker, and an LLM. It worked, and it ran in production for months.

It also accumulated obligations. Embeddings drift out of sync with documentation that changes weekly. The reranker had quota limits. Model swaps upstream changed behavior without warning. I was spending more time keeping retrieval fresh than improving answers.

The alternative was to stop managing a copy of the documentation and query the source directly through MCP. Rather than argue about it, I ran a bakeoff: seven representative questions through both architectures, scored side by side. It came out close to a tie — two wins each and three draws — with a clear pattern underneath. MCP won on freshness and never served a stale answer. The custom pipeline won where internal wiki content mattered, because the MCP endpoint only covered the public help center.

That result wrote the v2 spec: go MCP-based, but keep a direct path to Confluence.

Architecture

Version two is a hand-written Node application on Railway, connected to Slack over a socket rather than a public endpoint. A question runs through five stages:

1. 1

   Intent gate

   A small fast model decides whether this is actually a question. Banter gets banter back and exits early rather than triggering a documentation search.

2. 2

   Parallel retrieval and classification

   The help center (via MCP) and the internal wiki (via a scoped Confluence query) are searched at the same time as a classifier tags the question against a taxonomy of seven categories and seventy-nine sub-categories. Each call has its own timeout and fails independently — one dead backend never takes down the answer.

3. 3

   Synthesis

   The retrieved material is synthesized into one answer with tagged citations. The prompt forces a self-critique pass and requires contradictions to be surfaced rather than smoothed over: if the help center and the wiki disagree, the answer says so and cites both.

4. 4

   Failure detection

   A separate model call scores whether retrieval actually answered the question, sorting failures into no sources, partial sources, contradictory, or off-topic.

5. 5

   Routing and escalation

   Citations are turned into real Slack links by deterministic post-processing, never by the model, so a citation link is never broken. If the failure detector is confident the answer failed, the question is escalated to the product owner for that category.

Two details I would point at in an interview:

The classification taxonomy and the escalation routing map share their keys. The classifier’s output is the routing decision. There is no translation layer to drift out of sync.

Each pipeline stage reads its model and prompt from the database, cached briefly. Changing which model handles synthesis is a row update, not a deploy — which matters when a provider changes a model’s behavior underneath you on a Tuesday.

Auto-escalation is deliberately hard to trigger: the failure detector must be confident above a threshold, the question must not be low-signal, and the synthesis must not have been confident. That threshold started at 0.75 and moved to 0.85 after a week of watching it in production, and suppressed escalations log their reason so the gate itself can be tuned.

The bot has a companion admin console — a question log with escalation trends and repeat-question detection, a knowledge-health view showing coverage gaps and stale content, and a configuration surface for the prompt and model slots the bot reads live.

Outcome

Over eleven weeks the assistant answered 1,090 questions for about 32 people a week — roughly 99 questions a week that nobody had to interrupt a colleague to ask.

The number I care about more is that people kept asking. An internal tool nobody is required to use gets exactly as much traffic as it earns, and this one held about 32 askers a week across three teams for the whole period.

Two decisions did most of that work. Citations meant any answer could be checked in one click instead of taken on faith, so being occasionally wrong did not poison the whole tool. And escalation meant a question it could not answer went to the person who actually owned that area instead of dead-ending — the failure case still moved someone forward, which is the difference between a bot people tolerate and one they keep opening.