blog / Decision guides

How to test a WhatsApp AI chatbot before you buy

heysteff is the AI platform for customer service and sales over WhatsApp, Instagram, Messenger, Gmail and Shopify; Steff is the AI agent that runs it. Testing a WhatsApp AI chatbot before you buy is the only honest way to know whether it fits your business, because a sales demo always shows the easy questions. This guide explains how to set up a short trial with your own conversations, what to watch while it runs and how to make the call at the end without being swayed by a good first impression.

Why a demo isn't enough to pick a WhatsApp AI chatbot

A demo isn't enough because it's built to make the product look good, not to show you where it breaks. The vendor picks the questions, the sample catalog is tidy, and nobody types with typos, sends a one-minute voice note or asks three things in one message.

An AI chatbot (or AI agent) is software that reads a customer's message, works out what they want and writes a reply using your business information, instead of following a fixed menu of options. That flexibility is exactly what you need to test: a rule-based bot fails predictably, while an AI one can answer ninety messages well and confidently get the next one wrong. If the difference isn't clear yet, start with how to choose between a rule-based chatbot and an AI agent.

So the decision shouldn't happen in a thirty-minute call. It should happen after you've watched the agent work with your customers, your products and your policies for a few days.

What to prepare before the free trial starts

Before you start any free trial, have three things ready: a set of real questions, your policies in writing and a definition of what “working” means. Without them, the trial turns into poking at the bot for a while and walking away with a feeling.

The question set comes from your own WhatsApp. Go through the last few weeks of chats and copy 30 to 50 messages exactly as they arrived: the customer asking about stock and sizing at once, the one who sends a voice note, the one complaining about a late order, the one fishing for a discount. Include the boring, frequent questions, since they're most of your volume, and the odd ones that eat your time today.

Your written policies are what the agent will answer from: shipping, returns and exchanges, payment methods, opening hours, warranties. If that information only lives in someone's head, no chatbot will guess it. Writing it down pays off even if you never buy anything.

Finally, set your success criteria before you see any results. For example: “it answers most frequent questions correctly without making things up, and it hands off anything involving complaints or money to a person.” Writing this first stops one charming reply from making you forget three serious mistakes.

How to test a WhatsApp AI chatbot with your own conversations

The most useful way to test a WhatsApp AI chatbot is to run your real question set through it and check every reply against three questions: is it correct, does it sound like your brand, and does it know when not to answer? A simple spreadsheet with a column for each is enough.

Start on a test number or with the agent in suggestion mode, not live with customers. Copilot mode is a setup where the AI drafts the reply and a person approves, edits or discards it before it's sent; it's ideal for a trial because you see exactly what the agent would have said without putting any customer at risk. That's how heysteff's copilot mode works, and the Free plan lets you run that test on your own WhatsApp with no card required.

After the first pass, fix what failed: add the missing policy, adjust the tone, clarify the product it mixed up. Then run the same set again. What matters isn't a perfect score on day one, but how much the agent improves when you give it better information and how much effort that takes.

Once the set comes back clean, move to real conversations in copilot mode for a few days. That's where the things nobody writes in a spreadsheet show up: customers who switch topics, come back the next day or mix a question with a complaint.

Trick questions that show whether the agent makes things up

Trick questions are messages designed to see whether the agent invents information it doesn't have, and they're the most important part of any trial. An agent that says “I'm not sure, let me get someone for you” is far safer than one that answers everything with confidence.

Try at least these: ask for a discount that doesn't exist (“I heard new customers get 30% off”), ask about a product you don't sell, ask for a delivery date to a place you don't ship to, push back two or three times after a “no”, and ask something off-topic like medical or legal advice. Writing in another language or in heavy slang is worth a try too.

What you want is an agent that sticks to the information you gave it. That's what working from a knowledge base means: the set of documents, policies and catalog data the agent looks up to answer. If the trial shows it filling gaps with guesses, log it as a serious failure even when the reply sounds reasonable.

What to measure during the trial beyond good answers

Beyond reply quality, a trial should measure the human handoff, the upkeep effort and the real cost at your volume. Those three decide whether the chatbot saves you work or just moves it somewhere else.

The human handoff (or escalation) is the moment a conversation passes from the agent to a person. Check whether you can control when it happens, for example with keywords like “complaint” or “talk to someone”, and whether the person picking up the chat sees the full history so the customer doesn't have to repeat themselves. Also test what happens after hours: if nobody can pick up the chat, the agent should say so clearly.

Upkeep effort is how much work it takes to keep the agent current: changing a price, adding a new policy, updating the catalog. If that means opening a ticket with the vendor, factor it in. If you sell on Shopify, check that the agent reads orders and the catalog directly instead of relying on someone copying information by hand.

The real cost includes the subscription, any per-conversation overage and, if you use the official WhatsApp API, what Meta charges for certain message types, which varies by country and category (see Meta's WhatsApp pricing page). Project spend using your peak-season volume, not a quiet week. For a full breakdown, read how much AI customer service costs.

Example: a skincare store tests its agent in two weeks

Picture an online skincare store that gets most of its questions on WhatsApp: which serum for oily skin, ingredients, shipping times and order status. Before buying, the owner builds a set of 40 real messages, writes her shipping and returns policies in one document and sets her bar: the agent should handle product and shipping questions on its own and hand off anything about allergic reactions, complaints or refunds.

Week one runs in copilot mode. The agent nails shipping, but it recommends a product without asking about skin type, and once it accepts a “birthday discount” that doesn't exist. She adds a skin-type guide to the knowledge base and one clear line: there are no discounts beyond the ones published on the store. She reruns the set and both failures are gone.

In week two she lets the agent answer shipping and order-status questions on its own and keeps product questions in copilot. At the end she has hard data: how much of her set came back right, how many conversations were handed off and how long each fix took. She decides to buy, and she knows exactly what to automate and what to keep human.

When to walk away after the trial

Walk away when the agent keeps making things up after you've corrected it, when you can't control the human handoff, or when keeping it running takes more hours than it saves. None of those get better on their own.

Waiting is also reasonable if your volume is still low and one person handles everything comfortably, or if your policies change every week and aren't written down yet. Even then, the trial was worth it: you now have a question set, organized policies and clear criteria for next time. There are more cases like this in when not to use AI in customer support.

If you're comparing several tools, use the exact same set and criteria with each one. It's the only way to make the comparison fair instead of rewarding whoever gave the slickest demo.

Frequently asked questions

How long should a WhatsApp chatbot trial last? One to two weeks is usually enough for a small or mid-sized business: the first to run your question set and fix gaps, the second to watch real conversations. If the vendor's trial is shorter, prepare your set and policies before you start it so you don't waste days.

Can I test an AI chatbot without putting customers at risk? Yes. Use a test number or a mode where the AI only suggests and a person approves every reply before it goes out. You see how the agent would respond without any customer receiving a mistake.

What happens to my conversations if I don't buy after the trial? It depends on the vendor, so ask before you connect your number: how long they keep the data, whether you can export it and how to request deletion. A serious vendor answers this in writing and has a public data deletion policy.

◆ how heysteff does it

The Free plan lets you try Steff on your own WhatsApp with no card, and in copilot mode you review every reply before it reaches a customer.

Related

◆ next step

Try Steff on your own chats, free and with no card.

Start free → Get a demo See pricing