Over the last year or so, I experimented with different ways of getting customer feedback for Checkly. This post is about what worked for us and how I was totally wrong about chat widgets.

We'll look at some common tools like Hotjar, Drift, SurveyMonkey, straight up email and even Github. This is a pretty anecdotal story, no science was hurt in the process of writing this post, YMMV and all that.

Also, I focussed mostly on the free(mium) parts of the tools mentioned above, as Checkly is an early stage company. No tengo dinero. Yet!

The private beta period

We launched a private beta period in February 2018. Most signups came through a combination of Betalist, Hackernews, a tiny bit of Twitter and some Reddit posts.

Shamefully, I did not really have a strategy to collect, manage and structure customer feedback outside of installing a Hotjar widget and just following along with the recommendations from the onboarding emails.

Surprisingly, this was quite successful but only because of Hotjar's Incoming feature. Incoming allows users to express how happy they are by selecting an emoticon and optionally point out details on a page by adding some text and a (partial) screenshot. You've seen it on a million sites by now.

Over the weeks, I collected about 50 to 75 responses ranging from gushing praise and super nice tips to sharp critiques on obvious bugs and even some colorful profanity!

I weighed the feedback, adjust my backlog and reported updates and fixes over Twitter. This directly highlights a pretty big issue with Incoming: It's only incoming! Silliness aside, users can leave an email address, but any back and forth is disconnected from the initial report.

Using Incoming was really helpful, but I struggled with almost all of Hotjar's other features. Admittedly, I was a noob on configuring this but I'm also a pretty quick learner. After some iteration and tweaking of copy, placement etc. I still was not getting any value out of the Survey's and Poll's features. I got one response to a survey and a maximum of 15 responses to a poll.

I used Hotjar's Analytics suite of tools a bit, but strictly speaking they are not customer feedback tools, at least by my own definition. Something for another post.

People also emailed me directly, but Hotjar took care of most direct application feedback. Remember, no one was paying yet and there was no pricing to haggle about.

All in all, Incoming is pretty useful, especially at an early stage / beta launch. Its entry barrier is super low, so non invested users (i.e. not paying or thinking about paying) easily leave pretty interesting feedback. However, Incoming really means just incoming.

Total score: ⭐️⭐️⭐️⭐️  recommended

Adding chat and a survey

Near the end of the private beta period, I added two things:

1. A Drift chat widget. This seemed to be the hot thing to do. Why Drift? They have a pretty generous free tier. I turned off the *ping* sound.
2. A Survey Monkey survey send out to all users when they are at the end of their trial.

Chat

I was SUPER skeptical about the chat widget. I personally found them annoying, overbearing and could not imagine them working for customer support or sales.

Boy was I wrong. As in completely, utterly, totally wrong.

Within the roughly one year since I installed the chat widget it has become my number one communication channel with customers and trial users.

Here is a random overview of chats from my Conversations tab in Drift right now. Notice they go from feature suggestions, to sales enquiries to actual complaints.

• Feature request: Customer suggests a tweak to how read-only users can have better access to some parts of the application. The customer added a screenshot to clarify the point. This was implemented and rolled out the next week.
• Feature request: Customer asks about exposing 95 and 99 percentile metrics on our Prometheus integration. This is a bit harder. Pushed to our backlog.
• Complaint: Customer requests how we deal with refunds because we totally dropped the ball on one part of our alerting service. Case was discussed and refund was issued to said customer within the hour.
• Sales: Customer asks if there is an option to customize her plan to add just a bit more browser checks without having to jump to the next plan. Yes, this is an option and we modify the plan on the fly.

At this point, I'm not actually using any of the other Drift features yet, except for pushing some custom attributes like accountID and planName so I can quickly see what type of customer I'm dealing with.

Interestingly, I now always opt for the chat widget myself when reaching out for support and I now judge the customer service of a SaaS app by how well their chat integration works.

Total score: ⭐️⭐️⭐️⭐️⭐️ A++ would buy again.

Survey

We send out a "trial expiry reminder" email using Mailchimp. It's a pretty short, friendly voiced email with a very friendly call to action button to fill out a 2-minute survey on how the trial experience was.

After 521 sends I stopped sending that specific email as the goal of the email — collecting feedback about the trial period — was clearly not being met.

The SurveyMonkey survey was filled out eight times in total. The survey itself was actually put together quite carefully and followed all the best practices and examples I could find.

Maybe I was still really bad at making surveys, or maybe surveys just have a super low response rate? In any case, the survey was not working for me at this moment. I will try out Mailchimp's built in one-click survey pretty soon and see if that lowers the bar enough for users to actually click.

Users still get a heads up about their trial expiring, just so you know.

Total score: 🍍🐼  not sure, I probably did something wrong.

Adding a public roadmap with voting

After a pretty intense six to eight months of iteration and listening to direct feedback I hit a new phase in Checkly's lifecycle. There were a couple dozen paying customers and some were very passionate and vocal about Checkly's roadmap.

A pattern started appearing.

1. Many paying customers requested very similar features or tweaks.
2. Many of these features/tweak were already in my private back log.
3. I sometimes had a hard time recognizing what customers deemed important.

How to align all of this? How to communicate to customers you are aware of their needs and allow them to vote and interact with the respective backlog items? Public roadmap to the rescue!

Now there a gazillion tools out there that fulfill this need but I started out with two simple, free things:

1. A Github project with an issue template and project Kanban board. Users can add issues in the normal issue view and use the voting feature Github added fairly recently. Using some tags that describe the feature areas such as dashboards and API checks issues are organized. The board has three columns just to keep things as simple as possible.
   

2.  A small addition to the UI to draw attention to the roadmap and prompt users to suggest feature. This was 100% stolen from inspired by Hotjar.

To be completely honest, it took a while for this to catch on, but over the last months more and more customers are actively contributing and voting. I guess you need a certain critical mass of invested, paying customers for this to work and I just hit that mass recently.

I also actively direct questions raised on Twitter and in the Drift chat widget to issues in the public roadmap and customers seem to be totally fine with that.

Yes, you need a Github account to contribute to this. And yes, there is no direct link between a Checkly user and a Github user. However, this has not been a problem yet but might account for the slow uptake.

What about email?

Customers and trial users still actively email about all kinds of things. Replying to email and interacting with customers should just be a basic hygiene factor.

Just one pet peeve — Don't use a "no-reply" address on your automated mails. People should just be able to reply to a Mailchimp mail and get a human on the line.

banner image: detail from "People enjoying spring on a picnic", Katshushika Hokusai, Japan. Reproduction from circa 1960. Source