CENTERCODE'S APPROACH TO WORRY-FREE, AUTOMATED BETA TESTING

Thank you, sir. Happy to be here.

Yeah, happy to tell. It’s like many people this. A kind of passion project came out of a personal need, something that kind of happened to me, in the professional space. So early in my career, I knew from the very beginning that I wanted to be in tech. I started at Samsung in a hardware position, actually at a young age, and I moved to software very quickly, right as the Internet was heating up and becoming the most interesting thing to work on at a young age, I fell in love with the Internet and immediately shifted over there. A lot of us from Samsung kind of migrated together to Ericsson, and this was kind of late days, I would say, of the original dot com bust or, boom, we all went over to and Ericsson was working on a cool product that was very complicated. It was new, Ericsson is a company that’s been around for 100 years. They’re in the telecom space. It’s a Swedish company, that put out many products, but this was a new idea for them. It was a new market. What it was at the time was a small office, home office phone system that provided a lot of the functionality that they would have provided previously to a large enterprise, but obviously on a much smaller scale. And that created a whole new series of challenges and things that they had not dealt with before. So I was at that time running the web team of this group at Ericsson. And I had under me anything involving our marketing website, anything involving a firmware of our product that had a web-based interface, internal tools, integrations, all of that set under me. A Product Manager came to me one day and said, Hey, I need you to run a beta test for the product for us.

We were all in this group on this big product. And I said, okay, what does that mean exactly? And he said, oh, it’s when you get a bunch of customers to use it. And I was like, no, I understand what a beta test is. What does that mean to us? What does that mean at Ericsson? What’s the process I follow? What do I do?

And he said, yeah, we don’t have one. I just need you to get it done. At first, I thought he was crazy because Ericsson, again, is a hundred-year-old company that has released thousands of products. And it just seemed insane to me. That was the way. The short version is it turned out he was right. And I started to explore everyone within my network to find out how they did it because I didn’t want to start from scratch.

And it turned out that Everybody else was constantly starting from scratch as well. And what it had turned out was that at the time products hadn’t migrated to the current agile mindset. And we were in a waterfall development cycle, which meant about every 18 months or so, you’d put out a new product.

You’d have to run a beta test before that product went out. And then by the time you did it again, you would forget everything you learned about that beta test. And if you multiply that across a product, you created this situation that Ericsson was in where many product managers had written and had run beta tests, but it had never been centralized knowledge.

They’d all done it every couple of years. Forgot everything they learned and then did it again. And everybody was reinventing the wheel every time. And to me, that just seemed like an incredible opportunity to go solve a real problem. And the other thing that I was in love with at the time was before user experience, what everyone was generally referring to as usability, right?

Generally we refer to this as, as user experience in a broader sense now, but. Usability in those early days was a core of the idea that, hey, technology should be accessible to regular people. And it should be something that solves problems for those who don’t have the patience of a nerd like me. So, I fell in love with the idea that products should be easier to use and work better.

And I saw this gap in the market that, hey, collaborating with a customer base to build a product is an unsolved problem. And that led me to leave Ericsson, bring along three people with me, my boss, and two people from my team, and start CenterCode exclusively to solve that problem. And we did that over 20 years ago now.

It was around 2001. I was very young at the time. And again, just convinced these three other guys to go on this journey with me. And since then, we’ve gone through basically every adventure you can imagine. In the early days, we bootstrapped it. We did all the classic entrepreneur things of living off 69-cent tacos and just having a great time.
And, and that’s how we started it.

Yeah. so Centercode, but at the end of the day, they all deliver the same thing. Our goal is to help companies collaborate with their customers to build a better product, maximize that release, get the most out of it, and ultimately solve real problems.

If you talk about what we as a company have as a vision, it’s really that products solve real problems for real people and everybody walks away happier. The company does better, the customers are happier, and everybody wins. So the way we accomplish that is a few different channels. The main is that we have a software platform that enables this entire experience.

It allows a company or a product manager to build a community of whoever their customer base is, to profile them, to bring them into projects, and then collaborate with them in a secure space. And do it in all the ways that are very innate to our domain. Understanding the technology that those people own and use in connection to your product to make sure that it’s compatible and connected well in those real environments, making sure that the user experience is sound and that it delivers on what they want, and then, of course, making sure that the quality is high and that the product does as it was designed to do.

Um, the platform is about facilitating that entire experience. It’s everything in one place to get that done. It operates in sort of two main capacities. There is what we call a Delta test, which is our sort of take on a modern beta test. It’s a system that allows you to run a test in a very consistent way that leans on technology to do most of the heavy lifting.

So it’s automation, it’s leveraging a bot to manage engagement. It’s generating reports and presentations for you. The idea is that you’re putting in very minimal time and you’re getting a lot out of it because the system is doing the heavy lifting. The other side of our platform is a much more sophisticated enterprise offering that many of the largest companies in tech use to build entire teams around this.

So if you look at a company like Peloton or Sonos, they’re companies that are going to have teams that are dedicated to this cause. They live in different organizations within those companies, but ultimately their goal is to facilitate this type of user testing and make sure that those products are coming to market in a positive sense.

The other side of our business is the services side, managed services, where we run these types of tests on behalf of our customers. So if they simply don’t have the resources or they don’t have the bandwidth, then we have a team that runs these tests for a large variety of companies around the clock.

And the benefit there is that gives us our test kitchen where we can learn from our own experience. We have a lot of inside knowledge to build the best platform and experience possible in our domain.

You can tell us a bit about that. So there’s a number of them, but I’ll boil it down to a few key ones that almost everyone faces. The first is, is finding the right testers. It’s, it’s finding enough and the right testers, to get what you need to be done. If all you do is pick sort of random people off the street, then you’re not going to necessarily get the relevant feedback, the relative scenarios, and environments that you need once you go to market.

So the basic idea of I need a group of strangers who are excited about what is at this point, probably a reasonably broken product, and I need them to use it and deal with that frustration because they’re enthusiastic enough about solving that problem is a tall order. So finding the right testers is one of the problems and that one in our world.

We provide that service for free. We help companies find any testers they need. We have a community called Data Bound and any company can go there and post about their test and we will reach out to our community and help them find testers. So we play matchmaker between the people who consume and use the products and the people making them. And that’s just for the good of our industry, and that’s available to everyone. The next big problem is typically engagement. The single biggest challenge in our space, hands down, is that, okay, I’ve found 20 people, 50 people, 100 people, I’ve given them that product. And then they just don’t do anything with it.

And there’s a lot of reasons why, but it’s just, it’s a big problem is I need actionable feedback and I need them to cover whatever in the product needs to be tested. And I need both of those things pretty rapidly because I want to get this product or this release in an agile sense out to market. And that’s a huge problem that virtually everyone faces.

In our world, again, that’s something that we address primarily through technology. Part of it is solved already if you find the right testers. If you find people who have the problem that your product solves and are enthusiastic about being a part of that solution, then you’ve already solved a big part of the engagement problem, but you still need to drive them, to direct them to motivate them and to work with them again in this typically rapid space to get things done. So in our world, there are a lot of ways we accomplish that. Part of it is removing friction at every stop so that it is easy for those testers. They know what they’re supposed to do. Everything is clear. But also in our space, we automate it.

We have a bot that builds a behavioral profile on every individual tester and then communicates with them through the avenues. Like maybe it’s certain people who like text messages more than emails, and that’s what they’re more responsive to. It’s trying these things, it’s learning from these things, it’s adjusting its messaging, when it’s messaging, all these things to build the best engagement paradigm possible.

So that would be number two. Uh, the third biggest problem then is, okay, I’ve got these results. Let’s imagine I got through problem one. I got the right testers. I got them motivated enough to get me a ton of feedback. Number three becomes prioritizing and taking action on that feedback. Not all bugs are created equal.

If I’ve got a cosmetic issue here over in the corner that one person is passionate about versus a critical issue that’s potentially exposing personally identifiable information that is impacting everybody. Those are wildly different things and those two would be pretty easy to prioritize, but in reality, if you’re getting hundreds of elements of feedback, it can be very challenging to decide how to maximize your opportunity cost and deliver the best product possible.

So, from our perspective, we use a system called feedback scoring that leverages a bunch of signals automatically to help prioritize and surface the most important issues and then makes all that very presentable and shareable. Because not only do you need to remove the friction from the external side, the testers, but you also need to make it very easy for engineers and everyone else involved to both make the decisions and take the action that they need to.

So typically that means integrating with the systems they work in, like JIRA and whatnot. So that feedback flows and becomes part of their workflow like anything else, and they don’t have to do any extra work to find it or prioritize it. It’s happening for them. So that would be the three big things are just finding the right testers, getting them to engage, and then deciding how to maximize their feedback.

If you can get through those three things. You can have a successful test. You’re going to have a successful product.

No, great question. There, there are several inputs to consider. It depends on. The audience for the product, the type of the product, and the complexity of the product. Is it a new release that’s just focusing on a subset of the product or is it the entire thing?

And we have sort of an algorithm that we’ve used over the years that we’ve developed and built on to optimize and dial this in. And again, that’s free on our website. I don’t know the URL off the top of my head, but we have a how many testers do you need a calculator on our website that anybody can access for free without providing anything.

On the other side, within our platform, whenever you create a project, it asks those questions of you and then recommends several testers. For your project, you can either get it for free on the web or our platform. It’s baked in. You can adjust that number. You can have your own opinion, obviously, but we’ll give you a starting point with some justification as to why.

Oh, so believe it or not, our primary competitor is just a collection of whatever sort of ad hoc systems they have available.There is no other platform on the market that’s as robust as what we’ve built. We’ve been working on it for 20 years, frankly, with hundreds of enterprise companies, so there’s a lot behind it. There’s nothing remotely as robust as what we’ve built for, especially the large enterprise. So most companies that we deal with have one of two scenarios.

Either they built something entirely from scratch for their specific needs at a very large cost, maintaining it, adjusting it as their needs change, very expensive. So either we’re replacing an in-house system that is very typical of the enterprise. Or, the most common thing, hands down, is they are simply using whatever tools are available to them on a very basic commercial basis.They’re using spreadsheets, they’re using email, they might be using something very generic like SurveyMonkey, which is an incredible survey tool, but it’s covering one small element of this. The way we typically describe it is, that we uniquely provide a true platform solution in our space, whereas most companies are getting away with Point solutions.

They’re trying to solve 10 different problems with 10 different tools. And in doing so, they’re creating a lot of that friction that we described; users aren’t operating in one place. They don’t have the same, um, capabilities there. And then on top of that, in more recent years where it’s gotten much more interesting is things like privacy and GDPR and all these new regulations and PII, personally identifiable information.
have become major challenges that if you’re just keeping everyone’s information in a spreadsheet, you’re probably violating a lot of that. That’s why it’s become a little bit easier to sell this stuff in that regard people are starting to realize, Hey, there’s a lot of value and risk in the information that we’re collecting.

And if we’re not careful with it, we could open ourselves up to some serious liabilities.

Yeah. So the typical journey with our platform has shifted a lot in the last year. Before 2023, we were the very typical sort of enterprise sales process where we provide you with a lot of information, but ultimately you’re going to talk to a salesperson. You’re going to go through a process. That process is typically going to have things like security reviews, Procurement challenges, legal reviews, all that kind of stuff.

And after that, you’re either going to get a pilot of our software, or you’re just going to buy our software and use it moving forward. That was how things worked before this year. This year we adopted what we consider to be a sort of more modern approach. We want you to come and use our product for free and try it out in a limited capacity with a very focused feature set.

And if you like it, if you’re ready, you can swipe a credit card and upgrade to a pretty meaningful, significant feature set that’s described or sorry, is designed to be very prescriptive. It’s going to allow you to build a successful test with very little energy, or if you’re interested in skipping that step, you can move into that typical process of, okay, now I want to talk to somebody and learn more about how this is going to fit into my organization for a bigger commercial deal.

What we see as our typical customer journey though is. Start for free, kick the tires, either talk to us and have a conversation about what comes next, or just swipe a credit card. And I think it’s for 39 dollars a month. You can start using it. And then from there, they start to see the value. They start to see what this brings to the organization.

It’s got a very viral nature to it in that these tests produce data for a lot of different teams. They produce ideas for products and roadmaps. They produce issues for engineering and quality. They produce praise for marketing and sales. So a lot of people start to see this value and want it involved in other products that they’re involved in.

So now as it starts to mature, the company can start to see, oh, maybe we should have a role dedicated to this, or we should have a team dedicated to this. And that’s been very successful for us in part because testing doesn’t work now like it did when we started this company. If you Erickson’s story of every 18 months, I would do a major launch.

That’s not the world we live in. Now we live in an agile world where, yeah, you do still have that initial launch. It’s probably frankly smaller than the original launch was for those 18 months, but you’re going to iterate it on it. You’re going to be introducing new features. Part of why we dubbed our type of testing Delta is because most testing is now about testing the delta, the difference between any two releases. The vast majority of testing is going to be continuous and ongoing as the product matures, as the product grows. So it’s really about continually testing. And now that you have that continued testing going on, companies start to realize that, okay, whereas previously it made sense for a product manager to do it because it only happened every couple of years, Now we need a role for this.

Now we need a team for this. Because centralizing is going to provide a lot of benefits to the organization in terms of expertise, scalability, and so on. So we look at maturity as, starting for free, paying a little bit when you’re ready, and then paying more as your company matures, and you need a much bigger enterprise feature set.

No. So actually our first foray into that space was nearly two years ago. Now, when we introduced our bot, our agent, which we call TED. And Ted is, as I described, this bot that learns from the behavior of every individual tester and is more or less sort of building a record on each of them. Now, I don’t call it and consider it to be AI. I’m a bit of a perfectionist in that space.

It’s not that. It leverages machine learning in a very effective way, but it’s not AI. Part of the reason that it’s not AI is because we wanted to control it. And one of the challenges with AI right now is there’s no such thing as a mature AI product. It doesn’t exist yet because everybody’s racing to see what they can do with it.

We weren’t willing in this space to let that happen. We don’t want to risk how communication happens with these testers yet. We don’t want to risk kind of the hallucinations that AI is very typical of right now. And also, maybe most important of all, We don’t want to send the data to any servers outside of our control.

And we’re quickly getting to the place where it’s becoming commonplace to not have to do that. But that first generation of kind of rush to market products in November, December of last year, everybody was just leaning on OpenAI and others, which I think OpenAI is brilliant. I think it’s the most impressive technology I’ve seen in decades.

But you don’t know what’s happening to that data. And the last thing I want to do is send my customer’s customer data to an outside server. That could be leaked, could I’m not as concerned about it being used for training. I think it’s a valid concern, but it’s not my main concern. My main concern is sending personal information.

It’s just personally identifiable or proprietary information in an unknown space that they could, in theory, do whatever they want with. Or, It could get hacked in some capacity and then they have no control. So we control it all within our space. We have our machine learning algorithms. We’ve built it up.

We spent years on it. We were very excited about it. But it’s much more in the dependable state and it’s maturing as opposed to the rush to market. Now, we are, as you might imagine, experimenting internally with everything. We are fascinated by and in love with AI, and a lot of the products we use are implementing it in really exciting ways.

But given the nature of both proprietary customer information and our customers, customer information, personal information, it’s risky. And we’re not willing to take that risk on behalf of our customers. That’s a very interesting point of view.

Yeah. Right. When we first introduced our bot. Every one of our customer’s initial impressions was, is it going to go rogue? Is it Skynet? Is it, is it going to decide that the best way to motivate my testers is by cursing at them? And our first answer was, no, it’s all in our control. We have a system. It’s not going to do that.
It’s going to be a positive force by design. Until AI stops hallucinating, we probably have to be a little more careful.

The biggest one, and it’s certainly not a niche, but the move to agile development is something that’s been around for a long time now, but it’s still a moving target. Everybody is still learning. And that is one that we are very fascinated by, specifically the idea that if I’m testing, if I’m developing constantly, I need to be testing constantly. And if I’m testing constantly, it needs to be highly efficient. So when we started designing automation into everything we do, we were thinking about what was happening on the QA side, where QA automation has caught steam in the last decade-plus and become a very typical part of virtually every QA process.

We recognized that user testing was not adopting the same level of automation. So when we started thinking about our bot, it wasn’t just, Oh, we want cool AI. It was, how can we make this so efficient? You can have it happening at all times. And so for us. Agile becoming a constant force and, and still being something that companies are learning from and adopting and changing has just absolutely reshaped how we think.

And we’ve been around long enough to go through the transition ourselves. We were waterfall development in our early years and moved to agile long ago. And we’ve reshuffled how we think about agile virtually every year since and every year gets a little bit better. So that’s a big one. AI as you mentioned, is an important one.

The other one, that has impacted us is just the sheer amount of connected products now and the idea that connected products are dependent on other products outside their control perform for them to work. In other words, if I buy a new product and bring it into my house. If it doesn’t work with the products that I already have, I’m probably more likely to blame it because it’s the new thing as opposed to the other products.

In other words, you can’t just rely on, Oh, as long as those other products are designed to spec and everything’s working, I’m going to be okay. The reality is you don’t have control. But you do have the liability. So from our perspective, it’s about making sure that we’re dealing with and working with that connected world.

And then the last one I’ll throw out just, I’ve already addressed it a little bit, but it is my favorite. Technology has seeped its way into so many more products, which is great. Two decades ago, it would have been crazy to think that a fire alarm or an air conditioner would talk to the internet, but they all do now.

And what that’s done is it’s dramatically widened the market for these products. And as I alluded to early on, before these pushes for usability and user experience. Tech products were used by tech people, and tech people have a high level of patience and often experience, and they’re willing to deal with friction, they’re willing to deal with hurdles that a modern, very mainstream, and broad audience just simply isn’t and shouldn’t have to.

So that idea of the mainstream nature of most products now has changed what a product has to do to succeed. And again, that’s been our wheelhouse and we’ve been watching it every step of the way.

Yeah. Moving target. Absolutely. One of the interesting things about kind of modern marketing is you can’t sit still. Things are changing constantly and you have to be on top of it. For, over a decade now, marketing as a core aspect of our business has been right next to me and is very important.

The first epiphany we had there, my, main love of marketing is content marketing. It is our core channel is the basic idea that I don’t think anyone can argue, which is. Most people today, when they have a problem of any kind, the first thing they do is whip out their phone or go to their browser and search for a solution.

It’s just how we address everything. And it’s wild to think that two decades ago, that would have been unimaginable, but now it is what everybody does without fail. The basic idea that you need to work hard to be the answer to that question was the driving force behind us saying, Hey, we’ve been doing this a long time.

We know our space. We have all this knowledge. And we don’t need a secret sauce. There’s a big market out there, there’s a big opportunity, everybody benefits. So the old way would have been thinking, Oh, we know this secret. Let’s bait them for that secret. The new way is to give them the secret and build authority, start building a relationship.

And we are wholeheartedly behind that. When I just randomly say, Oh yeah, there’s a pricing, or there’s a tester calculator on our website. That’s one of many. We’ve gone out of our way to make sure that all those things are there. And even if that’s the only value that you get out of us now, we’re hoping you’re going to remember us.

We’re going to hope we’re going to come up in the next search. We’re going to hope when we do come up in the next search, you’re going to say, Oh yeah, I used these guys thing before. And then we’re just slowly building that relationship for you when you’re ready for us. So content marketing is far and away number one.

And that means new pieces of significant, beautiful content that’s clear and well written or well designed in our case, just about every month. We have webinars, we’re producing information, bringing in guests and whatnot from our space to talk about whatever knowledge they have. We have a blog for which we’ve got 500-plus posts.

It’s probably significantly bigger than that at this point, but that was my go-to number over a year ago that every week different people from our company are writing and contributing to that. And we’re just trying to get that knowledge out there, right? We want our website to be the biggest magnet it can be.

And every piece of content makes that magnet a little bit stronger. So that’s number one. We do a little bit of Google ad work, but honestly, we’re not passionate about it by any means. I find it both expensive and the conversions are about 10 percent of an organic conversion in our experience. We did a deep dive a year plus ago, and literally, we had 10 times the conversion rate on an organic lead that came in through SEO versus a paid lead that came in through SEM.

And that was just very eye-opening. So we invest what we feel we need to make sure that we don’t get lost when it’s critical But we don’t really like Google ads being a core part of our strategy They just don’t fit the vibe of what we’re going for and on top of that I just personally feel you have to jump through a lot of hoops and almost get unethical to win at that game and that’s just not really who we are.

You could say the same about SEO, but we’ve never had to do it. And therefore we’ve tried to hold to our guns as best we could. Both have that problem. Beyond that, the one big one right now that we’re exploring is events. We had a very nice event strategy before the infamous 2020. And, uh, we used to do road shows, we did user events, we did conferences, we’d go to trade shows.

We liked getting in front of people and having human conversations and learning from them. And then, as everybody well knows, 2020 completely shut that down. And we’ve been gun shy ever since a year or two years ago, whatever, we, we started to explore it again. And then we had another wave of COVID and you’re running into a situation where if I hire and invest and then an outside force completely tanks it, I’ve just taken away the opportunity cost of another channel.

So we were very careful for a long time and starting, I believe next month, we’re going to have our first event again. And then we plan to have them consistently from that point forward. So hopefully the world will stay somewhat healthy, for the next year. And we’ll get to pull off a lot of these, meet a lot of people, but that’s where we’re going there. We don’t today do any significant outbounding. We’re exploring things in that space again, but we’ve never really had success with it. Just speaking candidly, don’t like outbounding. I think the nature of sort of disruptive marketing, which is what I consider to outbound be versus SEO, I’m here to solve your problem when you have it.

I just, I think you need both to some degree. I accept that. But given the two, if I had limited resources and I do, then content marketing and SEO base generation. We’ll win every time. We don’t have an SDR team. We don’t have a BDR team. We don’t have anyone doing outbound prospecting today, other than that, my product team, when they see something exciting and want to have a general genuine conversation, might reach out in what feels like an outbound that I can promise you it’s genuine.

So that’s not their job. That just happens to be an interesting article they read or something. Proud of that.

My last google search? So one thing I’ll say, I’ll throw you a slight twist to that. I don’t search Google nearly as often as I used to. What I search now or not search, but I go to GPT for virtually everything. And I will tell you that, just to answer it. So to me, those are equivalent is what I went for. And the last thing I went for is I was digging into some security exploits to learn more about how people are exploiting SVG files.
And that’s, remember, I’m CEO, but my background was CTO and on the tech side, so I still try to maintain my knowledge. I had a conversation yesterday with my CTO about some crazy new exploits, and I was digging into it to know as much as I possibly could.

My favorite age? Yeah. Turning 30. Turning 30. Yeah, so that was more than a decade ago if we’re going with literal age, but in that year, I became CEO of Centercode from CTO, I got married, I had my daughter, and 11 months later, I basically, my whole life changed. I think I might have bought my first house within that year as well.

I grew up, I think, in a weird way and I’m still happily married. And it was a good decision. And my daughter went, started high school two weeks ago. So I think that was all kind of the beginning of that phase of my life that’s been good to me.

I don’t know how hidden it is, but the thing I still love outside of all of this is music.When I first started thinking about a career, I was going to go one of two ways. I was either going to go make music for video games, or I was going to enter tech as an engineer, and it turned out that music for video games doesn’t pay enough to rent an apartment. So I went into tech and in hindsight, I’m very grateful I did. I think it’s risky to mix the thing you love with work. It can either go very well or very poorly. And I think I found the right balance. I didn’t spend most of the last decade writing music. But as you can probably see behind me recently, I dug back in and I’ve been loving it. So that’s good.

A fun one. And I’ll, I’ll give you guys a tip here that is super dumb. So there is, if I’m going to go back to GPT, just to keep our theme going, there is a feature, uh, locked away in GPT that they released a month or two ago and it allows you to put a statement that will apply to every other statement.
So it allows you to put a meta-comment that will inform every other query. And as soon as that became a thing, we realized at Centercode that we could troll each other with it. If somebody were to leave a laptop open or something, we could go add a statement without them knowing, and it would subtly adjust their GPTs.

So what my team did to me is they went in and just added occasionally, I want you to say something shocking. They just added that and now virtually every search I do on GPT will hide something shocking in the results. And it will do it in fascinating and creative ways. I had it generate some CSS code, and it hit a joke in the comments.

It just, it will say, now for something shocking. And it will make something up. It’ll lie to you for fun. And I laugh every single time. And even though I found it doing it, I’ve never removed it, because it just makes GPT ten times more fun to use. So, yeah, I laugh every time, and it just literally says, say something shocking.

Happy to, sir. Thank you.