Why Don't People Use Your Software Framework?

Fade in. Exterior. Unnamed city. Day.

- Three months earlier -

"... and today's AI is all about engineering. With that, let's wrap up this episode. If you like our channel plea..."

You gasp, slamming the 'pause' button on your Airpods and skidding to a halt. A brilliant concept had just blossomed in your mind:

"I need to build an AI framework!", you exclaim to yourself, filled with excitement and a sudden burst of energy. You can't believe you never thought of it before!

Your mind is already racing, laying out the steps and moving parts needed to launch this product into reality. With shaking hands, you hastily pull off your running shoes, ignoring the small puddle of sweat pooling behind your ears.

The world around you stops suddenly: the birds stop chirping; the cars stop honking; the other joggers fade away. It's like you've been transported away from the hustle and bustle of the city, and into a world of possibility and success.

Your fingers move furiously, swiping and tapping your phone as you type out notes, ideas and random thoughts. You dream of potential investors, new applications, and the narrative of your first launch on Hacker News and Product Hunt. Your heart races as you write, almost like you could manifest success with every keystroke. Your mind fizzes with excitement, and nothing, NOTHING, can stop you now. Your Airpods drop to the ground as you take off, running toward the bright future and your newfound success.

- This is not your day -

The initial hype was great. Within the first few days of its launch, thousands of users were trying out your framework and sharing their experiences. You got hundreds of GitHub stars and fifty upvotes on r/machinelearning. Not too shabby. Not too shabby at all.

But then...something happened. Something unexpected - user growth started to plateau and then drop. Suddenly, weeks had flown by and you hadn't seen any real growth. You started to worry that the software you'd poured your heart and soul into just wasn't sustainable.

Surely this couldn't really be true. There must be some problem with Google Analytics or the telemetry you set up. Maybe it's just a funny time of year - lots of people are on holiday right now and not using your software at work. You'd done everything by the book. There's no way this could be going off the rails so soon.

But deep down, your heart sank. You knew these are just flimsy excuses.

Now you meander down the street, your head bowed and your hands in your pockets, your mood as gray as the cloudy sky above. You think back to the launch of your software and the excitement and anticipation of that day, as you would the birth of a bouncing baby boy. You created something that you were passionate about, something that you believed in, and yet despite the initial buzz and hype, it was a flop. But you trudge on, determined to get to the bottom of what went wrong.

"Why don't people use it?" You stare up at the sky and plead, trying to make sense of what had happened. God knows you'd put in the effort — maybe (probably. definitely) too much. You just wish you could have done something, anything, differently, to make your software take off.

Eventually, you find yourself on a bench in the park. The sun begins to peek out from behind the clouds, almost as if it knows your mood and is trying to brighten your day. You let out a long sigh that you didn't know you'd been keeping in, and let your eyes close, feeling the warm sun on your face and the cool breeze on your skin.

You sit in silence, pondering your next move.

Fade out.

- The end? -

Understanding three common mistakes

Success stories are often similar; failures, on the other hand, are often unique.

When it comes to software like AI frameworks with low adoption, there are three common mistakes in the early stage: bad focusing, poor onboarding, and lack of features. In this article, I'll explain them step by step with illustrations and, most importantly, show you how to fix them.

The AI solution space

Every real problem needs a solution.

There are many AI problems, big or small, easy or complicated; as long as they're real, they all need solutions.

Let's use dots to represent a solution. You could picture the AI solution space as below:

In reality, solutions are not uniformly distributed. They're clustered. This is because different problems may end up with similar solutions. For example, sentiment analysis and text classification can both be solved using one language model with different training labels. Image segmentation and object detection can be solved by training a ResNet. As a consequence, the real AI solution space looks more like this:

The clustered nature of the AI solution space justifies the existence of software frameworks, abstractions that provide generic functionality shared by a cluster of solutions, and can be selectively changed by additional user-written code, providing a standard way to build and deploy solutions. If solutions were uniformly distributed in the AI solution space, then no common pattern could be abstracted, leaving no need for frameworks.

Let's recap the relationship between the problem, the solution, and the framework in the illustration below:

Having those well-defined illustrative concepts helps us recognize the first mistake when building a software framework.

Mistake 1: Bad focus on a tiny niche market

In the figure below, the bubbles A, B, C, and D represent four frameworks. In terms of focus, it is easy to see that A > B > C > D, as A enables the most solutions and meets the demands of the most real-world problems. D is the worst as it has fictional demands that match no real problems. Therefore, A will have the highest adoption in the long run, while D will have low/no adoption regardless of how many posts you submit to Hacker News and how many stars you get on GitHub.

"Who would be stupid enough to choose C or D?" you might ask.

The disturbing truth is if you get a sparkling idea during jogging, or if you're building a new framework when reading this article, you're very likely in C or D. Here's why:

• You can't see the full picture of the AI solution space. Your observable space is limited by your knowledge and expertise in AI and the industry.
• The markets behind A and B are oceans filled with big (and hungry) fish. You don't have the resources to swim that fast or bite that hard.
• Your experience and knowledge make you overestimate the number of solutions and requirements covered by C and D.
• As an engineer, you are easily obsessed with the framework idea itself, ignoring that it lacks any practical use in the real world.

In any case, don't be D. Don't be sad if you're in C. The solution space is huge and dynamic; it continuously changes over time. New problems and requirements emerge; old requirements will be unnecessary. One good example is the creative AI market in 2021 vs. 2022.

Say you have a good focus and your framework conceptually covers some real needs. Congratulations! Let's move on and see the other two common traps you may fall into.

Mistake 2: Poor onboarding experience and education resources

Let's zoom in on a framework "bubble":

The first thing to highlight is the knowledge gap between you and your users. Your framework may cover all the functionalities that users need. However, due to the lack of docs and poor onboarding experience, users don't know your framework can solve their problems. The consequence? Low adoption.

Unlike you (as the know-it-all of your framework), your users observe the framework from a completely different angle: they can learn it only from the docs, tutorials, references, videos, and code snippets that you expose to them. In reality, your impatient users will only learn from well-written resources. If they feel onboarding is rough, they'll immediately give up and seek alternatives. They are not as determined and passionate as you.

One common mistake developers make is ignoring the importance of educational resources and DocOps when building a framework. They naively believe that as long as they commit a function to the master branch, users will use it; as long as you write a few vague keywords about a feature in the commit message, users will understand the feature; as long as you make breaking changes, users will adapt their code accordingly.

Users won't.

• Users won't spend time digging into your unit tests to understand how a function works.
• Users won't read your commit message riddle and the pull request's title to guess your new feature.
• Users won't check spam filters to read the newsletter where you announce the breaking changes.
• Users won't read poorly written docs with non-executable code snippets and report bugs to you.

Users won't waste their time on your software if you won't "waste" your time on educating them.

So, you've pulled out all the stops to minimize the knowledge gap: good docs, detailed tutorials, instructional videos, offline/online meetups, conference presentations, daily tweets and weekly newsletters.

That's that dealt with. And you already fixed your focus (from mistake one). But you still see low adoption. What is it this time?

Mistake 3: Outdated and missing features

AI is evolving fast, and the market is highly competitive. You, on the other hand, are outdated. Obsolete. A relic.

Every day, hundreds of new papers are published on ArXiv; benchmark leaderboards get refreshed; new repositories trend on GitHub. If you can't keep your knowledge up-to-date, your framework quickly becomes obsolete. It won't catch users' attention anymore.

Compared to the second mistake, where users don't know they can do it with your framework, in this case, users know they can't do it with your framework. That's game over for you.

Tired of this competition? Well, you chose it, right? It's the hot market flooded with opportunities and money. Are you going to deal with that heat or get cold feet and run away? What about your insert_country_name dream?

How to fix those mistakes

History is full of mistakes; that's how we learn.

Don't be afraid of those mistakes. Confronting them is just a part of your life.

The first step is to recognize which mistake you made. It sounds easy, but under the huge pressure of low adoption, you'll very likely jump to the conclusion that you made all the mistakes. Confessing you made all the mistakes is as easy as saying "it doesn't work". Without breaking it down, this confession leads to no action points. It's up to you to prioritize and solve problems one by one, given your limited resources.

Retarget for bigger demands

If you have a bad focus, you need to retarget. Learn more about what people need and abandon the obsession of making a framework. Build more ad-hoc solutions and recognize their connections before working on the framework. Do more user research and market analysis, find the slice of cake that's big and interesting enough.

It's also possible your intentional focus is good enough, but your narrative and branding suck, which makes users think the framework doesn't match their needs. In this case, reconsider the narrative, promote the features that really matter, and hide the complexities that nobody (besides you) cares about.

Revamp docs to improve onboarding

If your onboarding experience is poor, you must revamp the docs and organize educational resources. A software framework needs educational resources at multiple levels – an API reference, tutorials, code snippets, high-level applications, and industry benchmarks. Just having an API reference auto-generated from docstrings is nowhere near enough.

Think back to how you learn a second language: You use a dictionary to look up vocabulary (i.e. API reference). Your first-year book is filled with basic daily conversations (i.e. code snippets) to teach you how to greet and buy things. Your second-year book has longer stories (i.e. high-level applications) to help you master grammar and descriptive writing. To level up your skills, you watch movies or read books from famous writers (i.e. industry benchmarks).

Your users are the same. On day one, they're just novices. But they learn over time, making progress by consuming the knowledge you give them. Your task is to help users grow as fast as possible, by providing all levels of resources that they need. When they reach the master level, you can relax a bit, as they can now advocate to others; they can produce high-quality educational content without you.

Don't just stick to text; go multimodal! Try podcasts, videos, meetups, and conference presentations, if you have the resources.

Move fast with lifelong learning

If your features are outdated, you need to learn more about the state-of-the-art and update your framework. Learn APIs, design patterns and algorithms from popular open-source frameworks. Keep an eye on their new pull requests and new features in the release notes, and think if it makes sense to add them to your framework. Always keep your knowledge and your codebase up to date. From time to time, do some retrospective on the feature roadmap: are you moving in the right direction? Are you adding this feature because people really need it?

In the open source world, there is no absolute advance. You can always learn from other software, and it's all about velocity. If you fall behind but move faster than the pack, you can catch up eventually. If you're leading but stop updating, a predator will eat you for lunch. Velocity is the lifeline of open source software.