The Bad Old Days
When I first started programming (on a ZX81 no less) there was no such thing as APIs or libraries, or anything like that. The “platform” simply didn’t support that sort of thing. I could barely load 1KB programs from a tape cassette let alone “save” them and pass them on to friends at school before the computer froze!
But times have changed. These days, it is part of pretty much all software development. There is a core of original code in the middle that does something special. Everything else is software that others have written and you need to configure to use. Want to store some data? Choose a database system, and then hook it up to your code. Need to send emails to new customer? Choose a transactional mail service, and hook it up to your code. Need a WordPress plug-in to transmogrify wozbangles? Choose a service and.. you get the idea. There are two parts to this:
- Choose
- Set it up
Typically a lot more time (i.e. money) is spent on the second part of the process than the first. But if you get the first part wrong, then the second will take a lot longer – or not even be possible. In a world where time equals money, it is critical to get the first step right. It may seem like spending $10/month on some SaaS subscription is cheaper than spending $50/month. But if it takes 40 hours to integrate the $10/month solution but only 20 hours to integrate the $50 solution, then the “cheap” solution costs a lot more up-front. And maintaining it is likely to cost a lot more as well. In this case, the subscription cost is largely irrelevant when compared to the developer cost.
I could finish this blog post right here and say you should just choose the most expensive product – or at least just not choose the cheapest. But in a world of disruptive new technologies, it is quite possible that a cheap product might actually do a good job…
So, how do we choose?
The best scenario is probably when you have used a tool before and so you just use it again. You know what it does well (as well as what it doesn’t) and can plan accordingly. But with the plethora of new APIs and tools, as developers, a lot of time we have to evaluate technologies we have never touched to enable us (not to mention the folks with the cheque books) to choose the right tool. Broadly speaking, the two things they care about are:
- Will it do what we want?
- What will it cost us?
To answer those questions, we look at the feature lists of potential services, read developer documentation, maybe some source code if possible, check out on-line reviews and, after looking at the pricing page, try to estimate what the on-going costs will be. If it’s a big ticket item, and they have a free trial, we might even try a quick hack to get the service running.
After all that, a decision is made (purely on technical merits, right?) and we hope we made the right call. I say “hope” because a lot of the time, you just don’t know. The feature list might say that it can do what you want, but often only when you try to implement things properly do you discover the little gotchas that tie you up in knots. And that’s when all those cost projections can go out the window. Then you start to wonder if you had gone with (more expensive) Option B whether it might have all been a lot easier – and ultimately a lot cheaper in developer time. But the more time you invest in the integration, the more harder it is to switch providers…
Of course, this not a problem limited to software. Whether you are looking for a house, partner or vacuum cleaner, we almost always have to evaluate the options, make a choice and then figure out how to live with the new addition. Changing your mind in the analogue world is no easier than in the digital.
What about machine learning?
Machine learning is typically handy when you have a lot of variables and want to know which ones are the most significant. At first blush, it seems like there may be a match. Could we mix together the different techniques I mentioned above for evaluating a product (be that an API, SaaS or something else), and spit out a number relating to how hard it is going to be for a given developer (or team) to use?
Here are some of the variables that we could factor in:
- How widely used it is
- Quality of the documentation
- Quality and complexity of the source code
- Number of related questions on developer forums (lots of questions would indicate it is widely used – but could also indicate it is hard to use, especially if the questions are asked by somewhat experienced developers)
- Number of open (and closed issues/PRs) on GitHub.
- Some measure of the number of support tickets and how quickly and well they are responded to. For many commercial products, this info is just not going to be available.
- Maturity of the product, number of releases, and their regularity.
- Ratings and reviews.
- Cost of developers that specialise in this software.
- The skill level and experience of the developer(s) that will most likely be using the product i.e. Probably you!
Note that I haven’t included any information you might find on the pricing page of a product here since that doesn’t generally affect how easy it will be for a developer to use it – although the cost and availability of priority support from the vendor may be a factor. 10 minutes on a live chat could save an afternoon of googling Stack Overflow…
Is this actually feasible?
When we look at the list of factors above, some of them could easily be turned into the sorts of numbers that ML algorithms need. But turning others into meaningful numbers is going to be tough. Maybe we can leverage other sophisticated natural language ML tools to rate and rank some of this stuff (e.g. reviews, documentation) to some degree, but it’s going to take a lot of work – although others may have already done this.
Secondly, for a lot of products we might want to compare, there is going to be missing data in some of these categories. There are lots of “this vs that” comparison sites out there trying to solve this problem in a different way. Even if the categories line up (they rarely do – even for stuff as easy to rank as laptop dimensions from the same manufacturer!), they often have missing data. There are ways round this, but it’s not ideal. By limiting the field of products we want to work with (e.g. packages on npm, WordPress plug-ins say) we could gather data with less gaps – but since those repositories already have some of these kinds of stats available (e.g. npm), we aren’t really adding anything new.
But the biggest problem here is that for supervised ML, we need training examples. We need lots of examples of various products with numerical values for the different variables labelled with “correct” answers relating to how easy the product was to use for different developers. As far as I know, that data simply doesn’t exist – and creating it (along with encoding the other factors) would be a mammoth task! Maybe ML is not such a great fit.
What do other developers think?
One of the most valuable indicators for me about the value of a given product or service is the opinion of other people that I trust. If another developer I know with skills that I rate in a relevant area tells me that they had a great experience with Product X, I am going to believe them over and above Product X’s marketing team’s blurb. In fact, this kind of information pretty much trumps every other factor in the above list. At the very least, it puts the product on my evaluation short-list. Not quite so reliable are the experiences of developers I don’t really know, but there is still value there. Here is the hierarchy:
- My experience (but I don’t have any – which is why I am searching)
- The experiences of developers I know and trust
- The opinions and experiences of developers I don’t really know
- The opinion of Product X’s marketing team
The key thing here is that the better I know someone and how closely their situation matches mine, the more likely I am to trust them. If I could find someone who matches my skills, experiences and the task I have to do in every way, except they have used Product X but I haven’t, then I would follow their recommendations for Product X to the letter.
Of course, there is no-one else in the world with EXACTLY my skills and experiences. But there are a zillion developers out there who match me to varying extents. And this is exactly the sort of thing that machine learning algorithms can solve! We just need to gather the experiences of those zillion other developers along with their opinions of all the different products they used and for which purposes they used them. Armed with that information, such a system could then predict whether any given product would be good for someone with my experiences and opinions of various products to use in the project I am working on.
Needless to say, gathering that kind of data would be rather tricky (!) but still a whole lot easier than the initial idea. The folks at The State of JavaScript have actually attempted something like this – albeit just for the world of JavaScript. They have managed to get data points from 20,000 developers, so this sort of thing is possible.
With that kind of data (and maybe a few extra questions), would it be possible for me to describe myself and the task I am trying to accomplish and then see a report of what the best and worst JavaScript technologies for doing that job would be? That, and maybe the email addresses of five other JavaScript developers who are the most like me!
Edit: These guys say they are already using AI to find the best products (although not yet APIs!)
ZX81 image from http://sebastien.andrivet.com/
