The Alternative Implementation Problem

In this post, I want to talk about a dynamic that I’ve seen play itself over and over again in the software world. In fact, I would venture a guess that this kind of situation probably happens in the hardware world as well, but I’ll speak about software systems since this is where my experience lies. This discussion is going to touch a bit on human psychology, and outline a common trap so that you can hopefully avoid getting stuck in it.

Most of my career, both in academia and industry, has been spent trying to optimize dynamically-typed programming languages. During my master’s I worked on a simple optimizing JIT for MATLAB. For my PhD I worked on a JIT for JavaScript. Today I’m working on YJIT, an optimizing JIT for Ruby which has now been upstreamed into CRuby.

During my PhD, while working on my own JavaScript JIT, I read many papers and blog posts about JIT compilers for other dynamic languages. I read about the design of HotSpot, Self, LuaJIT, PyPy, TruffleJS, V8, SpiderMonkey, and JavaScriptCore among others. I also had the chance to interact with and meet face to face with many of the really smart people behind these projects.

One of the things that struck me is that the PyPy project was kind of stuck in a weird place. They had developed an advanced JIT compiler for Python which could produce great speedups over CPython. By all accounts many people could benefit from these performance gains, but PyPy was seeing very little use in the “real world”. One of the challenges that they faced is that Python is a moving target. New versions of CPython come out regularly, always adding many new features, and PyPy struggles to keep up, is always several Python versions behind. If you want you Python software to be PyPy-compatible, you’re much more limited in terms of which Python features you use, and most Python programmers don’t want to have to think about that.

Reading about LuaJIT, I found that it was and still is highly regarded. Many people regard its creator, Mike Pall, as an incredible programmer. LuaJIT offers great performance gains over the default, interpreted Lua implementation, and has seen some decent adoption in the wild. However, I again saw that there are a number of Lua users who do not want to use LuaJIT because the Lua language keeps adding new features and LuaJIT is several versions behind. This is a bit strange considering that Lua is a language that is known for its minimalism. It seems like they could have made an effort to slow down the addition of new features and/or coordinate with Mike Pall, but this wasn’t done.

Almost 4 years ago, I joined Shopify to work on Ruby. For some reason, the space of Ruby JITs has been particularly competitive, and there had been a number of projects to build Ruby JITs. The TruffleRuby JIT boasted the most impressive performance numbers, but again, had seen limited deployments. There were some practical reasons for this, the warm up time of TruffleRuby is much longer than that of CRuby, but I also saw a similar dynamic to that of PyPy and LuaJIT, where CRuby kept adding features, and TruffleRuby contributors had to work hard to try and keep up. It didn’t really matter if TruffleRuby could be quite a bit faster, because Ruby users would always view CRuby as the canonical implementation, and anything that wasn’t fully compatible wasn’t seen as worthy of consideration.

Hopefully, at this point, you see where I’m going with this. What I’ve concluded, based on experience, is that positioning your project as an alternative implementation of something is a losing proposition. It doesn’t matter how smart you are. It doesn’t matter how hard you work. The problem is, when you build an alternative implementation, you’ve made yourself subject to the whims of the canonical implementation. They have control over the direction of the project, and all you can do is try to keep up. In the case of JITted implementations of traditionally interpreted languages, there’s a bit of a weird dynamic, because it’s much faster to implement new features in an interpreter. The implementers of the canonical implementation may see you as competition they are trying to outrun. You may be stuck trying to ice skate uphill.

Almost 4 years ago, with support from Shopify, two dedicated colleagues and I started a project to build YJIT, yet another Ruby JIT. The difference is that we made the key choice to build YJIT not as an alternative implementation, but directly inside CRuby itself. This came with a number of design tradeoffs, but critically, YJIT could be 100% compatible with every CRuby feature from the start. YJIT is now the “official” Ruby JIT, and is deployed at Shopify, Discourse and GitHub among others. If you’ve visited github.com today, or any Shopify store, you’ve interacted with YJIT. We’ve had more success than any other Ruby JIT compiler so far, and compatibility has been key in achieving this.

You may read this and think that the key lesson of this post follows the old adage that “if you can’t beat them, join them”. In some ways, I suppose it does. What I want to say is that if you start a project to try and position yourself as an alternative but better implementation of something, you are likely to find yourself stuck in a spot where you’re always playing catch up and living in the shadow of the canonical implementation. The canonical project keeps evolving, and you have no choice but to follow along with limited decisional power over where your own project is headed. That’s no fun. You may have better luck trying to join up with the canonical implementation instead. However, that’s only part of the answer.

In the Ruby space, there is also Crystal, a Ruby-like language that is statically compiled with type inference. This language is intentionally not Ruby-compatible, it has chosen to diverge from Ruby, but has still seen limited success. I think this is interesting because it gives us a broader perspective. Rubyists don’t like Crystal because it’s almost-Ruby-but-not-quite. It looks like Ruby, syntactically, but has many subtle differences and is very much incompatible in practice. This just confuses people, it breaks their expectations. Crystal probably would have had better luck if it had never marketed itself as being similar to Ruby in the first place.

Peter Thiel has a saying that “competition is for losers”. His main point is that you shouldn’t put yourself in a position where you’re forced to compete if you don’t have to. My advice to younger programmers would be, if you’re thinking of creating your own programming language, for example, then don’t go trying to create a subset of Python, or something superficially very close to an existing language. Do your own thing. That way, you can evolve your system at your own pace and in your own direction, without being chained by expectations that your language should have to match the performance, feature set, or library ecosystem of another implementation.

I’ll finish with some caveats. What I said above applies when you have a situation where there is a canonical implementation of a language or system. It doesn’t apply in a space where you have open standards. For example, if you want to implement your own JSON parser, there is a clearly defined specification that is relatively small and doesn’t evolve very fast. This is very much something you can achieve. You also have a situation where there are multiple browser-based implementations of JavaScript. This is possible in part because there is an external standard body that governs the JS specification, and the people working on the JS standard understand that JIT-compiled implementations are critical for performance and guide the evolution of the language accordingly. They are not in the game of adding many new features as fast as possible.