How do you select the right technology for your platform?

It's the people, not the tech!

When designing applications and platforms people think they need to compare and assess technologies against each other. They should be comparing themselves against the technology.

Geeks love to obsess about technology. This is not just limited to computers. As a teenager I overheard conversations comparing the latest car models from Ford and Holden (a now defunct Australian car brand), more recently it was discussions of who is aligned with whom in the cloak-and-dagger intrigue of Westeros, and detailed hypotheses about why DrDisrespect is no longer on Twitch!

Simply, geeks love to obsess. They love to delve into the minutia and details, whether it be engines: V6 vs. V8; supercharger, turbocharger, or both; or what did Varys in fact hear in the flames? If you replace cylinder with CPU core you could be forgiven for mistaking these as the one conversation.

Every technology has its advantages and disadvantages: and it usually does not matter. Any programming language may have its own idioms or language features that make it more suitable to a task. Every framework has its design and implementation details. Ultimately, this almost never matters. The most difficult part of incorporating a technology into a process or platform is not about overcoming technological limitations but human ones.

Today we programme by poking. We are no longer building platforms from the ground up, developing every individual piece. We have free operating systems, languages, libraries, commodity hardware, infrastructure, and managed services. The only thing left is to connect that into a product. A developer’s work is twofold: to figure out how these pieces work and to tie them together.

You will be better at the first part if you are smarter or harder working than others: taking the time to read manuals and documentation or being able to intuit, learn, and remember the design and interfaces of platforms. For the rest the process is the same, but just takes longer. It is the way that experience can sometimes help where raw talent lacks.

Python and JavaScript are starting to become the lingua franca of backend development for the web, Python more so for data engineering and machine learning. They are almost certainly not the best tools for the job. Both have a late-binding type system (often seen as a curse, rather than a blessing), Python’s multi-threaded code cannot make use of more than 1 CPU core at a time, the list of criticisms goes on yet is also irrelevant!

These languages are accessible to most developers. The additional cost to developer productivity from using good but not perfect tools is eclipsed by the loss of productivity that comes from using an unfamiliar tool, no matter how effective once mastered.

JSON beat XML as the de facto standard for data exchange on the web and the Internet at large because it was easier for a developer to read JSON than XML. JSON could be copy/pasted into a JavaScript source file, or Python with very trivial changes. Compare XML which is an entirely new language (assuming no familiarity with HTML) and harder for programmers to read no matter their experience.

When languages or frameworks are similar the difference will only show at the margin. The LAMP (Linux Apache MySQL PHP/Perl/Python) stack ran the web during the 90s and 2000s and is still popular. NoSQL databases have their advantages, but selecting the database engine is not going to make or break your application. If MySQL is able to run the web it will probably work for you. The advantage of a NoSQL database at scale will only matter when an application has grown to need it. The skills to query and interact with the database are required from day one. The ease of directly adding JSON documents is overshadowed if the API and query language are new to the developer. If you already have experience with MySQL, defining table schemas and data relationships, the decision is easy.

Some language features are a nice to have. Pattern matching in Scala and Haskell is a great feature not available in Python (but coming soon). The inconvenience of a cumbersome alternative adds minutes or hours to development, where unfamiliarity with Scala or Haskell represents a complete roadblock to getting started.

Software's problem is that it is so difficult to construct. This problem is the reason why Excel still runs so much of the corporate and accounting world. Spreadsheets are a remarkable tool that allow intuitive access to numerical computation. From within minutes you can use all of the basic computation skills you learnt using calculators. Further, the DSL (Domain Specific Language) of spreadsheets, essentially just function application, is simple and allows the user’s skills to grow as they learn new functions.

These ideas are shared by Peter Norvig, forwarding anecdotes about Lisp vs. C++. He even goes on to share his preferences for Lisp, but concedes that alternatives are better suited to large projects approaching 100s of developers. A project that has grown to 100s of contributors is remarkable. Nonetheless the principles still hold, in fact moreso. The cost for 100s of developers to learn or adapt to a foreign language is a much bigger hurdle and expense than the inefficiencies that would be brought along from choosing C++ over Java/C#/Go/else.

A domain will often have its popular languages and 3rd party packages. Python already has multiple machine learning, data analysis, web, and other frameworks and modules. No matter its language features it is going to be the correct choice for a lot of domains. Python is easy to recommend. More controversial is to use C/C++ for a desktop application. These languages have mature GUI and other libraries. Even C, for those familiar, makes sense as the language to build desktop applications where most of the work has already been done by the GUI toolkit.

The technological hurdles have been surmounted. The task is to connect together these technologies. If you are deciding between languages or frameworks prioritise the following:

• Choose what you know. If you are already familiar with a tool then you immediately skip the first of the two jobs when programming by poking. You are done with poking and can immediately get to developing.

• Choose what your neighbours know. No matter how proficient you are at working through manuals and documentation, scouring Stack Overflow for answers, it is always easier to look to the person next to you with immediate answers, often because they have just solved your problem themselves.

• Choose the technology that has already done the work for you. Beyond esoteric language features look for a language in which the important protocols and algorithms already have implementations. Reinventing the wheel in the right language is less productive than using someone else's wheel in the wrong one.

When new technology is created by tying together existing tools and products—when the tools are already here and available to use—the difficult part rests in learning how they work and how to use them. This difficulty is apparent in how clearly useful and widely applicable the skill of software engineering is, and how rare that skill is in the population.