What your developers are using: the application stack
In this unpredictable, chaotic world, it's hard to tell the difference between real and shpiel; but fear not. We'll run through the 4 major parts of shipping and maintaining an application and explore what they are, as well as which products, methods, and services developers use to accomplish them. But first, a few notes:
The process isn't really linear
Most developers aren't building applications from scratch, writing a company's first line of code, or spinning up their first server. Software development works in cycles of create, test, ship, and iterate. Every application starts with someone writing code, but the later parts of the workflow – like deploying code – also require their own writing, testing, and iteration.
Nobody actually knows what they're doing
Contrary to popular belief, there's no "playbook" for building great apps. In fact, most developers openly admit that they don't really know what they're doing: they're figuring it out just like you are, googling when they need help, and asking their colleagues for suggestions and feedback. Then there are the people yelling on HackerNews that they indeed are all-knowing, but don't be afraid of them.
Every company is different
No two apps are the same, but sometimes they rhyme. There's no blueprint for building an app from scratch, and every company approaches it different. Usually, circumstances determine the end product more than thinking from first principles: this engineer had experience with this framework, we randomly had credit for this cloud provider, etc.
With all of that out of the way, let's get started. Developer tasks fall into four semi-discrete steps:
- Writing code: IDE, Language, Frameworks, Package Managers, SDKs
- Deciding on code: Version control, CI/CD
- Deploying code: Infrastructure, Containers and orchestration
- Monitoring code: Log storage, APM tools
Devtools go through horizontal to vertical cycles. Every five years or so, groups of startups try to pile multiple of these steps into one product, arguing that the developer would prefer an integrated platform. At some point they realize that this doesn't work and takes too long to sell, so they try instead to do a really good job automating one particular developer task. As this continues to happen, you'll be able to notice it now.
The easiest way to start is to visualize what a typical modern developer workflow looks like. Let's pretend we're building a web app in React, and deploying it on AWS infrastructure:
- Decide on code using Github and CircleCI
- Deploy code using EC2, RDS, Docker, and Kubernetes
- Monitor code using Datadog and Prometheus
More generally, here's how things stack up.
1) Writing code
Starts when: well, this is kind of the beginning.
Ends when: you think your code is ready to see the light of day.
What it is: Developers write code in software that supercharges their writing abilities with formatting, integrations, and organization. IDE stands for Integrated Development Environment: in practice, it's the same thing as Microsoft Word or Google Docs, but with features that lend themselves more towards building applications.
What developers use: the most popular IDE right now is VSCode, by most estimations. It's free from Microsoft (surprised? Me too), pretty clean to use, and agnostic to which programming language you're using. Another popular provider is JetBrains, makers of the popular IntelliJ IDE for Java. Hometown favorites for those looking for a simpler experience are SublimeText and Atom (these aren't technically IDEs).
What it is: all code eventually gets compiled or translated to something that your computer can understand at a low level. Above that, it's a 2-people-100-opinions situation: there are thousands of programming languages and tens of legitimately popular ones, and there's no universally agreed upon best practice. Some languages are fast, some are easy to write in, and some are powerful. Few are all.
What it is: a lot of applications need to use the same things over and over again. People package up their code into reusable blocks and share them with other developers, so people don't need to do the same work 100 times. Those are called "modules" or "frameworks" or "packages" or "libraries" – these exist across the whole developer workflow (not just writing code) but this is probably the most natural place to cover them.
What it is: modern codebases use a lot of frameworks – it's not uncommon to see hundreds, and those frameworks are built on other frameworks, which are built on other frameworks...you get the idea. Managing these dependencies and making sure things are always compatible and up to date is the job of package managers.
What it is: Software Development Kits (SDKs) are pretty poorly defined, but the simplest way to think of them is a collection of related APIs that you use to build something useful. Some SDKs are free, some are paid, some are open source, and some are provided by companies, to you, so you can build for their product.
What developers use: because SDKs are so use case specific, there's nothing to point to as "the most popular" or anything like that. An example of an SDK is Snapchat's Snap Kit – it's a development package that you use to add Bitmoji, stories, and ads to your app. Along similar lines, the Google Maps SDK lets you add and work with maps in your app, and Apple's ARKit lets you add augmented reality to your iOS apps.
2) Deciding on code
Starts when: you think your code is ready to see the light of day.
Ends when: the code you're ready to deploy is organized and agreed upon.
What it is: version control is a central part of modern application development, and is a little bit like supercharging the "save" button in Excel. When multiple developers are working on a single application, managing how they change things, what gets kept and what gets thrown out, and how changes get smoothly merged in is all under the purview of version control.
What developers use: everyone uses Git for version control – it's an open source protocol from the guy who built Linux, and it's notoriously difficult to understand and use. Github, Gitlab, and Bitbucket offer managed solutions that host your code on a remote server, and add interesting functionality around collaboration and management.
Continuous integration and delivery (CI/CD)
What it is: to quote RedHat, CI "means new code changes to an app are regularly built, tested, and merged to a shared repository." In plain english, it adds automation to the code changes you're making: you can run automated tests (are there security vulnerabilities? Does it break anything?) and merge / deploy your changes rapidly. CI/CD is both a philosophy and a software product category.
What developers use: the two most popular solutions for CI are CircleCI and Jenkins. Technically though, Github and Co. provide a lot of functionality that resembles CI/CD, and I would categorize them under this bucket as well as under version control.
3) Deploying code
Starts when: the code you're ready to deploy is organized and agreed upon.
Ends when: your code is running and users can interact with it.
Your code needs to run somewhere that's not your laptop if you want it to work well. There's an entire ecosystem of these services (it's one of the biggest markets in software, if not the biggest) so we'll need to break it up into a few pieces.
What it is: these are the famous "cloud providers" you keep hearing about. They make their money off of people renting servers from them and deploying software on those servers. Cloud providers offer products across two major categories – compute and storage – and fill in the gaps with other services in between. When you pay for infrastructure as a service (IaaS), you're getting building blocks with very little abstraction built on top.
What developers use: Amazon Web Services is the most popular public cloud. Their main IaaS offering consists of EC2 for compute and EBS for storage, although most mid-large customers use tens of their products. Google Cloud Platform (GCP) and Microsoft Azure have been gaining ground quickly though, and there are smaller IaaS providers like DigitalOcean (my former employer) who target smaller contracts who prioritize simplicity and ease of use.
What it is: because using bare bones compute and storage is kind of annoying sometimes (a lot of operational overhead, maintenance, etc.), companies have built platforms on top of those building blocks that simplify infra management for their customers. These will usually automatically deploy from a Github repo, have extra GUI beef for doing things in a console instead of via an API, and charge a lot more money.
What developers use: the OG PaaS is Heroku, which got acquired by Salesforce back in the day (sad. very sad). Today, the most popular PaaS actually comes from IaaS providers: AWS, for example, has products like RDS, SQS, and Lambda that further abstract infrastructure management for a price. This is also a space where independent companies have succeeded in recent years: MongoDB, Snowflake, and Elastic are good examples in storage, while Confluent and Netlify are showing promise in compute.
Containers and orchestration
What it is: developers don't usually just drop code on a server by itself, because that server's configuration ("environment") is different from that of where they developed the code: their laptop. Today, containers are an increasingly popular way to develop and deploy code: they isolate all of your code and dependencies in a little container so that they'll behave the same no matter where you put them. Container orchestrators take care of creating, destroying, and networking those containers together into a software symphony.
What developers use: the dominant container engine is Docker, which is open source. Docker does develop an orchestration service called Swarm, but its popularity pales in comparison to Kubernetes (ooh! aah!). K8s, as it's affectionately called, was originally developed by Google and released in 2014. It's getting really popular, and major IaaS providers are pushing managed K8s services like AWS's EKS and GCP's GKE.
4) Monitoring code
Starts when: your code is running and users can interact with it.
Ends when: you have visibility into your deployed code and know what it's doing.
Log storage and visualization
What it is: all of the servers that you've deployed your code on – whether you're managing them directly or through a platform – create tons of data about what's going on. Those are usually called logs (although technically, a log is just a type of file), and they're incredibly valuable to developers, because they contain the information they need about why things went wrong, where failure happened, and how they can fix it. Log data is voluminous and difficult to parse, and products are emerging to organize them and "glean insights."
What developers use: the most popular log management solution is Elasticsearch – it's open source software built off of Apache Lucene that lets you store, index, and most importantly, search your log data. Elastic is the (now public) company behind the software, and they offer managed versions hosted on public clouds, as well as other open source integrations like Kibana for visualization.
What it is: application performance management (APM) is the holistic management of your deployed applications (logs are a part of the broader umbrella). While cloud providers often give you some visibility into how your servers are performing, developers often want more power and granularity. There are some open source solutions here, but the big winners have been SaaS providers.
What developers use: NewRelic and Datadog are the two most popular APM (monitoring as a service) vendors. You basically just hook these up to your cloud provider(s) and you're good to go with metrics, alerts, and visualizations. There are good open source solutions as well: Prometheus and Grafana are monitoring servers built for time series and metrics storage.