There is a better way to deal with computers. Hackers have been developing and using it for at least 40 years: it is the way of the command line, of the conversation with the computer in languages that are neither human nor machine, but a carefully crafted middle ground. Hackers, by the way, is how we call people who know well their tools, who dare to explore, and who need to build things. In the words of Paul Graham, “To programmers, ‘hacker’ connotes mastery in the most literal sense: someone who can make a computer do what he wants”.
You’ll find the tools of the hackers in Linux, in macOS, in a Raspberry Pi, and running most of the web. By learning them you’ll become much more productive, and a realm of things that were previously unthinkable will become possible. You will became able to talk your computer into doing what you want. And you will be mastering the tools that scientists, programmers, engineers, statisticians, mathematicians and analysts all over the world are using.
Why this booklet
This book focuses on the fundamentals of the UNIX toolset. Whether building a website, programming a Raspberry Pi, or analyzing data, developers need to master the same core tools that professionals use daily.
The most valuable lesson is to explore, to prototype, and not to be afraid of trying things out. From editing files to organizing projects, from version control to deployment — these tools form the foundation of modern software development.
The UNIX tools and environment
Over the last three decades the UNIX environment and toolset has become increasingly prevalent in research institutions and development labs. Programmers find it powerful and efficient, and they often go for it when given the choice; scientists find its tools the best suited for most computing tasks.
It used to be that programmers had to develop in whatever operating system their users had (Windows, in most cases). But the web runs mostly in web servers and web browsers; most servers are UNIX-like, and the browsers need to work everywhere. That has given the people programming for the web the freedom to chose their environment, and many have chosen Linux and macOS, both flavors of UNIX. When people learn how to use them, as you will have by the end of this book, they find themselves much more productive (and much more powerful) than they used to be. There’s no going back.
Scientists have also gravitated towards UNIX-like environments. They are rock solid, can be made very fast, and the readily available set of tools allow them to get the job done. If your job requires data analysis and exploration, or building models, it will pay you to be at home in a UNIX-like environment. If you want to work in a world-class development or research organization you’ll most likely be expected to be proficient using it.
The rise of open source
The UNIX toolset has been the foundation of professional development for decades. What started in research labs and universities has become the standard across the industry. Today, whether you’re using Linux, macOS, or WSL on Windows, you have access to the same powerful tools that run most of the internet’s infrastructure.
The tools covered here are, with few exceptions, the same ones in use today. It used to be that only hard-core CS majors and mathematicians would be familiar with them; nowadays, many developers from various science disciplines are at home in Linux, and expect to be able to work with the UNIX toolset.
What you will learn
By the end of this booklet you’ll be familiar with the command line interface to your computer (and to your server) and the classic UNIX tools you can run on them. You’ll know how to edit files using Emacs and Vi, the two classic UNIX editors. You’ll be organizing your files in an efficient manner and managing them using git and github. You’ll have learnt how to go about writing programs in Python, and how to approach building software projects systematically.
The scope is large and the coverage will be shallow. I shall not attempt to write a full reference to these tools: only to give you enough of a push to get you going.
There are many things to learn, but none is of great complexity. Some, like the commands your editor understands and a bunch of UNIX utilities, you need to interiorize in muscle memory until they become automatic. Others you’ll just have to know they exist, and where to look them up.
The choices presented here reflect common professional preferences. They are not particularly controversial: many professional developers nowadays are using similar setups to the ones shown here. Some recommendations are fundamental: you should use a UNIX-style environment like Linux or macOS. Others are a matter of personal taste: some prefer Emacs over Vi for editing, but both are excellent choices. Some probably reflect fashion: the fact that most people are using git instead of mercurial for version control is probably due to the success of github, for example. The important thing is that you know what version control is, and that you make it part of your work-flow.
Some parts of the toolset are more stable than others. The core UNIX tools have remained remarkably stable — commands like grep, sed, awk, and editors like Vim and Emacs work essentially the same today as they did decades ago. While version control has evolved from RCS through CVS and subversion to git, Each one of them has been a large improvement over its predecessor. But today git is here to stay, as are the classic editors and tools, and it’s well worth your time to learn them well.
You should learn by doing, and you should explore. Do not be afraid of trying things out. Do not concern yourself too much with breaking things. Thanks to git —and to the good working practices you will be acquiring— you’ll always be able to go back in time and recover an earlier working version. The first law of software development is that you don’t understand your problem until you have implemented a solution, so don’t be shy about it.
The environment
The tools covered in this book work across all UNIX-like systems: Linux, macOS, and even Windows through WSL (Windows Subsystem for Linux). Whether you’re using a high-end MacBook, a Linux workstation, or a Raspberry Pi, the commands and concepts remain the same.
The development process
Prototype and adapt
It is in the nature of the design of complex things that you never know in advance what you’ll really want it to look like when you are done. Surely you have an idea of what problem you want to solve, and how you are going to do it; but as you build it you’ll become aware of all sorts of complexities and crevices hiding beneath the surface, and you’ll adapt, and you’ll learn about the problem you are trying to solve. In most cases you’ll end up with something that is not what you first envisioned. And that is a good thing. You’d otherwise have missed an excellent opportunity to learn.
This fluidity of the design and construction process is most apparent in building software, but it was ever thus. In more traditional engineering disciplines it happened during the design part of the development cycle, in the form of changing specs and design drawings, long before the concrete was cast. What makes software special is that the drawings are also the product.
Start small, be nimble
Don’t set up to build a new facebook. Don’t be concerned about scalability, or millions of users. Focus instead of testing your ideas, on having online something you can learn from as soon as possible, and on building it in such a way that you can change it fast.
The tools you’ll be learning about in this book will enable you to go back in time to an earlier version of your projects; explore different options by keeping different versions of your work and moving between them; put them online fast and easy, and setup and manage your servers; edit your work efficiently, and manage it properly. The goal is to help you do the things that need doing and move along.
The Way of the Future
The UNIX tools and environment remain the foundation of modern computing. While new utilities and programming languages continue to appear, they build upon and integrate with the UNIX philosophy. Learning these tools provides a solid foundation for any technical career - from software development pto data science, from system administration to research.