Multi-tasking in Python: Speed up your program 10x by executing things simultaneously

Step-by-step guide to apply threads and processes to speed up your code

An army of workers to help us execute faster (image by Brian McGowan on Unsplash)

This article focuses on speeding up your program by making it do multiple things at the same time. We don’t have to idle while our program waits for and API response e.g; we can do something else in that same time! We’ll also get into how to apply more CPU’s to speed up calculation times. At the end of this article you’ll:

• understand the difference ways of multi-tasking
• know when to apply a which technique
• be able to speed up your own code by using the code examples

Before we begin I’d strongly suggest to check out the article below. It explains how Python works under the hood and why it isn’t as fast as other languages. Also it reveals why isn’t Python multi-threaded to begin with? You’ll have a better understanding of what the problem we’re trying to solve in this article. Let’s code!

Why Python is so slow and how to speed it up

Threads and processes

Python can multi-task in two ways: threading and multiprocessing. On the surface they appear very alike but are fundamentally different. In the parts below we’ll examine both by using two simple metaphors. Our goal is to get an understanding of the differences between threads and processes so that we know when to use which.

Threading is like making breakfast

Let’s make some breakfast: we’ll need a boiled egg, some toast and a cup of coffee so we have 4 tasks:

1. toast bread
2. boil water
3. boil egg
4. switch on the coffee maker

This is what we’re trying to make in as little time as possible (image by Eiliv-Sonas Aceron on Unsplash)