Distributed Python

Build your Celery (the distributed task queue, not the vegetable) skills with weekly tutorials,
how-tos and articles about common gotchas.

June 26, 2018

Estimated reading time: 4 minutes

Distributed Monte Carlo simulation with Celery chords

Posted by Bjoern Stiel

Celery chords are one of the six Celery workflow primitives. A Celery workflow defines the order in which individual Celery tasks are executed asynchronously. A chord consists of a Celery group (called chord header) and a callback. A group is a list of tasks that are executed in parallel. After all group tasks have finished successfully, the callback is executed,...


June 12, 2018

Estimated reading time: 5 minutes

How to Dockerize a Celery App With Django And RabbitMQ

Posted by Bjoern Stiel

Docker simplifies building, testing, deploying and running applications. Docker allows developers to package up an application with everything it needs, such as libraries and other dependencies, and ship it all out as one package. This package, which is essentially a build artifact, is called a Docker image. As a general Docker design principle, you should follow the 12factor design principles....


June 05, 2018

Estimated reading time: 3 minutes

Dynamic Task Routing in Celery

Posted by Bjoern Stiel

In our previous blog post we looked into custom queues and task routing. We had to configure per task which queue we want Celery to task a route to. While this approach works well for a simple setup, it does not scale well for applications and micro-services where many Celery tasks need to be routed to a number of different...


May 29, 2018

Estimated reading time: 2 minutes

Task Routing in Celery

Posted by Bjoern Stiel

By default, Celery routes all tasks to a single queue and all workers consume this default queue. With Celery queues, you can control which Celery workers process which tasks. This can be useful if you have a slow and a fast task and you want the slow tasks not to interfere with the fast tasks. Or if you need to...


Get distributed Python tips straight to your inbox