Deploy a Celery Worker

In the course of processing web requests, you may have to offload tasks to an asynchronous, background process (typically called a worker). Render makes this easy to do through Background Workers, which work like any other service.

For this quick start, we’ll use Celery, a popular distributed task queue for Python with a Flask frontend to submit sample tasks. We will also deploy Flower to view tasks as they’re processed by Celery.

Celery can use Redis for its processing backend, so we’ll set up a Redis service first.

Deploy to Render

There are two ways to deploy your application on Render, either by declaring your services within your repository in a render.yaml file, or by manually setting up your services using the dashboard. In the following steps, we will walk you through both options.

We will be deploying four services:

1. Flask Web Service that lets you create new asynchronous tasks for Celery to process
2. Celery Background Worker that will process tasks
3. Render Redis instance as the Celery Broker that handles reliable communication of tasks between clients (Flask Web Service) and workers (Celery Background Worker)
4. Flower Web Service to monitor Celery. Flower is a web based tool to monitor and administer Celery. You can use it to view tasks and results

Use render.yaml to Deploy

1. You can fork our render-examples/celery on GitHub which contains a render.yaml file that sets up the four services.

2. On the Render Dashboard, go to the Blueprint page and click the New Blueprint Instance button. Select your repository (after giving Render the permission to access it, if you haven’t already). Or alternatively, you can click the Deploy To Render button in the Readme of the forked repo.

3. In the deploy window, click Approve.

That’s it! Your app will be live on your .onrender.com URL as soon as the build finishes. Try out the web services using their Render URLs and play around with the small application.

Note: Flower’s dashboard is not meant for public consumption, so we recommend securing it by adding authentication as described in the Flower docs.

Deploy manually

1. Create a new Redis server with the following settings using the Redis deployment guide and make a note of the Internal Redis URL which will look like redis://red-xxxxxxxxxxxxxxxxxxxx:6379.

   Maxmemory Policy	noeviction (recommended for queues/persistence)
   Plan	Starter

   We choose the noeviction maxmemory policy to ensure Redis does not delete scheduled tasks upon reaching memory capacity by preventing the creation of new tasks. Choosing allkeys-lru instead would allow the creation of new tasks by discarding the oldest potentially incompleted tasks.

   We choose the Starter instance type as it is the smallest instance type with persistence, which will ensure that we retain tasks when Redis restarts.

2. Fork render-examples/celery on GitHub and create a new Background Worker on Render using your fork.

   Runtime	Python 3
   Build Command	pip install -r requirements.txt
   Start Command	celery --app tasks worker --loglevel info --concurrency 4

   Add the following environment variable to the background worker:

   Key	Value
   CELERY_BROKER_URL	<Internal Redis URL>, the Internal Redis URL from step 1.

3. Create a new Web Service for Flask using the same repo you created in Step 2.

   Runtime	Python 3
   Build Command	pip install -r requirements.txt
   Start Command	gunicorn app:app

   Add the following environment variable to the web service:

   Key	Value
   CELERY_BROKER_URL	<Internal Redis URL>, the Internal Redis URL from step 1.

4. Once the Flask web service is live, navigate to its URL. Fill out and submit the form to create a new Celery task.

5. Create a new Web Service for Flower using the same repo you created in Step 2.

   Runtime	Python 3
   Build Command	pip install -r requirements.txt
   Start Command	celery flower --app tasks --loglevel info

   Add the following environment variable for the web service:

   Key	Value
   CELERY_BROKER_URL	<Internal Redis URL>, the Internal Redis URL from step 1.

   Note: Flower’s dashboard is not meant for public consumption, so we recommend securing it by adding authentication as described in the Flower docs.

6. Once the Flower web service is live, navigate to its URL and browse the tabs. You can view information such as how many worker processes are running and how many tasks have been completed.

You can also use a Celery background worker from Django. See our Django quick start and the Django integration guide in Celery docs.

See Specifying a Python Version if you need to customize the version of Python used for your app.