How to Install Apache Airflow in Docker for Efficient Workflow Management

Apache Airflow is a powerful open-source platform used for orchestrating complex data workflows. By containerizing it with Docker, you can ensure portability, reproducibility, and easier management of your Airflow instances. In this tutorial, we’ll walk you through the process of how to install Apache Airflow in Docker, enabling you to streamline your workflow management.

Before we dive into the installation process, make sure you have the following prerequisites in place:

1. Docker installed on your system.
2. Basic knowledge of using the command line.

Set Up a Working Directory

Start by creating a directory where you’ll store the necessary files and configurations. Open your terminal and execute the following commands:

mkdir airflow-docker
cd airflow-docker

Create Docker Compose File

Inside the airflow-docker directory, create a docker-compose.yaml file using your preferred text editor. This file will define the services required for Airflow:

version: '3'
services:
  webserver:
    image: apache/airflow:latest
    ports:
      - "8080:8080"
    environment:
      - AIRFLOW__CORE__SQL_ALCHEMY_CONN=sqlite:////usr/local/airflow/airflow.db
      - AIRFLOW__WEBSERVER__SECRET_KEY=your_secret_key
    volumes:
      - ./dags:/usr/local/airflow/dags
      - ./logs:/usr/local/airflow/logs
      - ./plugins:/usr/local/airflow/plugins
  scheduler:
    image: apache/airflow:latest
    volumes:
      - ./dags:/usr/local/airflow/dags
      - ./logs:/usr/local/airflow/logs
      - ./plugins:/usr/local/airflow/plugins

Customize Airflow Configuration

Create an airflow.cfg file in the airflow-docker directory to customize Airflow’s configuration:

wget https://raw.githubusercontent.com/apache/airflow/main/airflow/config_templates/default_airflow.cfg -O airflow.cfg

Edit the airflow.cfg file to your preferences. For example, you can change the executor to CeleryExecutor for distributed task execution.

Start Airflow Back in the terminal, navigate to the airflow-docker directory and run the following command:

docker-compose up

Access Airflow Web UI

Open your web browser and go to http://localhost:8080 to access the Airflow Web UI. Log in using the credentials you specified in the docker-compose.yaml file.

Congratulations! You’ve successfully installed Apache Airflow in Docker, allowing you to efficiently manage and orchestrate your data workflows. This setup provides flexibility, scalability, and ease of use for your workflow automation needs. Explore Airflow’s extensive features and customize your workflows to streamline your data processing tasks.