On macOS, Airflow tasks or the DAG processor sometimes die with a segmentation fault (SIGSEGV) or an abort, often only now and then. The code of the task is rarely the problem. A child process created with fork() touches a macOS system library that is not safe to use after a fork, and macOS kills it. Two environment variables, set before Airflow starts, fix most cases.

Copy and paste

export NO_PROXY="*"
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
export PYTHONFAULTHANDLER=true      # print a Python traceback if it still crashes
airflow standalone

Set them in the same terminal before starting Airflow, or add them to ~/.zshrc. They have no effect on processes that are already running, so stop Airflow and start it again.

What the crash looks like

In Airflow 3 the task log contains a banner:

******************************************* Received SIGSEGV *******************************************
SIGSEGV (Segmentation Violation) signal indicates Segmentation Fault error which refers to
an attempt by a program/library to write or read outside its allocated memory.

In Airflow 2 the message was Task exited with return code Negsignal.SIGSEGV. The macOS-specific signs are messages that mention +[NSNumber initialize], _scproxy, or a class that “may have been in progress in another thread when fork() was called”.

Why it happens

Apple’s Objective-C runtime and system frameworks are not safe to use in a process created by fork() unless the process immediately starts a new program with exec(). Airflow forks to run tasks, parse DAG files and run the triggerer. When the child then resolves a host name, looks up the system proxy settings or sets up SSL, it reaches those frameworks and crashes.

Airflow 3.3 runs tasks with fork followed by exec on macOS, which avoids the problem for task execution. The note in its source code says that the DAG processor and the triggerer still use a plain fork, so on 3.3 the crash is more likely while DAG files are parsed or in deferred tasks.

The fixes, in order

  1. Upgrade. Airflow 3.3.2 has the fork-and-exec change for tasks. On older versions every task is exposed.
  2. NO_PROXY="*". When a Python library looks up proxy settings on macOS, it asks the system through _scproxy, one of the usual triggers. With NO_PROXY="*" that lookup is skipped. It also bypasses any proxy, so do not use it on a network that needs one.
  3. OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES. It turns off the Objective-C check that kills the child process. It must be in the environment before the process starts. Python developers point out that it does not make every fork safe, so treat it as a workaround.
  4. PYTHONFAULTHANDLER=true. When a crash remains, Python prints the traceback of the thread that crashed, which names the library involved. Airflow’s own crash message suggests this step.
  5. Check the architecture. On Apple Silicon, python -c "import platform; print(platform.machine())" should print arm64. A Python running under Rosetta, or libraries with C extensions built for the other architecture, can crash for reasons unrelated to fork.
  6. Run Airflow in Linux. Airflow supports only Linux in production. If nothing else helps, run the local environment in Docker or in a Linux virtual machine.

The causes and fixes come from Airflow’s crash message and source code (3.3.2) and from the reports linked below. I could not reproduce the crash on Airflow 3.3.2.

References