Airflow provides a built‑in object storage backend in the common‑io provider package. To enable it, set the xcom_backend configuration:
Historically, Airflow allowed XCom values to be serialized using Python's pickle module, which could lead to security vulnerabilities and version incompatibilities. Modern Airflow , and pickling support is deprecated. Always ensure your XCom values are JSON‑serializable unless you have a very good reason to do otherwise.
To make Airflow use your custom backend, configure the environment variable or update your airflow.cfg : airflow xcom exclusive
# In airflow.cfg or helm chart xcom_backend = airflow.providers.google.cloud.xcom.backends.cloud_storage.CloudStorageXCom
In this example, task1 shares customer data through XCom, and task2 retrieves the data and processes it. Airflow provides a built‑in object storage backend in
XCom communication relies on an explicit key-value schema tied directly to a specific dag_id , task_id , and run_id .
task2 = BashOperator( task_id='task2', bash_command='echo task_instance.xcom_pull("greeting") ', dag=dag, ) task2 = BashOperator( task_id='task2'
Automated cloud databases can hit storage limits quickly, locking up the entire cluster.