airflow.providers.apache.arrow.hooks.adbc

Classes

AdbcHook

General-purpose Airflow hook for interacting with databases via the Arrow Database Connectivity (ADBC) standard.

Functions

fetch_all_handler(cursor)

Return results for DbApiHook.run().

replace_placeholders(sql, placeholder)

Module Contents

airflow.providers.apache.arrow.hooks.adbc.fetch_all_handler(cursor)[source]

Return results for DbApiHook.run().

airflow.providers.apache.arrow.hooks.adbc.replace_placeholders(sql, placeholder)[source]
class airflow.providers.apache.arrow.hooks.adbc.AdbcHook(*args, schema=None, log_sql=True, **kwargs)[source]

Bases: airflow.providers.common.sql.hooks.sql.DbApiHook

General-purpose Airflow hook for interacting with databases via the Arrow Database Connectivity (ADBC) standard.

This hook enables connections to any database supported by an ADBC driver, using the Python ADBC driver manager. It provides methods for executing SQL queries, inserting rows in bulk, and handling Arrow-native data transfers.

Key Features:

  • Supports chunked and batched inserts using Apache Arrow RecordBatches for efficient data transfer.

  • Discovers and loads ADBC drivers dynamically based on connection extras or naming conventions.

  • Handles dialect-specific connection URIs and driver entrypoints.

  • Integrates with Airflow’s connection system (conn_id, extras, etc.).

  • Provides custom placeholder replacement for parameterized SQL queries.

  • Loads rows with the driver’s bulk ingest (adbc_ingest, e.g. COPY on PostgreSQL) by default, or binds them with DBAPI executemany (fast_executemany=False).

  • Exposes configuration via connection extras: driver, entrypoint, db_kwargs, conn_kwargs, dialect.

Connection Extras:

  • driver: Name of the ADBC driver to use (e.g., “adbc_driver_postgresql”).

  • entrypoint: Optional Python entrypoint for the driver.

  • db_kwargs: Driver-specific database initialization options passed to AdbcDatabase. Keys and value types are defined by the driver (e.g., "username", "password" for the PostgreSQL driver). Do not put ADBC connection options here — they belong in conn_kwargs.

  • conn_kwargs: ADBC connection options as string key-value pairs. Keys must use the canonical dotted ADBC option names, for example: "adbc.connection.autocommit": "true", "adbc.connection.read_only": "true", "adbc.connection.current_catalog": "my_catalog", "adbc.connection.current_db_schema": "my_schema". Short names such as autocommit or read_only are not recognized by ADBC drivers and will raise NotSupportedError.

  • dialect: SQL dialect name (default: “default”).

Example usage:

hook = AdbcHook(adbc_conn_id=”my_adbc_conn”) records = hook.get_records(“SELECT * FROM my_table”)

For more details, see:
conn_name_attr = 'adbc_conn_id'[source]
default_conn_name = 'adbc_default'[source]
conn_type = 'adbc'[source]
hook_name = 'ADBC Connection'[source]
supports_autocommit = True[source]
classmethod get_ui_field_behaviour()[source]

Get custom field behaviour.

property uri: str[source]

Return the URI handed to the driver.

A host that already is a URI or a path (postgresql://user:pass@host:5432/db, file::memory:, :memory:, /data/app.db) is passed through unchanged. Otherwise the URI is built as <dialect>://login:password@host:port/schema from the connection’s fields. Airflow’s own connection URI is of no use here: its query string carries the extras, which drivers such as libpq refuse as unknown parameters.

property driver: str[source]
property entrypoint: str | None[source]
property db_kwargs: dict[source]
property conn_kwargs: dict[source]
property dialect_name: str[source]
get_conn()[source]

Return a connection object.

set_autocommit(conn, autocommit)[source]

Set autocommit on the ADBC connection.

The DBAPI attribute conn.autocommit has no effect on the underlying ADBC driver; the real lever is the adbc.connection.autocommit option. This override applies the option at the driver level, then calls super() to keep the Python-level bookkeeping attribute in sync so that the base class get_autocommit() and run() commit-guard behave correctly.

get_records(sql, parameters=None)[source]

Execute the sql and return a set of records.

Parameters:
insert_rows(table, rows, target_fields=None, commit_every=1000, replace=False, *, executemany=False, fast_executemany=True, autocommit=False, **kwargs)[source]

Insert a collection of tuples into a table.

Rows are converted to Arrow RecordBatch objects and inserted in chunks; each chunk (of size commit_every) is done in a new transaction.

Parameters:

  • table – Name of the target table, optionally prefixed with its schema (schema.table).

  • rows – The rows to insert into the table

  • target_fields – The names of the columns to fill in the table

  • commit_every – The maximum number of rows to insert in one transaction. Set to 0 to insert all rows in one transaction.

  • replace – Whether to replace instead of insert. Replacing always binds the rows to the generated statement, because bulk ingest cannot express it.

  • executemany – Kept for signature compatibility with insert_rows(); rows are always sent as one Arrow batch per chunk.

  • fast_executemany – If True (default), load each batch with the driver’s bulk ingest (cursor.adbc_ingest, COPY on PostgreSQL). If the driver does not implement bulk ingest, falls back to binding. If False, bind each batch to the INSERT statement with cursor.executemany.

  • autocommit – What to set the connection’s autocommit setting to before executing the query.

Was this entry helpful?