ADBC connection¶

The ADBC connection type enables connection to any database that has an Arrow Database Connectivity (ADBC) driver, such as PostgreSQL, SQLite, DuckDB, Snowflake, BigQuery, and Flight SQL servers.

Connections of this type are used by AdbcHook, which builds on top of DbApiHook and transfers data as Apache Arrow RecordBatch objects for efficient, zero-copy bulk loads.

Default Connection ID¶

adbc_default

Configuring the Connection¶

Host or URI (Host field)

Either the host name of the database, or the complete URI the driver expects (e.g. postgresql://user:pass@host:5432/db or file::memory:). A value that contains :// or ::, or that starts with /, : or file:, is passed to the driver unchanged.

Login / Password / Port / Schema

Used when the Host field holds a host name: the hook then builds the URI <dialect>://login:password@host:port/schema from these fields. The extras never end up in that URI.

Extra (JSON)

A JSON object with the following recognized keys:

driver (string)

Python package name of the ADBC driver to load, e.g. "adbc_driver_postgresql" or "adbc_driver_sqlite". When omitted the hook derives the name from the dialect as adbc_driver_<dialect>.

entrypoint (string, optional)

Fully-qualified Python symbol used as the driver entrypoint, e.g. "adbc_driver_sqlite.dbapi.connect". Most drivers do not need this; it is only required when the automatic entrypoint discovery built into adbc_driver_manager cannot locate the function.

dialect (string, optional)

SQL dialect name used to build the URI scheme and derive the default driver name. Defaults to "default".

db_kwargs (object, optional)

Keyword arguments forwarded verbatim to the ADBC AdbcDatabase constructor (database-level connection settings). uri is always injected automatically from the Host field and must not be repeated here. Common keys:

Key	Type	Description
`username`	string	Database user name (alternative to encoding it in the URI).
`password`	string	Database password (alternative to encoding it in the URI).
`adbc.sqlite.query.batch_rows`	int	SQLite: number of rows fetched per Arrow batch.

All other keys are driver-specific. Consult the documentation for your ADBC driver for the full list. See also the ADBC driver documentation.

conn_kwargs (object, optional)

Keyword arguments forwarded verbatim to the ADBC AdbcConnection constructor (connection-level settings within the already-open database). Common keys defined by the ADBC specification:

Key	Type	Description
`adbc.connection.autocommit`	string	Enable autocommit for this connection. Use `"true"` or `"false"`.
`adbc.connection.read_only`	string	Open the connection in read-only mode. Use `"true"` or `"false"`.
`adbc.connection.current_catalog`	string	Default catalog for unqualified table names.
`adbc.connection.current_db_schema`	string	Default schema for unqualified table names.

All other keys are driver-specific.

Warning

Security — blast radius of db_kwargs / conn_kwargs

Both db_kwargs and conn_kwargs are passed through to the ADBC driver without filtering. Any user or role that can edit this connection in the Airflow UI or via the Connections API can therefore supply arbitrary driver-level options. Depending on the driver, this may include options that:

load extensions from arbitrary file-system paths (e.g. SQLite’s load_extension or DuckDB’s extension loading);
configure network endpoints or proxy addresses;
change authentication parameters.

Restrict access to the Connections UI and API (Airflow RBAC Connections resource) to trusted operators only. Do not expose connection-editing permissions to untrusted Dag authors.

Similarly, the driver and entrypoint extras load shared libraries or Python symbols by name. Treat them with the same level of trust as executable code.

Examples¶

SQLite in-memory database:

{
  "driver": "adbc_driver_sqlite"
}

PostgreSQL with extra driver options:

{
  "driver": "adbc_driver_postgresql",
  "dialect": "postgresql",
  "db_kwargs": {
    "adbc.postgresql.query.batch_rows": 1000
  },
  "conn_kwargs": {
    "adbc.connection.autocommit": "true"
  }
}

Usage¶

Use AdbcHook in a task to query or load data:

from airflow.providers.apache.arrow.hooks.adbc import AdbcHook


def load_data():
    hook = AdbcHook(adbc_conn_id="my_adbc_conn")

    # Run a query and get results as a list of tuples
    records = hook.get_records("SELECT id, name FROM users WHERE active = 1")

    # Insert rows in chunks of 5 000. By default each chunk is loaded with the
    # driver's bulk ingest (e.g. COPY on PostgreSQL); pass fast_executemany=False
    # to bind the rows to an INSERT statement instead.
    rows = [(1, "Alice"), (2, "Bob")]
    hook.insert_rows(
        table="users",
        rows=rows,
        target_fields=["id", "name"],
        commit_every=5000,
    )