Connect to an Iceberg catalog

Builds a catalog handle. For type = "rest" and type = "glue" this does not contact the server: iceberg-rust opens the connection lazily, on the first operation that needs it. So a mistyped uri, an unreachable host or a missing credential all return a handle here and fail later, at icebergr_list_namespaces() or icebergr_table(), which can look like a fault in those functions rather than in the connection.

Usage

icebergr_catalog(
  type = c("rest", "memory", "glue"),
  uri = NULL,
  warehouse = NULL,
  ...,
  storage = c("auto", "local", "s3"),
  name = "icebergr"
)

Arguments

type

Catalog type. "rest" for an Iceberg REST catalog, "memory" for an in-process catalog over a local warehouse directory, "glue" for AWS Glue.

There is deliberately no "hadoop" option: iceberg-rust does not implement a Hadoop or filesystem catalog. Use type = "memory" with warehouse for a table on local disk.

uri

Catalog URI. Required for type = "rest", ignored otherwise.

warehouse

Warehouse location. A directory for type = "memory"; for REST catalogs, the warehouse name or location the server expects.

...

Further catalog properties, passed through to iceberg-rust as name-value pairs. Use this for non-secret configuration such as "s3.path-style-access" or "prefix". TRUE and FALSE are sent as "true" and "false", and a number is written out in full rather than in scientific notation. A property iceberg-rust does not know is ignored without a word, so check the spelling against its documentation.

storage

Storage backend. "auto" infers it from warehouse, "local" forces the local filesystem, "s3" forces object storage. S3 requires the package to have been compiled with the s3 Cargo feature.

name

A label for the connection, used in error messages.

Value

An icebergr_catalog object.

Details

To find out straight away, ask the catalog something: icebergr_list_namespaces(catalog) is the cheapest round trip.

type = "memory" is the exception, and is checked here: its warehouse has to be an existing directory, because there is no server to ask later.

Credentials

Credentials are read from environment variables, never from arguments:

ICEBERGR_REST_TOKEN

Bearer token for a REST catalog.

ICEBERGR_REST_CREDENTIAL

OAuth2 client credential.

ICEBERGR_REST_OAUTH2_SERVER_URI

OAuth2 token endpoint.

ICEBERGR_REST_SCOPE

OAuth2 scope.

ICEBERGR_S3_ACCESS_KEY_ID, ICEBERGR_S3_SECRET_ACCESS_KEY, ICEBERGR_S3_SESSION_TOKEN

Object storage credentials.

The standard AWS_* variables are a fallback for the ICEBERGR_S3_* ones, but only for a connection that addresses object storage: storage = "s3", type = "glue", or an s3:// warehouse. They are not forwarded to a catalog that has no object storage in sight, because a third-party REST catalog controls each table's location and may answer with its own s3.endpoint, at which point ambient keys would sign requests to a host it chose. Set storage = "s3" if a REST catalog identified by name needs them.

A credential is never sent over an unencrypted connection: an http:// uri or OAuth2 endpoint is an error whenever any credential property is populated. An Authorization, Proxy-Authorization, Cookie or X-Api-Key header passed as a header. property counts as one. A loopback address is exempt, since developing against a local catalog is ordinary, and ICEBERGR_ALLOW_INSECURE_CREDENTIALS=true overrides the check.

Catalog properties are never printed, logged or included in error messages, and user:password@ in a uri is redacted when a catalog is printed. A credential property passed through ... anyway is accepted but warned about, since a script is the one place it should not be.

Examples

# A local warehouse needs no catalog server and no credentials.
warehouse <- tempfile("warehouse")
dir.create(warehouse)
catalog <- icebergr_catalog("memory", warehouse = warehouse)
catalog
#> <icebergr_catalog>
#>   type:      memory
#>   name:      icebergr
#>   warehouse: /tmp/RtmpCcblIw/warehouse28fd4a0f3fdb

if (FALSE) { # \dontrun{
# A REST catalog. The token comes from the environment, not from here.
Sys.setenv(ICEBERGR_REST_TOKEN = "...")
catalog <- icebergr_catalog("rest", uri = "https://catalog.example.com")
} # }