/
DB Guide: Cheat Sheet

DB Guide: Cheat Sheet

A quick reference for how DOT manages Databricks and how projects are structured.

How DOT Uses Databricks

DOT manages Databricks centrally through a dedicated support team. Rather than each team running their own instance, all projects share a managed set of environments with the support team handling infrastructure and access.

Environments

DOT maintains three Databricks workspaces available to project teams:

  • dev — Where all active project development happens. Most projects will live here permanently.

  • stage — A pre-production environment for projects with formal production workflows. Not all projects will use this.

  • prod — Live production environment. Only projects with validated, stable workflows are promoted here.

Projects and Catalogs

In DOT's Databricks setup, every project maps to exactly one catalog — this is how we keep projects isolated and organized. When your project is onboarded, a catalog is created for you.

Databricks Concepts

  • Catalog — The top-level container for your project. Think of it as your project's namespace within Databricks. Each project gets one.

  • Schema — A logical grouping within a catalog, similar to a folder or database schema. Use schemas to organize data by domain, pipeline stage, or team convention.

  • Table — Structured data stored within a schema. Tables can be either managed (data stored and maintained within Databricks) or external (data physically resides in S3 or another source and is referenced by Databricks).

  • Volume — A storage location for unstructured or semi-structured files (CSVs, JSONs, PDFs, etc.) within a catalog. Useful for landing raw data before it's processed into tables.

External Connections

Projects can connect to data sources outside of Databricks as needed:

  • External S3 Buckets — Connect to S3 buckets in DOT AWS accounts or partner accounts (e.g., state DOTs, universities) to read or write data directly.

  • Foreign Database Connections — Connect to external databases (RDS, PostgreSQL, etc.) for querying or ingestion workflows.

These are set up by the DOT Databricks Support Team on request. Submit a ticket to get started.