What is Databricks?

Databricks is a data analytics platform available for use at the USDOT. It combines data engineering, data science, and business intelligence in a single platform. Teams can work together in a workspace on data processing, analytics, and machine learning projects. Since Databricks is built on Apache Spark, it can process large volumes of data very efficiently.

How Can My Team Use Databricks?

Each team member is provided a workspace where they can create notebooks that run Python, SQL, R, or Scala to process data and create visualizations. Users have access to powerful compute resources which are provisioned securely in our AWS environment. Users can also create and manage tables just like in a data warehouse but with the flexibility of a data lake. Lastly, users have access to machine learning tools to aid in experimentation and building custom ai/ml applications.

How Much Does It Cost?

There are no license costs for having a Databricks account. Instead, users are charged based on what they consume. Our team can work with you to establish guard rails on your usage to help manage costs.

How Do I Get Started?

Fill out the Databricks request form to help our team provision you for success. The following provides some guidance for completing the form.

Environments

Our Databricks platform consists of two distinct environments (dev and prod) each of which corresponds to a separate AWS account. Each team will be initially provisioned in this dev environment. Data engineers and data scientists who are using notebooks and workspaces will only ever be provisioned in dev. It’s not uncommon that a project will only ever need to work in the dev workspace. However, applications or data lakes that will be used by a group of end users are promoted from dev → prod. Resources in production are provisioned using service principals and are only accessed by users or developers for testing or troubleshooting.

User Groups and Roles

Users in Databricks are sorted into groups to provide permissions to various resources across the platform. Groups are named according to the following convention: <mode>_<project>_<role>_<env>. Organization/Mode and Environment are critical components of the group name, while domain, and to a lesser extent role, can be flexed to suit your team’s need. Our users are first grouped by mode such as ost_sandbox_user_dev (note this doesn’t include a role since it’s a parent group for all roles). Additional groups are provisioned based on the required roles and environments. The following roles are provisioned for your team:

• admin: Manages the organization members and permissions. Can create, read, update, and delete on all resources.

• architect: Manages the Unity Catalog objects for their organization. Can create, read, update, and delete on unity catalog objects (schema, table, volume, model).

• engineer: Can create in objects such as tables or views (limited to subject area in unity catalog e.g. ost_aiml_dev) and read/write to them.

• contributor: Writes data but does not create tables/schemas.

• tester: Reads data in test environment (should only be assigned resources required for testing).

• analyst: Reads data to produce visualizations/reports.

Occasionally teams will need access to a more specific set of resources and be excluded from others - such as a team dedicated to a particular project or use case. In this case groups can be created to match that domain e.g., ost_useCase01_data_engineer_sandbox is a group for data engineers working on a particular use case in the sandbox environment.

Our support team can work with you to build out groups as needed by your project.

Catalogs

Unity Catalog is the central data governance feature in Databricks. It is a collection of catalogs which contain schemas, tables, models, and volumes. Each catalog is mapped to an individual s3 bucket in AWS which allows the storage of a variety of object types. Schemas hold tables, models, and volumes. Volumes allow for storing unstructured data such as images, video, pdfs, and many other file types. Your team will be provisioned with catalogs which you own and manage. Each team is initially provided with a sandbox catalog such as, ost_sandbox_dev. The naming convention is similar to user groups: <mode>_<project>_<subproject>_<env>. For example, you may want a separate catalog and bucket for the data science work your team is doing, so we would create a catalog such as ost_data_science_dev. Our team can help advise on how to name and organize your data in the Databricks Unity Catalog.

Tagging Policies

Tagging is an important aspect our shared service offering of Databricks. Each mode is required to tag all resources with that mode so that charges can be charged back to the appropriate organization. Since it is essential we maintain proper tagging, we have policies in place to enforce tagging wherever applicable. This relieves some of the burden on the users and helps us maintain consistent tagging. We will always include the following tags and need your help in setting the values:

• Mode

• FISMA ID - this defaults to “N/A” if not provided

• Project - could be the same as the team

• Fed Owner

• Program Office

Our team can work with you to determine additional tagging requirements specific to your mode/project. 

Advanced Compute

While all compute options are available, we implement guardrails to manage costs. These limits can be increased to support big data workloads or advanced data science/machine learning projects. Additionally, we can provide guidance on using multi-node compute clusters to improve efficiency.

Data Engineering and Pipelines

Databricks provides robust support for building data pipelines with advanced features such as change data capture, Delta Lake, conditional workflows, notifications, and more. We recommend using a medallion architecture in building your data pipelines and can assist in some of the data architecture decisions. However, you have a lot of flexibility with your Databricks space and can set up your catalogs, schemas, and tables to best leverage your data. Our platform has access to CI/CD through DOT’s enterprise GitHub and Jenkins offerings and our team can help direct you to those resources, as needed.