Course Outline
Databricks Platform and Lakehouse Fundamentals
- Databricks Lakehouse architecture and components.
- Organizing workspaces and catalogs.
Databricks Workspace and Notebooks
- Workspace navigation and notebook-based development.
- Structuring code into reusable notebooks.
Apache Spark Architecture and Execution
- Spark runtime architecture and execution model.
- Lazy evaluation and the job DAG.
PySpark DataFrames and the DataFrame API
- DataFrame abstractions and schemas.
- Core DataFrame operations and column expressions.
Translating SQL to PySpark DataFrames
- Translating core SQL clauses to DataFrame operations.
- Window functions and aggregations in PySpark.
Reading and Writing Data in Databricks
- Reading from common file and database sources.
- Writing and partitioning data within the Lakehouse.
Delta Lake and Table Management
- Delta tables and ACID transactions.
- Time travel and schema evolution.
Data Cleaning and Transformation Patterns
- Data cleaning and type conversion.
- Building reusable transformation logic.
User-Defined Functions and Modular Code
- Python UDFs and pandas UDFs.
- Modularizing procedural logic into functions.
Performance Tuning and Optimization
- Partitioning and caching strategies.
- Diagnosing bottlenecks with the Spark UI.
Structured Streaming Fundamentals
- Batch versus streaming processing models.
- Streaming DataFrames and basic aggregations.
Databricks Jobs and Workflow Orchestration
- Scheduling notebooks as jobs and tasks.
- Building multi-step workflows with dependencies.
Unity Catalog and Data Governance
- Unity Catalog architecture and namespaces.
- Access control and data lineage.
Testing, Debugging, and Production Practices
- Unit testing PySpark logic.
- Debugging and code quality standards.
End-to-End Financial Services Use Cases
- Building an end-to-end banking ETL pipeline.
- Translating legacy SQL processes to PySpark.
Migrating SQL Workloads to PySpark
- Migration strategy and planning patterns.
- Incremental conversion of SQL workflows to PySpark.
Requirements
- Experience with Python programming, including functions and data types.
- An understanding of SQL, covering joins, aggregations, and subqueries.
- No prior experience with Databricks or PySpark is required.
Audience
- Data engineers, data analysts, and other data professionals.
- Teams looking to migrate existing SQL-based workflows to Databricks and PySpark.
Testimonials (2)
All the topics covered, although many were very quick, give us an idea of what we will need to delve into further. Additionally, I liked that we got to do some hands-on practice, although I still believe the course deserves more.
Sandra Mariela Lopez Bernal - Kueski
Course - Databricks
Machine Translated
I liked that it was practical. Loved to apply the theoretical knowledge with practical examples.
