Time: Tuesday and Thursday, 11:30AM - 12:50PM
Location: MC 2038
Instructor: Jimmy Lin
TAs: Arash Khalaji (mkhalaji), Kacie Phillippo (k2philli), and Felix Wang (t577wang)
Piazza: Link

Overview

The data flywheel characterizes the virtuous cycle by which an organization builds a useful product, analyzes user behavior from it to derive insights, and transforms those insights into actions to improve the product. Google, Meta, Amazon, and nearly all consumer technology companies today can be characterized in this manner.

Today, machine learning techniques are ubiquitous in "closing the loop", in deriving insights from massive amounts of data and putting them into action. This course provides an overview of the infrastructure that enables this, what can generically be termed a data platform. In industry this is often called a lakehouse (a portmanteau of a data lake and data warehouse). In modern parlance, this course focuses on data engineering.

Data engineering is a practical endeavor for grappling with the volume, velocity, variety, and veracity of data. While there are best practices, there are no solutions, only tradeoffs (e.g., between time and space, between latency and throughput, etc.) that need to be managed based on different usage scenarios. Outside of brand-new startups, data platforms likely already exist within an organization, and typically represent complex, messy, and evolving entities comprising a myriad of technologies, processes, and constraints (both technical and non-technical). Thus, a key competency in data engineering is the ability to understand and evaluate existing designs and the tradeoffs they represent; only then can data platforms be improved to accelerate the data flywheel.

Having described what this course is, it might be similarly helpful to describe what this course is not:

• This course is not about machine learning, but touches on machine learning to the extent necessary to understand how data drives model training, inference, etc.
• This course is not about data science, but about providing the data infrastructure to support such activities.
• This course is not about algorithms and methods for distributed processing, but touches on concepts relevant to data engineering such as data partitioning, replication, and consensus.
• This course is not about building database applications, but touches on relevant aspects such as ELT pipelines from operational databases.

Objectives

By the end of this course, students will be able to:

• Understand the complexities and appreciate the challenges associated with the volume, velocity, variety, and veracity of data.
• Understand how core computer science concepts such as partitioning and replication contribute to the design of data platforms.
• Identify and explain the role of different components in a modern data lake, including differences in how operational and analytical infrastructure are organized.
• Articulate and evaluate the tradeoffs between different designs and approaches to data engineering with respect to best practices and desired outcomes.
• Rapidly map the contents of this course to software, platforms, and technologies used in industry.
• Use coding agents effectively and responsibly in pursuit of the above objectives.

Fork me on GitHub!