Data Engineering for Humans • Training 03

Article-Training • Data Engineering for Humans

Storage & Databases

Choose the Right Place for Every Kind of Data

Good storage design matches the workload instead of forcing every workload into the same system.

Workload → Structure → Scale → Storage Pattern → Performance & Cost

Sources

→

Ingest

→

Land

Batch

Streaming

CDC

8learning modules

24interactive practices

5rapid review questions

50%certificate threshold

Participant name

Learning target

Choose storage architecture based on workload, data shape and access pattern.

Practice progress0 / 24

Complete 12 of 24 practices (50%) and enter your name to unlock the Certificate of Participation.

MODULE 01

🗄️

Storage Foundations

Storage architecture decides where data should live based on workload, structure, latency, scale, durability, access patterns, governance, and cost.

🗄️

Key idea

Different workloads need different storage patterns • Operational and analytical systems optimize for different behaviors • Durability, availability, latency, scale, and cost must be balanced

Core ideas

Different workloads need different storage patterns
Operational and analytical systems optimize for different behaviors
Durability, availability, latency, scale, and cost must be balanced

Conceptual model

Workload
   ↓
Access Pattern
   ↓
Structure + Scale
   ↓
Storage Choice
   ↓
Performance + Cost

✅

Start with the workload and access pattern—not with a favorite database product.

Practice — 3 cases

Practice 1 / Práctica 1

What is the main goal of a storage architecture?

AMatch storage to workload and access needsBUse one database for every use caseCStore everything only in spreadsheets

Best answer: A. Match storage to workload and access needs

Practice 2 / Práctica 2

Which characteristic most clearly distinguishes storage choices?

AThe color of the applicationBHow the workload reads and writes dataCThe number of monitors

Best answer: B. How the workload reads and writes data

Practice 3 / Práctica 3

Which principle is strongest?

AAlways choose the newest platformBAlways maximize storage sizeCChoose the simplest pattern that reliably meets the workload

Best answer: C. Choose the simplest pattern that reliably meets the workload

MODULE 02

⚖️

OLTP vs OLAP

OLTP systems optimize short, frequent transactions. OLAP systems optimize large analytical scans, aggregations, trends, and historical analysis.

⚖️

Key idea

OLTP favors fast inserts, updates, and point lookups • OLAP favors large reads, joins, aggregations, and history • Separate operational and analytical workloads when their needs conflict

Core ideas

OLTP favors fast inserts, updates, and point lookups
OLAP favors large reads, joins, aggregations, and history
Separate operational and analytical workloads when their needs conflict

Conceptual model

OLTP → Orders / Payments / Updates
OLAP → Trends / Aggregations / History

✅

Do not force heavy analytics onto a transactional database when it threatens operational performance.

Practice — 3 cases

Practice 4 / Práctica 4

Which workload is most associated with OLTP?

AFrequent small inserts and updatesBLong analytical scansCStatic archival files

Best answer: A. Frequent small inserts and updates

Practice 5 / Práctica 5

Which workload is most associated with OLAP?

ASingle-row transaction updatesBComplex analytical queries over large datasetsCKeyboard input validation

Best answer: B. Complex analytical queries over large datasets

Practice 6 / Práctica 6

Why separate OLTP and OLAP workloads?

ABecause SQL cannot do analyticsBBecause warehouses cannot store tablesCBecause transactional and analytical workloads optimize for different behavior

Best answer: C. Because transactional and analytical workloads optimize for different behavior

MODULE 03

🏢

Data Warehouse

A data warehouse stores curated, integrated, historical data optimized for analytics, reporting, KPI calculation, and repeatable business definitions.

🏢

Key idea

Curated and modeled data supports consistent analytics • Historical data enables trends and comparisons • Warehouses prioritize analytical performance over row-by-row transactions

Core ideas

Curated and modeled data supports consistent analytics
Historical data enables trends and comparisons
Warehouses prioritize analytical performance over row-by-row transactions

Conceptual model

Operational Sources
   ↓
Transform / Model
   ↓
Data Warehouse
   ↓
BI / Reporting

✅

Use a warehouse when trusted, repeatable analytical structure matters more than raw-data flexibility.

Practice — 3 cases

Practice 7 / Práctica 7

What is a data warehouse primarily optimized for?

AAnalytics over curated integrated dataBRaw unstructured landing onlyCDevice firmware storage

Best answer: A. Analytics over curated integrated data

Practice 8 / Práctica 8

Why is historical data valuable in a warehouse?

AIt removes all governance needsBIt supports trends, comparisons, and period analysisCIt prevents joins

Best answer: B. It supports trends, comparisons, and period analysis

Practice 9 / Práctica 9

Which consumer commonly uses warehouse data?

AKeyboard driversBBIOS updatesCBI dashboards and analytical reports

Best answer: C. BI dashboards and analytical reports

MODULE 04

🌊

Data Lake

A data lake stores large volumes of raw or lightly processed structured, semi-structured, and unstructured data, usually on scalable object storage.

🌊

Key idea

Lakes preserve raw data for future uses • Schema-on-read allows interpretation at consumption time • Without governance, a lake can become a data swamp

Core ideas

Lakes preserve raw data for future uses
Schema-on-read allows interpretation at consumption time
Without governance, a lake can become a data swamp

Conceptual model

Files + Events + Tables
   ↓
Object Storage / Data Lake
   ↓
SQL / Spark / ML / Exploration

✅

A lake is flexible, but flexibility still requires metadata, lineage, access control, and lifecycle management.

Practice — 3 cases

Practice 10 / Práctica 10

How is a data lake best described?

AStorage for raw structured and unstructured dataBOnly a transactional banking databaseCA dashboard tool

Best answer: A. Storage for raw structured and unstructured data

Practice 11 / Práctica 11

What does schema-on-read mean?

AEvery file must be normalized before landingBStructure is interpreted when data is consumedCOnly SQL Server can read the data

Best answer: B. Structure is interpreted when data is consumed

Practice 12 / Práctica 12

What is a risk of a poorly governed data lake?

AIt automatically becomes a warehouseBIt guarantees perfect qualityCIt can become a data swamp

Best answer: C. It can become a data swamp

MODULE 05

🏠

Lakehouse

A lakehouse combines low-cost flexible lake storage with warehouse-like table management, reliability, governance, and analytical performance.

🏠

Key idea

Open or columnar files can be managed as reliable tables • Transaction layers add consistency and table semantics • One platform can support BI, data science, and ML over shared data

Core ideas

Open or columnar files can be managed as reliable tables
Transaction layers add consistency and table semantics
One platform can support BI, data science, and ML over shared data

Conceptual model

Object Storage
   +
Table / Transaction Layer
   ↓
Lakehouse
   ↓
BI + SQL + ML

✅

A lakehouse is useful when you want lake flexibility without giving up reliable analytical table behavior.

Practice — 3 cases

Practice 13 / Práctica 13

What is the core lakehouse idea?

ACombine lake flexibility with warehouse-style managementBReplace all object storage with spreadsheetsCRemove analytical tables

Best answer: A. Combine lake flexibility with warehouse-style management

Practice 14 / Práctica 14

What capability does a lakehouse add over a raw lake?

ALess metadataBStronger table management and transactional reliabilityCNo need for governance

Best answer: B. Stronger table management and transactional reliability

Practice 15 / Práctica 15

Who can benefit from shared lakehouse data?

AOnly email clientsBOnly device driversCBI, SQL analytics, data science, and ML workloads

Best answer: C. BI, SQL analytics, data science, and ML workloads

MODULE 06

🧩

SQL vs NoSQL

Relational SQL databases excel when relationships, schema, transactions, and consistency matter. NoSQL databases trade some relational structure for flexible models, scale patterns, or specialized access.

🧩

Key idea

SQL is strong for structured relationships and ACID transactions • NoSQL includes document, key-value, wide-column, and graph patterns • Choose based on data model and access pattern, not fashion

Core ideas

SQL is strong for structured relationships and ACID transactions
NoSQL includes document, key-value, wide-column, and graph patterns
Choose based on data model and access pattern, not fashion

Conceptual model

Relational Need → SQL
Flexible / Specialized Access → NoSQL
Hybrid Workload → Polyglot Persistence

✅

SQL and NoSQL are not enemies; mature architectures often use each where it fits best.

Practice — 3 cases

Practice 16 / Práctica 16

Why choose a relational SQL database?

AStrong schema, relationships, and transactional integrityBBecause it cannot use indexesCBecause it only stores images

Best answer: A. Strong schema, relationships, and transactional integrity

Practice 17 / Práctica 17

When can NoSQL be a strong fit?

AWhen every workload needs complex joinsBWhen flexible or specialized high-scale access patterns matterCOnly when there is no network

Best answer: B. When flexible or specialized high-scale access patterns matter

Practice 18 / Práctica 18

What should drive SQL vs NoSQL selection?

ABrand popularity onlyBThe newest marketing termCData model, consistency needs, scale, and access pattern

Best answer: C. Data model, consistency needs, scale, and access pattern

MODULE 07

📦

File & Object Storage

File and object storage are foundational for datasets, backups, logs, media, model artifacts, and lake architectures. Formats and partitioning strongly affect analytical performance.

📦

Key idea

Object storage scales well for large collections of files and blobs • Columnar formats such as Parquet reduce scan and storage cost for analytics • Partitioning helps engines skip irrelevant data

Core ideas

Object storage scales well for large collections of files and blobs
Columnar formats such as Parquet reduce scan and storage cost for analytics
Partitioning helps engines skip irrelevant data

Conceptual model

Object Storage
  /year=2026/month=09/
     part-001.parquet
     part-002.parquet

✅

For analytical files, format and partition design can matter almost as much as where the files are stored.

Practice — 3 cases

Practice 19 / Práctica 19

What is object storage especially good for?

ALarge-scale files, blobs, backups, and data lake objectsBLow-latency row lockingCSpreadsheet formulas

Best answer: A. Large-scale files, blobs, backups, and data lake objects

Practice 20 / Práctica 20

Why can Parquet improve analytics?

AIt forces row-by-row scansBIts columnar format can improve compression and reduce scanned dataCIt removes the need for schemas forever

Best answer: B. Its columnar format can improve compression and reduce scanned data

Practice 21 / Práctica 21

Why does partitioning matter in large datasets?

AIt guarantees zero storage costBIt removes the need for metadataCIt can reduce scanned data and improve manageability

Best answer: C. It can reduce scanned data and improve manageability

MODULE 08

🎯

Choosing the Right Storage

The final storage decision balances performance, latency, durability, availability, retention, governance, cost, interoperability, and operational complexity.

🎯

Key idea

Hot data needs fast access; cold data can use cheaper tiers • Retention follows business, legal, and operational requirements • Hybrid architectures are normal when workloads differ

Core ideas

Hot data needs fast access; cold data can use cheaper tiers
Retention follows business, legal, and operational requirements
Hybrid architectures are normal when workloads differ

Conceptual model

Transactions → Operational DB
Analytics → Warehouse / Lakehouse
Raw / Archive → Object Storage
Specialized Access → NoSQL

✅

The right architecture is the least-complex combination that meets reliability, performance, governance, and cost requirements.

Practice — 3 cases

Practice 22 / Práctica 22

What should guide storage selection first?

AWorkload and access patternBLaptop screen sizeCLogo color

Best answer: A. Workload and access pattern

Practice 23 / Práctica 23

What is the purpose of storage tiers?

AMake all data equally expensiveBBalance performance, retention, and costCPrevent backups

Best answer: B. Balance performance, retention, and cost

Practice 24 / Práctica 24

What does durability describe?

AHow quickly a dashboard loadsBHow many users have accountsCHow reliably data remains intact over time

Best answer: C. How reliably data remains intact over time

5-Question Knowledge Check

Can you explain why storage design changes with workload?

OLTP and OLAP optimize different workloads.

OLTP favors short transactions and point operations; OLAP favors scans, joins, aggregations, and historical analysis.

A warehouse is curated for analytics.

Warehouses integrate and model trusted historical data for reporting, KPIs, and analytical queries.

A lake preserves flexible raw data.

Lakes support structured, semi-structured, and unstructured data.

A lakehouse adds reliable table behavior to lake storage.

Lakehouse designs combine flexibility with table reliability.

Storage selection starts with workload and access pattern.

Latency, consistency, scale, durability, retention, governance, cost, and operational complexity refine the final choice.

Storage Decision Map

Choose storage by workload, access pattern, structure, latency, and governance.

Need	Strong candidate
High-volume operational transactions	Relational OLTP database
Curated historical analytics	Data Warehouse
Raw / semi-structured / unstructured data at scale	Data Lake / Object Storage
Lake flexibility + managed analytical tables	Lakehouse
Flexible or specialized non-relational access	NoSQL
Low-cost archive / large files / model artifacts	Object Storage

Certificate of Participation

Complete at least 12 of the 24 practice cases (50%) and enter your name.

0 / 24 • 0%

Production note

Good storage architecture matches the workload, data shape, scale, latency, durability, governance, and access pattern instead of forcing every use case into one system.

Storage & Databases

Storage Foundations

Core ideas

Practice — 3 cases

OLTP vs OLAP

Core ideas

Practice — 3 cases

Data Warehouse

Core ideas

Practice — 3 cases

Data Lake

Core ideas

Practice — 3 cases

Lakehouse

Core ideas

Practice — 3 cases

SQL vs NoSQL

Core ideas

Practice — 3 cases

File & Object Storage

Core ideas

Practice — 3 cases

Choosing the Right Storage

Core ideas

Practice — 3 cases

5-Question Knowledge Check

Storage Decision Map

Certificate of Participation

Storage is a design decision, not just a destination

Production note