Data Engineering

Storage

Database Fundamentals

• SQL
• Normalization
• Concepts: relational, noSQL, data warehouse, data lake, data lakehouse, database Dump
• ACID transactions
• CAP theorem
• Dead locks
• Database design

- Modeling - How choose database?

Relational Database

Python examples connections, disconnection, execution query:

• MySQL
• Postgres
• SQLServer
• Oracle
• DB2

Non-relational Database

Python examples connections, disconnection, execution query:

• DynamoDB
• ElasticSearch
• Apache Hive
• Redis

Data Projects

Small Projects

• Blocked sort based index

[//]: # (- Apache Airflow: generating multiples DAG) [//]: # (- Apache Airflow: generating multiples tasks in DAG)

Exploratory Data Analysis (EDA)

• The Spending of Brazilians Senators
• Finding Interesting Business Metrics

Data Lakes

• Apache Hadoop (YARN, Hive, HDFS), Tez, Spark, Sqoop and Azkaban
• Apache Hadoop (YARN, Hive, HDFS), Impala, Spark, Oracle database, Redis, and Airflow

Data Warehouses

• Data Catalog in Databricks (Unity Catalog)

Data QA

• Data Quality Assurance in Medallion Architecture

---

References

• 1: Coding Roadmap

---