Data Engineer Course Projects

| Project | Stack, tools, libraries | | :-------------------- | :--------------------- | | Data quality cheks. RFM datamart. | SQL, Common Table Expression, Window Functions, PostgreSQL, cloudbeaver | Modifying DWH. Migration to the new model. | SQL, Window Functions, PostgreSQL, cloudbeaver | Modifying ETL and datamarts. Implementing idempotency. | AirFlow, SQL, PostgreSQL, cloudbeaver, bash, pandas, SQLAlchemy, PostgresOperator, BashOperator | Data quality checks in ETL | AirFlow, SQL, PostgreSQL | Datamart in DWH based on multiple sources | Airflow, PostgreSQL, MongoDB Compass, pendulum, Jupyter Notebook, bash, SQLAlchemy, PostgresHook | | Datamart based on Analytical Database Vertica | AirFlow, Yandex S3 Storage, Common Table Expression, SQL, Vertica, cloudbeaver, pandas | Working with PySpark in Hadoop. Working with HDFS. | Hadoop, Spark, PySpark, YARN, MapReduce, Window Functions, HDFS, Airflow, SparkSubmitOperator, Parquet | Processing stream data with Spark | Kafka, PySpark, AirFlow, kcat, Jupyter Notebook, SQL, PostgreSQL, Spark Streaming | | Cloud services | Yandex Cloud Services, Datalense, Kubernetes, kubectl, Kafka, kcat, confluentkafka, flask, Docker Compose, Helm, Redis | Combining data streams. Analytics datamart. | Yandex S3, DWH, Vertica, boto3, Airflow, TriggerDagRunOperator, Metabase ---