#Dataengineering
Showing 28 of 28 repositories tagged #dataengineering, ranked by stars
Scalable and efficient data transformation framework - backwards compatible with dbt.
Compare tables within or across databases
Scalable master data management, identity resolution, entity resolution, and deduplication using ML
The agent harness for building analytics into your app on top of ClickHouse, Redpanda and other high-performance analytical infrastructure
Roadmap for Data Engineering
Code and data for the Modern Polars book
Все, о чем меня когда-либо спрашивали на собеседованиях, и другие полезные знания в кратком формате
PDFStract - Extract, Chunking and Embedding Layer in Your RAG Pipeline - Available as CLI - WEBUI - API
A Data Platform built for AWS, powered by Kubernetes.
Resources about data science, machine learning, deep learning, data engineering, and SQL.
Duke MIDS: Data Engineering and DataOps Course
Found a data engineering challenge or participated in a selection process ? Share with us!
Build, test, deploy, iterate - Dev and prod tool for data science pipelines
Sample project that use Dagster, dbt, DuckDB and Dash to visualize car and motorcycle Spanish market
Forecasting Solar Power: Analysis of using a LSTM Neural Network
Apply for a job at Olist's Data Team: https://olist.gupy.io/
This project serves as a comprehensive guide to building an end-to-end data engineering pipeline using TCP/IP Socket, Apache Spark, OpenAI LLM, Kafka and Elasticsearch. It covers each stage from data acquisition, processing, sentiment analysis with ChatGPT, production to kafka topic and connection to elasticsearch.
Instant search for and access to many datasets in Pyspark.
Code/Notes for the Data Engineering Zoomcamp by DataTalksClub
An end-to-end data engineering pipeline that fetches data from Wikipedia, cleans and transforms it with Apache Airflow and saves it on Azure Data Lake. Other processing takes place on Azure Data Factory, Azure Synapse and Tableau.
Challenge Data Engineer
Scheduling Big Data Workloads and Data Pipelines in the Cloud with pyDag
Learn the basic commands to use Pandas in Jupyter-Notebook to accomplish the most important Data Enginnering tasks. Read the underlying article on Medium:
Data-Science Python courses from DataCamp
In this project I have built etl pipline which scraps the trending repository based on month,week and day LIVE extract other related information using API and transform it into star schema and load to postgresql then analyzed it over PowerBI . Deployment repo link: