ee046211-deep-learning

Technion ECE 046211 - Deep Learning

Tal Daniel • Daniel Soudry

Jupyter Notebook tutorials for the Technion's ECE 046211 course "Deep Learning"

Student Projects Website • Video Tutorials (Winter 2024)

• ee046211-deep-learning

Agenda

|File | Topics Covered | Video | |----------------|---------|-------| |Setting Up The Working Environment.pdf| Guide for installing Anaconda locally with Python 3 and PyTorch, integration with PyCharm and using GPU on Google Colab |-| |ee046211tutorial01
machinelearning_recap.ipynb/pdf| Supervised and Unsupervised Learning, Model Evaluation, Bias-Variance Tradeoff, Feature Scaling, Linear Regression, Gradient Descent, Regularization (Ridge, LASSO)|Video Link| |ee046211tutorial02
singleneuronrecap.ipynb/pdf| Discriminative models, Perceptron, Logistic Regression (also in PyTorch), Softmax Regression, Activation functions|Video Link| |ee046211tutorial03
optimizationgradient_descent.ipynb/pdf|Unimodal functions, Convexity, Hessain, Gradient Descent, SGD, Learning Rate, LR Scheculing / Annealing, Momentum, Nesterov Momentum, Adaptive Learning Rate Methods, Adagrad, RMSprop, Adam, AdaBelief, MADGRAD, Adan, Schedule-free Optimization (SGD, Adam)|Video Link - Part 1

Video Link - Part 2| |ee046211tutorial04
differentiationautograd.ipynb/pdf|Lagrange Multipliers, Automatic Differentiation (AutoDiff) Forward Mode and Reverese Mode, PyTorch Autograd|Video Link| |ee046211tutorial05
multilayernn.ipynb/pdf|Multi-Layer Perceptron (MLP), Backpropagation, Neural Netwroks in PyTorch, Weights Initialization - Xavier (Glorot), Kaiming (He), Deep Double Descent|Video Link| |ee046211tutorial06
convnetsvisualtasks.ipynb/pdf|2D Convolution (Cross-correlation), Convolution-based Classification, Convolutional Neural Networks (CNNs), Regularization and Overfitting, Dropout, Data Augmentation, CIFAR-10 dataset, Visualizing Filters, Applications of CNNs, The problems with CNNs (adversarial attacks, poor generalization, fairness-undesirable biases)|Video Link - Part 1

Video Link - Part 2| |ee046211tutorial07
sequentialtasksrnn.ipynb/pdf|Sequential Tasks, Natural Language Processing (NLP), Language Model, Perplexity, BLEU, Recurrent Neural Network (RNN), Backpropagation Through Time (BPTT), Long Term Short Memory (LSTM), Gated Recurrent Unit (GRU), RWKV, xLSTM, Multi-head Self-Attention, Transformer, BERT and GPT, Teacher Forcing, torchtext, Sentiment Analysis, Transformers Warmup, Intialization, GLU variants, Pre-norm and Post-norm, RMSNorm, SandwichNorm, ReZero, Rectified Adam (RAdam), Relative Positional Encoding/Embedding|Video Link - Part 1

Video Link - Part 2

Video Link - Part 3| |ee046211tutorial08
trainingmethods.ipynb/pdf|Feature Scaling, Normalization, Standardization, Batch Normalization, Layer Normalization, Instance Normalization, Group Normalization, Vanishing Gradients, Exploding Gradients, Skip-Connection, Residual Nlock, ResNet, DenseNet, U-Net, Hyper-parameter Tuning: Grid Search, Random Search, Bayesian Tuning, Optuna with PyTorch|Video Link

Video Link - Optuna Tutorial| |ee046211tutorial09
selfsupervisedrepresentationlearning.ipynb/pdf|Transfer Learning, Domain Adaptation, Pre-trained Networks, Sim2Real, BERT, Low-rank Adaptation - LoRA, DoRA, Representation Learning, Self-Supervised Learning, Autoencoders, Contrastive Learning, Contrastive Predictive Coding (CPC), Simple Framework for Contrastive Learning of Visual Representations (SimCLR), Momentum Contrast (MoCo), Bootstrap Your Own Latent (BYOL), DINO, CLIP| Video Link - Part 1 - Transfer Learning

Video Link - Part 2 - Self-supervised Learning| |ee046211tutorial10
compressionpruningamp.ipynb/pdf|Resource Efficiency in DL, Automatic Mixed Precision (AMP), Quantization (Dynamic, Static), Quantization Aware Training (QAT), LLM Quantization, Pruning, The Lottery Ticket Hypothesis|Video Link| |pytorchmaximizecpugpuutilization.ipynb/pdf|Tips and Tricks for efficient coding in PyTorch, Maximizing the CPU and GPU utilization, nvidia-smi, PyTorch Profiler, AMP, Multi-GPU training, HF Accelerate, RL libraries|Video Link|