100 Essential Questions

Deep Learning is characterized by hierarchical compositionality (layers learning increasingly abstract features), end-to-end learning (raw data to output without manual feature engineering), and distributed representations (concepts coded by a distributed pattern rather than a single neuron).

In classical ML, humans manually engineer features (like HOG or MFCC), whereas Deep Learning automatically learns hierarchical representations directly from raw data.

In feed-forward networks (like CNNs), information flows strictly from input to output, while feed-back architectures (like RNNs) use feedback loops to re-inject outputs as inputs.

Without a non-linear activation function, stacking multiple layers would be mathematically equivalent to a single linear transformation, preventing the model from learning complex functions.

The four steps are the forward pass (computing predictions), loss calculation (measuring error), backpropagation (calculating gradients via the chain rule), and the weight update using gradient descent.

Mini-batch SGD provides an ideal compromise by being faster and parallelizable, whereas Batch GD computes the gradient over the entire dataset, which is stable but impractically slow.

ReLU assigns a gradient of zero to negative inputs, causing neurons to "die" and stop learning; Leaky ReLU solves this by assigning a very small, non-zero gradient ($\alpha x$) for negative values.

Softmax transforms a vector of raw scores into a valid probability distribution, where all values sum to 1, making it ideal for predicting a single class out of many.

The training set adjusts weights, the validation set tunes hyperparameters and detects overfitting during training, and the test set provides an unbiased final evaluation of the model.

L1 regularization promotes sparsity by driving some weights exactly to zero, while L2 regularization limits excessively large weights without strictly enforcing sparsity.

Dropout randomly deactivates a fraction of neurons during training, forcing the network to learn redundant and robust representations rather than relying on a few specific neurons.

On a dataset heavily skewed toward one class (e.g., 95% to 5%), a model can achieve 95% accuracy just by predicting the majority class, making metrics like Precision, Recall, and F1-score necessary.

AUC-ROC measures the model's overall capacity to correctly separate and distinguish between classes across all possible decision thresholds.

Early stopping automatically halts training when the model's performance on the validation loss stops improving, ensuring the model doesn't start memorizing training noise.

Hidden layer biases should be initialized to zero, while the output layer bias is optimally initialized to the average of the target values.

MLPs suffer from the "curse of dimensionality" requiring millions of parameters, lack spatial invariance, and destroy local pixel relationships by flattening the image.

CNNs utilize local receptive fields (neurons connect only to small local regions) and weight sharing (the same filter slides across the whole image).

"Same" padding adds zeros around the image boundaries to maintain the original spatial dimensions, whereas "valid" padding applies no padding, resulting in reduced output dimensions.

where $I$ is input size, $K$ is kernel size, $P$ is padding, and $S$ is stride.

Max Pooling reduces spatial dimensions, focuses on the most salient features (like edges or corners), and provides invariance to small spatial translations without adding learnable parameters.

Dense layers can account for over 85% of a network's parameters, so Global Average Pooling is used to drastically reduce parameter count and mitigate overfitting.

The number of filters should be doubled (e.g., from 64 to 128) each time spatial resolution is halved, maintaining a constant computational cost.

Two stacked 3×3 filters provide the same 5×5 receptive field but require 28% fewer parameters, making the network deeper and more efficient.

ResNet introduced residual connections (skip connections), which help bypass layers and allow gradients to flow freely through extremely deep networks.

Feature Extraction involves freezing the entire pre-trained convolutional base and only training a newly added classification head on the specific dataset.

Fine-tuning (unfreezing the top layers and training with a very low learning rate) should be used when the dataset is medium-sized and the precision from pure feature extraction is insufficient.

Data augmentation creates artificial variations to prevent overfitting; applying it to the validation or test sets would distort the unbiased evaluation of the model's true performance.

Recall (Sensitivity) is critical because minimizing false negatives (missing an actual disease) is far more important than avoiding false alarms.

Inputs must be normalized (to $[0,1]$ or standardized); feeding large, unscaled values creates instability and slows down gradient descent convergence.

They lack temporal memory, require inputs of fixed length, and cannot share learned temporal patterns across different positions in a sequence.

An RNN maintains a hidden state ($h_t$) through a feedback loop, which is updated at each time step and acts as a memory of previously seen inputs.

BPTT is the algorithm used to train RNNs, where the network is unrolled across the entire sequence length, and gradients are propagated backward through the shared weights over time.

During BPTT, gradients are repeatedly multiplied by the same weight matrices; over long sequences (>10–15 steps), these gradients exponentially shrink toward zero, causing the network to forget long-term dependencies.

Gradient clipping is heavily utilized, which caps the maximum value of gradients to ensure stable training without numerical overflow.

The LSTM maintains a visible working memory called the hidden state ($h_t$) and a protected, long-term memory called the cell state ($C_t$).

The Forget gate decides what to erase from the cell state, the Input gate decides what new information to add, and the Output gate controls what is emitted as the new hidden state.

The cell state updates via element-wise addition and multiplication, acting as a "conveyor belt" that allows gradients to flow backward almost unchanged without suffering repeated matrix multiplications.

The GRU merges the cell state and hidden state into a single vector and reduces the architecture to just two gates (Update and Reset), requiring about 25% fewer parameters than an LSTM.

A GRU is preferred when seeking an optimal compromise between speed and performance, particularly for edge devices, mobile computing, or real-time systems.

A BiLSTM consists of two independent RNNs processing the sequence in both chronological and reverse order, allowing it to leverage both past and future context simultaneously.

The encoder is forced to compress the entire input sequence into a single, fixed-size context vector, which becomes a severe information bottleneck for long sequences.

RNNs suffer from a sequential bottleneck because computing the state at time $t$ strictly depends on the computed state from time $t-1$, making parallelization impossible during training.

RNNs are still ideal for real-time streaming inference (as they process elements one by one with $O(1)$ memory), microcontrollers, and irregularly spaced time series.

A low temperature ($T \sim 0.3$) creates highly coherent but repetitive text, whereas a high temperature ($T \sim 1.3$) increases lexical diversity at the risk of generating incoherent text.

NLU takes raw text and extracts structured representations (like entities or intent), whereas NLG takes structured data or prompts and produces fluent, human-readable text.

It illustrates syntactic ambiguity; the sentence produces two valid syntax trees depending on whether the observer or the man holds the telescope.

The first steps generally include converting to lowercase, removing HTML tags via regex, and stripping URLs.

They decompose unknown words into smaller, known sub-word components (e.g., "unhappiness" into "un", "##happi", "##ness"), thereby mitigating the Out-Of-Vocabulary problem.

Words like "not", "no", and "never" are often flagged as stop words; removing them destroys negation and entirely flips the sentiment of phrases like "not good".

Stemming uses rigid heuristic rules to chop off suffixes (often leaving non-words like "studi"), while Lemmatization performs morphological analysis to return a valid dictionary root (like "study").

A word's lemma depends on its grammar; without POS tagging, the lemmatizer wouldn't know if "better" should be reduced to "good" (adjective) or left as "better" (verb).

It assigns a grammatical category (e.g., noun, verb, adjective, preposition) to each token based on its precise context in the sentence.

Named Entity Recognition (NER).

Dependency parsing creates a directed tree that extracts grammatical relationships, identifying the head of a word to answer "who does what to whom" (e.g., nominal subjects and direct objects).

A CFG is a formal system using non-terminal rewriting rules (e.g., $S \rightarrow NP\ VP$) to describe the structural syntax of a language and generate derivation trees.

CFGs generate multiple valid syntax trees for ambiguous sentences but completely lack the contextual or probabilistic awareness to select the correct meaning.

Out-Of-Vocabulary, which refers to words that the model never encountered during its training phase.

BERT inherently learns the nuances between inflected forms through their surrounding context; stripping words to radicals destroys these valuable contextual markers.

While it removes noise, it also aggressively strips all punctuation, numbers, and symbols, which could contain critical data like dates, monetary values, or structural sentence markers.

It creates a highly sparse matrix (often >99% zeros) based purely on word counts, completely losing the order of words and semantic nuances.

By capturing sequences of $N$ consecutive words (like bigrams), they can process "not good" as a single distinct feature rather than two disconnected, conflicting words.

The Inverse Document Frequency (IDF) component heavily penalizes words that appear universally across the corpus, reducing their final weight to near zero.

TF-IDF explicitly weights words by their rarity and discriminative power, highlighting strong keywords while attenuating background noise.

It is the linguistic theory that the semantic meaning of a word can be mathematically inferred from the company it keeps (its surrounding context words).

CBOW (Continuous Bag of Words, predicting a target word from its context) and Skip-gram (predicting the surrounding context from a target word).

While Word2Vec uses a local predictive neural network window, GloVe constructs a massive global co-occurrence matrix for the entire corpus and applies SVD factorization.

FastText represents a word as the sum of its character n-grams (e.g., <te, ter, err> for "terrible"), allowing it to assemble plausible vectors for misspelled or entirely new words.

They assign exactly one fixed vector per word type, meaning polysemic words (like "bank" as a river or "bank" as an institution) receive the exact same representation regardless of context.

Mean pooling gives equal mathematical weight to all words, diluting strong keywords in semantic noise, whereas TF-IDF heavily favors the discriminative terms crucial for classification.

BERT generates contextual embeddings, meaning the exact same word generates completely different vectors depending on the surrounding sentence structure.

Unlike causal models (like GPT) that read strictly left-to-right, BERT simultaneously accesses both the left and right context of a token to compute its representation.

Masked Language Modeling (MLM, predicting 15% of tokens that have been hidden) and Next Sentence Prediction (NSP).

Feature Extraction freezes BERT's internal weights and uses its outputs in a basic classifier, while Fine-Tuning updates all of BERT's parameters specifically for the downstream task.

It is computationally massive, demanding a GPU for real-time performance and exhibiting inference latencies up to 100–1000 times slower than classical methods on a CPU.

It eliminated sequential processing (allowing massive parallelization), solved the vanishing gradient for long-range dependencies, and removed the fixed context vector bottleneck.

Self-attention allows each token to dynamically compute a relevance score with every other token in the sequence, weighting their importance to build an enriched representation.

They represent the Query (what the token seeks), the Key (what information the token offers), and the Value (the actual content transmitted).

Multiple heads allow the model to specialize simultaneously in different linguistic dimensions, such as subject-verb relations, coreferences, and semantic similarity.

The pure attention mechanism is permutation invariant and has no innate concept of word order; positional embeddings inject necessary sequence data.

It masks future positions (setting them to $-\infty$) during auto-regressive generation to prevent the model from "cheating" by looking ahead at words it hasn't generated yet.

Multi-head attention, a feed-forward network, and residual connections encompassing Layer Normalization (Add & Norm).

BERT utilizes an Encoder-only structure, GPT utilizes a Decoder-only structure, and T5 utilizes the full Encoder-Decoder architecture.

Scaling refers to the massive increase in both parameter count (up to hundreds of billions) and training corpus size, which leads to emergent, qualitatively new capabilities.

They are complex abilities the model was never explicitly programmed for that appear at scale, such as In-Context Learning (few-shot) and Chain-of-Thought reasoning.

To perform auto-regressive language modeling (predicting the next token) on massive unannotated corpora, allowing the model to acquire broad encyclopedic knowledge and syntax.

The model learns to follow specific instructions and adopt a conversational format using human-validated instruction-response pairs.

Reinforcement Learning from Human Feedback aligns the model with human values (helpfulness, harmlessness, honesty) to minimize toxicity and increase safety.

RAG queries an external vector database for precise, up-to-date documents and injects these passages directly into the prompt to ground the LLM's response.

LoRA freezes all original model weights and injects small, trainable low-rank matrices into the attention layers, updating only a fraction of a percent of parameters and saving massive GPU memory.

It compares the analytically computed backpropagation gradients against a numerical approximation to verify there are no implementation bugs in the code.

You intentionally try to overfit the model on a tiny subset of data; if the loss does not reach near zero, the architecture lacks expressivity and must be expanded.

ReduceLROnPlateau automatically reduces the learning rate when the validation loss stops decreasing.

Because "film" is a generic, domain-neutral word that appears in almost every movie review, providing no discriminative signal between positive and negative sentiments.

It indicates the model is failing to maintain context and is stuck in a loop, generating the same sequence repeatedly, which is a common flaw in vanilla RNNs.

It provides a robust, reliable estimate of the model's performance without permanently sacrificing a fixed portion of the already limited data for a single validation set.

A generator applies random transformations (like rotations and flips) directly in memory for each batch, enriching the dataset infinitely without consuming physical disk storage.

It measures the lexical diversity of the text by comparing the number of unique words (types) against the total number of generated words (tokens).

Classical methods like TF-IDF or BoW, which execute in ~1–3 ms per document entirely on CPU infrastructure.

Always establish a simple, interpretable baseline (like TF-IDF) first; only introduce complex architectures (like BERT or LLMs) if they demonstrate a clearly measurable gain on the target metrics.