DeepLearning

• all data as input once
• loss graph is quite smooth
• need great calculation power

• input each time only one data sample
• cost will fluctuate over the training
• fast update the weights

• only a subset of all data points in each

Universal Approximation Theorem Perceptrons solves linearly separable problem One hidden layer is enough to approximate any continuons funtion, to an arbitrary degree of accuracy

Perceptron Learning Algorithm(PLA): \(\omega \gets 0\) Iterate over training examples until convergence \(\hat{y}^{i} \gets \omega^{T} x^i\) \(e \gets y^{i}-\hat{y}^{i}\) \(\omega \gets \omega + e \cdot x^i\)

Gradient-based optimization does neither always find the absolute minimum, nor does it find the optimal direction

Sigmoids vanish for large postive and negative inputs Rectified Linear Unit(ReLu) LeakyReLU Exponential linear unit Absolute value activation

valid: \(M=\lfloor \frac{N - k}{s} \rfloor +1\) padding: \(M=\lfloor \frac{N - k +2p}{s} \rfloor+1\) N: input size M: output size p: padding k: kernel size s: stride size

\[RF = 1 + \sum^{L}_{l=1}(k_{l}-1)*s \] \[ RF_{i} = (RF_{i+1} -1)*s + k \]

From where I want to calcalete to the input layer. and set the current calcalete layer's RF as 1

The layer can extract image features, and finally determine the convolution kernel parameters through backpropagation to obtain the final features

For Vanishing/exploding gradients: each example in layer all data are normalized \[ \mu = \frac{1}{N} \sum_{i=1} ^{N} x_{i, j}\]

\[ \sigma_{j}^{2} = \frac{1}{N} \sum_{i=1}^{n} (x_{i, j} - \mu_{j})^{2}\]

\[\hat{x}_{i,j} = \frac{x_{i, j} - \mu_{j}}{\sqrt{\sigma_{j}^{2} + \epsilon}} \]

\[ y_{i, j} = \gamma_{j}\hat{x}_{i, j} + \beta_{j}\]

• Batch Normalization norm each channel
• Layer Normalization norm each sample
• Instance Normalization norm each sample and each channel
• Group Normalization norm multi channel and sample

• Depthwise Convolution: channel-wise
• Pointwise Convolution: 1*1 convolution

dacay schudle

small learing rate increase very fast, and decay slowly can deal with bad initialization

• training N model, and take the averate
• take N snapshots of training

• randomly initialization of G and D
• inputs (Distribution: \({z_{1}, z_{2}, z_{3}, z_{4}... } : Z\)) from known distribution to G get rough outputs (Distribution: \({z^{'}_{1}, z^{'}_{2}, z^{'}_{3}, z^{'}_{4}... } : Z^{'}\))
• rough outputs and real image (Examples: \({x_{1}, x_{2}, x_{3}, x_{4}..... } : X\)) feed to D
• training D to classify them with mark, and update D Max \(V = \frac{1}{m} \sum^{m}_{i=1} log D(X)\) to 1 Min \(V = \frac{1}{m} \sum^{m}_{i=1} log D(G(Z))\) to 0, so \[max_{d}[E_{x\backsim data} \log D_{d}(x) + E_{z \backsim p(z)} \log (1-D_{d}(G_{g}(z)))]\]

• fix D, feed new inputs from known distribution to G
• get rough outputs again, and pass them to D, and evaluated with mark

• training G, for getting better mark

  max \(V = \frac{1}{m} \sum^{m}_{i=1} log D(G(Z)) = \frac{1}{m} \sum^{m}_{i=1} log D(Z')\) to 1, so \[min_{g}[E_{z \backsim p(z)} \log (1-D_{d}(G_{g}(z)))]\]

  Just like training normal neural network with minimum cross enteopy

\[min_{g}max_{d}[E_{x\backsim data} \log D_{d}(x) + E_{z \backsim p(z)} \log (1-D_{d}(G_{g}(z)))]\]

consider the consistency loss on all examples between Student model and Teacher model.

1. training the student model with labeled data as usual,
2. difference augmented view methodes(scala, rotate…) applying on each unlabled data.
3. passing augmented views(x', x'') of the same data(x) to student and teacher model
4. minimizing the consistency loss of both output \(L_{\mu} = ||f(x') -g(x'')||^{2}\).
5. updating weight of teacher model, \(\Theta'_{t} = \alpha \Theta''_{t-1} + (1-\alpha)\Theta_{t}\)

1. training Teacher model with labeled data as usual
2. using well trianed teacher model to predict unlabled data
3. taking over the confident prediciton(threshold) lable as new labeled data
4. training student model with original and new label data
5. passing the student model weights to teacher model, and predict all data again

Reference Frame \(f_{i}\), color \(c_{i}\), Target Frame \(f_{j}\), predect color \(y_{j}\). \[A_{ij} = \frac{exp(f_{i}^{T}f_{j})}{\sum_{k}exp(f_{k}^{T}f_{j})}, y_{j}=\sum_{i}A_{ij}c_{i}\]

picture divied into patches, predict the relative position of patches.

• Gap between batches, jitter location of patches
• Chromatic abberation, predict the absolute position of patches

Idea: Learn to predict future embeddings linearly. \(Z_{t+k} = W_{k}C_{t}\) Loss: mean squared error not helpful, because encoding = 0 will give perfect loss, positive example are close, and negative example are distant

Maxmize agreement between representations of two views, good contrastive learning need many negative examples.

• MoCo: \(\theta_{k} <- m\theta_{k} +(1-m)\theta_{q}\), decouples batch size of large number of negative exsamples, more complex model
• BYOL: no need for negative examples

• in CNN model, replace the last fully connected layer with 1x1 converlutions layer
• at last upsampling to original size
• ouput original weight * original height * class number with one-hot coding.
• loss funtion, cross entry of pixel-wise: \(\frac{1}{N} \sum_{ij}\sum_{k} \log p_{ij}^{k} t_{ij}^{k}\),
  • imbalanced background not work good for target prediciton, using balanced loss function
  • weight factor r inverse to class frequency
  • dice loss
  • Focal loss
• upsampling: nearest neighbor interpolation, transposed convolutions
• upsampling combining with the corresponding pooling

tranposed convolution can cause artifacts, can avoid by using fixed upsampling(nearest neighbor)

• Contraction: extract semantic information
• Expansion: produce detail segmentation
• Skip connection: copy high-resolution information into decoder

combine feathers representations at multiple scale atrous converlution: dilate filter by implicit zeros in between kenerl elements

Faster R-CNN:

• Loss = \(\sum\) classification loss + \(\sum\) region proposal loss
• RoI pooling: all object map to Rol convolutional features(C*H*W) for region proposal

change the mask size, and predect all at once