Machine Learning Notes

These are my notes made during watching Andrej Karpathy’s tutorials on YouTube. I have also more theoretical (overview) notes about NLP and ML.

The tutorials are these:

• backpropagation
• language modelling I
• language modelling II
• language modelling III

(Py)Torch

• Tensor
  • .view(<shape>) will take the internal linear (memory) representation of tensor and layout it according to the wanted shape
    • very efficient (better than e.g. torch.cat(torch.unbind(tensor, 1), 1)
• arange similar to range in Python
• randn(<shape> will fill with numbers from normal distribution
• -1 in shape tells torch to infer the dimension
• squeeze
• torch.linspace(from, to, steps) is like range in Python but works for floats
• sum per row: P.sum(1, keepdim=False)
• broadcasting semantics
  • from numpy
  • when a binary operation is defined for two tensors:
    • both dimensions are equal
    • one of them is 1
    • one of them doesn’t exist
• toch.nn.functional.one_hot
• common way of importing functional: import torch.nn.functional as F
• @ is vector multiplication
• indexing with a range: x[torch.arange(10), y]
• with torch.no_grad(): ... tells torch to not include what follows in backpropagation
• torch.zeros_like(tensor) will create a new tensor with the shape of tensor with all zeros
• torch.all_close(t1, t2) will compare tensors with some tolerance