Join the attention revolution! Learn how to build attention-based models, and gain intuition about how they work
Why Another Transformer Tutorial?
Since they were first introduced in Attention Is All You Need (2017), Transformers have been the state-of-the-art for natural language processing. Recently, we have also seen Transformers applied to computer vision tasks with very promising results (see DETR, ViT). Vision Transformers, for example, now outperform all CNN-based models for image classification! Many people in the deep learning community (myself included) believe that an attention revolution is imminent — that is, that attention-based models will soon replace most of the existing state-of-the-art methods.
All deep learning practitioners should familiarize themselves with Transformers in the near future. Plenty of other Transformer articles…
I show that Graph Convolutional Networks are (almost) a special case of Transformers, and discuss how that affects our interpretation of each.
Introduction
Transformers and Graph Convolutional Networks (GCNs) have grown tremendously in popularity over the past few years. At first glance, these two model types appear very different. They were designed to solve two very different problems:
- Transformers improved on attention-based models in NLP.
- GCNs operate on unstructured data types, such as 3D point clouds.
GCNs have received a small amount attention in NLP, but otherwise, the two models are used for different applications. In this article, I will argue…
Deep learning papers, blog posts, Github repos, etc. that I liked this week.
This is the fourth edition of my weekly update on deep learning. Every Thursday, I’ll release a new batch of research papers, blog posts, Github repos, etc. that I liked over the past week. Links are provided for each featured project, so you can dive in and learn about whatever catches your eye. If you missed the last few editions, you can find them here:
It’s hard to find reliable sources of information when starting out in deep learning. These will save you time, effort, and headaches.
There are tons of free machine learning resources out there, but the sheer volume makes it difficult to sift through them. Not all of them are reliable or well-written. When you’re starting out in deep learning, a poorly written tutorial can do more harm than good. I sincerely hope that never happens to you, but it’s affected me more than a few times.
These are my favorite resources for deep learning news, which I’ve accumulated over the past…
Deep learning papers, blog posts, Github repos, etc. that I liked this week
This is the third edition of my weekly update on deep learning. Every Thursday, I’ll release a new batch of research papers, blog posts, Github repos, etc. that I liked over the past week. Links are provided for each featured project, so you can dive in and learn about whatever catches your eye. If you missed the last few editions, you can find them here:
All thoughts and opinions…
A New Medium Publication for Deep Learning
I’m excited to introduce The DL— a Medium publication with a focus on deep learning. Deep learning has absolutely exploded in recent years. Seemingly every tech company/startup boasts a killer AI feature, and ML-related skills are in extremely high demand. Deep learning research is also evolving very quickly, and new groundbreaking research is published almost every week. This makes it difficult for practitioners to stay up-to-date on the latest algorithms, frameworks, and best practices.
Deep learning papers, blog posts, Github repos, etc. that I liked this week
This is the second edition of my weekly update on deep learning. Every Thursday, I’ll release a new batch of research papers, blog posts, Github repos, etc. that I liked over the past week. Links are provided for each featured project, so you can dive in and learn about whatever catches your eye. If you missed last week’s edition, you can find it here. All thoughts and opinions are my own. Follow me or check back next week for more. Enjoy!
Very Deep VAEs
[ArXiv][Github]
OpenAI recently showed that Variational AutoEncoders can outperform other likelihood-based generative models at creating realistic 2D images. Although the…
Deep learning papers, blog posts, Github repos, etc. that I liked this week.
Every week, I’ll be posting some of the projects in deep learning that caught my eye, ranging from research papers to blog posts and Github repos. All thoughts and opinions are my own. Follow me or check back next week for more. Enjoy!
Stylized Neural Painting
[Project Site][Paper][Github]
This project completely blew me away. Stylized Neural Painting uses deep learning to generate realistic painting artworks. Similar to previous methods like Neural Style Transfer, it can control the style of the generated artwork by providing a style image. But neural style generates stylized images on a pixel-by-pixel basis, whereas SNP actually predicts the brushstrokes needed…
Math and code for efficiently computing large convolutions with FFTs.
Note: Complete methods for 1D, 2D, and 3D Fourier convolutions are provided in this Github repo. I also provide PyTorch modules, for easily adding Fourier convolutions to a trainable model.
Convolutions
Convolutions are ubiquitous in data analysis. For decades, they’ve been used in signal and image processing. More recently, they became an important ingredient in modern neural networks. You’ve probably encountered convolutions if you work with data at all.
Mathematically, convolutions are expressed as:
BYOL is a surprisingly simple method to leverage unlabeled image data and improve your deep learning models for computer vision.
Note: All code from this article is available in this Google Colab notebook. You can use Colab’s free GPU to run/modify the experiments yourself.
Self-Supervised Learning
Too often in deep learning, there just isn’t enough labelled data. Manually labeling data is too time intensive, and outsourcing the labor can be prohibitively expensive for small companies or individuals. Self-supervised learning is a nascent sub-field of deep learning, which aims to alleviate your data problems by learning from unlabeled samples. The goal is simple: train a model so that similar samples have similar representations. …
