FP1 - Project Proposal

Due date: 16/10, 11:59PM

Introduction

Transformers are one of the most important neural architectures in Deep Learning recently. They have been initially proposed to solve Machine Translation problems, but today they have been applied in a variety of areas within Artificial Intelligence (AI), including Natural Language Processing, Computer Vision, Robotics, among others. In your final project you will have to implement a Transformer Network “from scratch” in PyTorch to reproduce the results of a paper of your interest.

Instructions

1. Find a Transformer Paper

Your first task is to find a paper that uses a Transformer that solves a problem of your interest. This paper should have public datasets that you can download to train your own transformer. If the authors (or the community) don’t provide a dataset but you already have a similar one, you can use your own data. You have only one month to work on this project, so you won’t have time to curate your own dataset. Ideally, this paper should also provide the code and weights of the model proposed in the paper, so you can compare your results against the ones from the authors. In summary, you need:

1. A published paper proposing a transformer for a problem of your interest;
2. A public dataset to train the proposed transformer;
3. A github repository with the code and weights of the proposed transformer (optional but strongly recommended).

If you don’t have a paper in mind yet, you can search Google Scholar or arXiv for transformer papers in your research area, or look up the following AI/ML conferences:

• AAAI Conference on Artificial Intelligence (AAAI)
• International Joint Conferences on Artificial Intelligence (IJCAI)
• Conference on Neural Information Processing Systems (NeurIPS)
• International Conference on Machine Learning (ICML)
• International Conference on Learning Representations (ICLR)
• The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR).
• Annual Meeting of the Association for Computational Linguistics (ACL)

Your paper doesn’t have to be from one of these conferences. This is just a list of some of the top tier Deep Learning conferences where you can find excellent papers. Here is a few examples of papers that fit well the final project:

• Vaswani, A., et al. “Attention is all you need.” Advances in Neural Information Processing Systems. 2017. - This paper introduced the transformer architecture.
• Parmar, N., et al. “Image Transformers.” International Conference on Machine Learning. 2018.
• Huang, C. Z. A. et al. “Music Transformer: Generating Music with Long-Term Structure.” International Conference on Machine Learning. 2019.

2. Read the paper

Once you have identified a paper you want to reproduce, read the paper carefully and annotate the main contributions of the paper.

3. Write a proposal

Write a proposal including the title of the paper you selected and the following two sections:

1. Introduction (2-5 paragraphs)

Describe the paper motivation, the dataset that was used, and summarize its main contributions in terms of methods and results. Include a footnote with a link to the dataset and/or github repository.

2. Schedule

Include a table with a schedule organized in 4 weeks, where you describe your tasks per week. If you are working in a group (of two), you should differentiate the tasks per author.

Use the NeurIPS template to write your proposal. You don’t need to write an abstract for your proposal. Your proposal should have a maximum of 2 pages.

4. Submission

Submit your proposal as a pdf file to the FP1: Project Proposal task on Moodle.