![]() ![]() ![]() The attention operation can be thought of as a retrieval process as well.Īs mentioned in the paper you referenced ( Neural Machine Translation by Jointly Learning to Align and Translate), attention by definition is just a weighted average of values, For example, when you search for videos on Youtube, the search engine will map your query (text in the search bar) against a set of keys (video title, description, etc.) associated with candidate videos in their database, then present you the best matched videos ( values). The key/value/query concept is analogous to retrieval systems. How should one understand the queries, keys, and values The key/value/query formulation of attention is from the paper Attention Is All You Need.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |