Attention Is All You Need - Part 4

1 4 25
calendar_today agoschedule3 min read
— Originally published at dev.to

Hello, I'm Ganesh. I'm building git-lrc, an AI code reviewer that runs on every commit. It is free, unlimited, and source-available on Github. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.

In a previous article, I explained about the embedding and preprocessing of input.

In this article we will discuss step 2 of the transformer model, i.e. position encoding.

What is Positional Encoding?

Positional encoding is a technique used to encode the position of a word in a sentence. It is used to overcome the limitation of the transformer model that it cannot process the input in parallel.

Basically whenever the input text is given to the transformer model.

say

The Lion attacked Deer.
The Deer attacked Lion.

The token generated almost like this

791, 33199, 18855, 64191, 627, 791, 64191, 18855, 33199, 13

The meaning is different but the token created same.

Which will effect vector embeddings

How Positional Encoding Will Solve?

So, basically tokens of "Lion" and "attacked" will be same in both the sentences.

This will lead to the loss of position information. To overcome this, positional encoding is used.

We use position encoding for encoding position of words. We assign a vector to each position.

To get rid of this problem we use positional encoding.

This is the formula for positional encoding:

In this next step, we will get the positional information for each word correctly without losing any information.

So, we completed step 2 of the transformer model.

Conclusion

We found out text that has different meanings has the same tokens and loses information of the position of the word in the sentence.

To overcome this, we use positional encoding.

Reference: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

git-lrc

Any feedback or contributors are welcome! It’s online, source-available, and ready for anyone to use.
⭐ Star it on GitHub: https://github.com/HexmosTech/git-lrc

1.1k Points30 Badges1 4 25
44Posts
5Comments
3Followers
3Connections
I am tech enthusiast, IoT innovator, software developer.
Build your own developer journey
Track progress. Share learning. Stay consistent.
🔥 Join developers growing publicly
Share your knowledge, build in public, and grow your developer presence with a global community.

More Posts

Attention Is All You Need - Part 5

Ganesh Kumar - Jun 1

Attention Is All You Need - Part 1

Ganesh Kumar - Apr 29

Attention Is All You Need - Part 2

Ganesh Kumar - Apr 29

Your Backup Data Knows More Than You Think. HYCU aiR Is Finally Asking It the Right Questions.

Tom Smithverified - May 14

Attention Is All You Need - Part 6

Ganesh Kumar - Jun 1
chevron_left

Related Jobs

View all jobs →

Commenters (This Week)

7 comments
5 comments
4 comments

Contribute meaningful comments to climb the leaderboard and earn badges!