objc.io

Swift Talk # 491

Building a Language Model: Tokenization

Preview

In the full episode21:21

  1. Introduction
  2. Implementing Token Vocabulary Generation
  3. Finding the Most Common Token Pairs
  4. Processing Frequencies and Replacing Tokens
  5. Switching to Larger Input
  6. Optimizing Performance with Word Frequency Counting
  7. Next Steps and Applications

This episode is exclusive to Subscribers

Become a subscriber to watch future and all 336 current subscriber-only episodes, plus enjoy access to episode video downloads and 30% discount for your team members.

Become a subscriber

Recent Episodes

See All

Unlock Full Access

Subscribe to Swift Talk

  • Watch All Episodes

    A new episode every week

  • icon-benefit-download Created with Sketch.

    Download Episodes

    Take Swift Talk with you when you're offline

  • Support Us

    With your help we can keep producing new episodes

Pricing & Sign Up