Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

A good way to build intuition for how much text fits in a token is by pasting a block of text into a tokenizer playground, like this one: https://huggingface.co/spaces/Xenova/the-tokenizer-playgroun...


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: