Can't I use the max_tokens (set to 1) and logit_bias parameters? Not saying I wa... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		Kiro on Aug 15, 2023 \| parent \| context \| favorite \| on: Show HN: LLMs can generate valid JSON 100% of the ... Can't I use the max_tokens (set to 1) and logit_bias parameters? Not saying I want to do this. I just want to understand how this works.

baobabKoodaa on Aug 16, 2023 [–]

Not sure exactly what is logit_bias, but after Googling for 5 seconds it seems to be an OpenAI parameter that's not available in HuggingFace transformers?

Anyway, if your idea is to make one API call per token, the biggest problem with that approach is that it would be really slow to do that.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact