Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

It updates token logits (probabilities) after every token before sampling. I don't think this is very common yet.


Right, there are many folks (dozens of us!) yelling about logit processors and building them into various frameworks.

The mostly widely accessible form of this is probably BNF grammar biasing in llama.cpp: https://github.com/ggerganov/llama.cpp/blob/master/grammars/...


Still rare, but I wrote a whole paper last year about what happens when you use this functionality (a lot, including defeating any kind of RLHF!)

https://aclanthology.org/2022.cai-1.2.pdf


anecdotal counter evidence, I've seen multiple projects / papers manipulating the logits, it's a very common thing to think of doing now to improve performance (by eliminating bad options from consideration)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: