But I revisited it to fact-check myself, and the ~60B floor is related to stability of personality induced via a system prompt. The actual floor seems to be in the ~8-13B range; in the linked paper they found that the instruction-tuned version of llama 2 showed some personality consistency and in this one (arxiv.org/html/2501.15427v1) llama 3 8B matches gpt-4o's performance in persona gym.
I am now more confused though. First, I don't know what to make to of the benchmarks, and second, seems like there are a lot of basic questions that seem to not have been investigated at all??
"I don’t think very many people try very hard with model identity" is spot on.
Isn’t model identity unstable for models below ~60-70B params?
Thanks, I don't recall this paper, looks interesting even if incomplete!
Lots of things are worse in smaller models, but I’ve never heard this before. I don’t think very many people try very hard with model identity.
Can you point me at something?
So the paper I was referencing is this one: https://arxiv.org/pdf/2307.00184
But I revisited it to fact-check myself, and the ~60B floor is related to stability of personality induced via a system prompt. The actual floor seems to be in the ~8-13B range; in the linked paper they found that the instruction-tuned version of llama 2 showed some personality consistency and in this one (arxiv.org/html/2501.15427v1) llama 3 8B matches gpt-4o's performance in persona gym.
I am now more confused though. First, I don't know what to make to of the benchmarks, and second, seems like there are a lot of basic questions that seem to not have been investigated at all??
"I don’t think very many people try very hard with model identity" is spot on.