[FT] To keep target_for_fewshot_sorting
for another purpose
#240
Labels
feature request
New feature/request
target_for_fewshot_sorting
for another purpose
#240
Hi there!
In the code it's stated that
target_for_fewshot_sorting
might be removed in the future, maybe becauseline["__few_shots"]
is introduced.lighteval/src/lighteval/tasks/requests.py
Line 172 in 506abda
I suggest keeping
target_for_fewshot_sorting
, but for another purpose. To let the user determine the criteria by which fewshot examples are sorted and picked in fewshot manager.My concrete example is in fewshot structured generation in which model is to generate function call json representations. I want to pick the examples, in a
balanced
setting, according to the name of the functions but currently they're picked according to the whole representation of the function call which doesn't make sense as all of them are different.The text was updated successfully, but these errors were encountered: