About the effect of varying the number of QA pairs per image

[qibao77]

Thank you for sharing this nice work. Could you explain why the number of QA pairs per image has a negative impact? Off the top of my head, a single QA pair can only represent one small factor regarding the quality of a caption—so more QA pairs should be able to reflect the quality better. Why this method can avoid reward hacking?

[Cooperx521]

Great question! Thanks for your interest in our work.

1. In fact, we didn’t claim in the paper that “the number of QA pairs per image has a negative impact.” The experiment in Table 4 only shows that when the number of QAs reaches 2, the model’s captioning ability seems close to saturation.

2. Regarding “why more QA pairs are not working”. There’s no doubt that when the QAs become more numerous and diverse, (assuming the QA quality remains high), the reward they provide for evaluating captions should theoretically be higher. The relationship between the number of QAs and performance should, in theory, follow a scaling-law-like curve. The reason why adding more QAs doesn’t yield additional gains might be threefold:
   (1) The number of QAs we currently provide is still insufficient, as the x-axis in such scaling laws is often logarithmic.
   (2) The diversity of QAs we provide is limited; if the QAs are relatively homogeneous, the effective number of unique QAs doesn’t really increase.
   (3) This scaling law saturates quickly. From the perspective of the entire 75k-image dataset, one or two QAs per image are already sufficient to cover nearly all question types. In that case, increasing the diversity of image types would be a more effective way to improve performance.

These are all interesting directions for future work.

3. As for “why not reward hacking?”. The model doesn’t know in advance what each QA pair will ask about. Therefore, the only way to answer the questions correctly is to make the caption as comprehensive as possible, ensuring the visual information is fully conveyed to the LLM that answers the questions.

Best,
Long