I have been establishing baselines for a project at the prime intellect residency and we were using swe-multimodal v2 to run ablations. I found out that GPT 6 Luna xhigh performed on the benchmark similarly whether I used normal images or OCR or no images at all. This was a bit baffling at first but based on further investigation (and burning a lot of tokens), I concluded that most of the information required to solve the tasks was provided in the text part of the tasks or was derivable from code. In other words, swe-multimodal is not really testing the...