I have been establishing baselines for a project at the prime intellect residency and we were using swe-multimodal v2 for performing ablations. I found out that gpt 6 Luna xhigh performed on the benchmark similarly whether I used normal images or OCR or no images at all. This was bit baffling at first but based on some further slop-investigation, I concluded that most of the information required to solve the tasks was provided in the text part of the tasks or was derivable from code. In other words, swe-multimodal is not really testing the multimodality. In this post, I will...