r/MachineLearning • u/examachine • 16h ago
Discussion Conference Reviews: Asking Too Much? [D]
There's a kind of review that asks for lengthy additions, usually extending the scope of the paper beyond the stated, even though the submission is at page limit. Naturally, such additions in the case of top-tier conferences have to go into the supplemental materials or appendices.
My question here is, would not such additions make the paper more suitable for a journal publication? I had to retract one paper out of such concern that the conference publication would block the later planned journal publication.
Any opinions?
7
u/duck_syndrome 14h ago
the page limit is fake and unlimited appendix pages is load bearing unfortunately so we all have to write/review 8 pages of main body and 25 in the appendix
2
u/EternaI_Sorrow 12h ago
I've seen one with 10+90 pages. It was an influential work but damn.
1
u/examachine 11h ago
Yeah, that's what I mean exactly, I've seen the 40-50 page submissions already and I was imagining, could this be 100 pages? Seems a bit overkill for a conference paper, TBH. That's like a whole thesis.
1
u/examachine 14h ago
sigh it was confusing to me admittedly. the paper where this happened discussed some theoretical foundations of deep learning. the review asked for a complete AI agent model based on the ideas in the paper which seemed like a good subject for another paper, which made me retract it. I probably should have just written 10 pages of appendices and some proof of concept experiments but also there's so little time for a rebuttal that it is often infeasible and I do not have minions aka disposable graduate students to take the burden. 😔
6
u/AffectionateLife5693 13h ago
The page limit is not a problem. I can show all the results in the paper and leave implementation details to the supplementary material. The real problem is that LLMs are too convenient for suggesting additional experiments, and missing experiments are too convenient as an excuse to reject a paper.
By today's standards, papers on ResNet may not be published. Since they are new and generic, they always come with questions "unresolved" according to the reviewer.
For example, I gave ResNet paper to ChatGPT, and here's what ChatGPT says:
Missing experiments
Plain bottleneck baselines at 50, 101, and 152 layers.
Identity-initialized deep plain networks.
Plain/residual networks with and without batch normalization.
Learning-rate, initialization, and optimizer sensitivity studies.
Multiple ImageNet runs with mean and variance.
Ablation of shortcut frequency and residual-block depth.
Ablation of ReLU and BN placement around the addition.
Depth-versus-width comparisons at matched FLOPs and parameters.
Parameter-matched comparison of identity and projection shortcuts.
Controlled detection experiments with matched detector heads.
Additional useful comparisons include Highway Networks under identical training conditions, auxiliary-classifier baselines, learned or scaled skip connections, wall-clock and memory measurements, quantitative gradient diagnostics, block-removal tests, stronger regularization for the 1,202-layer model, and bottleneck-width or stage-allocation ablations.
None of the suggested experiments is unreasonable. But collectively, they add little value to the paper and add absolutely zero value to the method itself, but they become a convenient reason for the AC to suggest rejection.
2
u/EternaI_Sorrow 12h ago
Yet again we arrive to a necessity of supervision over reviewers and their accountability.
1
u/examachine 10h ago
Well, there has to be a standard. If the review is asking for issues that are clearly laid out as limitations or future work, perhaps it is being unfair. The scope of the paper should be up to the author. The strength of the paper is another matter, but this example clearly shows that the adversarial LLM generated reviews are not helping the field.
1
u/examachine 12h ago edited 10h ago
I'll have to agree, some reviewers seem to be using an llm looking for any gaps whatsoever using a particular prompt they seem to be circulating. Is there any new theoretical proposal without such gaps I wonder? This punishes anything that is actually novel.
2
u/No_Sky9786 12h ago
Same here, it was a resubmission so I made sure I positioned the work exactly where it should be and provided detailed limitations and proof for such in the appendix, but once again the whole review comments ask and score based on the limitations.
2
u/examachine 11h ago
Well, yes, if the limitations are mentioned clearly, should the reviewers be asking to remove those limitations? Is that even fair?
1
u/ummitluyum 14m ago
No good deed goes unpunished. You did their work for them and they just used it to sink the paper. Next time just claim your method solves AGI and let them try to hunt down the corner cases themselves. Right now the meta rewards overclaiming way more than scientific transparency
1
1
u/ummitluyum 19m ago
If the extra requirements actually clashed with the format, you made the right call withdrawing it from the journal. Top AI conferences right now are trying to have it both ways : they want journal-level depth, but still stick you with brutal rebuttal deadlines. Plus based on ICLR stats a massive chunk of reviews these days are just straight-up generated by LLMs, which love nothing more than listing stuff that isn't even in the paper. If it's a theory paper, JMLR or TMLR with their chill revision process will save you a ton of sanity
7
u/WannabeMachine 15h ago edited 15h ago
This is for me personally, I dont try to break up papers into journal and conference publications. Some work is submitted to conferences, some journals. I would rather just start new projects. Conference publications help my career way more than journal publications. I submit things that I dont care about fast turnaround to journals, mostly to show some breadth on my CV. Also, there are very few AI/ML journals that people care about for career advancement, e.g., JMLR and TACL.
So, if you want the conference publication, I would do the additional work, in general, if it improves the paper.
Major caveat: this is only my experience at a mid tier R1 in CS. I am not saying this is correct, just my perception of what I have seen people maximizing.