skip to main content
ACM Digital Library home
ACM Association for Computing Machinery corporate logo
Advanced Search

Reviewer guidelines

 

Reviewing Procedure

Our timeline for submissions is as follows.

  1. When a paper is submitted to TOIS, it is checked by the Assistant to the Editor-in-Chief, usually within 48 hours.

  2. The Editor-in-Chief (EiC) then scans the paper. If the EiC finds that the paper is not in the scope of the journal or clearly below its standards, the paper can be rejected without review.

  3. The EiC assigns the paper to an Associate Editor (AE), usually within 7 days after submission.

  4. The AE assigns the paper to three reviewers, usually within 7 days after the paper has been assigned to the AE.

  5. The reviewers write detailed technical reviews of the paper. The reviewers usually take 1-2 months to review a submission, but some papers may take longer, e.g., when they are especially complex or are received during the review period of major conferences.

  6. If two reviews come and they both recommend to reject the paper, and the AE agrees, with the reviews, then the AE may decide to reject the paper based on the two reviews. The AE may decide to wait for the third review or even commission additional reviews.

  7. Once the AE has received enough reviews to be able to formulate a recommendation, the AE has one week to recommend a decision to the EiC.

  8. The EiC usually reaches a decision within 3 days after receiving the AEs recommendation.

  9. Possible decisions are Accept, Minor revision, Major revision, and Reject.

  10. Revisions of papers with a "Major revision" decision should be resubmitted within at most three months. Revisions of papers with a "Minor revision" should be resubmitted with a month. Submissions of revisions of papers should always be accompanied with a statement detailing what revisions were implemented and what actions were taken in response to the reviews.

  11. In exceptional cases, revision periods can be extended by two weeks at most.

  12. Revisions of papers with a ‘major revision’ decision can only receive an ‘accept’, ‘minor revision’ or ‘reject’ decision. In other words, no paper can receive a ‘major revision’ decision in a row.

 

Reviewing Guidelines

As a reviewer, you have four weeks to complete your review. If for some reason you find out that you are not able to complete your review within the agreed upon period, please contact the Associate Editor who invited you immediately.

In your review, please touch upon as many of the following points as is practical:

    • Goals: What are the research goals and questions? What is the information retrieval task?

    • Description: Is the description sufficiently detailed for others to replicate the work? Is it clearly written? Is it clearly structured? Do the authors make effective use of examples and illustrations? Papers describing systems should clearly describe the contributions or the principles underlying the system. Papers describing theoretical results should also evaluate their findings experimentally.

    • Evaluation: Do the authors evaluate against suitable state-of-the-art baselines? Do they use state-of-the-art datasets? Are all claims clearly articulated and supported either by empirical experiments or theoretical analyses? Are the experimental results analyzed? Have appropriate significance tests been performed?

    • Importance: Does the paper constitute a significant, technically correct contribution to the field that is appropriate for ACM TOIS? Is it clear how the work advances the current state of understanding, and why the advance matters?

    • Related Work: Do the authors clearly acknowledge and identify the contributions of their predecessors? Is the paper sufficiently different from prior published work (by the author or others) to merit a new publication?

    • Discussion: Are strength and limitations and generality of the research adequately discussed, in particular in relation to related work?

    • Clarity: Is the paper written in a way such that an interested reader with a background in information retrieval, but no special knowledge of the paper's subject, could understand and appreciate the paper's results? In particular,

      • Is it written in a clear, readable style, with good grammar and few (if any) typographical errors?

      • Are the goals and contributions of the work clearly and correctly stated?

      • Are the problem description, approach and evaluation adequately detailed for others to replicate the work?

      • If the paper introduces new terminology or techniques, does it explain why current terminology or techniques are insufficient?

    • Recommendation: Please also recommend a decision: accept, major revision, minor revision or reject. If you recommend major revision or minor revision, please provide a clear list of changes that can be checked by you and the Associate Editor handling the paper upon resubmission.

Thanks to the Journal of Machine Learning Research for inspirations for these guidelines.

 

 

General ACM policies

For more information on specific topics, see the ACM publications policies.