Skip to content
Live
Trending storyDeveloping

Evaluating the Rigor of AI Reviewer Training Assessment Papers

1 reports1 outlets13 hours agoUpdate

The story so far

AI overview

Hugging Face Daily Papers reports that the PaperCritic AI reviewer released by qwen, trained through reinforcement learning, outperforms state-of-the-art models and other training methods in jointly maximizing the precision and recall of rigorous review points. It can also transfer its capabilities to other review fields. The model is trained using the PaperSound-23K dataset, constructed with 1K ICLR submissions and 23K human and LLM-generated review points, and performs excellently on the in-domain test set. Learning ICLR reviews has also been transferred to cross-domain tasks such as Nature journal papers and code reviews. As the number of AI conference submissions grows rapidly, AI reviewers are being considered as an alternative. PaperCritic focuses solely on evaluating rigor, defining the learning objective as jointly maximizing (1) the precision of review points based on rigor and substance, and (2) the recall rate of high-quality reference review points. A quality-labeled dataset PaperSound-23K is constructed using 1K ICLR submissions and 23K human and LLM-generated review points. The AI reviewer model is trained through reinforcement learning, with the F1 score of precision and recall as the reward.

AI-written from the reports · Updated 17 minutes ago

Coverage timeline

Follow the coverage to see every side of the story.

17 Nov
  1. Hugging Face Daily Papers
    PaperCritic: Training AI Reviewers to Assess the Soundness of Papers

    PaperCritic AI reviewers trained through reinforcement learning outperform leading models and other training methods in jointly maximizing the precision and recall of rigorous review points, and their capabilities can be transferred to other review fields. The model is trained using the PaperSound-23K dataset, which includes 1K ICLR submissions and 23K human and LLM-generated review points, and performs excellently on the domain-specific test set. Learning ICLR reviews has also been transferred to cross-domain tasks such as Nature journal papers and code reviews.

Heat over time

Not enough data yet to draw a trend.