Evaluating scientific papers

Evaluating scientific papers

Scientific journals constitute the main publication channel in many domains, and the international scientific community has through peer review developed a quality control system for the content of these journals. Although not always water-tight, the fact that their articles have been screened and approved by (in general) two experts in the field constitutes an external positive evaluation of the described research results. The same can be said about papers in conference proceedings, either in book or in journal format, although it is known that the peer review here is in general somewhat less strict. In all other cases, evaluators should be aware of the necessity to try to obtain themselves some expert evaluation of the quality of the paper; we give some hints for doing this below.

The international scientific community is also paying much attention to quantitative data that are supposed to measure the impact of a paper or of the journal in which the paper is published, based primarily on the number of citations that the paper or the journal receives in other publications worldwide. We refer to the page about bibliometric methods for a more detailed description and critical analysis of these methods.

We will now describe some elements that one should consider in the evaluation of a scientific publication. The reader should have no difficulty in adapting the meaning of the statements to other formats of publication, wherever the term “paper” or “article” is used.

A. The content

The true evaluation of the content of a paper can, of course, be performed only by an expert in the field. We list the most important aspects to be considered during an independent evaluation of the content:

• What is the scientific quality of the paper? Does the author have a clear understanding of the present state of knowledge in his domain? Did the author follow the appropriate steps to attain his results? Is his conclusion satisfactory?
• Was the approach of the research sufficiently comprehensive? Did the author take into account all important aspects of the problem?
• What is the originality contributed by this paper? Does the paper provide new insights in the domain it is dealing with
• What kind of impact may be expected for both the short and the long term?

B. The manuscript

Having performed successfully a high-quality research project is a necessary but not a sufficient condition for achieving a good paper. Through his publication, the researcher is sharing his findings with the worldwide scientific community, and for this reason the message should be clear and focused. Writing a good paper is an art in itself. Special points of attention are:

• Is the title well chosen (explaining the content, yet not too long)?
• Is the abstract well written and properly structured (generally: objective(s), methods, results, conclusions) and does it summarize the most important aspects of the paper?
• Are the keywords appropriate?
• Is the overall structure of the paper clear and appropriate (generally: introduction, methods, results, discussion, conclusion)?
• Does the introduction clearly identify the underlying problem in the context of the present knowledge in this domain? Does it clearly state the aim(s) of the paper?
• Does the author acknowledge related published research by others?
• Does the author clearly explain the methodology followed for tackling this problem, and the reasons for using this specific methodology?
• If the paper is based on a statistical analysis, is this performed in the correct way? Is the sampling unbiased and sufficiently large?
• How good is the quality of the figures? Is the scale of the figures clearly indicated?
• Where available, are numerical data presented in figures or tables, and are these well-organized and structured?
• Is there a database with original data linked to this paper? What is its value?
• Is the conclusion well motivated and sufficiently discussed?
• Is the language used clear and correct (vocabulary, orthography, grammar, ...)?
• Is there no unnecessary repetition of data (text, figures, tables)?
• Are the references relevant and complete, up-to-date and correctly formatted? Are the cited items publicly accessible (although not necessarily for free)?

C. Availability and visibility

• Is the selected journal the appropriate publication channel for reaching the community or public for which it is intended? (Remark: The publication in local journals – if necessary in a local language and style – may be desirable for typical development research on applied topics; this can be done in parallel with a more international publication.
• Is the journal covered by major international bibliographic databases, so that readers can easily be alerted about its contents?
• Is the journal published by one of the large international publishers or by a local enterprise (with limited distribution channels)?
• Is the paper published in an ‘open access’ journal, with global availability without delay or financial barriers? Such open access papers are available free of charge from the Internet - either via open access journals (the ‘golden road’) or via freely accessible institutional or subject repositories indexed by the major open access harvesters (the ‘green road’). While many researchers worldwide still use the selective ISI group of journals as both their main source of scientific information and as their favored outlet for publishing research results, there is an increasing amount of evidence that open access publications are generally cited more often than papers available via a paid subscription only, thus delivering enhanced visibility, use and scientific impact.
• Is the paper - simultaneously with its publication in a journal - also deposited in an institutional repository that follows the OAIMHP (Open Archives Initiative Metadata Harvesting Protocol), so that it will turn up in searches by the major international open access repository harvesters (e.g. BASE, openAIRE, OAIster) and Google Scholar?
• Does the author specify under which copyright provisions his results are published?(If nothing is specified, one should in principle ask the author’s permission before applying or further elaborating on his results, which may be an impediment to useful applications in the developing world. Many of the Open Access journals and repositories automatically include a license that allows using the results – under the condition of proper citation of the original work – as specified by one of the Creative Commons licenses.)
• Has the paper received a persistent label or identifier such as the Digital Object Identifier (DOI)?

•  
• Scopus

  • 

  Multidisciplinary abstract and citation database: journals, conference papers, trade publications, book series, patents. Scopus quick guide | User rights | More information about the database
• Google Scholar

  A web search engine for scientific information. When you log in with your University of Oulu user ID, you have access to the licensed electronic resources also from outside the University of Oulu Network. Google Scholar quick guide
• SciFinder-n (Chemical Abstracts)

  • 

  An easy-to-use interface for the world's leading chemical information sources, millions of literature and patent references, as well as chemical compounds and reactions containing information of CAS databases, such as the Chemical Abstracts database. Also contains Medline. Only the students and staff of University of Oulu are entitled to use this database. All users need to create a personal account to use SciFinder-n. You will get instructions for registration from [email protected]. Note: the IP address controls have been lifted until January 31, 2022 for easier access to SciFinder-n also off campus.
  SciFinder-n quick guide | Rights to use | SciFinder-n Content
• CiteSeerX An evolving scientific literature digital library and search engine that focuses primarily on literature in computer and information science.
• MathSciNet

  • 

  MathSciNet is a comprehensive database covering the world's mathematical literature of the past 61 years. Quick Guide

Field-Weighted Citation Impact (FWCI)

Field-Weighted Citation Impact (FWCI) is the ratio of the total citations actually received by the denominator’s output, and the total citations that would be expected based on the average of the subject field. To Scopus database it is sourced directly from SciVal.

• Scopus
  • 
  Multidisciplinary abstract and citation database: journals, conference papers, trade publications, book series, patents. Scopus quick guide | User rights | More information about the database
• SciVal
  • 
  SciVal offers access to the research performance of 4,600 research institutions and 220 countries worldwide. It is a powerful tool for benchmarking and characterizing research and research collaboration of different entities such as researcher, research group, research institution or country. SciVal uses Scopus content from 1996 onwards. Registration is needed. SciVal metrics guidebook | Snowball metrics | More information

Examples of top cited articles listing

CiteSeerX most cited articles in the field of computer science

ScienceWatch: hot papers