Bled 25th Anniversary Special Section
Guidelines for Reviewers

Roger Clarke **

19 December 2011

© Xamax Consultancy Pty Ltd, 2006-11

Available under an AEShareNet Free
for Education licence or a Creative Commons 'Some
Rights Reserved' licence.

This document is at http://www.rogerclarke.com/Bled25/PaperRev.html


Introduction

The 25th Anniversary of the Bled eConference in June 2012 is being celebrated with the publication of a Volume that will include a Special Section of refereed papers.

The editorial team (Roger Clarke, Andreja Pucihar and Joze Gricar), and the Bled Community as a whole, greatly appreciate your assistance in relation to the reviewing of papers submitted for the Special Section.

The sections below contain general and specific guidance for reviewers.

Specific Guidance

For general guidance to reviewers, see below.

The following additional guidelines apply to this Special Section.

A. Desirable Characteristics of Papers for the Special Section

Authors were requested specifically to:

  1. reflect on, and build on, the content of the 24 Bled conferences held to date - using this source-material
  2. focus on a theme that has persisted across multiple conferences
  3. be future-oriented, i.e. identify the lessons from the Conference's rich history

Details are in the Invitation to Propose Papers.

Please take these points into account while performing your review.

B. Double-Open Process

Many journals and conferences in the IS discipline and associated research domains use the 'double-blind' approach, i.e. reviewers are not provided with the authors' names (and are supposed to remain ignorant of the authors' identities and affiliations); and authors are not provided with reviewers' names and affiliations.

The 'double-blind' approach may be conventional, and it may have some advantages, but it certainly suffers many serious deficiencies. In this case, it is clearly inappropriate, because the authors are generally well-established, and the context is one of community, and the editors want to encourage constructive communications among authors, reviewers and editors with a view to achieving both the (vital, but somewhat negative) objective of quality assurance and the (more positive) objective of quality improvement.

The approach adopted is accordingly double-open. Please note the following features:

  1. reviewers:
  2. authors:

C. Evaluation

Could you please provide comments on the paper, reflecting the Guidelines below.

Could you please also provide:

The remainder of this document contains the standard Guidelines for Reviewers of Bled Conference Papers.

The Function of Reviewing

The reviewing of papers is an act of quality assurance. The dimensions of quality that a journal or conference is concerned about vary, particularly in the weighting given to them. The quality factors that are considered by the Bled Outstanding Paper Award Committee are as follows:

Regrettably, some journals have sacrificed Relevance in recent years and prioritised Rigour for its own sake. Some journals, and most conferences, adopt a more balanced approach. The reviewer needs to take into account the the particular venue's attitude towards these quality factors.

Who is the Reviewer Working For?

There are several customers or stakeholders whose interests need to be taken into account.

Clearly, the immediate purpose of a review is to provide the Editor or Program Chair with information about the paper's acceptability or otherwise. But, important as those people are, they operate as a proxy for the journal's readers or the conference delegates.

The next purpose of a review is to inform the Author(s) about what needs to be done to improve the paper, possibly to the level needed to get across the threshhold and achieve acceptance, possibly not.

But it's advantageous to think about the Community with whom the reviewer is engaging. This may be formed around the nucleus of a discipline, a research domain, a geographical region, or intersections among two or more of them. All members of that community stand to gain from a professional approach to reviewing.

Indicators of Quality in a Review

A treatise could be written on what constitutes a good review; but there are a couple of key characteristics that enable a review to serve the needs of the editor, the author and the community alike:

  1. Demonstrated Understanding of the Paper. A lengthy recapitulation is unnecessary, but the reviewer needs to either summarise the paper in a couple of sentences, or to convey by other means that they've grasped what the author was trying to do. The reason this matters is that if the readers of a review aren't satisfied that the reviewer understood the paper, they will de-value the comments made
  2. Positive Features of the Paper, identified briefly. This should not be overdone, but it should be present. It's very rare that a paper reaches a reviewer without at least some redeeming features. Especially where the list of criticisms is substantial, it's important to convey to the reader that they and their effort aren't worthless. Apart from the morality of the matter, criticism is more likely to be effective if it's cushioned by some recognition of worth
  3. Critique, Expressed Constructively. This is the central feature: a good review must lead somewhere. The editor or program chair needs to make a decision; and the author needs suggestions about what they can do about the features the reviewer doesn't like. If the reviewer's opinion is that the paper isn't appropriate for this venue and never will be, where does it belong? Or, at the very least, what next steps should the author take in order to improve their work?
  4. Value-Add. It's a cliché, but it's applicable. It's linked to another cliché: performing a review is an act of collaboration not competition. The self-confident reviewer doesn't limit themselves to identifying and helping to eradicate the inadequacies; they also offer additional perspectives, references and inferences

What Not To Do

There are also some traps that it's important for reviewers to avoid. Three are ethical:

There are also a couple of corollaries of the four quality indicators discussed above:


Some Examples of Good and Bad Content in Reviews
Bad

Good


Bibliography

Directly relevant articles and notes from within the IS discipline are:

Bieber M. (1997?) 'How to Review' at http://www-ec.njit.edu/~bieber/review.html

Davison R.M., de Vreede G-J. & Briggs R.O. (2006) 'On Peer Review Standards For The Information Systems Literature' Commun. AIS 16, 49 (2005) 967-980, at http://cais.isworld.org/articles/default.asp?vol=16&art=49

Koh C. (2003) 'IS journal review process: a survey on IS research practices and journal review issues' Infor. & Mngt 40 (2003) 743-756

Lee A.S. (1995) 'Reviewing a Manuscript for Publication' Invited Note in J. Ops Mngt 13, 1 (July 1995) 87-92, at http://www.people.vcu.edu/~aslee/referee.htm

Zmud R. (1998) 'A Personal Perspective on the State of Journal Refereeing' MIS Qtly 22, 2 (September 1998), at http://www.misq.org/archivist/vol/no22/issue3/edstat.html

A couple of relevant items from other disciplines are:

Black N., van Rooyen S., Godlee F., Smith R. & Evans S. (1998) 'What Makes a Good Reviewer and a Good Review for a General Medical Journal' J. Am. Med. Assoc. 280, 3 (1998) 231-233

Finney D.J. (1997) 'The Responsible Referee' Biometrics 53, 2 (June 1997) 715-719

Hames I. (2007) 'Peer Review and Manuscript Management in Scientific Journals' Blackwell/ALPSP, 2007, from http://www.blackwellpublishing.com/book.asp?ref=9781405131599

Smith, A.J. (1990)  'The task of the referee' IEEE Computer 23, 4 (April 1990) 65-71


Author Affiliations

Roger Clarke is Principal of Xamax Consultancy Pty Ltd, Canberra. He is also a Visiting Professor in the Cyberspace Law & Policy Centre at the University of N.S.W., and a Visiting Professor in the Research School of Computer Science at the Australian National University.