Roger
Clarke
**
19 December 2011
©
Xamax Consultancy Pty Ltd, 2006-11
Available under an AEShareNet licence or a Creative
Commons licence.
This document is at http://www.rogerclarke.com/Bled25/PaperRev.html
The 25th Anniversary of the Bled eConference in June 2012 is being
celebrated with the publication of a Volume that will include a Special Section
of refereed papers.
The editorial team (Roger Clarke, Andreja Pucihar and Joze Gricar), and the
Bled Community as a whole, greatly appreciate your assistance in relation to
the reviewing of papers submitted for the Special Section.
The sections below contain general and specific guidance for reviewers.
For general guidance to reviewers, see below.
The following additional guidelines apply to this Special Section.
Authors were requested specifically to:
- reflect on, and build on, the content of the 24 Bled conferences held to
date - using this
source-material
- focus on a theme that has persisted across multiple conferences
- be future-oriented, i.e. identify the lessons from the Conference's rich
history
Details are in
the
Invitation to Propose Papers.
Please take these points into account while performing your review.
Many journals and conferences in the IS discipline and associated research
domains use the 'double-blind' approach, i.e. reviewers are not provided with
the authors' names (and are supposed to remain ignorant of the authors'
identities and affiliations); and authors are not provided with reviewers'
names and affiliations.
The 'double-blind' approach may be conventional, and it may have some
advantages, but it certainly suffers many serious deficiencies. In this case,
it is clearly inappropriate, because the authors are generally
well-established, and the context is one of community, and the editors want to
encourage constructive communications among authors, reviewers and editors with
a view to achieving both the (vital, but somewhat negative) objective of
quality assurance and the (more positive) objective of quality improvement.
The approach adopted is accordingly double-open. Please note the following
features:
- reviewers:
- are requested to write their comments openly, for the editors and author(s)
- are requested not to contact the author(s) directly, but to conduct all
communications about the paper via the editors
- are welcome to send additional, confidential comments to the editors,
where they consider it appropriate to do so
- authors:
- are requested not to contact reviewers directly, but to conduct all
communications about the paper via the editors
- are welcome to send additional, confidential comments to the editors,
where they consider it appropriate to do so
Could you please provide comments on the paper, reflecting the Guidelines
below.
Could you please also provide:
- an assessment of the Quality of Technical Content, on a 5-point scale (5
is High)
- an assessment of the Quality of Presentation, on a 5-point scale (5 is
High)
- a Recommendation to Accept as-is, Accept subject to minor or substantial
revisions, Accept after Major Revisions, or Reject.
The remainder of this document contains the standard Guidelines for
Reviewers of Bled Conference Papers.
The reviewing of papers is an act of quality assurance. The dimensions of
quality that a journal or conference is concerned about vary, particularly in
the weighting given to them. The quality factors that are considered by the
Bled Outstanding Paper Award Committee are as follows:
- R elevance, or Real World Quality
- A mbition, or Contribution Quality
- R igour, or Academic Quality
- E ase of Access, or Presentation Quality
Regrettably, some journals have sacrificed Relevance in recent years and
prioritised Rigour for its own sake. Some journals, and most conferences,
adopt a more balanced approach. The reviewer needs to take into account the
the particular venue's attitude towards these quality factors.
There are several customers or stakeholders whose interests need to be taken
into account.
Clearly, the immediate purpose of a review is to provide the Editor or
Program Chair with information about the paper's acceptability or
otherwise. But, important as those people are, they operate as a proxy for the
journal's readers or the conference delegates.
The next purpose of a review is to inform the Author(s) about
what needs to be done to improve the paper, possibly to the level needed to get
across the threshhold and achieve acceptance, possibly not.
But it's advantageous to think about the Community with whom
the reviewer is engaging. This may be formed around the nucleus of a
discipline, a research domain, a geographical region, or intersections among
two or more of them. All members of that community stand to gain from a
professional approach to reviewing.
A treatise could be written on what constitutes a good review; but there
are a couple of key characteristics that enable a review to serve the needs of
the editor, the author and the community alike:
- Demonstrated Understanding of the Paper. A lengthy
recapitulation is unnecessary, but the reviewer needs to either summarise the
paper in a couple of sentences, or to convey by other means that they've
grasped what the author was trying to do. The reason this matters is that if
the readers of a review aren't satisfied that the reviewer understood the
paper, they will de-value the comments made
- Positive Features of the Paper, identified briefly. This
should not be overdone, but it should be present. It's very rare that a paper
reaches a reviewer without at least some redeeming features. Especially
where the list of criticisms is substantial, it's important to convey to the
reader that they and their effort aren't worthless. Apart from the morality of
the matter, criticism is more likely to be effective if it's cushioned by some
recognition of worth
- Critique, Expressed Constructively. This is the central
feature: a good review must lead somewhere. The editor or program chair needs
to make a decision; and the author needs suggestions about what they can do
about the features the reviewer doesn't like. If the reviewer's opinion is
that the paper isn't appropriate for this venue and never will be, where does
it belong? Or, at the very least, what next steps should the author take in
order to improve their work?
- Value-Add. It's a cliché, but it's applicable.
It's linked to another cliché: performing a review is an act of
collaboration not competition. The self-confident reviewer doesn't limit
themselves to identifying and helping to eradicate the inadequacies; they also
offer additional perspectives, references and inferences
There are also some traps that it's important for reviewers to avoid. Three
are ethical:
- Don't fail to declare lack of expertise. Reviewers
should decline invitations to review papers in areas they are not sufficiently
familiar with. Where the lack of expertise is partial (e.g. in relation to a
particular research technique), a simple declaration of the reviewer's
limitations, and hence deference to other reviewers on that aspect, is
sufficient
- Don't fail to declare a conflict of interest. Examples
include an association with the research reported on, or with one of the
authors of the paper
- Don't take premature advantage of content. Examples
include providing copies to a colleague or student prior to publication. This
doesn't preclude approaching the editor to seek the author's consent for a
targeted and limited distribution
- Don't delegate unless you exercise control over the quality of the
review. It's unfair to the Program Chair and the Author(s) to assign
the task to a junior colleague or a student, unless you review their
draft comments and the paper itself, adapt the draft as necessary, and take
personal responsibility for the review that is submitted in your name
There are also a couple of corollaries of the four quality indicators
discussed above:
- Don't be emotive, aggressive, sarcastic or demeaning.
Not only do such stylistic problems shake the confidence of junior or otherwise
insecure authors, they also reduce the effectiveness of the advice provided
- Don't be vague about the paper's weaknesses. It helps
neither editor nor author to say that 'this paper makes too limited a
contribution to be considered for this venue', 'the method is weak', 'the
literature search is too narrow', or 'the implications are insufficiently
developed', unless the theme is supported and developed
- Don't criticise without 'actionable advice'. This term
is Allen Lee's. He goes further than merely urging that criticism be expressed
constructively. Criticism shouldn't even be made unless it culminates in a
suggestion about what might be done to address the problem
- The paper is not up to the standard expected for this conference. (Vague,
not actionable)
- The paper needs to be structured better. (Vague, not actionable)
- There is room for improvement in the literature review and theory building
areas. (Vague, not actionable)
- Problem definition, argumentation and conclusions are far away from what
is expected. (The authors clearly need guidance, but they haven't been given
any)
- A number of language mistakes and incoherent sentences could be solved.
(Which??)
- Some parts of the paper could be rewritten in such a way that they improve
comprehension to the less-initiated. (Which??)
- Some paragraphs had to be read more than once, in order to be fully
comprehensible. (Which??)
- The author/s need to spell out briefly the aims and objectives of the
research at an early stage (ideally in the introduction) in order to set a
framework for the rest of the paper
- [In Figure 1,] the semantics of the boxes, circles, and lines should be
more precisely articulated
- The big problem of the paper is that the authors failed to clearly define
the factors listed [in Figure 1]
- The paper is a mix of a research project description & a discussion of
empirical findings & thus lacks focus. I would be interested in both
aspects but since neither part provides an in-depth discussion, I found the
result not very satisfying
- The paper's weaknesses are the lack of in-depth case analysis and the lack
of a strong theoretical discussion. My suggestion is to make more explicit what
the theoretical focus of the article is and build the story line around that
focus
- We learn very little about the set-up of the study. Who were the
participants? How were they selected? What sites did they visit? How often?
When? What were the demographics, and did participants have differing
experience and skills with respect to online buying? Can the study still be
representative, given the low response rate? Was a non-response-bias
identified? And in what ways (if any) can the results be generalised outside
the geographical area in which the study was performed?
Bibliography
Directly relevant articles and notes from within the IS discipline are:
Bieber M. (1997?) 'How to Review' at
http://www-ec.njit.edu/~bieber/review.html
Davison R.M., de Vreede G-J. & Briggs R.O. (2006) 'On Peer Review
Standards For The Information Systems Literature' Commun. AIS 16, 49 (2005)
967-980, at
http://cais.isworld.org/articles/default.asp?vol=16&art=49
- Drawing on the literature, Davison et al. (2006, p. 15) identifies
attributes of a good reviewer as being humaneness, competence, openness,
unbiassedness and absence of prejudice, ethicality, timeliness, persuasiveness
and diligence
Koh C. (2003) 'IS journal review process: a survey on IS research practices
and journal review issues' Infor. & Mngt 40 (2003) 743-756
Lee A.S. (1995) 'Reviewing a Manuscript for Publication' Invited Note in J.
Ops Mngt 13, 1 (July 1995) 87-92, at
http://www.people.vcu.edu/~aslee/referee.htm
Zmud R. (1998) 'A Personal Perspective on the State of Journal Refereeing'
MIS Qtly 22, 2 (September 1998), at
http://www.misq.org/archivist/vol/no22/issue3/edstat.html
A couple of relevant items from other disciplines are:
Black N., van Rooyen S., Godlee F., Smith R. & Evans S. (1998) 'What Makes
a Good Reviewer and a Good Review for a General Medical Journal' J. Am. Med.
Assoc. 280, 3 (1998) 231-233
Finney D.J. (1997) 'The Responsible Referee' Biometrics 53, 2 (June 1997)
715-719
Hames I. (2007) 'Peer Review and Manuscript Management in Scientific Journals'
Blackwell/ALPSP, 2007, from
http://www.blackwellpublishing.com/book.asp?ref=9781405131599
Smith, A.J. (1990) 'The task of the referee' IEEE Computer
23, 4 (April 1990) 65-71
Author
Affiliations
Roger Clarke is Principal of
Xamax
Consultancy Pty Ltd, Canberra. He is also a Visiting Professor in the
Cyberspace
Law & Policy Centre at the
University
of N.S.W., and a Visiting Professor in the
Research
School of Computer Science at the
Australian
National University.