AES and Its Papers

[bluesman]

This is an explanation and defense of the peer review process followed by JAES and hundreds of top medical and scientific journals.  It is not in any way a defense of AES, which I left about 20 years ago after being a member for about that long. They do it the same way it's done by every top publication and specialty society with which I'm associated, including JAMA, NEJM etc.  I've been the editor of 2, associate or section editor of 5, and reviewer for 15 or so of the world's top medical journals over the last 40 years, and JAES' PR guidelines are identical to those of the best of the best.  Here's a link to the AES submission review process description, which clearly describes the fact that papers submitted for presentation are subjected to the same review process used for pure journal submissions.  Unless this has been revised since I left the AES, it's still in effect.

 

For virtually all top quality scientific journals, reviewer group members are (as stated in another post) annually thanked and identified along with publication of annual submission / review / acceptance stats.  For example, here's the 2019 version in the Journal of the American Medical Association with publication stats, and here's the actual reviewer list (access is usually for members only, but these links should open a publicly accessible version made available by JAMA). But the identity of reviewers of specific submissions is widely kept confidential, for reasons I'm about to discuss

 

The two major sets of recommendations and guidelines for top quality scientific journalism are the "Uniform requirements for manuscripts submitted to biomedical journals" and the "Best Practice and Guidance" (guidelines, guidance, and other documents) of the Committee on Publication Ethics.  Here's a link to their Ethical Guidelines for Peer Reviewers, which clearly state on page 4 of 12 that peer reviews are to be anonymous for pre-publication review ("Peer reviews are published but not signed").  I wrote the guidelines for publication at the university in which I'm still a professor.  We have a full range of basic science programs (undergraduate, Masters, PhD, fellowships and postdoc) as well as a medical school.  I've researched this intensively, and I know of no authority that recommends open disclosure of the pre-publication reviewers of specific papers.

 

Anonymity is an intergral part of many processes that seriously affect us every day but are totally transparent to most of you, e.g. IRBs (Institutional Review Boards) that approve and oversee clinical research trials, grand jury deliberations that determine whether or not an accused person will be indicted, etc.  The reasons for this are manifold and have been agreed upon by countless people whose accomplishments and reputations are sterling.  Both sides of the coin are represented, in that peer reviewing offers opportunity for retribution, IP theft, inappropriate support and promotion, competitive interference etc - in both directions.  Authors are identified to potential reviewers in order to identify potential COI.  And although this is admittedly on an "honor system" basis, the senior editorial staff of reputable scientific journals are generally quite familiar with their reviewers, their interests, and areas of active research and collaboration. The initial choice of potential reviewers is guided by this knowledge.

 

Does it always work well?  Of course it doesn't.  Some reviewers only blow their own horns, some are way out of their leaegues, and some are trading on reputations they either don't deserve or ceased to justify years before.  Here's a little story you'll find amusing.  Several years ago, I discovered that a fairly common complication of an operation done frequently around the world had been misdiagnosed historically and was being treated incorrectly for decades. What was thought to be a mechanical distortion of a specific cartilage that has to be cut during the procedure turned out to be a benign cartilaginous tumor triggered by the trauma of surgery.  When reoperating the second such case in my own patients, I sent the tissue to our pathology department - and the surgical pathologist with whom I worked most closely (who was also Chief of Surgical Pathology) confirmed with several specialized tests that it was a traumatic chondroma, as I suspected.

 

I then gathered info from colleagues around the country on similar cases and learned that simple reshaping was doomed to failure.  So I did an IRB supervised prospective review of every such patient I treated, removing the offending cartilage for pathologic examination and reconstructing the area with cartilage from the ear.  When we had 9 cases with identical pathologic findings, we wrote it up and submitted it to one of the most prestigious journals in my field.  It was rejected on the basis of this reviewer's comment: "I do not see any evidence of the pathologic process postulated by the authors.  I suggest that they have their specimens reviewd by a competent pathologist".

 

Once the steam stopped coming out of our ears, I sent the manuscript and all the slides to the gentleman who was senior head and neck pathologist for the Armed Forces Institute of Pathology at the time - his nickname for decades was "dean of American head and neck pathologists", and he was in fact that good.  His response was sent directly to the editor of the journal with a copy to me - it read, "I do not understand why this was at all necessary.  I agree completely with the authors and suggest immediate publication".  And that's what happened.

 

I don't know who the blowhard was who recommended rejection, and it's probably better if I never do.  He or she was clearly another surgeon with an axe to grind but little or no pathologic expertise (and may even have done the wrong operation for this so many times that there was concern about litigation if this was published).  So I have no problem identifying with the righteous indignation that should follow a flawed peer review.  But the process is a well established and long proven approach that produces far more good than bad results.

[bluesman]

9 hours ago, yakman said:

However the process for convention paper are more relaxed compared to journal.

If that's now the case, it's changed since I was a member.  The peer review document used throughout my years as a member (linked in my post above and again HERE) clearly says that the review process for full papers submitted for pre-publication presentation at a conference is "...as a rule, the same as for a direct submission to JAES".  Exceptions are only made for a few unusual meetings, e.g. "new and emerging topics", for which peer review is left to the organizing committee for the conference.

 

Abstracts and precis that are submitted for presentation and subsequently written up as formal papers are also given full peer review before publication.  I don't recall that JAES had any obligatory publications when I was a member.

 

Of course, this could have changed - but I can't find any documentation of change if it did occur, and the standards for scientific journal publication have not been relaxed anywhere else.  They've been tightened.

[bluesman]

5 hours ago, The Computer Audiophile said:

Reviewers give their stamp of approval, which equates to a very powerful message for those who aren’t learned in a subject. Because peer reviewed papers are often used as badges of honor toward consumers, we should know who stamped it.

That's not the way it works, Chris.  As Kal points out, the purpose of peer review is not to validate the conclusions drawn by the authors - it's to validate the  study's premise, design, methodology, clarity, scientific soundness, originality, and educational value to the intended readership(s) and community.  A good reviewer knows the subject and its literature well enough to evaluate each submission for these and other qualities. I've reviewed over 1000 submissions to the top journals in my fields since 1979 (when I finished my residency).  My recommended rejection rate (a stat kept by the best journal editors for each of their reviewers) has run between 75% and 95% over 40 years, which is typical for editorial board members and reviewers at these journals.  But more important is the fact that even though I disagreed with the conclusions of at least half of the authors whose works I reviewed, I recomended their publication because they were well done, well written, and reasonable - and the main editor agreed with me on each.  The existence of multiple well done, peer reviewed studies that seem to contradict each other is a strong indication for more research.

 

A scientific study is only publishable if it adds useful information to the existing body of knowledge, i.e. if it identifies a new or novel finding, if it confirms either a prior study done on a scale too small to justify generaliztion or a study whose reported findings and interpretation go against conventional or curently accepted thought, it raises reasonable doubts about previously accepted findings, etc.  A good scientist is always open to new data and sound interpretation that contradict his or her findings and beliefs.  We can all be wrong and welcome sufficient input to clarify, supplement, or correct what we think we know.

 

Top quality peer reviewed scientific publications are not intended to support consumer or industry activities - they're only intended to ask a question and help answer it.  Any study with the occult but intended purpose of influencing market behavior in a given direction is biased beyond objectivity and is both unethical and grounds for academic discipline.  Every industry works with academic partners to study and develop new things - but every study that's done at any reputable institution must have sufficient scientific merit and importance to merit peer reviewed publication. 

 

Industry does sponsor a lot of reasearch.  But reputable researchers submit their work for publication, and meritorious submissions are published by top journals, regardless of the value of the information to the sponsor.  One of my departmental colleagues and his team were sued by a pharmaceutical company who sponsored a study in which we learned that waiting 12 weeks was as effective in resolving the problem of interest as was taking the sponsor's new drug for 12 weeks. The sponsor actually sued him to prevent publication - and they lost.

 

I've held professorial rank at the two largest and best universities in our region with full complements of educational and clinical programs in healthcare  (Assistant Prof 1979-82, Associate Prof 1982-93, Professor from 1993 to the present).  Several of my colleagues have been disciplined, demoted, and even dismissed for published improprieties.  Unfortunately, we can only control this from the supply side.  Once a study is published and someone gains legal access to it, there's no legal way to prevent its use for profit or other personal aggrandizement short of provable theft if IP, patent violation, etc.

 

We (i.e the academic and scientific community) have historically tried to prevent public dissemination of a lot of basic research for exactly this reason. It was not until recently that you could even subscribe to or otherwise access top peer reviewed medical and scientific journals without a credentialed subscription or through credentialed membership in libraries with institutional subscriptions.  For a lot of reasons (most purely economic - the scientific publishing business has been grossly altered by the internet), you can now buy subscriptions to good journals and even buy individual articles from many.  It's a tug of war for sure - but changing the peer review process offers little or no hope of improving anything I can identify.