Sage Journals: Discover world-class research

Abstract

Objectives:

High-quality research has a tangible impact on patient care and should inform all medical decision-makings. Appraising and benchmarking of research is necessary in evidence-based medicine and allocation of funding. The aim of this review is to demonstrate how evidence may be gathered by quantifying the amount and type of research by a group of surgeons over a 20-year period.

Methods:

Members of the Colorectal Surgical Society of Australia and New Zealand were identified in April 2020. A search of the Scopus database was conducted to quantify each surgeon’s research output from 1999 to 2020. Authorship details such as the Hirsch index and number of papers published were recorded, as were publication-related details.

Results:

226 colorectal surgeons were included for analysis, producing a total of 5053 publications. The most frequent colorectal topics were colorectal cancer (32%, n = 1617 of all publications), followed by pelvic floor disorders (4.3%, n = 217) and inflammatory bowel disease (3.5%, n = 177). 56% (n = 2830) of all publications were case series audits (21%, n = 1061), expert opinion pieces (20%, n = 1011) and cohort studies (15%, n = 758). 7% (n = 354) were randomised control or non-randomised control trials, 3% (n = 152) were systematic reviews and 1% (n = 50) were meta-analyses. The top 10% (n = 23) of authors accounted for more than half (54%, n = 2729) of manuscripts published.

Conclusion:

Australasian colorectal surgeons made a significant contribution to the medical literature over the past 20 years and the number of publications is increasing over time. A greater output of higher-level evidence research is needed. This information may be used to better allocate researcher funding and grants for future projects.

Keywords

Colorectal surgery research evidence-based medicine benchmarking publications

Introduction

High-quality research is necessary to drive developments in surgery, contribute to the field of evidence-based medicine and, ultimately, improve medical decision-making and patient outcomes. Surgical research involves a variety of pre-clinical and clinical experimental studies, the outcomes of which are presented at congresses and published in scientific journals. It is important to assess both the history and current state of scientific publications by surgeons in order to evaluate research prospects, document areas of research need, and identify which surgeons are contributing most to research output. This information is made more important when considering that benchmarking of research output is progressively becoming a necessity in the quest for research funds and grants.¹ Furthermore, these data provide an objective measure of which surgeons to include in expert advisory panels and working groups on national/international guidelines.

While there have been a number of studies looking at the research output of various surgical specialties such as plastic surgery, orthopaedic surgery and surgical oncology,^2–6 no study to date has assessed the research output of a cohort of surgeons as well as individual surgeons. In this study, we quantified the amount and type of research being published by a subset of Australasian (Australia and New Zealand) colorectal surgeons. In addition, we aimed to evaluate which individual surgeons were contributing most to research output, based on both volume of publications and the surgeons’ Hirsch (H) index. The H-index is an objective measure reflecting both the number of publications and number of citations per publication, and is recognised as a simple and effective way to summarise an individual’s scientific research output.⁷

Methods

Literature search

All current members of the CSSANZ were identified in April 2020 using the CSSANZ ‘find a surgeon’ website tool. A retrospective search of the Scopus database for medical peer-reviewed publications was conducted to collate, assess and define each surgeon’s research output. Scopus was the research database chosen as it employs an algorithm using name, location and affiliation to determine authorship and also provides detailed author profiles. Publications were deemed as belonging to the surgeon if the health service recorded on Scopus matched the health service of the surgeon, either current or previous. Only publications over the 20-year period between 1999 and 2020 were included.

Analysis

Data recorded included the total number of publications and citations per author, publication year, type of publication, topic of publication and the surgeons’ H-index. The type of study (systematic reviews, meta-analyses, randomised or non-randomised controlled trials, observational studies, etc.) was recorded, and publications were grouped according to the main topic covered (colorectal cancer, inflammatory bowel disease, diverticular disease, etc.). If a paper related to colorectal disease did not fit a main topic category (e.g. a rare colorectal pathology discussed in a case report), it was categorised as ‘other’. Papers unrelated to colorectal disease (surgical education, perioperative management, etc.) were classified as ‘non-colorectal’.

H-index was used to quantify each surgeon’s individual research contribution. It is an objective mathematical model, defined as the number (h) of publications which are cited at least h times. This allows adjustment for both the quantity and quality of an author’s work, and has been shown to be an accurate measure of an individual’s academic contribution.⁷ All statistical measurements were performed using STATA v15 software.⁸

Results

Of the 251 colorectal surgeons found via the CSSANZ website, 226 were found to have publications in Scopus. These 226 surgeons contributed to a total of 5053 publications over the past 20 years. Of the major colorectal pathologies, the most popular topic was colorectal cancer (32%, n = 1617 of all publications), followed by pelvic floor disorders (4.3%, n = 217) and inflammatory bowel disease (3.5%, n = 177) (Table 1). Other popular topics included endoscopy/polyps (3.3%, n = 167), perianal abscesses and fistulae disease (2.2%, n = 111) and diverticular disease (2%, n = 511). Nearly one-quarter (24%, n = 1213) of all research works over the past 20 years were unable to be categorised into predetermined topics, while 19% (n = 960) of publications by colorectal surgeons were not related to colorectal surgery. There has been a very clear increasing trend in overall publications from 1999 to 2020, with an increase in papers published from just over 100 in 1999 to over 450 in 2020 (Figure 1).

Table 1.

Research topics by percentage of all publications over the 20-year period.

Topic	% of all publications	Number
Non-colorectal	19.4%	980
Cancer
Colorectal cancer	15.9%	803
Rectal cancer	10.3%	520
Colon cancer	5.7%	288
Pelvic floor disorders	4.3%	217
Inflammatory bowel disease	3.5%	177
Endoscopy/colonoscopy and colonic polyps	3.3%	167
Perianal abscess and fistula	2.2%	111
Diverticular disease	2.0%	101
Radiology	1.8%	91
Constipation	1.4%	71
Haemorrhoids	1.2%	61
Anal cancer	1.1%	55
Anal fissures	1.0%	50
Trauma	0.9%	45
Gastrointestinal bleeding	0.9%	45
Endometriosis	0.6%	30
Pilonidal disease	0.4%	20
Anorectal	0.2%	10
Presacral tumours	0.2%	10
Volvulus	0.0%	0
Pruritus ani	0.0%	0
Other	23.8%	1203

Figure 1.

Trend of research output in colorectal surgery over the 20-year period.

More than half (56%, n = 2830) of all publications were from the following three categories: case series audits (21%, n = 1061), expert opinion pieces (20%, n = 1011) and cohort studies (15%, n = 758). 11% involved experimental studies: 7% were randomised control or non-randomised control trials, 3% were systematic reviews and 1% were meta-analyses. 56 papers (1.4%, n = 71) demonstrated novel surgical techniques (Figure 2).

Figure 2.

Distribution of colorectal surgery publications over 20 years by type of study.

The top 10% (n = 23) of authors had ⩾43 publications each and accounted for more than half (54%, n = 2729) of the manuscripts published. There was a significant difference in publication age between the top 10% of authors and the rest of the cohort (Table 2). The top 10% (n = 22)* of authors by H-index (H-index ⩾19) accounted for 47% (n = 2375) of manuscripts published. Six study topics were significantly associated with having greater H-index (p ⩽ 0.05) (Table 3). For example, each meta-analysis published is associated with a 0.69 increase in H-index. For an author publishing 10 meta-analyses, he or she would be expected to have a 6.9 higher H-index than a similar author publishing no meta-analyses. For every 1-year increase in publication age, there was an associated 0.25 increase in H-index. Females had slightly lower H-index than males on average, but this difference was not statistically significant (p = 0.58).

Table 2.

Descriptive statistics of authors in the top 10% of total publications.

Characteristic	Top 10% (n = 23, 10.0%)	All other authors (n = 207, 90.0%)	p-value
Sex			0.05
Male	23 (100%)	178 (86.0%)
Female	0 (0.0%)	29 (14.0%)
H-index	26 (18, 30)	4 (2,9)	<0.0001
Number of documents	103 (65, 153)	9 (3,19)	<0.0001
Total citations	2618 (1213, 3509)	78 (20, 280)	<0.0001
Total number of publications	81 (54, 111)	5 (2,13)	<0.0001
Publication age	28 (19,33)	12 (6,19)	<0.0001
Expert opinion	14 (10,25)	0 (0,2)	<0.0001
Survey questionnaire	4 (1,8)	0 (0,1)	<0.0001
Literature review	6 (2,11)	0 (0,1)	<0.0001
Basic science	5 (1,11)	0 (0,0)	<0.0001
Description of novel surgical technique	1 (0,2)	0 (0,0)	<0.0001
Case report	4 (2,7)	1 (0,2)	<0.0001
Case series audit	14 (8,22)	1 (0,3)	<0.0001
Case–control study	2 (1,3)	0 (0,0)	<0.0001
Cohort study	10 (8,16)	1 (0,2)	<0.0001
RCT or non-RCT	5 (1,9)	0 (0,0)	<0.0001
Meta-analysis	1 (0,3)	0 (0,1)	<0.0001
Systematic review	2 (0,6)	0 (0,1)	<0.0001

RCT: randomised clinical trial.

Table 3.

The impact of certain publications on author H-index.

Parameter	Estimate^a	Lower 95% CI	Upper 95% CI	p-value
Meta-analysis	+0.69	+0.25	+1.12	<0.01
Case–control study	+0.59	−0.01	+1.18	0.05
RCT or clinical trial	+0.56	+0.40	+0.73	<0.0001
Cohort study	+0.32	+0.17	+0.47	<0.0001
Basic science	+0.27	+0.07	+0.47	0.01
Case series audits	+0.25	+0.13	+0.38	<0.001
Publication age (1-year increase)	+0.25	+0.21	+0.30	<0.0001

CI: confidence interval; RCT: randomised clinical trial.

It shows the increase (+) or decrease (−) in H-index for each parameter.

Discussion

What did we find?

Research output by Australasian colorectal surgeons has been increasing over the past 20 years. This finding is reflected in studies worldwide⁹ and locally, such as that by Chua et al., who demonstrated that research within surgical oncology is rising in Australia.⁶ These findings are encouraging given that van Rossum et al. reported that Australia is inferior to some European countries, the United States and the United Kingdom in terms of publications per 10⁶ inhabitants, total number of publications and publication mean impact factor.¹⁰ We also found publication age to be directly associated with the number of publications produced and H-index. Hence, encouraging quality research in the generation of surgeons now would have a positive impact on research output in the future.

Colorectal cancer was the most highly represented research topic. This is not surprising given it is the second most common cancer diagnosed in both men and women and accounts for 9% of all cancer deaths in Australia.¹¹ Such interest is reflected by ongoing research in new therapy options for colorectal cancer, such as immunotherapy and genetic therapy.¹² To date, there has been no review of the research output of colorectal surgeons (or other general surgeons), and so comparing the output of Australasian surgeons to those from other regions is not possible.

Our study has chosen to include all forms of research output by Australasian surgeons, including opinion pieces and case reports. By focusing not only on high-impact factor journals or high-level evidence trials, our study has taken a broad look at colorectal research, providing a comprehensive review of what has been published. This may ultimately help guide directions for future research by gauging which topics need more input from the scientific community.

Despite that fact that Australasian colorectal surgeons have been involved in a substantial amount of publications over the past 20 years, the lack of higher-evidence publications is somewhat concerning and raises questions about the quality of research. A low prevalence of high-level clinical studies in the surgical literature is not unique to the cohort of clinicians analysed in this study and has been reported previously for other groups of surgeons.^13,14 This should, however, be seen as an opportunity for improvement, not only as a stimulus for increasing the number of clinical trials but also the quality of trials. Surgeons should strive to publish trials that procure a clinically relevant information gain¹⁵ and are patient centred.¹⁶ This lack of ‘useful’ research is a problem not facing only Australasian researchers, but is an issue that requires reform and improvement worldwide.¹⁷ Possible reasons for the lack of randomised trials in surgery include the reluctance to randomise while a procedure is being developed, reluctance to randomise when a procedure is already being routinely practised, challenges in standardising techniques of operative procedures among surgeons, difficulties in accruing sufficient patient numbers and the lack of funding in comparison to drug trials that are often supported by pharmaceutical companies.⁶

The top 10% (n = 23) of authors accounted for more than half of the manuscripts published. These authors had a significantly higher publication age when compared with the rest of the cohort (28 vs 12 years) suggesting that colorectal surgeons are better able to publish higher volumes in the latter part of their consultant career.

What are the limitations of this study?

One of the limitations of this study is the use of Scopus as the sole source of finding publications; while quick and largely reliable,¹⁸ this resource may have omitted relevant publications or authors from the analysis. Furthermore, authors from the cohort of colorectal surgeons analysed in our study shared names with researchers in other fields, which had the potential to confuse findings. Hurdles due to homonyms were kept to a minimum because the Scopus identification number was applied for author identification, and all references were accurately checked and any incorrect references were removed.

The use of the H-index is a somewhat flawed tool and has been criticised as a measure of the impact of an individual researcher. For example, the H-index does not take in to account author placement in the authors’ list;^19,20 it may be manipulated through self-citations²¹ and numbers cannot be compared across disciplines because citation habits differ.²² However, a variety of studies have shown that the H-index by and large agrees with other objective and subjective measures of scientific quality in a variety of different disciplines and that the H-index is also effective in discriminating among scientists who will perform well and less well in the future; this makes it a useful indicator of scientific quality that can be used (together with other criteria) to assist in academic appointment processes and to allocate research resources.⁷

What are the directions for future research?

Our study demonstrates a straightforward, rapid and accessible way of quantifying the research output of a group of surgeons over a long period of time. This study utilises freely available data gathered directly from online sources, and is thus free from bias. It is also easily reproducible for any other cohort of surgeons, and may be used as part of a wider study on publishing habits of all Australasian surgeons. Coupled with the use of the H-index as an objective marker of the impact of an individual surgeon’s research output, these data may help to stratify surgeons when allocation of research grants and funding is at stake. For example, a colorectal surgeon with a strong H-index and a track record of quality publications on pelvic floor disorders would be considered first for a major research grant on pelvic floor disorders, despite the fact that they may not be as senior or experienced as some of their other colleagues. Such a transparent process could help to negate biases that may exist with allocating researcher funding. Further work needs to be done to assess how these data may be utilised by policy-makers and those responsible for allocation of research funds.

Conclusion

Australasian colorectal surgeons have made significant contributions to medical literature over the past 20 years. The majority of research output by this cohort of surgeons is directed towards colorectal cancer, followed by inflammatory bowel disease and pelvic floor disorders. There is a lack of higher-level publications such as clinical trials, systematic reviews and meta-analyses, highlighting the need for improvement in this area. The top 10% of authors accounted for more than half of research output; further work should be conducted into how this information may be used to better allocate researcher funding and grants for future projects.

Footnotes

Author contributions

Dr J.R. BMBS contributed to writing of paper and submission. Dr A.L. MBBS contributed to data collection. Dr M.(M.) R. BBiomedSc,BSc (Hons),MD,PGDipSurgAnat contributed to statistical analysis. Dr P.B.S. MBBS contributed to data collection. Mr S.W. contributed to study design and editing of final draft. Professor A.H. contributed to research and writing of paper. Professor N.Z. contributed to review and editing of paper. Dr M.S. contributed to data collection. Dr P.S. MBBS,FRACS contributed to study design and co-ordination.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research,authorship,and/or publication of this article: This study was supported by the Epworth Research Institute Major Research (grant no. 11.952.000.80982). This paper is not based on a communication to a society or meeting.

ORCID iD

Jessica Rahme

References

Adams

Griliches

Measuring science: an exploration. Proc Natl Acad Sci USA 1996; 93: 12663–12670.

Singh

Qadir

Deshmukh

RG.

Research output of a cohort of orthopaedic consultants in Great Britain. Ann R Coll Surg Engl 2003; 85(1): 36–39.

Loonen

MPJ

Hage

Kon

Publications of plastic surgery research 1972 through 2004: a longitudinal trend analysis of three international journals. JPRAS 2007; 60: 934–945.

Croker

Lobo

Croker

, et al. Who, where, what and where to now? A snapshot of publishing patterns in Australian orthopaedic surgery. ANZ J Surg 2017; 87: 1044–1047.

Kawaguchi

Guarise

Silva

Quadros

, et al. Analysis of scientific output by spine surgeons from Japan: January 2000 to December 2013. J Orthop Sci 2016; 21(1): 13–18.

Terence

Chua

PJC

David

, et al. Trends in surgical oncology research in Australia during the period 1998–2009 – a bibliometric review. J Surg Onc 2011; 104: 216–219.

Hirsch

JE.

Does the h index have predictive power?

PNAS 2007; 104: 19193–19198.

StataCorp. Stata statistical software: Release 15. College Station, TX: StataCorp, 2017.

McCulloch

Feinberg

Philippou

, et al. Progress in clinical research in surgery and IDEAL. Lancet 2018; 392(10141): 88–94.

10.

van Rossum

Bosker

Pierik

, et al. Geographic origin of publications in surgical journals. Br J Surg 2007; 94(2): 244–247.

11.

Welfare AIoHa. Cancer in Australia: an overview 2014. [Version updated 16 April 2015]. Volume cancer series No 90. Cat. no. CAN 88. Canberra, ACT, Australia: AIHW, 2014.

12.

Valentini

Mantini

Turriziani

, et al. Research trends in the treatment of colorectal cancer. Rays 2000; 25(3): 393–395.

13.

CYSJ

Sack

Chang

, et al. Reporting randomized, controlled trials: where quality of reporting may be improved. Dis Colon Rectum 2002; 45(4): 443–447.

14.

Chang

Matsen

Simpkins

CE.

Why should surgeons care about clinical research methodology. J Am Coll Surg 2006; 203(6): 827–830.

15.

Evangelou

ESK

Siontis

Pfeiffer

, et al. Perceived information gain from randomized trials correlates with publication in high-impact factor journals. J Clin Epidemiol 2012; 65(12): 1274–1281.

16.

Mullins

Vandigo

Zheng

, et al. Patient-centeredness in the design of clinical trials. Value Health 2014; 17(4): 471–475.

17.

Ioannidis

JPA

. Why most clinical research is not useful. PLoS Med 2016; 13: e1002049.

18.

Meho

Yang

. Impact of data sources on citation counts and rankings of LIS faculty: Web of Science vs. Scopus and Google Scholar. J Am Soc Inform Sci Tech 2007; 58: 2105–2125.

19.

Zhang

CT.

A proposal for calculating weighted citations based on author rank. EMBO Rep 2009; 10(5): 416–417.

20.

Sekercioglu

CH.

Quantifying coauthor contributions. Science 2008; 322: 371.

21.

Bartneck

Kokkelmans

Detecting h-index manipulation through self-citation analysis. Scientometrics 2011; 87(1): 85–98.

22.

Bornmann

Daniel

HD.

What do citation counts measure? A review of studies on citing behavior. J Doc 2008; 64: 45–80.

Review of research output of Australian and New Zealand colorectal surgeons over the past 20 years