Sage Journals: Discover world-class research

Abstract

Setting

The internet is an important source of health information but is unregulated. Little research has focused on the assessment of digital information related to nutrition.

Aim

To develop and validate a novel online quality assessment tool (OQAT) for quality assessment of online nutrition information.

Method

The OQAT was developed and validated in six distinct stages. After reviewing the literature, a framework and criteria were developed and formalised. Next, the quality assessment criteria were piloted on a subset of data and criteria refined. The established criteria were then validated against a previously validated assessment tool, and reliability was tested. Finally, the validated OQAT was used to assess the quality of articles from a 24-h collection period, 19 April 2021.

Results

The final OQAT consisted of 10 key questions. Twenty-six news articles were assessed independently by two raters. Comparison of scores found moderate internal consistency (α = 0.382). Cohen's Kappa coefficient demonstrated high interrater agreement (k = 0.653, p < 0.001). The OQAT was tested on 291 relevant Uniform Resource Locators (URLs), which were determined to be either poor 3% (n = 9), satisfactory 49% (n = 144), or high-quality 48% (n = 139) articles. There was a statistically significant difference in OQAT scores between blogs, news articles, and press releases, χ²(2) = 23.22, p < 0.001, with a mean rank OQAT score of 138.2 for blogs, 216.6 for news articles, and 188.7 for press releases.

Conclusion

This novel tool provides a reliable and objective method for assessing the quality of nutrition content online. It could potentially be used by researchers to assess the quality of online information in different settings and by organisations to inform readers of the quality of information being accessed.

Keywords

Nutrition quality assessment quality assessment tool digital health online information validation

Introduction

The internet, including social media, is one of the most important sources of nutrition information for the general population.¹ However, it is largely unregulated.² The quality of online nutrition information is important as misinformation leads consumers to believe that the scientific evidence is contradictory³ and negatively influences consumer beliefs.⁴ Additionally, contradictory information increases uncertainty about health research more generally. Furthermore, the perception of conflicting health reports in the media can increase negative attitudes towards health policy and reduce adherence to guidelines.⁵ Nutrition guidelines are a key component of the wider health agenda with the World Health Organization making recommendations on diet quality aimed at individuals and the wider food environment. However, there is a rise in self-proclaimed experts online that share misleading information about food and nutrition that can shape perceptions and influence diet.⁶ People are therefore regularly exposed to conflicting nutrition information from television, online news, and social media⁷ that undermines these recommendations. Prolonged exposure to inconsistent nutrition information over a period of time has been shown to have detrimental effects on consumer beliefs and healthy dietary intentions.⁸ Inconsistent dietary information may also reduce engagement with recommended nutrition behaviours such as fruit and vegetable consumption.⁹

Previous research has assessed the quality of different types of health information but rarely online health information and even less frequently nutrition information online. For example, patient-facing treatment information,¹⁰ health news in print newspapers,¹¹ and patient-orientated websites¹² have all been evaluated systematically. A tool developed and applied to health information in newspapers by Robinson et al.¹¹ has subsequently been used to assess the quality of print newspapers, including nutrition information,¹³ Brexit-related food issues in the UK,¹⁴ and media reporting on air pollution-related health outcomes.¹⁵ In addition, alternative quality assessment criteria have been used to assess media coverage of childhood obesity in UAE newspapers¹⁶ and the veracity of information shared by social media influencers on Instagram.¹⁷ Blogs, including recipes, have also been considered, including homemade infant milk,¹⁸ healthy living blogs,¹⁹ and the nutrient profile of ‘clean eating’ blogs.^20,21 In response to the increase of blogs as a public-facing source of nutrition information, the construct of blogs written by Dietitians has been considered,²² and content analysis of Dietitian- versus non-Dietitian-authored blogs,²³ but neither study extended to other types of online content. Studies assessing the quality of online nutrition information have used either newly developed, unvalidated assessment criteria or criteria designed to assess patient-facing health information, which do not transfer effectively to nutrition information and make quality comparison across studies challenging. The DISCERN tool¹⁰ has been widely used to assess the quality of nutrition-related information online. However, while it is validated with high interrater reliability, it is designed to be patient facing and to assess clinical information such as treatment options, risks/benefits of treatment, and quality of life. These assessment criteria are not suitable for public health nutrition information in a non-clinical setting.

Existing reviews have evaluated existing quality assessment tools for assessing the quality of general online health information, as opposed to nutrition specifically,²⁴ and educational blogs aimed at healthcare professionals.²⁵ Paterson et al.²⁵ identified 151 quality indicators in the literature, noting that most scoring systems lacked evidence of validity. Consistent with prior reviews,²⁶ Zhang et al.²⁴ concluded that quality was defined and measured differently with different studies using different criteria. Both reviews found that the quality of information varied across websites and concluded that overall quality was problematic. To our knowledge, the existing quality assessment tools currently used to assess the quality of online nutrition information are based on medical guidelines, and although these instruments were designed as a guide for consumers to evaluate health information quality, the number of criteria included makes them impractical to use and questions have been raised about their validity.²⁴ A more recent systematic review of the suitability of existing criteria and instruments used in evaluating health information on social media highlighted the need for future research to identify specific assessment tools and quality evaluation criteria for information shared on social media.²⁷

Disseminating news through social media has become an integral part of online news distribution and consumption, with users contributing as both content creators and content distributors.²⁸ However, social media has been criticised for having ‘disturbed media power structures’, that is, a structure that enables reduced influence of professional media and allows public actors to play a greater role in shaping debate.²⁹ Online communication can amplify political misinformation³⁰ and encourage unconstructive discussion.³¹ Therefore, in the context of widespread sharing of misinformation and disinformation, it is important to understand the quality of the information that has the potential to be widely shared.³²

Among social media and microblogging platforms, Twitter is a popular social network for discussing news and nutrition-related information globally.³³ One crucial function of Twitter is as a platform for information sharing, including Uniform Resource Locators (URLs)³⁴ to external online content. The act of sharing content on Twitter is active and demonstrates engagement with the content as research shows that not all online content generates active participation.³⁵ On Twitter, information sharing is considered either ‘first-degree sharing’, that is, generating original content and/or posting from an external source, or ‘second-degree sharing’, that is, retweeting a tweet.³⁶ Mixed-methods frameworks have been used to carry out qualitative analysis in other areas, such as on climate change commentary on Twitter,³⁴ and have used content analysis to assess the emotion of tweets.^36,37 While these frameworks considered thematic analysis of the narrative, they did not look at the quality of the information shared.

In summary, there is a lack of suitable standardised quality assessment criteria to assess the quality of online nutrition information.^25,27 Where quality assessment tools have been developed for online nutrition-related information, they have been more specifically focused towards clinical information¹⁰ or have assumed use by informed readers with existing knowledge.³⁸ In addition, source credibility is a key consideration online as this can impact the virality of content.³⁹ Therefore, assessment of the media source type, as well as the content type, is essential to understand where high- and low-quality information is being published online.

Aims

Based on the diverse and often unvalidated quality assessment criteria used in the literature and the lack of a universally accepted tool for assessing online nutrition information, this study aimed to:

develop a novel tool for objective assessment of the quality of online nutrition information;

validate the novel assessment tool;

assess the novel assessment for interrater reliability and face validity;

pilot test to assess the quality of a sample of online nutrition information published on Twitter and assess the relationship between the source and the quality.

Methods

The online quality assessment tool (OQAT) was developed and validated in six distinct stages and then used to assess the quality of articles from one 24-h period. First, a literature search was carried out searching for validated tools designed to assess the quality of online information. Second, a framework and criteria were developed based on the literature mapping the quality assessment criteria on the framework. Third, the criteria were discussed and agreed within the research team. Fourth, the quality assessment criteria were piloted on a subset of data and the wording of the criteria was refined and criteria were removed if deemed to be duplication. In the fifth stage, the established criteria were validated against an existing print media assessment tool, and reliability was tested. Finally, upon completion of the validation and reliability testing, the validated OQAT was used to assess the quality of articles from a 24-h collection period, 19 April 2021.

Development of a novel OQAT

A literature search was conducted to identify articles using tools to assess the quality of online and print nutrition and health information on the Web of Science, PubMed, and the Association for Computing Machinery (ACM) Digital Library in 2020 and updated in 2021. The literature search focused on validated and un-validated tools assessing quality of nutrition information or health information more generally, online and in print news. Papers were excluded if they were not health or nutrition related, they assessed nutrition or health literacy, or they assessed videos, images, or audios such as podcasts.

To develop section one of the OQAT, a framework of quality evaluation criteria, and corresponding indicators, critical to the assessment of online information was constructed, informed by previously evaluated quality assessment tools.²⁴ The framework was based on two key research papers from the literature: the validated Robinson tool,^11,13 which the authors were already familiar with and had previously been used to assess nutrition information in print news, and a systematic review,²⁴ which categorised the criteria and indicators used in 165 research studies to assess the quality of online health information for consumers. The Robinson tool was selected as it has been used widely to assess nutrition-specific information in newspapers¹³ and includes objective questions that do not assume the rater has a large amount of prior knowledge of nutrition. The tool also assesses public-facing information rather than information for clinical patients, or practitioners, which differs from other tools, and places emphasis on evidence-based reporting. These were deemed to be important criteria for creating an informative novel tool to assess online nutrition information. The systematic review was selected as it is a comprehensive review of existing validated and unvalidated assessment criteria used to assess health information online.

Five of the 21 questions on the Robinson tool were selected as they were relevant for online information (full details are documented in Supplemental Material S1). Three of which were not explicitly stated by Zhang et al. Misleading news and headlines and causal inference⁴⁰ can have detrimental effects on public health, and therefore, the authors felt it was important to include these questions from the Robinson tool. Two major categories were identified by Zhang et al., content and design. This study only considers content; therefore, all criteria categorised as design were discounted at the first stage. Content-related measures were further classified into five criteria with 28 corresponding indications. The criteria and indicators were selected for the OQAT based on relevance to an individual article as opposed to the wider site, and those being represented in at least 50% of the articles reviewed to ensure robust indicators were selected. Initially, 13 indicators were selected.

Once the framework, criteria, and indicators were defined, the quality indicators were initially piloted on 20 randomly selected URLs from the 24-h collection period, 19 April 2021, and assessed by two trained raters. These represented news articles and blogs. Both raters had formal nutrition education. The criteria were refined based on discussion within the team on the relevance of the criteria and the existence of a high correlation between indicators. The resulting 10 quality assessment indicators were grouped into the three relevant categories as per Zhang et al.: currency, credibility, and reliability, as described below.

Currency refers to whether the content is up to date. The main indicators include the publication date and when the article was last updated.

Credibility criteria consider authoritativeness and trustworthiness. Authoritativeness refers to whether the content was contributed by creditable sources and cites credible sources. Trustworthiness is whether a source is truthful or biased. The credibility indicators, as identified by Zhang et al.,²⁴ overlap with the technical criteria identified by Eysenbach et al.²⁶

Reliability refers to whether the content of a webpage is understandable for general consumers without a nutrition or science background. It does not consider readability and accessibility of the whole website.

Given the infinite nature of online content, the inclusion of currency was deemed necessary to evaluate whether the article includes up-to-date scientific evidence and policy information. Positive responses to all 10 indicators were considered essential for a high-quality source of nutrition information. Articles were scored positively if they met the criteria and zero if they did not. There were no negative scores. From a minimum of zero, the maximum score achievable was 10. A full breakdown of the marking criteria and instructions can be found in Supplemental Material S2.

After scoring, articles were categorised as poor, satisfactory, and high quality based on the quality score: 0–2 indicated poor quality, 3–6 indicated satisfactory quality, and 7–10 indicated high quality. The three cut-offs were selected based on Rasch analysis (see ‘Statistical analysis’ section), which identified the minimum requirements for each category.

To enable the content analysis and comparison of content type, section two of the OQAT was developed to capture the type of information shared (i.e. news article, blog, press release, video, social media, and promotional) and the original source. This differs from other research evaluating social media content, which has focused on the social media user⁴¹ and network analysis.⁴² In the context of validating the OQAT, the source and content type were determined by manually evaluating the webpage, with two trained raters, both with formal nutrition training, independently reviewing a subset of URLs and meeting to discuss discrepancies. Content was categorised as per section two of the OQAT, with each article reviewed assigned to a category for media source type, that is, professional blog, news article, or non-governmental organisation (NGO), and a category for content type, that is, blog, news, advertising, or video (Supplemental Material S1).

Expert panel evaluation

Face validity indicates whether the criteria measure what the developers intended them to measure.⁴³ This was assessed by two independent experts as per the literature.^43–45 The panel was selected based on their publication record in the area of online information quality, the quality of media reporting, or the role of social media in information literacy. The panel reviewed the assessment criteria and related instructions and provided comments on the clarity and content.

Validation of the novel OQAT

The novel OQAT was validated against an existing tool developed to assess quality of health information in the print idea.¹¹ It was not possible to validate with a high-quality existing tool to assess online nutrition information as this does not currently exist. Therefore, only news articles were used for validation against the validated tool developed to assess UK print news using URLs shared on two randomly selected dates in 2021, 19 April and 12 June. Two trained researchers independently assessed all nutrition-related news articles excluding information not categorised as news (such as blogs). Any significant discrepancies were discussed, and consensus was reached.

Inter-rater reliability

Inter-rater reliability was carried out to ensure the measure was independent of the raters and could therefore be repeated with different raters. To test reliability, two trained raters, both with formal nutrition training, used the OQAT independently to score a randomly selected subset of the URLs shared on 19 April 2021. A minimum of 50 observations is recommended for reliability testing.⁴⁶ Therefore, this was the minimum number included by authors. This included a random subset of blogs (due to the large number), all news articles, and all press releases. Any significant discrepancies between the two raters were identified, the articles were discussed, and consensus was reached.

Pilot data collection

Twitter was used to collect URLs³⁴ for analysis. Tweet Archiver⁴⁷ was used to automatically webscrape Twitter for posts containing the term ‘nutrition’ or #nutrition during a randomly selected 24-h period in 2021, 19 April 2021, using Random.org. The data were then screened, and those without a URL were discounted. The URLs were then screened for eligibility and relevance, discounting advertising, recipes, original research papers, and articles, which did not relate to human health.

Two trained raters, both with formal nutrition training, used the tool independently to score the relevant URLs identified during the 24-h period. Any discrepancies were discussed, and consensus was reached. Articles were excluded if topically irrelevant, were linked to social media, or were advertising or product promotion. Articles on climate change, animal nutrition, food, and agricultural policy were discounted if they did not directly relate to nutrition and human health. In addition, URLs were discounted if they were part of discussion forums or were in video format.

Statistical analysis

The Statistical Package for the Social Sciences (SPSS) version 28.0 was used for statistical analysis. The internal consistency of the OQAT quality score was calculated using Cronbach’s alpha. This indicates the degree to which items measuring the same general construct produce similar scores. Validity of quality scores for news articles was determined by the intraclass correlation coefficient. This statistic allows for the calculation of the agreement between the OQAT and the Robinson tool.

As the data were ordinal, a weighted Kappa coefficient⁴⁸ was used to measure the agreement between the two raters. Kappa was chosen to assess the interrater agreement rather than correlation coefficients as correlation coefficients only assess covariation in data and do not reflect agreement on the actual scores. Cohen's Kappa can range from −1 to +1, where 0 represents the amount of agreement that can be expected from random chance, and 1 represents perfect agreement between the raters. Scores > 0.81 can be considered a near-perfect agreement.

For the quality analysis, the Shapiro–Wilk's test was used to test for normal distribution, and a p-value of <0.01 was taken for significance. The non-parametric Kruskal–Wallis H test was used to compare the OQAT scores of blogs, news articles, and press releases due to the non-normality of the data. Rasch analysis was used for the estimation of cut-off levels that represented distinct levels of the quality of information, using Winsteps.⁴⁹ The dichotomous Rasch model was fitted to two sets of data, one for Rater 1 and another for Rater 2, using the Joint Maximum Likelihood Estimation (JMLE) method, and model assumptions were checked for unidimensionality using the principal component analysis of Rasch residuals (PCAR), and other fit statistics were used to evaluate item fit.⁵⁰ Cut-off points were estimated using a score-to-measure table that compared the relationship between each raw score, from 0 to 10, with Rasch model estimates of the location (JMLE measures) and their standard errors (SE) according to the procedure suggested by Wright.⁵¹

Results

Development of the OQAT

Based on a review of quality assessment tools in the literature designed to assess online health information and print news, the following elements were defined for section one:

Three criteria: rules by which the quality of information is judged. Criteria reflect the values held by the evaluator regarding what is important for determining the quality of nutrition information.

Ten indicators: observable attributes associated with the webpage content, which indicate whether the webpage content meets a given criterion.

Initially, 14 indicators were identified. However, during the development phase, the criteria were refined based on discussion within the team on the relevance of the criteria and the existence of a high correlation between indicators. The highly correlated indicators, such as publication date, date of last review, and date of next review, were removed, as was citing an author and citing a journal, as this could be determined by asking if the article contained links to high-quality sources.

The final version of section one of the novel OQAT consisted of 10 key quality indicators adapted from Robinson et al.¹¹ and additional sources. To ensure suitability for online use, the three criteria were based on the quality assessment from Zhang et al.²⁴ (Table 1). All indicators were scored positively, and an article could score between the values of 0 and 10.

Table 1.

Quality assessment criteria for the novel online quality assessment tool (OQAT). Section one: quality assessment.

Criteria	Indicators
Currency (whether the content is up to date)	Publication date or date of last update
Credibility (authoritativeness and trustworthiness)	Authorship – author nameAuthorship – credentialsAttribution – high-quality peer-reviewed referencesAttribution – quote a specialistDisclosure – financial or professional disclosures, bias disclosure
Reliability	Adequate and accurate backgroundHeadline – true reflection of the article and evidenceDoes not generalise – from animal or lab studiesDoes not have the potential to cause undue harm or optimism

Section two of the OQAT is a content analysis codebook (Table 2). URLs were manually reviewed and categorised by media source type and content type to allow for quality comparison by content type. Instructions on categorising URLs are available in Supplemental Material S1.

Table 2.

Content analysis codebook. Section two: media source type and content type.

Media source type

1. Blog – personal

2. Blog – professional

3. Company (products and services)

4. Government organisation (e.g. PHE, FDA)

5. Magazine

6. Non-governmental organisation (NGO)

7. Professional news (e.g. CNN, The Guardian, The Huffington Post, BBC)

8. Research institute/university

9. Social media (e.g. YouTube, Instagram, etc.)

10. Unrelated

Content type

1. News article

2. Blog

3. Scientific report – out of scope for analysis

4. Press release

5. Video – out of scope for analysis

6. Picture – out of scope for analysis

7. Social media (e.g. Twitter/Facebook status) – out of scope for analysis

8. Promotional – out of scope for analysis

As part of the OQAT development, the criteria were sent to two independent experts for review. Suggestions were made that the OQAT could be improved by scoring poorly if the article is more than 5 years old. The authors agree with this suggestion given the infinite nature of online content and initially included the ‘date of last update’ as a quality indicator; however, this was highly correlated with publication date and therefore was removed. The inclusion of a publication date allows readers to make an informed decision on the relevance of the evidence. However, highlighting that an article is out of date or updating the content regularly is made as a recommendation in the discussion. Additionally, the reviewers suggested that it would be valuable to better identify articles with multiple links to multiple content, both high quality and low quality. Again, this is discussed in the discussion as part of the recommendations.

Validation of the OQAT

Following comments from the expert panel, the instructions were refined for clarity and to avoid ambiguity. The panel did not suggest modifying the criteria or indicators; therefore, these were accepted as having face validity. The OQAT was validated against an existing validated tool designed to measure the quality of health information in UK print newspapers¹¹ using data scraped from Twitter on 19 April 2021 and 12 June 2021 (due to the limited number of news articles shared on 19 April 2021). Over this randomly selected 24-h period, 2894 tweets were collected from Twitter posts that contained the word ‘nutrition’ or #nutrition, and 1007 posts included a URL. Each URL was reviewed manually (Figure 1) and categorised as per the OQAT codebook (Table 2).

Figure 1.

Flow diagram of identification and screening of tweets from 19 April 2021 for analysis to test the novel tool for assessing the quality of online nutrition information.

To validate the OQAT, only news articles were assessed as this type of information is what the previously validated tool was designed to assess. A total of n = 26 news articles were assessed by two trained raters independently using both the OQAT and the existing Robinson tool.¹¹ Raters met to discuss and agree on any discrepancies. Comparison of scores (Table 3) using Cronbach's alpha found moderate internal consistency (α = 0.502).

Table 3.

Tabulation of scores on the Robinson tool and the online quality assessment tool (OQAT) by tertile for news articles.

		OQAT			Total
		Poor	Average	High	Total
Robinson tool	Poor	0	2	3	5
	Average	0	6	9	15
	High	0	0	6	6
Total		0	8	18	26

Reliability testing

After reviewing manually, 291 posts with URLs remained. These represented a total of 260 blogs, 26 news, and 5 press releases. News articles and press releases were oversampled, and 100% of these sources were included due to the small numbers in these categories. Out of the 260 blogs, 100 blogs were randomly selected using Excel’s random generator. Six blogs were excluded as the URLs were broken at the time of review. Therefore, in the final sample to assess interrater reliability, 94 blogs, 26 news articles, and 5 press releases shared on 19 April 2021 (total n = 125) were assessed. Discrepancies were identified and discussed, and consensus was reached (Table 4). The weighted Kappa coefficient demonstrated high interrater agreement (k = 0.653, p < 0.001, 95% CI 0.524–0.782).

Table 4.

Interrater reliability by tertile for all data assessed.

Before		Rater 2			Total
Before		Poor	Average	High	Total
Rater 1	Poor	2	1	1	4
	Average	1	45	10	56
	High	0	10	55	65
Total		3	56	66	125

Fit

The Rasch analysis of the data from Rater 1 indicated that 9 out of the 10 items complied with the recommended OUTFIT mean squares between 0.5 and 1.5 for being ‘productive for measurement’ (Table 5). Item 9 had a very low OUTFIT mean square below 0.5 and is ‘less productive for measurement but not degrading’, that is, it does not affect the general fit of the items to the Rasch model. There was a slight difference in the item fits for Rater 2, with most of the same items fitted in the same range, except for item 6 that was in the range of 1.5–2.0 and considered to be ‘unproductive for measurement but not degrading’. All items within these three OUTFIT ranges were acceptable, and no further attempt was made to improve the fit of items. The Rasch model assumption of a single measure that represents a single dimension was confirmed with separate PCAR. The data from Rater 1 had an unexplained variance in the first contrast of 1.83, and the data from Rater 2 had a value of 1.93, both of which was smaller than 2.0 that has been used to indicate likelihood of whether the Rasch measure was unidimensional.⁴⁹ An additional check of unidimensionality examined the values for disattenuated correlation from the comparisons between sets of items that were classified after PCAR. All values were the recommended value of 0.87, indicating that the measure was an adequate measurement of a single latent variable.⁵⁰

Table 5.

Item fit statistics for the dichotomous Rasch model.

Item^a	JMLE measures ± SE^b		OUTFIT mean square
	Rater 1	Rater 2	Rater 1	Rater 2
Disclosure – 6. Does the article disclose any financial or professional conflict?	8.33 ± 0.28	8.35 ± 0.24	1.24	1.94
Specialist – 5. Does the article quote a specialist?	7.44 ± 0.23	7.61 ± 0.21	1.29	1.34
Reference – 4. Does the article include references to high-quality peer-reviewed resources that can be accessed in one click?	6.73 ± 0.22	6.82 ± 0.20	0.99	0.78
Credentials – 3. Does the article state the authors’ credentials or provide access to a biography?	5.74 ± 0.22	6.74 ± 0.20	0.68	0.71
Result – 10. The article does NOT have the potential to cause undue harm or optimism.	5.07 ± 0.23	5.02 ± 0.23	0.79	0.86
Author – 2. Does the article state the authors’ name?	4.33 ± 0.25	5.24 ± 0.22	0.79	0.78
Accurate – 7. Does the article provide adequate and accurate background?	3.75 ± 0.29	4.85 ± 0.24	0.92	0.74
Representative – 8. Is the headline a true reflection of the article and evidence?	3.67 ± 0.29	4.21 ± 0.28	1.14	1.44
Date – 1. Does the article state the publication date or date of last update?	3.58 ± 0.30	4.12 ± 0.29	0.78	0.79
Generalise – 9. The article does NOT make generalisations from animal or lab studies?	1.4 ± 0.60	0.97 ± 0.98	0.16	0.26

Question items (indicators) have been ordered based on their order of fit. The estimates for items 10 and 2 were reversed for Rater 2 compared to Rater 1.

The Joint Maximum Likelihood Estimation (JMLE) estimates have been rescored from their logit values to a measure range of 0–10.

Estimation of cut-off points

The Wright (person-to-item) Map allows the representation of the questions (ITEMS) and information sources (PERSON) on the single latent Rasch measure. Three statistically independent levels were found for both sets of data independently with the rescaled Rasch cut-off points shown in Supplemental Material as lines in Figures 1 and 2 (Supplemental Material S3). The corresponding raw scores for both sets of data were estimated independently and were low (0–2), medium (3–6), and high (7–10).

Testing the OQAT for quality assessment of online information

The data scraped from Twitter on 19 April 2021 (n = 2894) were reviewed for eligibility (Figure 1) and manually categorised as per the OQAT codebook (Table 2). Articles (n = 361) not related to nutrition and human health were excluded. A total of n = 291 articles were analysed, and these represented 260 blogs, 5 press releases, and 26 news articles.

The OQAT was pilot tested on the relevant URLs (n = 291). The scores generated were not normally distributed, as assessed by the Shapiro–Wilk test (p < 0.001). In total, 3% (n = 9) of articles were categorised as poor, 49% (n = 144) as satisfactory, and 48% (n = 139) as high quality (Figure 2).

Figure 2.

Percentage of articles in each category as assessed by the novel online quality assessment tool (OQAT).

A Kruskal–Wallis H test showed that there was a significant difference in OQAT scores between blogs, news, and press releases, χ²(2) = 23.22, p < 0.001, with a mean rank OQAT score of 138.2 for blogs, 216.6 for news articles, and 188.7 for press releases.

To allow for analysis by criteria, mean scores were calculated for each quality assessment criterion to allow for comparison by criteria and article type. Scores were categorised as positive if they were >0.66 or negative <0.33. Blogs were least likely to state author credentials, quote a specialist, or disclose any financial bias. Across all article types, a specialist quote was least likely to be included as well as disclosure on financial conflict or bias (Figure 3).

Figure 3.

Mean online quality assessment tool (OQAT) score per criterion by content type.

Discussion

Successful development, reliability testing, and validation of this novel OQAT have addressed a gap in the existing literature by providing a validated quality assessment tool suitable for assessing online nutrition information. It is vital that a means of assessing the quality and credibility of online nutrition information is developed due to the lack of current regulation on the internet, the increasingly common access to poor-quality nutrition information, and the growing dependence on online sources for nutrition information. A quality assessment tool is needed to assess changes in the quality of information shared on social media over time if steps are taken in the future to intervene and restrict the spread of nutrition-related misinformation online.

Currently, no such mechanism exists for evaluating nutrition-related online information, although several studies have attempted to assess the quality of a variety of different types of online information. These have included assessment of specific dietary advice for cancer patients,¹² vegan diets,¹⁷ and impact of social media influencers.³⁵ However, many of these assessment criteria remain unstandardised and/or unvalidated.^24,25 In addition to the lack of standardisation, most of the tools discussed in the literature were designed and tested before the vast expansion of social media usage and therefore are not well suited to assess the quality of information shared in this way.²⁷ Importantly, the initial testing of our novel OQAT instrument, which compared the quality of nutrition information from different sources, demonstrates that our new tool is sufficiently sensitive to detect differences in quality of online information.

An important characteristic of online content is the ability to update the content at any time and its infinite duration. The OQAT was designed to better assess the unique characteristics and wider range of online, freely available public-facing content (which is easy to access, can be written by anyone at any time, and is often discursive yet informative). This differs from other tools designed to assess print news^11,13 or patient-facing information.¹⁰ The OQAT captures this key currency information, which is necessarily different to other available tools. The OQAT is also designed to capture the unique online capability of utilising hyperlinks to external sources, such as research journals and author biographies, both of which are key to determining the credibility of the online article and have been shown to impact spread of information.³⁹

Validation was found to have moderate internal consistency but perhaps would not be expected to be higher given that online news is provided in a different format than paper-based editions. The validation results show that the OQAT is more suitable for online content than current tools but perhaps would not be suitable for nutrition-related news in print newspapers. It is more inclusive, agile, and more suitable for assessing content that is not based on a traditional press release. Nevertheless, high interrater reliability was reported providing evidence that the assessment tool was robust. The high interrater reliability suggests that the OQAT met the objective to be an objective tool that could be used by trained raters. Face validity was essential to judge the understanding of the criteria and the associated instructions. The expert panel did not recommend any major changes to the criteria; instead, they suggested improving the wording of questions to avoid ambiguity, as seen in other studies in order to improve reliability.^43–45

Our data show, at least on the day sampled, that blogs were the most prevalent nutrition-related content type shared via Twitter. This has also been found in the context of obesity-related content⁵² but not in the context of online content related to climate change, where news was the most shared³⁶ and professional media outlets were the most prolific actors.⁵³ This indicates that nutrition articles are different from other topics in the news such as climate change⁵³ and politics,^30,54 which may therefore require a different approach towards the assessment of quality.

An initial assessment of the quality of information indicated that blogs were not only the most prevalent but also the lowest-quality article type. This supports previous work in the literature,¹² where information in blogs has been measured against dietary advice and found to score poorly on providing scientific evidence and including expert opinion.^12,17,22 The lack of evidence-based information in blogs found in our study was consistent with the literature pertaining to print news,¹³ obesity,⁵² anti-climate change blogs,²⁹ and public-authored political blogs.³¹ All of which identified the damage poor-quality non-expert written blogs can have on public debate. Seeking expert opinion, a sign that the writer was aware of the importance of peer review, was also lacking in many sources, consistent with the published literature.¹¹ More encouragingly, the vast majority of articles scored positively on listing an author, an assessment criterion that has previously been shown to positively affect article quality.^11,13

The main strength of the tool is that it provides a set of standardised assessment criteria, as called for by Afful-Dadzie et al.,²⁷ to assess the quality of online content. The quality assessment criteria could expand the OQAT relevance beyond researchers as it could be employed as a checklist by content writers or as a framework for consumers to assess the quality of online nutrition information, providing a motivation for publishing higher-quality information. Similarly, the OQAT may later be suitable for other evidence-based online articles such as more general preventative health information, pending further research.

Further strengths of this study include the OQAT development, which was based on previously validated criteria¹¹ and methodology^1,25 developed and made relevant for online content. However, validation against a tool that was used to inform the development of the OQAT is a limitation. However, there is no gold standard tool to validate against, and there is a lack of validated tools in the literature; therefore, validating the tool using the Robinson tool was deemed the most appropriate method. Data collection was novel in that it used Twitter as the source of URLs, enabling objective selection of a cross-section of content designed to disseminate nutrition information. As Twitter has over 200 million active users, using Twitter ensures that the URLs being assessed have been interacted with. This is preferable to a Google search, which may return content that does not stimulate reader engagement.³⁵ By creating a tool that can be used for all nutrition-related online content, the OQAT also builds on recent studies that have previously categorised the positive characteristics of Dietitian-authored blogs²² and compared the quality of the blogs to those from lay authors.⁵⁵

This study had several limitations. While a thorough literature review was conducted to identify online and print quality assessment tools and extended to wider health information, some tools may have been missed as a systematic review was not conducted as part of this research. The uniqueness of the OQAT created challenges during the validation process. As the previously validated tool was designed to assess nutrition-related news,¹¹ it was necessary to validate the OQAT using only news articles. However, this type of information is not commonly shared on Twitter. This further supports the need for a quality assessment tool that can assess diverse types of online content, as articles categorised as ‘blogs’ are shared more frequently than ‘news’ on Twitter. Similarly, the disproportionately high number of blogs, while representing the type of content being shared, did not allow for a comprehensive comparison of the quality of all content types. A further limitation of the quality assessment methods is that the raters were not blind to the article source. One possible effect of this may have been to moderate the article score if the source was trusted, or not trusted, by the rater. However, the questions were worded as clearly as possible to reduce the risk of bias. A further limitation is that the indicators were not weighted. Rasch analysis indicated that the unweighted items were broadly adequate to assess quality and meet the objective of classifying articles into three quality levels without the need for weighting. However, this is the first iteration of the OQAT and future refined and improved versions may consider weighting.

A limitation of the study, and the OQAT more generally, is that only webpages are considered. Therefore, the wider limitations of a website are not considered, for example, if references cannot be accepted or if author credentials cannot be prominently displayed.

This is because the OQAT was designed to measure online articles as they relate to evidence-based nutrition and not the usability or accessibility of websites, which could include other information such as events or advertising, which were out of scope. Similarly, the OQAT does not include readability scores as these can be assessed by external software such as the Flesch–Kincaid readability test. The OQAT was validated using articles identified with the word and hashtag nutrition and written in English. Relevant information could have been missed if a tweet used alternative descriptive words such as ‘diet’ or ‘healthy lifestyle’. Future research should consider broader search terms. Finally, apart from publication date, the OQAT does not consider how up to date the article is as this is challenging to determine in nutrition as some research and guidelines are relevant 30 years later. Future versions of the OQAT should consider how to reliably deal with this.

Importantly however, the successful development and validation of the OQAT have led to a number of recommendations for practice. Online content and blogs, in particular, are a popular source of nutrition information for the public,^17,22 but they vary widely in quality.^18,19,52 Based on the findings from the OQAT development and validation and the wider literature, a series of recommendations to content writers are suggested. Online content that gives dietary advice must be evidence based and provide the evidence to the reader through references or hyperlinks. References and hyperlinks should link to scientific evidence rather than circular links within the website – it is best practice to include an identifying feature for scientifically validated weblinks.

Given the infinite lifespan of online content, articles should be reviewed and updated regularly (annually as a minimum) and include a warning or caveat if the content is more than 5 years old or be removed from the website, so the reader is informed on how up to date the evidence is and not unknowingly exposed to out-of-date nutrition information. Blog authors need to give a brief, referenced summary of the evidence ensuring the most up-to-date evidence is stated to ensure the reader has a comprehensive background of the topic. Additionally, authors should not overstate the evidence. Notably, many blogs reviewed by the OQAT scored poorly as they suggested health can be improved by regularly eating one nutrient or food type over a short period of time, or similar overstatements leading to increased risk of causing undue harm or optimism. Finally, any funding should be explicitly stated so that a reader is informed whether the author has been paid to write about a certain food or topic. Further research is needed to determine whether nutrition information is more likely to be shared if it is of lower quality.

Conclusions

The development and validation of this novel OQAT add to a body of literature assessing quality of information in the media and online. This study contributes to the methodology of assessing the quality of online information. It has further developed existing tools and guidelines to create a tool that is designed to be simple to use and, with further testing, could be used by non-nutritionists to measure the distinct characteristics of online information. This tool is a reliable and objective method that can be used in future research and practice, either by researchers to assess the quality of online information in different settings or by organisations to inform readers of the quality of information being accessed. While this tool was validated using nutrition information, it may also be suitable for other evidence-based online articles such as more general health information.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076231187249 - Supplemental material for Development and validation of a quality assessment tool to assess online nutrition information

Supplemental material, sj-docx-1-dhj-10.1177_20552076231187249 for Development and validation of a quality assessment tool to assess online nutrition information by Cassandra H Ellis, J Bernadette Moore, Peter Ho and Charlotte EL Evans in DIGITAL HEALTH

Footnotes

Acknowledgements

The authors would like to thank Dr Alice Kininmonth and Ms Christina Sabbagh for their advice and guidance on the first draft of this online quality assessment tool.

Contributorship

CHE researched the literature,conceived the study,and wrote the first draft of the manuscript. CELE was involved in the study design,protocol development,and data analysis. PH advised on,and contributed to,the statistical analysis. JBM was involved in the study design and protocol development. All authors reviewed and edited the manuscript and approved the final version of the manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research,authorship,and/or publication of this article.

Ethical approval

This study did not require ethical approval as all data were open-source.

Funding

The author(s) received no financial support for the research,authorship,and/or publication of this article.

Guarantor

CELE.

Informed consent

This study did not use patients;therefore,patient consent was not required. All data used in this study were publicly available,and the Twitter user is not identifiable in this manuscript.

ORCID iD

Cassandra H Ellis

Supplemental material

Supplemental material for this article is available online.

References

Adamski

Truby

Klassen

, et al. Using the internet: nutrition information-seeking behaviours of lay people enrolled in a massive online nutrition course. Nutrients 2020; 12: 750. DOI: 10.3390/nu12030750.

Zhang

Kuhnle

Smith

, et al. Fight under uncertainty: restraining misinformation and pushing out the truth. ACM Int Conf Adv Soc Networks Anal Mining 2018 266–273.

Mozaffarian

Forouhi

. Dietary guidelines and health—is nutrition science up to the task? Br Med J 2018: 360. DOI: 10.1136/bmj.k822.

Wakefield

Loken

Hornik

. Use of mass media campaigns to change health behaviour. Lancet 2010; 376: 1261–1271. DOI: 10.1016/S0140-6736(10)60809-4.

Chang

. Men's and women's responses to two-sided health news coverage: a moderated mediation model. J Health Commun 2013; 18: 1326–1344. DOI: 10.1080/10810730.2013.778363.

Diekman

Ryan

Oliver

. Misinformation and disinformation in food science and nutrition: impact on practice. J Nutr 2022. DOI: 10.1016/j.tjnut.2022.10.001.

Vijaykumar

McNeill

Simpson

. Associations between conflicting nutrition information, nutrition confusion and backlash among consumers in the UK. Public Health Nutr 2021; 24: 914–923. DOI: 10.1017/S1368980021000124.

Clark

Nagler

Niederdeppe

. Confusion and nutritional backlash from news media exposure to contradictory information about carbohydrates and dietary fats. Public Health Nutr 2019; 22: 3336–3348. DOI: 10.1017/S1368980019002866.

Lee

. In: Peifer

Lee

Metzgar

(eds) Perceived influence of fake news and its consequences. ProQuest Dissertations Publishing, 2018.

10.

Charnock

Shepperd

Needham

, et al. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health (1979) 1999; 53: 105–111. DOI: 10.1136/jech.53.2.105.

11.

Robinson

Coutinho

Bryden

, et al. Analysis of health stories in daily newspapers in the UK. Public Health 2013; 127: 39–45. https://doi.org/10.1016/j.puhe.2012.10.001.

12.

El Jassar

Kritsotakis

. Assessment of quality of information available over the internet about vegan diet. 2019.

13.

Kininmonth

Jamil

Almatrouk

, et al. Quality assessment of nutrition coverage in the media: a 6-week survey of five popular UK newspapers. BMJ Open 2017; 7. DOI: 10.1136/bmjopen-2016-014633.

14.

Strong

Wells

. Brexit-related food issues in the UK print media: setting the agenda for post-Brexit food policy. Br Food J 2020; 122: 2187–2201. DOI: 10.1108/BFJ-08-2019-0582.

15.

Sun

Cang

Ruan

, et al. Reporting gaps between news media and scientific papers on outdoor air pollution–related health outcomes: a content analysis. Int J Health Plann Manage 2020; 35: 221–232. DOI: 10.1002/hpm.2894.

16.

Awofeso

Imam

Ahmed

. Content analysis of media coverage of childhood obesity topics in UAE newspapers and popular social media platforms, 2014-2017. Int J Health Policy Manag 2019; 8: 81–89. DOI: 10.15171/ijhpm.2018.100.

17.

Sabbagh

Boyland

Hankey

, et al. Analysing credibility of UK social media influencers’ weight-management blogs: a pilot study. Int J Environ Res Public Health 2020; 17: 9022. DOI: 10.3390/ijerph17239022.

18.

Davis

Knol

Crowe-White

, et al. Homemade infant formula recipes may contain harmful ingredients: a quantitative content analysis of blogs. Public Health Nutr 2020; 23: 1334–1339. DOI: 10.1017/S136898001900421X.

19.

Boepple

Thompson

. A content analysis of healthy living blogs: evidence of content thematically consistent with dysfunctional eating attitudes and behaviors. Int J Eat Disord 2014; 47: 362–367. DOI: 10.1002/eat.22244.

20.

Dickinson

Watson

Prichard

. Are clean eating blogs a source of healthy recipes? A comparative study of the nutrient composition of foods with and without clean eating claims. Nutrients 2018: 10. DOI: 10.3390/nu10101440.

21.

Allen

Dickinson

Prichard

. The dirt on clean eating: a cross sectional analysis of dietary intake, restrained eating and opinions about clean eating among women. Nutrients 2018: 10. DOI: 10.3390/nu10091266.

22.

Mete

Curlewis

Shield

, et al. Reframing healthy food choices: a content analysis of Australian healthy eating blogs. BMC Public Health 2019; 19: 1711–1711. DOI: 10.1186/s12889-019-8064-7.

23.

Chan

Drake

Vollmer

. A qualitative research study comparing nutrition advice communicated by registered dietitian and non-registered dietitian bloggers. J Commun Healthc 2020; 13: 55–63. DOI: 10.1080/17538068.2020.1749351.

24.

Zhang

Sun

Xie

. Quality of health information for consumers on the web: a systematic review of indicators, criteria, tools, and evaluation results. J Assoc Inf Sci Technol 2015; 66: 2071–2084. DOI: 10.1002/asi.23311.

25.

Paterson

Thoma

Milne

, et al. A systematic review and qualitative analysis to determine quality indicators for health professions education blogs and podcasts. J Grad Med Educ 2015; 7: 549–554. DOI: 10.4300/JGME-D-14-00728.1.

26.

Eysenbach

Powell

Kuss

, et al. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. Jama 2002; 287: 2691–2700. DOI: 10.1001/jama.287.20.2691.

27.

Afful-Dadzie

Egala

. Social media in health communication: a literature review of information quality. Health Inf Manage J 2021. DOI: 10.1177/1833358321992683.

28.

Kalsnes

Larsson

. Understanding news sharing across social media: detailing distribution on Facebook and Twitter. Journalism Stud (London, England) 2018; 19: 1669–1688. DOI: 10.1080/1461670X.2017.1297686.

29.

Pearce

Niederer

Özkula

, et al. The social media life of climate change: platforms, publics, and future imaginaries. Wiley Interdiscip Rev Clim Change 2019; 10: n/a. DOI: 10.1002/wcc.569.

30.

Perl

Howlett

Ramesh

. Policy-making and truthiness: can existing policy models cope with politicized evidence and willful ignorance in a “post-fact” world? Policy Sci 2018; 51: 581–600. DOI: 10.1007/s11077-018-9334-4.

31.

Koop

Jansen

. Political blogs and blogrolls in Canada: forums for democratic deliberation? Soc Sci Comput Rev 2009; 27: 155–173. DOI: 10.1177/0894439308326297.

32.

Borah

. Who is the agenda setter? Examining the intermedia agenda-setting effect between Twitter and newspapers. JInf Technol Polit 2019; 16: 236–249. DOI: 10.1080/19331681.2019.1641451.

33.

Karami

Dahl

Turner-Mcgrievy

, et al. Characterizing diabetes, diet, exercise, and obesity comments on Twitter. Int J Inf Manage 2018; 38: 1–6. DOI: 10.1016/j.ijinfomgt.2017.08.002.

34.

Andreotta

Nugroho

Hurlstone

, et al. Analyzing social media data: a mixed-methods framework combining computational and qualitative text analysis. Behav Res Methods 2019; 51: 1766–1781. DOI: 10.3758/s13428-019-01202-8.

35.

Pantic

. Engagement with live blogs: when passive consumption overpowers participation. Electron News (Mahwah, NJ) 2020; 14: 22–36. DOI: 10.1177/1931243120910449.

36.

Veltri

Atanasova

. Climate change on Twitter: content, media ecology and information sharing behaviour. Public Understanding Sci (Bristol, England) 2017; 26: 721–737. DOI: 10.1177/0963662515613702.

37.

Fownes

Margolin

. Twitter and climate change. Sociol Compass 2018; 12: e12587–n/a. DOI: 10.1111/soc4.12587.

38.

Saltaouras

Lightowler

Coe

, et al. Availability and quality assessment of online nutrition information materials for pelvic cancer patients in the UK. Eur J Cancer Care 2019; 28: e13039–n/a. DOI: 10.1111/ecc.13039.

39.

Wang

Chen

Shi

, et al.

What makes cancer information viral on social media?

Comput Human Behav 2019; 93: 149–156. DOI: 10.1016/j.chb.2018.12.024.

40.

Haber

Smith

Moscoe

, et al. Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): a systematic review. PLoS One 2018; 13: e0196346. DOI: 10.1371/journal.pone.0196346.

41.

Warner-Søderholm

Bertsch

Sawe

, et al.

Who trusts social media?

Comput Human Behav 2018; 81: 303–315. DOI: 10.1016/j.chb.2017.12.026.

42.

Himelboim

Smith

Rainie

, et al. Classifying Twitter topic-networks using social network analysis. Social Media + Society 2017; 3. DOI: 10.1177/2056305117691545.

43.

Bradette-Laplante

Carbonneau

Provencher

, et al. Development and validation of a nutrition knowledge questionnaire for a Canadian population. Public Health Nutr 2017; 20: 1184–1192. DOI: 10.1017/s1368980016003372.

44.

Anderson

Bell

Adamson

, et al. A questionnaire assessment of nutrition knowledge–validity and reliability issues. Public Health Nutr 2002; 5: 497–503. DOI: 10.1079/PHNPHN2001307.

45.

Mikhail

Rolls

Yost

, et al. Development and validation testing of a weight management nutrition knowledge questionnaire for adults. Int J Obes 2020; 44: 579–589. DOI: 10.1038/s41366-019-0510-1.

46.

Vet HCWd. Measurement in medicine : a practical guide. Cambridge: Cambridge University Press, 2011.

47.

Digital Inspiration. https://digitalinspiration.com/product/twitter-archiver/ (accessed October 2021 2021).

48.

McHugh

. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012; 22: 276–282. doi:10.11613/BM.2012.031.

49.

Linacre

. Winsteps® Rasch measurement computer program (version 5.4.0). Portland, Oregon: Winstepscom, 2023.

50.

. A new approach to measuring overall liking with the many-facet Rasch model. Food Qual Prefer 2019; 74: 100–111. https://doi.org/10.1016/j.foodqual.2019.01.015.

51.

Wright

. Separation, reliability and skewed distributions: statistically different sample-independent levels of performance. Rasch Meas Trans 2001; 14: 786.

52.

Ellis

Evans

CEL

Moore

. Systematic assessment of obesity information on the microblogging platform Twitter. Proc Nutr Soc 2020 79: E791. DOI: 10.1017/S0029665120007776.

53.

Newman

. Tracking the release of IPCC AR5 on Twitter: users, comments, and sources following the release of the Working Group I Summary for Policymakers. Public Understanding Sci (Bristol, England) 2017; 26: 815–825. DOI: 10.1177/0963662516628477.

54.

Chadwick

Vaccari

O’Loughlin

. Do tabloids poison the well of social media? Explaining democratically dysfunctional news sharing. New Media Soc 2018; 20: 4255–4274. DOI: 10.1177/1461444818769689.

55.

Chan

AKM

Nickson

Rudolph

, et al. Social media for rapid knowledge dissemination: early experience from the COVID-19 pandemic. Anaesthesia 2020; 75: 1579–1582. DOI: 10.1111/anae.15057.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.92 MB

0.00 MB