SiteLines - Ideas About Web Searching

The Challenge of Evaluating Health Search Tools

Tony Gentile of Healthline has posted a long and interesting comment on my review of Healthline.com at http://www.buzzhit.com/2005/10/rita-doesnt-dig-us.html

Near the end of his article, Gentile asks,

“Is your fundamental believe [sic] that only ad-free content can be trusted? If so, unfortunately, that would put many companies out of business.”

I believe that there is good information on health matters provided by all kinds of web sources -- educational institutions, commercial web sources, and maybe even in some wiki-style sources (although even Wikipedia’s founder has publicly admitted to serious problems with content quality).

In my view, the issue of trustworthiness of search tools isn't simple. Evaluators like me are concerned more about disclosure, the content of the index, and the relevance and quality of results. Trustworthiness of search tools emerges when levels of disclosure, content, relevance and quality are satisfied. It's a package deal, and it doesn't matter if the tool is commercial or non-commercial in origin.

THe vast majority of commercial search tools on the web earn their revenue from different sources in order to deliver free information – and that includes not just sidebar ads, but also things like paid inclusion of content, paid placement for top results, partnerships, sponsorships, and other types of mechanisms that lead to preferred inclusion, or improve the positioning of content of paying partners. That immediately distorts the playing field of information indexed and retrieved by these search tools.

More disclosure, and more visible disclosure (i.e. alongside the actual result) would help a lot. I'm not the only one who feels this way: Consumerwebwatch.org has commissioned several studies on disclosure, and continues to concluded that there isn't enough of it, and that more of it, more prominently placed, would help consumers a lot.

Unfortunately it rarely happens, because there’s no outright requirement to disclose, except for lightweight US FTC guidelines, which govern only US-domiciled search tools.

Search tool marketers aren't any different from marketers of any stripe -- if they do their job right, people will believe that their product is somehow better, different, more comprehensive, and more relevant – in other words, more trustworthy. Trust is particularly important for consumer health information. Sometimes on-the-page marketing content is really helpful in building trust, often it's a little over-zealous, sometimes it is downright misleading, and sometimes there is no information at all. It's my job to scratch the surface and figure out if the search tool can actually deliver the goods.

Reviewers like me face challenges when reviewing commercial search tools. Without sufficient on-the-page disclosure, we must look at results of searches, in order to figure out the underlying index, the content delivery structure, and compare those results to what we could find in existing search tools.

Let’s look at the disclosures and promotional information in a tool like Healthline, as an illustrative example. According to the web site, 1,100 doctors are associated with Healthline. They helped to build Healthline's healthmaps, synonym and taxnonomy structure, and are associated with Healthline's original content. Healthline's editorial policy speaks of physician and editorial involvement in original content creation and selection of licensed content.

I couldn't find any original Healthline-created medical content anywhere in the Healthline site during the three times I checked it (my last check was October 27) -- the best I could find was third-party licensed content. In his blog post, Gentile also that there is much original Healthline content that isn't yet on the site -- built by those same doctors. I hope that the original content is added soon, because Healthline’s editorial policy page leads readers to believe that the original content is there already.

In the How We're Different section of the help file, Healthline states:

."..when you search with Healthline, we use the collective knowledge of over 1100 doctors to give you more precise search options, all just a click away (we call this Medically Guided Search)."

And in the Guided Tour, this statement appears:

"What if you could have over one thousand doctors help you search for health information on the Internet?"

In my view, Healthline's statements about physician involvement can make searchers believe that whatever they see linked from the Healthline site is either selected or written by those doctors. That’s not the case, and although the exact nature of physician involvement is disclosed in the site's help file, it took me three careful re-reads to realize that much -- even most -- of the content in Healthline's search index isn't vetted by those physicians.

There's also no disclosure that I can find anywhere on the site of just who some of those 1,100 Healthline-associated physicians are. Where are the names and credentials?

What about Healthline's healthmaps? Healthline claims that at least some of those 1,100 doctors helped build those healthmaps.

Some of the healthmaps are so generic that they look like placeholders. Perhaps they are, and will be updated in future builds of Healthline content. (I ran this check on October 27.) For example. see the Healthline healthmap for LUPUS NEPHRITIS, which looks fairly generic. In fact, it’s identical to the one for HEADACHE. Like the healthmap for HEADACHE, one of the Healthline healthmap links for Lupus Nephritis is to PREVENTION, an odd choice for a condition where Healthline's licensed ADAM-derived reference article states that there is no known way to prevent the disease.

And what happens when you click on PREVENTION in the Lupus Nephritis healthmap? You get links from the Healthline index that match the keywords lupus+nephritis+prevention but not, as the healthmap would lead you to believe, about prevention of lupus nephritis specifically. In fact, I only spotted one link, from a 2002 research article, that tangentially mentioned the possibility that heparin could prevent the occurrence of nephritis in experimental lupus.

Searching for another, more developed, healthmap for a disease, I went to the ACNE healthmap. When you click on a link for SYMPTOMS in the ACNE healthmap, the links presented come from the general index, using the pre-encoded search keywords ACNE SYMPTOMS. To see if the mapped links would be any different from the links I would get from a keyword search in Healthline, I went back and did another search in the top page of Healthline for the keywords acne symptoms. I got exactly the same 13 results that were delivered by the healthmap.

What good is a visual taxonomy if the linked information isn't any better than one could get with a simple keyword search? The idea of a medical taxonomy is to deliver information results that keywords alone couldn’t achieve.

Ultimately, it's the quality of the search results for ACNE SYMPTOMS that matter most to health information searchers. Healthline tells us:

"Healthline only searches the top health sites on the web, so you only receive the best health information, without having to sift through pages of unnecessary and unrelated results."

Among the 13 results delivered from a search on the healthmap for acne symptoms, I got links that included basic dictionary-type acne information from a yoga site; an online questionnaire that I could submit to a Rosacea-treatment manufacturer on my Rosacea symptoms; and an "Acne Knowledge Map" from an alternative treatment shopping site, GoldBamboo.com.

When one compares the quality of results in Healthline to those in other comparable health search tools, like MedHunt (http://www.hon.ch/MedHunt/) it’s easy to see the difference. MedHunt’s search for the keywords acne symptoms reveals 717 results (compared to 13 for Healthline) that are clearly differentiated in useful ways. Commercial sites are noted. Some are reviewed, others not, and that too is disclosed. Hits are organized by relevance but there are alternative ways to display them. Disclosure statements and content descriptions are easy to find in MedHunt's About Us section.

The web-based health information world already has good search tools – MedHunt and Medline Plus are just two examples, but I could provide more. I hope that more good search tools will continue to be developed and existing ones refined. When we review new search tools, it’s important to compare new tools to best-of-breed tools that already exist, and to test all claims of quality, comprehensiveness, relevance, and authority.

UPDATE October 28 2005: Somehow I made an error citing the contact for Healthline. I have corrected the inforamtion in this post today. Oops. Thanks for Tony of Healthline for spotting this and letting me know.

Scratching Under the Surface of a "New" Health Search Engine

Lots of buzz this week about Healthline.com, a new vertical search engine for medical information. Chris Sherman, in his SearchDay review, quotes the company's promotional material, which indicates that the site covers "62,000 web sites with between 45-50 million pages... [and] hosted content licensed from reliable content providers."

However, my own initial examination showed a site that offers little to rival the best quality ad-and-sponsorship-free medical content on the web through sites like Medline Plus. Healthline relies principally on content from popular pre-existing 3rd party .com sources that could be obtained from any commercial search engine.

I conducted a search of the keywords lung cancer in order to obtain results. The first link, Lung cancer - small cell (Doctor-Reviewed information) led to a brief definition of the term, reviewed by Allen J. Blaivas, D.O., Division of Pulmonary and Critical Care Medicine, UMDNJ-New Jersey Medical School, Newark, NJ, and updated in early 2005. A small unlinked logo on the right side of the page suggested that the content was derived from A.D.A.M., a popular consumer health encyclopedia which is also used in Medline Plus and many others. A close comparison revealed exact duplication of content, and attribution to Blaivas, in both the Healthline entry and the A.D.A.M. entry.

Returning to the list of top results in Healthline.com for lung cancer, I wondered about the inclusion of lung cancer links from the site Worldhistory.com. What's the link between lung cancer and history? Answer: nothing. The site is simply a domain name that repurposes the content of the Wikipedia, word for word.

Another link, to lung cancer links from Goldbamboo.com, offer "a comprehensive online source that combines Eastern and Western health and wellness information." The links to lung cancer information are largely sourceless, although it's clear from a cursory examination that all or almost all the informational content is repurposed from pre-existing sources that contain the keywords "lung cancer". In the about page of Goldbamboo.com, it becomes clear that the purpose of the site is to offer paid inclusion of content: "Each page view includes highly relevant product and advertising information tailored to the consumer's stated interests. "

Most of the remaining links on the Healthline search results page represented commercial sites such as Medicinenet.com, Healthwise (in this case repurposed through the Everett Medical Clinic, a chain of private medical clinics in Washington State), and Emedicine.com -- three popular commercial information sites whose content is repurposed in many other information sites, all well-linked in major search engines. The news links are derived from general news sources repurposed from Topix.net, a well-known commercial provider of news services.

The only site in the list of links that lacked advertising or some other form of paid sponsorship or repurposing of pre-existing commercial partnered content was NLM's MedlinePlus.gov.

I don't mind companies like Healthline trying to market themselves with a little puffery during launch (like asserting that the site was "created in collaboration with 1,100 physician specialists") but it's important for reviewers not to believe everything in the press kit. Clearly the vast majority of those "physician specialists" in Healthline come from somewhere else, likely the 3rd party content providers who supply much of the Healthline-branded content.

If plain-language lay-level searching is important, there are plenty of better options out there for non-medical searchers. You don't have to speak "medicalese" to MedlinePlus, the Canadian Health Network, or OMNI, either, and even the research-focused PubMed offers spelling checks, terminology alternatives, and sophisticated back-end query reformulations to non-experts.

And what about the content of Healthline? Much of it is derived from common, well-known and well-positioned consumer web content, and there is little about the search results that look different from anything else on the commercial, free web. Surely, if sites like Wikipedia -- where volunteers with minimal editorial oversight contribute content, and even its CEO concedes serious quality problems -- qualify as "high-quality, authoritative information", what does that say about the standards that information professionals set for the quality of health information?

Doctors are Blogging on Medical Searching

Thanks to my colleague Dean Giustini at University of British Columbia's Health Sciences Library for cluing me into the (relatively) new world of doctor blogging. A quick glance at two blogs, KevinMD and California Medicine Man reveals that doctors are sharing all kinds of information and ideas in their blogs, including thoughts on searching for quality medical information.

Librarians can learn a lot from reading these blogs. If you're one, expect an initial shocked-and-appalled reaction when you realize just how heavily docs rely on tools like Google and Google Scholar for searching for quality medical information.

In a recent entry, John Ford (aka California Medicine Man), who is an assistant prof at UCLA School of Medicine, recounted how he tweaks Google to get better results by running his search in Google and adding the keyword differential to get better results. KevinMD, aka Kevin Pho, an internal medicine physician at the Nashua Medical Group in Boston, recounts numerous keyword tips and tricks to squeeze better information out of Google (e.g. adding keywords like AAFP or ICD9 to tweak results.

It appears that many actually prefer Google to PubMed, mainly because it's fast and its relevance algorithm is deemed preferable to PubMed's last-in-first-out approach. Of course, most docs aren't also information professionals, so they won't know (unless we tell them) that Google's relevance algorithms prefer the most popular, and that its coverage of PubMed content can be significantly out of date. While there is nothing intrinsically wrong with using Google or its subsets to find what I call a something (which roughly translates into something that looks like "good enough" for the task at hand), it absolutely cannot be relied on for practice excellence.

I posted a comment on Ford's weblog today - you can read it at: http://www.blogger.com/comment.g?blogID=10965136&postID=111881456515423089&isPopup=true

Researching Medical Literature on the Internet - 2005 Update

In the latest issue of LLRX.com, Gloria Miccioli has produced an update on resources for finding health and medical literature on the web. In addition to PubMed and the NLM Gateway to Medline and other health databases, she also covers a range of commercial and non-commercial portal sites. For those unfamiliar with PubMed or the NLM Gateway, her consise summary is a great place to start.

Digitization of important historical medical journals

A large collection of important historical medical journals will be digitized and made available beginning in late 2004. From the press release:

"The Wellcome Trust, the Joint Information Systems Committee (JISC), and the
U.S. National Library of Medicine (NLM) are joining forces to digitise the complete backfiles of a number of important and historically significant medical journals. The digitised content will be made freely available on the Internet - via PubMed Central and augment the content already available there.

The NLM will manage the project, host the archive and ensure that the digital files are preserved in perpetuity.

The list of journals to be digitised will include the Annals of Surgery,
Biochemical Journal, Journal of Physiology and Medical History. Digitisation
will commence in Summer 2004 and the first titles will be online early in
2005."

Recommended Resource: NutritionSource

NutritionSource, from the Harvard School of Public Health, is designed to "provide timely information on diet and nutrition for clinicians, allied health professionals, and the public." The site is well designed and presented, with PDF options for printing on every page. Many recommendations are supported with references to major studies.

Keeping Up With Health Trends Online Through E-Letters

In the March issue of LLRX.com, Lois Ambash contributes an annotated roundup of recommended newsletters to help health professionals and researchers keep up to date with health-related issues, events policies and trends.

New from the AARP - Internet Resources on Aging

This is a new collection developed by the American Association of Retired Persons (AARP). I previously featured AgeSource in an earlier issue of SiteLines: this is not the same thing. Internet Resources on Aging contains 600 web sites of interest to older adults in the areas of health, caregiving, housing, services, and more. AgeSource includes aging-related research resources such as databases and statistical sources.

AARP's AgeLine database also recently added a customized feature "AgeLine Research to Go." Users can click on any of over 50 high-interest aging-related topics and get up to 20 recent citations/abstracts from AgeLine. New topics are added regularly. A companion features "AgeLine Searches to Go" covers consumer literature written for the general public interested in aging-related topics. (Thank you Margaret Eccles of the AARP for clarifying the differences between these tools!)

Ghostwriting in Medical Journals

In the Sunday December 7 Guardian, Anthony Barnett's article "Revealed: how drug firms 'hoodwink' medical journals" describes a widespread practice where large drug companies hire medical writers to write review articles on particular drugs, then submit the articles to reputable medical journals with the claim that they are written by academics or physicians. Barnett suggests that "almost half of all articles published in journals are by ghostwriters." The article includes statements from medical writing firms confirming the practice, and one particularly striking example revealed by Dr. David Healey, an eminent UK psychiatrist, critic of the psychiatric drug industry and author of Let Them Eat Prozac.

The problem of medical ghostwriting is not particularly new. The CBC profiled the practice in a 2003 episode of Marketplace, and articles on ghostwriting can be found as early as 1994 in the PubMed database. The CBC site has a particularly good section of links to related articles on the topic, for further review.

Thank you Dan Dagostino of the University of Toronto Library for drawing the Observer article to my attention.

FLASH Tutorial for PubMed

The National Library of Medicine has created a Flash-enhanced tutorial on how to use PubMed. The tutorial is nicely laid out, with a sidebar menu of major sections. Each section is subdivided using a series of horozontal file tabs. Plenty of "show me" options enhance interactivity, along with exercises and suggested answers.

Recommended Resource: National Electronic Library for Health (NeLH)

North Americans often forget the wealth of English language resources from Europe and focus on North American-produced resources. The National Electronic Library for Health(NeLH) is one of those often-overlooked resources and deserves a place on your bookmark list. The electronic library is produced by the UK National Health Service, and attempts to provide health care professionals with medical information. Patient information is provided through the companion site NHS Direct Online, and the relationship between the two sites is similar to the Medline/MedlinePLUS sites in the US. Even the professional site's material is practical and useful for patients. A worthy addition to general medical and health link lists. (Spotted on RLG Shelflife, November 26 2003)

Recommended Resource: Ethnomed

This valuable new link list is a joint project of the University of Washington Health Sciences Library and the Harborview Medical Center's Community House Calls Program. Ethnomed contains information related to health care of selected specific cultural groups. The dozen-plus groups are representative of "recent immigrants to Seattle [homebase of Ethnomed] or the US, many of whom are refugees fleeing war-torn parts of the world." The site includes a cultural profile for each population group, selected clinical topics, and helpful foreign-language patient information handouts. Note that the site is rather underpopulated with links at this time. However, the structure of the site is fully visible and I look forward to seeing more resources linked in the near future. Contributors who wish to create materials for Ethnomed are invited, and the site has instructions and templates for content creation.

Stress-Free Web Sites

Librarians Index to the Internet added some interesting sites related to coping with holiday stress: (note: descriptions are drawn directly from LII)
---------------------------------------------------------------------
Stress Management: Managing Holiday Travel Stress
Provides basic tips for managing stress while traveling by car and airplane during the holiday season. Includes information on planning before traveling, and advice for parents of young children. Maintained for About.com by "a physician, researcher, and writer with an interest in stress and its effects on the human body."
---------------------------------------------------------------------
Family and Relationships: Get the Facts: Coping with Holiday Stress
Questions and answers about coping with stress and family problems during the holiday season (Halloween through Christmas). By psychologist Dorothy Cantor, a former president of the American Psychological Association (APA).

Problems with Information on Health Web Sites

ConsumerWebWatch recently published an important study on the quality of information on health-related web sites. This is essential reading for health specialists and those who serve the information needs of the public seeking health infomation online.

Some interesting observations from the report:

• Whereas the medical literature is consumed with the importance of whether information is accurate on a web site, the popular press pay little attention to accuracy.
  
• The popular press -- even the biz-tech press, has ignored the story of the possible relationship between health web sites and pharmaceutical companies. "For example, when Yahoo! created its health Web site health.yahoo.com both ZDNet and CNet published stories about it. They discussed how this would be a new revenue stream for Yahoo! because pharmaceutical companies would be interested in advertising on
  the site. ...Neither piece was at all concerned about how ad revenue from health firms might influence health content, despite mentioning that the site would give links to prescribed medication right next to facts about symptoms and news headlines about each ailment.
  

Determining the quality and reliability of commercial health information is a big job, even for journalists who regularly investigate sources of information. Most commercial web sites disclose little about their ownership. Ownership of web sites (or investment in those sites) by pharmaceutical companies or their officers could have an impact on the information offered -- and if not the information, certainly the advertising appearing on the page. It is also possible -- and difficult to figure out -- what information about a disease, condition, or drug is MISSING from a health-related web site, and if that omission is intentional.

Database of International Rehabilitation Research

Spotted on Gary Price's Resource Shelf:

Produced by the Center for International Rehabilitation Research Information and Exchange (CIRRIE), at SUNY Buffalo, the
Database of International Rehabilitation Research contains "over 21,000 citations of international rehabilitation research published between 1990 and the present." You can search by subject heading (and there is a thesaurus available), author, title keywords, research area, geographic area, and language. A handy quick search link makes major topical areas browsable - a great feature for those searchers who can't figure out the search functions.

This is a handy database alternative for those who can't access licensed databases for rehabilitation resources.

Older Medine Citations to Be Added to PubMed

The National Library of Medicine reported that over 1.5 million old Medline citations dating from 1953 to 1965 will be added to the PubMed database. Because the data lacks certain coding conventions, there are some quirks involved in retrieving OLDMEDLINE records in PubMed. Serious PubMed searchers should consult the NLM Technical Bulletin related to the announcement.

Bioethics Web

Another excellent affiliate site from the UK-based Resource Discovery Network, BioethicsWeb is a "gateway to evaluated, quality Internet resources relating to biomedical ethics, including ethical, social, legal and public policy questions arising from advances in medicine and biology, issues relating to the conduct of biomedical research and approaches to bioethics." The site was launched about a year ago and now has links to just under 300 resources -- all carefully evaluated and fully annotated. The site is browsable through a guided hierarchy. Entries are also searchable through a typical search template.

My only gripe is a typical one with library-developed gateways -- the presentation of browse/search in an alphabetical list arranged by title.

Recommended Resource: Hitting the Headlines

Just how important is that latest health news report? Do you ever question the reliability of mass media health news reporting? How can you evaluate the evidence behind the news?

Hitting the Headlines may be able to help. Starting in mid-2001 and sponsored by the UK's National Electronic Library for Health, staff from the NHS Centre for Reviews and Dissemination were commissioned by the NELH to assess the reliability of both the journalists' reporting of health stories and the research on which they are based.

CRD staff provide a rapid assessment of the original research behind the news story and evaluate how accurately the journalists have reported the findings of the research. The summaries are produced within 2 days of publication. The site also provides an FAQ and background on the project.

The site is keyword-searchable. This is an excellent companion to Biomedicine and Health in the News, reported in the April 6 issue of SiteLines.

AgeSource Worldwide

AgeSource Worldwide is a new free database on aging from the American Association of Retired Persons (AARP). It's a nice compliment to the better-known AgeLine (also available form the AARP) which indexes aging-related information in books, journals, and videos.

AgeSource Worldwide is not a journal index, rather it is a metasite that includes links to "clearinghouses, databases, libraries, directories, statistical resources, bibliographies and reading lists, texts, and Web "metasites" focused on aging or closely allied subjects." This is an excellent resource that belongs in any link list related to aging, health or wellness, and a great model of how high quality links can be compiled and delivered over the web.

As a new resource, it's not large yet - only 200 resources, and although the project is international in scope, over half originate in the U.S. Don't be put off by the apparently small number -- each link represents a sizeable collection of items.

AgeSource enables topic selection through a series of checkboxes -- a big bonus when search language can be variable -- and it's also keyword-searchable. Helpful annotations precede entry into any of the links. The site is also available in French and Spanish.

NLM Implements Educational Clearinghouse

Another winning initiative from the US National Network of Libraries of Medicine -- an Educational Clearinghouse Database as part of their National Training Center and Clearinghouse initiative. The database links to information, training resources and available courses related to biomedical topics and tools from the National Library of Medicine, the National Network of Libraries of Medicine and other governmental, educational and not-for profit sites.

There isn't much in the database yet (I counted about 100 entries today) but that will likely change quickly. Medical information searchers and librarians will find many of the resources immediately useful. One cautionary note -- as a clearinghouse there is some variability of content quality, so caveat lector.

Finding the Original Source for Health News

It's been around since 1997, but Biomedicine and Health in the News is a great tool for finding the research behind the headlines in health care news.

Two similar services -- one for the Minneapolis Star Tribune and another for the New York Times -- link health news headlines in these papers to the original research articles. Most news stories never precisely cite the original source of medical breakthroughs, and this tool is a great way to fill in the gap.

Evaluating Health Information on the Web

Thanks to Gary Price for alerting me to an excellent toolkit on evaluating health informaation on the Internet -- another superb guide from the National Library of Medicine's MEDLINEPlus.

Top 10 Medical Web Sites from the Medical Library Association

From the Medical Library Association, their list of "Top 10" health-related web sites. It's a pretty solid list, and to it I would add the Hardin Meta Directory and for Canadian readers, the Canada Health Network