SiteLines - Ideas About Web Searching

How Relevant (or Even Different) is User-Selected Relevance from Machine-Generated Relevance?

In April, Rustysearch.com created a site for its "Search Engine Relevancy Challenge" -- a white-branded site that randomly compiled the results form 4 different search engines and provided options for users to rate the site. Ahead by a hair - and JUST a hair -- Yahoo! (closely followed by Google). (See the early results at: http://www.seroundtable.com/archives/001900.html)

But really, just how relevant is this relevance test? Surely questions of searcher behavior (fast, interactive, tend to select the top stuff first) works into this alleged "blind" relevance test.

Asking users to rate content in order to determine relevance isn't new. The ongoing problem with relying on user preference and ratings is that most searchers can't adequately judge quality. They can only judge what appears to be relevant to their own information needs. And what's more, the typical user only selects from the fewest (and usually top-appearing hits) to determine relevance, and may well leave more relevant, higher quality resources unrated because they just didn't take the time to look at them.

What's New when the Internet Goes to College 2004-05

Steve Jones of the University of Illinois, and a research associate with the Pew Internet and American Life Project, produced a Powerpoint update of the 2002 Pew Internet report "The Internet Goes to College" with new data updated from 2004 and 2005. Some of the more interesting findings related to information-seeking by college students include:

Faculty don't see the Internet as improving the quality of student work -- rather, they see the quality of student work worsened or at best unaffected by the web (slide 19); and 44% of faculty feel that plagiarism has increased as the result of Internet access (slide 20).

Many of the comments that Jones supplies demonstrate the gulf between faculty and student views. While faculty are startled at the inability of students to discern the difference between quality and junk online, students seem to use whatever it takes to either make the prof happy or get the job done in the minimum time:

“I use Google because I heard it searches for more things” (than other sources)"
“I believe I can find anything on the Internet. There hasn’t been anything I haven’t been able to find.”
“Because I’m lazy.”
"Books have “so much information that no one can go through it all.”
"I use “the Internet first is because it is more convenient.”
"I go to the library “because that’s what teachers like.”
“Google has gotten me through college.”

Knowledge Workers Turn Away From the Open Web - Survey

Earlier this month, Outsell Inc. has released details of a major new survey that suggests that knowledge workers are relying less on the open web and more on informal sources of information such as colleagues, alerting services and other conveniences.

With that headline, you'd think that time spent searching for information was decreasing. Wrong: searchers are spending almost 30% more time in search-related activities today compared to four years ago.

That serious searchers are finding less on the open web isn't a surprise. In our experience working with business searchers, there are two signficant problems with their web search behavior: 1) relying almost exclusively on inadequate, popular search tools that rank results in similar ways; and 2) a search approach that relies almost entirely on keyword-guessing of search terms (which are often wrong).

Are Chinese searchers more "serious" than US searchers?

In a March 29 2005 paper, Ira Machefsky and John Fernandez of search engine Accoona, clue us in to some key differences between searchers in China and the US. The authors compared search terms used in the US and Chinese versions of Accoona. Whereas US searchers focus on news, gossip, and entertainment, Chinese searches show a strong focus on business information, particularly manufacturing.

From the report:

"Chinese searchers are focused on manufacturing and education while U.S. searchers are focused on entertainment, celebrities and games (and looking for ways to cheat at those games). ...Between those extremes there is an overlap of searches for current events and holidays, but U.S. searchers tend to focus on one-time catastrophes and current holidays, while Chinese searchers tend to focus on longer-term political issues, and especially in the recent past on ageing and emigration. Chinese consumer product searches tend to be for durable goods such as cars, while U.S. searchers tend to look for consumer electronics.

These search preferences tell a lot about the different preoccupations of two different cultures and civilizations. They reflect the mindsets of a free, established economically secure West with leisure time to spend and a striving, achieving, manufacturing-based China, looking for a more tolerant and open society. Our preoccupations make us what we will be, even as they reflect what we already are. Search terms give us a glimpse into these evolving societies."

What search engines do search engine companies use?

This report from VisitorVille claims to track the search engine preferences of employees of major search engines, presumably using tracking by IP address blocks. Slashdot picked out a few goodies in their summary of the report:

"Microsoft employees use Google for their searches 66% of the time, but MSN Search only 20% of the time... Google employees use Google as their search engine 100% of the time ... Apple employees like Google best... "

ConsumerWebWatch Report on Search Engine Disclosure of Paid Placement

This week, an important new report on the disclosure practices of search engines was released by ConsumerWebWatch. The report tracks the changes and improvements that search engines have made to their disclosure practices since the FTC warned them to do so in 2002. However the report also notes that consumers remain largely unaware of how paid placement can influence the display of search results.

Interestingly, the report's testers included information professionals, and even they remarked on the confusion that still remains.

"Even the report's testers, information professionals by trade, found disclosure and transparency practices among many search engines confusing and confounding. That increases the likelihood consumers may have a difficult time distinguishing objective search results from paid advertising."

Search engine technology compromised by the increase in web pages

"Information Overload, Retrieval Strategies and Internet User Empowerment" by Christopher Carlson, discusses how the promise of search engine technology has been compromised by the rapid increase of Internet web pages.

Carlson confirms every smart searcher's suspicion that traditional retrieval technologies (like search engines) are providing increasingly poor results as database size grows ever larger. Even techological advances (personalization, algorithmic ranking, clustering of results) have limited utility. Interestingly, Carlson concludes (and I agree) that the ultimate solution that will empower users will not be technological in scope -- in other words, providing better personalization, larger databases, more advanced search options, better clustering or better algorithms -- won't help matters much. Rather, "...it seems clear that user empowerment in terms of Internet retrieval will largely come from increased self-reliance and self-teaching of retrieval-relevant skills."

Canadians use search engines more than Americans

According to a report by eMarketer covered in the Search Engine Journal, Canadians use search engines more than Americans.

It found that Canadians performed 575 million searches in the month of April 2004,versus 3.6 billion searches conducted by people in the US during the previous month. However, taking into account differences in population, the report claims that the average Canadian performed more searches per month than the average person in the US.

This data comparison assumes that Canadians and Americans have equal web access and are equal users of the web. But is that true? I was suspicious so I went searching.

It's difficult to find comparable facts and figures for different countries. However, according to figures produced by Clickz and aggregated from several sources, Canada's online population of 20.45 million users represents about 63% of the total population, and is virtually identical to the 185.90 million US users, which represent about 64% of the US's total population.

Of course, errors might still exist -- geomapping of IP addresses isn't always reliable, and it's conceivable that some searchers in both countries conducted searches in search tools that were not included in the count. In the final analysis, while the differences between countries are still reasonably significant, what do they tell us about the differences between our nations' information seeking habits?

CIL 2004: Some Key Themes

At this year's annual Computers in Libraries conference, there seemed to be a couple of key recurring themes.

Trend 1: Search engine bashing, or the increasing irrelevance of commerial ad-supported search engines.

Some experts are shifting their views on search engine capbility. -- rather than telling us about new ways to manipulate search engines and other commercial free search tools, some experts are suggesting that search engines aren't doing their job very well any more. (I heard the term "Google-bashing" a lot from conference attendees, who seemed to catch the same trend.)

David Seuss, CEO of Northern Light, provided a thoughtful keynote on web search, indicating that after 10 years, the web search challenge is nowhere near solved. Seuss highlighted the increasing inability of search engines to deliver relevant search results to many queries; that as search engine databases grow, relevance declines as algorithms increasingly fail to recognize what researchers are truly looking for; that improvements in relevance algorithms can't really make a difference in an ad-driven search engine world. Seuss envisioned a world of fragmented, selective, specialized datasets/databases that serious searchers could select from and pay for --which sounded a lot like the old DIALOG search options of many years ago, and likely forshadowed Northern Light's new focus. View his presentation online at http://www.infotoday.com/cil2004/presentations/default.shtml

Search Engine Showdown's Greg Notess delivered similar warnings, reminding us that while the search engines promote their databases and search capability, they are increasingly turning into ad machines. I echoed those same views in my presentation on the business of search engines.

Trend: Still no serious takeup of RSS

Although several presentations on RSS capabilities were featured at CIL, it was abundantly clear that, among the library community, there is hardly any takeup of RSS. It's possible that the presentations assumed a level of understanding that just wasn't there, but there is a lot of work to be done at the introductory level just to help people get the basic concepts.

Workplace Internet abuse isn't as sexy as you think

Spotted in the August 19 issue of the Toronto Star: a study by Paul Mastrangelo of Genesee Survey Services -- presented at last week's American Psychological Association meeting held in Toronto -- estimates that office workers use the Internet -- a LOT. Mastrangelo estimates that "between 80-85% of employees use work computers for personal use" and that much of this use is for daily tasks like paying bills or shopping or looking up something on Google. He goes on to suggest that the answer for employers isn't simple, since some things (like banking) take way less time online than jumping into your car and leaving the office. A more rational approach is recommended, like training employees to use the Internet more efficiently.

If you're interested in reading beyond the headlines, Genesse Survey Services produced an interesting and far more detailed white paper (in WORD format) on the same topic.