Vortals Making a Comeback

Vertical search engines are back because its cheaper and easier than ever to cluster and correlate information, writes CIO Update guest columnist Raul Valdes-Perez of Vivisimo.

In the late 1990’s, topic-specific portals or vortals (vertical portals) were all the rage. Vortals were needed because general Web search engines such as Altavista, Google, and others returned too much information of spotty relevance and credibility.

A search for golf courses in Florida might return a Florida State University student’s home page recounting his last summer’s internship at a golf course. A golf.com vortal could instead lead searchers to more-reliably relevant search results.

There are five reasons why Web vortals declined five years ago and why they are now set for a comeback. After listing these, I will discuss how enterprises can borrow these ideas and jump early into emerging enterprise vortals.

Bandwidth and hosting were costly. Not anymore. High-profile business writers are constantly commenting on how the “cheap revolution” is making new applications possible. Vortals are a great example.

Web-crawling enterprise search products were difficult to use or expensive. Modern enterprise search software is surprisingly affordable and easy to use for crawling and indexing both internal content and external Websites — wherever the right information is stored.

To build an information service, the previous mindset was that the content first had to be brought in-house and only then stored in the corporate database or indexed by the enterprise search engine.

Now, modern meta (federated) search software can simply meta-search remote search services, under a subscription if needed, rather than license their content and bring it in house. Then, search results from one’s own search engine can be blended with the remote search results into one overall list. Web meta-search engines do this blending quite effectively.

In the past, building a vortal always involved building a directory of the information, such as Yahoo and Open Directory did for the general Web. Directory-building, also known as taxonomy-building, is expensive and laborious and requires constant maintenance as new topics become important.

Today, directory building is being superseded by search technology combined with clustering of search results, which give the benefits of organized information without the endless costs and headaches of directories.

Web business models were shaky in the late 90’s. Revenue was usually based on non-targeted banner ads. Today, search-based paid listings supplied by the major Web properties or their value-added resellers provide significant revenue shares.

Now that Web vortals are poised for a resurgence, what are the implications of Web vortal technology for the enterprise, and how does it all come together?

The advertising business-model issue (No.5) and bandwidth costs are less relevant for the enterprise but the other improved circumstances fit squarely with enterprise needs and opportunities.

Enterprises can now develop topic-specific search portals within weeks, for the benefit of either their own knowledge workers or their customers. As an example, a company whose main product is finite-element-methods software for materials engineers can follow these steps:

Engineers and librarians cooperate to prepare a list of the important external and internal resources: internal documents, external corporate Websites, research pages, etc.

If a resource is already served by a good, focused search engine, then meta-search it. If a resource is not covered by a good search engine, or if it’s covered by a search engine that also indexes irrelevant material, then use an enterprise search engine to crawl the pages.

Use enterprise meta-search software to build a single search box that forwards user queries to the search engines identified or newly built in step No.2.

Cluster the combined search results, which may be quite heterogeneous in form as well as content; a challenge that text clustering ideally solves.

The entire process can take as little as a few weeks. Further time can be spent identifying new resources and adding them modularly to the vortal.

The cost involves the time of engineers and librarians, for whom the task is interesting and motivating, the enterprise software costs of search, meta-search, and clustering, and a server or two.

The example was finite element methods, but any business or company, or department or research group within a large corporation, has its specialties. Those search boxes perfectly embodies the idea of a topic-specific Web portal with crawls of external Web sites and its own content, meta-search of other search engines, and clustering of the joint search results.

Web vortals are back and enterprise vortals are set to emerge, driven by the tremendous leaps in technical convenience and affordability and the competitive need for information advantage.

Raul Valdes-Perez is CEO of Vivisimo, a maker of search and clustering software. He can be reached at [email protected].