Market Forces May Solve Search Engines' "Tragedy of the Commons"

Email this to a friend   Print this story

The 'death of free submission' is a good thing. Paid inclusion should lead to less cluttered search indexes and more solvent search engine companies In the old days (1999!), submission was a numbers game. The more sites you could properly submit to, the more traffic you'd get. But as the internet matures, this is rapidly changing. The big difference is that these days, it costs to get a listing in the major indexes (Yahoo, LookSmart and NBCi), and the search engines are also getting into the act -- Inktomi now has a paid-submission option, and the other major search engines are all considering following along. There has been a lot of blathering o­n various discussion forums lately about the "death of free submission," but quite frankly I think that it's a good thing. I know that sounds weird, but bear with me for a minute. From the standpoint of a commercial website, the problem with the net is that it is drowning in other websites. The number of webpages is growing rapidly, which makes it harder for people to find your website in the sea of other, similar websites. When people want to find a website, they typically go to o­ne of the major indexes or search engines. From your standpoint as a commercial website operator, would you like their search to list 10,000 websites (submitted for free), or 100 websites (each of which has paid $199 to get listed)? Obviously, the latter! At the same time, as long as free submissions are available for non-commercial sites (and all the majors still permit this), non-commercial sites are not penalized, because they are not competing with commercial sites. So if you are not trying to make money o­n the web, fear not -- nothing has changed for you. But if you are serious about doing business o­n the web, then paid submissions are a good thing. They cut out all the chaff. Or to put it more bluntly, "Money talks, B*llsh*t walks!" The Economics of Paid Submission If you make a profit of $10 when someone buys something o­n your site, and 1% of your visitors buy something, then each new visitor is worth 10 cents to you. If you can get visitors for 9 cents each, you make money; if they cost you 11 cents, you lose money. It's that simple. There are three main sources for paid clicks that you should consider; paid submission to indexes (like Yahoo), paid inclusion in a search engine (currently o­nly Inktomi provides this), and sponsored listings in a pay-per-click search engine (the biggest being Goto.com). All can be excellent sources of cost-effective traffic. Paid Submission to Indexes Three major indexes are now offering paid submission, and some of them require it in their business and shopping categories. It is important to understand that paid submission does not guarantee a listing -- if your site isn't good enough, they'll reject it and you're out the money (though you can appeal). I have an extensive tutorial o­n how to submit to Yahoo and the major indexes  that should minimize your chances of a rejection and maximize your chances of a great listing that generates real traffic. I paid $199 for my Yahoo listing, and last year Yahoo sent me over 25,000 visitors. $199 / 30,000 = .8 cents a visitor. That's dirt cheap, and this year I get them for nothing, since a Yahoo registration is a 1-time thing. If you're selling o­n the web, that $199 is the best investment you'll ever make (other than the money you send to me, of course!) Yahoo has just started offering Sponsored Listings for between $25 to $300 a month, depending o­n category. 5 sponsored listings are displayed at the top of category pages (if more than 5 people buy sponsored listings, they rotate randomly). In order to get a sponsored listing, you must first get a normal listing in Yahoo, then you can apply for a sponsored listing in the category your listing is in. You can't use this to change your listing title or description, by the way; it just gets you "up top." Is it worth it? Hard to say, at present. I've applied for o­ne and I'm going to be watching it closely. o­ne problem may be that as more people sign up for a sponsored listing o­n a particular page, each listing appears less often (for example, if 10 people sign up, o­n average each will get their link displayed o­n o­nly half of the page views) - and it is unclear if Yahoo is going to adjust pricing to compensate for this. If you see a "more sponsored sites" link, then there are more than 5 sponsored listings for the page, click o­n the link to see how many. To apply for a sponsored listing, visit the Yahoo category page that contains your listing and click o­n the "what is a sponsored listing?" link. Yahoo has also introduced "Most Popular" listings underneath the sponsored listings; so far it is unclear how a site becomes a Most Popular site. The same goes, to a lesser degree, for the paid-inclusion deal at LookSmart. My $199 LookSmart listing generated about 1600 clicks last year (about 12.5 cents a click). Not nearly as economical as Yahoo, but not bad -- and again, this year the clicks will be free. On the other hand, I do not recommend paying for a listing at NBCi. My experience is that they don't generate much traffic at all, and most of it comes from their "LiveDirectory" listings that you can submit to for free. There are other, 2nd-tier indexes that offer paid submissions. None of them are worth it. Paid Inclusion o­n Search Engines Currently, o­nly Inktomi offers a paid inclusion service through a company called PositionTech. For $30 a year, PositionTech will guarantee your URL stays in the Inktomi database, and check it every couple of days for changes. The advantages of coughing up the cash are: <DIR><DIR> Your listing is always fresh, containing your latest content. If you play with your keywords, you get to see how your changes affect your ranking in a couple of days instead of a couple of months. Inktomi provides search engine results for many other search engines (such as HotBot and AOL), so you get broad placement. Inktomi currently penalizes pages that are submitted through their free Add URL service; pages submitted through the paid inclusion service or stumbled upon by the Inktomi spider (which independantly wanders around the web) do not suffer this penalty. Note however, that if you've submitted a URL through the free service and Inktomi later finds it o­n its own, the penalty is removed. This ranking penalty is Inktomi's way of dealing with "gateway" pages and spam pages; they claim that the vast majority of page submitted via the free Add URL service are spam. In addition, according to Positiontech, paid inclusion is the o­nly way to get "dynamic urls" (ie: o­nes with ? in them) into Inktomi, as the regular spider will not spider them. </DIR></DIR> I have tested the Inktomi paid inclusion service and it works as advertised. I recommend that commercial sites pay $30 and list their homepage using the service. While you can spend extra and list other URLs o­n your site (at a discount) via paid inclusion, in most cases this isn't necessary. It appears to be the case that paid inclusion URLs are often used as starting points by the Inktomi spider, and so your other URLs tend to be quickly found and indexed with no penalty. Also, o­nce your page has been found by the spider, resubmitting it via the free Add URL does not cause it to be penalized! So my advice is pay for your homepage, submit all your other pages via the free Add URL, and in a few weeks the spider will have found most of your pages and removed the penalty. I should note that PositionTech says that their paid inclusion service won't cause other URLs to be spidered, but my experience is otherwise. It may be that it doesn't cause your related URLs to be preferentially spidered. The jury is still out o­n this o­ne. As with every rule, there are exceptions, and if there are important pages o­n your site that are constantly changing, paying a little extra to have Inktomi refresh its listing more often may be a good investment. Sponsored Listings o­n a Pay-Per-Click Search Engine If the average visitor to your site earns you 10 cents in profit, and you can get o­ne by paying 5 cents, then you're in a position to make money. That's the idea behind pay-per-click. Search engine results are ranked not by keywords, but by how much you are willing to pay for a click. You bid for each keyword you want to sponsor. The two leaders in the field are Goto.com  (by far the big kahuna) and FindWhat The other sites aren't worth much, in my opinion. But the top two can make you money if you use them correctly, and I strongly recommend that commercial sites consider using them. I've written a detailed tutorial devoted to techniques for efficiently bidding o­n these sites, as well as Secret Net Tools (available to contributors) that help you find appropriate keywords to bid o­n and manage your bids. Future Trends Looking into my crystal ball, I think it's pretty clear that the days of effective free submission to the major search engines are numbered. This is a classic example of "the tragedy of the commons ". If a resource is freely available to everyone, then nobody has any incentive to conserve it, and everyone will try and exploit it as much as they can before the other guy ruins it. The result, in a village common, is overgrazing; o­n search engines, it is spam. If o­n the other hand, you charge people a small amount for listings, they get a lot more selective about what they submit. The quality of listings goes up. Spam goes down. This is a good thing. Furthermore, let's be honest, it's in our interests for the search engines to make money, because a bankrupt search engine is no help to anyone. Since at the present time, most of the search engines (except for Yahoo) are losing money hand over fist, I think that all the major o­nes will adopt paid inclusion plans similar to Inktomi. I think they'd be stupid not to. Going out further o­n a limb, I will make a stunning prediction. I predict that Yahoo will announce that they will start charging commercial sites a yearly maintenance fee for listings, in addition to their $199 listing fee. And by the way, Yahoo guys, if you end up doing this because I suggested it, you owe me - small, unmarked, nonsequentially numbered bills, please. Please Click if you like this on google plus.