sizing the web: domains, sites, hosts

This page examines the size and shape of the web: estimates of the number and growth of domains and hosts.

It covers -

• metrics challenges
• three epochs?
• how many domains and servers
• how many registrations
• how many live registrations and non-renewals
• what is the number of internet hosts
• where is growth occurring

     metrics challenges

Identification of the total number of sites on the web is problematical, one reason for the inconsistency of some figures cited in this guide.

Figures about registrations (live or otherwise) are more certain than those for the number of sites online at any one time. The implications of those figures are even more problematical, as Andrew Odlyzko notes in a cogent paper on Internet Growth: Myth & Reality, Use & Abuse in the November 2000 issue of Information Impacts magazine.

Figures for domains are more certain than those for the number of pages or the number of links. Disagreement about terminology, data collection challenges and the commercial incentive to look on the bright side - or simply add a digit or two - mean that figures on number of viewers, audience demographics or the consequences of their visits are yet more contentious.

We have highlighted particular data collection and analysis challenges later in this guide.

     three epochs?

In terms of number of domains the web has experienced three epochs:

• 1991-1997 - explosive growth, at a rate of between 1,600% and 850% per year
• 1998-2001 - rapid growth, at a rate of 150% per year
• 2002-2006 - maturing growth, at a rate of 25% per year.

At the peak in 1994-5 the web grew from 700 sites to 12,000 sites.

If current growth continues the web will increase to 200 million sites by 2010.

     how many domains and servers

The October 2001 Web Characterisation report from the OCLC claimed that there were 8,745,000 unique sites, of which 3,119,000 are 'public', 2,078,000 are 'private' and 3,246,000 are 'provisional'.

As of 10 June 2000 one global figure for active domains - ie those that were live - was 17.75 million, including 9.48 million dotcom domains. Those figures come from the DomainStats site. The OCLC had estimated that there were 7.12 million unique sites as of June 2000, up from 4.6 million the preceding year.

Netcraft argued that there were over 15 million domains in 2000, with slower growth of registrations in that year (around 10% per month) and the disappearance of 330,000 domains. By late 2006 Netcraft identified 101,435,253 sites, including 'parked' domains and abandoned blogs or other sites that had not been updated for two years.

One 2008 study analysed distribution of IP address allocation in 238 countries. As of 2007 the US held 37.73% of IP addresses worldwide, followed by the UK (12.83%), Japan (7.64%), China (5.74%), Germany (3.81%), France (3.65%), Canada (2.81%), S Korea (2.74%), Netherlands (2.00%), Italy (1.67%) and Australia (1.62%). Those nations had over 80% of total allocated IP address ranges.

     registrations

We have highlighted some domain registration figures in a supplementary profile on Domains & the DNS.

As of October 2002 there were over 21.2 million dot-com registrations, for example, along with 3.6 million dot-net registrations, 740,000 dot-biz, 80,000 dot-name, 920,000 dot-info and around 2.3 million dot-orgs.

Not all of those registrations were active. Some had gone offline forever, others were temporarily unavailable, others had been registered by domain name speculators in the hope of making a profit from transfer to a new name holder, others were being used by registrars and other entities engaged in domain name tasting.

As at December 2002 there were 310,733 registrations in the overall dot-au space (analysed in our auDA profile), with over 390,000 by December 2003 and 500,000 by early 2005. The number of registrations for some dot-au 2LDs was -

Nov 95 Dec 96 Dec 97 Dec 98 Dec 99 Dec 00 Oct 01 Dec 02 Dec 03 Dec 04 Dec 05 Dec 06

com 2,573 13,555 31,657 62,898 126,591 202,484 229,339 278,903 340,589 425,698 539,248 697,763

net 17,384 17,383 15,849 27,812 38,939 51,897 69,713

org 63 520 1,520 3,000 4,850 6,700 7,841 11,218 15,479 16,247 20,773 25,637

asn 1,983 2,532 3,022 3,377 2,664

id 510 1,741 3,063 5,197

As at August 2002 there were around 113,781 active domains in the dot-nz space (discussed in a more detailed profile).

Estimates by Domains Worldwide of other ccTLD registrations as of October 2002 were -

ccTLD	Registrations (m)
Germany Italy Netherlands Switzerland Belgium S Korea Tuvalu Brazil UK	5.45 0.68 0.61 0.44 0.19 0.40 0.40 0.40 3.50

We have supplied some other figures in a more detailed note on domain name sizes.

     how many registrations are live?

There is considerable industry and academic disagreement about how many of the millions of domain registrations are live at any one time and are associated with unique sites, ie visitors to the address find a distinct site rather than a 'coming soon' marker, an automated redirection to the 'real' site at another address or simply nothing at all.

That disagreement is evident in skepticism about "domain name exhaustion" (claims that the supply of names is about to run out, so ICANN or ccTLD administrators must mint new TLDs or 2LDs) and suggestions that several million names in popular TLDs such as dot-com or dot-info have been registered by speculators or on a defensive basis by major corporate interests.

On the basis of a small-scale sample an October 2002 report by State of the Domain (SoTD), a subsidiary of a major domain registrar, offered indicative figures for some gTLDs -

gTLD	live (%)	redirected (%)	inactive (%)	'parked' (%)
com org net biz info name	44.6 41.4 30.6 25.8 31.5 34.7	1.0 1.9 2.4 1.1 5.7 0.6	25.5 26.6 33.1 33.3 31.5 34.7	19.4 16.3 22.7 21.8 18.6 38.0

A similar report from registry operator Afilias in July 2002 suggested that 24% of dot-info sites were live, 9% were redirected, 12% were parked and 35% were inactive.

The DeletedDomains service claimed that a total of 18,395,664 dot-com, dot-net and dot-org registrations had not been renewed by registrants, of which 13,783,454 were dot-coms. In October 2005 Netcraft reported 74.4 million 'sites', including single-page sites used by domain registrars to promote individual unsold domain names. Reports on domain name tasting (aka domain kiting) indicate that around 90% of registrations in some gTLDs were evanescent, exploiting loopholes in registration rules that allow registrars to register and then delete names within short grace period (typically five to 30 days).

Figures for the number of blogs and other types of sites are problematical. In mid-2003 Blogcount estimated that there were between 2.4 million to 2.9 million active blogs - discussed here - although other estimates were significantly lower.

     number of hosts

The January 2000 Domain Survey by the Internet Software Consortium (ISC) suggested that there were upwards of 88 million hosts on the net, expected to increase to around 500 million by early 2003.

The ITU reports (PDF) that in 2001 there were 141 million hosts across the globe, of which 2.28 million were in Australia, 0.4 million were in New Zealand, 2.89 million were in Canada and 106.2 million were in the US. Africa had 0.274 million (of which 0.238 million were in South Africa).

The OECD believes that there are around 52 thousand secure servers - tools for electronic commerce - in the USA, and upwards of 74 thousand in the OECD as a whole (a growth of 95% over the preceding year).

Thoughtful comments on host versus domain counts are made in Matthew Zook's 2000 article Internet Metrics: Using Hosts & Domain Counts To Map The Internet, complemented by the 1998 OECD report on Internet Infrastructure Indicators (PDF).

The World Internetworking Alliance, a now-defunct advocacy group, suggested that the number of hosts almost doubled each year to 1997 -

Year 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997

Hosts 2k 5k 28k 56k 159k 313k 617k 1.13m 2.05m 3.86m 6.64m 12.88m 22.00m

The OECD reports that the regional distribution of hosts per 1,000 inhabitants was as follows -

Region N America Oceania Europe Latin America Asia Africa

1997 46.28 26.8 6.13 0.48 0.53 0.17

1998 69.74 34.76 9.45 0.91 0.87 0.21

1999 116.41 43.84 13.41 1.67 1.28 0.28

2000 168.68 59.76 20.22 2.53 1.96 0.31

As we have noted in discussing digital divides, such 'regionalisation' can produce substantial distortions: much of Oceania for example is comparable to Africa when figures for Australia and New Zealand are removed.

Early growth of the web is highlighted by another estimate of the number of sites -

Year Jun 1993 Dec 1993 Jun 1994 Dec 1994 Jun 1995 Dec 1995

Sites 130 623 2,738 10,022 23,500 90,000

     where is the growth occurring?

The October 2001 Web Characterisation report from the OCLC claimed that growth of the web was slowing, with an estimated 8.7 million unique sites.

The Mosaic Group at the University of Omaha has a project on Global Diffusion of the Internet, measuring growth of the net on a global and nation by nation basis.

The Internet Geography Project (IGP) at the University of California, under Matthew Zook, offers authoritative maps and a number of excellent papers.

The UN Development Program 1999 Human Development Report (HDR) includes statistics about internet diffusion in the third and fourth worlds.

The 2003 paper Trends in the Evolution of the Public Web 1998-2002 by Edward O'Neill, Brian Lavoie & Rick Bennett suggested that the publicly-accessible web, as of June 2002, contained 3,080,000 sites, an estimated 35% of the overall web. Growth had continued to slow since 2001.

They commented that

Examination of year-on-year growth rates (measured in terms of the number of Web sites) for the period 1998 - 2002 reveals this decline: between 1998 and 1999, the public Web expanded by more than 50 percent; between 2000 and 2001, the growth rate had dropped to only 6 percent, and between 2001 and 2002, the public Web actually shrank slightly in size. The slowdown in growth of the public Web is even more dramatically evident in absolute terms. Between 1998 and 1999, the public Web exhibited a net growth of 772,000 sites; a similar number (713,000) were added between 1999 and 2000. After this point, however, absolute growth dropped off precipitously: between 2000 and 2001, only 177,000 new public Web sites were added, and between 2001 and 2002, the public Web shrank by 39,000 sites.

   next page (content)