Web site and web page persistence and change : a longitudinal study

The World Wide Web is, by most accounts, growing and changing rapidly. This research addresses the Web entity issues of life and death, and change over time. This project is concerned with those elements associated with the Web site and Web page URLs and structures that provide insights into Web page and Web site constancy and persistence. It does not address nor analyze the content or meaning of Web pages and Web sites except to the degree that such information can be inferred from the URL.

This analysis is necessarily a longitudinal study. Two data collection periods were established for harvesting the Web site data: December 1996 to February 1997 for the first data capture, and July and August 1997 for the second. Web page data were taken on a weekly basis beginning in early January 1997 and for the purposes of this thesis, ending in late August 1997.

This research also addresses Web entity taxonomies or structures. There has been scant attention paid to Web page or Web site structures. It is suggested that different types of Web entities behave differently. Web pages and Web sites can be distinguished in several ways. This thesis focuses on Web entity attributes that can be determined from an analysis of the URL as well as by measures of Web object types and byte-weight.

It is found that Web sites and Web pages undergo significant changes over time. These changes include the redistribution of object types within Web sites, additions and deletions to text and graphic objects, and additions and deletions of hypertext links to other Web pages. It is also demonstrated that Web site and Web page typologies can help predict constancy and permanence behaviors. The study also suggests that as Web entities mature, their constancy and permanence behaviors moderate somewhat. From one week to the next, approximately twenty percent of Web pages will undergo some degree of change. At the same time, approximately five percent will be intermittently comatose.