The Journal of Web Science > Vol 1 > Issue 1

The Graph Structure in the Web – Analyzed on Different Aggregation Levels

Robert Meusel, Data and Web Science Group, University of Mannheim, Germany, Sebastiano Vigna, 2Laboratory for Web Algorithmics, Università degli Studi di Milano, Italy, Oliver Lehmberg, Data and Web Science Group, University of Mannheim, Germany, Christian Bizer, Data and Web Science Group, University of Mannheim, Germany
 
Suggested Citation
Robert Meusel, Sebastiano Vigna, Oliver Lehmberg and Christian Bizer (2015), "The Graph Structure in the Web – Analyzed on Different Aggregation Levels", The Journal of Web Science: Vol. 1: No. 1, pp 33-47. http://dx.doi.org/10.1561/106.00000003

Publication Date: 13 Aug 2015
© 2015 R. Meusel, S Vigna, O. Lehmberg, and C. Bizer
 
Subjects
 

Share

Open Access

This is published under the terms of CC BY-NC-ND 2.0.

In this article:
1. Introduction 
2. Definitions 
3. Related Work 
4. Datasets and Methodology 
5. Analysis of the Page Graph 
6. Analysis of the Host Graph 
7. Analysis of the PLD Graph 
Discussion 
Conclusion 
References 

Abstract

Knowledge about the general graph structure of theWorldWideWeb is important for understanding the social mechanisms that govern its growth, for designing ranking methods, for devising better crawling algorithms, and for creating accurate models of its structure. In this paper, we analyze a large web graph. The graph was extracted from a large publicly accessible web crawl that was gathered by the Common Crawl Foundation in 2012. The graph covers over 3:5 billion web pages and 128:7 billion hyperlinks. We analyze and compare, among other features, degree distributions, connectivity, average distances, and the structure of weakly/strongly connected components. We conduct our analysis on three different levels of aggregation: page, host, and pay-level domain (PLD) (one “dot level” above public suffixes). Our analysis shows that, as evidenced by previous research (Serrano et al., 2007), some of the features previously observed by Broder et al., 2000 are very dependent on artifacts of the crawling process, whereas other appear to be more structural. We confirm the existence of a giant strongly connected component; we however find, as observed by other researchers (Donato et al., 2005; Boldi et al., 2002; Baeza-Yates and Poblete, 2003), very different proportions of nodes that can reach or that can be reached from the giant component, suggesting that the “bow-tie structure” as described by Broder et al. is strongly dependent on the crawling process, and to the best of our current knowledge is not a structural property of the Web. More importantly, statistical testing and visual inspection of size-rank plots show that the distributions of indegree, outdegree and sizes of strongly connected components of the page and host graph are not power laws, contrarily to what was previously reported for much smaller crawls, although they might be heavy tailed. If we aggregate at pay-level domain, however, a power law emerges. We also provide for the first time accurate measurement of distance-based features, using recently introduced algorithms that scale to the size of our crawl (Boldi and Vigna, 2013).

DOI:10.1561/106.00000003