Skip to main content

Internet Archive Web Crawls

The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine.

920,042
RESULTS
rss


PART OF
Web Crawls
Media Type
108
collections
918,730
web
1,204
data
Availability
920,042
Always Available
Topics & Subjects
796,361
crawldata
2,263
no404
1,452
wikipedia
811
wordpress
252
amazonbooks
6
end of term
More right-solid
Collection
920,042
Web Crawls
920,042
Internet Archive Web Crawls
469,184
Worldwide Web Crawls
281,687
Youtube Videos
73,846
YouTube 2007 Crawl
71,730
Wide Crawl Number 14 started March 2016
More right-solid
Creator
790,269
internet archive
73,846
internetarchive
32,949
alexa internet
31
google, inc.
3
lekash@archive.org
3
ximm@archive.org
More right-solid
Language
751
English
SHOW DETAILS
up-solid down-solid
eye
Title
Date Archived
Creator
Worldwide Web Crawls
collection
469,183
ITEMS
3.3B
VIEWS
collection
eye 3.3B
Wide crawls of the Internet conducted by Internet Archive. Please visit the Wayback Machine to explore archived web sites. Since September 10th, 2010, the Internet Archive has been running Worldwide Web Crawls of the global web, capturing web elements, pages, sites and parts of sites. Each Worldwide Web Crawl was initiated from one or more lists of URLs that are known as "Seed Lists". Descriptions of the Seed Lists associated with each crawl may be provided as part of the metadata for...
Survey Crawls
collection
65,663
ITEMS
1.9B
VIEWS
collection
eye 1.9B
Survey crawls are run about twice a year, on average, and attempt to capture the content of the front page of every web host ever seen by the Internet Archive since 1996.
Topic: survey crawls
Live Web Proxy Crawls
collection
17,899
ITEMS
1.7B
VIEWS
collection
eye 1.7B
Content crawled via the Wayback Machine Live Proxy mostly by the Save Page Now feature on web.archive.org. Liveweb proxy is a component of Internet Archive’s wayback machine project. The liveweb proxy captures the content of a web page in real time, archives it into a ARC or WARC file and returns the ARC/WARC record back to the wayback machine to process. The recorded ARC/WARC file becomes part of the wayback machine in due course of time.
Survey Crawl April 2013
collection
16,282
ITEMS
624.3M
VIEWS
collection
eye 624.3M
Survey crawl of domains started April 2013. This data is currently not publicly accessible.
Wide Crawl started April 2013
collection
25,005
ITEMS
448.9M
VIEWS
collection
eye 448.9M
Web wide crawl with initial seedlist and crawler configuration from April 2013.
Wayback Indexes
collection
554
ITEMS
410.9M
VIEWS
collection
eye 410.9M
Wayback indexes. This data is currently not publicly accessible.
Survey Crawl December 2014
collection
13,890
ITEMS
396.3M
VIEWS
collection
eye 396.3M
Survey crawl of domains started December 2014. This data is currently not publicly accessible.
Wide Crawl started June 2014
collection
45,313
ITEMS
316.6M
VIEWS
collection
eye 316.6M
Web wide crawl with initial seedlist and crawler configuration from June 2014.
Wide Crawl Number 12 - started March, 14th 2015
collection
49,621
ITEMS
283.8M
VIEWS
collection
eye 283.8M
Web wide crawl with initial seedlist and crawler configuration from January 2015.
Wide Crawl started August 2013
collection
21,911
ITEMS
281.5M
VIEWS
collection
eye 281.5M
Web wide crawl with initial seedlist and crawler configuration from August 2013.
Wide Crawl started January 2012
collection
30,362
ITEMS
271.5M
VIEWS
collection
eye 271.5M
Web wide crawl with initial seedlist and crawler configuration from January 2012 using HQ software.
Wide Crawl Number 14 started March 2016
collection
71,730
ITEMS
262.2M
VIEWS
collection
eye 262.2M
Web wide crawl.
Wide Crawl started April 2012
collection
39,252
ITEMS
240.9M
VIEWS
collection
eye 240.9M
Web wide crawl with initial seedlist and crawler configuration from April 2012.
Survey Crawl
collection
12,622
ITEMS
240.3M
VIEWS
collection
eye 240.3M
Survey crawl of domains. This data is currently not publicly accessible.
Survey Crawl started July 2015
collection
10,137
ITEMS
222.1M
VIEWS
collection
eye 222.1M
Survey crawl of domains. This data is currently not publicly accessible.
Survey Crawl May 2014
collection
6,909
ITEMS
199.7M
VIEWS
collection
eye 199.7M
Survey crawl of domains started May 2014. This data is currently not publicly accessible.
Wide Crawl started October 2010
collection
15,839
ITEMS
176.1M
VIEWS
collection
eye 176.1M
Web wide crawl with initial seedlist and crawler configuration from October 2010
Wide Crawl Started January 2013
collection
15,138
ITEMS
173.8M
VIEWS
collection
eye 173.8M
Wide crawls of the Internet conducted by Internet Archive. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
Wide Crawl started September 2012
collection
22,402
ITEMS
173.4M
VIEWS
collection
eye 173.4M
Web wide crawl with initial seedlist and crawler configuration from September 2012.
Wide Crawl Number 13
collection
46,049
ITEMS
171.5M
VIEWS
collection
eye 171.5M
Web Wide Crawl Number 13
Wide Crawl started October 2011
collection
12,648
ITEMS
154.7M
VIEWS
collection
eye 154.7M
Web wide crawl with initial seedlist and crawler configuration from March 2011 using HQ software.
Around The World Crawl
collection
2,150
ITEMS
154.6M
VIEWS
collection
eye 154.6M
Data crawled by Sloan Foundation on behalf of Internet Archive
Wide Crawl started February 2014
collection
9,789
ITEMS
143.5M
VIEWS
collection
eye 143.5M
Web wide crawl with initial seedlist and crawler configuration from February 2014.
.com survey started January 2011
collection
2,535
ITEMS
142.5M
VIEWS
collection
eye 142.5M
Survey crawl of .com domains started January 2011.
Topic: webcrawl
Wide Crawl started March 2011
collection
8,528
ITEMS
138.9M
VIEWS
collection
eye 138.9M
Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi. What’s in the data set: Crawl start date: 09 March, 2011 Crawl end date: 23 December, 2011 Number of captures: 2,713,676,341 Number of unique URLs: 2,273,840,159 Number of hosts: 29,032,069 The seed list for this crawl was a list of Alexa’s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT)...
Wikipedia Outlinks March 2016
collection
19,267
ITEMS
128.3M
VIEWS
collection
eye 128.3M
Crawl of outlinks from wikipedia.org started March, 2016. These files are currently not publicly accessible. Properties of this collection. It has been several years since the last time we did this. For this collection, several things were done: 1. Turned off duplicate detection. This collection will be complete, as there is a good chance we will share the data, and sharing data with pointers to random other collections, is a complex problem. 2. For the first time, did all the different wikis....
Wikipedia Outlinks February 2012
collection
2,951
ITEMS
111.3M
VIEWS
collection
eye 111.3M
Crawl of outlinks from wikipedia.org started February, 2012. These files are currently not publicly accessible.
GDELT
collection
27,518
ITEMS
67.6M
VIEWS
collection
eye 67.6M
A daily crawl of more than 200,000 home pages of news sites, including the pages linked from those home pages. Site list provided by The GDELT Project
Topics: GDELT, News
Shallow Crawls
collection
1,042
ITEMS
58.8M
VIEWS
collection
eye 58.8M
Shallow crawls that collect content 1 level deep including embeds. This data is currently not publicly accessible.
International News Crawls
collection
4,271
ITEMS
56.1M
VIEWS
collection
eye 56.1M
Crawls of International News Sites
Wikipedia Outlinks May 2011
collection
1,638
ITEMS
50.9M
VIEWS
collection
eye 50.9M
Crawl of outlinks from wikipedia.org started May, 2011. These files are currently not publicly accessible.
Wide Crawl Number 15
collection
38,757
ITEMS
49M
VIEWS
collection
eye 49M
Web wide crawl.
Wikipedia Outlinks July 2011
collection
1,011
ITEMS
35.7M
VIEWS
collection
eye 35.7M
Crawl of outlinks from wikipedia.org started July, 2011. These files are currently not publicly accessible.
Hackernews crawl number 0
collection
6,154
ITEMS
28.3M
VIEWS
collection
eye 28.3M
Hacker News Crawl of their links.
End of Term Web Crawls
collection
13,846
ITEMS
28.2M
VIEWS
by IA
collection
eye 28.2M
This collection includes web crawls of the Federal Executive, Legislative, and Judicial branches of government performed at the end of US presidential terms of office.
Topics: web, end of term, US, federal government
COM Survey Crawl 2009-2010
collection
729
ITEMS
24.7M
VIEWS
collection
eye 24.7M
COM survey crawl data collected by Internet Archive in 2009-2010. This data is currently not publicly accessible.
Geocities Closing Crawl
collection
149
ITEMS
24.2M
VIEWS
collection
eye 24.2M
Geocities crawl performed by Internet Archive. This data is currently not publicly accessible. from Wikipedia : Yahoo! GeoCities is a Web hosting service. GeoCities was originally founded by David Bohnett and John Rezner in late 1994 as Beverly Hills Internet (BHI), and by 1999 GeoCities was the third-most visited Web site on the World Wide Web. In its original form, site users selected a "city" in which to place their Web pages. The "cities" were metonymously named after...
Wayback CDX Shards
collection
1,200
ITEMS
24M
VIEWS
collection
eye 24M
CDX Index shards for the Wayback Machine. The Wayback Machine works by looking for historic URL's based on a query. This is done by searching an index of all the web objects (pages, images, etc) that have been archived over the years. This collection holds the index used for this purpose, which is broken up into 300 pieces so they fit into items more naturally and distribute the lookup load. Each of these 300 pieces is stored in at least 2 items, and then those are also stored on the backup...
Shallow Crawl Started 2013
collection
544
ITEMS
22.5M
VIEWS
collection
eye 22.5M
Shallow crawl started 2013 that collects content 1 level deep, including embeds. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
Shallow Crawl Started 2013
collection
252
ITEMS
22.2M
VIEWS
collection
eye 22.2M
Shallow crawl started 2013 that collects content 1 level deep, including embeds. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
nsdlweb
collection
91
ITEMS
19.3M
VIEWS
collection
eye 19.3M
this data is currently not publicly accessible.
survey_net00000
collection
300
ITEMS
17.2M
VIEWS
collection
eye 17.2M
Survey crawl of .net domains started December 2010.
Topic: webcrawl
Collections news crawls v2
collection
820
ITEMS
15.1M
VIEWS
by ximm@archive.org
collection
eye 15.1M
Collections News Crawls v3
collection
2,131
ITEMS
14.3M
VIEWS
by ximm@archive.org
collection
eye 14.3M
Miscellaneous high-value news sites 
Topics: World news, US news, news
End of Term 2012 Web Crawls
collection
2,381
ITEMS
14.3M
VIEWS
collection
eye 14.3M
This collection contains web crawls performed on the US Federal Executive, Legislative & Judicial branches of government in 2012-2013.
Topics: end of term, US, Federal government, 2012, Obama
Youtube Videos
collection
281,656
ITEMS
13.6M
VIEWS
collection
eye 13.6M
Captures of pages from YouTube. Currently these are discovered by searching for YouTube links on Twitter.
Topics: YouTube, Twitter, Video
collection
eye 12M
Crawl of International News Sites with initial seedlist and crawler configuration from Sep 1, 2010.
Survey Crawl
collection
1,787
ITEMS
11.9M
VIEWS
collection
eye 11.9M
Survey crawl of domains. This data is currently not publicly accessible.
ORG Survey Crawls
collection
191
ITEMS
10.5M
VIEWS
collection
eye 10.5M
Survey of .org domains. This data is currently not publicly accessible.
Live Web Proxy Crawls
web
eye 9.5M
favorite 0
comment 0
Internet Archive Liveweb Capture from WaybackMachine, captured by wwwb-proxy0.us.archive.org:wbm from Sun Mar 27 22:10:09 PDT 2011 to Mon Mar 28 05:27:05 PDT 2011.
Topic: crawldata
Shallow Crawl Started February 2013
collection
86
ITEMS
9.1M
VIEWS
collection
eye 9.1M
Shallow crawl started February 2013 that collects content 1 level deep, including embeds. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.
survey_net00001
collection
165
ITEMS
7.3M
VIEWS
collection
eye 7.3M
Survey crawl of .net domains started October 2011.
Topics: webwidecrawl, net
Edu & Gov Crawl, June 2010
collection
704
ITEMS
6.5M
VIEWS
collection
eye 6.5M
TEST COLLECTION: Crawl of .edu and .gov sites started in June 2010.
Topic: crawldata
collection
eye 6.1M
End of term 2008 crawl data gathered by Internet Archive on behalf of the California Digital Library. This data is currently not publicly accessible.
2004 Election
collection
178
ITEMS
5.2M
VIEWS
collection
eye 5.2M
2004 Election crawl performed by Internet Archive. This data is currently not publicly accessible.
Live Web Proxy Crawls
web
eye 5.2M
favorite 0
comment 0
Crawl Data
collection
32,956
ITEMS
5.2M
VIEWS
collection
eye 5.2M
Crawl Data. This data is currently not publicly accessible.
crawl_UNK
collection
32,949
ITEMS
5.1M
VIEWS
collection
eye 5.1M
Crawl data. This data is currently not publicly accessible.
World Wars Crawl
collection
13
ITEMS
5M
VIEWS
collection
eye 5M
Web data related to World Wars I and II collected by Internet Archive in an experimental crawl sponsored by National Endowment for the Humanities and JISC. This data is currently not publicly accessible.
Wide Crawl started September 2010
collection
332
ITEMS
4.9M
VIEWS
collection
eye 4.9M
Web wide crawl with initial seedlist and crawler configuration from September 2010
YouTube 2007 Crawl
collection
73,846
ITEMS
4.4M
VIEWS
collection
eye 4.4M
Data crawled from YouTube.com in 2007 by Internet Archive. These files are not currently accessible.
Wide Crawl started January 2012
web
eye 4.2M
favorite 0
comment 0
Internet Archive crawldata from Webwide Crawl, captured by crawl423.us.archive.org:wide from Tue Jan 17 08:02:53 PST 2012 to Tue Jan 17 01:16:20 PST 2012.
Topic: crawldata
Shallow Crawl Started November 2012
collection
137
ITEMS
3.9M
VIEWS
collection
eye 3.9M
Shallow crawl started November 2012 that collects content 1 level deep, including embeds. This data is currently not publicly accessible.
Live Web Proxy Crawls
web
eye 3.7M
favorite 0
comment 0
Internet Archive Liveweb Capture from WaybackMachine, captured by wwwb-proxy0.us.archive.org:wbm from Sun May 8 07:07:52 PDT 2011 to Sun May 8 08:00:29 PDT 2011.
Topic: crawldata
Hurricane Katrina
collection
111
ITEMS
3.7M
VIEWS
collection
eye 3.7M
Data related to Hurricane Katrina collected in 2005 by Internet Archive. This data is currently not publicly accessible. from Wikipedia : Hurricane Katrina was the deadliest and most destructive Atlantic hurricane of the 2005 Atlantic hurricane season. It was the costliest natural disaster, as well as one of the five deadliest hurricanes, in the history of the United States. Among recorded Atlantic hurricanes, it was the sixth strongest overall. At least 1,833 people died in the hurricane and...
Data crawled by Internet Archive on behalf of Internet Archive from Fri Nov 01 06:23:33 PDT 2002 to Tue Nov 19 23:24:07 PDT 2002
Topic: crawldata
End of Term 2008 UNT Dot Gov Crawl
collection
281
ITEMS
3.6M
VIEWS
collection
eye 3.6M
End of term 2008 crawl of .gov domains gathered by University of North Texas . This data is currently not publicly accessible. UNT is a student-focused, public, research university located in Denton, Texas.
Data crawled by National Endowment for the Humanities and JISC on behalf of Internet Archive from Fri Aug 08 00:17:40 PDT 2008 to Thu Jun 26 05:29:33 PDT 2008
Topic: crawldata
Live Web Proxy Crawls
web
eye 3.4M
favorite 0
comment 0
Internet Archive Liveweb Capture from WaybackMachine, captured by wwwb-proxy0.us.archive.org:wbm from Mon Mar 28 12:43:47 PDT 2011 to Mon Mar 28 16:56:17 PDT 2011.
Topic: crawldata
Live Web Proxy Crawls
web
eye 3M
favorite 0
comment 0
Internet Archive Liveweb Capture from WaybackMachine, captured by wwwb-proxy0.us.archive.org:wbm from Fri May 20 00:54:34 PDT 2011 to Fri May 20 04:55:47 PDT 2011.
Topic: crawldata
Live Web Proxy Crawls
web
eye 2.9M
favorite 0
comment 0
Internet Archive Liveweb Capture from WaybackMachine, captured by wwwb-proxy0.us.archive.org:wbm from Thu May 19 17:19:06 PDT 2011 to Thu May 19 17:46:28 PDT 2011.
Topic: crawldata
Live Web Proxy Crawls
web
eye 2.8M
favorite 0
comment 0
Internet Archive Liveweb Capture from WaybackMachine, captured by wwwb-proxy0.us.archive.org:wbm from Tue Mar 29 00:12:24 PDT 2011 to Tue Mar 29 07:24:41 PDT 2011.
Topic: crawldata
Live Web Proxy Crawls
web
eye 2.7M
favorite 0
comment 0
Internet Archive Liveweb Capture from WayBack Machine, captured by wwwb-live1.us.archive.org from 2013-03-29T09:54:55 UTC to 2013-03-29T13:00:26 UTC.
Topic: crawldata
Live Web Proxy Crawls
web
eye 2.6M
favorite 0
comment 0
Internet Archive Liveweb Capture from WaybackMachine, captured by wwwb-proxy0.us.archive.org:wbm from Wed Mar 30 03:11:49 PDT 2011 to Wed Mar 30 10:24:04 PDT 2011.
Topic: crawldata
Live Web Proxy Crawls
web
eye 2.5M
favorite 0
comment 0
Internet Archive Liveweb Capture from WayBack Machine, captured by wwwb-live0.us.archive.org from 2013-03-29T09:35:58 UTC to 2013-03-29T15:03:34 UTC.
Topic: crawldata