Common Crawl Index Server

Fork me on GitHub

Please see the PyWB CDX Server API Reference for more examples on how to use the query API (please replace the API endpoint coll/cdx by one of the API endpoints listed in the table below). Alternatively, you may use one of the command-line tools based on this API: Ilya Kreymer's Common Crawl Index Client, Greg Lindahl's cdx-toolkit or Corben Leo's getallurls (gau).

Common Crawl data is stored on Amazon Web Services' Public Data Sets. All data and index files are free to download — run your own index server or analyze the index offline!
Please do not overload the URL index server for bulk downloads (e.g. all records of the entire .com top-level domain), see the download instructions. Alternatively, check the columnar index which allows for efficient aggregations and filtering on any field/column.
More information about this URL index is found in our announcement of the Common Crawl index. For help and support, please visit the Common Crawl user forum.

Currently available index collections (also as JSON list):

Search Page Crawl API endpoint Index File List on
s3://commoncrawl/
/CC-MAIN-2021-31 July 2021 /CC-MAIN-2021-31-index CC-MAIN-2021-31/cc-index.paths.gz
/CC-MAIN-2021-25 June 2021 /CC-MAIN-2021-25-index CC-MAIN-2021-25/cc-index.paths.gz
/CC-MAIN-2021-21 May 2021 /CC-MAIN-2021-21-index CC-MAIN-2021-21/cc-index.paths.gz
/CC-MAIN-2021-17 April 2021 /CC-MAIN-2021-17-index CC-MAIN-2021-17/cc-index.paths.gz
/CC-MAIN-2021-10 February 2021 /CC-MAIN-2021-10-index CC-MAIN-2021-10/cc-index.paths.gz
/CC-MAIN-2021-04 January 2021 /CC-MAIN-2021-04-index CC-MAIN-2021-04/cc-index.paths.gz
/CC-MAIN-2020-50 November 2020 /CC-MAIN-2020-50-index CC-MAIN-2020-50/cc-index.paths.gz
/CC-MAIN-2020-45 October 2020 /CC-MAIN-2020-45-index CC-MAIN-2020-45/cc-index.paths.gz
/CC-MAIN-2020-40 September 2020 /CC-MAIN-2020-40-index CC-MAIN-2020-40/cc-index.paths.gz
/CC-MAIN-2020-34 August 2020 /CC-MAIN-2020-34-index CC-MAIN-2020-34/cc-index.paths.gz
/CC-MAIN-2020-29 July 2020 /CC-MAIN-2020-29-index CC-MAIN-2020-29/cc-index.paths.gz
/CC-MAIN-2020-24 May 2020 /CC-MAIN-2020-24-index CC-MAIN-2020-24/cc-index.paths.gz
/CC-MAIN-2020-16 March 2020 /CC-MAIN-2020-16-index CC-MAIN-2020-16/cc-index.paths.gz
/CC-MAIN-2020-10 February 2020 /CC-MAIN-2020-10-index CC-MAIN-2020-10/cc-index.paths.gz
/CC-MAIN-2020-05 January 2020 /CC-MAIN-2020-05-index CC-MAIN-2020-05/cc-index.paths.gz
/CC-MAIN-2019-51 December 2019 /CC-MAIN-2019-51-index CC-MAIN-2019-51/cc-index.paths.gz
/CC-MAIN-2019-47 November 2019 /CC-MAIN-2019-47-index CC-MAIN-2019-47/cc-index.paths.gz
/CC-MAIN-2019-43 October 2019 /CC-MAIN-2019-43-index CC-MAIN-2019-43/cc-index.paths.gz
/CC-MAIN-2019-39 September 2019 /CC-MAIN-2019-39-index CC-MAIN-2019-39/cc-index.paths.gz
/CC-MAIN-2019-35 August 2019 /CC-MAIN-2019-35-index CC-MAIN-2019-35/cc-index.paths.gz
/CC-MAIN-2019-30 July 2019 /CC-MAIN-2019-30-index CC-MAIN-2019-30/cc-index.paths.gz
/CC-MAIN-2019-26 June 2019 /CC-MAIN-2019-26-index CC-MAIN-2019-26/cc-index.paths.gz
/CC-MAIN-2019-22 May 2019 /CC-MAIN-2019-22-index CC-MAIN-2019-22/cc-index.paths.gz
/CC-MAIN-2019-18 April 2019 /CC-MAIN-2019-18-index CC-MAIN-2019-18/cc-index.paths.gz
/CC-MAIN-2019-13 March 2019 /CC-MAIN-2019-13-index CC-MAIN-2019-13/cc-index.paths.gz
/CC-MAIN-2019-09 February 2019 /CC-MAIN-2019-09-index CC-MAIN-2019-09/cc-index.paths.gz
/CC-MAIN-2019-04 January 2019 /CC-MAIN-2019-04-index CC-MAIN-2019-04/cc-index.paths.gz
/CC-MAIN-2018-51 December 2018 /CC-MAIN-2018-51-index CC-MAIN-2018-51/cc-index.paths.gz
/CC-MAIN-2018-47 November 2018 /CC-MAIN-2018-47-index CC-MAIN-2018-47/cc-index.paths.gz
/CC-MAIN-2018-43 October 2018 /CC-MAIN-2018-43-index CC-MAIN-2018-43/cc-index.paths.gz
/CC-MAIN-2018-39 September 2018 /CC-MAIN-2018-39-index CC-MAIN-2018-39/cc-index.paths.gz
/CC-MAIN-2018-34 August 2018 /CC-MAIN-2018-34-index CC-MAIN-2018-34/cc-index.paths.gz
/CC-MAIN-2018-30 July 2018 /CC-MAIN-2018-30-index CC-MAIN-2018-30/cc-index.paths.gz
/CC-MAIN-2018-26 June 2018 /CC-MAIN-2018-26-index CC-MAIN-2018-26/cc-index.paths.gz
/CC-MAIN-2018-22 May 2018 /CC-MAIN-2018-22-index CC-MAIN-2018-22/cc-index.paths.gz
/CC-MAIN-2018-17 April 2018 /CC-MAIN-2018-17-index CC-MAIN-2018-17/cc-index.paths.gz
/CC-MAIN-2018-13 March 2018 /CC-MAIN-2018-13-index CC-MAIN-2018-13/cc-index.paths.gz
/CC-MAIN-2018-09 February 2018 /CC-MAIN-2018-09-index CC-MAIN-2018-09/cc-index.paths.gz
/CC-MAIN-2018-05 January 2018 /CC-MAIN-2018-05-index CC-MAIN-2018-05/cc-index.paths.gz
/CC-MAIN-2017-51 December 2017 /CC-MAIN-2017-51-index CC-MAIN-2017-51/cc-index.paths.gz
/CC-MAIN-2017-47 November 2017 /CC-MAIN-2017-47-index CC-MAIN-2017-47/cc-index.paths.gz
/CC-MAIN-2017-43 October 2017 /CC-MAIN-2017-43-index CC-MAIN-2017-43/cc-index.paths.gz
/CC-MAIN-2017-39 September 2017 /CC-MAIN-2017-39-index CC-MAIN-2017-39/cc-index.paths.gz
/CC-MAIN-2017-34 August 2017 /CC-MAIN-2017-34-index CC-MAIN-2017-34/cc-index.paths.gz
/CC-MAIN-2017-30 July 2017 /CC-MAIN-2017-30-index CC-MAIN-2017-30/cc-index.paths.gz
/CC-MAIN-2017-26 June 2017 /CC-MAIN-2017-26-index CC-MAIN-2017-26/cc-index.paths.gz
/CC-MAIN-2017-22 May 2017 /CC-MAIN-2017-22-index CC-MAIN-2017-22/cc-index.paths.gz
/CC-MAIN-2017-17 April 2017 /CC-MAIN-2017-17-index CC-MAIN-2017-17/cc-index.paths.gz
/CC-MAIN-2017-13 March 2017 /CC-MAIN-2017-13-index CC-MAIN-2017-13/cc-index.paths.gz
/CC-MAIN-2017-09 February 2017 /CC-MAIN-2017-09-index CC-MAIN-2017-09/cc-index.paths.gz
/CC-MAIN-2017-04 January 2017 /CC-MAIN-2017-04-index CC-MAIN-2017-04/cc-index.paths.gz
/CC-MAIN-2016-50 December 2016 /CC-MAIN-2016-50-index CC-MAIN-2016-50/cc-index.paths.gz
/CC-MAIN-2016-44 October 2016 /CC-MAIN-2016-44-index CC-MAIN-2016-44/cc-index.paths.gz
/CC-MAIN-2016-40 September 2016 /CC-MAIN-2016-40-index CC-MAIN-2016-40/cc-index.paths.gz
/CC-MAIN-2016-36 August 2016 /CC-MAIN-2016-36-index CC-MAIN-2016-36/cc-index.paths.gz
/CC-MAIN-2016-30 July 2016 /CC-MAIN-2016-30-index CC-MAIN-2016-30/cc-index.paths.gz
/CC-MAIN-2016-26 June 2016 /CC-MAIN-2016-26-index CC-MAIN-2016-26/cc-index.paths.gz
/CC-MAIN-2016-22 May 2016 /CC-MAIN-2016-22-index CC-MAIN-2016-22/cc-index.paths.gz
/CC-MAIN-2016-18 April 2016 /CC-MAIN-2016-18-index CC-MAIN-2016-18/cc-index.paths.gz
/CC-MAIN-2016-07 February 2016 /CC-MAIN-2016-07-index CC-MAIN-2016-07/cc-index.paths.gz
/CC-MAIN-2015-48 November 2015 /CC-MAIN-2015-48-index CC-MAIN-2015-48/cc-index.paths.gz
/CC-MAIN-2015-40 September 2015 /CC-MAIN-2015-40-index CC-MAIN-2015-40/cc-index.paths.gz
/CC-MAIN-2015-35 August 2015 /CC-MAIN-2015-35-index CC-MAIN-2015-35/cc-index.paths.gz
/CC-MAIN-2015-32 July 2015 /CC-MAIN-2015-32-index CC-MAIN-2015-32/cc-index.paths.gz
/CC-MAIN-2015-27 June 2015 /CC-MAIN-2015-27-index CC-MAIN-2015-27/cc-index.paths.gz
/CC-MAIN-2015-22 May 2015 /CC-MAIN-2015-22-index CC-MAIN-2015-22/cc-index.paths.gz
/CC-MAIN-2015-18 April 2015 /CC-MAIN-2015-18-index CC-MAIN-2015-18/cc-index.paths.gz
/CC-MAIN-2015-14 March 2015 /CC-MAIN-2015-14-index CC-MAIN-2015-14/cc-index.paths.gz
/CC-MAIN-2015-11 February 2015 /CC-MAIN-2015-11-index CC-MAIN-2015-11/cc-index.paths.gz
/CC-MAIN-2015-06 January 2015 /CC-MAIN-2015-06-index CC-MAIN-2015-06/cc-index.paths.gz
/CC-MAIN-2014-52 December 2014 /CC-MAIN-2014-52-index CC-MAIN-2014-52/cc-index.paths.gz
/CC-MAIN-2014-49 November 2014 /CC-MAIN-2014-49-index CC-MAIN-2014-49/cc-index.paths.gz
/CC-MAIN-2014-42 October 2014 /CC-MAIN-2014-42-index CC-MAIN-2014-42/cc-index.paths.gz
/CC-MAIN-2014-41 September 2014 /CC-MAIN-2014-41-index CC-MAIN-2014-41/cc-index.paths.gz
/CC-MAIN-2014-35 August 2014 /CC-MAIN-2014-35-index CC-MAIN-2014-35/cc-index.paths.gz
/CC-MAIN-2014-23 July 2014 /CC-MAIN-2014-23-index CC-MAIN-2014-23/cc-index.paths.gz
/CC-MAIN-2014-15 April 2014 /CC-MAIN-2014-15-index CC-MAIN-2014-15/cc-index.paths.gz
/CC-MAIN-2014-10 March 2014 /CC-MAIN-2014-10-index CC-MAIN-2014-10/cc-index.paths.gz
/CC-MAIN-2013-48 Winter 2013 /CC-MAIN-2013-48-index CC-MAIN-2013-48/cc-index.paths.gz
/CC-MAIN-2013-20 Summer 2013 /CC-MAIN-2013-20-index CC-MAIN-2013-20/cc-index.paths.gz
/CC-MAIN-2012 Index of 2012 ARC files /CC-MAIN-2012-index CC-MAIN-2012/cc-index.paths.gz
/CC-MAIN-2009-2010 Index of 2009 - 2010 ARC files /CC-MAIN-2009-2010-index CC-MAIN-2009-2010/cc-index.paths.gz
/CC-MAIN-2008-2009 Index of 2008 - 2009 ARC files /CC-MAIN-2008-2009-index CC-MAIN-2008-2009/cc-index.paths.gz