Common Crawl Index Server

Fork me on GitHub

Please see the CDX Server API Reference for more examples on how to use the query api. Alternatively, you may use the command-line tools based on this API: Ilya Kreymer's Common Crawl Index Client or Greg Lindahl's cdx-toolkit.

Common Crawl data is stored on Amazon Web Services' Public Data Sets. All data and index files are free to download — run your own index server or analyze the index offline!
Please do not overload the URL index server for bulk downloads (e.g. all records of the entire .com top-level domain), see the download instructions. Alternatively, check the columnar index.
More information about this URL index is found in our announcement of the Common Crawl index. For help and support, please visit the Common Crawl user forum.

Currently available index collections (also as JSON list):

Search Page Crawl API endpoint Index File List on
s3://commoncrawl/
/CC-MAIN-2018-51 December 2018 Index /CC-MAIN-2018-51-index CC-MAIN-2018-51/cc-index.paths.gz
/CC-MAIN-2018-47 November 2018 Index /CC-MAIN-2018-47-index CC-MAIN-2018-47/cc-index.paths.gz
/CC-MAIN-2018-43 October 2018 Index /CC-MAIN-2018-43-index CC-MAIN-2018-43/cc-index.paths.gz
/CC-MAIN-2018-39 September 2018 Index /CC-MAIN-2018-39-index CC-MAIN-2018-39/cc-index.paths.gz
/CC-MAIN-2018-34 August 2018 Index /CC-MAIN-2018-34-index CC-MAIN-2018-34/cc-index.paths.gz
/CC-MAIN-2018-30 July 2018 Index /CC-MAIN-2018-30-index CC-MAIN-2018-30/cc-index.paths.gz
/CC-MAIN-2018-26 June 2018 Index /CC-MAIN-2018-26-index CC-MAIN-2018-26/cc-index.paths.gz
/CC-MAIN-2018-22 May 2018 Index /CC-MAIN-2018-22-index CC-MAIN-2018-22/cc-index.paths.gz
/CC-MAIN-2018-17 April 2018 Index /CC-MAIN-2018-17-index CC-MAIN-2018-17/cc-index.paths.gz
/CC-MAIN-2018-13 March 2018 Index /CC-MAIN-2018-13-index CC-MAIN-2018-13/cc-index.paths.gz
/CC-MAIN-2018-09 February 2018 Index /CC-MAIN-2018-09-index CC-MAIN-2018-09/cc-index.paths.gz
/CC-MAIN-2018-05 January 2018 Index /CC-MAIN-2018-05-index CC-MAIN-2018-05/cc-index.paths.gz
/CC-MAIN-2017-51 December 2017 Index /CC-MAIN-2017-51-index CC-MAIN-2017-51/cc-index.paths.gz
/CC-MAIN-2017-47 November 2017 Index /CC-MAIN-2017-47-index CC-MAIN-2017-47/cc-index.paths.gz
/CC-MAIN-2017-43 October 2017 Index /CC-MAIN-2017-43-index CC-MAIN-2017-43/cc-index.paths.gz
/CC-MAIN-2017-39 September 2017 Index /CC-MAIN-2017-39-index CC-MAIN-2017-39/cc-index.paths.gz
/CC-MAIN-2017-34 August 2017 Index /CC-MAIN-2017-34-index CC-MAIN-2017-34/cc-index.paths.gz
/CC-MAIN-2017-30 July 2017 Index /CC-MAIN-2017-30-index CC-MAIN-2017-30/cc-index.paths.gz
/CC-MAIN-2017-26 June 2017 Index /CC-MAIN-2017-26-index CC-MAIN-2017-26/cc-index.paths.gz
/CC-MAIN-2017-22 May 2017 Index /CC-MAIN-2017-22-index CC-MAIN-2017-22/cc-index.paths.gz
/CC-MAIN-2017-17 April 2017 Index /CC-MAIN-2017-17-index CC-MAIN-2017-17/cc-index.paths.gz
/CC-MAIN-2017-13 March 2017 Index /CC-MAIN-2017-13-index CC-MAIN-2017-13/cc-index.paths.gz
/CC-MAIN-2017-09 February 2017 Index /CC-MAIN-2017-09-index CC-MAIN-2017-09/cc-index.paths.gz
/CC-MAIN-2017-04 January 2017 Index /CC-MAIN-2017-04-index CC-MAIN-2017-04/cc-index.paths.gz
/CC-MAIN-2016-50 December 2016 Index /CC-MAIN-2016-50-index CC-MAIN-2016-50/cc-index.paths.gz
/CC-MAIN-2016-44 October 2016 Index /CC-MAIN-2016-44-index CC-MAIN-2016-44/cc-index.paths.gz
/CC-MAIN-2016-40 September 2016 Index /CC-MAIN-2016-40-index CC-MAIN-2016-40/cc-index.paths.gz
/CC-MAIN-2016-36 August 2016 Index /CC-MAIN-2016-36-index CC-MAIN-2016-36/cc-index.paths.gz
/CC-MAIN-2016-30 July 2016 Index /CC-MAIN-2016-30-index CC-MAIN-2016-30/cc-index.paths.gz
/CC-MAIN-2016-26 June 2016 Index /CC-MAIN-2016-26-index CC-MAIN-2016-26/cc-index.paths.gz
/CC-MAIN-2016-22 May 2016 Index /CC-MAIN-2016-22-index CC-MAIN-2016-22/cc-index.paths.gz
/CC-MAIN-2016-18 April 2016 Index /CC-MAIN-2016-18-index CC-MAIN-2016-18/cc-index.paths.gz
/CC-MAIN-2016-07 February 2016 Index /CC-MAIN-2016-07-index CC-MAIN-2016-07/cc-index.paths.gz
/CC-MAIN-2015-48 November 2015 Index /CC-MAIN-2015-48-index CC-MAIN-2015-48/cc-index.paths.gz
/CC-MAIN-2015-40 September 2015 Index /CC-MAIN-2015-40-index CC-MAIN-2015-40/cc-index.paths.gz
/CC-MAIN-2015-35 August 2015 Index /CC-MAIN-2015-35-index CC-MAIN-2015-35/cc-index.paths.gz
/CC-MAIN-2015-32 July 2015 Index /CC-MAIN-2015-32-index CC-MAIN-2015-32/cc-index.paths.gz
/CC-MAIN-2015-27 June 2015 Index /CC-MAIN-2015-27-index CC-MAIN-2015-27/cc-index.paths.gz
/CC-MAIN-2015-22 May 2015 Index /CC-MAIN-2015-22-index CC-MAIN-2015-22/cc-index.paths.gz
/CC-MAIN-2015-18 April 2015 Index /CC-MAIN-2015-18-index CC-MAIN-2015-18/cc-index.paths.gz
/CC-MAIN-2015-14 March 2015 Index /CC-MAIN-2015-14-index CC-MAIN-2015-14/cc-index.paths.gz
/CC-MAIN-2015-11 February 2015 Index /CC-MAIN-2015-11-index CC-MAIN-2015-11/cc-index.paths.gz
/CC-MAIN-2015-06 January 2015 Index /CC-MAIN-2015-06-index CC-MAIN-2015-06/cc-index.paths.gz
/CC-MAIN-2014-52 December 2014 Index /CC-MAIN-2014-52-index CC-MAIN-2014-52/cc-index.paths.gz
/CC-MAIN-2014-49 November 2014 Index /CC-MAIN-2014-49-index CC-MAIN-2014-49/cc-index.paths.gz
/CC-MAIN-2014-42 October 2014 Index /CC-MAIN-2014-42-index CC-MAIN-2014-42/cc-index.paths.gz
/CC-MAIN-2014-41 September 2014 Index /CC-MAIN-2014-41-index CC-MAIN-2014-41/cc-index.paths.gz
/CC-MAIN-2014-35 August 2014 Index /CC-MAIN-2014-35-index CC-MAIN-2014-35/cc-index.paths.gz
/CC-MAIN-2014-23 July 2014 Index /CC-MAIN-2014-23-index CC-MAIN-2014-23/cc-index.paths.gz
/CC-MAIN-2014-15 April 2014 Index /CC-MAIN-2014-15-index CC-MAIN-2014-15/cc-index.paths.gz
/CC-MAIN-2014-10 March 2014 Index /CC-MAIN-2014-10-index CC-MAIN-2014-10/cc-index.paths.gz
/CC-MAIN-2013-48 Winter 2013 Index /CC-MAIN-2013-48-index CC-MAIN-2013-48/cc-index.paths.gz
/CC-MAIN-2013-20 Summer 2013 Index /CC-MAIN-2013-20-index CC-MAIN-2013-20/cc-index.paths.gz

Powered by pywb