Common Crawl Index Server

Fork me on GitHub

Please see the CDX Server API Reference for more examples on how to use the query api. Alternatively, you may use the command-line tools based on this API: Ilya Kreymer's Common Crawl Index Client or Greg Lindahl's cdx-toolkit.

Common Crawl data is stored on Amazon Web Services' Public Data Sets. All data and index files are free to download — run your own index server or analyze the index offline!
Please do not overload the URL index server for bulk downloads (e.g. all records of the entire .com top-level domain), see the download instructions. Alternatively, check the columnar index which allows for efficient aggregations and filtering on any field/column.
More information about this URL index is found in our announcement of the Common Crawl index. For help and support, please visit the Common Crawl user forum.

Currently available index collections (also as JSON list):

Search Page Crawl API endpoint Index File List on
s3://commoncrawl/
/CC-MAIN-2020-50 November 2020 Index /CC-MAIN-2020-50-index CC-MAIN-2020-50/cc-index.paths.gz
/CC-MAIN-2020-45 October 2020 Index /CC-MAIN-2020-45-index CC-MAIN-2020-45/cc-index.paths.gz
/CC-MAIN-2020-40 September 2020 Index /CC-MAIN-2020-40-index CC-MAIN-2020-40/cc-index.paths.gz
/CC-MAIN-2020-34 August 2020 Index /CC-MAIN-2020-34-index CC-MAIN-2020-34/cc-index.paths.gz
/CC-MAIN-2020-29 July 2020 Index /CC-MAIN-2020-29-index CC-MAIN-2020-29/cc-index.paths.gz
/CC-MAIN-2020-24 May 2020 Index /CC-MAIN-2020-24-index CC-MAIN-2020-24/cc-index.paths.gz
/CC-MAIN-2020-16 March 2020 Index /CC-MAIN-2020-16-index CC-MAIN-2020-16/cc-index.paths.gz
/CC-MAIN-2020-10 February 2020 Index /CC-MAIN-2020-10-index CC-MAIN-2020-10/cc-index.paths.gz
/CC-MAIN-2020-05 January 2020 Index /CC-MAIN-2020-05-index CC-MAIN-2020-05/cc-index.paths.gz
/CC-MAIN-2019-51 December 2019 Index /CC-MAIN-2019-51-index CC-MAIN-2019-51/cc-index.paths.gz
/CC-MAIN-2019-47 November 2019 Index /CC-MAIN-2019-47-index CC-MAIN-2019-47/cc-index.paths.gz
/CC-MAIN-2019-43 October 2019 Index /CC-MAIN-2019-43-index CC-MAIN-2019-43/cc-index.paths.gz
/CC-MAIN-2019-39 September 2019 Index /CC-MAIN-2019-39-index CC-MAIN-2019-39/cc-index.paths.gz
/CC-MAIN-2019-35 August 2019 Index /CC-MAIN-2019-35-index CC-MAIN-2019-35/cc-index.paths.gz
/CC-MAIN-2019-30 July 2019 Index /CC-MAIN-2019-30-index CC-MAIN-2019-30/cc-index.paths.gz
/CC-MAIN-2019-26 June 2019 Index /CC-MAIN-2019-26-index CC-MAIN-2019-26/cc-index.paths.gz
/CC-MAIN-2019-22 May 2019 Index /CC-MAIN-2019-22-index CC-MAIN-2019-22/cc-index.paths.gz
/CC-MAIN-2019-18 April 2019 Index /CC-MAIN-2019-18-index CC-MAIN-2019-18/cc-index.paths.gz
/CC-MAIN-2019-13 March 2019 Index /CC-MAIN-2019-13-index CC-MAIN-2019-13/cc-index.paths.gz
/CC-MAIN-2019-09 February 2019 Index /CC-MAIN-2019-09-index CC-MAIN-2019-09/cc-index.paths.gz
/CC-MAIN-2019-04 January 2019 Index /CC-MAIN-2019-04-index CC-MAIN-2019-04/cc-index.paths.gz
/CC-MAIN-2018-51 December 2018 Index /CC-MAIN-2018-51-index CC-MAIN-2018-51/cc-index.paths.gz
/CC-MAIN-2018-47 November 2018 Index /CC-MAIN-2018-47-index CC-MAIN-2018-47/cc-index.paths.gz
/CC-MAIN-2018-43 October 2018 Index /CC-MAIN-2018-43-index CC-MAIN-2018-43/cc-index.paths.gz
/CC-MAIN-2018-39 September 2018 Index /CC-MAIN-2018-39-index CC-MAIN-2018-39/cc-index.paths.gz
/CC-MAIN-2018-34 August 2018 Index /CC-MAIN-2018-34-index CC-MAIN-2018-34/cc-index.paths.gz
/CC-MAIN-2018-30 July 2018 Index /CC-MAIN-2018-30-index CC-MAIN-2018-30/cc-index.paths.gz
/CC-MAIN-2018-26 June 2018 Index /CC-MAIN-2018-26-index CC-MAIN-2018-26/cc-index.paths.gz
/CC-MAIN-2018-22 May 2018 Index /CC-MAIN-2018-22-index CC-MAIN-2018-22/cc-index.paths.gz
/CC-MAIN-2018-17 April 2018 Index /CC-MAIN-2018-17-index CC-MAIN-2018-17/cc-index.paths.gz
/CC-MAIN-2018-13 March 2018 Index /CC-MAIN-2018-13-index CC-MAIN-2018-13/cc-index.paths.gz
/CC-MAIN-2018-09 February 2018 Index /CC-MAIN-2018-09-index CC-MAIN-2018-09/cc-index.paths.gz
/CC-MAIN-2018-05 January 2018 Index /CC-MAIN-2018-05-index CC-MAIN-2018-05/cc-index.paths.gz
/CC-MAIN-2017-51 December 2017 Index /CC-MAIN-2017-51-index CC-MAIN-2017-51/cc-index.paths.gz
/CC-MAIN-2017-47 November 2017 Index /CC-MAIN-2017-47-index CC-MAIN-2017-47/cc-index.paths.gz
/CC-MAIN-2017-43 October 2017 Index /CC-MAIN-2017-43-index CC-MAIN-2017-43/cc-index.paths.gz
/CC-MAIN-2017-39 September 2017 Index /CC-MAIN-2017-39-index CC-MAIN-2017-39/cc-index.paths.gz
/CC-MAIN-2017-34 August 2017 Index /CC-MAIN-2017-34-index CC-MAIN-2017-34/cc-index.paths.gz
/CC-MAIN-2017-30 July 2017 Index /CC-MAIN-2017-30-index CC-MAIN-2017-30/cc-index.paths.gz
/CC-MAIN-2017-26 June 2017 Index /CC-MAIN-2017-26-index CC-MAIN-2017-26/cc-index.paths.gz
/CC-MAIN-2017-22 May 2017 Index /CC-MAIN-2017-22-index CC-MAIN-2017-22/cc-index.paths.gz
/CC-MAIN-2017-17 April 2017 Index /CC-MAIN-2017-17-index CC-MAIN-2017-17/cc-index.paths.gz
/CC-MAIN-2017-13 March 2017 Index /CC-MAIN-2017-13-index CC-MAIN-2017-13/cc-index.paths.gz
/CC-MAIN-2017-09 February 2017 Index /CC-MAIN-2017-09-index CC-MAIN-2017-09/cc-index.paths.gz
/CC-MAIN-2017-04 January 2017 Index /CC-MAIN-2017-04-index CC-MAIN-2017-04/cc-index.paths.gz
/CC-MAIN-2016-50 December 2016 Index /CC-MAIN-2016-50-index CC-MAIN-2016-50/cc-index.paths.gz
/CC-MAIN-2016-44 October 2016 Index /CC-MAIN-2016-44-index CC-MAIN-2016-44/cc-index.paths.gz
/CC-MAIN-2016-40 September 2016 Index /CC-MAIN-2016-40-index CC-MAIN-2016-40/cc-index.paths.gz
/CC-MAIN-2016-36 August 2016 Index /CC-MAIN-2016-36-index CC-MAIN-2016-36/cc-index.paths.gz
/CC-MAIN-2016-30 July 2016 Index /CC-MAIN-2016-30-index CC-MAIN-2016-30/cc-index.paths.gz
/CC-MAIN-2016-26 June 2016 Index /CC-MAIN-2016-26-index CC-MAIN-2016-26/cc-index.paths.gz
/CC-MAIN-2016-22 May 2016 Index /CC-MAIN-2016-22-index CC-MAIN-2016-22/cc-index.paths.gz
/CC-MAIN-2016-18 April 2016 Index /CC-MAIN-2016-18-index CC-MAIN-2016-18/cc-index.paths.gz
/CC-MAIN-2016-07 February 2016 Index /CC-MAIN-2016-07-index CC-MAIN-2016-07/cc-index.paths.gz
/CC-MAIN-2015-48 November 2015 Index /CC-MAIN-2015-48-index CC-MAIN-2015-48/cc-index.paths.gz
/CC-MAIN-2015-40 September 2015 Index /CC-MAIN-2015-40-index CC-MAIN-2015-40/cc-index.paths.gz
/CC-MAIN-2015-35 August 2015 Index /CC-MAIN-2015-35-index CC-MAIN-2015-35/cc-index.paths.gz
/CC-MAIN-2015-32 July 2015 Index /CC-MAIN-2015-32-index CC-MAIN-2015-32/cc-index.paths.gz
/CC-MAIN-2015-27 June 2015 Index /CC-MAIN-2015-27-index CC-MAIN-2015-27/cc-index.paths.gz
/CC-MAIN-2015-22 May 2015 Index /CC-MAIN-2015-22-index CC-MAIN-2015-22/cc-index.paths.gz
/CC-MAIN-2015-18 April 2015 Index /CC-MAIN-2015-18-index CC-MAIN-2015-18/cc-index.paths.gz
/CC-MAIN-2015-14 March 2015 Index /CC-MAIN-2015-14-index CC-MAIN-2015-14/cc-index.paths.gz
/CC-MAIN-2015-11 February 2015 Index /CC-MAIN-2015-11-index CC-MAIN-2015-11/cc-index.paths.gz
/CC-MAIN-2015-06 January 2015 Index /CC-MAIN-2015-06-index CC-MAIN-2015-06/cc-index.paths.gz
/CC-MAIN-2014-52 December 2014 Index /CC-MAIN-2014-52-index CC-MAIN-2014-52/cc-index.paths.gz
/CC-MAIN-2014-49 November 2014 Index /CC-MAIN-2014-49-index CC-MAIN-2014-49/cc-index.paths.gz
/CC-MAIN-2014-42 October 2014 Index /CC-MAIN-2014-42-index CC-MAIN-2014-42/cc-index.paths.gz
/CC-MAIN-2014-41 September 2014 Index /CC-MAIN-2014-41-index CC-MAIN-2014-41/cc-index.paths.gz
/CC-MAIN-2014-35 August 2014 Index /CC-MAIN-2014-35-index CC-MAIN-2014-35/cc-index.paths.gz
/CC-MAIN-2014-23 July 2014 Index /CC-MAIN-2014-23-index CC-MAIN-2014-23/cc-index.paths.gz
/CC-MAIN-2014-15 April 2014 Index /CC-MAIN-2014-15-index CC-MAIN-2014-15/cc-index.paths.gz
/CC-MAIN-2014-10 March 2014 Index /CC-MAIN-2014-10-index CC-MAIN-2014-10/cc-index.paths.gz
/CC-MAIN-2013-48 Winter 2013 Index /CC-MAIN-2013-48-index CC-MAIN-2013-48/cc-index.paths.gz
/CC-MAIN-2013-20 Summer 2013 Index /CC-MAIN-2013-20-index CC-MAIN-2013-20/cc-index.paths.gz
/CC-MAIN-2012 Index of 2012 ARC files /CC-MAIN-2012-index CC-MAIN-2012/cc-index.paths.gz
/CC-MAIN-2009-2010 Index of 2009 - 2010 ARC files /CC-MAIN-2009-2010-index CC-MAIN-2009-2010/cc-index.paths.gz
/CC-MAIN-2008-2009 Index of 2008 - 2009 ARC files /CC-MAIN-2008-2009-index CC-MAIN-2008-2009/cc-index.paths.gz

Powered by pywb