Please see the PyWB CDX Server API Reference for more examples on how to use the query API. Replace the API endpoint coll/cdx with one of the API endpoints listed below (also as JSON list):
Alternatively, you may use one of the command-line tools based on this API:
Common Crawl data is stored on Amazon Web Services' Public Data Sets. All data and index files are free to download. Feel free to run your own index server, or analyze the index offline.
Please do not overload the URL index server for bulk downloads (e.g. all records of the entire .com top-level domain), see the download instructions. Alternatively, check the columnar index which allows for efficient aggregations and filtering on any field/column.
More information about this URL index is found in our announcement of the Common Crawl index. For help and support, please visit the Common Crawl user forum, or Discord Server.
For further information please see Getting Started.