A number of years in the past, Creative Commons tasked me with constructing an internet crawler able to downloading 500 million images.
Crawling anything past a few thousand URLs demands a fast distributed system. Moreover, it is not enough to be fast; moral, legal, and sensible considerations demand that a crawler be polite: a crawler should be rigorously designed to keep away from exhausting the sources of its targets.
Finally, there may be the matter of analyzing and indexing the dataset produced by the crawler. Achieving these aims on the dimensions of a number of hundred million images is a serious problem; the issues of charge limiting and task scheduling change into far more difficult when state is unfold throughout a number of nodes.

In this text, I focus on the process of designing, implementing, and deploying a big scale picture crawler, with a couple of code snippets and diagrams alongside the way in which. The complete supply code is available on GitHub beneath the MIT License. With CC Search (now Openverse), Creative Commons (CC) got down to index all of the CC licensed works on the internet, starting with images.