Data Structure

A data structure organizes data for efficient access. For example, store crawled site maps in a graph for BFS traversal (see next entry), all fetched via rotating IPs to avoid throttling.