url-normalization
Clean, filter and sample URLs to optimize data collection – Python & command-line – Deduplication, spam, content and language filters
Remove clutter from URLs and return a canonicalized version
