pub fn normalize_urls(urls: Vec<Url>) -> Vec<Url>Expand description
Normalizes a list of URLs to avoid duplicates.
This function removes URL fragments and ensures each URL ends with a trailing slash if it doesn’t have a path or if the path is just “/”. It also logs a warning if duplicate URLs are found after normalization. Invalid URLs (those not using http or https schemes) are filtered out.
§Arguments
urls- A vector of URLs to normalize
§Returns
A vector of normalized unique URLs, in the order each first appears