Skip to main content

normalize_urls

Function normalize_urls 

Source
pub fn normalize_urls(urls: Vec<Url>) -> Vec<Url>
Expand description

Normalizes a list of URLs to avoid duplicates.

This function removes URL fragments and ensures each URL ends with a trailing slash if it doesn’t have a path or if the path is just “/”. It also logs a warning if duplicate URLs are found after normalization. Invalid URLs (those not using http or https schemes) are filtered out.

§Arguments

  • urls - A vector of URLs to normalize

§Returns

A vector of normalized unique URLs, in the order each first appears