Web Scraping

Html Agility Pack (HAP) is a common library used in .NET for web scraping. The HAP HTML parser is written in C# to read/write DOM and supports plain XPATH or XSLT. They have recently added the .NET Core version also for web scraping. 

• Html Agility Pack (html-agility-pack.net) website

Web scraping basically works like this :

• Fetch the HTML page
• Analyze the HTML
• Get the data that matches class names, divs, or whatever you've specified

Load Html From Web

HtmlWeb.Load method gets an HTML document from an internet resource.

Example

The following example loads an Html from the web.

var html = @"http://html-agility-pack.net/";
HtmlWeb web = new HtmlWeb();
var htmlDoc = web.Load(html);
var node = htmlDoc.DocumentNode.SelectSingleNode("//head/title");
Console.WriteLine("Node Name: " + node.Name + "\n" + node.OuterHtml);

Source: Load from Web Method in Html Agility Pack

For an end-to-end sample see Easily Do Web Scraping In .NET Core 6.0

Some additional articles and resources with information about Web Scraping

• Easily Do Web Scraping In .NET Core 6.0 (c-sharpcorner.com)
• Web Scraping in C#: Building Your First C-Sharp Scraper from Scratch - ScraperAPI
• Web Scraping with C# | ScrapingBee