Web Scraping

Web scraping basically works like this :

• Fetch the HTML page
• Analyze the HTML
• Get the data that matches class names, divs, or whatever you've specified

Html Agility Pack (HAP) is a common library used in .NET for web scraping. The HAP HTML parser is written in C# to read/write DOM and supports plain XPATH or XSLT. They have recently added the .NET Core version also for web scraping. 

• Html Agility Pack (html-agility-pack.net) website

Load Html From Web

HtmlWeb.Load method gets an HTML document from an internet resource.

Example

The following example loads an Html from the web.

var html = @"http://html-agility-pack.net/";
HtmlWeb web = new HtmlWeb();
var htmlDoc = web.Load(html);
var node = htmlDoc.DocumentNode.SelectSingleNode("//head/title");
Console.WriteLine("Node Name: " + node.Name + "\n" + node.OuterHtml);

Source: Load from Web Method in Html Agility Pack

For an end-to-end sample see Easily Do Web Scraping In .NET Core 6.0

Some additional articles and resources with information about Web Scraping

• Easily Do Web Scraping In .NET Core 6.0 (c-sharpcorner.com)
• Web Scraping in C#: Building Your First C-Sharp Scraper from Scratch - ScraperAPI
• Web Scraping with C# | ScrapingBee