Fetch contents of a web page using C#

If you are looking for some code help which can help you fetch contents of a web page using C# in asp.net then you’re at the right place. Sometimes you do need to read a web page programmatically so you could process the contents of the page.

In order to perform this task you need to import three Microsoft.net namespaces:

using System.IO;
using System.Net;
using System.Text;

First of all lets review the concept of web page request and response mechanism. When ever you type a URL in a browser window to get a web page, you’re actually making a request to the server to get a particular page. To perform this task programmatically we’ll use WebRequest class of System.Net namespace. We’ll create a request to simulate a manual request.

WebRequest objReq = WebRequest.Create("www.yoururl.com/anypage");

When a request is sent to the server, the application server or say web server returns a response with the required page contents. In manual request the browser itself receives the response and shows it to the user, but in our case we’ll receive this response in a specific response object of .net’s System.Net namespace called WebResponse.

WebResponse objResp = objReq.GetResponse();

First we need to create a stream from this response object so can prepare our page contents to be read by our stream reader which will read the actuall contents.

Stream objStream = objResp.GetResponseStream();

Now, we need to get the contents from this stream object. We’ll get a stream of the contents or text in ASCII format in a stream object.

StreamReader objSR = new StreamReader(objStream,Encoding.ASCII);

Finally, you simply need to get the text from this stream reader into a string variable.

String content = objSR.ReadToEnd();

Now, you can either save the contents or can display it on some other page. You can also process the contents of the page output. Remember the output text will contain the whole HTML format page – I mean with the head and body tags etc. If you want to display this text on some other page just remove those extra tags and just copy the actuall contents required. You can also get a subscring at some particular location. It’s all up to you now how you play with these simple statements. :)