Fetch contents of a web page using C#

If you are looking for some code help which can help you fetch contents of a web page using C# in asp.net then you’re at the right place. Sometimes you do need to read a web page programmatically so you could process the contents of the page.

In order to perform this task you need to import three Microsoft.net namespaces:

using System.IO;
using System.Net;
using System.Text;

First of all lets review the concept of web page request and response mechanism. When ever you type a URL in a browser window to get a web page, you’re actually making a request to the server to get a particular page. To perform this task programmatically we’ll use WebRequest class of System.Net namespace. We’ll create a request to simulate a manual request.

WebRequest objReq = WebRequest.Create("www.yoururl.com/anypage");

When a request is sent to the server, the application server or say web server returns a response with the required page contents. In manual request the browser itself receives the response and shows it to the user, but in our case we’ll receive this response in a specific response object of .net’s System.Net namespace called WebResponse.

WebResponse objResp = objReq.GetResponse();

First we need to create a stream from this response object so can prepare our page contents to be read by our stream reader which will read the actuall contents.

Stream objStream = objResp.GetResponseStream();

Now, we need to get the contents from this stream object. We’ll get a stream of the contents or text in ASCII format in a stream object.

StreamReader objSR = new StreamReader(objStream,Encoding.ASCII);

Finally, you simply need to get the text from this stream reader into a string variable.

String content = objSR.ReadToEnd();

Now, you can either save the contents or can display it on some other page. You can also process the contents of the page output. Remember the output text will contain the whole HTML format page – I mean with the head and body tags etc. If you want to display this text on some other page just remove those extra tags and just copy the actuall contents required. You can also get a subscring at some particular location. It’s all up to you now how you play with these simple statements. :)

Get file list from the server using ftp web request in C#

Sometimes developers need to get a list of all the files on a server, and this needs to be done using ftp. Microsoft .Net provides classes to handle ftp requests. I'll show you some other functions performed using these classes in my later posts. Today, I'll focus only on getting a list of files from ftp server. I'll use FTPWebRequest class to fetch the list of files, and a StreadReader class to read the stream returned by FTPWebRequest method. Code is in C# (C-Sharp).

First of all you need to include two namespaces:

using System.Collections.Generic;
using System.Net;

System.Net namespace provides FTPWebRequest class to make a web request to an ftp server.

Now, declare a generic list of strings to keep the list of the fetched files:

List strList = new List();

In order to understand how to use generic list you can view my following post:
Using a generic list of strings or other class objects

Declare an object of type FTPWebRequest, and create an instance using the ftp server IP.
Here ftpServerIP is the IP of FTP server from where you want to fetch the list of files.

FtpWebRequest fwr = (FtpWebRequest)FtpWebRequest.Create(new Uri("ftp://" + ftpServerIP));

FTPWebRequest method requires proper credentials to authenticate the request. In the statement below ftpUserID and ftpPassword variables serve this purpose.

fwr.Credentials = new NetworkCredential(ftpUserID, ftpPassword);

WebRequestMethod in the following code line tells the FTP Web Request object the operation you want to perform. In our current scenario we want to get the list of all the files in the root directory at the ftp server, so we'll use ListDirectory method type.

fwr.Method = WebRequestMethods.Ftp.ListDirectory;

Now, we'll create an object of type StreamReader as given below:

StreamReader sr = new StreamReader(fwr.GetResponse().GetResponseStream());

The important thing to understand here is 'fwr.GetResponse().GetResponseStream()'. As you already know that fwr is an FTPWebRequest type object and GetResponse method of this object returns an object of type WebResponse, so call an other method named GetResponseStream to create a StreamReader object. It shouldn't be confusing; nevertheless I'll discuss streams and related stuff in some later posts.

Okay, fellas! Now, we're ready to reap the benefits of all the labor we have done until now. :-)

We'll read the stream one line at a time - which will be a string - and we'll add these strings in the strList collection.

string str = sr.ReadLine();
while (str != null)
{
strList.Add(str);
str = sr.ReadLine();
}

Remember that the stream we received from the ftp server contained a list of files that is one file in one line!

So, we're almost done with the task we were supposed to do, but don't forget to impress your development manager or technical lead by the following statements - be smart and don't leave the labor for the garbage collector!

sr.Close();
sr = null;
fwr = null;