Perl literacy course

lecture #8

Doing in Perl things that browsers do

Shlomo Yona <yona@cs.technion.ac.il> http://yeda.cs.technion.ac.il/~yona/

Today's lecture

We will deal with Web-Server applications in this lecture, and we will see how we can write HTTP-Clients to communicate with such Web-Server applications.

1. Some basics of HTTP
2. Some basics of the WWW library in Perl

The Hypertext Transfer Protocol

It is a generic, stateless, protocol which can be used for many tasks beyond its use for hypertext, such as name servers and distributed object management systems, through extension of its request methods, error codes and headers.

A feature of HTTP is the typing and negotiation of data representation, allowing systems to be built independently of the data being transferred.

More details on RFC2616.

Some basics of HTTP

HTTP is a stateless protocol.

Communication is based on HTTP-Requests and HTTP-Responses.

Requests are made by clients, and responses are made by servers.

In many cases the same object can be a client and a server (e.g. a Proxy).

Some basics of HTTP (cont.)

HTTP requests/responses are made of a HEADER followed by an (optional) BODY.

The header contains information such as addresses, what to do, agent information and more.

The body usually contains the data to transfer. Sometimes the actual data is in the header.

GET

The GET method means retrieve whatever information (in the form of an entity) is identified by the Request-URI.

If the Request-URI refers to a data-producing process, it is the produced data which shall be returned as the entity in the response and not the source text of the process, unless that text happens to be the output of the process.

The request data is encoded within the URI. This is the strange stuff you see on URLs after the actuall site address, in some cases.

HEAD

The HEAD method is identical to GET except that the server MUST NOT return a message-body in the response.

The metainformation contained in the HTTP headers in response to a HEAD request SHOULD be identical to the information sent in response to a GET request.

This method can be used for obtaining metainformation about the entity implied by the request without transferring the entity-body itself.

This method is often used for testing hypertext links for validity, accessibility, and recent modification.

POST

The POST method is used to request that the origin server accept the entity enclosed in the request as a new subordinate of the resource identified by the Request-URI in the Request-Line.

POST is designed to allow a uniform method to cover the following functions:

• Annotation of existing resources;
• Posting a message to a bulletin board, newsgroup, mailing list, or similar group of articles;
• Providing a block of data, such as the result of submitting a form, to a data-handling process;
• Extending a database through an append operation.

Some practical differences betweeb GET and POST

GET requests should always be idempotent on the server. This means that whereas one GET request might (rarely) change some state on the Server, two or more identical requests will have no further effect.

If a user hits "reload" on his/her browser, an identical request will be sent to the server, potentially resulting in two identical database or guestbook entries, counter increments, etc.

Browsers may reload a GET URL automatically, particularly if cacheing is disabled (as is usually the case with Web-Server application output), but will typically prompt the user before re-submitting a POST request.

GET is (in theory) the preferred method for idempotent operations, such as querying a database, though it matters little if you're using a form.

Some practical differences betweeb GET and POST (cont.)

There is a further practical constraint that many systems have builtin limits to the length of a GET request they can handle: when the total size of a request (URL+params) approaches or exceeds 1Kb, you are well-advised to use POST in any case.

In terms of mechanics, they differ in how parameters are passed to the Web-Server application.

In the case of a POST request, form data is passed on STDIN, so the Web-Server application should read from there (the number of bytes to be read is given by the Content-length header).

In the case of GET, the data is passed in the environment variable QUERY_STRING.

The content-type (application/x-www-form-urlencoded) is identical for GET and POST requests.

Typical HTTP Responses

HTTP Responses contain Status Code Definitions

• 100 <= Status Codes < 200 are for "Information".
• 200 <= Status Codes < 300 are for "Successful".
• 300 <= Status Codes < 400 are for "Redirection".
• 400 <= Status Codes < 500 are for "Client Error".
• 500 <= Status Codes < 600 are for "Server Error".

Status Code -- Information

• 100 Continue
• 101 Switching Protocols

Status Code -- Successful

• 200 OK
• 201 Created
• 202 Accepted
• 203 Non-Authoritative Information
• 204 No Content
• 205 Reset Content
• 206 Partial Content

Status Code -- Redirection

• 300 Multiple Choices
• 301 Moved Permanently
• 302 Found
• 303 See Other
• 304 Not Modified
• 305 Use Proxy
• 306 (Unused)
• 307 Temporary Redirect

Status Code -- Client Error

• 400 Bad Request
• 401 Unauthorized
• 402 Payment Required
• 403 Forbidden
• 404 Not Found
• 405 Method Not Allowed
• 406 Not Acceptable
• 407 Proxy Authentication Required
• 408 Request Timeout
• 409 Conflict
• 410 Gone
• 411 Length Required
• 412 Precondition Failed
• 413 Request Entity Too Large
• 414 Request-URI Too Long
• 415 Unsupported Media Type
• 416 Requested Range Not Satisfiable
• 417 Expectation Failed

Status Code -- Server Error

• 500 Internal Server Error
• 501 Not Implemented
• 502 Bad Gateway
• 503 Service Unavailable
• 504 Gateway Timeout
• 505 HTTP Version Not Supported

Writing HTTP clients with Perl

One can always do some socket programming, reinvent the wheel and create one's own propriety HTTP client.

The better way is to use existing tools. Some of them are quite excellent, and unless you need something fundamentally different, you'd better use the existing tools.

Library for WWW in Perl

LWP (short for "Library for WWW in Perl") is a popular group of Perl modules for accessing data on the Web.

Like most Perl module-distributions, each of LWP's component modules comes with documentation that is a complete reference to its interface.

However, there are so many modules in LWP that it's hard to know where to look for information on doing even the simplest things.

Introducing you to using LWP would require a whole book--a book that just happens to exist, called Perl & LWP.

Getting Documents with LWP::Simple

If you just want to access what's at a particular URL, the simplest way to do it is to use LWP::Simple's functions.

In a Perl program, you can call its get($url) function.

It will try getting that URL's content.

If it works, then it'll return the content; but if there's some error, it'll return undef.

Getting Documents with LWP::Simple (cont.)

#!/usr/bin/perl -w
use strict;

my $url = 'http://yeda.cs.technion.ac.il/~yona/perl/'; # example URL

use LWP::Simple;
my $content = get $url;
die "Couldn't get $url" unless defined $content;

# Then go do things with $content, like this:
$content =~ m/Next weekly lecture.*?Tuesday.*?(\d+.\d+.\d+)/is;
if( $1 ) {
	print "The next Perl lecture will take place on $1\n";
} else {
	print "Hmpf... probably Shlomo changes the format of the page...\n";
}

Getting Documents with LWP::Simple (cont.)

The handiest variant on get is getprint, which is useful in Perl one-liners. If it can get the page whose URL you provide, it sends it to STDOUT; otherwise it complains to STDERR.

% perl -MLWP::Simple -e "getprint 'http://cpan.org/RECENT'"

This is the URL of a plain-text file. It lists new files in CPAN in the past two weeks. You can easily make it part of a tidy little shell command, like this one that mails you the list of new Acme:: modules:

% perl -MLWP::Simple -e "getprint 'http://cpan.org/RECENT'"  \
| grep "/by-module/Acme" | mail -s "New Acme modules! Joy!" $USER

Getting Documents with LWP::Simple (cont.)

There are other useful functions in LWP::Simple, including one function for running a HEAD request on a URL (useful for checking links, or getting the last-revised time of a URL), and two functions for saving and mirroring a URL to a local file. See the LWP::Simple documentation for the full details.

The Basics of the LWP Class Model

LWP::Simple's functions are handy for simple cases, but its functions don't support cookies or authorization; they don't support setting header lines in the HTTP request; and generally, they don't support reading header lines in the HTTP response (most notably the full HTTP error message, in case of an error). To get at all those features, you'll have to use the full LWP class model.

While LWP consists of dozens of classes, the two that you have to understand are LWP::UserAgent and HTTP::Response. LWP::UserAgent is a class for "virtual browsers," which you use for performing requests, and HTTP::Response is a class for the responses (or error messages) that you get back from those requests.

The Basics of the LWP Class Model (cont.)

The basic idiom is $response = $browser->get($url), or fully illustrated:

# Early in your program:
use LWP 5.64;	# Loads all important LWP classes, and makes
		#  sure your version is reasonably recent.
my $browser = LWP::UserAgent->new;
# ...
# Then later, whenever you need to make a get request:
my $url = 'http://freshair.npr.org/dayFA.cfm?todayDate=current';

my $response = $browser->get( $url );
die "Can't get $url -- ", $response->status_line 
	unless $response->is_success;

die "Hey, I was expecting HTML, not ", $response->content_type
	unless $response->content_type eq 'text/html';
	# or whatever content-type you're equipped to deal with

# Otherwise, process the content somehow:
if($response->content =~ m/jazz/i) {
	print "They're talking about jazz today on Fresh Air!\n";
} else {
	print "Fresh Air is apparently jazzless today.\n";
}

The Basics of the LWP Class Model (cont.)

There are two objects involved: $browser, which holds an object of the class LWP::UserAgent, and then the $response object, which is of the class HTTP::Response.

You really need only one browser object per program; but every time you make a request, you get back a new HTTP::Response object, which will have some interesting attributes:

• A status code indicating success or failure (which you can test with $response->is_success).
• An HTTP status line, which I hope is informative if there is a failure (which you can see with $response->status_line, and which returns something like "404 Not Found").
• A MIME content-type like "text/html", "image/gif", "application/xml", and so on, which you can see with $response->content_type
• The actual content of the response, in $response->content. If the response is HTML, that's where the HTML source will be; if it's a GIF, then $response->content will be the binary GIF data.
• And dozens of other convenient and more specific methods that are documented in the docs for HTTP::Response, and its superclasses, HTTP::Message and HTTP::Headers.

Adding Other HTTP Request Headers

The most commonly used syntax for requests is $response = $browser->get($url), but in truth, you can add extra HTTP header lines to the request by adding a list of key-value pairs after the URL, like so:

$response = $browser->get( $url, $key1, $value1, $key2, $value2, ... );

Adding Other HTTP Request Headers (cont.)

For example, here's how to send more Netscape-like headers, in case you're dealing with a site that would otherwise reject your request:

my @ns_headers = (
	'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',
	'Accept' => 'image/gif, image/x-xbitmap, image/jpeg, 
	image/pjpeg, image/png, */*',
	'Accept-Charset' => 'iso-8859-1,*,utf-8',
	'Accept-Language' => 'en-US',
);

...

$response = $browser->get($url, @ns_headers);

Adding Other HTTP Request Headers (cont.)

If you weren't reusing that array, you could just go ahead and do this:

$response = $browser->get($url,
	'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',
	'Accept' => 'image/gif, image/x-xbitmap, image/jpeg, 
	image/pjpeg, image/png, */*',
	'Accept-Charset' => 'iso-8859-1,*,utf-8',
	'Accept-Language' => 'en-US',
);

Adding Other HTTP Request Headers (cont.)

If you were only going to change the 'User-Agent' line, you could just change the $browser object's default line from "libwww-perl/5.65" (or the like) to whatever you like, using LWP::UserAgent's agent method:

$browser->agent('Mozilla/4.76 [en] (Win98; U)');

Enabling Cookies

A default LWP::UserAgent object acts like a browser with its cookies support turned off.

There are various ways of turning it on, by setting its cookie_jar attribute.

A "cookie jar" is an object representing a little database of all the HTTP cookies that a browser can know about.

It can correspond to a file on disk (the way Netscape uses its cookies.txt file), or it can be just an in-memory object that starts out empty, and whose collection of cookies will disappear once the program is finished running.

Enabling Cookies (cont.)

To give a browser an in-memory empty cookie jar, you set its cookie_jar attribute like so:

$browser->cookie_jar({});

Enabling Cookies (cont.)

To give it a copy that will be read from a file on disk, and will be saved to it when the program is finished running, set the cookie_jar attribute like this:

use HTTP::Cookies;
$browser->cookie_jar( HTTP::Cookies->new(
	'file' => '/some/where/cookies.lwp',
	# where to read/write cookies
	'autosave' => 1,
	# save it to disk when done
));

Enabling Cookies (cont.)

That file will be an LWP-specific format. If you want to access the cookies in your Netscape cookies file, you can use the HTTP::Cookies::Netscape class:

use HTTP::Cookies;
# yes, loads HTTP::Cookies::Netscape too

$browser->cookie_jar( HTTP::Cookies::Netscape->new(
	'file' => 'c:/Program Files/Netscape/Users/DIR-NAME-HERE/cookies.txt',
	# where to read cookies
));

You could add an 'autosave' => 1 line as we did earlier, but at time of writing, it's uncertain whether Netscape might discard some of the cookies you could be writing back to disk.

Posting Form Data

Many HTML forms send data to their server using an HTTP POST request, which you can send with this syntax:

$response = $browser->post( $url,
	[
		formkey1 => value1, 
		formkey2 => value2, 
		# ...
	],
);

Or if you need to send HTTP headers:

$response = $browser->post( $url,
	[
		formkey1 => value1, 
		formkey2 => value2, 
		# ...
	],
	headerkey1 => value1, 
	headerkey2 => value2, 
);

Posting Form Data (cont.)

For example, the following program makes a search request to AltaVista (by sending some form data via an HTTP POST request), and extracts from the HTML the report of the number of matches:

use strict;
use warnings;
use LWP 5.64;
my $browser = LWP::UserAgent->new;

my $word = 'tarragon';

my $url = 'http://www.altavista.com/sites/search/web';
my $response = $browser->post( $url,
	[ 
		'q' => $word,  # the Altavista query string
		'pg' => 'q', 'avkw' => 'tgz', 'kl' => 'XX',
	]
);
die "$url error: ", $response->status_line
unless $response->is_success;
die "Weird content type at $url -- ", $response->content_type
unless $response->content_type eq 'text/html';

if( $response->content =~ m{AltaVista found ([0-9,]+) results} ) {
	# The substring will be like "AltaVista found 2,345 results"
	print "$word: $1\n";
} else {
	print "Couldn't find the match-string in the response\n";
}

Sending GET Form Data

Some HTML forms convey their form data not by sending the data in an HTTP POST request, but by making a normal GET request with the data stuck on the end of the URL. For example, if you went to imdb.com and ran a search on Blade Runner, the URL you'd see in your browser window would be:

http://us.imdb.com/Tsearch?title=Blade%20Runner&restrict=Movies+and+TV

Sending GET Form Data (cont.)

To run the same search with LWP, you'd use this idiom, which involves the URI class:

use URI;
my $url = URI->new( 'http://us.imdb.com/Tsearch' );
	# makes an object representing the URL

$url->query_form(  # And here the form data pairs:
	'title'    => 'Blade Runner',
	'restrict' => 'Movies and TV',
);

my $response = $browser->get($url);

Absolutizing URLs

The URI class that we just mentioned above provides all sorts of methods for accessing and modifying parts of URLs (such as asking sort of URL it is with $url->scheme, and asking what host it refers to with $url->host, and so on, as described in the docs for the URI class.

However, the methods of most immediate interest are the query_form method seen above, and now the new_abs method for taking a probably relative URL string (like "../foo.html") and getting back an absolute URL (like "http://www.perl.com/stuff/foo.html"), as shown here:

use URI;
$abs = URI->new_abs($maybe_relative, $base);

Absolutizing URLs (cont.)

For example, consider this program that matches URLs in the HTML list of new modules in CPAN:

use strict;
use warnings;
use LWP 5.64;
my $browser = LWP::UserAgent->new;

my $url = 'http://www.cpan.org/RECENT.html';
my $response = $browser->get($url);
die "Can't get $url -- ", $response->status_line
	unless $response->is_success;

my $html = $response->content;
while( $html =~ m/<A HREF=\"(.*?)\"/g ) {    
	print "$1\n";  
}

Absolutizing URLs (cont.)

MIRRORING.FROM
RECENT
RECENT.html
authors/00whois.html
authors/01mailrc.txt.gz
authors/id/A/AA/AASSAD/CHECKSUMS
...

Absolutizing URLs (cont.)

However, if you actually want to have those be absolute URLs, you can use the URI module's new_abs method, by changing the while loop to this:

while( $html =~ m/<A HREF=\"(.*?)\"/g ) {    
	print URI->new_abs( $1, $response->base ) ,"\n";
}

(The $response->base method from HTTP::Message is for returning the URL that should be used for resolving relative URLs--it's usually just the same as the URL that you requested.)

Absolutizing URLs (cont.)

http://www.cpan.org/MIRRORING.FROM
http://www.cpan.org/RECENT
http://www.cpan.org/RECENT.html
http://www.cpan.org/authors/00whois.html
http://www.cpan.org/authors/01mailrc.txt.gz
http://www.cpan.org/authors/id/A/AA/AASSAD/CHECKSUMS

Of course, using a regexp to match hrefs is a bit simplistic, and for more robust programs, you'll probably want to use an HTML-parsing module like HTML::LinkExtor, or HTML::TokeParser, or even maybe HTML::TreeBuilder.

Other Browser Attributes

LWP::UserAgent objects have many attributes for controlling how they work. Here are a few notable ones:

Other Browser Attributes -- timeout

$browser->timeout(15);

This sets this browser object to give up on requests that don't answer within 15 seconds.

Other Browser Attributes -- protocol restriction

$browser->protocols_allowed( [ 'http', 'gopher'] ); 

This sets this browser object to not speak any protocols other than HTTP and gopher.

If it tries accessing any other kind of URL (like an "ftp:" or "mailto:" or "news:" URL), then it won't actually try connecting, but instead will immediately return an error code 500, with a message like "Access to ftp URIs has been disabled".

Other Browser Attributes -- Keep-Alive

use LWP::ConnCache;
$browser->conn_cache(LWP::ConnCache->new());

This tells the browser object to try using the HTTP/1.1 "Keep-Alive" feature, which speeds up requests by reusing the same socket connection for multiple requests to the same server.

Other Browser Attributes -- User-Agent

$browser->agent( 'SomeName/1.23 (more info here maybe)' ); 

This changes how the browser object identifies itself in the default "User-Agent" line is its HTTP requests.

By default, it'll send "libwww-perl/versionnumber", like "libwww-perl/5.65".

You can change that to something more descriptive like this:

$browser->agent( 'SomeName/3.14 (contact@robotplexus.int)' );

Or if need be, you can go in disguise, like this:

$browser->agent( 'Mozilla/4.0 (compatible; MSIE 5.12; Mac_PowerPC)' );

Other Browser Attributes -- obeying redirection responses

push @{ $ua->requests_redirectable }, 'POST'; 

This tells this browser to obey redirection responses to POST requests (like most modern interactive browsers), even though the HTTP RFC says that should not normally be done.

Writing Polite Robots

If you want to make sure that your LWP-based program respects robots.txt files and doesn't make too many requests too fast, you can use the LWP::RobotUA class instead of the LWP::UserAgent class.

LWP::RobotUA class is just like LWP::UserAgent, and you can use it like so:

my $response = $browser->get($url);

Writing Polite Robots (cont.)

But LWP::RobotUA adds these features:

If the robots.txt on $url's server forbids you from accessing $url, then the $browser object (assuming it's of the class LWP::RobotUA) won't actually request it, but instead will give you back (in $response) a 403 error with a message "Forbidden by robots.txt". That is, if you have this line:

die "$url -- ", $response->status_line, "\nAborted" 
	unless $response->is_success;

then the program would die with an error message like this:

http://whatever.site.int/pith/x.html -- 403 Forbidden 
by robots.txt
Aborted at whateverprogram.pl line 1234

Writing Polite Robots (cont.)

But LWP::RobotUA adds these features (cont'):

If this $browser object sees that the last time it talked to $url's server was too recently, then it will pause (via sleep) to avoid making too many requests too often.

Writing Polite Robots (cont.)

How long it will pause for, is by default one minute--but you can control it with the $browser->delay( minutes ) attribute.

For example, this code:

$browser->delay( 7/60 );

means that this browser will pause when it needs to avoid talking to any given server more than once every 7 seconds.

Using Proxies

In some cases, you will want to (or will have to) use proxies for accessing certain sites or for using certain protocols. This is most commonly the case when your LWP program is running (or could be running) on a machine that is behind a firewall.

To make a browser object use proxies that are defined in the usual environment variables (HTTP_PROXY), just call the env_proxy on a user-agent object before you go making any requests on it. Specifically:

use LWP::UserAgent;
my $browser = LWP::UserAgent->new;

# And before you go making any requests:
$browser->env_proxy;

HTTP Authentication

Many Web sites restrict access to documents by using "HTTP Authentication". This isn't just any form of "enter your password" restriction, but is a specific mechanism where the HTTP server sends the browser an HTTP code that says "That document is part of a protected 'realm', and you can access it only if you re-request it and add some special authorization headers to your request".

For example, the Unicode.org administrators stop email-harvesting bots from harvesting the contents of their mailing list archives by protecting them with HTTP Authentication, and then publicly stating the username and password (at http://www.unicode.org/mail-arch/ ) -- namely username "unicode-ml" and password "unicode".

HTTP Authentication (cont.)

For example, consider this URL, which is part of the protected area of the Web site:

http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html

If you access that with a browser, you'll get a prompt like "Enter username and password for 'Unicode-MailList-Archives' at server 'www.unicode.org'", or in a graphical browser, something like this.

HTTP Authentication (cont.)

In LWP, if you just request that URL, like this:

use LWP 5.64;
my $browser = LWP::UserAgent->new;

my $url = 'http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html';
my $response = $browser->get($url);

die "Error: ", $response->header('WWW-Authenticate') || 'Error accessing',
	#  ('WWW-Authenticate' is the realm-name)
			"\n ", $response->status_line, "\n at $url\n Aborting"
	unless $response->is_success;

HTTP Authentication (cont.)

Then you'll get this error:

Error: Basic realm="Unicode-MailList-Archives"
401 Authorization Required
at http://www.unicode.org/mail-arch/unicode-ml/y2002-m08/0067.html
Aborting at auth1.pl line 8.  [or wherever]

because the $browser doesn't know any the username and password for that realm ("Unicode-MailList-Archives") at that host ("www.unicode.org").

The simplest way to let the browser know about this is to use the credentials method to let it know about a username and password that it can try using for that realm at that host.

HTTP Authentication (cont.)

The syntax is:

$browser->credentials(
	'servername:portnumber',
	'realm-name',
	'username' => 'password'
);

HTTP Authentication (cont.)

In most cases, the port number is 80, the default TCP/IP port for HTTP; and you usually call the credentials method before you make any requests. For example:

$browser->credentials(
	'reports.mybazouki.com:80',
	'web_server_usage_reports',
	'plinky' => 'banjo123'
);

HTTP Authentication (cont.)

So if we add the following to the program above, right after the $browser = LWP::UserAgent->new; line:

$browser->credentials(  # add this to our $browser 's "key ring"
	'www.unicode.org:80',
	'Unicode-MailList-Archives',
	'unicode-ml' => 'unicode'
);

and then when we run it, the request succeeds, instead of causing the die to be called.

Accessing HTTPS URLs

When you access an HTTPS URL, it'll work for you just like an HTTP URL would--if your LWP installation has HTTPS support (via an appropriate Secure Sockets Layer library). For example:

use LWP 5.64;
my $url = 'https://www.paypal.com/';   # Yes, HTTPS!
my $browser = LWP::UserAgent->new;
my $response = $browser->get($url);
die "Error at $url\n ", $response->status_line, "\n Aborting"
	unless $response->is_success;
print "Whee, it worked!  I got that ",
$response->content_type, " document!\n";

Accessing HTTPS URLs (cont.)

If your LWP installation doesn't have HTTPS support set up, then the response will be unsuccessful, and you'll get this error message:

Error at https://www.paypal.com/
501 Protocol scheme 'https' is not supported
Aborting at paypal.pl line 7.   [or whatever program and line]

If your LWP installation does have HTTPS support installed, then the response should be successful, and you should be able to consult $response just like with any normal HTTP response.

Getting Large Documents

When you're requesting a large (or at least potentially large) document, a problem with the normal way of using the request methods (like $response = $browser->get($url)) is that the response object in memory will have to hold the whole document--in memory. If the response is a 30-megabyte file, this is likely to be quite an imposition on this process's memory usage.

Getting Large Documents (cont.)

A notable alternative is to have LWP save the content to a file on disk, instead of saving it up in memory. This is the syntax to use:

$response = $ua->get($url, ':content_file' => $filespec,);

For example,

$response = $ua->get('http://search.cpan.org/', ':content_file' => '/tmp/sco.html');

Getting Large Documents (cont.)

When you use this :content_file option, the $response will have all the normal header lines, but $response->content will be empty.

Note that this ":content_file" option isn't supported under older versions of LWP. If you need to be compatible with older LWP versions, then use this syntax, which does the same thing:

use HTTP::Request::Common;
$response = $ua->request( GET($url), $filespec );

Presenting Several CPAN modules

WWW::Search http://search.cpan.org/author/MTHURN/WWW-Search-2.37/lib/WWW/Search.pm

WWW::Search::Ebay http://search.cpan.org/author/MTHURN/WWW-Search-Ebay-2.11/lib/WWW/Search/Ebay.pm

WWW::Search::Google http://search.cpan.org/author/LBROCARD/WWW-Search-Google-0.20/Google.pm

WWW::Search::Metapedia http://search.cpan.org/author/MTHURN/WWW-Search-Backends-1.01/lib/WWW/Search/Metapedia.pm

WWW::Search::RpmFind http://search.cpan.org/author/ALIAN/WWW-Search-RpmFind-1.2/RpmFind.pm

WWW::Amazon::Wishlist http://search.cpan.org/author/SIMONW/WWW-Amazon-Wishlist-0.85/Wishlist.pm

WWW::Mechanize http://search.cpan.org/author/PETDANCE/WWW-Mechanize-0.32/lib/WWW/Mechanize.pm

References

1. Web Basics with LWP by Sean M. Burke
2. RFC 2616 Hypertext Transfer Protocol -- HTTP/1.1
3. perldoc LWP::Simple
4. perldoc LWP::UserAgent
5. perldoc LWP
6. perldoc HTTP::Response
7. perldoc HTTP::Message
8. perldoc HTTP::Headers
9. perldoc URI
10. URI::Escape
11. The Robots Exclusion Protocol http://www.robotstxt.org/
12. perldoc LWP::RobotUA

Thank you