diff -Nru libhttp-recorder-perl-0.06/CHANGES libhttp-recorder-perl-0.07/CHANGES --- libhttp-recorder-perl-0.06/CHANGES 2012-04-23 14:19:49.000000000 +0000 +++ libhttp-recorder-perl-0.07/CHANGES 2013-04-05 03:51:50.000000000 +0000 @@ -1,100 +1,104 @@ -Revision history for HTTP::Recorder ------------------------------------ - -0.06 - New maintanner - added a script httprecorder.pl for simple recordering - Added HTTP::Proxy dependacy to Makefile.PL - Added patch by SREZIC for encoding follow_link commands bugid:8243 - Added patch by Marek.Rouchal bugid:15998 fixing the unmodify sub - that sometimes gets a string instead of an object - Added patch by david to prevent susbsitute on undef value, bugid:36841 - Moved the injected JS inside the head tag, bugid:48574 - -0.05 17 August 2005 - Fixed a bug where link text wasn't being un-escaped before being - logged. - - Significant changes in the use of the Control Panel: - - The showwindow option is gone (no popups any more). - - If you want to use the control panel, you need to start there. - - The control panel now spawns a window for navigation. - - The navigation window updates the Control Panel via Javascript. - - Download button downloads the script file directly. - -0.04 17 August 2005 - Parse and manipulate parameters with URI::QueryParam and - HTTP::Request::Params, rather than doing it by hand. - -0.03_03 16 August 2005 - Several improvements to the Control Panel UI - - background color for easier reading - - script textarea resizes with window - - JavaScript confirmation before deleting script - - Recording changes - - Log 0, false, and empty values - - Use click() instead of submit_form() - - Changed the format of the %fields argument to - Logger::SetFieldsAndSubmit. This is unlikely to affect you unless - you've made a subclass or are using that method directly. - -0.03_02 9 August 2005 - - Support forms with multiple Submit buttons. - - Always use single quotes around arguments. - - Applied a patch from alex@kapranoff.ru (rt.cpan.org #6711) to - optionally ignore gets for favicon.ico. On by default. - - Applied a patch from alex@kapranoff.ru (rt.cpan.org #6646) so - that the output script doesn't try to set hidden fields. - -0.03_01 1 March 2004 - - Support for recording SSL - - Added a UI - - Available via control URL rather than a JS popup - - Update, reset clear, download features - - "Goto page" functionality - - Rewriting improvements - - Support 'link' tag - - Support link anchors - - Set base URL if necessary (so relative links will be properly followed) - - Don't try to set the value of submit buttons in a form - - support multipart/form-data - - Additional documentation - - Logging improvements - - Add SetFieldsAndSubmit method to Logger; use it - - Set form name before trying to set form fields. - - Use form names when available, rather than form numbers. - -0.02 15 February 2004 - - Improved rewriting for pages with JavaScript: - - Don't rewrite href="javascript:XXX" links or text inside them - - Change the way forms are re-written, so that JavaScript won't fail. - - Use link indices as well as names (supports multiple links of the - same name on a single page). - - Preserve page titles. - - Use form names when available, rather than form numbers. - - Keep attributes in their original order on rewrite. - - Support links with quotes (") in them. - - Only try to rewrite text/ content (images, etc. won't be - corrupted). - - ** Many thanks to Jason Gessner for his - patches and feedback. +Revision history for HTTP::Recorder +----------------------------------- + +0.07 + Quelch 'unitialized' warning in HTTP::Recorder (thanks sjn) + Make http_recorder script give a nicer output at startup + +0.06_01 + New maintanner + added a script httprecorder.pl for simple recordering + Added HTTP::Proxy dependacy to Makefile.PL + Added patch by SREZIC for encoding follow_link commands bugid:8243 + Added patch by Marek.Rouchal bugid:15998 fixing the unmodify sub + that sometimes gets a string instead of an object + Added patch by david to prevent susbsitute on undef value, bugid:36841 + Moved the injected JS inside the head tag, bugid:48574 + +0.05 17 August 2005 + Fixed a bug where link text wasn't being un-escaped before being + logged. + + Significant changes in the use of the Control Panel: + - The showwindow option is gone (no popups any more). + - If you want to use the control panel, you need to start there. + - The control panel now spawns a window for navigation. + - The navigation window updates the Control Panel via Javascript. + + Download button downloads the script file directly. + +0.04 17 August 2005 + Parse and manipulate parameters with URI::QueryParam and + HTTP::Request::Params, rather than doing it by hand. + +0.03_03 16 August 2005 + Several improvements to the Control Panel UI + - background color for easier reading + - script textarea resizes with window + - JavaScript confirmation before deleting script + + Recording changes + - Log 0, false, and empty values + - Use click() instead of submit_form() + + Changed the format of the %fields argument to + Logger::SetFieldsAndSubmit. This is unlikely to affect you unless + you've made a subclass or are using that method directly. + +0.03_02 9 August 2005 + + Support forms with multiple Submit buttons. + + Always use single quotes around arguments. + + Applied a patch from alex@kapranoff.ru (rt.cpan.org #6711) to + optionally ignore gets for favicon.ico. On by default. + + Applied a patch from alex@kapranoff.ru (rt.cpan.org #6646) so + that the output script doesn't try to set hidden fields. + +0.03_01 1 March 2004 + + Support for recording SSL + + Added a UI + - Available via control URL rather than a JS popup + - Update, reset clear, download features + - "Goto page" functionality + + Rewriting improvements + - Support 'link' tag + - Support link anchors + - Set base URL if necessary (so relative links will be properly followed) + - Don't try to set the value of submit buttons in a form + - support multipart/form-data + + Additional documentation + + Logging improvements + - Add SetFieldsAndSubmit method to Logger; use it + - Set form name before trying to set form fields. + - Use form names when available, rather than form numbers. + +0.02 15 February 2004 + + Improved rewriting for pages with JavaScript: + - Don't rewrite href="javascript:XXX" links or text inside them + - Change the way forms are re-written, so that JavaScript won't fail. + + Use link indices as well as names (supports multiple links of the + same name on a single page). + + Preserve page titles. + + Use form names when available, rather than form numbers. + + Keep attributes in their original order on rewrite. + + Support links with quotes (") in them. + + Only try to rewrite text/ content (images, etc. won't be + corrupted). + + ** Many thanks to Jason Gessner for his + patches and feedback. diff -Nru libhttp-recorder-perl-0.06/MANIFEST libhttp-recorder-perl-0.07/MANIFEST --- libhttp-recorder-perl-0.06/MANIFEST 2011-07-16 14:59:54.000000000 +0000 +++ libhttp-recorder-perl-0.07/MANIFEST 2013-04-05 03:56:45.000000000 +0000 @@ -1,10 +1,11 @@ -lib/HTTP/Recorder.pm -lib/HTTP/Recorder/Logger.pm -bin/httprecorder -Makefile.PL -t/load.t -t/pod.t -MANIFEST -CHANGES -README -META.yml Module meta-data (added by MakeMaker) +lib/HTTP/Recorder.pm +lib/HTTP/Recorder/Logger.pm +bin/httprecorder +Makefile.PL +t/load.t +t/pod.t +MANIFEST +CHANGES +README +META.yml Module YAML meta-data (added by MakeMaker) +META.json Module JSON meta-data (added by MakeMaker) diff -Nru libhttp-recorder-perl-0.06/META.json libhttp-recorder-perl-0.07/META.json --- libhttp-recorder-perl-0.06/META.json 1970-01-01 00:00:00.000000000 +0000 +++ libhttp-recorder-perl-0.07/META.json 2013-04-05 03:56:45.000000000 +0000 @@ -0,0 +1,54 @@ +{ + "abstract" : "unknown", + "author" : [ + "Shmuel Fomberg (semuelf@cpan.org)" + ], + "dynamic_config" : 1, + "generated_by" : "ExtUtils::MakeMaker version 6.62, CPAN::Meta::Converter version 2.120921", + "license" : [ + "unknown" + ], + "meta-spec" : { + "url" : "http://search.cpan.org/perldoc?CPAN::Meta::Spec", + "version" : "2" + }, + "name" : "HTTP-Recorder", + "no_index" : { + "directory" : [ + "t", + "inc" + ] + }, + "prereqs" : { + "build" : { + "requires" : { + "ExtUtils::MakeMaker" : "0" + } + }, + "configure" : { + "requires" : { + "ExtUtils::MakeMaker" : "0" + } + }, + "runtime" : { + "requires" : { + "Getopt::Long" : "0", + "HTML::Parser" : "0", + "HTML::TokeParser" : "0", + "HTTP::Proxy" : "0", + "HTTP::Request::Params" : "0", + "LWP::UserAgent" : "0", + "Pod::Usage" : "0", + "URI::Escape" : "0", + "URI::QueryParam" : "0" + } + } + }, + "release_status" : "stable", + "resources" : { + "repository" : { + "url" : "https://github.com/semuel/perl-cpan-http-recorder.git" + } + }, + "version" : "0.07" +} diff -Nru libhttp-recorder-perl-0.06/META.yml libhttp-recorder-perl-0.07/META.yml --- libhttp-recorder-perl-0.06/META.yml 2012-04-23 14:22:44.000000000 +0000 +++ libhttp-recorder-perl-0.07/META.yml 2013-04-05 03:56:45.000000000 +0000 @@ -1,30 +1,32 @@ ---- #YAML:1.0 -name: HTTP-Recorder -version: 0.06 -abstract: ~ -author: - - Shmuel Fomberg (semuelf@cpan.org) -license: unknown -distribution_type: module -configure_requires: - ExtUtils::MakeMaker: 0 -build_requires: - ExtUtils::MakeMaker: 0 -requires: - Getopt::Long: 0 - HTML::Parser: 0 - HTML::TokeParser: 0 - HTTP::Proxy: 0 - HTTP::Request::Params: 0 - LWP::UserAgent: 0 - Pod::Usage: 0 - URI::Escape: 0 - URI::QueryParam: 0 -no_index: - directory: - - t - - inc -generated_by: ExtUtils::MakeMaker version 6.56 -meta-spec: - url: http://module-build.sourceforge.net/META-spec-v1.4.html - version: 1.4 +--- +abstract: unknown +author: + - 'Shmuel Fomberg (semuelf@cpan.org)' +build_requires: + ExtUtils::MakeMaker: 0 +configure_requires: + ExtUtils::MakeMaker: 0 +dynamic_config: 1 +generated_by: 'ExtUtils::MakeMaker version 6.62, CPAN::Meta::Converter version 2.120921' +license: unknown +meta-spec: + url: http://module-build.sourceforge.net/META-spec-v1.4.html + version: 1.4 +name: HTTP-Recorder +no_index: + directory: + - t + - inc +requires: + Getopt::Long: 0 + HTML::Parser: 0 + HTML::TokeParser: 0 + HTTP::Proxy: 0 + HTTP::Request::Params: 0 + LWP::UserAgent: 0 + Pod::Usage: 0 + URI::Escape: 0 + URI::QueryParam: 0 +resources: + repository: https://github.com/semuel/perl-cpan-http-recorder.git +version: 0.07 diff -Nru libhttp-recorder-perl-0.06/Makefile.PL libhttp-recorder-perl-0.07/Makefile.PL --- libhttp-recorder-perl-0.06/Makefile.PL 2011-07-16 14:56:35.000000000 +0000 +++ libhttp-recorder-perl-0.07/Makefile.PL 2013-04-05 03:44:04.000000000 +0000 @@ -1,21 +1,27 @@ -use ExtUtils::MakeMaker; -# See lib/ExtUtils/MakeMaker.pm for details of how to influence -# the contents of the Makefile that is written. -WriteMakefile( - NAME => 'HTTP::Recorder', - VERSION_FROM => 'lib/HTTP/Recorder.pm', - AUTHOR => 'Shmuel Fomberg (semuelf@cpan.org)', - PREREQ_PM => { LWP::UserAgent => 0, - HTML::TokeParser => 0, - URI::Escape => 0, - URI::QueryParam => 0, - HTTP::Request::Params => 0, - HTTP::Proxy => 0, - Getopt::Long => 0, - Pod::Usage => 0, - HTML::Parser => 0, - }, - EXE_FILES => [qw( - bin/httprecorder - )], -); +use ExtUtils::MakeMaker; + +# See lib/ExtUtils/MakeMaker.pm for details of how to influence +# the contents of the Makefile that is written. +WriteMakefile( + NAME => 'HTTP::Recorder', + VERSION_FROM => 'lib/HTTP/Recorder.pm', + AUTHOR => 'Shmuel Fomberg (semuelf@cpan.org)', + PREREQ_PM => { + LWP::UserAgent => 0, + HTML::TokeParser => 0, + URI::Escape => 0, + URI::QueryParam => 0, + HTTP::Request::Params => 0, + HTTP::Proxy => 0, + Getopt::Long => 0, + Pod::Usage => 0, + HTML::Parser => 0, + }, + EXE_FILES => [ qw( + bin/httprecorder + ) ], + META_MERGE => { + resources => + {repository => 'https://github.com/semuel/perl-cpan-http-recorder.git',}, + }, +); diff -Nru libhttp-recorder-perl-0.06/bin/httprecorder libhttp-recorder-perl-0.07/bin/httprecorder --- libhttp-recorder-perl-0.06/bin/httprecorder 2011-07-03 13:34:58.000000000 +0000 +++ libhttp-recorder-perl-0.07/bin/httprecorder 2013-04-05 03:44:04.000000000 +0000 @@ -1,64 +1,65 @@ -#!/usr/bin/perl - -use strict; -use warnings; - -use HTTP::Proxy; -use HTTP::Recorder; -use Getopt::Long; -use Pod::Usage; - -my $port = 8080; -my $filename = "http_traffic"; -my $show_help = ''; - -GetOptions ("port=i" => \$port, "file=s" => \$filename, "help" => \$show_help); -pod2usage(1) if ($show_help); - -print "httprecorder - quick script for recording HTTP traffic\n"; -print "Starting Proxy Server on Port $port\n"; -print "Recording to file: $filename\n"; - -my $proxy = HTTP::Proxy->new( port => $port ); - -# create a new HTTP::Recorder object -my $agent = new HTTP::Recorder; - -# set the log file (optional) -$agent->file($filename); - -# set HTTP::Recorder as the agent for the proxy -$proxy->agent( $agent ); - -# start the proxy -$proxy->start(); - -=head1 NAME - -httprecorder - quick script for recording HTTP traffic - -=head1 SYNOPSIS - - httprecorder [ --port=8080 ] [ --file=http_traffic ] [ --help ] - - Options: - --help brief help message - --port Port number to use for the proxy server (default 8080) - --file filename to record the traffic to (default "http_traffic") - -=head1 DESCRIPTION - -This is a quick script for using HTTP::Recorder module. If you need customizing -please see the module itself - -=head1 Author - -Shmuel Fomberg - -=head1 COPYRIGHT AND LICENSE - -Copyright 2011 Shmuel Fomberg. - -This program is free software; Released under the GNU Public License. - -=cut +#!/usr/bin/env perl + +use strict; +use warnings; + +use HTTP::Proxy; +use HTTP::Recorder; +use Getopt::Long; +use Pod::Usage; + +my $port = 8080; +my $filename = "http_traffic"; +my $show_help = ''; + +GetOptions ("port=i" => \$port, "file=s" => \$filename, "help" => \$show_help); +pod2usage(1) if ($show_help); + +print "httprecorder - quick script for recording HTTP traffic\n"; + +my $proxy = HTTP::Proxy->new( port => $port ); +print "Proxy server host:port ".$proxy->host.":".$proxy->port."\n"; + +# create a new HTTP::Recorder object +my $agent = new HTTP::Recorder; +print "Proxy control panel http://" . $agent->control . "/\n"; + +# set the log file (optional) +$agent->file($filename); +print "Recording to file $filename\n"; + +# set HTTP::Recorder as the agent for the proxy +$proxy->agent( $agent ); + +# start the proxy +$proxy->start(); + +=head1 NAME + +httprecorder - quick script for recording HTTP traffic + +=head1 SYNOPSIS + + httprecorder [ --port=8080 ] [ --file=http_traffic ] [ --help ] + + Options: + --help brief help message + --port Port number to use for the proxy server (default 8080) + --file filename to record the traffic to (default "http_traffic") + +=head1 DESCRIPTION + +This is a quick script for using HTTP::Recorder module. If you need customizing +please see the module itself + +=head1 Author + +Shmuel Fomberg + +=head1 COPYRIGHT AND LICENSE + +Copyright 2011 Shmuel Fomberg. + +This program is free software; Released under the GNU Public License. + +=cut diff -Nru libhttp-recorder-perl-0.06/debian/changelog libhttp-recorder-perl-0.07/debian/changelog --- libhttp-recorder-perl-0.06/debian/changelog 2012-05-05 01:25:04.000000000 +0000 +++ libhttp-recorder-perl-0.07/debian/changelog 2013-05-25 17:00:39.000000000 +0000 @@ -1,3 +1,17 @@ +libhttp-recorder-perl (0.07-1) unstable; urgency=low + + [ Salvatore Bonaccorso ] + * Change Vcs-Git to canonical URI (git://anonscm.debian.org) + * Change search.cpan.org based URIs to metacpan.org based URIs + + [ Florian Schlichting ] + * Import Upstream version 0.07 + * Email change: Florian Schlichting -> fsfs@debian.org + * Bump Standards-Version to 3.9.4 (no change) + * Bump copyright years + + -- Florian Schlichting Sat, 25 May 2013 19:00:19 +0200 + libhttp-recorder-perl (0.06-1) unstable; urgency=low [ gregor herrmann ] diff -Nru libhttp-recorder-perl-0.06/debian/control libhttp-recorder-perl-0.07/debian/control --- libhttp-recorder-perl-0.06/debian/control 2012-05-05 01:25:04.000000000 +0000 +++ libhttp-recorder-perl-0.07/debian/control 2013-05-25 16:56:47.000000000 +0000 @@ -11,11 +11,11 @@ libtest-pod-perl Maintainer: Debian Perl Group Uploaders: Damyan Ivanov , - Florian Schlichting + Florian Schlichting Homepage: http://www.bitmistress.org/ -Vcs-Git: git://git.debian.org/pkg-perl/packages/libhttp-recorder-perl.git +Vcs-Git: git://anonscm.debian.org/pkg-perl/packages/libhttp-recorder-perl.git Vcs-Browser: http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libhttp-recorder-perl.git -Standards-Version: 3.9.3 +Standards-Version: 3.9.4 Package: libhttp-recorder-perl Architecture: all diff -Nru libhttp-recorder-perl-0.06/debian/copyright libhttp-recorder-perl-0.07/debian/copyright --- libhttp-recorder-perl-0.06/debian/copyright 2012-05-05 01:25:04.000000000 +0000 +++ libhttp-recorder-perl-0.07/debian/copyright 2013-05-25 16:58:18.000000000 +0000 @@ -1,17 +1,17 @@ Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ Upstream-Name: HTTP-Recorder Upstream-Contact: Shmuel Fomberg -Source: http://search.cpan.org/dist/HTTP-Recorder/ +Source: https://metacpan.org/release/HTTP-Recorder/ Files: * Copyright: 2003-2005 by Linda Julien - 2011, Shmuel Fomberg + 2011-2013, Shmuel Fomberg License: GPL-1+ Files: debian/* Copyright: 2005-2008, Ernesto Hernández-Novich (USB) 2007, Damyan Ivanov - 2012, Florian Schlichting + 2012-2013, Florian Schlichting License: Artistic or GPL-1+ License: Artistic diff -Nru libhttp-recorder-perl-0.06/debian/watch libhttp-recorder-perl-0.07/debian/watch --- libhttp-recorder-perl-0.06/debian/watch 2012-05-05 01:25:04.000000000 +0000 +++ libhttp-recorder-perl-0.07/debian/watch 2013-01-27 17:51:43.000000000 +0000 @@ -1,3 +1,3 @@ # format version number, currently 3; this line is compulsory! version=3 -http://search.cpan.org/dist/HTTP-Recorder/ .+/HTTP-Recorder-([[:digit:]].*)\.tar\.gz +https://metacpan.org/release/HTTP-Recorder/ .+/HTTP-Recorder-([[:digit:]].*)\.tar\.gz diff -Nru libhttp-recorder-perl-0.06/lib/HTTP/Recorder/Logger.pm libhttp-recorder-perl-0.07/lib/HTTP/Recorder/Logger.pm --- libhttp-recorder-perl-0.06/lib/HTTP/Recorder/Logger.pm 2011-07-04 13:02:59.000000000 +0000 +++ libhttp-recorder-perl-0.07/lib/HTTP/Recorder/Logger.pm 2013-04-05 03:44:04.000000000 +0000 @@ -1,254 +1,254 @@ -package HTTP::Recorder::Logger; - -use strict; -use warnings; -use LWP::MemberMixin; -use HTML::Entities qw(decode_entities); -our @ISA = qw( LWP::MemberMixin ); - -sub new { - my $class = shift; - - my %args = ( - @_ - ); - - my $self = bless ({}, ref ($class) || $class); - - $self->{'file'} = $args{'file'} || "/tmp/scriptfile"; - - $self->{agentname} = "\$agent"; - - return $self; -} - -sub agentname { shift->_elem('agentname', @_); } -sub file { shift->_elem('file', @_); } - -sub GetScript { - my $self = shift; - - if (open (SCRIPT, $self->{file})) { - my @script = - - -

Start Recording

-

Type a url into the browser's adddress field to begin recording. - -EOF - - return $content; -} - -sub get_recorder_content { - my $self = shift; - - my @script = $self->{logger}->GetScript(); - my $script = ""; - foreach my $line (@script) { - next unless $line; - $line =~ s/\n//g; - $script .= "$line\n"; - } - - my $content = < - - - - - -HTTP::Recorder Control Panel - - - - - - - - - - - - -
- Goto page: - -
-
- - - - - - - - - - - - - - - -
-Current Script: -
- -
- - - -
- - -
-
- -EOF - - return $content; -} - -sub script_update { - my $self = shift; - - my $url = "http://" . $self->control . "/"; - my $js = < - - -EOF -} - -=head1 Bugs, Missing Features, and other Oddities - -=head2 Javascript - -L can't play back Javascript actions, and -L doesn't record them. - -=head2 Why are my images corrupted? - -HTTP::Recorder only tries to rewrite responses that are of type -text/*, which it determines by reading the Content-Type header of the -HTTP::Response object. However, if the received image gives the wrong -Content-Type header, it may be corrupted by the recorder. While this -may not be pleasant to look at, it shouldn't have an effect on your -recording session. - -=head1 See Also - -See also L, L, L. - -=head1 Requests & Bugs - -Please submit any feature requests, suggestions, bugs, or patches at -http://rt.cpan.org/, or email to bug-HTTP-Recorder@rt.cpan.org. - -If you're submitting a bug of the type "X doesn't record correctly," -be sure to include a (preferably short and simple) HTML page that -demonstrates the problem, and a clear explanation of a) what it does -that it shouldn't, and b) what it should do instead. - -=head1 Author - -Copyright 2003-2005 by Linda Julien - -Maintained by Shmuel Fomberg - -Released under the GNU Public License. - -=cut - -1; +package HTTP::Recorder; + +our $VERSION = "0.07"; + +=head1 NAME + +HTTP::Recorder - record interaction with websites + +=head1 SYNOPSIS + +=head2 This module is deprecated + +It works by tagging links in a page, and then when a link is clicked +looking on the submitted tag to see which link was clicked + +It can not handle Javascript-created links or JS manipulation of the page +so it works only for fairly static websites + +For better options check out Selenium + +Patchs are welcome, and I'll fix bugs as much as I can, but please don't +expect me to implement new features + +=head2 Using HTTP::Recorder as a Web Proxy + +Set HTTP::Recorder as the user agent for a proxy, and it rewrites HTTP +responses so that additional requests can be recorded. + +=head3 The Proxy Script + +For quick start, run the httprecorder script + + httprecorder + +This will open a local proxy on port 8080, and will dump the recorded traffic +to a file named http_traffic in the current directory. use the -help parameter +for usage info + +Start the proxy script, then change the settings in your web browser +so that it will use this proxy for web requests. For more information +about proxy settings and the default port, see L. + +The script will be recorded in the specified file, and can be viewed +and modified via the control panel. + +For better control, use this example: + + #!/usr/bin/perl + + use HTTP::Proxy; + use HTTP::Recorder; + + my $proxy = HTTP::Proxy->new(); + + # create a new HTTP::Recorder object + my $agent = new HTTP::Recorder; + + # set the log file (optional) + $agent->file("/tmp/myfile"); + + # set HTTP::Recorder as the agent for the proxy + $proxy->agent( $agent ); + + # start the proxy + $proxy->start(); + +=head3 Start Recording + +Now you can use your browser as your normally would, and your actions +will be recorded in the file you specified. Alternatively, you can +start recording from the Control Panel. + +=head3 Using the Control Panel + +If you have Javascript enabled in your browser, go to the +L control URL (http://http-recorder by default), +optionally type a URL into the "Goto page" field, and click "Go". + +In the new window, interact with web sites as you normally do, +including typing a new address into the address field. The Control +Panel will be updated after each recorded action. + +The Control Panel allows you to modify, delete, or save your script. + +=head2 SSL sessions + +As of version 0.03, L can record SSL sessions. + +To begin recording an SSL session, go to the control URL +(http://http-recorder/ by default), and enter the initial URL. +Then, interact with the web site as usual. + +=head2 Script output + +By default, L outputs L scripts. + +However, you can override HTTP::Recorder::Logger to output other types +of scripts. + +=cut + +use strict; +use warnings; +use LWP::UserAgent; +use HTML::TokeParser; +use HTTP::Recorder::Logger; +use URI::Escape qw(uri_escape uri_unescape); +use URI::QueryParam; +use HTTP::Request::Params; + +our @ISA = qw( LWP::UserAgent ); + +=head1 Functions + +=head2 new + +Creates and returns a new L object, referred to as the 'agent'. + +=cut + +sub new { + my $class = shift; + + my %args = ( @_ ); + + my $self = $class->SUPER::new( %args ); + bless $self, $class; + + $self->{prefix} = $args{prefix} || "rec"; + $self->{control} = $args{control} || "http-recorder"; + $self->{logger} = $args{logger} || + new HTTP::Recorder::Logger(file => $args{file}); + $self->{ignore_favicon} = $args{ignore_favicon} || 1; + + return $self; +} + +=head2 $agent->prefix([$value]) + +Get or set the prefix string that L uses for rewriting +responses. + +=cut + +sub prefix { shift->_elem('prefix', @_); } + +=head2 $agent->control([$value]) + +Get or set the URL of the control panel. By default, the control URL +is 'http-recorder'. + +The control URL will display a control panel which will allow you to +view and edit the current script. + +=cut + +sub control { shift->_elem('control', @_); } + +=head2 $agent->logger([$value]) + +Get or set the logger object. The default logger is a +L, which generates L scripts. + +=cut + +sub logger { + my $self = shift; + $self->_elem('logger', @_); +} + +=head2 $agent->ignore_favicon([0|1]) + +Get or set ignore_favicon flag that causes L to skip +logging requests favicon.ico files. The value is 1 by default. + +=cut + +sub ignore_favicon { shift->_elem('ignore_favicon', @_); } + +=head2 $agent->file([$value]) + +Get or set the filename for generated scripts. The default is +'/tmp/scriptfile'. + +=cut + +sub file { + my $self = shift; + my $file = shift; + + $self->{logger}->file($file) if $file; +} + +sub send_request { + my $self = shift; + my $request = shift; + + my $response; + + # special handling if the URL is the control URL + if ($request->uri->host eq $self->{control}) { + + # get the arguments passed from the form + my $arghash; + $arghash = extract_values($request); + + # there may be an action we need to perform + if (exists $arghash->{updatescript}) { + my $script = $arghash->{ScriptContent}; + $self->{logger}->SetScript($script || ''); + } elsif (exists $arghash->{clearscript}) { + $self->{logger}->SetScript(""); + } + + my ($h, $content); + if (exists $arghash->{goto}) { + my $url = $arghash->{url}; + + if ($url) { + my $r = new HTTP::Request("GET", $url); + my $response = $self->send_request( $r ); + + return $response; + } else { + $h = HTTP::Headers->new(Content_Type => 'text/html'); + $content = $self->get_start_page(); + } + } elsif (exists $arghash->{savescript}) { + $h = HTTP::Headers->new(Content_Type => 'text/plain', + Content_Disposition => 'attachment; filename=recorder-script.pl'); + my @script = $self->{logger}->GetScript(); + $content = join('', @script); + } else { + $h = HTTP::Headers->new(Content_Type => 'text/html'); + $content = $self->get_recorder_content(); + } + + $response = HTTP::Response->new(200, + "", + $h, + $content, + ); + } else { + $request = $self->modify_request ($request) + unless $self->{ignore_favicon} + && $request->uri->path =~ /favicon\.ico$/i; + + $response = $self->SUPER::send_request( $request ); + + my $content_type = $response->headers->header('Content-type') || ""; + + # don't try to modify the content unless it's text/ + if ($content_type =~ m#^text/#i) { + $self->modify_response($response); + } + } + + return $response; +} + +sub modify_request { + my $self = shift; + my $request = shift; + + my $values = extract_values($request); + + # log the actions + my $action = $values->{"$self->{prefix}-action"}; + + my $referer = $request->headers->referer; + if (!$action) { + if (!$referer) { + my $uri = $self->unmodify($request->uri);; + + # log a blank line to give the code a little breathing room + $self->{logger}->LogLine(); + $self->{logger}->GotoPage(url => $uri); + } + } elsif ($action eq "follow") { + $self->{logger}->FollowLink(text => $values->{"$self->{prefix}-text"} || "", + index => $values->{"$self->{prefix}-index"} || "", + url => $values->{"$self->{prefix}-url"}); + } elsif ($action eq "submitform") { + my %fields; + my ($btn_name, $btn_value, $btn_number); + foreach my $param (keys %$values) { + my %fieldhash; + my ($fieldtype, $fieldname); + if ($param =~ /^$self->{prefix}-form(\d+)-(\w+)-(.*)$/) { + $fieldtype = $2; + $fieldname = $3; + + if ($fieldtype eq 'submit') { + next unless $values->{$fieldname}; + $btn_name = $fieldname; + $btn_value = $values->{$fieldname}; + } else { + next if ($fieldtype eq 'hidden'); + next unless $fieldname && exists $values->{$fieldname}; + $fieldhash{'name'} = $fieldname; + $fieldhash{'type'} = $fieldtype; + if (ref($values->{$fieldname}) eq 'ARRAY') { + my @tempvalues = @{$values->{$fieldname}}; + for (my $i = 0 ; $i < scalar @tempvalues ; $i++) { + $fieldhash{'value'} = $tempvalues[$i]; + my %temphash = %fieldhash; + $fields{"$fieldname-$i"} = \%temphash; + } + } else { + $fieldhash{'value'} = $values->{$fieldname}; + $fields{$fieldname} = \%fieldhash; + } + } + } + } + + $self->{logger}->SetFieldsAndSubmit(name => $values->{"$self->{prefix}-formname"}, + number => $values->{"$self->{prefix}-formnumber"}, + fields => \%fields, + button_name => $btn_name, + button_value => $btn_value); + + # log a blank line to give the code a little breathing room + $self->{logger}->LogLine(); + } + + # undo what we've done + $request->uri($self->unmodify($request->uri)); + $request->content($self->unmodify($request->content)); + + # reset the Content-Length (if needed) to prevent warnings from + # HTTP::Protocol + if ($action && ($action eq "submitform")) { + $request->headers->header('Content-Length' => length($request->content()) ); + + } + + my $https = $values->{"$self->{prefix}-https"}; + if ( $https && $https == 1) { + my $uri = $request->uri; + $uri->scheme('https') if $uri->scheme eq 'http'; + + $request = new HTTP::Request($request->method, + $uri, + $request->headers, + $request->content); + + } + + return $request; +} + +sub unmodify { + my $self = shift; + my $content = shift; + + return $content unless $content; + + # get rid of the arguments we added + my $prefix = $self->{prefix}; + + # workaround: the content can be a simple string + if (not ref $content) { + $content =~ s/(?:^|(?<=\&))\Q$prefix\E-[^=]+=[^\&]*(\&|$)//g; + return $content; + } + + for my $key ($content->query_param) { + if ($key =~ /^$prefix-/) { + $content->query_param_delete($key); + } + } + return $content; +} + +sub extract_values { + my $request = shift; + + my $parser = HTTP::Request::Params->new({ + req => $request, + }); + + # un-escape all params + for my $key (keys %{$parser->params}) { + $parser->params->{$key} = uri_unescape($parser->params->{$key}); + } + + return $parser->params; +} + +sub modify_response { + my $self = shift; + my $response = shift; + my $formcount = 0; + my $formnumber = 0; + my $linknumber = 1; + + $response->headers->push_header('Cache-Control', 'no-store, no-cache'); + $response->headers->push_header('Pragma', 'no-cache'); + + my $content = $response->content(); + my $p = HTML::TokeParser->new(\$content); + my $newcontent = ""; + my %links; + my $formname; + + my $js_href = 0; + my $in_head = 0; + my $basehref; + while (my $token = $p->get_token()) { + if (@$token[0] eq 'S') { + my $tagname = @$token[1]; + my $attrs = @$token[2]; + my $oldaction; + my $text; + + if ($tagname eq 'head') { + $in_head = 1; + } elsif ($in_head && $tagname eq 'base') { + $basehref = new URI($attrs->{'base'}); + } elsif (($tagname eq 'a' || $tagname eq 'link') && $attrs->{'href'}) { + my $t = $p->get_token(); + if (@$t[0] eq 'T') { + $text = @$t[1]; + } else { + undef $text; + } + $p->unget_token($t); + + # up the counter for links with the same text + my $index; + if (defined $text) { + $links{$text} = 0 if !(exists $links{$text}); + $links{$text}++; + $index = $links{$text}; + } else { + $index = $linknumber; + } + if ($attrs->{'href'} =~ m/^javascript:/i) { + $js_href = 1; + } else { + if ($tagname eq 'a') { + $attrs->{'href'} = + $self->rewrite_href($attrs->{'href'}, + $text, + $index, + $response->base); + } elsif ($tagname eq 'link') { + $attrs->{'href'} = + $self->rewrite_linkhref($attrs->{'href'}, + $response->base); + } + } + $linknumber++; + } elsif ($tagname eq 'form') { + $formcount++; + $formnumber++; + } + + # put the hidden field before the real field + # so that it won't be inside + if (!$js_href && $tagname ne 'form' && ($formcount == 1)) { + my ($formfield, $fieldprefix, $fieldtype, $fieldname); + $fieldprefix = "$self->{prefix}-form" . $formnumber; + $fieldtype = lc($attrs->{type} || 'unknown'); + if ($attrs->{name}) { + $fieldname = $attrs->{name}; + $formfield = ($fieldprefix . '-' . + $fieldtype . '-' . $fieldname); + $newcontent .= "\n"; + } + } + + $newcontent .= ("<".$tagname); + + # keep the attributes in their original order + my $attrlist = @$token[3]; + foreach my $attr (@$attrlist) { + # only rewrite if + # - it's not part of a javascript link + # - it's not a hidden field + $newcontent .= (" ".$attr."=\"".$attrs->{$attr}."\""); + } + $newcontent .= (">\n"); + if ($tagname eq 'head') { + # add the javascript to update the script, right after the head opening tag + $newcontent .= $self->script_update(); + } + if ($tagname eq 'form') { + if ($formcount == 1) { + $newcontent .= $self->rewrite_form_content($attrs->{name} || "", $formnumber, $response->base); + } + } + } elsif (@$token[0] eq 'E') { + my $tagname = @$token[1]; + if ($tagname eq 'head') { + if (!$basehref) { + $basehref = $response->base; + $basehref->scheme('http') if $basehref->scheme eq 'https'; + $newcontent .= "\n"; + } + $basehref = ""; + $in_head = 0; + } + $newcontent .= ("\n"); + if ($tagname eq 'form') { + $formcount--; + } elsif ($tagname eq 'a' || $tagname eq 'link') { + $js_href = 0; + } + } elsif (@$token[0] eq 'PI') { + $newcontent .= (@$token[2]); + } else { + $newcontent .= (@$token[1]); + } + } + + $response->content($newcontent); + + return; +} + +sub rewrite_href { + my $self = shift; + my $href = shift || ""; + my $text = shift || ""; + my $index = shift || 1; + my $base = shift; + + my $newhref = new URI($href); + my $prefix = $self->{prefix}; + + if ($base->scheme eq 'https') { + $newhref->query_param_append( "$prefix-https", 1); + $newhref->scheme('http'); + } + + # the original URL + $newhref->query_param_append( "$prefix-url", uri_escape($href)); + + # the action (i.e. follow link) + $newhref->query_param_append( "$prefix-action", 'follow'); + + # the link information + $text = uri_escape($text); # might have special characters + $newhref->query_param_append( "$prefix-text", $text); + $newhref->query_param_append( "$prefix-index", $index); + + return $newhref; +} + +sub rewrite_linkhref { + my $self = shift; + my $href = shift || ""; + my $base = shift; + + my $newhref = new URI($href); + my $prefix = $self->{prefix}; + + $newhref->query_param_append( "$prefix-https", 1) + if $base->scheme eq 'https'; + + # the original URL + $newhref->query_param_append( "$prefix-url", uri_escape($href)); + + # the action (i.e. don't record) + $newhref->query_param_append( "$prefix-action", 'norecord'); + + return $newhref; +} + +sub rewrite_form_content { + my $self = shift; + my $name = shift || ""; + my $number = shift; + my $url = shift; + my $fields; + + my $https = 1 if ($url->scheme eq 'https'); + + $fields .= ("{prefix}-action\" value=\"submitform\">\n"); + $fields .= ("{prefix}-formname\" value=\"$name\">\n"); + $fields .= ("{prefix}-formnumber\" value=\"$number\">\n"); + if ($https) { + $fields .= ("{prefix}-https\" value=\"$https\">\n"); + } + + return $fields; +} + +sub get_start_page { + my $self = shift; + + my $content = < + +HTTP::Recorder Start Page + + + +

Start Recording

+

Type a url into the browser's adddress field to begin recording. + +EOF + + return $content; +} + +sub get_recorder_content { + my $self = shift; + + my @script = $self->{logger}->GetScript(); + my $script = ""; + foreach my $line (@script) { + next unless $line; + $line =~ s/\n//g; + $script .= "$line\n"; + } + + my $content = < + + + + + +HTTP::Recorder Control Panel + + + + + + + + + + + + +
+ Goto page: + +
+
+ + + + + + + + + + + + + + + +
+Current Script: +
+ +
+ + + +
+ + +
+
+ +EOF + + return $content; +} + +sub script_update { + my $self = shift; + + my $url = "http://" . $self->control . "/"; + my $js = < + + +EOF +} + +=head1 Bugs, Missing Features, and other Oddities + +=head2 Javascript + +L can't play back Javascript actions, and +L doesn't record them. + +=head2 Why are my images corrupted? + +HTTP::Recorder only tries to rewrite responses that are of type +text/*, which it determines by reading the Content-Type header of the +HTTP::Response object. However, if the received image gives the wrong +Content-Type header, it may be corrupted by the recorder. While this +may not be pleasant to look at, it shouldn't have an effect on your +recording session. + +=head1 See Also + +See also L, L, L. + +=head1 Requests & Bugs + +Please submit any feature requests, suggestions, bugs, or patches at +http://rt.cpan.org/, or email to bug-HTTP-Recorder@rt.cpan.org. + +If you're submitting a bug of the type "X doesn't record correctly," +be sure to include a (preferably short and simple) HTML page that +demonstrates the problem, and a clear explanation of a) what it does +that it shouldn't, and b) what it should do instead. + +=head1 Author + +Copyright 2003-2005 by Linda Julien + +Maintained by Shmuel Fomberg + +Released under the GNU Public License. + +=cut + +1;