diff -Nru pgbadger-3.3/ChangeLog pgbadger-5.0/ChangeLog --- pgbadger-3.3/ChangeLog 2013-05-01 16:03:04.000000000 +0000 +++ pgbadger-5.0/ChangeLog 2014-02-06 15:29:04.000000000 +0000 @@ -1,7 +1,383 @@ +2014-02-05 version 5.0 + +This new major release adds some new features like incremental mode and SQL +queries times histogram. There is also a hourly graphic representation of the +count and average duration of top normalized queries. Same for errors or events, +you will be able to see graphically at which hours they are occuring the most +often. + +The incremental mode is an old request issued at PgCon Ottawa 2012 that concern +the ability to construct incremental reports with successive runs of pgBadger. +It is now possible to run pgbadger each days or even more, each hours, and have +cumulatives reports per day and per week. A top index page allow you to go +directly to the weekly and daily reports. + +This mode have been build with simplicity in mind so running pgbadger by cron +as follow: + + 0 23 * * * pgbadger -q -I -O /var/www/pgbadger/ /var/log/postgresql.log + +is enough to have daily and weelky reports viewable using your browser. + +You can take a look at a sample report at http://dalibo.github.io/pgbadger/demov5/index.html + +There's also a useful improvement to allow pgBadger to seek directly to the +last position in the same log file after a successive execution. This feature +is only available using the incremental mode or the -l option and parsing a +single log file. Let's say you have a weekly rotated log file and want to run +pgBadger each days. With 2GB of log per day, pgbadger was spending 5 minutes +per block of 2 GB to reach the last position in the log, so at the end of the +week this feature will save you 35 minutes. Now pgBadger will start parsing +new log entries immediatly. This feature is compatible with the multiprocess +mode using -j option (n processes for one log file). + +Histogram of query times is a new report in top queries slide that shows the +query times distribution during the analyzed period. For example: + + Range Count Percentage + -------------------------------------------- + 0-1ms 10,367,313 53.52% + 1-5ms 799,883 4.13% + 5-10ms 451,646 2.33% + 10-25ms 2,965,883 15.31% + 25-50ms 4,510,258 23.28% + 50-100ms 180,975 0.93% + 100-500ms 87,613 0.45% + 500-1000ms 5,856 0.03% + 1000-10000ms 2,697 0.01% + > 10000ms 74 0.00% + + +There is also some graphic and report improvements, like the mouse tracker +formatting that have been reviewed. It now shows a vertical crosshair and +all dataset values at a time when mouse pointer moves over series. Automatic +queries formatting has also been changed, it is now done on double click +event as simple click was painful when you want to copy some part of the +queries. + +The report "Simultaneous Connections" has been relabeled into "Established +Connections", it is less confusing as many people think that this is the number +of simultaneous sessions, which is not the case. It only count the number of +connections established at same time. + +Autovacuum reports now associate database name to the autovacuum and autoanalyze +entries. Statistics now refer to "dbname.schema.table", previous versions was only +showing the pair "schema.table". + +This release also adds Session peak information and a report about Simultaneous +sessions. Parameters log_connections and log_disconnections must be enabled in +postgresql.conf. + +Complete ChangeLog: + + - Fix size of SQL queries columns to prevent exceeding screen width. + - Add new histogram reports on top normalized queries and top errors + or event. It shows at what hours and in which quantity the queries + or errors appears. + - Add seeking to last parser position in log file in incremental mode. + This prevent parsing all the file to find the last line parse from + previous run. This only works when parsing a single flat file, -j + option is permitted. Thanks to ioguix for the kick. + - Rewrite reloading of last log time from binary files. + - Fix missing statistics of last parsed queries in incremental mode. + - Fix bug in incremental mode that prevent reindexing a previous day. + Thanks to Martin Prochazka for the great help. + - Fix missing label "Avg duration" on column header in details of Most + frequent queries (N). + - Add vertical crosshair on graph. + - Fix case where queries and events was not updated when using -b and + -e command line. Thanks to Nicolas Thauvin for the report. + - Fix week sorting on incremental report main index page. Thanks to + Martin Prochazka for the report. + - Add "Histogram of query times" report to show statistics like + 0-100ms : 80%, 100-500ms :14%, 500-1000ms : 3%, > 1000ms : 1%. + Thanks to tmihail for the feature request. + - Format mouse tracker on graphs to show all dataset value at a time. + - Add control of -o vs -O option with incremental mode to prevent + wrong use. + - Change log level of missing LAST_PARSED.tmp file to WARNING and + add a HINT. + - Update copyright date to 2014 + - Fix empty reports of connections. Thanks to Reeshna Ramakrishnan + for the report. + - Fix display of connections peak when no connection was reported. + - Fix warning on META_MERGE for ExtUtils::MakeMaker < 6.46. Thanks + to Julien Rouhaud for the patch. + - Add documentation about automatic incremental mode. + - Add incremental mode to pgBadger. This mode will build a report + per day and a cumulative report per week. It also create an index + interface to easiest access to the different report. Must be run, + for example, as: + pgbadger /var/log/postgresql.log.1 -I -O /var/www/pgbadger/ + after a daily PostgreSQL log file rotation. + - Add -O | --outdir path to specify the directory where out file + must be saved. + - Automatic queries formatting is now done on double click event, + simple click was painful when you want to copy some part of the + queries. Thanks to Guillaume Smet for the feature request. + - Remove calls of binmode to force html file output to be utf8 as + there is some bad side effect. Thanks to akorotkov for the report. + - Remove use of Time::HiRes Perl module as some distributions does + not include this module by default in core Perl install. + - Fix "Wide character in print" Perl message by setting binmode + to :utf8. Thanks to Casey Allen Shobe for the report. + - Fix application name search regex to handle application name with + space like "pgAdmin III - Query Tool". + - Fix wrong timestamps saved with top queries. Thanks to Herve Werner + for the report. + - Fix missing logs types statitics when using binary mode. Thanks to + Herve Werner for the report. + - Fix Queries by application table column header: Database replaced + by Application. Thanks to Herve Werner for the report. + - Add "Max number of times the same event was reported" report in + Global stats Events tab. + - Replace "Number of errors" by "Number of ERROR entries" and add + "Number of FATAL entries". + - Replace "Number of errors" by "Number of events" and "Total errors + found" by "Total events found" in Events reports. Thanks to Herve + Werner for the report. + - Fix title error in Sessions per database. + - Fix clicking on the info link to not go back to the top of the page. + Thanks to Guillaume Smet for the report and solution. + - Fix incremental report from binary output where binary data was not + loaded if no queries were present in log file. Thanks to Herve Werner + for the report. + - Fix parsing issue when log_error_verbosity = verbose. Thanks to vorko + for the report. + - Add Session peak information and a report about Simultaneous sessions. + log_connections+log_disconnections must be enabled in postgresql.conf. + - Fix wrong requests number in Queries by user and by host. Thanks to + Jehan-Guillaume de Rorthais for the report. + - Fix issue with rsyslog format failing to parse logs. Thanks to Tim + Sampson for the report. + - Associate autovacuum and autoanalyze log entry to the corresponding + database name. Thanks to Herve Werner for the feature request. + - Change "Simultaneous Connections" label into "Established Connections", + it is less confusing as many people think that this is the number of + simultaneous sessions, which is not the case. It only count the number + of connections established at same time. Thanks to Ronan Dunklau for + the report. + +2013-11-08 version 4.1 + +This release fixes two major bugs and some others minor issues. There's also a +new command line option --exclude-appname that allow exclusion from the report +of queries generated by a specific program, like pg_dump. Documentation have +been updated with a new chapter about building incremental reports. + + - Add log_autovacuum_min_duration into documentation in chapter about + postgresql configuration directives. Thanks to Herve Werner for the + report. + - Add chapter about "Incremental reports" into documentation. + - Fix reports with per minutes average where last time fraction was + not reported. Thanks to Ludovic Levesque and Vincent Laborie for the + report. + - Fix unterminated comment in information popup. Thanks to Ronan + Dunklau for the patch. + - Add --exclude-appname command line option to eliminate unwanted + traffic generated by a specific application. Thanks to Steve Crawford + for the feature request. + - Allow external links use into URL to go to a specific report. Thanks + to Hubert depesz Lubaczewski for the feature request. + - Fix empty reports when parsing compressed files with the -j option + which is not allowed with compressed file. Thanks to Vincent Laborie + for the report. + - Prevent progress bar length to increase after 100% when real size is + greater than estimated size (issue found with huge compressed file). + - Correct some spelling and grammar in ChangeLog and pgbadger. Thanks + to Thom Brown for the patch. + - Fix major bug on SQL traffic reports with wrong min value and bad + average value on select reports, add min/max for select queries. + Thanks to Vincent Laborie for the report. + +2013-10-31 - Version 4.0 + +This major release is the "Say goodbye to the fouine" release. With a full +rewrite of the reports design, pgBadger has now turned the HTML reports into +a more intuitive user experience and professional look. + +The report is now driven by a dynamic menu with the help of the embedded +Bootstrap library. Every main menu corresponds to a hidden slide that is +revealed when the menu or one of its submenus is activated. There's +also the embedded font Font Awesome webfont to beautify the report. + +Every statistics report now includes a key value section that immediately +shows you some of the relevant information. Pie charts have also been +separated from their data tables using two tabs, one for the chart and the +other one for the data. + +Tables reporting hourly statistics have been moved to a multiple tabs report +following the data. This is used with General (queries, connections, sessions), +Checkpoints (buffer, files, warnings), Temporary files and Vacuums activities. + +There's some new useful information shown in the key value sections. Peak +information shows the number and datetime of the highest activity. Here is the +list of those reports: + + - Queries peak + - Read queries peak + - Write queries peak + - Connections peak + - Checkpoints peak + - WAL files usage Peak + - Checkpoints warnings peak + - Temporary file size peak + - Temporary file number peak + +Reports about Checkpoints and Restartpoints have been merged into a single report. +These are almost one in the same event, except that restartpoints occur on a slave +cluster, so there was no need to distinguish between the two. + +Recent PostgreSQL versions add additional information about checkpoints, the +number of synced files, the longest sync and the average of sync time per file. +pgBadger collects and shows this information in the Checkpoint Activity report. + +There's also some new reports: + + - Prepared queries ratio (execute vs prepare) + - Prepared over normal queries + - Queries (select, insert, update, delete) per user/host/application + - Pie charts for tables with the most tuples and pages removed during vacuum. + +The vacuum report will now highlight the costly tables during a vacuum or +analyze of a database. + +The errors are now highlighted by a different color following the level. +A LOG level will be green, HINT will be yellow, WARNING orange, ERROR red +and FATAL dark red. + +Some changes in the binary format are not backward compatible and the option +--client has been removed as it has been superseded by --dbclient for a long time now. + +If you are running a pg_dump or some batch process with very slow queries, your +report analysis will be hindered by those queries having unwanted prominence in the +report. Before this release it was a pain to exclude those queries from the +report. Now you can use the --exclude-time command line option to exclude all +traces matching the given time regexp from the report. For example, let's say +you have a pg_dump at 13:00 each day during half an hour, you can use pgbadger +as follows: + + pgbadger --exclude-time "2013-09-.* 13:.*" postgresql.log + +If you are also running a pg_dump at night, let's say 22:00, you can write it +as follows: + + pgbadger --exclude-time '2013-09-\d+ 13:[0-3]' --exclude-time '2013-09-\d+ 22:[0-3]' postgresql.log + +or more shortly: + + pgbadger --exclude-time '2013-09-\d+ (13|22):[0-3]' postgresql.log + +Exclude time always requires the iso notation yyyy-mm-dd hh:mm:ss, even if log +format is syslog. This is the same for all time-related options. Use this option +with care as it has a high cost on the parser performance. + + +2013-09-17 - version 3.6 + +Still an other version in 3.x branch to fix two major bugs in vacuum and checkpoint +graphs. Some other minors bugs has also been fixed. + + - Fix grammar in --quiet usage. Thanks to stephen-a-ingram for the report. + - Fix reporting period to starts after the last --last-parsed value instead + of the first log line. Thanks to Keith Fiske for the report. + - Add --csv-separator command line usage to documentation. + - Fix CSV log parser and add --csv-separator command line option to allow + change of the default csv field separator, coma, in any other character. + - Avoid "negative look behind not implemented" errors on perl 5.16/5.18. + Thanks to Marco Baringer for the patch. + - Support timestamps for begin/end with fractional seconds (so it'll handle + postgresql's normal string representation of timestamps). + - When using negative look behind set sub-regexp to -i (not case insensitive) + to avoid issues where some upper case letter sequence, like SS or ST. + - Change shebang from /usr/bin/perl to /usr/bin/env perl so that user-local + (perlbrew) perls will get used. + - Fix empty graph of autovacuum and autoanalyze. + - Fix checkpoint graphs that was not displayed any more. + + +2013-07-11 - Version 3.5 + +Last release of the 3.x branch, this is a bug fix release that also adds some +pretty print of Y axis number on graphs and a new graph that groups queries +duration series that was shown as second Y axis on graphs, as well as a new +graph with number of temporary file that was also used as second Y axis. + + - Split temporary files report into two graphs (files size and number + of file) to no more used a second Y axis with flotr2 - mouse tracker + is not working as expected. + - Duration series representing the second Y axis in queries graph have + been removed and are now drawn in a new "Average queries duration" + independant graph. + - Add pretty print of numbers in Y axis and mouse tracker output with + PB, TB, GB, KB, B units, and seconds, microseconds. Number without + unit are shown with P, T, M, K suffix for easiest very long number + reading. + - Remove Query type reports when log only contains duration. + - Fix display of checkpoint hourly report with no entry. + - Fix count in Query type report. + - Fix minimal statistics output when nothing was load from log file. + Thanks to Herve Werner for the report. + - Fix several bug in log line parser. Thanks to Den Untevskiy for the + report. + - Fix bug in last parsed storage when log files was not provided in the + right order. Thanks to Herve Werner for the report. + - Fix orphan lines wrongly associated to previous queries instead of + temporary file and lock logged statement. Thanks to Den Untevskiy for + the report. + - Fix number of different samples shown in events report. + - Escape HTML tags on error messages examples. Thanks to Mael Rimbault + for the report. + - Remove some temporary debug informations used with some LOG messages + reported as events. + - Fix several issues with restartpoint and temporary files reports. + Thanks to Guillaume Lelarge for the report. + - Fix issue when an absolute path was given to the incremental file. + Thanks to Herve Werner for the report. + - Remove creation of incremental temp file $tmp_last_parsed when not + running in multiprocess mode. Thanks to Herve Werner for the report. + + +2013-06-18 - Version 3.4 + +This release adds lot of graphic improvements and a better rendering with logs +over few hours. There's also some bug fixes especially on report of queries that +generate the most temporary files. + + - Update flotr2.min.js to latest github code. + - Add mouse tracking over y2axis. + - Add label/legend information to ticks displayed on mouseover graphs. + - Fix documentation about log_statement and log_min_duration_statement. + Thanks to Herve Werner for the report. + - Fix missing top queries for locks and temporary files in multiprocess + mode. + - Cleanup code to remove storage of unused information about connection. + - Divide the huge dump_as_html() method with one method per each report. + - Checkpoints, restart points and temporary files are now drawn using a + period of 5 minutes per default instead of one hour. Thanks to Josh + Berkus for the feature request. + - Change fixed increment of one hour to five minutes on queries graphs + "SELECT queries" and "Write queries". Remove graph "All queries" as, + with a five minutes increment, it duplicates the "Queries per second". + Thanks to Josh Berkus for the feature request. + - Fix typos. Thanks to Arsen Stasic for the patch. + - Add default HTML charset to utf-8 and a command line option --charset + to be able to change the default. Thanks to thomas hankeuhh for the + feature request. + - Fix missing temporary files query reports in some conditions. Thanks + to Guillaume Lelarge and Thomas Reiss for the report. + - Fix some parsing issue with log generated by pg 7.4. + - Update documentation about missing new reports introduced in previous + version 3.3. + +Note that it should be the last release of the 3.x branch unless there's major +bug fixes, but next one will be a major release with a completely new design. + + 2013-05-01 - Version 3.3 This release adds four more useful reports about queries that generate locks and -temporary files. An other new report about restartpoint on slaves and several +temporary files. An other new report about restart point on slaves and several bugs fix or cosmetic change. Support to parallel processing under Windows OS has been removed. diff -Nru pgbadger-3.3/debian/bzr-builder.manifest pgbadger-5.0/debian/bzr-builder.manifest --- pgbadger-3.3/debian/bzr-builder.manifest 1970-01-01 00:00:00.000000000 +0000 +++ pgbadger-5.0/debian/bzr-builder.manifest 2014-02-17 09:15:16.000000000 +0000 @@ -0,0 +1,2 @@ +# bzr-builder format 0.3 deb-version {debupstream}-0~5 +lp:~stub/ubuntu/precise/pgbadger/devel revid:stuart@stuartbishop.net-20140217090428-83ciqqie8u8l5b3d diff -Nru pgbadger-3.3/debian/changelog pgbadger-5.0/debian/changelog --- pgbadger-3.3/debian/changelog 2014-02-17 09:15:16.000000000 +0000 +++ pgbadger-5.0/debian/changelog 2014-02-17 09:15:16.000000000 +0000 @@ -1,3 +1,15 @@ +pgbadger (5.0-0~5~ubuntu14.04.1) trusty; urgency=low + + * Auto build. + + -- Stuart Bishop Mon, 17 Feb 2014 09:13:47 +0000 + +pgbadger (5.0-0) precise; urgency=low + + * New upstream release. + + -- Stuart Bishop (Work) Mon, 17 Feb 2014 16:02:24 +0700 + pgbadger (3.3-2) unstable; urgency=low * Fixed debian/copyright file diff -Nru pgbadger-3.3/doc/pgBadger.pod pgbadger-5.0/doc/pgBadger.pod --- pgbadger-3.3/doc/pgBadger.pod 2013-05-01 16:03:04.000000000 +0000 +++ pgbadger-5.0/doc/pgBadger.pod 2014-02-06 15:29:03.000000000 +0000 @@ -27,6 +27,8 @@ -G | --nograph : disable graphs on HTML output. Enable by default. -h | --help : show this message and exit. -i | --ident name : programname used as syslog ident. Default: postgres + -I | --incremental : use incremental mode, reports will be generated by + days in a separate directory, --outdir must be set. -j | --jobs number : number of jobs to run on parallel on each log file. Default is 1, run as single process. -J | --Jobs number : number of log file to parse in parallel. Default @@ -42,6 +44,7 @@ -o | --outfile filename: define the filename for output. Default depends on the output format: out.html, out.txt or out.tsung. To dump output to stdout use - as filename. + -O | --outdir path : directory where out file must be saved. -p | --prefix string : give here the value of your custom log_line_prefix defined in your postgresql.conf. Only use it if you aren't using one of the standard prefixes specified @@ -49,7 +52,7 @@ includes additional variables like client ip or application name. See examples below. -P | --no-prettify : disable SQL queries prettify formatter. - -q | --quiet : don't print anything to stdout, even not a progress bar. + -q | --quiet : don't print anything to stdout, not even a progress bar. -s | --sample number : number of query samples to store/display. Default: 3 -S | --select-only : use it if you want to report select queries only. -t | --top number : number of queries to store/display. Default: 20 @@ -84,6 +87,13 @@ --disable-temporary : do not generate temporary report. --disable-checkpoint : do not generate checkpoint report. --disable-autovacuum : do not generate autovacuum report. + --charset : used to set the HTML charset to be used. Default: utf-8. + --csv-separator : used to set the CSV field separator, default: , + --exclude-time regex : any timestamp matching the given regex will be + excluded from the report. Example: "2013-04-12 .*" + You can use this option multiple times. + --exclude-appname name : exclude entries for the specified application name + from report. Example: "pg_dump". Examples: @@ -120,21 +130,44 @@ This supposes that your log file and HTML report are also rotated every week. +Or better, use the auto-generated incremental reports: + + 0 4 * * * /usr/bin/pgbadger -I -q /var/log/postgresql/postgresql.log.1 \ + -O /var/www/pg_reports/ + +will generate a report per day and per week in the given output directory. + +If you have a pg_dump at 23:00 and 13:00 each day during half an hour, you can +use pgbadger as follow to exclude these periods from the report: + + pgbadger --exclude-time "2013-09-.* (23|13):.*" postgresql.log + +This will help to not have all COPY order on top of slowest queries. You can +also use --exclude-appname "pg_dump" to solve this problem in a more simple way. + =head1 DESCRIPTION -pgBadger is a PostgreSQL log analyzer built for speed with fully detailed reports from your PostgreSQL log file. It's a single and small Perl script that aims to replace and out-perform the old PHP script pgFouine. +pgBadger is a PostgreSQL log analyzer build for speed with fully detailed reports from your PostgreSQL log file. It's a single and small Perl script that outperform any other PostgreSQL log analyzer. -By the way, we would like to thank Guillaume Smet for all the work he has done on this really nice tool. We've been using it a long time, it is a really great tool! +It is written in pure Perl language and uses a javascript library (flotr2) to draw graphs so that you don't need to install any additional Perl modules or other packages. Furthermore, this library gives us more features such as zooming. pgBadger also uses the Bootstrap javascript library and the FontAwesome webfont for better design. Everything is embedded. -pgBadger is written in pure Perl language. It uses a Javascript library to draw graphs so that you don't need additional Perl modules or any other package to install. Furthermore, this library gives us additional features, such as zooming. +pgBadger is able to autodetect your log file format (syslog, stderr or csvlog). It is designed to parse huge log files as well as gzip compressed file. See a complete list of features below. -pgBadger is able to autodetect your log file format (syslog, stderr or csvlog). It is designed to parse huge log files, as well as gzip, zip or bzip2 compressed files. See a complete list of features below. +All charts are zoomable and can be saved as PNG images. + +You can also limit pgBadger to only report errors or remove any part of the report using command line options. + +pgBadger supports any custom format set into log_line_prefix of your postgresql.conf file provide that you use the %t, %p and %l patterns. + +pgBadger allow parallel processing on a single log file and multiple files through the use of the -j option and the number of CPUs as value. + +If you want to save system performance you can also use log_duration instead of log_min_duration_statement to have reports on duration and number of queries only. =head1 FEATURE pgBadger reports everything about your SQL queries: - Overall statistics. + Overall statistics The most frequent waiting queries. Queries that waited the most. Queries generating the most temporary files. @@ -143,15 +176,20 @@ Queries that took up the most time. The most frequent queries. The most frequent errors. + Histogram of query times. + +The following reports are also available with hourly charts divide by periods of +five minutes: + + SQL queries statistics. + Temporary file statistics. + Checkpoints statistics. + Autovacuum and autoanalyze statistics. -The following reports are also available with hourly charts: +There's also some pie reports of distribution about: - Hourly queries statistics. - Hourly temporary file statistics. - Hourly checkpoints statistics. - Hourly restartpoints statistics. Locks statistics. - Queries by type (select/insert/update/delete). + ueries by type (select/insert/update/delete). Distribution of queries type per database/application Sessions per database/user/client. Connections per database/user/client. @@ -159,6 +197,8 @@ All charts are zoomable and can be saved as PNG images. SQL queries reported are highlighted and beautified automatically. +You can also have incremental reports with one report per day and a cumulative report per week. + =head1 REQUIREMENT pgBadger comes as a single Perl script - you do not need anything other than a modern Perl distribution. Charts are rendered using a Javascript library so you don't need anything. Your browser will do all the work. @@ -187,6 +227,29 @@ Note that multiprocessing can not be used with compressed files or CSV files as well as under Windows platform. +=head1 INSTALLATION + +Download the tarball from github and unpack the archive as follow: + + tar xzf pgbadger-4.x.tar.gz + cd pgbadger-4.x/ + perl Makefile.PL + make && sudo make install + +This will copy the Perl script pgbadger to /usr/local/bin/pgbadger by default and the +man page into /usr/local/share/man/man1/pgbadger.1. Those are the default installation +directories for 'site' install. + +If you want to install all under /usr/ location, use INSTALLDIRS='perl' as an argument +of Makefile.PL. The script will be installed into /usr/bin/pgbadger and the manpage +into /usr/share/man/man1/pgbadger.1. + +For example, to install everything just like Debian does, proceed as follows: + + perl Makefile.PL INSTALLDIRS=vendor + +By default INSTALLDIRS is set to site. + =head1 POSTGRESQL CONFIGURATION You must enable and set some configuration directives in your postgresql.conf @@ -197,8 +260,8 @@ log_min_duration_statement = 0 Here every statement will be logged, on busy server you may want to increase -this value to only log queries with a higher duration time. Note that if you -have log_statement set to 'all' nothing will be logged with log_line_prefix. +this value to only log queries with a higher duration time. Note that if you have +log_statement set to 'all' nothing will be logged through log_min_duration_statement. See next chapter for more information. With 'stderr' log format, log_line_prefix must be at least: @@ -228,6 +291,7 @@ log_disconnections = on log_lock_waits = on log_temp_files = 0 + log_autovacuum_min_duration = 0 Do not enable log_statement as their log format will not be parsed by pgBadger. @@ -253,7 +317,7 @@ set to 'all' nothing will be logged with log_line_prefix. -=head1 Parallel processing +=head1 PARALLEL PROCESSING To enable parallel processing you just have to use the -j N option where N is the number of cores you want to use. @@ -306,34 +370,78 @@ not remove those files unless pgbadger is not running. They are all named with the following template tmp_pgbadgerXXXX.bin so they can be easily identified. -=head1 INSTALLATION +=head1 INCREMENTAL REPORTS -Download the tarball from github and unpack the archive as follow: +pgBadger include an automatic incremental report mode using option -I or +--incremental. When running in this mode, pgBadger will generate one report +per day and a cumulative report per week. Output is first done in binary +format into the mandatory output directory (see option -O or --outdir), +then in HTML format for daily and weekly reports with a main index file. - tar xzf pgbadger-3.x.tar.gz - cd pgbadger-3.x/ - perl Makefile.PL - make && sudo make install +The main index file will show a dropdown menu per week with a link to the week +report and links to daily reports of this week. -This will copy the Perl script pgbadger to /usr/local/bin/pgbadger by default and the man page into /usr/local/share/man/man1/pgbadger.1. Those are the default installation directories for 'site' install. +For example, if you run pgBadger as follow based on a daily rotated file: -If you want to install all under /usr/ location, use INSTALLDIRS='perl' as an argument of Makefile.PL. The script will be installed into /usr/bin/pgbadger and the manpage into /usr/share/man/man1/pgbadger.1. + 0 4 * * * /usr/bin/pgbadger -I -q /var/log/postgresql/postgresql.log.1 \ + -O /var/www/pg_reports/ -For example, to install everything just like Debian does, proceed as follows: +you will have all daily and weekly reports for the full running period. - perl Makefile.PL INSTALLDIRS=vendor +In this mode pgBagder will create an automatic incremental file into the +output directory, so you don't have to use the -l option unless you want +to change the path of that file. This mean that you can run pgBadger in +this mode each days on a log file rotated each week, it will not count +the log entries twice. -By default INSTALLDIRS is set to site. +=head1 BINARY FORMAT + +Using the binary format it is possible to create custom incremental and +cumulative reports. For example, if you want to refresh a pgbadger report +each hour from a daily PostgreSQl log file, you can proceed by running each +hour the following commands: + + pgbadder --last-parsed .pgbadger_last_state_file -o sunday/hourX.bin /var/log/pgsql/postgresql-Sun.log + +to generate the incremental data files in binary format. And to generate the fresh HTML +report from that binary file: + + pgbadder sunday/*.bin + +Or an other example, if you have one log file per hour and you want a reports to be +rebuild each time the log file is switched. Proceed as follow: + + pgbadger -o day1/hour01.bin /var/log/pgsql/pglog/postgresql-2012-03-23_10.log + pgbadger -o day1/hour02.bin /var/log/pgsql/pglog/postgresql-2012-03-23_11.log + pgbadger -o day1/hour03.bin /var/log/pgsql/pglog/postgresql-2012-03-23_12.log + ... + +When you want to refresh the HTML report, for example each time after a new binary file +is generated, just do the following: + + pgbadger -o day1_report.html day1/*.bin + +Adjust the commands following your needs. =head1 AUTHORS -pgBadger is an original work from Gilles Darold. It is maintained by the good folk at Dalibo and everyone who wants to contribute. +pgBadger is an original work from Gilles Darold. + +The pgBadger logo is an original creation of Damien Clochard. + +The pgBadger v4.x design comes from the "Art is code" company. + +This web site is a work of Gilles Darold. + +pgBadger is maintained by Gilles Darold, the good folks at Dalibo, and every one who wants to contribute. + +Many people have contributed to pgBadger, they are all quoted in the Changelog file. =head1 LICENSE pgBadger is free software distributed under the PostgreSQL Licence. -Copyright (c) 2012-2013, Dalibo +Copyright (c) 2012-2014, Dalibo A modified version of the SQL::Beautify Perl Module is embedded in pgBadger with copyright (C) 2009 by Jonas Kramer and is published under the terms of diff -Nru pgbadger-3.3/LICENSE pgbadger-5.0/LICENSE --- pgbadger-3.3/LICENSE 2013-05-01 16:03:04.000000000 +0000 +++ pgbadger-5.0/LICENSE 2014-02-06 15:29:03.000000000 +0000 @@ -1,4 +1,4 @@ -Copyright (c) 2012-2013, Dalibo +Copyright (c) 2012-2014, Dalibo Permission to use, copy, modify, and distribute this software and its documentation for any purpose, without fee, and without a written agreement diff -Nru pgbadger-3.3/Makefile.PL pgbadger-5.0/Makefile.PL --- pgbadger-3.3/Makefile.PL 2014-02-17 09:15:16.000000000 +0000 +++ pgbadger-5.0/Makefile.PL 2014-02-17 09:15:16.000000000 +0000 @@ -34,7 +34,7 @@ 'DESTDIR' => $DESTDIR, 'INSTALLDIRS' => $INSTALLDIRS, 'clean' => {}, - 'META_MERGE' => { + ($ExtUtils::MakeMaker::VERSION < 6.46 ? () : 'META_MERGE' => { resources => { homepage => 'http://projects.dalibo.org/pgbadger', repository => { @@ -44,5 +44,6 @@ }, }, } + ) ); diff -Nru pgbadger-3.3/pgbadger pgbadger-5.0/pgbadger --- pgbadger-3.3/pgbadger 2013-05-01 16:03:04.000000000 +0000 +++ pgbadger-5.0/pgbadger 2014-02-06 15:29:03.000000000 +0000 @@ -1,4 +1,4 @@ -#!/usr/bin/perl +#!/usr/bin/env perl #------------------------------------------------------------------------------ # # pgBadger - Advanced PostgreSQL log analyzer @@ -35,16 +35,15 @@ use File::Basename; use Storable qw(store_fd fd_retrieve); use Time::Local 'timegm_nocheck'; -use POSIX qw(locale_h sys_wait_h _exit); +use POSIX qw(locale_h sys_wait_h _exit strftime); setlocale(LC_NUMERIC, ''); setlocale(LC_ALL, 'C'); use File::Spec qw/ tmpdir /; use File::Temp qw/ tempfile /; use IO::Handle; use IO::Pipe; -use Time::HiRes qw/usleep/; -$VERSION = '3.3'; +$VERSION = '5.0'; $SIG{'CHLD'} = 'DEFAULT'; @@ -54,6 +53,14 @@ my $parent_pid = $$; my $interrupt = 0; my $tmp_last_parsed = ''; +my @SQL_ACTION = ('SELECT', 'INSERT', 'UPDATE', 'DELETE'); +my $graphid = 1; +my $NODATA = '
NO DATASET
'; + +my $pgbadger_logo = + ''; +my $pgbadger_ico = + 'data:image/x-icon;base64,AAABAAEAEBAAAAEAIABoBAAAFgAAACgAAAAQAAAAIAAAAAEAIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIiIiAAAAAAA////BxQUFH0FBQWyExMT4wQEBM4ODg7DFRUVe/T09AsAAAAAg4ODAAAAAAAAAAAAAAAAAFJSUgAAAAAAKioqcQICAv9aWlr/MjIy/omJif6Li4v+HR0d/ldXV/8DAwP/Ly8veAAAAABZWVkAAAAAAF9fXwAAAAAAKioqvh4eHv/n5+f/qqqq/s3Nzf4FBQX/AAAA/5KSkv+Kior/6urq/h0dHf8uLi7DAAAAAG1tbQAAAAAAJiYmeCEhIf//////6enp/iMjI//t7e3/l5eX/2dnZ//U1NT/S0tL/9XV1f7/////Hx8f/y8vL34AAAAAvb29HAEBAf/s7Oz+3d3d/j09Pf9TU1P/JCQk//z8/P//////oaGh/xsbG/95eXn/+Pj4/unp6f4DAwP/vLy8GiQkJJJjY2P95+fn/1NTU/9ubm7/UFBQ/yEhIf/m5ub//f39/yEhIf8SEhL/h4eH/4yMjP//////YWFh/w4ODn8CAgLLv7+//09PT/68vLz/ZGRk/zQ0NP8sLCz/9/f3/+Hh4f9gYGD/hYWF/0VFRf9nZ2f//v7+/rm5uf4CAgLLCwsL9efn5/9zc3P/dHR0/wUFBf8MDAz/ampq///////9/f3/FBQU/0BAQP88PDz/WFhY//Ly8v/j4+P+AAAA1QAAANyFhYX+5+fn/09PT/8AAAD/AgIC/9jY2P/+/v7//////y4uLv8AAAD/AAAA/5ycnP+Kior/6Ojo/gQEBOkAAADKVVVV/xgYGP8AAAD/AAAA/yAgIP///////v7+//////87Ozv/AAAA/wAAAP8wMDD/OTk5/7+/v/4AAADMJiYmn5mZmf0AAAD+AAAA/wAAAP9BQUH///////7+/v//////KCgo/wAAAP8AAAD/AAAA/5CQkP9ubm7+MjIynzMzMyFFRUX/zc3N/zw8PP4DAwP/BgYG/9bW1v//////mJiY/wAAAP8AAAD/AAAA/wAAAP5gYGD+BAQE/0JCQiMAAAAAIiIishEREf719fX+6Ojo/3Jycv9XV1f/ZGRk/3d3d/+5ubn/WVlZ/2dnZ/7MzMz+xMTE/h4eHrIAAAAATU1NAAAAAAALCwvhSUlJ/v7+/v7////+//////////////////////////+YmJj/QUFB/g8PD+EAAAAAVVVVAAAAAAAAAAAA////AR8fH70GBgb/goKC/tzc3P7////+/////9zc3P6CgoL+BgYG/xsbG7YAAAAAAAAAAAAAAAAAAAAAAAAAAD8/PwAAAAAARkZGNBkZGa4ODg73AAAA/wAAAP8NDQ31GhoasWFhYUAAAAAAOzs7AAAAAAAAAAAA/D8AAPAPAADAAwAAwAMAAIABAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAQAAgAEAAMADAADgBwAA+B8AAA'; #### # method used to fork as many child as wanted @@ -82,34 +89,6 @@ exit &$coderef(); } -# Informa the parent that it should stop iterate on parsing other files -sub stop_parsing -{ - $interrupt = 1; -} - -# With multiprocess we need to wait all childs -sub wait_child -{ - my $sig = shift; - print STDERR "Received terminating signal ($sig).\n"; - if ($^O !~ /MSWin32|dos/i) { - 1 while wait != -1; - $SIG{INT} = \&wait_child; - $SIG{TERM} = \&wait_child; - foreach my $f (@tempfiles) { - unlink("$f->[1]") if (-e "$f->[1]"); - } - unlink("$tmp_last_parsed") if ($tmp_last_parsed); - } - _exit(0); -} -$SIG{INT} = \&wait_child; -$SIG{TERM} = \&wait_child; -$SIG{USR2} = \&stop_parsing; - -$| = 1; - # Command line options my $zcat_cmd = 'gunzip -c'; my $zcat = $zcat_cmd; @@ -120,14 +99,15 @@ my $format = ''; my $outfile = ''; my $outdir = ''; +my $incremental = ''; my $help = ''; my $ver = ''; my @dbname = (); my @dbuser = (); my @dbclient = (); -my @dbclient2 = (); my @dbappname = (); my @exclude_user = (); +my @exclude_appname = (); my $ident = ''; my $top = 0; my $sample = 0; @@ -144,6 +124,7 @@ my $progress = 1; my $error_only = 0; my @exclude_query = (); +my @exclude_time = (); my $exclude_file = ''; my @include_query = (); my $include_file = ''; @@ -159,19 +140,22 @@ my $disable_autovacuum = 0; my $avg_minutes = 5; my $last_parsed = ''; -my $report_title = 'pgBadger: PostgreSQL log analyzer'; +my $report_title = 'PostgreSQL log analyzer'; my $log_line_prefix = ''; my $compiled_prefix = ''; my $project_url = 'http://dalibo.github.com/pgbadger/'; my $t_min = 0; my $t_max = 0; -my $t_min_hour = 0; -my $t_max_hour = 0; my $remove_comment = 0; my $select_only = 0; my $tsung_queries = 0; my $queue_size = 0; my $job_per_file = 0; +my $charset = 'utf-8'; +my $csv_sep_char = ','; +my %current_sessions = (); +my $incr_date = ''; +my $last_incr_date = ''; my $NUMPROGRESS = 10000; my @DIMENSIONS = (800, 300); @@ -179,6 +163,10 @@ my $img_format = 'png'; my @log_files = (); my %prefix_vars = (); + +# Load the DATA part of the script +my @jscode = ; + my $sql_prettified; # Do not display data in pie where percentage is lower than this value @@ -190,6 +178,39 @@ my $num_sep = ','; $num_sep = ' ' if ($n =~ /,/); +# Inform the parent that it should stop iterate on parsing other files +sub stop_parsing +{ + $interrupt = 1; +} + +# With multiprocess we need to wait all childs +sub wait_child +{ + my $sig = shift; + print STDERR "Received terminating signal ($sig).\n"; + if ($^O !~ /MSWin32|dos/i) { + 1 while wait != -1; + $SIG{INT} = \&wait_child; + $SIG{TERM} = \&wait_child; + foreach my $f (@tempfiles) { + unlink("$f->[1]") if (-e "$f->[1]"); + } + } + if ($last_parsed && -e $tmp_last_parsed) { + unlink("$tmp_last_parsed"); + } + if ($last_parsed && -e "$last_parsed.tmp") { + unlink("$last_parsed.tmp"); + } + _exit(0); +} +$SIG{INT} = \&wait_child; +$SIG{TERM} = \&wait_child; +$SIG{USR2} = \&stop_parsing; + +$| = 1; + # get the command line parameters my $result = GetOptions( "a|average=i" => \$avg_minutes, @@ -202,6 +223,7 @@ "G|nograph!" => \$nograph, "h|help!" => \$help, "i|ident=s" => \$ident, + "I|incremental!" => \$incremental, "j|jobs=i" => \$queue_size, "J|job_per_file=i" => \$job_per_file, "l|last-parsed=s" => \$last_parsed, @@ -209,6 +231,7 @@ "N|appname=s" => \@dbappname, "n|nohighlight!" => \$nohighlight, "o|outfile=s" => \$outfile, + "O|outdir=s" => \$outdir, "p|prefix=s" => \$log_line_prefix, "P|no-prettify!" => \$noprettify, "q|quiet!" => \$quiet, @@ -227,6 +250,7 @@ "image-format=s" => \$img_format, "exclude-query=s" => \@exclude_query, "exclude-file=s" => \$exclude_file, + "exclude-appname=s" => \@exclude_appname, "include-query=s" => \@include_query, "include-file=s" => \$include_file, "disable-error!" => \$disable_error, @@ -239,12 +263,12 @@ "disable-temporary!" => \$disable_temporary, "disable-checkpoint!" => \$disable_checkpoint, "disable-autovacuum!" => \$disable_autovacuum, - "client=s" => \@dbclient2, # Backward compatibility + "charset=s" => \$charset, + "csv-separator=s" => \$csv_sep_char, + "exclude-time=s" => \@exclude_time, ); die "FATAL: use pgbadger --help\n" if (not $result); -push(@dbclient, @dbclient2); # Backward compatibility - if ($ver) { print "pgBadger version $VERSION\n"; exit 0; @@ -281,6 +305,15 @@ $avg_minutes ||= 5; $avg_minutes = 60 if ($avg_minutes > 60); $avg_minutes = 1 if ($avg_minutes < 1); +my @avgs = (); +for (my $i = 0 ; $i < 60 ; $i += $avg_minutes) { + push(@avgs, sprintf("%02d", $i)); +} + +# Set error like log level regex +my $parse_regex = qr/^(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|HINT|STATEMENT|CONTEXT)/; +my $full_error_regex = qr/^(WARNING|ERROR|FATAL|PANIC|DETAIL|HINT|STATEMENT|CONTEXT)/; +my $main_error_regex = qr/^(WARNING|ERROR|FATAL|PANIC)/; # Set syslog prefix regex my $other_syslog_line = @@ -293,8 +326,8 @@ if ($format eq 'syslog2') { $other_syslog_line = - qr/^(\d+)-(\d+)-(\d+)T(\d+):(\d+):(\d+)(?:.[^\s]+)?\s([^\s]+)\s([^\s\[]+)\[(\d+)\]:(?:\s\[[^\]]+\])?\s\[(\d+)\-\d+\]\s*(.*)/; - $orphan_syslog_line = qr/^(\d+)-(\d+)-(\d+)T(\d+):(\d+):(\d+)(?:.[^\s]+)?\s([^\s]+)\s([^\s\[]+)\[(\d+)\]:/; + qr/^(\d+-\d+)-(\d+)T(\d+):(\d+):(\d+)(?:.[^\s]+)?\s([^\s]+)\s(?:[^\s]+\s)?([^\s\[]+)\[(\d+)\]:(?:\s\[[^\]]+\])?\s\[(\d+)\-\d+\]\s*(.*)/; + $orphan_syslog_line = qr/^(\d+-\d+)-(\d+)T(\d+):(\d+):(\d+)(?:.[^\s]+)?\s([^\s]+)\s(?:[^\s]+\s)?([^\s\[]+)\[(\d+)\]:/; } # Set default top query @@ -337,8 +370,22 @@ # Extract the output directory from outfile so that graphs will # be created in the same directory -my @infs = fileparse($outfile); -$outdir = $infs[1] . '/'; +if ($outfile ne '-') { + if (!$outdir) { + my @infs = fileparse($outfile); + if ($infs[0] ne '') { + $outdir = $infs[1]; + } else { + # maybe a confusion between -O and -o + die "FATAL: output file $outfile is a directory, should be a file\nor maybe you want to use -O | --outdir option instead.\n"; + } + } elsif (!-d "$outdir") { + # An output directory have been passed as command line parameter + die "FATAL: $outdir is not a directory or doesn't exist.\n"; + } + $outfile = basename($outfile); + $outfile = $outdir . '/' . $outfile; +} # Remove graph support if output is not html $graph = 0 unless ($extension eq 'html' or $extension eq 'binary' ); @@ -407,6 +454,13 @@ } } +# Testing regex syntax +if ($#exclude_time >= 0) { + foreach my $r (@exclude_time) { + &check_regex($r, '--exclude-time'); + } +} + # Loading included query from file if any if ($include_file) { open(IN, "$include_file") or die "FATAL: can't read file $include_file: $!\n"; @@ -444,39 +498,39 @@ $log_line_prefix = '^(...)\s+(\d+)\s(\d+):(\d+):(\d+)(?:\s[^\s]+)?\s([^\s]+)\s([^\s\[]+)\[(\d+)\]:(?:\s\[[^\]]+\])?\s\[(\d+)\-\d+\]\s*' . $log_line_prefix - . '\s*(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT):\s+(.*)'; + . '\s*(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT):\s+(?:[0-9A-Z]{5}:\s+)?(.*)'; $compiled_prefix = qr/$log_line_prefix/; unshift(@prefix_params, 't_month', 't_day', 't_hour', 't_min', 't_sec', 't_host', 't_ident', 't_pid', 't_session_line'); push(@prefix_params, 't_loglevel', 't_query'); } elsif ($format eq 'syslog2') { $format = 'syslog'; $log_line_prefix = - '^(\d+)-(\d+)-(\d+)T\d+:\d+:\d+(?:.[^\s]+)?\s([^\s]+)\s([^\s\[]+)\[(\d+)\]:(?:\s\[[^\]]+\])?\s\[(\d+)\-\d+\]\s*' + '^(\d+)-(\d+)-(\d+)T(\d+):(\d+):(\d+)(?:.[^\s]+)?\s([^\s]+)\s(?:[^\s]+\s)?([^\s\[]+)\[(\d+)\]:(?:\s\[[^\]]+\])?\s\[(\d+)\-\d+\]\s*' . $log_line_prefix - . '\s*(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT):\s+(.*)'; + . '\s*(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT):\s+(?:[0-9A-Z]{5}:\s+)?(.*)'; $compiled_prefix = qr/$log_line_prefix/; unshift(@prefix_params, 't_year', 't_month', 't_day', 't_hour', 't_min', 't_sec', 't_host', 't_ident', 't_pid', 't_session_line'); push(@prefix_params, 't_loglevel', 't_query'); } elsif ($format eq 'stderr') { $orphan_stderr_line = qr/$log_line_prefix/; - $log_line_prefix = '^' . $log_line_prefix . '\s*(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT):\s+(.*)'; + $log_line_prefix = '^' . $log_line_prefix . '\s*(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT):\s+(?:[0-9A-Z]{5}:\s+)?(.*)'; $compiled_prefix = qr/$log_line_prefix/; push(@prefix_params, 't_loglevel', 't_query'); } } elsif ($format eq 'syslog') { $compiled_prefix = -qr/^(...)\s+(\d+)\s(\d+):(\d+):(\d+)(?:\s[^\s]+)?\s([^\s]+)\s([^\s\[]+)\[(\d+)\]:(?:\s\[[^\]]+\])?\s\[(\d+)\-\d+\]\s*(.*?)\s*(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT):\s+(.*)/; +qr/^(...)\s+(\d+)\s(\d+):(\d+):(\d+)(?:\s[^\s]+)?\s([^\s]+)\s([^\s\[]+)\[(\d+)\]:(?:\s\[[^\]]+\])?\s\[(\d+)\-\d+\]\s*(.*?)\s*(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT):\s+(?:[0-9A-Z]{5}:\s+)?(.*)/; push(@prefix_params, 't_month', 't_day', 't_hour', 't_min', 't_sec', 't_host', 't_ident', 't_pid', 't_session_line', 't_logprefix', 't_loglevel', 't_query'); } elsif ($format eq 'syslog2') { $format = 'syslog'; $compiled_prefix = -qr/^(\d+)-(\d+)-(\d+)T(\d+):(\d+):(\d+)(?:.[^\s]+)?\s([^\s]+)\s([^\s\[]+)\[(\d+)\]:(?:\s\[[^\]]+\])?\s\[(\d+)\-\d+\]\s*(.*?)\s*(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT):\s+(.*)/; +qr/^(\d+)-(\d+)-(\d+)T(\d+):(\d+):(\d+)(?:.[^\s]+)?\s([^\s]+)\s(?:[^\s]+\s)?([^\s\[]+)\[(\d+)\]:(?:\s\[[^\]]+\])?\s\[(\d+)\-\d+\]\s*(.*?)\s*(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT):\s+(?:[0-9A-Z]{5}:\s+)?(.*)/; push(@prefix_params, 't_year', 't_month', 't_day', 't_hour', 't_min', 't_sec', 't_host', 't_ident', 't_pid', 't_session_line', 't_logprefix', 't_loglevel', 't_query'); } elsif ($format eq 'stderr') { $compiled_prefix = -qr/^(\d+-\d+-\d+\s\d+:\d+:\d+)[\.\d]*(?: [A-Z\d]{3,6})?\s\[(\d+)\]:\s\[(\d+)\-\d+\]\s*(.*?)\s*(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT):\s+(.*)/; +qr/^(\d+-\d+-\d+\s\d+:\d+:\d+)[\.\d]*(?: [A-Z\d]{3,6})?\s\[(\d+)\]:\s\[(\d+)\-\d+\]\s*(.*?)\s*(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT):\s+(?:[0-9A-Z]{5}:\s+)?(.*)/; push(@prefix_params, 't_timestamp', 't_pid', 't_session_line', 't_logprefix', 't_loglevel', 't_query'); $orphan_stderr_line = qr/^(\d+-\d+-\d+\s\d+:\d+:\d+)[\.\d]*(?: [A-Z\d]{3,6})?\s\[(\d+)\]:\s\[(\d+)\-\d+\]\s*(.*?)\s*/; } @@ -493,14 +547,21 @@ # Check start/end date time if ($from) { - if ($from !~ /^(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})$/) { - die "FATAL: bad format for begin datetime, should be yyyy-mm-dd hh:mm:ss\n"; - } + if ($from !~ /^(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})([.]\d+([+-]\d+)?)?$/) { + die "FATAL: bad format for begin datetime, should be yyyy-mm-dd hh:mm:ss.l+tz\n"; + } else { + my $fractional_seconds = $7 || "0"; + $from = "$1-$2-$3 $4:$5:$6.$7" + } + } if ($to) { - if ($to !~ /^(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})$/) { - die "FATAL: bad format for ending datetime, should be yyyy-mm-dd hh:mm:ss\n"; - } + if ($to !~ /^(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})([.]\d+([+-]\d+)?)?$/) { + die "FATAL: bad format for ending datetime, should be yyyy-mm-dd hh:mm:ss.l+tz\n"; + } else { + my $fractional_seconds = $7 || "0"; + $to = "$1-$2-$3 $4:$5:$6.$7" + } } # Stores the last parsed line from log file to allow incremental parsing @@ -542,7 +603,7 @@ KEYS KILL KEY LINES LOAD LOCAL LOCK LOW_PRIORITY LANGUAGE LEAST LOGIN MODIFY NULLIF NOSUPERUSER NOCREATEDB NOCREATEROLE OPTIMIZE OPTION OPTIONALLY OUTFILE OWNER PROCEDURE PROCEDURAL READ REGEXP RENAME RETURN REVOKE RLIKE ROLE ROLLBACK SHOW SONAME STATUS - STRAIGHT_JOIN SET SEQUENCE TABLES TEMINATED TRUNCATE TEMPORARY TRIGGER TRUSTED UNLOCK + STRAIGHT_JOIN SET SEQUENCE TABLES TEMINATED TRUNCATE TEMPORARY TRIGGER TRUSTED UN$filenumLOCK USE UPDATE UNSIGNED VALUES VARIABLES VIEW VACUUM WRITE ZEROFILL XOR ABORT ABSOLUTE ACCESS ACTION ADMIN AFTER AGGREGATE ALSO ALWAYS ASSERTION ASSIGNMENT AT ATTRIBUTE BACKWARD BEFORE BIGINT CACHE CALLED CASCADE CASCADED CATALOG CHAIN CHARACTER CHARACTERISTICS @@ -599,23 +660,39 @@ my @BRACKETS = ('(', ')'); map {$_ = quotemeta($_)} @BRACKETS; +# Inbounds of query times histogram +my @histogram_query_time = (0, 1, 5, 10, 25, 50, 100, 500, 1000, 10000); + +# Get inbounds of query times histogram +sub get_hist_inbound +{ + my $duration = shift; + + for (my $i = 0; $i <= $#histogram_query_time; $i++) { + return $histogram_query_time[$i-1] if ($histogram_query_time[$i] > $duration); + } + + return -1; +} + # Where statistics are stored my %overall_stat = (); +my %overall_checkpoint = (); my @top_slowest = (); my %normalyzed_info = (); my %error_info = (); my %logs_type = (); -my %per_hour_info = (); my %per_minute_info = (); my %lock_info = (); my %tempfile_info = (); my %connection_info = (); my %database_info = (); my %application_info = (); +my %user_info = (); +my %host_info = (); my %session_info = (); my %conn_received = (); my %checkpoint_info = (); -my %restartpoint_info = (); my %autovacuum_info = (); my %autoanalyze_info = (); my @graph_values = (); @@ -628,26 +705,65 @@ my %tsung_session = (); my @top_locked_info = (); my @top_tempfile_info = (); +my %drawn_graphs = (); my $t0 = Benchmark->new; +# Automatically set parameters with incremental mode +if ($incremental) { + # In incremental mode an output directory must be set + if (!$outdir) { + die "FATAL: you must specify an output directory with incremental mode, see -O or --outdir.\n" + } + # Ensure this is not a relative path + if (dirname($outdir) eq '.') { + die "FATAL: output directory ($outdir) is not an absolute path.\n"; + } + # Ensure that the directory already exists + if (!-d $outdir) { + die "FATAL: output directory $outdir does not exists\n"; + } + # Set default last parsed file in incremental mode + if (!$last_parsed) { + $last_parsed = $outdir . '/LAST_PARSED'; + } + $outfile = 'index.html'; + # Set default output format + $extension = 'binary'; +} + # Reading last line parsed if ($last_parsed && -e $last_parsed) { if (open(IN, "$last_parsed")) { my $line = ; close(IN); - ($saved_last_line{datetime}, $saved_last_line{orig}) = split(/\t/, $line, 2); + ($saved_last_line{datetime}, $saved_last_line{current_pos}, $saved_last_line{orig}) = split(/\t/, $line, 3); + # Preserve backward compatibility with version < 5 + if ($saved_last_line{current_pos} =~ /\D/) { + $saved_last_line{orig} = $saved_last_line{current_pos} . "\t" . $saved_last_line{orig}; + $saved_last_line{current_pos} = 0; + } + if ( ($format eq 'binary') || ($format eq 'csv') ) { + $saved_last_line{current_pos} = 0; + } + } else { die "FATAL: can't read last parsed line from $last_parsed, $!\n"; } } -$tmp_last_parsed = 'tmp_' . $last_parsed if ($last_parsed); - +$tmp_last_parsed = 'tmp_' . basename($last_parsed) if ($last_parsed); + # Main loop reading log files my $global_totalsize = 0; my @given_log_files = ( @log_files ); +# Verify that the file have not changed for incremental move +if ( ($saved_last_line{current_pos} > 0) && ($#given_log_files == 0)) { + $saved_last_line{current_pos} = 0 if (&check_file_changed($given_log_files[0], $saved_last_line{datetime})); + $saved_last_line{current_pos}++ if ($saved_last_line{current_pos} > 0); +} + # log files must be erase when loading stats from binary format if ($format eq 'binary') { $queue_size = 1; @@ -657,6 +773,9 @@ my $pipe; +# Seeking to an old log position is not possible when multiple file are provided +$saved_last_line{current_pos} = 0 if (!$last_parsed && ($#given_log_files > 0)); + # Start parsing all given files using multiprocess if ( ($queue_size > 1) || ($job_per_file > 1) ) { @@ -694,7 +813,7 @@ $child_count--; delete $RUNNING_PIDS{$kid}; } - usleep(500000); + sleep(1); } # Do not use split method with compressed files if ( ($queue_size > 1) && ($logfile !~ /\.(gz|bz2|zip)/i) ) { @@ -707,7 +826,7 @@ $child_count--; delete $RUNNING_PIDS{$kid}; } - usleep(500000); + sleep(1); } push(@tempfiles, [ tempfile('tmp_pgbadgerXXXX', SUFFIX => '.bin', DIR => $TMP_DIR, UNLINK => 1 ) ]); spawn sub { @@ -737,7 +856,7 @@ if ($kid > 0) { delete $RUNNING_PIDS{$kid}; } - usleep(500000); + sleep(1); } # Terminate the process logger foreach my $k (keys %RUNNING_PIDS) { @@ -745,8 +864,10 @@ %RUNNING_PIDS = (); } - # Load all data gathered by all the differents processes + # Clear previous statistics &init_stats_vars(); + + # Load all data gathered by all the differents processes foreach my $f (@tempfiles) { next if (!-e "$f->[1]" || -z "$f->[1]"); my $fht = new IO::File; @@ -755,34 +876,40 @@ $fht->close(); } - # Get last line parsed from all process - if ($last_parsed) { - if (open(IN, "$tmp_last_parsed") ) { - while (my $line = ) { - chomp($line); - my ($d, $l) = split(/\t/, $line, 2); - if (!$last_line{datetime} || ($d gt $last_line{datetime})) { - $last_line{datetime} = $d; - $last_line{orig} = $l; - } - } - close(IN); - } - unlink("$tmp_last_parsed"); - } - } else { # Multiprocessing disabled, parse log files one by one foreach my $logfile ( @given_log_files ) { - last if (&process_file($logfile)); + last if (&process_file($logfile, '', $saved_last_line{current_pos})); + } +} + +# Get last line parsed from all process +if ($last_parsed) { + if (open(IN, "$tmp_last_parsed") ) { + while (my $line = ) { + chomp($line); + my ($d, $p, $l) = split(/\t/, $line, 3); + if (!$last_line{datetime} || ($d gt $last_line{datetime})) { + $last_line{datetime} = $d; + if ($p =~ /^\d+$/) { + $last_line{orig} = $l; + $last_line{current_pos} = $p; + } else { + $last_line{orig} = $p . "\t" . $l; + } + } + } + close(IN); } + unlink("$tmp_last_parsed"); } # Save last line parsed if ($last_parsed && scalar keys %last_line) { if (open(OUT, ">$last_parsed")) { - print OUT "$last_line{datetime}\t$last_line{orig}\n"; + $last_line{current_pos} ||= 0; + print OUT "$last_line{datetime}\t$last_line{current_pos}\t$last_line{orig}\n"; close(OUT); } else { &logmsg('ERROR', "can't save last parsed line into $last_parsed, $!"); @@ -793,44 +920,273 @@ my $td = timediff($t1, $t0); &logmsg('DEBUG', "the log statistics gathering took:" . timestr($td)); -&logmsg('LOG', "Ok, generating $extension report..."); - -# Open filehandle +# Global output filehandle my $fh = undef; -if ($extension ne 'tsung') { - $fh = new IO::File ">$outfile"; - if (not defined $fh) { - die "FATAL: can't write to $outfile, $!\n"; - } - if (($extension eq 'text') || ($extension eq 'txt')) { - if ($error_only) { - &dump_error_as_text(); + +if (!$incremental) { + + &logmsg('LOG', "Ok, generating $extension report..."); + + if ($extension ne 'tsung') { + $fh = new IO::File ">$outfile"; + if (not defined $fh) { + die "FATAL: can't write to $outfile, $!\n"; + } + if (($extension eq 'text') || ($extension eq 'txt')) { + if ($error_only) { + &dump_error_as_text(); + } else { + &dump_as_text(); + } + } elsif ($extension eq 'binary') { + &dump_as_binary($fh); } else { - &dump_as_text(); + # Create instance to prettify SQL query + if (!$noprettify) { + $sql_prettified = SQL::Beautify->new(keywords => \@pg_keywords); + } + &dump_as_html(); + } + $fh->close; + } else { + + # Open filehandle + $fh = new IO::File ">>$outfile"; + if (not defined $fh) { + die "FATAL: can't write to $outfile, $!\n"; + } + print $fh "\n"; + $fh->close(); + } + +} else { + + # Build a report per day + my %weeks_directories = (); + my @build_directories = (); + if (open(IN, "$last_parsed.tmp")) { + while (my $l = ) { + chomp($l); + push(@build_directories, $l) if (!grep(/^$l$/, @build_directories)); } - } elsif ($extension eq 'binary') { - &dump_as_binary($fh); + close(IN); + unlink("$last_parsed.tmp"); } else { + &logmsg('WARNING', "can't read file $last_parsed.tmp, $!"); + &logmsg('HINT', "maybe there's no new entries in your log since last run."); + } + foreach $incr_date (@build_directories) { + + $last_incr_date = $incr_date; + + # Set the path to binary files + my $bpath = $incr_date; + $bpath =~ s/\-/\//g; + $incr_date =~ /^(\d+)-(\d+)\-(\d+)$/; + + # Get the week number following the date + my $wn = &get_week_number($1, $2, $3); + $weeks_directories{$wn} = "$1-$2" if (!exists $weeks_directories{$wn}); + + # First clear previous stored statistics + &init_stats_vars(); + + # Load all data gathered by all the differents processes + unless(opendir(DIR, "$outdir/$bpath")) { + die "Error: can't opendir $outdir/$bpath: $!"; + } + my @mfiles = grep { !/^\./ && ($_ =~ /\.bin$/) } readdir(DIR); + closedir DIR; + foreach my $f (@mfiles) { + my $fht = new IO::File; + $fht->open("< $outdir/$bpath/$f") or die "FATAL: can't open file $outdir/$bpath/$f, $!\n"; + &load_stats($fht); + $fht->close(); + } + + &logmsg('LOG', "Ok, generating HTML daily report into $outdir/$bpath/..."); + + $fh = new IO::File ">$outdir/$bpath/$outfile"; + if (not defined $fh) { + die "FATAL: can't write to $outdir/$bpath/$outfile, $!\n"; + } # Create instance to prettify SQL query if (!$noprettify) { $sql_prettified = SQL::Beautify->new(keywords => \@pg_keywords); } - if ($error_only) { - &dump_error_as_html(); - } else { - &dump_as_html(); + &dump_as_html(); + $fh->close; + } + + # Build a report per week + foreach my $wn (sort { $a <=> $b } keys %weeks_directories) { + &init_stats_vars(); + + # Get all days of the current week + my @wdays = &get_wdays_per_month($wn - 1, $weeks_directories{$wn}); + my $wdir = ''; + + # Load data per day + foreach $incr_date (@wdays) { + my $bpath = $incr_date; + $bpath =~ s/\-/\//g; + $incr_date =~ /^(\d+)-(\d+)\-(\d+)$/; + $wdir = "$1/week-$wn"; + + # Load all data gathered by all the differents processes + if (-e "$outdir/$bpath") { + unless(opendir(DIR, "$outdir/$bpath")) { + die "Error: can't opendir $outdir/$bpath: $!"; + } + my @mfiles = grep { !/^\./ && ($_ =~ /\.bin$/) } readdir(DIR); + closedir DIR; + foreach my $f (@mfiles) { + my $fht = new IO::File; + $fht->open("< $outdir/$bpath/$f") or die "FATAL: can't open file $outdir/$bpath/$f, $!\n"; + &load_stats($fht); + $fht->close(); + } + } + } + + &logmsg('LOG', "Ok, generating HTML weekly report into $outdir/$wdir/..."); + if (!-d "$outdir/$wdir") { + mkdir("$outdir/$wdir"); + } + $fh = new IO::File ">$outdir/$wdir/$outfile"; + if (not defined $fh) { + die "FATAL: can't write to $outdir/$wdir/$outfile, $!\n"; + } + # Create instance to prettify SQL query + if (!$noprettify) { + $sql_prettified = SQL::Beautify->new(keywords => \@pg_keywords); } + &dump_as_html(); + $fh->close; + } - $fh->close; -} else { - # Open filehandle - $fh = new IO::File ">>$outfile"; + &logmsg('LOG', "Ok, generating global index to access incremental reports..."); + + $fh = new IO::File ">$outdir/index.html"; if (not defined $fh) { - die "FATAL: can't write to $outfile, $!\n"; + die "FATAL: can't write to $outdir/index.html, $!\n"; } - print $fh "\n"; - $fh->close(); + my $date = localtime(time); + print $fh qq{ + + +pgBadger :: Global Index on incremental reports + + + + + + +@jscode + + + + + + +


+
+ + + + +}; + # get years directories + unless(opendir(DIR, "$outdir")) { + die "Error: can't opendir $outdir: $!"; + } + my @dyears = grep { !/^\./ && /^\d{4}$/ } readdir(DIR); + closedir DIR; + my @day_names = ('Mon','Tue','Wed','Thu','Fri','Sat','Sun'); + my @month_names = ('Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sept','Oct','Nov','Dec'); + foreach my $y (sort { $b <=> $a } @dyears) { + print $fh qq{ +

Year $y

+
+
    +}; + # foreach year directory look for week directories + unless(opendir(DIR, "$outdir/$y")) { + die "Error: can't opendir $outdir/$y: $!"; + } + + my @yweeks = grep { !/^\./ && /^week-\d+$/ } readdir(DIR); + closedir DIR; + my %ywdays = &get_wdays_per_year($y); + foreach my $w (sort { &sort_by_week($a, $b); } @yweeks) { + $w =~ /week\-(\d+)/; + my $week = "Week $1"; + # foreach week add link to daily reports + my $wn = sprintf("%02d", $1 - 1); + my @wdays = @{$ywdays{$wn}}; + my $data_content = ''; + for (my $i = 0; $i <= $#wdays; $i++) { + my $bpath = $wdays[$i]; + $bpath =~ s/(\d+)\-(\d+)\-(\d+)/$1\/$2\/$3/g; + my $mmonth = $month_names[$2 - 1]; + my $dday = $3; + if (-e "$outdir/$bpath/index.html") { + $data_content .= "$day_names[$i] $mmonth $dday $y
    "; + } else { + $data_content .= "$day_names[$i] $mmonth $dday $y
    "; + } + } + print $fh qq{ +

  • $week
    • +}; + } + print $fh qq{ +
+
+}; + } + print $fh qq{ +
+ + + +
+ +
+ + + +}; + + + + $fh->close; + } my $t2 = Benchmark->new; @@ -870,6 +1226,8 @@ -G | --nograph : disable graphs on HTML output. Enable by default. -h | --help : show this message and exit. -i | --ident name : programname used as syslog ident. Default: postgres + -I | --incremental : use incremental mode, reports will be generated by + days in a separate directory, --outdir must be set. -j | --jobs number : number of jobs to run at same time. Default is 1, run as single process. -l | --last-parsed file: allow incremental log parsing by registering the @@ -883,6 +1241,7 @@ -o | --outfile filename: define the filename for the output. Default depends on the output format: out.html, out.txt or out.tsung. To dump output to stdout use - as filename. + -O | --outdir path : directory where out file must be saved. -p | --prefix string : give here the value of your custom log_line_prefix defined in your postgresql.conf. Only use it if you aren't using one of the standard prefixes specified @@ -890,7 +1249,7 @@ includes additional variables like client ip or application name. See examples below. -P | --no-prettify : disable SQL queries prettify formatter. - -q | --quiet : don't print anything to stdout, even not a progress bar. + -q | --quiet : don't print anything to stdout, not even a progress bar. -s | --sample number : number of query samples to store/display. Default: 3 -S | --select-only : use it if you want to report select queries only. -t | --top number : number of queries to store/display. Default: 20 @@ -916,15 +1275,22 @@ queries to include from the report. One regex per line. --disable-error : do not generate error report. --disable-hourly : do not generate hourly report. - --disable-type : do not generate query type report. + --disable-type : do not generate report of queries by type, database... --disable-query : do not generate query reports (slowest, most - frequent, ...). + frequent, queries by users, by database, ...). --disable-session : do not generate session report. --disable-connection : do not generate connection report. --disable-lock : do not generate lock report. --disable-temporary : do not generate temporary report. --disable-checkpoint : do not generate checkpoint/restartpoint report. --disable-autovacuum : do not generate autovacuum report. + --charset : used to set the HTML charset to be used. Default: utf-8. + --csv-separator : used to set the CSV field separator, default: , + --exclude-time regex : any timestamp matching the given regex will be + excluded from the report. Example: "2013-04-12 .*" + You can use this option multiple times. + --exclude-appname name : exclude entries for the specified application name + from report. Example: "pg_dump". Examples: @@ -962,20 +1328,51 @@ This supposes that your log file and HTML report are also rotated every week. +Or better, use the auto-generated incremental reports: + + 0 4 * * * /usr/bin/pgbadger -I -q /var/log/postgresql/postgresql.log.1 \ + -O /var/www/pg_reports/ + +will generate a report per day and per week. + +If you have a pg_dump at 23:00 and 13:00 each day during half an hour, you can +use pgbadger as follow to exclude these period from the report: + + pgbadger --exclude-time "2013-09-.* (23|13):.*" postgresql.log + +This will help to not have all COPY order on top of slowest queries. You can +also use --exclude-appname "pg_dump" to solve this problem in a more simple way. + }; exit 0; } +sub sort_by_week +{ + my $curr = shift; + my $next = shift; + + $a =~ /week\-(\d+)/; + $curr = $1; + $b =~ /week\-(\d+)/; + $next = $1; + + return $next <=> $curr; +} + sub init_stats_vars { + # Empty where statistics are stored %overall_stat = (); + %overall_checkpoint = (); @top_slowest = (); + @top_tempfile_info = (); + @top_locked_info = (); %normalyzed_info = (); %error_info = (); %logs_type = (); - %per_hour_info = (); %per_minute_info = (); %lock_info = (); %tempfile_info = (); @@ -985,7 +1382,6 @@ %session_info = (); %conn_received = (); %checkpoint_info = (); - %restartpoint_info = (); %autovacuum_info = (); %autoanalyze_info = (); @graph_values = (); @@ -1074,7 +1470,7 @@ if ($format eq 'csv') { require Text::CSV_XS; - my $csv = Text::CSV_XS->new({binary => 1, eol => $/}); + my $csv = Text::CSV_XS->new({binary => 1, eol => $/, sep_char => $csv_sep_char}); # Parse csvlog lines while (my $row = $csv->getline($lfile)) { @@ -1101,9 +1497,7 @@ $cursize = 0; } } - - # Process only relevant lines - next if ($row->[11] !~ /^(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT)$/); + next if ($row->[11] !~ $parse_regex); # Extract the date $row->[0] =~ m/^(\d+)-(\d+)-(\d+)\s+(\d+):(\d+):(\d+)\.(\d+)/; @@ -1120,7 +1514,7 @@ $old_errors_count = $overall_stat{'errors_number'}; $cursize = 0; } - $getout = 1; + $getout = 2; last; } @@ -1142,12 +1536,12 @@ $prefix_vars{'t_session_line'} =~ s/\..*//; $prefix_vars{'t_loglevel'} = $row->[11]; $prefix_vars{'t_query'} = $row->[13]; - # Set ERROR additional informations + # Set ERROR additional information $prefix_vars{'t_detail'} = $row->[14]; $prefix_vars{'t_hint'} = $row->[15]; $prefix_vars{'t_context'} = $row->[18]; $prefix_vars{'t_statement'} = $row->[19]; - + # Check if the log line should be excluded from the report if (&validate_log_line($prefix_vars{'t_pid'})) { @@ -1171,6 +1565,7 @@ my $cur_pid = ''; my @matches = (); my $goon = 0; + &logmsg('DEBUG', "Start parsing at offset $start_offset of file $logfile"); if ($start_offset) { $lfile->seek($start_offset, 0); } @@ -1223,6 +1618,9 @@ # skip non postgresql lines next if ($prefix_vars{'t_ident'} ne $ident); + # Stores temporary files and locks information + &store_temporary_and_lock_infos($cur_pid); + # Standard syslog format does not have year information, months are # three letters and day are not always with 2 digit. if ($prefix_vars{'t_month'} !~ /\d/) { @@ -1238,6 +1636,14 @@ "$prefix_vars{'t_year'}-$prefix_vars{'t_month'}-$prefix_vars{'t_day'} $prefix_vars{'t_hour'}:$prefix_vars{'t_min'}:$prefix_vars{'t_sec'}"; # Skip unwanted lines + if ($#exclude_time >= 0) { + foreach (@exclude_time) { + if ($prefix_vars{'t_timestamp'} =~ /$_/) { + return; + } + } + } + next if ($from && ($from gt $prefix_vars{'t_timestamp'})); if ($to && ($to lt $prefix_vars{'t_timestamp'})) { if ($tmpoutfile) { @@ -1246,7 +1652,7 @@ $old_errors_count = $overall_stat{'errors_number'}; $cursize = 0; } - $getout = 1; + $getout = 2; last; } @@ -1274,39 +1680,57 @@ $cur_pid = $8; my $t_query = $10; - $t_query = $11 if ($format eq 'syslog-ng'); $t_query =~ s/#011/\t/g; next if ($t_query eq "\t"); + if ($cur_info{$cur_pid}{vacuum} && ($t_query =~ /^\t(pages|tuples|buffer usage|avg read rate|system usage):/)) { if ($t_query =~ /^\t(pages|tuples): (\d+) removed, (\d+) remain/) { $autovacuum_info{tables}{$cur_info{$cur_pid}{vacuum}}{$1}{removed} += $2; } + if ($t_query =~ m#^\tsystem usage: CPU .* sec elapsed (.*) sec#) { + if ($1 > $autovacuum_info{peak}{system_usage}{elapsed}) { + $autovacuum_info{peak}{system_usage}{elapsed} = $1; + $autovacuum_info{peak}{system_usage}{table} = $cur_info{$cur_pid}{vacuum}; + $autovacuum_info{peak}{system_usage}{date} = + "$cur_info{$cur_pid}{year}-$cur_info{$cur_pid}{month}-$cur_info{$cur_pid}{day} " . + "$cur_info{$cur_pid}{hour}:$cur_info{$cur_pid}{min}:$cur_info{$cur_pid}{sec}"; + } + } next; } elsif ( $cur_info{$cur_pid}{parameters} && (($t_query =~ /[,\s]*\$(\d+)\s=\s/) || ($t_query =~ /^('[^']*')$/)) ) { # stores bind parameters if any $cur_info{$cur_pid}{parameters} .= " $t_query"; next; } - if ($cur_info{$cur_pid}{statement}) { + + if (exists $cur_temp_info{$cur_pid}{query}) { + $cur_temp_info{$cur_pid}{query} .= "\n" . $t_query; + } elsif (exists $cur_lock_info{$cur_pid}{query}) { + $cur_lock_info{$cur_pid}{query} .= "\n" . $t_query; + } elsif (exists $cur_info{$cur_pid}{statement}) { $cur_info{$cur_pid}{statement} .= "\n" . $t_query; - } elsif ($cur_info{$cur_pid}{context}) { + } elsif (exists $cur_info{$cur_pid}{context}) { $cur_info{$cur_pid}{context} .= "\n" . $t_query; - } elsif ($cur_info{$cur_pid}{detail}) { + } elsif (exists $cur_info{$cur_pid}{detail}) { $cur_info{$cur_pid}{detail} .= "\n" . $t_query; - } else { + } elsif (exists $cur_info{$cur_pid}{query}) { $cur_info{$cur_pid}{query} .= "\n" . $t_query; } - # Collect orphans lines of multiline queries + # Collect orphans lines of multiline queries } elsif ($cur_pid && ($line !~ $orphan_syslog_line)) { - if ($cur_info{$cur_pid}{statement}) { + if (exists $cur_temp_info{$cur_pid}{query}) { + $cur_temp_info{$cur_pid}{query} .= "\n" . $line; + } elsif (exists $cur_lock_info{$cur_pid}{query}) { + $cur_lock_info{$cur_pid}{query} .= "\n" . $line; + } elsif (exists $cur_info{$cur_pid}{statement}) { $cur_info{$cur_pid}{statement} .= "\n" . $line; - } elsif ($cur_info{$cur_pid}{context}) { + } elsif (exists $cur_info{$cur_pid}{context}) { $cur_info{$cur_pid}{context} .= "\n" . $line; - } elsif ($cur_info{$cur_pid}{detail}) { + } elsif (exists $cur_info{$cur_pid}{detail}) { $cur_info{$cur_pid}{detail} .= "\n" . $line; - } else { + } elsif (exists $cur_info{$cur_pid}{query}) { $cur_info{$cur_pid}{query} .= "\n" . $line; } @@ -1321,6 +1745,10 @@ for (my $i = 0 ; $i <= $#prefix_params ; $i++) { $prefix_vars{$prefix_params[$i]} = $matches[$i]; } + + # Stores temporary files and locks information + &store_temporary_and_lock_infos($cur_pid); + if (!$prefix_vars{'t_timestamp'} && $prefix_vars{'t_mtimestamp'}) { $prefix_vars{'t_timestamp'} = $prefix_vars{'t_mtimestamp'}; } elsif (!$prefix_vars{'t_timestamp'} && $prefix_vars{'t_session_timestamp'}) { @@ -1330,6 +1758,13 @@ $prefix_vars{'t_min'}, $prefix_vars{'t_sec'}) = ($prefix_vars{'t_timestamp'} =~ $time_pattern); # Skip unwanted lines + if ($#exclude_time >= 0) { + foreach (@exclude_time) { + if ($prefix_vars{'t_timestamp'} =~ /$_/) { + return; + } + } + } next if ($from && ($from gt $prefix_vars{'t_timestamp'})); if ($to && ($to lt $prefix_vars{'t_timestamp'})) { if ($tmpoutfile) { @@ -1338,7 +1773,7 @@ $old_errors_count = $overall_stat{'errors_number'}; $cursize = 0; } - $getout = 1; + $getout = 2; last; } @@ -1365,23 +1800,40 @@ # Collect additional query information } elsif ($cur_pid && ($line !~ $orphan_stderr_line)) { - if ($cur_info{$cur_pid}{vacuum} && ($line =~ /^\t(pages|tuples|buffer usage|avg read rate|system usage):/)) { + if ($line =~ s/^(STATEMENT|DETAIL|HINT):\s+//) { + $line =~ s/ERROR:\s+//; + $cur_info{$cur_pid}{"\L$1\E"} = $line; + next; + } elsif ($cur_info{$cur_pid}{vacuum} && ($line =~ /^\t(pages|tuples|buffer usage|avg read rate|system usage):/)) { if ($line =~ /^\t(pages|tuples): (\d+) removed, (\d+) remain/) { $autovacuum_info{tables}{$cur_info{$cur_pid}{vacuum}}{$1}{removed} += $2; } + if ($line =~ m#^\tsystem usage: CPU .* sec elapsed (.*) sec#) { + if ($1 > $autovacuum_info{peak}{system_usage}{elapsed}) { + $autovacuum_info{peak}{system_usage}{elapsed} = $1; + $autovacuum_info{peak}{system_usage}{table} = $cur_info{$cur_pid}{vacuum}; + $autovacuum_info{peak}{system_usage}{date} = + "$cur_info{$cur_pid}{year}-$cur_info{$cur_pid}{month}-$cur_info{$cur_pid}{day} " . + "$cur_info{$cur_pid}{hour}:$cur_info{$cur_pid}{min}:$cur_info{$cur_pid}{sec}"; + } + } next; } elsif ( $cur_info{$cur_pid}{parameters} && (($line =~ /[,\s]*\$(\d+)\s=\s/) || ($line =~ /^'[^']*'$/)) ) { # stores bind parameters if any $cur_info{$cur_pid}{parameters} .= " $line"; next; } - if (exists $cur_info{$cur_pid}{statement}) { + if (exists $cur_temp_info{$cur_pid}{query}) { + $cur_temp_info{$cur_pid}{query} .= "\n" . $line; + } elsif (exists $cur_lock_info{$cur_pid}{query}) { + $cur_lock_info{$cur_pid}{query} .= "\n" . $line; + } elsif (exists $cur_info{$cur_pid}{statement}) { $cur_info{$cur_pid}{statement} .= "\n" . $line; } elsif (exists $cur_info{$cur_pid}{context}) { $cur_info{$cur_pid}{context} .= "\n" . $line; } elsif (exists $cur_info{$cur_pid}{detail}) { $cur_info{$cur_pid}{detail} .= "\n" . $line; - } else { + } elsif (exists $cur_info{$cur_pid}{query}) { $cur_info{$cur_pid}{query} .= "\n" . $line; } @@ -1399,28 +1851,41 @@ } last if (($stop_offset > 0) && ($current_offset > $stop_offset)); } + $last_line{current_pos} = $current_offset if ($last_parsed && ($#given_log_files == 0)); + } close $lfile; # Get stats from all pending temporary storage foreach my $pid (sort {$cur_info{$a}{date} <=> $cur_info{$b}{date}} keys %cur_info) { + # Stores last queries information &store_queries($pid); + + } + # Stores last temporary files and locks information + foreach my $pid (keys %cur_temp_info) { + &store_temporary_and_lock_infos($pid); + } + # Stores last temporary files and locks information + foreach my $pid (keys %cur_lock_info) { + &store_temporary_and_lock_infos($pid); } + if ($extension eq 'tsung') { foreach my $pid (sort {$a <=> $b} keys %tsung_session) { &store_tsung_session($pid); } } - if ($progress && !$getout) { + if ($progress && ($getout != 1)) { if (!$tmpoutfile) { if ($totalsize) { if (($stop_offset > 0) && ($format ne 'csv')) { - print STDERR &progress_bar($cursize - $start_offset, $stop_offset, 25, '=',$overall_stat{'queries_number'},$overall_stat{'errors_number'}); + print STDERR &progress_bar($cursize - $start_offset, $stop_offset, 25, '=',$overall_stat{'queries_number'},$overall_stat{'errors_number'}, $logfile); } elsif ($extension eq 'tsung') { print STDERR &progress_bar($cursize, $totalsize, 25, '=', $logfile); } else { - print STDERR &progress_bar($cursize, $totalsize, 25, '=', $overall_stat{'queries_number'},$overall_stat{'errors_number'}); + print STDERR &progress_bar($cursize, $totalsize, 25, '=', $overall_stat{'queries_number'},$overall_stat{'errors_number'}, $logfile); } print STDERR "\n"; } @@ -1431,27 +1896,61 @@ %cur_info = (); - if ($tmpoutfile) { - &dump_as_binary($tmpoutfile); - $tmpoutfile->close(); - } + # In incremental mode data are saved to disk per day + if ($incremental && $last_line{datetime}) { + $incr_date = $last_line{datetime}; + $incr_date =~ s/\s.*$//; + # set path and create subdirectories + my $bpath = $incr_date; + while ($bpath =~ s/([^\-]+)\-/$1\//) { + mkdir("$outdir/$1") if (!-d "$outdir/$1"); + } + mkdir("$outdir/$bpath") if (!-d "$outdir/$bpath"); - # Inform the parent that it should stop parsing other files - if ($getout) { - kill(12, $parent_pid); + # Mark the directory as needing index update + if (open(OUT, ">>$last_parsed.tmp")) { + flock(OUT, 2) || return $getout; + print OUT "$incr_date\n"; + close(OUT); + } else { + &logmsg('ERROR', "can't save last parsed line into $last_parsed.tmp, $!"); + } + + # Save binary data + my $filenum = $$; + $filenum++ while (-e "$outdir/$bpath/$incr_date-$filenum.bin"); + my $fhb = new IO::File ">$outdir/$bpath/$incr_date-$filenum.bin"; + if (not defined $fhb) { + die "FATAL: can't write to $outdir/$bpath/$incr_date-$filenum.bin, $!\n"; + } + &dump_as_binary($fhb); + $fhb->close; + &init_stats_vars(); + + } elsif ($tmpoutfile) { + + &dump_as_binary($tmpoutfile); + $tmpoutfile->close(); + + } + + # Inform the parent that it should stop parsing other files + if ($getout) { + kill(12, $parent_pid); } # Save last line into temporary file if ($last_parsed && scalar keys %last_line) { if (open(OUT, ">>$tmp_last_parsed")) { flock(OUT, 2) || return $getout; - print OUT "$last_line{datetime}\t$last_line{orig}\n"; + $last_line{current_pos} ||= 0; + print OUT "$last_line{datetime}\t$last_line{current_pos}\t$last_line{orig}\n"; close(OUT); } else { - &logmsg('ERROR', "can't save last parsed line into $last_parsed, $!"); + &logmsg('ERROR', "can't save last parsed line into $tmp_last_parsed, $!"); } } - + return $getout; } @@ -1471,6 +1970,88 @@ } } +# Method used to check if the file stores logs after the last incremental position or not +# This position should have been saved in the incremental file and read in the $last_parsed at +# start up. Here we just verify that the first date in file is before the last incremental date. +sub check_file_changed +{ + my ($file, $saved_date) = @_; + + my ($lfile, $totalsize, $iscompressed) = &get_log_file($file); + + # Compressed files do not allow seeking + if ($iscompressed) { + close($lfile); + return 1; + # do not seek if filesize is smaller than the seek position + } elsif ($saved_last_line{current_pos} > $totalsize) { + close($lfile); + return 1; + } + + my ($gsec, $gmin, $ghour, $gmday, $gmon, $gyear, $gwday, $gyday, $gisdst) = localtime(time); + $gyear += 1900; + my $CURRENT_DATE = $gyear . sprintf("%02d", $gmon + 1) . sprintf("%02d", $gmday); + + %prefix_vars = (); + while (my $line = <$lfile>) { + + if ($format =~ /syslog/) { + + my @matches = ($line =~ $compiled_prefix); + if ($#matches >= 0) { + + for (my $i = 0 ; $i <= $#prefix_params ; $i++) { + $prefix_vars{$prefix_params[$i]} = $matches[$i]; + } + # Standard syslog format does not have year information, months are + # three letters and day are not always with 2 digit. + if ($prefix_vars{'t_month'} !~ /\d/) { + $prefix_vars{'t_year'} = $gyear; + $prefix_vars{'t_day'} = sprintf("%02d", $prefix_vars{'t_day'}); + $prefix_vars{'t_month'} = $month_abbr{$prefix_vars{'t_month'}}; + # Take care of year overlapping + if ("$prefix_vars{'t_year'}$prefix_vars{'t_month'}$prefix_vars{'t_day'}" > $CURRENT_DATE) { + $prefix_vars{'t_year'} = substr($CURRENT_DATE, 0, 4) - 1; + } + } + $prefix_vars{'t_timestamp'} = +"$prefix_vars{'t_year'}-$prefix_vars{'t_month'}-$prefix_vars{'t_day'} $prefix_vars{'t_hour'}:$prefix_vars{'t_min'}:$prefix_vars{'t_sec'}"; + if ($saved_date gt $prefix_vars{'t_timestamp'}) { + close($lfile); + return 0; + } else { + last; + } + } + + } elsif ($format eq 'stderr') { + + my @matches = ($line =~ $compiled_prefix); + if ($#matches >= 0) { + for (my $i = 0 ; $i <= $#prefix_params ; $i++) { + $prefix_vars{$prefix_params[$i]} = $matches[$i]; + } + if (!$prefix_vars{'t_timestamp'} && $prefix_vars{'t_mtimestamp'}) { + $prefix_vars{'t_timestamp'} = $prefix_vars{'t_mtimestamp'}; + } elsif (!$prefix_vars{'t_timestamp'} && $prefix_vars{'t_session_timestamp'}) { + $prefix_vars{'t_timestamp'} = $prefix_vars{'t_session_timestamp'}; + } + } + if ($saved_date gt $prefix_vars{'t_timestamp'}) { + close($lfile); + return 0; + } else { + last; + } + } + } + close($lfile); + + return 1; +} + + # Method used to check if we have already reach the last parsing position in incremental mode # This position should have been saved in the incremental file and read in the $last_parsed at # start up. @@ -1490,6 +2071,63 @@ $last_line{orig} = $line; } + # In incremental mode data are saved to disk per day + if ($incremental) { + $cur_date =~ s/\s.*$//; + # Check if the current day has changed, if so save data + $incr_date = $cur_date if (!$incr_date); + if ($cur_date gt $incr_date) { + + # Get stats from all pending temporary storage + foreach my $pid (sort {$cur_info{$a}{date} <=> $cur_info{$b}{date}} keys %cur_info) { + # Stores last queries information + &store_queries($pid); + } + # Stores last temporary files and locks information + foreach my $pid (keys %cur_temp_info) { + &store_temporary_and_lock_infos($pid); + } + # Stores last temporary files and locks information + foreach my $pid (keys %cur_lock_info) { + &store_temporary_and_lock_infos($pid); + } + + if ($extension eq 'tsung') { + foreach my $pid (sort {$a <=> $b} keys %tsung_session) { + &store_tsung_session($pid); + } + } + + # set path and create subdirectories + my $bpath = $incr_date; + while ($bpath =~ s/([^\-]+)\-/$1\//) { + mkdir("$outdir/$1") if (!-d "$outdir/$1"); + } + mkdir("$outdir/$bpath") if (!-d "$outdir/$bpath"); + + # Mark this directory as needing a reindex + if (open(OUT, ">>$last_parsed.tmp")) { + flock(OUT, 2) || return 1; + print OUT "$incr_date\n"; + close(OUT); + } else { + &logmsg('ERROR', "can't save last parsed line into $last_parsed.tmp, $!"); + } + + # Save binary data + my $filenum = $$; + $filenum++ while (-e "$outdir/$bpath/$incr_date-$filenum.bin"); + my $fhb = new IO::File ">$outdir/$bpath/$incr_date-$filenum.bin"; + if (not defined $fhb) { + die "FATAL: can't write to $outdir/$bpath/$incr_date-$filenum.bin, $!\n"; + } + &dump_as_binary($fhb); + $fhb->close; + $incr_date = $cur_date; + &init_stats_vars(); + } + } + return 1; } @@ -1548,6 +2186,8 @@ { return 0 if ($#_ < 0); + return 0 if (!$_[0]); + my $text = reverse $_[0]; $text =~ s/(\d\d\d)(?=\d)(?!\d*\.)/$1$num_sep/g; @@ -1555,6 +2195,35 @@ return scalar reverse $text; } +# Format numbers with comma for better reading +sub pretty_print_size +{ + my $val = shift; + return 0 if (!$val); + + if ($val >= 1125899906842624) { + $val = ($val / 1125899906842624); + $val = sprintf("%0.2f", $val) . " PiB"; + } elsif ($val >= 1099511627776) { + $val = ($val / 1099511627776); + $val = sprintf("%0.2f", $val) . " TiB"; + } elsif ($val >= 1073741824) { + $val = ($val / 1073741824); + $val = sprintf("%0.2f", $val) . " GiB"; + } elsif ($val >= 1048576) { + $val = ($val / 1048576); + $val = sprintf("%0.2f", $val) . " MiB"; + } elsif ($val >= 1024) { + $val = ($val / 1024); + $val = sprintf("%0.2f", $val) . " KiB"; + } else { + $val = $val . " B"; + } + + return $val; +} + + # Format duration sub convert_time { @@ -1573,6 +2242,7 @@ $days = $days < 1 ? '' : $days . 'd'; $hours = $hours < 1 ? '' : $hours . 'h'; $minutes = $minutes < 1 ? '' : $minutes . 'm'; + $seconds =~ s/\.\d+$// if ($minutes); $time = $days . $hours . $minutes . $seconds . 's'; return $time; @@ -1652,7 +2322,7 @@ # Stop when we have our number of samples if (!exists $error_info{$q}{date} || ($#{$error_info{$q}{date}} < $sample)) { - if (($q =~ /deadlock detected/) || !grep(/\Q$real_error\E/, @{$error_info{$q}{error}})) { + if ( ($q =~ /deadlock detected/) || ($real_error && !grep(/\Q$real_error\E/, @{$error_info{$q}{error}})) ) { push(@{$error_info{$q}{date}}, $date); push(@{$error_info{$q}{detail}}, $detail); push(@{$error_info{$q}{context}}, $context); @@ -1678,7 +2348,7 @@ $logfile_str .= ', ..., ' . $log_files[-1]; } print $fh qq{ -$report_title +pgBadger :: $report_title - Global information --------------------------------------------------- @@ -1689,9 +2359,11 @@ }; # Overall statistics - my $fmt_unique = &comma_numbers(scalar keys %normalyzed_info) || 0; - my $fmt_queries = &comma_numbers($overall_stat{'queries_number'}) || 0; - my $fmt_duration = &convert_time($overall_stat{'queries_duration'}) || 0; + my $fmt_unique = &comma_numbers(scalar keys %normalyzed_info); + my $fmt_queries = &comma_numbers($overall_stat{'queries_number'}); + my $fmt_duration = &convert_time($overall_stat{'queries_duration'}); + $overall_stat{'first_query_ts'} ||= '-'; + $overall_stat{'last_query_ts'} ||= '-'; print $fh qq{ - Overall statistics --------------------------------------------------- @@ -1702,20 +2374,20 @@ First query: $overall_stat{'first_query_ts'} Last query: $overall_stat{'last_query_ts'} }; - foreach (sort {$overall_stat{'query_peak'}{$b} <=> $overall_stat{'query_peak'}{$a}} keys %{$overall_stat{'query_peak'}}) { - print $fh "Query peak: ", &comma_numbers($overall_stat{'query_peak'}{$_}), " queries/s at $_"; + foreach (sort {$overall_stat{'peak'}{$b}{query} <=> $overall_stat{'peak'}{$a}{query}} keys %{$overall_stat{'peak'}}) { + print $fh "Query peak: ", &comma_numbers($overall_stat{'peak'}{$_}{query}), " queries/s at $_"; last; } if (!$disable_error) { - my $fmt_errors = &comma_numbers($overall_stat{'errors_number'}) || 0; - my $fmt_unique_error = &comma_numbers(scalar keys %{$overall_stat{'unique_normalized_errors'}}) || 0; + my $fmt_errors = &comma_numbers($overall_stat{'errors_number'}); + my $fmt_unique_error = &comma_numbers(scalar keys %error_info); print $fh qq{ Number of events: $fmt_errors Number of unique normalized events: $fmt_unique_error }; } if ($tempfile_info{count}) { - my $fmt_temp_maxsise = &comma_numbers($tempfile_info{maxsize}) || 0; + my $fmt_temp_maxsise = &comma_numbers($tempfile_info{maxsize}); my $fmt_temp_avsize = &comma_numbers(sprintf("%.2f", ($tempfile_info{size} / $tempfile_info{count}))); print $fh qq{Number temporary files: $tempfile_info{count} Max size of temporary files: $fmt_temp_maxsise @@ -1729,9 +2401,19 @@ Total duration of sessions: $tot_session_duration Average duration of sessions: $avg_session_duration }; + foreach (sort {$overall_stat{'peak'}{$b}{session} <=> $overall_stat{'peak'}{$a}{session}} keys %{$overall_stat{'peak'}}) { + print $fh "Session peak: ", &comma_numbers($overall_stat{'peak'}{$_}{session}), " sessions at $_"; + last; + } } if (!$disable_connection && $connection_info{count}) { print $fh "Total number of connections: $connection_info{count}\n"; + foreach (sort {$overall_stat{'peak'}{$b}{connection} <=> $overall_stat{'peak'}{$a}{connection}} keys %{$overall_stat{'peak'}}) { + if ($overall_stat{'peak'}{$_}{connection} > 0) { + print $fh "Connection peak: ", &comma_numbers($overall_stat{'peak'}{$_}{connection}), " conn/s at $_"; + } + last; + } } if (scalar keys %database_info > 1) { print $fh "Total number of databases: ", scalar keys %database_info, "\n"; @@ -1747,21 +2429,19 @@ } # INSERT/DELETE/UPDATE/SELECT repartition - my $totala = $overall_stat{'SELECT'} + $overall_stat{'INSERT'} + $overall_stat{'UPDATE'} + $overall_stat{'DELETE'}; + my $totala = 0; + foreach my $a (@SQL_ACTION) { + $totala += $overall_stat{$a}; + } if (!$disable_type && $totala) { my $total = $overall_stat{'queries_number'} || 1; print $fh "\n- Queries by type ------------------------------------------------------\n\n"; print $fh "Type Count Percentage\n"; - print $fh "SELECT: ", &comma_numbers($overall_stat{'SELECT'}) || 0, " ", - sprintf("%0.2f", ($overall_stat{'SELECT'} * 100) / $total), "%\n"; - print $fh "INSERT: ", &comma_numbers($overall_stat{'INSERT'}) || 0, " ", - sprintf("%0.2f", ($overall_stat{'INSERT'} * 100) / $total), "%\n"; - print $fh "UPDATE: ", &comma_numbers($overall_stat{'UPDATE'}) || 0, " ", - sprintf("%0.2f", ($overall_stat{'UPDATE'} * 100) / $total), "%\n"; - print $fh "DELETE: ", &comma_numbers($overall_stat{'DELETE'}) || 0, " ", - sprintf("%0.2f", ($overall_stat{'DELETE'} * 100) / $total), "%\n"; - print $fh "OTHERS: ", &comma_numbers($total - $totala) || 0, " ", sprintf("%0.2f", (($total - $totala) * 100) / $total), "%\n" + foreach my $a (@SQL_ACTION) { + print $fh "$a: ", &comma_numbers($overall_stat{$a}), " ", sprintf("%0.2f", ($overall_stat{$a} * 100) / $total), "%\n"; + } + print $fh "OTHERS: ", &comma_numbers($total - $totala), " ", sprintf("%0.2f", (($total - $totala) * 100) / $total), "%\n" if (($total - $totala) > 0); print $fh "\n"; @@ -1790,6 +2470,35 @@ } } } + + # Show request per user statistics + if (scalar keys %user_info > 1) { + print $fh "\n- Request per user ------------------------------------------------------\n\n"; + print $fh "User Request type Count\n"; + foreach my $d (sort keys %user_info) { + print $fh "$d - ", &comma_numbers($user_info{$d}{count}), "\n"; + foreach my $r (sort keys %{$user_info{$d}}) { + next if ($r eq 'count'); + print $fh "\t$r ", &comma_numbers($user_info{$d}{$r}), "\n"; + } + } + } + + # Show request per user statistics + if (scalar keys %user_info > 1) { + print $fh "\n- Request per user ------------------------------------------------------\n\n"; + print $fh "Host Request type Count\n"; + foreach my $d (sort keys %user_info) { + print $fh "$d - ", &comma_numbers($user_info{$d}{count}), "\n"; + foreach my $r (sort keys %{$user_info{$d}}) { + next if ($r eq 'count'); + print $fh "\t$r ", &comma_numbers($user_info{$d}{$r}), "\n"; + } + } + } + + + } if (!$disable_lock && scalar keys %lock_info > 0) { @@ -1875,7 +2584,7 @@ } } - # Show lock wait detailed informations + # Show lock wait detailed information if (!$disable_lock && scalar keys %lock_info > 0) { my @top_locked_queries; @@ -1931,7 +2640,7 @@ print $fh "\n"; } - # Show temporary files detailed informations + # Show temporary files detailed information if (!$disable_temporary && scalar keys %tempfile_info > 0) { my @top_temporary; @@ -2136,7 +2845,7 @@ $logfile_str .= ', ..., ' . $log_files[-1]; } print $fh qq{ -$report_title +pgBadger :: $report_title - Global information --------------------------------------------------- @@ -2163,16 +2872,20 @@ last if ($idx > $top); if ($error_info{$k}{count} > 1) { my $msg = $k; - $msg =~ s/HINT: (parameter "[^"]+" changed to)/LOG: $1/; - $msg =~ s/HINT: (database system was shut down)/LOG: $1/; + $msg =~ s/ERROR: (parameter "[^"]+" changed to)/LOG: $1/; + $msg =~ s/ERROR: (database system was shut down)/LOG: $1/; + $msg =~ s/ERROR: (recovery has paused)/LOG: $1/; + $msg =~ s/ERROR: (database system was interrupted while in recovery)/LOG: $1/; print $fh "$idx) " . &comma_numbers($error_info{$k}{count}) . " - $msg\n"; print $fh "--\n"; my $j = 1; for (my $i = 0 ; $i <= $#{$error_info{$k}{date}} ; $i++) { - if ( ($error_info{$k}{error}[$i] =~ s/HINT: (parameter "[^"]+" changed to)/LOG: $1/) - || ($error_info{$k}{error}[$i] =~ s/HINT: (database system was shut down)/LOG: $1/)) + if ( ($error_info{$k}{error}[$i] =~ s/ERROR: (parameter "[^"]+" changed to)/LOG: $1/) + || ($error_info{$k}{error}[$i] =~ s/ERROR: (database system was shut down)/LOG: $1/) + || ($error_info{$k}{error}[$i] =~ s/ERROR: (database system was interrupted while in recovery)/LOG: $1/) + || ($error_info{$k}{error}[$i] =~ s/ERROR: (recovery has paused)/LOG: $1/)) { - $logs_type{HINT}--; + $logs_type{ERROR}--; $logs_type{LOG}++; } print $fh "\t- Example $j: $error_info{$k}{date}[$i] - $error_info{$k}{error}[$i]\n"; @@ -2184,10 +2897,12 @@ $j++; } } else { - if ( ($error_info{$k}{error}[0] =~ s/HINT: (parameter "[^"]+" changed to)/LOG: $1/) - || ($error_info{$k}{error}[0] =~ s/HINT: (database system was shut down)/LOG: $1/)) + if ( ($error_info{$k}{error}[0] =~ s/ERROR: (parameter "[^"]+" changed to)/LOG: $1/) + || ($error_info{$k}{error}[0] =~ s/ERROR: (database system was shut down)/LOG: $1/) + || ($error_info{$k}{error}[0] =~ s/ERROR: (database system was interrupted while in recovery)/LOG: $1/) + || ($error_info{$k}{error}[0] =~ s/ERROR: (recovery has paused)/LOG: $1/)) { - $logs_type{HINT}--; + $logs_type{ERROR}--; $logs_type{LOG}++; } print $fh "$idx) " . &comma_numbers($error_info{$k}{count}) . " - $error_info{$k}{error}[0]\n"; @@ -2217,421 +2932,220 @@ } } -sub get_page_style -{ - return qq{ - -}; - -} - - sub html_header { my $date = localtime(time); - my $style = &get_page_style(); - print $fh qq{ - + my $global_info = &print_global_information(); + + print $fh qq{ + -$report_title - - - - -$style -}; - if (!$nograph) { - my @jscode = ; - print $fh <pgBadger :: $report_title + + + + + + @jscode -EOF - } - print $fh qq{ -
- -

$report_title

-}; - print $fh qq{ -
-Overall statistics | +
+ + +


+
+ +
    +}; } sub html_footer { print $fh qq{ -

     

    -
    - Report generated by pgBadger $VERSION. -
    + +
+
+ + - -
-
Table of contents
+
+
-}; - print $fh qq{ + }; } -sub dump_as_html -{ - # Dump the html header - &html_header(); +# Create global information section +sub print_global_information +{ - # Global information my $curdate = localtime(time); my $fmt_nlines = &comma_numbers($nlines); my $total_time = timestr($td); @@ -2641,1835 +3155,4065 @@ if ($#log_files > 0) { $logfile_str .= ', ..., ' . $log_files[-1]; } - print $fh qq{ -
+ return qq{
  • Generated on $curdate
  • Log file: $logfile_str
  • Parsed $fmt_nlines log entries in $total_time
  • Log start from $overall_stat{'first_log_ts'} to $overall_stat{'last_log_ts'}
-
}; - # Overall statistics - my $fmt_unique = &comma_numbers(scalar keys %normalyzed_info) || 0; - my $fmt_queries = &comma_numbers($overall_stat{'queries_number'}) || 0; - my $fmt_duration = &convert_time($overall_stat{'queries_duration'}) || 0; - print $fh qq{ -
-

Overall statistics ^

-
- + + + + + + + }; + $count = &comma_numbers($cur_period_info{'SELECT'}{count}); + $average = &convert_time($cur_period_info{'SELECT'}{average}); + $select_queries .= qq{ + + + + + + }; + my $insert_count = &comma_numbers($cur_period_info{'INSERT'}{count}); + my $update_count = &comma_numbers($cur_period_info{'UPDATE'}{count}); + my $delete_count = &comma_numbers($cur_period_info{'DELETE'}{count}); + my $write_average = &convert_time($write_average_duration / ($write_average_count || 1)); + $write_queries .= qq{ + + + + + + + + }; + my $prepare_count = &comma_numbers($cur_period_info{prepare}); + my $execute_count = &comma_numbers($cur_period_info{execute}); + my $bind_prepare = &comma_numbers(sprintf("%.2f", $cur_period_info{execute}/($cur_period_info{prepare}||1))); + my $prepare_usual = &comma_numbers(sprintf("%.2f", ($cur_period_info{prepare}/($cur_period_info{usual}||1)) * 100)) . "%"; + $prepared_queries .= qq{ + + + + + + + + }; + $count = &comma_numbers($connection_info{chronos}{"$d"}{"$h"}{count}); + $average = &comma_numbers(sprintf("%0.2f", $connection_info{chronos}{"$d"}{"$h"}{count} / 3600)); + $connections .= qq{ + + + + + + }; + $count = &comma_numbers($session_info{chronos}{"$d"}{"$h"}{count}); + $cur_period_info{'session'}{average} = + $session_info{chronos}{"$d"}{"$h"}{duration} / ($session_info{chronos}{"$d"}{"$h"}{count} || 1); + $average = &convert_time($cur_period_info{'session'}{average}); + $sessions .= qq{ + + + + + + }; + } + } + + # Set default values + $queries = qq{} if (!$queries); + $select_queries = qq{} if (!$select_queries); + $write_queries = qq{} if (!$write_queries); + $prepared_queries = qq{} if (!$prepared_queries); + $connections = qq{} if (!$connections); + $sessions = qq{} if (!$sessions); + + print $fh qq{ +
+

General Activity

+
+ +
+
+
-
    -
  • Number of unique normalized queries: $fmt_unique
    • -
  • Number of queries: $fmt_queries
    • -
  • Total query duration: $fmt_duration
    • -
  • First query: $overall_stat{'first_query_ts'}
    • -
  • Last query: $overall_stat{'last_query_ts'}
    • -}; - foreach (sort {$overall_stat{'query_peak'}{$b} <=> $overall_stat{'query_peak'}{$a}} keys %{$overall_stat{'query_peak'}}) { - print $fh "
  • Query peak: ", &comma_numbers($overall_stat{'query_peak'}{$_}), " queries/s at $_
    • "; +} + +sub print_overall_statistics +{ + + my $fmt_unique = &comma_numbers(scalar keys %normalyzed_info); + my $fmt_queries = &comma_numbers($overall_stat{'queries_number'}); + my $fmt_duration = &convert_time($overall_stat{'queries_duration'}); + $overall_stat{'first_query_ts'} ||= '-'; + $overall_stat{'last_query_ts'} ||= '-'; + my $query_peak = 0; + my $query_peak_date = ''; + foreach (sort {$overall_stat{'peak'}{$b}{query} <=> $overall_stat{'peak'}{$a}{query}} keys %{$overall_stat{'peak'}}) { + $query_peak = &comma_numbers($overall_stat{'peak'}{$_}{query}); + $query_peak_date = $_; last; } - if (!$disable_error) { - my $fmt_errors = &comma_numbers($overall_stat{'errors_number'}) || 0; - my $fmt_unique_error = &comma_numbers(scalar keys %{$overall_stat{'unique_normalized_errors'}}) || 0; - print $fh qq{ -
  • Number of events: $fmt_errors
    • -
  • Number of unique normalized events: $fmt_unique_error
    • -}; - } - if ($autovacuum_info{count}) { - print $fh qq{ -
  • Total number of automatic vacuums: $autovacuum_info{count}
    • -}; + my $fmt_errors = &comma_numbers($overall_stat{'errors_number'}); + my $fmt_unique_error = &comma_numbers(scalar keys %error_info); + my $autovacuum_count = &comma_numbers($autovacuum_info{count}); + my $autoanalyze_count = &comma_numbers($autoanalyze_info{count}); + my $tempfile_count = &comma_numbers($tempfile_info{count}); + my $fmt_temp_maxsise = &comma_numbers($tempfile_info{maxsize}); + my $fmt_temp_avsize = &comma_numbers(sprintf("%.2f", $tempfile_info{size} / ($tempfile_info{count} || 1))); + my $session_count = &comma_numbers($session_info{count}); + my $avg_session_duration = &convert_time($session_info{duration} / ($session_info{count} || 1)); + my $tot_session_duration = &convert_time($session_info{duration}); + my $connection_count = &comma_numbers($connection_info{count}); + my $connection_peak = 0; + my $connection_peak_date = ''; + my $session_peak = 0; + my $session_peak_date = ''; + foreach (sort {$overall_stat{'peak'}{$b}{connection} <=> $overall_stat{'peak'}{$a}{connection}} keys %{$overall_stat{'peak'}}) { + $connection_peak = &comma_numbers($overall_stat{'peak'}{$_}{connection}); + $connection_peak_date = $_; + last; } - if ($autoanalyze_info{count}) { - print $fh qq{ -
  • Total number of automatic analyzes: $autoanalyze_info{count}
    • -}; + foreach (sort {$overall_stat{'peak'}{$b}{session} <=> $overall_stat{'peak'}{$a}{session}} keys %{$overall_stat{'peak'}}) { + $session_peak = &comma_numbers($overall_stat{'peak'}{$_}{session}); + $session_peak_date = $_; + last; } + my $main_error = 0; + my $total = 0; + foreach my $k (sort {$error_info{$b}{count} <=> $error_info{$a}{count}} keys %error_info) { + next if (!$error_info{$k}{count}); + $main_error = &comma_numbers($error_info{$k}{count}) if (!$main_error); + $total += $error_info{$k}{count}; + } + $total = &comma_numbers($total); + + my $db_count = scalar keys %database_info; print $fh qq{ -
-		 -
    +

    Overview

    + +
    +

    Global Stats

    +
    + +
    +
    +
      +
    • $fmt_unique Number of unique normalized queries
      • +
    • $fmt_queries Number of queries
      • +
    • $fmt_duration Total query duration
      • +
    • $overall_stat{'first_query_ts'} First query
      • +
    • $overall_stat{'last_query_ts'} Last query
      • +
    • $query_peak queries/s at $query_peak_date Query peak
      • +
    +
    +
    +
      +
    • $fmt_errors Number of events
      • +
    • $fmt_unique_error Number of unique normalized events
      • +
    • $main_error Max number of times the same event was reported
      • +
    +
    +
    +
      +
    • $autovacuum_count Total number of automatic vacuums
      • +
    • $autoanalyze_count Total number of automatic analyzes
      • +
    +
    +
    +
      +
    • $tempfile_count Number temporary file
      • +
    • $fmt_temp_maxsise Max size of temporary file
      • +
    • $fmt_temp_avsize Average size of temporary file
      • +
    +
    +
    +
      +
    • $session_count Total number of sessions
      • +
    • $session_peak sessions at $session_peak_date Session peak
      • +
    • $tot_session_duration Total duration of sessions
      • +
    • $avg_session_duration Average duration of sessions
      • +
    +
    +
    +
      +
    • $connection_count Total number of connections
      • }; - if ($tempfile_info{count}) { - my $fmt_temp_maxsise = &comma_numbers($tempfile_info{maxsize}) || 0; - my $fmt_temp_avsize = &comma_numbers(sprintf("%.2f", $tempfile_info{size} / $tempfile_info{count})); + if ($connection_count) { print $fh qq{ -
    • Number of temporary files: $tempfile_info{count}
      • -
    • Max size of temporary files: $fmt_temp_maxsise
      • -
    • Average size of temporary files: $fmt_temp_avsize
      • +
    • $connection_peak connections/s at $connection_peak_date Connection peak
      • }; } - if (!$disable_session && $session_info{count}) { - my $avg_session_duration = &convert_time($session_info{duration} / $session_info{count}); - my $tot_session_duration = &convert_time($session_info{duration}); - print $fh qq{ -
    • Total number of sessions: $session_info{count}
      • -
    • Total duration of sessions: $tot_session_duration
      • -
    • Average duration of sessions: $avg_session_duration
      • + print $fh qq{ +
    • $db_count Total number of databases
      • +
    +
    +
    +
    +
    }; - } - if (!$disable_connection && $connection_info{count}) { - print $fh qq{ -
  • Total number of connections: $connection_info{count}
    • + +} + +sub print_general_activity +{ + my $queries = ''; + my $select_queries = ''; + my $write_queries = ''; + my $prepared_queries = ''; + my $connections = ''; + my $sessions = ''; + foreach my $d (sort {$a <=> $b} keys %per_minute_info) { + my $c = 1; + $d =~ /^\d{4}(\d{2})(\d{2})$/; + my $zday = "$abbr_month{$1} $2"; + foreach my $h (sort {$a <=> $b} keys %{$per_minute_info{$d}}) { + my %cur_period_info = (); + my $write_average_duration = 0; + my $write_average_count = 0; + foreach my $m (keys %{$per_minute_info{$d}{$h}}) { + $cur_period_info{count} += ($per_minute_info{$d}{$h}{$m}{query}{count} || 0); + $cur_period_info{duration} += ($per_minute_info{$d}{$h}{$m}{query}{duration} || 0); + $cur_period_info{min} = $per_minute_info{$d}{$h}{$m}{query}{duration} if (!exists $cur_period_info{min} || ($per_minute_info{$d}{$h}{$m}{query}{duration} < $cur_period_info{min})); + $cur_period_info{max} = $per_minute_info{$d}{$h}{$m}{query}{duration} if (!exists $cur_period_info{max} || ($per_minute_info{$d}{$h}{$m}{query}{duration} > $cur_period_info{max})); + foreach my $a (@SQL_ACTION) { + $cur_period_info{$a}{count} += ($per_minute_info{$d}{$h}{$m}{$a}{count} || 0); + $cur_period_info{$a}{duration} += ($per_minute_info{$d}{$h}{$m}{$a}{duration} || 0); + $cur_period_info{usual} += ($per_minute_info{$d}{$h}{$m}{$a}{count} || 0); + } + $cur_period_info{prepare} += ($per_minute_info{$d}{$h}{$m}{prepare} || 0); + $cur_period_info{execute} += ($per_minute_info{$d}{$h}{$m}{execute} || 0); + } + + $cur_period_info{average} = $cur_period_info{duration} / ($cur_period_info{count} || 1); + $cur_period_info{'SELECT'}{average} = $cur_period_info{'SELECT'}{duration} / ($cur_period_info{'SELECT'}{count} || 1); + $write_average_duration = ($cur_period_info{'INSERT'}{duration} + + $cur_period_info{'UPDATE'}{duration} + + $cur_period_info{'DELETE'}{duration}); + $write_average_count = ($cur_period_info{'INSERT'}{count} + + $cur_period_info{'UPDATE'}{count} + + $cur_period_info{'DELETE'}{count}); + $zday = " " if ($c > 1); + $c++; + + my $count = &comma_numbers($cur_period_info{count}); + my $min = &convert_time($cur_period_info{min}); + my $max = &convert_time($cur_period_info{max}); + my $average = &convert_time($cur_period_info{average}); + $queries .= qq{ +
$zday$h$count$min$max$average
$zday$h$count$average
$zday$h$insert_count$update_count$delete_count$write_average
$zday$h$prepare_count$execute_count$bind_prepare$prepare_usual
$zday$h$count$average/s
$zday$h$count$average
$NODATA
$NODATA
$NODATA
$NODATA
$NODATA
$NODATA
+ + + + + + + + + + + $queries + +
DayHourCountMin durationMax durationAvg duration
+
+
+ + + + + + + + + + $select_queries + +
DayHourCountAverage Duration
+
+
+ + + + + + + + + + + + $write_queries + +
DayHourINSERTUPDATEDELETEAverage Duration
+
+
+ + + + + + + + + + + + $prepared_queries + +
DayHourPrepareBindBind/PreparePercentage of prepare
+
+
+ + + + + + + + + + $connections + +
DayHourCountAverage / Second
+
+
+ + + + + + + + + + $sessions + +
DayHourCountAverage Duration
+
+
+ ↑ Back to the top of the General Activity table + + + }; + +} + +sub print_sql_traffic +{ + + my $bind_vs_prepared = sprintf("%.2f", $overall_stat{'execute'} / ($overall_stat{'prepare'} || 1)); + my $total_usual_queries = 0; + map { $total_usual_queries += $overall_stat{$_}; } @SQL_ACTION; + my $prepared_vs_normal = sprintf("%.2f", ($overall_stat{'execute'} / ($total_usual_queries || 1))*100); + + my $query_peak = 0; + my $query_peak_date = ''; + foreach (sort {$overall_stat{'peak'}{$b}{query} <=> $overall_stat{'peak'}{$a}{query}} keys %{$overall_stat{'peak'}}) { + $query_peak = &comma_numbers($overall_stat{'peak'}{$_}{query}); + $query_peak_date = $_; + last; } - if (scalar keys %database_info > 1) { - my $db_count = scalar keys %database_info; - print $fh qq{ -
  • Total number of databases: $db_count
    • -}; + + my $select_peak = 0; + my $select_peak_date = ''; + foreach (sort {$overall_stat{'peak'}{$b}{select} <=> $overall_stat{'peak'}{$a}{select}} keys %{$overall_stat{'peak'}}) { + $select_peak = &comma_numbers($overall_stat{'peak'}{$_}{select}); + $select_peak_date = $_; + last; + } + + my $write_peak = 0; + my $write_peak_date = ''; + foreach (sort {$overall_stat{'peak'}{$b}{write} <=> $overall_stat{'peak'}{$a}{write}} keys %{$overall_stat{'peak'}}) { + $write_peak = &comma_numbers($overall_stat{'peak'}{$_}{write}); + $write_peak_date = $_; + last; } + my $fmt_duration = &convert_time($overall_stat{'queries_duration'}); print $fh qq{ - - - +
    +

    SQL Traffic

    +
    +

    Key values

    +
    +
      +
    • $query_peak queries/s Query Peak
      • +
    • $query_peak_date Date
      • +
    +
    +
    +
    +

    Queries per second ($avg_minutes minutes average)

    +$drawn_graphs{queriespersecond_graph} +
    +
    }; + delete $drawn_graphs{queriespersecond_graph}; - # Declare variables used to draw graphs - my @labels = (); - my @data1 = (); - my @data2 = (); - my @data3 = (); - my $d1 = ''; - my $d2 = ''; - my $d3 = ''; - my @avgs = (); - for (my $i = 0 ; $i < 59 ; $i += $avg_minutes) { - push(@avgs, sprintf("%02d", $i)); - } - push(@avgs, 59); - # Set graphs limits - $overall_stat{'first_log_ts'} =~ /(\d+)-(\d+)-(\d+) (\d+):(\d+):(\d+)/; - $t_min = timegm_nocheck(0, $5, $4, $3, $2 - 1, $1) * 1000; - $t_min -= ($avg_minutes * 60000); - $t_min_hour = timegm_nocheck(0, 0, $4, $3, $2 - 1, $1) * 1000; - $overall_stat{'last_log_ts'} =~ /(\d+)-(\d+)-(\d+) (\d+):(\d+):(\d+)/; - $t_max = timegm_nocheck(59, $5, $4, $3, $2 - 1, $1) * 1000; - $t_max += ($avg_minutes * 60000); - $t_max_hour = timegm_nocheck(0, 0, $4, $3, $2 - 1, $1) * 1000; - - # Start creating hourly reports - if (!$disable_hourly && ($overall_stat{'queries_number'} || exists $connection_info{chronos})) { - print $fh qq{ -

    Hourly statistics ^

    + print $fh qq{ +
    +

    SELECT Traffic

    +
    +

    Key values

    +
    +
      +
    • $select_peak queries/s Query Peak
      • +
    • $select_peak_date Date
      • +
    +
    +
    +
    +

    SELECT queries per second ($avg_minutes minutes average)

    +$drawn_graphs{selectqueries_graph} +
    +
    }; - } - if (!$disable_hourly && $overall_stat{'queries_number'}) { - print $fh qq{ - - - - - - - + delete $drawn_graphs{selectqueries_graph}; + + print $fh qq{ +
    +

    INSERT/UPDATE/DELETE Traffic

    +
    +

    Key values

    +
    +
      +
    • $write_peak queries/s Query Peak
      • +
    • $write_peak_date Date
      • +
    +
    +
    +
    +

    Write queries per second ($avg_minutes minutes average)

    +$drawn_graphs{writequeries_graph} +
    +
    }; - if (exists $connection_info{chronos}) { - print $fh " \n"; - } - if (exists $session_info{chronos}) { - print $fh " \n"; - } - print $fh qq{ - - - - - - - - - - + delete $drawn_graphs{writequeries_graph}; + + print $fh qq{ +
    +

    Queries duration

    +
    +

    Key values

    +
    +
      +
    • $fmt_duration Total query duration
      • +
    +
    +
    +
    +

    Average queries duration ($avg_minutes minutes average)

    +$drawn_graphs{durationqueries_graph} +
    +
    }; - if (exists $connection_info{chronos}) { - print $fh " \n"; - } - if (exists $session_info{chronos}) { - print $fh " \n"; - } - print $fh qq{ - + delete $drawn_graphs{durationqueries_graph}; + + print $fh qq{ +
    +

    Prepared queries ratio

    +
    +

    Key values

    +
    +
      +
    • $bind_vs_prepared Ratio of bind vs prepare
      • +
    • $prepared_vs_normal % Ratio between prepared and "usual" statements
      • +
    +
    +
    +
    +

    Ratio of bind vs prepare statements ($avg_minutes minutes average)

    +$drawn_graphs{bindpreparequeries_graph} +
    +
    }; + delete $drawn_graphs{bindpreparequeries_graph}; - foreach my $d (sort {$a <=> $b} keys %per_hour_info) { - my $c = 1; - $d =~ /^\d{4}(\d{2})(\d{2})$/; - my $zday = "$abbr_month{$1} $2"; - foreach my $h (sort {$a <=> $b} keys %{$per_hour_info{$d}}) { - my $colb = $c % 2; - $zday = " " if ($c > 1); - $per_hour_info{$d}{$h}{average} = $per_hour_info{$d}{$h}{duration} / ($per_hour_info{$d}{$h}{count} || 1); - $per_hour_info{$d}{$h}{'SELECT'}{average} = - $per_hour_info{$d}{$h}{'SELECT'}{duration} / ($per_hour_info{$d}{$h}{'SELECT'}{count} || 1); - my $write_average = ( - ( - $per_hour_info{$d}{$h}{'INSERT'}{duration} + - $per_hour_info{$d}{$h}{'UPDATE'}{duration} + - $per_hour_info{$d}{$h}{'DELETE'}{duration} - ) - || 0 - ) / ( - ( - $per_hour_info{$d}{$h}{'INSERT'}{count} + - $per_hour_info{$d}{$h}{'UPDATE'}{count} + - $per_hour_info{$d}{$h}{'DELETE'}{count} - ) - || 1 - ); - print $fh ""; - if (exists $connection_info{chronos}) { - print $fh ""; - } - if (exists $session_info{chronos}) { - $per_hour_info{$d}{$h}{'session'}{average} = - $session_info{chronos}{"$d"}{"$h"}{duration} / ($session_info{chronos}{"$d"}{"$h"}{count} || 1); - print $fh ""; - } - print $fh "\n"; - $c++; - } - } +} + +sub compute_query_graphs +{ + my %graph_data = (); + if ($graph) { + foreach my $tm (sort {$a <=> $b} keys %per_minute_info) { + $tm =~ /(\d{4})(\d{2})(\d{2})/; + my $y = $1 - 1900; + my $mo = $2 - 1; + my $d = $3; + foreach my $h ("00" .. "23") { + next if (!exists $per_minute_info{$tm}{$h}); + my %q_dataavg = (); + my %a_dataavg = (); + my %c_dataavg = (); + my %s_dataavg = (); + my %p_dataavg = (); + foreach my $m ("00" .. "59") { + next if (!exists $per_minute_info{$tm}{$h}{$m}); + + my $rd = &average_per_minutes($m, $avg_minutes); + + $p_dataavg{prepare}{"$rd"} += $per_minute_info{$tm}{$h}{$m}{prepare} + if (exists $per_minute_info{$tm}{$h}{$m}{prepare}); + $p_dataavg{prepare}{"$rd"} += $per_minute_info{$tm}{$h}{$m}{prepare} + if (exists $per_minute_info{$tm}{$h}{$m}{parse}); + $p_dataavg{execute}{"$rd"} += $per_minute_info{$tm}{$h}{$m}{execute} + if (exists $per_minute_info{$tm}{$h}{$m}{execute}); - print $fh "
    DayHourQueriesSELECT queriesWrite queriesConnectionsSessions
    CountMin/Max/Avg duration CountAvg duration INSERTUPDATEDELETEAvg duration CountAvg/sCountAvg duration 
    $zday$h", - &comma_numbers($per_hour_info{$d}{$h}{count}), "", - &convert_time($per_hour_info{$d}{$h}{min}),"/",&convert_time($per_hour_info{$d}{$h}{max}),"/",&convert_time($per_hour_info{$d}{$h}{average}), "", - &comma_numbers($per_hour_info{$d}{$h}{'SELECT'}{count} || 0), "", - &convert_time($per_hour_info{$d}{$h}{'SELECT'}{average} || 0), "", - &comma_numbers($per_hour_info{$d}{$h}{'INSERT'}{count} || 0), "", - &comma_numbers($per_hour_info{$d}{$h}{'UPDATE'}{count} || 0), "", - &comma_numbers($per_hour_info{$d}{$h}{'DELETE'}{count} || 0), "", - &convert_time($write_average), "", &comma_numbers($connection_info{chronos}{"$d"}{"$h"}{count} || 0), - "", - &comma_numbers(sprintf("%0.2f", $connection_info{chronos}{"$d"}{"$h"}{count} / 3600)), "/s", &comma_numbers($session_info{chronos}{"$d"}{"$h"}{count} || 0), - "", &convert_time($per_hour_info{$d}{$h}{'session'}{average}), "
    \n"; + if (exists $per_minute_info{$tm}{$h}{$m}{query}) { - if ($graph) { + # Average per minute + $q_dataavg{count}{"$rd"} += $per_minute_info{$tm}{$h}{$m}{query}{count}; + if (exists $per_minute_info{$tm}{$h}{$m}{query}{duration}) { + $q_dataavg{duration}{"$rd"} += $per_minute_info{$tm}{$h}{$m}{query}{duration}; + } - foreach my $tm (sort {$a <=> $b} keys %{$per_minute_info{query}}) { - $tm =~ /(\d{4})(\d{2})(\d{2})/; - my $y = $1 - 1900; - my $mo = $2 - 1; - my $d = $3; - foreach my $h ("00" .. "23") { - my %dataavg = (); - foreach my $m ("00" .. "59") { - my $rd = &average_per_minutes($m, $avg_minutes); - if (exists $per_minute_info{query}{$tm}{$h}{$m}) { - - # Average per minute - $dataavg{average}{"$rd"} += $per_minute_info{query}{$tm}{$h}{$m}{count}; - - # Search minimum and maximum during this minute - foreach my $s (keys %{$per_minute_info{query}{$tm}{$h}{$m}{second}}) { - $dataavg{max}{"$rd"} = $per_minute_info{query}{$tm}{$h}{$m}{second}{$s} - if ($per_minute_info{query}{$tm}{$h}{$m}{second}{$s} > $dataavg{max}{"$rd"}); - $dataavg{min}{"$rd"} = $per_minute_info{query}{$tm}{$h}{$m}{second}{$s} - if (not exists $dataavg{min}{"$rd"} - || ($per_minute_info{query}{$tm}{$h}{$m}{second}{$s} < $dataavg{min}{"$rd"})); + # Search minimum and maximum during this minute + $q_dataavg{max}{"$rd"} = 0 if (!$q_dataavg{max}{"$rd"}); + $q_dataavg{min}{"$rd"} = 0 if (!$q_dataavg{min}{"$rd"}); + foreach my $s (keys %{$per_minute_info{$tm}{$h}{$m}{query}{second}}) { + $q_dataavg{max}{"$rd"} = $per_minute_info{$tm}{$h}{$m}{query}{second}{$s} + if ($per_minute_info{$tm}{$h}{$m}{query}{second}{$s} > $q_dataavg{max}{"$rd"}); + $q_dataavg{min}{"$rd"} = $per_minute_info{$tm}{$h}{$m}{query}{second}{$s} + if ($per_minute_info{$tm}{$h}{$m}{query}{second}{$s} < $q_dataavg{min}{"$rd"}); + } + + if (!$disable_query) { + foreach my $action (@SQL_ACTION) { + next if (!$per_minute_info{$tm}{$h}{$m}{$action}{count}); + $a_dataavg{$action}{count}{"$rd"} += ($per_minute_info{$tm}{$h}{$m}{$action}{count} || 0); + $a_dataavg{$action}{duration}{"$rd"} += ($per_minute_info{$tm}{$h}{$m}{$action}{duration} || 0); + if ( ($action ne 'SELECT') && exists $per_minute_info{$tm}{$h}{$m}{$action}{count}) { + $a_dataavg{write}{count}{"$rd"} += ($per_minute_info{$tm}{$h}{$m}{$action}{count} || 0); + $a_dataavg{write}{duration}{"$rd"} += ($per_minute_info{$tm}{$h}{$m}{$action}{duration} || 0); + } + # Search minimum and maximum during this minute + $a_dataavg{$action}{max}{"$rd"} = 0 if (! exists $a_dataavg{$action}{max}{"$rd"}); + $a_dataavg{$action}{min}{"$rd"} = 0 if (! exists $a_dataavg{$action}{min}{"$rd"}); + foreach my $s (keys %{$per_minute_info{$tm}{$h}{$m}{$action}{second}}) { + $a_dataavg{$action}{max}{"$rd"} = $per_minute_info{$tm}{$h}{$m}{$action}{second}{$s} + if ($per_minute_info{$tm}{$h}{$m}{$action}{second}{$s} > $a_dataavg{$action}{max}{"$rd"}); + $a_dataavg{$action}{min}{"$rd"} = $per_minute_info{$tm}{$h}{$m}{$action}{second}{$s} + if ($per_minute_info{$tm}{$h}{$m}{$action}{second}{$s} < $a_dataavg{$action}{min}{"$rd"}); + } } } } - foreach my $rd (@avgs) { - my $t = timegm_nocheck(0, $rd, $h, $d, $mo, $y) * 1000; - next if ($t < $t_min); - last if ($t > $t_max); + if (exists $per_minute_info{$tm}{$h}{$m}{connection}) { # Average per minute - $d2 .= "[$t, " . int(($dataavg{average}{"$rd"} || 0) / (60 * $avg_minutes)) . "],"; - - # Maxi per minute - $d1 .= "[$t, " . ($dataavg{max}{"$rd"} || 0) . "],"; + $c_dataavg{average}{"$rd"} += $per_minute_info{$tm}{$h}{$m}{connection}{count}; - # Mini per minute - $d3 .= "[$t, " . ($dataavg{min}{"$rd"} || 0) . "],"; + # Search minimum and maximum during this minute + $c_dataavg{max}{"$rd"} = 0 if (!$c_dataavg{max}{"$rd"}); + $c_dataavg{min}{"$rd"} = 0 if (!$c_dataavg{min}{"$rd"}); + foreach my $s (keys %{$per_minute_info{$tm}{$h}{$m}{connection}{second}}) { + $c_dataavg{max}{"$rd"} = $per_minute_info{$tm}{$h}{$m}{connection}{second}{$s} + if ($per_minute_info{$tm}{$h}{$m}{connection}{second}{$s} > $c_dataavg{max}{"$rd"}); + $c_dataavg{min}{"$rd"} = $per_minute_info{$tm}{$h}{$m}{connection}{second}{$s} + if ($per_minute_info{$tm}{$h}{$m}{connection}{second}{$s} < $c_dataavg{min}{"$rd"}); + } + delete $per_minute_info{$tm}{$h}{$m}{connection}; } - } - } - delete $per_minute_info{query}; - $d1 =~ s/,$//; - $d2 =~ s/,$//; - $d3 =~ s/,$//; - &flotr2_graph( - 1, 'queriespersecond_graph', $d1, $d2, $d3, 'Queries per second (' . $avg_minutes . ' minutes average)', - 'Queries per second', 'Maximum', 'Average', 'Minimum' - ); - $d1 = ''; - $d2 = ''; - $d3 = ''; - } - } - if (!$disable_hourly && $connection_info{'count'}) { - if ($graph) { - if (exists $per_minute_info{connection}) { - foreach my $tm (sort {$a <=> $b} keys %{$per_minute_info{connection}}) { - $tm =~ /(\d{4})(\d{2})(\d{2})/; - my $y = $1 - 1900; - my $mo = $2 - 1; - my $d = $3; - foreach my $h ("00" .. "23") { - my %dataavg = (); - foreach my $m ("00" .. "59") { - my $rd = &average_per_minutes($m, $avg_minutes); - if (exists $per_minute_info{connection}{$tm}{$h}{$m}) { - # Average per minute - $dataavg{average}{"$rd"} += $per_minute_info{connection}{$tm}{$h}{$m}{count}; + if (exists $per_minute_info{$tm}{$h}{$m}{session}) { - # Search minimum and maximum during this minute - foreach my $s (keys %{$per_minute_info{connection}{$tm}{$h}{$m}{second}}) { - $dataavg{max}{"$rd"} = $per_minute_info{connection}{$tm}{$h}{$m}{second}{$s} - if ($per_minute_info{connection}{$tm}{$h}{$m}{second}{$s} > $dataavg{max}{"$rd"}); - $dataavg{min}{"$rd"} = $per_minute_info{connection}{$tm}{$h}{$m}{second}{$s} - if (not exists $dataavg{min}{"$rd"} - || ($per_minute_info{connection}{$tm}{$h}{$m}{second}{$s} < $dataavg{min}{"$rd"})); - } - } + # Average per minute + $s_dataavg{average}{"$rd"} += $per_minute_info{$tm}{$h}{$m}{session}{count}; + + # Search minimum and maximum during this minute + $s_dataavg{max}{"$rd"} = 0 if (!$s_dataavg{max}{"$rd"}); + $s_dataavg{min}{"$rd"} = 0 if (!$s_dataavg{min}{"$rd"}); + foreach my $s (keys %{$per_minute_info{$tm}{$h}{$m}{session}{second}}) { + $s_dataavg{max}{"$rd"} = $per_minute_info{$tm}{$h}{$m}{session}{second}{$s} + if ($per_minute_info{$tm}{$h}{$m}{session}{second}{$s} > $s_dataavg{max}{"$rd"}); + $s_dataavg{min}{"$rd"} = $per_minute_info{$tm}{$h}{$m}{session}{second}{$s} + if ($per_minute_info{$tm}{$h}{$m}{session}{second}{$s} < $s_dataavg{min}{"$rd"}); } - foreach my $rd (@avgs) { - my $t = timegm_nocheck(0, $rd, $h, $d, $mo, $y) * 1000; + delete $per_minute_info{$tm}{$h}{$m}{session}; + } + } - next if ($t < $t_min); - last if ($t > $t_max); + foreach my $rd (@avgs) { + my $t = timegm_nocheck(0, $rd, $h, $d, $mo, $y) * 1000; - # Average per minute - $d2 .= "[$t, " . int(($dataavg{average}{"$rd"} || 0) / (60 * $avg_minutes)) . "],"; + next if ($t < $t_min); + last if ($t > $t_max); - # Maxi per minute - $d1 .= "[$t, " . ($dataavg{max}{"$rd"} || 0) . "],"; + if (exists $q_dataavg{count}) { + # Average queries per minute + $graph_data{query} .= "[$t, " . int(($q_dataavg{count}{"$rd"} || 0) / (60 * $avg_minutes)) . "],"; + # Maxi queries per minute + $graph_data{'query-max'} .= "[$t, " . ($q_dataavg{max}{"$rd"} || 0) . "],"; + # Mini queries per minute + $graph_data{'query-min'} .= "[$t, " . ($q_dataavg{min}{"$rd"} || 0) . "],"; + # Average duration per minute + $graph_data{query4} .= "[$t, " . sprintf("%.3f", ($q_dataavg{duration}{"$rd"} || 0) / ($q_dataavg{count}{"$rd"} || 1)) . "],"; + } + if (scalar keys %c_dataavg) { + # Average connections per minute + $graph_data{conn_avg} .= "[$t, " . int(($c_dataavg{average}{"$rd"} || 0) / (60 * $avg_minutes)) . "],"; + # Maxi connections per minute + $graph_data{conn_max} .= "[$t, " . ($c_dataavg{max}{"$rd"} || 0) . "],"; - # Mini per minute - $d3 .= "[$t, " . ($dataavg{min}{"$rd"} || 0) . "],"; - } + # Mini connections per minute + $graph_data{conn_min} .= "[$t, " . ($c_dataavg{min}{"$rd"} || 0) . "],"; } - } - delete $per_minute_info{connection}; - $d1 =~ s/,$//; - $d2 =~ s/,$//; - $d3 =~ s/,$//; - &flotr2_graph( - 2, 'connectionspersecond_graph', $d1, $d2, $d3, 'Connections per second (' . $avg_minutes . ' minutes average)', - 'Connections per second', 'Maximum', 'Average', 'Minimum' - ); - $d1 = ''; - $d2 = ''; - $d3 = ''; - } - } - } + if (scalar keys %s_dataavg) { + # Average connections per minute + $graph_data{sess_avg} .= "[$t, " . int(($s_dataavg{average}{"$rd"} || 0) / (60 * $avg_minutes)) . "],"; + # Maxi connections per minute + $graph_data{sess_max} .= "[$t, " . ($s_dataavg{max}{"$rd"} || 0) . "],"; - if (!$disable_hourly && $overall_stat{'queries_number'}) { - if ($graph) { - # All queries - foreach my $tm (sort {$a <=> $b} keys %per_hour_info) { - $tm =~ /(\d{4})(\d{2})(\d{2})/; - my $y = $1 - 1900; - my $mo = $2 - 1; - my $d = $3; - foreach my $h ("00" .. "23") { - my $t = timegm_nocheck(0, 0, $h, $d, $mo, $y) * 1000; - next if ($t < $t_min_hour); - last if ($t > $t_max_hour); - $d1 .= "[$t, " . ($per_hour_info{$tm}{$h}{count} || 0) . "],"; - $d2 .= "[$t, " - . sprintf("%.2f", (($per_hour_info{$tm}{$h}{duration} || 0) / ($per_hour_info{$tm}{$h}{count} || 1)) / 1000) - . "],"; - } - } - $d1 =~ s/,$//; - $d2 =~ s/,$//; - &flotr2_graph( - 3, 'allqueries_graph', $d1, '', '', 'All queries', - 'Queries', 'Number of queries', '', '', 'Duration', $d2, 'Average duration (s)' - ); - $d1 = ''; - $d2 = ''; - - if (!$disable_query) { - - # Select queries - foreach my $tm (sort {$a <=> $b} keys %per_hour_info) { - $tm =~ /(\d{4})(\d{2})(\d{2})/; - my $y = $1 - 1900; - my $mo = $2 - 1; - my $d = $3; - foreach my $h ("00" .. "23") { - my $t = timegm_nocheck(0, 0, $h, $d, $mo, $y) * 1000; - next if ($t < $t_min_hour); - last if ($t > $t_max_hour); - $d1 .= "[$t, " . ($per_hour_info{$tm}{$h}{'SELECT'}{count} || 0) . "],"; - $d2 .= "[$t, " - . sprintf( - "%.2f", - (($per_hour_info{$tm}{$h}{'SELECT'}{duration} || 0) / ($per_hour_info{$tm}{$h}{'SELECT'}{count} || 1)) / - 1000 - ) . "],"; + # Mini connections per minute + $graph_data{sess_min} .= "[$t, " . ($s_dataavg{min}{"$rd"} || 0) . "],"; } - } - $d1 =~ s/,$//; - $d2 =~ s/,$//; - &flotr2_graph( - 4, 'selectqueries_graph', $d1, '', '', 'SELECT queries', - 'Queries', 'Number of queries', '', '', 'Duration', $d2, 'Average duration (s)' - ); - $d1 = ''; - $d2 = ''; - - # Write queries - if (!$select_only) { - my $d4 = ''; - foreach my $tm (sort {$a <=> $b} keys %per_hour_info) { - $tm =~ /(\d{4})(\d{2})(\d{2})/; - my $y = $1 - 1900; - my $mo = $2 - 1; - my $d = $3; - foreach my $h ("00" .. "23") { - my $t = timegm_nocheck(0, 0, $h, $d, $mo, $y) * 1000; - next if ($t < $t_min_hour); - last if ($t > $t_max_hour); - my $wcount = - $per_hour_info{$tm}{$h}{'UPDATE'}{count} + - $per_hour_info{$tm}{$h}{'DELETE'}{count} + - $per_hour_info{$tm}{$h}{'INSERT'}{count}; - my $wduration = - $per_hour_info{$tm}{$h}{'UPDATE'}{duration} + - $per_hour_info{$tm}{$h}{'DELETE'}{duration} + - $per_hour_info{$tm}{$h}{'INSERT'}{duration}; - $d1 .= "[$t, " . ($per_hour_info{$tm}{$h}{'DELETE'}{count} || 0) . "],"; - $d2 .= "[$t, " . ($per_hour_info{$tm}{$h}{'INSERT'}{count} || 0) . "],"; - $d3 .= "[$t, " . ($per_hour_info{$tm}{$h}{'UPDATE'}{count} || 0) . "],"; - $d4 .= "[$t, " . sprintf("%.2f", (($wduration || 0) / ($wcount || 1)) / 1000) . "],"; + if (!$disable_query && (scalar keys %a_dataavg > 0)) { + foreach my $action (@SQL_ACTION) { + next if ($select_only && ($action ne 'SELECT')); + + # Average queries per minute + $graph_data{"$action"} .= "[$t, " . int(($a_dataavg{$action}{count}{"$rd"} || 0) / (60 * $avg_minutes)) . "],"; + if ($action eq 'SELECT') { + # Maxi queries per minute + $graph_data{"$action-max"} .= "[$t, " . ($a_dataavg{$action}{max}{"$rd"} || 0) . "],"; + # Mini queries per minute + $graph_data{"$action-min"} .= "[$t, " . ($a_dataavg{$action}{min}{"$rd"} || 0) . "],"; + # Average query duration + $graph_data{"$action-2"} .= "[$t, " . sprintf("%.3f", ($a_dataavg{$action}{duration}{"$rd"} || 0) / ($a_dataavg{$action}{count}{"$rd"} || 1)) . "]," if ($action eq 'SELECT'); + } else { + # Average query duration + $graph_data{"write"} .= "[$t, " . sprintf("%.3f", ($a_dataavg{write}{duration}{"$rd"} || 0) / ($a_dataavg{write}{count}{"$rd"} || 1)) . "],"; + } } } - $d1 =~ s/,$//; - $d2 =~ s/,$//; - $d3 =~ s/,$//; - $d4 =~ s/,$//; - &flotr2_graph( - 5, 'writequeries_graph', $d1, $d2, $d3, 'Write queries', - 'Queries', 'DELETE queries', 'INSERT queries', 'UPDATE queries', 'Duration', $d4, 'Average duration (s)' - ); - $d1 = ''; - $d2 = ''; - $d3 = ''; - $d4 = ''; + if (!$disable_query && (scalar keys %p_dataavg> 0)) { + $graph_data{prepare} .= "[$t, " . ($p_dataavg{prepare}{"$rd"} || 0) . "],"; + $graph_data{execute} .= "[$t, " . ($p_dataavg{execute}{"$rd"} || 0) . "],"; + $graph_data{ratio_bind_prepare} .= "[$t, " . sprintf("%.2f", ($p_dataavg{execute}{"$rd"} || 0) / ($p_dataavg{prepare}{"$rd"} || 1)) . "],"; + } } } } + foreach (keys %graph_data) { + $graph_data{$_} =~ s/,$//; + } } + $drawn_graphs{'queriespersecond_graph'} = &flotr2_graph( $graphid++, 'queriespersecond_graph', $graph_data{'query-max'}, + $graph_data{query}, $graph_data{'query-min'}, 'Queries per second (' . $avg_minutes . ' minutes average)', + 'Queries per second', 'Maximum', 'Average', 'Minimum' + ); - if (!$disable_hourly && (scalar keys %per_hour_info > 0)) { - if ($tempfile_info{count} || exists $checkpoint_info{chronos} || exists $restartpoint_info{chronos} || exists $autovacuum_info{chronos} ) { - print $fh qq{}; - } - if ($tempfile_info{count}) { - print $fh qq{}; + $drawn_graphs{'connectionspersecond_graph'} = &flotr2_graph( $graphid++, 'connectionspersecond_graph', $graph_data{conn_max}, + $graph_data{conn_avg}, $graph_data{conn_min}, 'Connections per second (' . $avg_minutes . ' minutes average)', + 'Connections per second', 'Maximum', 'Average', 'Minimum' + ); + + $drawn_graphs{'sessionspersecond_graph'} = &flotr2_graph( $graphid++, 'sessionspersecond_graph', $graph_data{sess_max}, + $graph_data{sess_avg}, $graph_data{sess_min}, 'Number of sessions (' . $avg_minutes . ' minutes average)', + 'Sessions', 'Maximum', 'Average', 'Minimum' + ); + + $drawn_graphs{'selectqueries_graph'} = &flotr2_graph( $graphid++, 'selectqueries_graph', $graph_data{"SELECT-max"}, + $graph_data{"SELECT"}, $graph_data{"SELECT-min"}, + 'SELECT queries (' . $avg_minutes . ' minutes period)', + 'Queries per second', 'Maximum', 'Average', 'Minimum' + ); + + $drawn_graphs{'writequeries_graph'} = &flotr2_graph( + $graphid++, 'writequeries_graph', $graph_data{"DELETE"}, $graph_data{"INSERT"}, $graph_data{"UPDATE"}, 'Write queries (' . $avg_minutes . ' minutes period)', + 'Queries', 'DELETE queries', 'INSERT queries', 'UPDATE queries' + ); + + if (!$select_only) { + $drawn_graphs{'durationqueries_graph'} = &flotr2_graph( + $graphid++, 'durationqueries_graph', $graph_data{query4}, $graph_data{"SELECT-2"}, $graph_data{write}, 'Average queries duration (' . $avg_minutes . ' minutes average)', + 'Duration', 'All queries', 'Select queries', 'Write queries' + ); + } else { + $drawn_graphs{'durationqueries_graph'} = &flotr2_graph( + $graphid++, 'durationqueries_graph', $graph_data{query4}, '', '', 'Average queries duration (' . $avg_minutes . ' minutes average)', + 'Duration', 'Select queries' + ); + } + + $drawn_graphs{'bindpreparequeries_graph'} = &flotr2_graph( + $graphid++, 'bindpreparequeries_graph', $graph_data{prepare}, $graph_data{"execute"}, $graph_data{ratio_bind_prepare}, 'Bind versus prepare statements (' . $avg_minutes . ' minutes average)', + 'Number of statements', 'Prepare/Parse', 'Execute/Bind', 'Bind vs prepare' + ); + +} + +sub print_established_connection +{ + + my $connection_peak = 0; + my $connection_peak_date = ''; + foreach (sort {$overall_stat{'peak'}{$b}{connection} <=> $overall_stat{'peak'}{$a}{connection}} keys %{$overall_stat{'peak'}}) { + $connection_peak = &comma_numbers($overall_stat{'peak'}{$_}{connection}); + $connection_peak_date = $_; + last; + } + + print $fh qq{ +
    +

    Established Connections

    +
    +

    Key values

    +
    +
      +
    • $connection_peak connections Connection Peak
      • +
    • $connection_peak_date Date
      • +
    +
    +
    +
    +

    Connections per second ($avg_minutes minutes average)

    +$drawn_graphs{connectionspersecond_graph} +
    +
    +}; + delete $drawn_graphs{connectionspersecond_graph}; + +} + +sub print_user_connection +{ + + my %infos = (); + my $total_count = 0; + my $c = 0; + my $conn_user_info = ''; + my @main_user = ('unknown',0); + foreach my $u (sort keys %{$connection_info{user}}) { + $conn_user_info .= ""; + $total_count += $connection_info{user}{$u}; + if ($main_user[1] < $connection_info{user}{$u}) { + $main_user[0] = $u; + $main_user[1] = $connection_info{user}{$u}; } - if ($checkpoint_info{wbuffer}) { - if (exists $checkpoint_info{chronos}) { - print $fh qq{}; + } + if ($graph) { + my @small = (); + foreach my $d (sort keys %{$connection_info{user}}) { + if ((($connection_info{user}{$d} * 100) / ($total_count||1)) > $pie_percentage_limit) { + $infos{$d} = $connection_info{user}{$d} || 0; + } else { + $infos{"Sum connections < $pie_percentage_limit%"} += $connection_info{user}{$d} || 0; + push(@small, $d); } } - if (exists $checkpoint_info{warning}) { - print $fh qq{}; + if ($#small == 0) { + $infos{$small[0]} = $infos{"Sum connections < $pie_percentage_limit%"}; + delete $infos{"Sum connections < $pie_percentage_limit%"}; + } + } + $drawn_graphs{userconnections_graph} = &flotr2_piegraph($graphid++, 'userconnections_graph', 'Connections per user', %infos); + $total_count = &comma_numbers($total_count); + print $fh qq{ +
    +

    Connections per user

    +
    +

    Key values

    +
    +
      +
    • $main_user[0] Main User
      • +
    • $total_count connections Total
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{userconnections_graph} +
    +
    +
    DayHourTemporary files
    $u" . + &comma_numbers($connection_info{user}{$u}) . "
    CheckpointsCheckpoint warning
    + + + + + + + + $conn_user_info + +
    UserCount
    + + + + + +}; + delete $drawn_graphs{userconnections_graph}; +} + +sub print_host_connection +{ + my %infos = (); + my $total_count = 0; + my $c = 0; + my $conn_host_info = ''; + my @main_host = ('unknown',0); + foreach my $h (sort keys %{$connection_info{host}}) { + $conn_host_info .= "$h" . + &comma_numbers($connection_info{host}{$h}) . ""; + $total_count += $connection_info{host}{$h}; + if ($main_host[1] < $connection_info{host}{$h}) { + $main_host[0] = $h; + $main_host[1] = $connection_info{host}{$h}; } - if ($restartpoint_info{wbuffer}) { - if (exists $restartpoint_info{chronos}) { - print $fh qq{Restartpoints}; + } + if ($graph) { + my @small = (); + foreach my $d (sort keys %{$connection_info{host}}) { + if ((($connection_info{host}{$d} * 100) / ($total_count||1)) > $pie_percentage_limit) { + $infos{$d} = $connection_info{host}{$d} || 0; + } else { + $infos{"Sum connections < $pie_percentage_limit%"} += $connection_info{host}{$d} || 0; + push(@small, $d); } } - if (exists $autovacuum_info{chronos}) { - print $fh " Autovacuum\n"; + if ($#small == 0) { + $infos{$small[0]} = $infos{"Sum connections < $pie_percentage_limit%"}; + delete $infos{"Sum connections < $pie_percentage_limit%"}; + } + } + $drawn_graphs{hostconnections_graph} = &flotr2_piegraph($graphid++, 'hostconnections_graph', 'Connections per host', %infos); + $total_count = &comma_numbers($total_count); + print $fh qq{ +
    +

    Connections per host

    +
    +

    Key values

    +
    +
      +
    • $main_host[0] Main Host
      • +
    • $total_count connections Total
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{hostconnections_graph} +
    +
    + + + + + + + + + $conn_host_info + +
    HostCount
    +
    +
    +
    +
    +
    +}; + + delete $drawn_graphs{hostconnections_graph}; +} + +sub print_database_connection +{ + my %infos = (); + my $total_count = 0; + my $conn_database_info = ''; + my @main_database = ('unknown',0); + foreach my $d (sort keys %{$connection_info{database}}) { + $conn_database_info .= "$d " . + &comma_numbers($connection_info{database}{$d}) . ""; + $total_count += $connection_info{database}{$d}; + if ($main_database[1] < $connection_info{database}{$d}) { + $main_database[0] = $d; + $main_database[1] = $connection_info{database}{$d}; + } + foreach my $u (sort keys %{$connection_info{user}}) { + next if (!exists $connection_info{database_user}{$d}{$u}); + $conn_database_info .= " $u" . + &comma_numbers($connection_info{database_user}{$d}{$u}) . ""; } - if ($tempfile_info{count} || exists $checkpoint_info{chronos} || exists $restartpoint_info{chronos}) { - print $fh qq{}; + } + if ($graph) { + my @small = (); + foreach my $d (sort keys %{$connection_info{database}}) { + if ((($connection_info{database}{$d} * 100) / ($total_count||1)) > $pie_percentage_limit) { + $infos{$d} = $connection_info{database}{$d} || 0; + } else { + $infos{"Sum connections < $pie_percentage_limit%"} += $connection_info{database}{$d} || 0; + push(@small, $d); + } } - if ($tempfile_info{count}) { - print $fh qq{CountAvg size}; + if ($#small == 0) { + $infos{$small[0]} = $infos{"Sum connections < $pie_percentage_limit%"}; + delete $infos{"Sum connections < $pie_percentage_limit%"}; } - if ($checkpoint_info{wbuffer}) { - print $fh - qq{Written buffersAddedRemovedRecycledWrite time (sec)Sync time (sec)Total time (sec)}; + } + $drawn_graphs{databaseconnections_graph} = &flotr2_piegraph($graphid++, 'databaseconnections_graph', 'Connections per database', %infos); + $total_count = &comma_numbers($total_count); + print $fh qq{ +
    +

    Connections per database

    +
    +

    Key values

    +
    +
      +
    • $main_database[0] Main Database
      • +
    • $total_count connections Total
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{databaseconnections_graph} +
    +
    + + + + + + + + + + $conn_database_info + +
    DatabaseUserCount
    +
    +
    +
    +
    +
    +}; + delete $drawn_graphs{databaseconnections_graph}; +} + +sub print_simultaneous_session +{ + + my $session_peak = 0; + my $session_peak_date = ''; + foreach (sort {$overall_stat{'peak'}{$b}{session} <=> $overall_stat{'peak'}{$a}{session}} keys %{$overall_stat{'peak'}}) { + $session_peak = &comma_numbers($overall_stat{'peak'}{$_}{session}); + $session_peak_date = $_; + last; + } + + print $fh qq{ +
    +

    Simultaneous sessions

    +
    +

    Key values

    +
    +
      +
    • $session_peak sessions Session Peak
      • +
    • $session_peak_date Date
      • +
    +
    +
    +
    +

    Number of sessions ($avg_minutes minutes average)

    +$drawn_graphs{sessionspersecond_graph} +
    +
    +}; + delete $drawn_graphs{sessionspersecond_graph}; + +} + + +sub print_user_session +{ + my %infos = (); + my $total_count = 0; + my $c = 0; + my $sess_user_info = ''; + my @main_user = ('unknown',0); + foreach my $u (sort keys %{$session_info{user}}) { + $sess_user_info .= "$u" . &comma_numbers($session_info{user}{$u}{count}) . + "" . &convert_time($session_info{user}{$u}{duration}), "" . + &convert_time($session_info{user}{$u}{duration} / $session_info{user}{$u}{count}) . + ""; + $total_count += $session_info{user}{$u}{count}; + if ($main_user[1] < $session_info{user}{$u}{count}) { + $main_user[0] = $u; + $main_user[1] = $session_info{user}{$u}{count}; } - if (exists $checkpoint_info{warning}) { - print $fh qq{CountAvg time (sec)}; + } + if ($graph) { + my @small = (); + foreach my $d (sort keys %{$session_info{user}}) { + if ((($session_info{user}{$d}{count} * 100) / ($total_count||1)) > $pie_percentage_limit) { + $infos{$d} = $session_info{user}{$d}{count} || 0; + } else { + $infos{"Sum sessions < $pie_percentage_limit%"} += $session_info{user}{$d}{count} || 0; + push(@small, $d); + } } - if ($restartpoint_info{wbuffer}) { - print $fh - qq{Written buffersWrite time (sec)Sync time (sec)Total time (sec)}; + if ($#small == 0) { + $infos{$small[0]} = $infos{"Sum sessions < $pie_percentage_limit%"}; + delete $infos{"Sum sessions < $pie_percentage_limit%"}; } - if (exists $autovacuum_info{chronos}) { - print $fh " VACUUMsANALYZEs\n"; + } + $drawn_graphs{usersessions_graph} = &flotr2_piegraph($graphid++, 'usersessions_graph', 'Connections per user', %infos); + $sess_user_info = qq{$NODATA} if (!$total_count); + $total_count = &comma_numbers($total_count); + print $fh qq{ +
    +

    Sessions per user

    +
    +

    Key values

    +
    +
      +
    • $main_user[0] Main User
      • +
    • $total_count sessions Total
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{usersessions_graph} +
    +
    + + + + + + + + + + + $sess_user_info + +
    UserCountTotal DurationAverage Duration
    +
    +
    +
    +
    +
    +}; + delete $drawn_graphs{usersessions_graph}; +} + +sub print_host_session +{ + my %infos = (); + my $total_count = 0; + my $c = 0; + my $sess_host_info = ''; + my @main_host = ('unknown',0); + foreach my $h (sort keys %{$session_info{host}}) { + $sess_host_info .= "$h" . &comma_numbers($session_info{host}{$h}{count}) . + "" . &convert_time($session_info{host}{$h}{duration}) . "" . + &convert_time($session_info{host}{$h}{duration} / $session_info{host}{$h}{count}) . + ""; + $total_count += $session_info{host}{$h}{count}; + if ($main_host[1] < $session_info{host}{$h}{count}) { + $main_host[0] = $h; + $main_host[1] = $session_info{host}{$h}{count}; } - if ($tempfile_info{count} || exists $checkpoint_info{chronos} || exists $restartpoint_info{chronos}) { - print $fh qq{}; - foreach my $d (sort {$a <=> $b} keys %per_hour_info) { - my $c = 1; - $d =~ /^\d{4}(\d{2})(\d{2})$/; - my $zday = "$abbr_month{$1} $2"; - foreach my $h (sort {$a <=> $b} keys %{$per_hour_info{$d}}) { - my $colb = $c % 2; - $zday = " " if ($c > 1); - print $fh "$zday$h"; - if ($tempfile_info{count}) { - my $temp_average = '0'; - if ($tempfile_info{chronos}{$d}{$h}{count}) { - $temp_average = &comma_numbers( - sprintf("%.2f", $tempfile_info{chronos}{$d}{$h}{size} / $tempfile_info{chronos}{$d}{$h}{count})); - } - print $fh "", &comma_numbers($tempfile_info{chronos}{$d}{$h}{count} || 0), - "$temp_average"; - } - if (exists $checkpoint_info{chronos} && $checkpoint_info{wbuffer}) { - if (exists $checkpoint_info{chronos}{$d}{$h}) { - print $fh "", &comma_numbers($checkpoint_info{chronos}{$d}{$h}{wbuffer}) || 0, - "", &comma_numbers($checkpoint_info{chronos}{$d}{$h}{file_added}) || 0, - "", &comma_numbers($checkpoint_info{chronos}{$d}{$h}{file_removed}) || 0, - "", - &comma_numbers($checkpoint_info{chronos}{$d}{$h}{file_recycled}) || 0, - "", &comma_numbers($checkpoint_info{chronos}{$d}{$h}{write}) || 0, - "", &comma_numbers($checkpoint_info{chronos}{$d}{$h}{sync}) || 0, - "", &comma_numbers($checkpoint_info{chronos}{$d}{$h}{total}) || 0, - ""; - } else { - print $fh -"0000000"; - } - } - if (exists $checkpoint_info{chronos} && $checkpoint_info{warning}) { - if (exists $checkpoint_info{chronos}{$d}{$h}{warning}) { - print $fh "", &comma_numbers($checkpoint_info{chronos}{$d}{$h}{warning}) || 0, - "", - &comma_numbers( - sprintf( - "%.2f", - ($checkpoint_info{chronos}{$d}{$h}{warning_seconds} || 0) / - ($checkpoint_info{chronos}{$d}{$h}{warning} || 1) - ) - ) || 0, ""; - } else { - print $fh "00"; - } - } - if (exists $restartpoint_info{chronos} && $restartpoint_info{wbuffer}) { - if (exists $restartpoint_info{chronos}{$d}{$h}) { - print $fh "", &comma_numbers($restartpoint_info{chronos}{$d}{$h}{wbuffer}) || 0, - "", &comma_numbers($restartpoint_info{chronos}{$d}{$h}{write}) || 0, - "", &comma_numbers($restartpoint_info{chronos}{$d}{$h}{sync}) || 0, - "", &comma_numbers($restartpoint_info{chronos}{$d}{$h}{total}) || 0, - ""; - } else { - print $fh -"0000"; - } - } - if (exists $autovacuum_info{chronos}) { - print $fh "", &comma_numbers($autovacuum_info{chronos}{"$d"}{"$h"}{count} || 0), "", - "", &comma_numbers($autoanalyze_info{chronos}{"$d"}{"$h"}{count} || 0), ""; - } - print $fh "\n"; - $c++; - } + } + if ($graph) { + my @small = (); + foreach my $d (sort keys %{$session_info{host}}) { + if ((($session_info{host}{$d}{count} * 100) / ($total_count||1)) > $pie_percentage_limit) { + $infos{$d} = $session_info{host}{$d}{count} || 0; + } else { + $infos{"Sum sessions < $pie_percentage_limit%"} += $session_info{host}{$d}{count} || 0; + push(@small, $d); } - print $fh "\n"; + } + if ($#small == 0) { + $infos{$small[0]} = $infos{"Sum sessions < $pie_percentage_limit%"}; + delete $infos{"Sum sessions < $pie_percentage_limit%"}; + } + } + $drawn_graphs{hostsessions_graph} = &flotr2_piegraph($graphid++, 'hostsessions_graph', 'Connections per host', %infos); + $sess_host_info = qq{$NODATA} if (!$total_count); + $total_count = &comma_numbers($total_count); + print $fh qq{ +
    +

    Sessions per host

    +
    +

    Key values

    +
    +
      +
    • $main_host[0] Main Host
      • +
    • $total_count sessions Total
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{hostsessions_graph} +
    +
    + + + + + + + + + + + $sess_host_info + +
    HostCountTotal DurationAverage Duration
    +
    +
    +
    +
    +
    +}; + + delete $drawn_graphs{hostsessions_graph}; +} + +sub print_database_session +{ + my %infos = (); + my $total_count = 0; + my $sess_database_info = ''; + my @main_database = ('unknown',0); + foreach my $d (sort keys %{$session_info{database}}) { + $sess_database_info .= "$d" . &comma_numbers($session_info{database}{$d}{count}) . + "" . &convert_time($session_info{database}{$d}{duration}) . "" . + &convert_time($session_info{database}{$d}{duration} / $session_info{database}{$d}{count}) . + ""; + $total_count += $session_info{database}{$d}{count}; + if ($main_database[1] < $session_info{database}{$d}{count}) { + $main_database[0] = $d; + $main_database[1] = $session_info{database}{$d}{count}; } } - if (!$disable_hourly && $graph) { - # checkpoint size - if (exists $checkpoint_info{chronos} && $checkpoint_info{wbuffer}) { - foreach my $tm (sort {$a <=> $b} keys %{$checkpoint_info{chronos}}) { - $tm =~ /(\d{4})(\d{2})(\d{2})/; - my $y = $1 - 1900; - my $mo = $2 - 1; - my $d = $3; - foreach my $h ("00" .. "23") { - my $t = timegm_nocheck(0, 0, $h, $d, $mo, $y) * 1000; - next if ($t < $t_min_hour); - last if ($t > $t_max_hour); - $d1 .= "[$t, " . ($checkpoint_info{chronos}{$tm}{$h}{wbuffer} || 0) . "],"; - } - } - $d1 =~ s/,$//; - &flotr2_graph( - 6, 'checkpointwritebuffers_graph', $d1, '', '', 'Checkpoint write buffers', - 'Buffers', 'Write buffers', '', '' - ); - $d1 = ''; - - foreach my $tm (sort {$a <=> $b} keys %{$checkpoint_info{chronos}}) { - $tm =~ /(\d{4})(\d{2})(\d{2})/; - my $y = $1 - 1900; - my $mo = $2 - 1; - my $d = $3; - foreach my $h ("00" .. "23") { - my $t = timegm_nocheck(0, 0, $h, $d, $mo, $y) * 1000; - next if ($t < $t_min_hour); - last if ($t > $t_max_hour); - $d1 .= "[$t, " . ($checkpoint_info{chronos}{$tm}{$h}{file_added} || 0) . "],"; - $d2 .= "[$t, " . ($checkpoint_info{chronos}{$tm}{$h}{file_removed} || 0) . "],"; - $d3 .= "[$t, " . ($checkpoint_info{chronos}{$tm}{$h}{file_recycled} || 0) . "],"; - } - } - $d1 =~ s/,$//; - $d2 =~ s/,$//; - $d3 =~ s/,$//; - &flotr2_graph( - 7, 'checkpointfiles_graph', $d1, $d2, $d3, 'Checkpoint Wal files usage', - 'Number of files', 'Added', 'Removed', 'Recycled' - ); - $d1 = ''; - $d2 = ''; - $d3 = ''; - } - - # restartpoint size - if (exists $restartpoint_info{chronos} && $restartpoint_info{wbuffer}) { - foreach my $tm (sort {$a <=> $b} keys %{$restartpoint_info{chronos}}) { - $tm =~ /(\d{4})(\d{2})(\d{2})/; - my $y = $1 - 1900; - my $mo = $2 - 1; - my $d = $3; - foreach my $h ("00" .. "23") { - my $t = timegm_nocheck(0, 0, $h, $d, $mo, $y) * 1000; - next if ($t < $t_min_hour); - last if ($t > $t_max_hour); - $d1 .= "[$t, " . ($restartpoint_info{chronos}{$tm}{$h}{wbuffer} || 0) . "],"; - } - } - $d1 =~ s/,$//; - &flotr2_graph( - 6, 'restartpointwritebuffers_graph', $d1, '', '', 'Restartpoint write buffers', - 'Buffers', 'Write buffers', '', '' - ); - $d1 = ''; - } - - # Temporary file size - if (exists $tempfile_info{chronos}) { - foreach my $tm (sort {$a <=> $b} keys %{$tempfile_info{chronos}}) { - $tm =~ /(\d{4})(\d{2})(\d{2})/; - my $y = $1 - 1900; - my $mo = $2 - 1; - my $d = $3; - foreach my $h ("00" .. "23") { - my $t = timegm_nocheck(0, 0, $h, $d, $mo, $y) * 1000; - next if ($t < $t_min_hour); - last if ($t > $t_max_hour); - $d1 .= "[$t, " . ($tempfile_info{chronos}{$tm}{$h}{size} || 0) . "],"; - $d2 .= "[$t, " . ($tempfile_info{chronos}{$tm}{$h}{count} || 0) . "],"; - } - } - $d1 =~ s/,$//; - $d2 =~ s/,$//; - &flotr2_graph( - 8, 'temporaryfile_graph', $d1, '', '', 'Temporary files', - 'Size of files', 'Size of files', '', '', 'Number of files', $d2, 'Number of files' - ); - $d1 = ''; - $d2 = ''; - } - - # VACUUMs and ANALYZEs - if (exists $autovacuum_info{chronos}) { - foreach my $tm (sort {$a <=> $b} keys %{$autovacuum_info{chronos}}) { - $tm =~ /(\d{4})(\d{2})(\d{2})/; - my $y = $1 - 1900; - my $mo = $2 - 1; - my $d = $3; - foreach my $h ("00" .. "23") { - my $t = timegm_nocheck(0, 0, $h, $d, $mo, $y) * 1000; - next if ($t < $t_min_hour); - last if ($t > $t_max_hour); - $d1 .= "[$t, " . ($autovacuum_info{chronos}{$tm}{$h}{count} || 0) . "],"; - $d2 .= "[$t, " . ($autoanalyze_info{chronos}{$tm}{$h}{count} || 0) . "],"; - } - } - $d1 =~ s/,$//; - $d2 =~ s/,$//; - &flotr2_graph( - 9, 'autovacuum_graph', $d1, $d2, '', 'Autovacuum actions', - '', 'VACUUMs', 'ANALYZEs' - ); - $d1 = ''; - $d2 = ''; + if ($graph) { + my @small = (); + foreach my $d (sort keys %{$session_info{database}}) { + if ((($session_info{database}{$d}{count} * 100) / ($total_count||1)) > $pie_percentage_limit) { + $infos{$d} = $session_info{database}{$d}{count} || 0; + } else { + $infos{"Sum sessions < $pie_percentage_limit%"} += $session_info{database}{$d}{count} || 0; + push(@small, $d); + } + } + if ($#small == 0) { + $infos{$small[0]} = $infos{"Sum sessions < $pie_percentage_limit%"}; + delete $infos{"Sum sessions < $pie_percentage_limit%"}; } } + $drawn_graphs{databasesessions_graph} = &flotr2_piegraph($graphid++, 'databasesessions_graph', 'Connections per database', %infos); + $sess_database_info = qq{$NODATA} if (!$total_count); + + $total_count = &comma_numbers($total_count); + print $fh qq{ +
    +

    Sessions per database

    +
    +

    Key values

    +
    +
      +
    • $main_database[0] Main Database
      • +
    • $total_count sessions Total
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{databasesessions_graph} +
    +
    + + + + + + + + + + + + $sess_database_info + +
    DatabaseUserCountTotal DurationAverage Duration
    +
    +
    +
    +
    +
    +}; + delete $drawn_graphs{databasesessions_graph}; +} + +sub print_checkpoint +{ + # checkpoint + my %graph_data = (); if ($graph) { - # VACUUM stats per table - if ($autovacuum_info{count} > 0) { - print $fh qq{ -

    VACUUMs by table ^

    - -
    - - - - - - - - - }; - my $total_count = 0; - my $total_idxscan = 0; - my $total_tuples = 0; - my $total_pages = 0; - foreach my $t (sort keys %{$autovacuum_info{tables}}) { - print $fh "\n"; - $total_count += $autovacuum_info{tables}{$t}{vacuums}; - $total_idxscan += $autovacuum_info{tables}{$t}{idxscans}; - $total_tuples += $autovacuum_info{tables}{$t}{tuples}{removed}; - $total_pages += $autovacuum_info{tables}{$t}{pages}{removed}; - } - print $fh "\n"; - print $fh "
    TableVACUUMsIndex scansTuples removedPages removed
    ", $t, - "", $autovacuum_info{tables}{$t}{vacuums}, - "", $autovacuum_info{tables}{$t}{idxscans}, - "", $autovacuum_info{tables}{$t}{tuples}{removed}, - "", $autovacuum_info{tables}{$t}{pages}{removed}, - "
    Total", $total_count, - "", $total_idxscan, - "", $total_tuples, - "", $total_pages, "
    		\n"; - if ($graph && $total_count) { - my %data = (); - foreach my $t (sort keys %{$autovacuum_info{tables}}) { - if ((($autovacuum_info{tables}{$t}{vacuums} * 100) / $total_count) > $pie_percentage_limit) { - $data{$t} = $autovacuum_info{tables}{$t}{vacuums} || 0; - } else { - $data{"Others"} += $autovacuum_info{tables}{$t}{vacuums} || 0; + foreach my $tm (sort {$a <=> $b} keys %per_minute_info) { + $tm =~ /(\d{4})(\d{2})(\d{2})/; + my $y = $1 - 1900; + my $mo = $2 - 1; + my $d = $3; + foreach my $h ("00" .. "23") { + next if (!exists $per_minute_info{$tm}{$h}); + my %chk_dataavg = (); + my %t_dataavg = (); + my %v_dataavg = (); + foreach my $m ("00" .. "59") { + next if (!exists $per_minute_info{$tm}{$h}{$m}); + + my $rd = &average_per_minutes($m, $avg_minutes); + + if ($checkpoint_info{wbuffer}) { + if (exists $per_minute_info{$tm}{$h}{$m}{checkpoint}) { + $chk_dataavg{wbuffer}{"$rd"} += ($per_minute_info{$tm}{$h}{$m}{checkpoint}{wbuffer} || 0); + $chk_dataavg{file_added}{"$rd"} += ($per_minute_info{$tm}{$h}{$m}{checkpoint}{file_added} || 0); + $chk_dataavg{file_removed}{"$rd"} += ($per_minute_info{$tm}{$h}{$m}{checkpoint}{file_removed} || 0); + $chk_dataavg{file_recycled}{"$rd"} += ($per_minute_info{$tm}{$h}{$m}{checkpoint}{file_recycled} || 0); + } } } - &flotr2_piegraph(18, 'autovacuumbytable_graph', 'Autovacuum per table', %data); - %data = (); - if ($total_tuples) { - print $fh "
    \n"; - foreach my $t (sort keys %{$autovacuum_info{tables}}) { - if ((($autovacuum_info{tables}{$t}{tuples}{removed} * 100) / $total_tuples) > $pie_percentage_limit) { - $data{$t} = $autovacuum_info{tables}{$t}{tuples}{removed} || 0; - } else { - $data{"Others"} += $autovacuum_info{tables}{$t}{tuples}{removed} || 0; - } + + foreach my $rd (@avgs) { + my $t = timegm_nocheck(0, $rd, $h, $d, $mo, $y) * 1000; + + next if ($t < $t_min); + last if ($t > $t_max); + + # Average of written checkpoints buffers and wal files + if (exists $chk_dataavg{wbuffer}) { + $graph_data{wbuffer} .= "[$t, " . ($chk_dataavg{wbuffer}{"$rd"} || 0) . "],"; + $graph_data{file_added} .= "[$t, " . ($chk_dataavg{file_added}{"$rd"} || 0) . "],"; + $graph_data{file_removed} .= "[$t, " . ($chk_dataavg{file_removed}{"$rd"} || 0) . "],"; + $graph_data{file_recycled} .= "[$t, " . ($chk_dataavg{file_recycled}{"$rd"} || 0) . "],"; } - &flotr2_piegraph(19, 'autovacuumtuplesremoved_graph', 'Autovacuum tuples removed per table', %data); } } - print $fh "
    \n"; } - - # ANALYZE stats per table - if ($autoanalyze_info{count} > 0) { - print $fh qq{ -

    ANALYZEs by table ^

    - -
    - - - - - - }; - my $total_count = 0; - my $total_idxscan = 0; - foreach my $t (sort keys %{$autoanalyze_info{tables}}) { - print $fh "\n"; - $total_count += $autoanalyze_info{tables}{$t}{analyzes}; - } - print $fh "\n"; - print $fh "
    TableANALYZEs
    ", $t, - "", $autoanalyze_info{tables}{$t}{analyzes}, - "
    Total", $total_count, - "
    \n"; + foreach (keys %graph_data) { + $graph_data{$_} =~ s/,$//; } } + # Checkpoints buffers and files + $drawn_graphs{checkpointwritebuffers_graph} = + &flotr2_graph($graphid++, 'checkpointwritebuffers_graph', $graph_data{wbuffer}, '', '', + 'Checkpoint write buffers (' . $avg_minutes . ' minutes period)', + 'Buffers', 'Write buffers', '', '' + ); + $drawn_graphs{checkpointfiles_graph} = + &flotr2_graph($graphid++, 'checkpointfiles_graph', $graph_data{file_added}, + $graph_data{file_removed}, $graph_data{file_recycled}, 'Checkpoint Wal files usage', + 'Number of files', 'Added', 'Removed', 'Recycled' + ); - # INSERT/DELETE/UPDATE/SELECT repartition - $overall_stat{'SELECT'} ||= 0; - $overall_stat{'INSERT'} ||= 0; - $overall_stat{'UPDATE'} ||= 0; - $overall_stat{'DELETE'} ||= 0; - my $totala = $overall_stat{'SELECT'} + $overall_stat{'INSERT'} + $overall_stat{'UPDATE'} + $overall_stat{'DELETE'}; - if (!$disable_type && $totala) { - print $fh qq{ -

    Queries by type ^

    - -"; + $files .= ""; + $warnings .= ""; + $zday = ''; + my %cinf = (); + my %rinf = (); + my %cainf = (); + my %rainf = (); + foreach my $m (keys %{$per_minute_info{$d}{$h}}) { + + if (exists $per_minute_info{$d}{$h}{$m}{checkpoint}) { + $cinf{wbuffer} += $per_minute_info{$d}{$h}{$m}{checkpoint}{wbuffer}; + $cinf{file_added} += $per_minute_info{$d}{$h}{$m}{checkpoint}{file_added}; + $cinf{file_removed} += $per_minute_info{$d}{$h}{$m}{checkpoint}{file_removed}; + $cinf{file_recycled} += $per_minute_info{$d}{$h}{$m}{checkpoint}{file_recycled}; + $cinf{write} += $per_minute_info{$d}{$h}{$m}{checkpoint}{write}; + $cinf{sync} += $per_minute_info{$d}{$h}{$m}{checkpoint}{sync}; + $cinf{total} += $per_minute_info{$d}{$h}{$m}{checkpoint}{total}; + $cainf{sync_files} += $per_minute_info{$d}{$h}{$m}{checkpoint}{sync_files}; + $cainf{sync_avg} += $per_minute_info{$d}{$h}{$m}{checkpoint}{sync_avg}; + $cainf{sync_longest} = $per_minute_info{$d}{$h}{$m}{checkpoint}{sync_longest} + if ($per_minute_info{$d}{$h}{$m}{checkpoint}{sync_longest} > $cainf{sync_longest}); + } + if (exists $per_minute_info{$d}{$h}{$m}{checkpoint}{warning}) { + $cinf{warning} += $per_minute_info{$d}{$h}{$m}{checkpoint}{warning}; + $cinf{warning_seconds} += $per_minute_info{$d}{$h}{$m}{checkpoint}{warning_seconds}; + } + } + if (scalar keys %cinf) { + $buffers .= ""; + $files .= ""; + } else { + $buffers .= ""; + $files .= ""; } - if (((($total - $totala) * 100) / $total) > $pie_percentage_limit) { - $data{'Others'} = $total - $totala; + if (exists $cinf{warning}) { + $warnings .= ""; } else { - $data{"Sum types < $pie_percentage_limit%"} += $total - $totala; + $warnings .= ""; } - &flotr2_piegraph(22, 'queriesbytype_graph', 'Type of queries', %data); } - print $fh "
    - - - - - - - -}; - my $total = $overall_stat{'queries_number'} || 1; + my $checkpoint_wbuffer_peak = 0; + my $checkpoint_wbuffer_peak_date = ''; + foreach (sort {$overall_checkpoint{'peak'}{$b}{checkpoint_wbuffer} <=> $overall_checkpoint{'peak'}{$a}{checkpoint_wbuffer}} keys %{$overall_checkpoint{'peak'}}) { + $checkpoint_wbuffer_peak = &comma_numbers($overall_checkpoint{'peak'}{$_}{checkpoint_wbuffer}); + $checkpoint_wbuffer_peak_date = $_; + last; + } + my $walfile_usage_peak = 0; + my $walfile_usage_peak_date = ''; + foreach (sort {$overall_checkpoint{'peak'}{$b}{walfile_usage} <=> $overall_checkpoint{'peak'}{$a}{walfile_usage}} keys %{$overall_checkpoint{'peak'}}) { + $walfile_usage_peak = &comma_numbers($overall_checkpoint{'peak'}{$_}{walfile_usage}); + $walfile_usage_peak_date = $_; + last; + } - print $fh "\n"; - print $fh "\n"; - print $fh "\n"; - print $fh "\n"; - print $fh "\n" - if (($total - $totala) > 0); - print $fh "
    TypeCountPercentage
    SELECT", &comma_numbers($overall_stat{'SELECT'}), - "", sprintf("%0.2f", ($overall_stat{'SELECT'} * 100) / $total), "%
    INSERT", &comma_numbers($overall_stat{'INSERT'}), - "", sprintf("%0.2f", ($overall_stat{'INSERT'} * 100) / $total), "%
    UPDATE", &comma_numbers($overall_stat{'UPDATE'}), - "", sprintf("%0.2f", ($overall_stat{'UPDATE'} * 100) / $total), "%
    DELETE", &comma_numbers($overall_stat{'DELETE'}), - "", sprintf("%0.2f", ($overall_stat{'DELETE'} * 100) / $total), "%
    OTHERS", &comma_numbers($total - $totala), - "", sprintf("%0.2f", (($total - $totala) * 100) / $total), "%
    		\n"; + print $fh qq{ +

    Checkpoints / Restartpoints

    - if ($graph && $totala) { - my %data = (); - foreach my $t ('SELECT', 'INSERT', 'UPDATE', 'DELETE') { - if ((($overall_stat{$t} * 100) / $total) > $pie_percentage_limit) { - $data{$t} = $overall_stat{$t} || 0; - } else { - $data{"Sum types < $pie_percentage_limit%"} += $overall_stat{$t} || 0; - } +
    +

    Checkpoints Buffers

    +
    +

    Key values

    +
    +
      +
    • $checkpoint_wbuffer_peak buffers Checkpoint Peak
      • +
    • $checkpoint_wbuffer_peak_date Date
      • +
    • $overall_checkpoint{checkpoint_write} seconds Highest write time
      • +
    • $overall_checkpoint{checkpoint_sync} seconds Sync time
      • +
    +
    +
    +
    +

    Checkpoint write buffers ($avg_minutes minutes average)

    +$drawn_graphs{checkpointwritebuffers_graph} +
    +
    +}; + delete $drawn_graphs{checkpointwritebuffers_graph}; + + print $fh qq{ +
    +

    Checkpoints Wal files

    +
    +

    Key values

    +
    +
      +
    • $walfile_usage_peak files Wal files usage Peak
      • +
    • $walfile_usage_peak_date Date
      • +
    +
    +
    +
    +

    Checkpoint Wal files usage

    +$drawn_graphs{checkpointfiles_graph} +
    +
    +}; + delete $drawn_graphs{checkpointfiles_graph}; + + my $buffers = ''; + my $files = ''; + my $warnings = ''; + foreach my $d (sort {$a <=> $b} keys %per_minute_info) { + $d =~ /^\d{4}(\d{2})(\d{2})$/; + my $zday = "$abbr_month{$1} $2"; + foreach my $h (sort {$a <=> $b} keys %{$per_minute_info{$d}}) { + $buffers .= "
    $zday$h
    $zday$h
    $zday$h" . &comma_numbers($cinf{wbuffer}) . + "" . &comma_numbers($cinf{write}) . + "" . &comma_numbers($cinf{sync}) . + "" . &comma_numbers($cinf{total}) . + "
    " . &comma_numbers($cinf{file_added}) . + "" . &comma_numbers($cinf{file_removed}) . + "" . &comma_numbers($cinf{file_recycled}) . + "" . &comma_numbers($cainf{sync_files}) . + "" . &comma_numbers($cainf{sync_longest}) . + "" . &comma_numbers($cainf{sync_avg}) . + "
    0000
    000000
    " . &comma_numbers($cinf{warning}) . "" . + &comma_numbers(sprintf( "%.2f", ($cinf{warning_seconds} || 0) / ($cinf{warning} || 1))) . + "
    00
    \n"; + } - # Show request per database statistics - if (scalar keys %database_info > 1) { - print $fh qq{ -

    Queries per database ^

    - -\n"; - $total_count += $database_info{$d}{count}; - foreach my $r (sort keys %{$database_info{$d}}) { - next if ($r eq 'count'); - print $fh "\n"; - } - } - print $fh "
    - - - - - - + $buffers = qq{} if (!$buffers); + $files = qq{} if (!$files); + $warnings = qq{} if (!$warnings); + + print $fh qq{ +
    +

    Checkpoints Activity

    +
    + +
    +
    +
    DatabaseRequest typeCount
    $NODATA
    $NODATA
    $NODATA
    + + + + + + + + + + + $buffers + +
    DayHourWritten buffersWrite timeSync timeTotal time
    + +
    + + + + + + + + + + + + + + $files + +
    DayHourAddedRemovedRecycledSynced filesLongest syncAverage sync
    +
    +
    + + + + + + + + + + $warnings + +
    DayHourCountAvg time (sec)
    +
    + + ↑ Back to the top of the Checkpoint Activity table + + + }; - my $total_count = 0; - foreach my $d (sort keys %database_info) { - print $fh "
    $d", - &comma_numbers($database_info{$d}{count}), "
    $r", - &comma_numbers($database_info{$d}{$r}), "
    \n"; - if ($graph && $total_count) { - my %infos = (); - my @small = (); - foreach my $d (sort keys %database_info) { - if ((($database_info{$d}{count} * 100) / $total_count) > $pie_percentage_limit) { - $infos{$d} = $database_info{$d}{count} || 0; - } else { - $infos{"Sum databases < $pie_percentage_limit%"} += $database_info{$d}{count} || 0; - push(@small, $d); - } - } - if ($#small == 0) { - $infos{$small[0]} = $infos{"Sum databases < $pie_percentage_limit%"}; - delete $infos{"Sum databases < $pie_percentage_limit%"}; - } - &flotr2_piegraph(20, 'requestsdatabases_graph', 'Queries per database', %infos); - } - print $fh "\n"; - } - # Show request per application statistics - if (scalar keys %application_info > 1) { - print $fh qq{ -

    Queries per application ^

    - -
    - - - - - - -}; - my $total_count = 0; - foreach my $d (sort keys %application_info) { - print $fh "\n"; - $total_count += $application_info{$d}{count}; - foreach my $r (sort keys %{$application_info{$d}}) { - next if ($r eq 'count'); - print $fh "\n"; - } - } - print $fh "
    DatabaseRequest typeCount
    $d", - &comma_numbers($application_info{$d}{count}), "
    $r", - &comma_numbers($application_info{$d}{$r}), "
    		\n"; - if ($graph && $total_count) { - my %infos = (); - my @small = (); - foreach my $d (sort keys %application_info) { - if ((($application_info{$d}{count} * 100) / $total_count) > $pie_percentage_limit) { - $infos{$d} = $application_info{$d}{count} || 0; - } else { - $infos{"Sum applications < $pie_percentage_limit%"} += $application_info{$d}{count} || 0; - push(@small, $d); +} + +sub print_temporary_file +{ + + # checkpoint + my %graph_data = (); + if ($graph) { + foreach my $tm (sort {$a <=> $b} keys %per_minute_info) { + $tm =~ /(\d{4})(\d{2})(\d{2})/; + my $y = $1 - 1900; + my $mo = $2 - 1; + my $d = $3; + foreach my $h ("00" .. "23") { + next if (!exists $per_minute_info{$tm}{$h}); + my %chk_dataavg = (); + my %t_dataavg = (); + my %v_dataavg = (); + foreach my $m ("00" .. "59") { + next if (!exists $per_minute_info{$tm}{$h}{$m}); + my $rd = &average_per_minutes($m, $avg_minutes); + if ($tempfile_info{count}) { + if (exists $per_minute_info{$tm}{$h}{$m}{tempfile}) { + $t_dataavg{size}{"$rd"} += ($per_minute_info{$tm}{$h}{$m}{tempfile}{size} || 0); + $t_dataavg{count}{"$rd"} += ($per_minute_info{$tm}{$h}{$m}{tempfile}{count} || 0); + } } } - if ($#small == 0) { - $infos{$small[0]} = $infos{"Sum applications < $pie_percentage_limit%"}; - delete $infos{"Sum applications < $pie_percentage_limit%"}; + + foreach my $rd (@avgs) { + my $t = timegm_nocheck(0, $rd, $h, $d, $mo, $y) * 1000; + + next if ($t < $t_min); + last if ($t > $t_max); + + if (exists $t_dataavg{size}) { + $graph_data{size} .= "[$t, " . ($t_dataavg{size}{"$rd"} || 0) . "],"; + $graph_data{count} .= "[$t, " . ($t_dataavg{count}{"$rd"} || 0) . "],"; + } } - &flotr2_piegraph(21, 'requestsapplications_graph', 'Queries per application', %infos); } - print $fh "
    \n"; + } + foreach (keys %graph_data) { + $graph_data{$_} =~ s/,$//; } } + # Temporary file size + $drawn_graphs{temporarydata_graph} = + &flotr2_graph($graphid++, 'temporarydata_graph', $graph_data{size}, '', '', + 'Size of temporary files (' . $avg_minutes . ' minutes period)', + 'Size of files', 'Size of files' + ); + # Temporary file number + $drawn_graphs{temporaryfile_graph} = + &flotr2_graph($graphid++, 'temporaryfile_graph', $graph_data{count}, '', '', + 'Number of temporary files (' . $avg_minutes . ' minutes period)', + 'Number of files', 'Number of files' + ); - # Lock stats per type - if (!$disable_lock && scalar keys %lock_info > 0) { - print $fh qq{ -

    Locks by type ^

    - -\n"; - foreach my $o (sort keys %{$lock_info{$t}}) { - next if (($o eq 'count') || ($o eq 'duration') || ($o eq 'chronos')); - print $fh "\n"; + +} + +sub print_analyze_per_table +{ + # ANALYZE stats per table + my %infos = (); + my $total_count = 0; + my $analyze_info = ''; + my @main_analyze = ('unknown',0); + foreach my $t (sort {$autoanalyze_info{tables}{$b}{analyzes} <=> $autoanalyze_info{tables}{$a}{analyzes}} keys %{$autoanalyze_info{tables}}) { + $analyze_info .= ""; + $total_count += $autoanalyze_info{tables}{$t}{analyzes}; + if ($main_analyze[1] < $autoanalyze_info{tables}{$t}{analyzes}) { + $main_analyze[0] = $t; + $main_analyze[1] = $autoanalyze_info{tables}{$t}{analyzes}; + } + } + $analyze_info .= ""; + + if ($graph) { + my @small = (); + foreach my $d (sort keys %{$autoanalyze_info{tables}}) { + if ((($autoanalyze_info{tables}{$d}{analyzes} * 100) / ($total_count||1)) > $pie_percentage_limit) { + $infos{$d} = $autoanalyze_info{tables}{$d}{analyzes} || 0; + } else { + $infos{"Sum analyzes < $pie_percentage_limit%"} += $autoanalyze_info{tables}{$d}{analyzes} || 0; + push(@small, $d); } - $total_count += $lock_info{$t}{count}; - $total_duration += $lock_info{$t}{duration}; } - print $fh "\n"; - print $fh "
    - - - - - - - - + my $tempfile_size_peak = 0; + my $tempfile_size_peak_date = ''; + foreach (sort {$overall_stat{'peak'}{$b}{tempfile_size} <=> $overall_stat{'peak'}{$a}{tempfile_size}} keys %{$overall_stat{'peak'}}) { + $tempfile_size_peak = &pretty_print_size($overall_stat{'peak'}{$_}{tempfile_size}); + $tempfile_size_peak_date = $_; + last; + } + print $fh qq{ +

    Temporary Files

    + +
    +

    Size of temporary files

    +
    +

    Key values

    +
    +
      +
    • $tempfile_size_peak Temp Files Peak
      • +
    • $tempfile_size_peak_date Date
      • +
    +
    +
    +
    +

    Size of temporary files ($avg_minutes minutes average)

    +$drawn_graphs{temporarydata_graph} +
    +
    +}; + delete $drawn_graphs{temporarydata_graph}; + + my $tempfile_count_peak = 0; + my $tempfile_count_peak_date = ''; + foreach (sort {$overall_stat{'peak'}{$b}{tempfile_count} <=> $overall_stat{'peak'}{$a}{tempfile_count}} keys %{$overall_stat{'peak'}}) { + $tempfile_count_peak = &comma_numbers($overall_stat{'peak'}{$_}{tempfile_count}); + $tempfile_count_peak_date = $_; + last; + } + print $fh qq{ +
    +

    Number of temporary files

    +
    +

    Key values

    +
    +
      +
    • $tempfile_count_peak per second Temp Files Peak
      • +
    • $tempfile_count_peak_date Date
      • +
    +
    +
    +
    +

    Number of temporary files ($avg_minutes minutes average)

    +$drawn_graphs{temporaryfile_graph} +
    +
    +}; + delete $drawn_graphs{temporaryfile_graph}; + + my $tempfiles_activity = ''; + foreach my $d (sort {$a <=> $b} keys %per_minute_info) { + $d =~ /^\d{4}(\d{2})(\d{2})$/; + my $zday = "$abbr_month{$1} $2"; + foreach my $h (sort {$a <=> $b} keys %{$per_minute_info{$d}}) { + $tempfiles_activity .= ""; + $zday = ""; + my %tinf = (); + foreach my $m (keys %{$per_minute_info{$d}{$h}}) { + if (exists $per_minute_info{$d}{$h}{$m}{tempfile}) { + $tinf{size} += $per_minute_info{$d}{$h}{$m}{tempfile}{size}; + $tinf{count} += $per_minute_info{$d}{$h}{$m}{tempfile}{count}; + } + } + if (scalar keys %tinf) { + my $temp_average = &pretty_print_size(sprintf("%.2f", $tinf{size} / $tinf{count})); + $tempfiles_activity .= ""; + } else { + $tempfiles_activity .= ""; + } + } + } + + $tempfiles_activity = qq{} if (!$tempfiles_activity); + + print $fh qq{ +
    +

    Temporary Files Activity

    +
    + +
    +
    +
    TypeObjectCountTotal DurationAvg duration (s)
    $zday$h" . &comma_numbers($tinf{count}) . + "$temp_average00
    $NODATA
    + + + + + + + + + $tempfiles_activity + +
    DayHourCountAverage size
    + + + ↑ Back to the top of the Temporay Files Activity table + + + }; - my $total_count = 0; - my $total_duration = 0; - foreach my $t (sort keys %lock_info) { - print $fh "
    $t", &comma_numbers($lock_info{$t}{count}), - "", &convert_time($lock_info{$t}{duration}), "", - &convert_time($lock_info{$t}{duration} / $lock_info{$t}{count}), "
    $o", - &comma_numbers($lock_info{$t}{$o}{count}), "", - &convert_time($lock_info{$t}{$o}{duration}), "", - &convert_time($lock_info{$t}{$o}{duration} / $lock_info{$t}{$o}{count}), "
    $t" . $autoanalyze_info{tables}{$t}{analyzes} . + "
    Total" . &comma_numbers($total_count) . "
    Total", &comma_numbers($total_count), - "", &convert_time($total_duration), "", - &convert_time($total_duration / ($total_count || 1)), "
    \n"; - if ($graph && $total_count) { - my %locktype = (); - my @small = (); - foreach my $d (sort keys %lock_info) { - if ((($lock_info{$d}{count} * 100) / $total_count) > $pie_percentage_limit) { - $locktype{$d} = $lock_info{$d}{count} || 0; - } else { - $locktype{"Sum types < $pie_percentage_limit%"} += $lock_info{$d}{count} || 0; - push(@small, $d); + if ($#small == 0) { + $infos{$small[0]} = $infos{"Sum analyzes < $pie_percentage_limit%"}; + delete $infos{"Sum analyzes < $pie_percentage_limit%"}; + } + } + $drawn_graphs{tableanalyzes_graph} = &flotr2_piegraph($graphid++, 'tableanalyzes_graph', 'Analyzes per tables', %infos); + $total_count = &comma_numbers($total_count); + my $database = ''; + if ($main_analyze[0] =~ s/^([^\.]+)\.//) { + $database = $1; + } + + $analyze_info = qq{$NODATA} if (!$total_count); + print $fh qq{ +
    +

    Analyses per table

    +
    +

    Key values

    +
    +
      +
    • $main_analyze[0] ($main_analyze[1]) Main table analyzed (database $database)
      • +
    • $total_count analyzes Total
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{tableanalyzes_graph} +
    +
    + + + + + + + + + $analyze_info + +
    TableNumber of analyzes
    +
    +
    +
    +
    +
    +}; + delete $drawn_graphs{tableanalyzes_graph}; + +} + +sub print_vacuum +{ + + # checkpoint + my %graph_data = (); + foreach my $tm (sort {$a <=> $b} keys %per_minute_info) { + $tm =~ /(\d{4})(\d{2})(\d{2})/; + my $y = $1 - 1900; + my $mo = $2 - 1; + my $d = $3; + foreach my $h ("00" .. "23") { + next if (!exists $per_minute_info{$tm}{$h}); + my %chk_dataavg = (); + my %t_dataavg = (); + my %v_dataavg = (); + foreach my $m ("00" .. "59") { + next if (!exists $per_minute_info{$tm}{$h}{$m}); + + my $rd = &average_per_minutes($m, $avg_minutes); + + if (exists $per_minute_info{$tm}{$h}{$m}{autovacuum}) { + $v_dataavg{vcount}{"$rd"} += ($per_minute_info{$tm}{$h}{$m}{autovacuum}{count} || 0); + } + if (exists $per_minute_info{$tm}{$h}{$m}{autoanalyze}) { + $v_dataavg{acount}{"$rd"} += ($per_minute_info{$tm}{$h}{$m}{autoanalyze}{count} || 0); + } + } + + foreach my $rd (@avgs) { + my $t = timegm_nocheck(0, $rd, $h, $d, $mo, $y) * 1000; + + next if ($t < $t_min); + last if ($t > $t_max); + + if (exists $v_dataavg{vcount}) { + $graph_data{vcount} .= "[$t, " . ($v_dataavg{vcount}{"$rd"} || 0) . "],"; + } + if (exists $v_dataavg{acount}) { + $graph_data{acount} .= "[$t, " . ($v_dataavg{acount}{"$rd"} || 0) . "],"; } + } - if ($#small == 0) { - $locktype{$small[0]} = $locktype{"Sum types < $pie_percentage_limit%"}; - delete $locktype{"Sum types < $pie_percentage_limit%"}; - } - &flotr2_piegraph(10, 'lockbytype_graph', 'Type of locks', %locktype); } - print $fh "\n"; } + foreach (keys %graph_data) { + $graph_data{$_} =~ s/,$//; + } + + # VACUUMs vs ANALYZEs chart + $drawn_graphs{autovacuum_graph} = + &flotr2_graph($graphid++, 'autovacuum_graph', $graph_data{vcount}, $graph_data{acount}, + '', 'Autovacuum actions (' . $avg_minutes . ' minutes period)', '', 'VACUUMs', 'ANALYZEs' + ); - # Show session per database statistics - if (!$disable_session && exists $session_info{database}) { - print $fh qq{ -

    Sessions per database ^

    - -"; + $zday = ""; + my %ainf = (); + foreach my $m (keys %{$per_minute_info{$d}{$h}}) { + + if (exists $per_minute_info{$d}{$h}{$m}{autovacuum}{count}) { + $ainf{vcount} += $per_minute_info{$d}{$h}{$m}{autovacuum}{count}; } + if (exists $per_minute_info{$d}{$h}{$m}{autoanalyze}{count}) { + $ainf{acount} += $per_minute_info{$d}{$h}{$m}{autoanalyze}{count}; + } + } - if ($#small == 0) { - $infos{$small[0]} = $infos{"Sum sessions < $pie_percentage_limit%"}; - delete $infos{"Sum sessions < $pie_percentage_limit%"}; + if (scalar keys %ainf) { + $vacuum_activity .= ""; + } else { + $vacuum_activity .= ""; + } + if (scalar keys %ainf) { + $vacuum_activity .= ""; + } else { + $vacuum_activity .= ""; } - &flotr2_piegraph(11, 'databasesessions_graph', 'Sessions per database', %infos); } - print $fh "
    - - - - - - - + my $vacuum_size_peak = 0; + my $vacuum_size_peak_date = ''; + foreach (sort {$overall_stat{'peak'}{$b}{vacuum_size} <=> $overall_stat{'peak'}{$a}{vacuum_size}} keys %{$overall_stat{'peak'}}) { + $vacuum_size_peak = &comma_numbers($overall_stat{'peak'}{$_}{vacuum_size}); + $vacuum_size_peak_date = $_; + last; + } + my $autovacuum_peak_system_usage_db = ''; + if ($autovacuum_info{peak}{system_usage}{table} =~ s/^([^\.]+)\.//) { + $autovacuum_peak_system_usage_db = $1; + } + my $autoanalyze_peak_system_usage_db = ''; + if ($autoanalyze_info{peak}{system_usage}{table} =~ s/^([^\.]+)\.//) { + $autoanalyze_peak_system_usage_db = $1; + } + $autovacuum_info{peak}{system_usage}{elapsed} ||= 0; + $autoanalyze_info{peak}{system_usage}{elapsed} ||= 0; + print $fh qq{ +

    Vacuums

    + +
    +

    Vacuums / Analyzes Distribution

    +
    +

    Key values

    +
    +
      +
    • $autovacuum_info{peak}{system_usage}{elapsed} sec More CPU costly vacuum
      Table $autovacuum_info{peak}{system_usage}{table}
      Database $autovacuum_peak_system_usage_db
      • +
    • $autovacuum_info{peak}{system_usage}{date} Date
      • +
    • $autoanalyze_info{peak}{system_usage}{elapsed} sec More CPU costly analyze
      Table $autoanalyze_info{peak}{system_usage}{table}
      Database $autovacuum_peak_system_usage_db
      • +
    • $autoanalyze_info{peak}{system_usage}{date} Date
      • +
    +
    +
    +
    +

    Autovacuum actions ($avg_minutes minutes average)

    +$drawn_graphs{autovacuum_graph} +
    +
    }; - my $total_count = 0; - my $c = 0; - foreach my $d (sort keys %{$session_info{database}}) { - my $colb = $c % 2; - print $fh "\n"; - $total_count += $session_info{database}{$d}{count}; - $c++; - } - print $fh "
    DatabaseCountTotal DurationAvg duration (s)
    $d", &comma_numbers($session_info{database}{$d}{count}), - "", &convert_time($session_info{database}{$d}{duration}), "", - &convert_time($session_info{database}{$d}{duration} / $session_info{database}{$d}{count}), "
    		\n"; - if ($graph && $total_count) { - my %infos = (); - my @small = (); - foreach my $d (sort keys %{$session_info{database}}) { - if ((($session_info{database}{$d}{count} * 100) / $total_count) > $pie_percentage_limit) { - $infos{$d} = $session_info{database}{$d}{count} || 0; - } else { - $infos{"Sum sessions < $pie_percentage_limit%"} += $session_info{database}{$d}{count} || 0; - push(@small, $d); + delete $drawn_graphs{autovacuum_graph}; + + # ANALYZE stats per table + &print_analyze_per_table(); + + # VACUUM stats per table + &print_vacuum_per_table(); + + # Show tuples and pages removed per table + &print_vacuum_tuple_removed; + &print_vacuum_page_removed; + + my $vacuum_activity = ''; + foreach my $d (sort {$a <=> $b} keys %per_minute_info) { + my $c = 1; + $d =~ /^\d{4}(\d{2})(\d{2})$/; + my $zday = "$abbr_month{$1} $2"; + foreach my $h (sort {$a <=> $b} keys %{$per_minute_info{$d}}) { + $vacuum_activity .= "
    $zday$h" . &comma_numbers($ainf{vcount}) . "0" . &comma_numbers($ainf{acount}) . "0
    \n"; } - # Show session per user statistics - if (!$disable_session && exists $session_info{user}) { - print $fh qq{ -

    Sessions per user ^

    - -\n"; - $c++; + +} + +sub print_vacuum_per_table +{ + # VACUUM stats per table + my $total_count = 0; + my $total_idxscan = 0; + my $vacuum_info = ''; + my @main_vacuum = ('unknown',0); + foreach my $t (sort {$autovacuum_info{tables}{$b}{vacuums} <=> $autovacuum_info{tables}{$a}{vacuums}} keys %{$autovacuum_info{tables}}) { + $vacuum_info .= ""; + $total_count += $autovacuum_info{tables}{$t}{vacuums}; + $total_idxscan += $autovacuum_info{tables}{$t}{idxscans}; + if ($main_vacuum[1] < $autovacuum_info{tables}{$t}{vacuums}) { + $main_vacuum[0] = $t; + $main_vacuum[1] = $autovacuum_info{tables}{$t}{vacuums}; } - print $fh "
    - - - - - - - + $vacuum_activity = qq{} if (!$vacuum_activity); + + print $fh qq{ +
    +

    Autovacuum Activity

    +
    + +
    +
    +
    UserCountTotal DurationAvg duration (s)
    $NODATA
    + + + + + + + + + $vacuum_activity + +
    DayHourVACUUMsANALYZEs
    + + + ↑ Back to the top of the Temporay Files Activity table + + + }; - my $total_count = 0; - my $c = 0; - foreach my $d (sort keys %{$session_info{user}}) { - my $colb = $c % 2; - $total_count += $session_info{user}{$d}{count}; - print $fh "
    $d", &comma_numbers($session_info{user}{$d}{count}), - "", &convert_time($session_info{user}{$d}{duration}), "", - &convert_time($session_info{user}{$d}{duration} / $session_info{user}{$d}{count}), "
    $t" . $autovacuum_info{tables}{$t}{vacuums} . + "" . $autovacuum_info{tables}{$t}{idxscans} . + "
    \n"; - if ($graph && $total_count) { - my %infos = (); - my @small = (); - foreach my $d (sort keys %{$session_info{user}}) { - if ((($session_info{user}{$d}{count} * 100) / $total_count) > $pie_percentage_limit) { - $infos{$d} = $session_info{user}{$d}{count} || 0; - } else { - $infos{"Sum sessions < $pie_percentage_limit%"} += $session_info{user}{$d}{count} || 0; - push(@small, $d); - } + } + $vacuum_info .= "Total" . &comma_numbers($total_count) . "" . &comma_numbers($total_idxscan) . ""; + + my %infos = (); + my @small = (); + foreach my $d (sort keys %{$autovacuum_info{tables}}) { + if ((($autovacuum_info{tables}{$d}{vacuums} * 100) / ($total_count||1)) > $pie_percentage_limit) { + $infos{$d} = $autovacuum_info{tables}{$d}{vacuums} || 0; + } else { + $infos{"Sum vacuums < $pie_percentage_limit%"} += $autovacuum_info{tables}{$d}{vacuums} || 0; + push(@small, $d); + } + } + if ($#small == 0) { + $infos{$small[0]} = $infos{"Sum vacuums < $pie_percentage_limit%"}; + delete $infos{"Sum vacuums < $pie_percentage_limit%"}; + } + $drawn_graphs{tablevacuums_graph} = &flotr2_piegraph($graphid++, 'tablevacuums_graph', 'Analyzes per tables', %infos); + $vacuum_info = qq{$NODATA} if (!$total_count); + $total_count = &comma_numbers($total_count); + my $database = ''; + if ($main_vacuum[0] =~ s/^([^\.]+)\.//) { + $database = $1; + } + print $fh qq{ +
    +

    Vacuums per table

    +
    +

    Key values

    +
    +
      +
    • $main_vacuum[0] ($main_vacuum[1]) Main table vacuumed on database $database
      • +
    • $total_count vacuums Total
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{tablevacuums_graph} +
    +
    + + + + + + + + + + $vacuum_info + +
    TableNumber of vacuumsIndex scans
    +
    +
    +
    +
    +
    +}; + delete $drawn_graphs{tablevacuums_graph}; + +} + +sub print_vacuum_tuple_removed +{ + # VACUUM stats per table + my $total_count = 0; + my $total_idxscan = 0; + my $total_tuple = 0; + my $total_page = 0; + my $vacuum_info = ''; + my @main_tuple = ('unknown',0); + foreach my $t (sort {$autovacuum_info{tables}{$b}{tuples}{removed} <=> $autovacuum_info{tables}{$a}{tuples}{removed}} keys %{$autovacuum_info{tables}}) { + $vacuum_info .= "$t" . $autovacuum_info{tables}{$t}{vacuums} . + "" . $autovacuum_info{tables}{$t}{idxscans} . + "" . $autovacuum_info{tables}{$t}{tuples}{removed} . + "" . $autovacuum_info{tables}{$t}{pages}{removed} . + ""; + $total_count += $autovacuum_info{tables}{$t}{vacuums}; + $total_idxscan += $autovacuum_info{tables}{$t}{idxscans}; + $total_tuple += $autovacuum_info{tables}{$t}{tuples}{removed}; + $total_page += $autovacuum_info{tables}{$t}{pages}{removed}; + if ($main_tuple[1] < $autovacuum_info{tables}{$t}{tuples}{removed}) { + $main_tuple[0] = $t; + $main_tuple[1] = $autovacuum_info{tables}{$t}{tuples}{removed}; + } + } + $vacuum_info .= "Total" . &comma_numbers($total_count) . "" . &comma_numbers($total_idxscan) . + "" . &comma_numbers($total_tuple) . "" . &comma_numbers($total_page) . ""; + + my %infos_tuple = (); + my @small = (); + foreach my $d (sort keys %{$autovacuum_info{tables}}) { + if ((($autovacuum_info{tables}{$d}{tuples}{removed} * 100) / ($total_tuple||1)) > $pie_percentage_limit) { + $infos_tuple{$d} = $autovacuum_info{tables}{$d}{tuples}{removed} || 0; + } else { + $infos_tuple{"Sum tuples removed < $pie_percentage_limit%"} += $autovacuum_info{tables}{$d}{tuples}{removed} || 0; + push(@small, $d); + } + } + if ($#small == 0) { + $infos_tuple{$small[0]} = $infos_tuple{"Sum tuples removed < $pie_percentage_limit%"}; + delete $infos_tuple{"Sum tuples removed < $pie_percentage_limit%"}; + } + $drawn_graphs{tuplevacuums_graph} = &flotr2_piegraph($graphid++, 'tuplevacuums_graph', 'Tuples removed per tables', %infos_tuple); + $vacuum_info = qq{$NODATA} if (!$total_count); + $total_count = &comma_numbers($total_count); + my $database = ''; + if ($main_tuple[0] =~ s/^([^\.]+)\.//) { + $database = $1; + } + print $fh qq{ +
    +

    Tuples removed per table

    +
    +

    Key values

    +
    +
      +
    • $main_tuple[0] ($main_tuple[1]) Main table with removed tuples on database $database
      • +
    • $total_tuple tuples Total removed
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{tuplevacuums_graph} +
    +
    + + + + + + + + + + + + $vacuum_info + +
    TableNumber of vacuumsIndex scansTuples removedPages removed
    +
    +
    +
    +
    +
    +}; + delete $drawn_graphs{tuplevacuums_graph}; + +} + +sub print_vacuum_page_removed +{ + # VACUUM stats per table + my $total_count = 0; + my $total_idxscan = 0; + my $total_tuple = 0; + my $total_page = 0; + my $vacuum_info = ''; + my @main_tuple = ('unknown',0); + my @main_page = ('unknown',0); + foreach my $t (sort {$autovacuum_info{tables}{$b}{pages}{removed} <=> $autovacuum_info{tables}{$a}{pages}{removed}} keys %{$autovacuum_info{tables}}) { + $vacuum_info .= "$t" . $autovacuum_info{tables}{$t}{vacuums} . + "" . $autovacuum_info{tables}{$t}{idxscans} . + "" . $autovacuum_info{tables}{$t}{tuples}{removed} . + "" . $autovacuum_info{tables}{$t}{pages}{removed} . + ""; + $total_count += $autovacuum_info{tables}{$t}{vacuums}; + $total_idxscan += $autovacuum_info{tables}{$t}{idxscans}; + $total_tuple += $autovacuum_info{tables}{$t}{tuples}{removed}; + $total_page += $autovacuum_info{tables}{$t}{pages}{removed}; + if ($main_page[1] < $autovacuum_info{tables}{$t}{pages}{removed}) { + $main_page[0] = $t; + $main_page[1] = $autovacuum_info{tables}{$t}{pages}{removed}; + } + } + $vacuum_info .= "Total" . &comma_numbers($total_count) . "" . &comma_numbers($total_idxscan) . + "" . &comma_numbers($total_tuple) . "" . &comma_numbers($total_page) . ""; + + my %infos_page = (); + my @small = (); + foreach my $d (sort keys %{$autovacuum_info{tables}}) { + if ((($autovacuum_info{tables}{$d}{pages}{removed} * 100) / ($total_page || 1)) > $pie_percentage_limit) { + $infos_page{$d} = $autovacuum_info{tables}{$d}{pages}{removed} || 0; + } else { + $infos_page{"Sum pages removed < $pie_percentage_limit%"} += $autovacuum_info{tables}{$d}{pages}{removed} || 0; + push(@small, $d); + } + } + if ($#small == 0) { + $infos_page{$small[0]} = $infos_page{"Sum pages removed < $pie_percentage_limit%"}; + delete $infos_page{"Sum pages removed < $pie_percentage_limit%"}; + } + $drawn_graphs{pagevacuums_graph} = &flotr2_piegraph($graphid++, 'pagevacuums_graph', 'Tuples removed per tables', %infos_page); + $vacuum_info = qq{$NODATA} if (!$total_count); + $total_count = &comma_numbers($total_count); + my $database = ''; + if ($main_page[0] =~ s/^([^\.]+)\.//) { + $database = $1; + } + print $fh qq{ +
    +

    Pages removed per table

    +
    +

    Key values

    +
    +
      +
    • $main_page[0] ($main_page[1]) Main table with removed pages on database $database
      • +
    • $total_page pages Total removed
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{pagevacuums_graph} +
    +
    + + + + + + + + + + + + $vacuum_info + +
    TableNumber of vacuumsIndex scansTuples removedPages removed
    +
    +
    +
    +
    +
    +}; + delete $drawn_graphs{pagevacuums_graph}; + +} + +sub print_lock_type +{ + my %locktype = (); + my $total_count = 0; + my $total_duration = 0; + my $locktype_info = ''; + my @main_locktype = ('unknown',0); + foreach my $t (sort keys %lock_info) { + $locktype_info .= "$t" . &comma_numbers($lock_info{$t}{count}) . + "" . &convert_time($lock_info{$t}{duration}) . "" . + &convert_time($lock_info{$t}{duration} / ($lock_info{$t}{count} || 1)) . ""; + $total_count += $lock_info{$t}{count}; + $total_duration += $lock_info{$t}{duration}; + if ($main_locktype[1] < $lock_info{$t}{count}) { + $main_locktype[0] = $t; + $main_locktype[1] = $lock_info{$t}{count}; + } + foreach my $o (sort keys %{$lock_info{$t}}) { + next if (($o eq 'count') || ($o eq 'duration') || ($o eq 'chronos')); + $locktype_info .= "$o" . &comma_numbers($lock_info{$t}{$o}{count}) . + "" . &convert_time($lock_info{$t}{$o}{duration}) . "" . + &convert_time($lock_info{$t}{$o}{duration} / $lock_info{$t}{$o}{count}) . + "\n"; + } + } + if ($total_count > 0) { + $locktype_info .= "Total" . &comma_numbers($total_count) . + "" . &convert_time($total_duration) . "" . + &convert_time($total_duration / ($total_count || 1)) . ""; + } else { + $locktype_info = qq{$NODATA}; + } + if ($graph) { + my @small = (); + foreach my $d (sort keys %lock_info) { + if ((($lock_info{$d}{count} * 100) / ($total_count||1)) > $pie_percentage_limit) { + $locktype{$d} = $lock_info{$d}{count} || 0; + } else { + $locktype{"Sum lock types < $pie_percentage_limit%"} += $lock_info{$d}{count} || 0; + push(@small, $d); + } - if ($#small == 0) { - $infos{$small[0]} = $infos{"Sum sessions < $pie_percentage_limit%"}; - delete $infos{"Sum sessions < $pie_percentage_limit%"}; + } + if ($#small == 0) { + $locktype{$small[0]} = $locktype{"Sum types < $pie_percentage_limit%"}; + delete $locktype{"Sum lock types < $pie_percentage_limit%"}; + } + } + $drawn_graphs{lockbytype_graph} = &flotr2_piegraph($graphid++, 'lockbytype_graph', 'Type of locks', %locktype); + $total_count = &comma_numbers($total_count); + print $fh qq{ +

    Locks

    +
    +

    Locks by types

    +
    +

    Key values

    +
    +
      +
    • $main_locktype[0] Main Lock Type
      • +
    • $total_count locks Total
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{lockbytype_graph} +
    +
    + + + + + + + + + + + + $locktype_info + +
    TypeObjectCountTotal DurationAverage Duration (s)
    +
    +
    +
    +
    +
    +}; + delete $drawn_graphs{lockbytype_graph}; +} + +sub print_query_type +{ + + my %data = (); + my $total_queries = 0; + my $total_select = 0; + my $total_write = 0; + foreach my $a (@SQL_ACTION) { + $total_queries += $overall_stat{$a}; + if ($a eq 'SELECT') { + $total_select += $overall_stat{$a}; + } elsif ($a ne 'OTHERS') { + $total_write += $overall_stat{$a}; + } + } + my $total = $overall_stat{'queries_number'}; + + my $querytype_info = ''; + foreach my $a (@SQL_ACTION) { + $querytype_info .= "$a" . &comma_numbers($overall_stat{$a}) . + "" . sprintf("%0.2f", ($overall_stat{$a} * 100) / ($total||1)) . "%"; + } + if (($total - $total_queries) > 0) { + $querytype_info .= "OTHERS" . &comma_numbers($total - $total_queries) . + "" . sprintf("%0.2f", (($total - $total_queries) * 100) / ($total||1)) . "%"; + } + $querytype_info = qq{$NODATA} if (!$total); + + if ($graph && $total) { + foreach my $t (@SQL_ACTION) { + if ((($overall_stat{$t} * 100) / ($total||1)) > $pie_percentage_limit) { + $data{$t} = $overall_stat{$t} || 0; + } else { + $data{"Sum query types < $pie_percentage_limit%"} += $overall_stat{$t} || 0; } - &flotr2_piegraph(12, 'usersessions_graph', 'Sessions per user', %infos); } - print $fh "\n"; + if (((($total - $total_queries) * 100) / ($total||1)) > $pie_percentage_limit) { + $data{'Others'} = $total - $total_queries; + } else { + $data{"Sum query types < $pie_percentage_limit%"} += $total - $total_queries; + } } + $drawn_graphs{queriesbytype_graph} = &flotr2_piegraph($graphid++, 'queriesbytype_graph', 'Type of queries', %data); - # Show session per host statistics - if (!$disable_session && exists $session_info{host}) { - print $fh qq{ -

    Sessions per host ^

    - -"; + $total_count += $database_info{$d}{count}; + if ($main_database[1] < $database_info{$d}{count}) { + $main_database[0] = $d; + $main_database[1] = $database_info{$d}{count}; + } + foreach my $r (sort keys %{$database_info{$d}}) { + next if ($r eq 'count'); + $query_database_info .= ""; } - print $fh "
    - - - - - - - -}; - my $total_count = 0; - my $c = 0; - foreach my $d (sort keys %{$session_info{host}}) { - my $colb = $c % 2; - $total_count += $session_info{host}{$d}{count}; - print $fh "\n"; - $c++; + $total_select = &comma_numbers($total_select); + $total_write = &comma_numbers($total_write); + print $fh qq{ +

    Queries

    +
    +

    Queries by type

    +
    +

    Key values

    +
    +
      +
    • $total_select Total read queries
      • +
    • $total_write Total write queries
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{queriesbytype_graph} +
    +
    +
    HostCountTotal DurationAvg duration (s)
    $d", &comma_numbers($session_info{host}{$d}{count}), - "", &convert_time($session_info{host}{$d}{duration}), "", - &convert_time($session_info{host}{$d}{duration} / $session_info{host}{$d}{count}), "
    + + + + + + + + + $querytype_info + +
    TypeCountPercentage
    + + + + + +}; + delete $drawn_graphs{queriesbytype_graph}; + +} + +sub print_query_per_database +{ + my %infos = (); + my $total_count = 0; + my $query_database_info = ''; + my @main_database = ('unknown', 0); + foreach my $d (sort keys %database_info) { + $query_database_info .= "
    $dTotal" . + &comma_numbers($database_info{$d}{count}) . "
    $r" . + &comma_numbers($database_info{$d}{$r}) . "
    \n"; - if ($graph && $total_count) { - my %infos = (); - my @small = (); - foreach my $d (sort keys %{$session_info{host}}) { - if ((($session_info{host}{$d}{count} * 100) / $total_count) > $pie_percentage_limit) { - $infos{$d} = $session_info{host}{$d}{count} || 0; - } else { - $infos{"Sum sessions < $pie_percentage_limit%"} += $session_info{host}{$d}{count} || 0; - push(@small, $d); - } - } - if ($#small == 0) { - $infos{$small[0]} = $infos{"Sum sessions < $pie_percentage_limit%"}; - delete $infos{"Sum sessions < $pie_percentage_limit%"}; + } + + $query_database_info = qq{$NODATA} if (!$total_count); + + if ($graph) { + my @small = (); + foreach my $d (sort keys %database_info) { + if ((($database_info{$d}{count} * 100) / ($total_count || 1)) > $pie_percentage_limit) { + $infos{$d} = $database_info{$d}{count} || 0; + } else { + $infos{"Sum queries per databases < $pie_percentage_limit%"} += $database_info{$d}{count} || 0; + push(@small, $d); } - &flotr2_piegraph(13, 'hostsessions_graph', 'Sessions per host', %infos); } - print $fh "\n"; + if ($#small == 0) { + $infos{$small[0]} = $infos{"Sum queries per databases < $pie_percentage_limit%"}; + delete $infos{"Sum queries per databases < $pie_percentage_limit%"}; + } } + $drawn_graphs{queriesbydatabase_graph} = &flotr2_piegraph($graphid++, 'queriesbydatabase_graph', 'Queries per database', %infos); - # Show connection per database statistics - if (!$disable_connection && exists $connection_info{database}) { - print $fh qq{ -

    Connections per database ^

    - -"; + $total_count += $application_info{$d}{count}; + if ($main_application[1] < $application_info{$d}{count}) { + $main_application[0] = $d; + $main_application[1] = $application_info{$d}{count}; + } + foreach my $r (sort keys %{$application_info{$d}}) { + next if ($r eq 'count'); + $query_application_info .= ""; } - print $fh "
    - - - - - - -}; - my $total_count = 0; - foreach my $d (sort keys %{$connection_info{database}}) { - print $fh "\n"; - $total_count += $connection_info{database}{$d}; - foreach my $u (sort keys %{$connection_info{user}}) { - next if (!exists $connection_info{database_user}{$d}{$u}); - print $fh "\n"; - } + $main_database[1] = &comma_numbers($main_database[1]); + print $fh qq{ +
    +

    Queries by database

    +
    +

    Key values

    +
    +
      +
    • $main_database[0] Main database
      • +
    • $main_database[1] Requests
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{queriesbydatabase_graph} +
    +
    +
    DatabaseUserCount
    $d", - &comma_numbers($connection_info{database}{$d}), "
    $u", - &comma_numbers($connection_info{database_user}{$d}{$u}), "
    + + + + + + + + + $query_database_info + +
    DatabaseRequest typeCount
    + + + + + +}; + delete $drawn_graphs{queriesbydatabase_graph}; + + +} + +sub print_query_per_application +{ + my %infos = (); + my $total_count = 0; + my $query_application_info = ''; + my @main_application = ('unknown', 0); + foreach my $d (sort keys %application_info) { + $query_application_info .= "
    $dTotal" . + &comma_numbers($application_info{$d}{count}) . "
    $r" . + &comma_numbers($application_info{$d}{$r}) . "
    \n"; - if ($graph && $total_count) { - my %infos = (); - my @small = (); - foreach my $d (sort keys %{$connection_info{database}}) { - if ((($connection_info{database}{$d} * 100) / $total_count) > $pie_percentage_limit) { - $infos{$d} = $connection_info{database}{$d} || 0; - } else { - $infos{"Sum connections < $pie_percentage_limit%"} += $connection_info{database}{$d} || 0; - push(@small, $d); - } - } - if ($#small == 0) { - $infos{$small[0]} = $infos{"Sum connections < $pie_percentage_limit%"}; - delete $infos{"Sum connections < $pie_percentage_limit%"}; + } + $query_application_info = qq{$NODATA} if (!$total_count); + if ($graph) { + my @small = (); + foreach my $d (sort keys %application_info) { + if ((($application_info{$d}{count} * 100) / ($total_count || 1)) > $pie_percentage_limit) { + $infos{$d} = $application_info{$d}{count} || 0; + } else { + $infos{"Sum queries per applications < $pie_percentage_limit%"} += $application_info{$d}{count} || 0; + push(@small, $d); } - &flotr2_piegraph(14, 'databaseconnections_graph', 'Connections per database', %infos); } - print $fh "\n"; + if ($#small == 0) { + $infos{$small[0]} = $infos{"Sum queries per applications < $pie_percentage_limit%"}; + delete $infos{"Sum queries per applications < $pie_percentage_limit%"}; + } } + $drawn_graphs{queriesbyapplication_graph} = &flotr2_piegraph($graphid++, 'queriesbyapplication_graph', 'Queries per application', %infos); - # Show connection per user statistics - if (!$disable_connection && exists $connection_info{user}) { - print $fh qq{ -

    Connections per user ^

    - -"; + $total_count += $user_info{$d}{count}; + if ($main_user[1] < $user_info{$d}{count}) { + $main_user[0] = $d; + $main_user[1] = $user_info{$d}{count}; + } + foreach my $r (sort keys %{$user_info{$d}}) { + next if ($r eq 'count'); + $query_user_info .= ""; } - print $fh "
    - - - - - -}; - - my $total_count = 0; - my $c = 0; - foreach my $u (sort keys %{$connection_info{user}}) { - my $colb = $c % 2; - print $fh "\n"; - $total_count += $connection_info{user}{$u}; - $c++; + $main_application[1] = &comma_numbers($main_application[1]); + print $fh qq{ +
    +

    Queries by application

    +
    +

    Key values

    +
    +
      +
    • $main_application[0] Main application
      • +
    • $main_application[1] Requests
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{queriesbyapplication_graph} +
    +
    +
    UserCount
    $u", &comma_numbers($connection_info{user}{$u}), - "
    + + + + + + + + + $query_application_info + +
    ApplicationRequest typeCount
    + + + + + +}; + delete $drawn_graphs{queriesbyapplication_graph}; + +} + +sub print_query_per_user +{ + my %infos = (); + my $total_count = 0; + my $query_user_info = ''; + my @main_user = ('unknown', 0); + foreach my $d (sort keys %user_info) { + $query_user_info .= "
    $dTotal" . + &comma_numbers($user_info{$d}{count}) . "
    $r" . + &comma_numbers($user_info{$d}{$r}) . "
    \n"; - if ($graph && $total_count) { - my %infos = (); - my @small = (); - foreach my $d (sort keys %{$connection_info{user}}) { - if ((($connection_info{user}{$d} * 100) / $total_count) > $pie_percentage_limit) { - $infos{$d} = $connection_info{user}{$d} || 0; - } else { - $infos{"Sum connections < $pie_percentage_limit%"} += $connection_info{user}{$d} || 0; - push(@small, $d); - } - } - if ($#small == 0) { - $infos{$small[0]} = $infos{"Sum connections < $pie_percentage_limit%"}; - delete $infos{"Sum connections < $pie_percentage_limit%"}; + } + $query_user_info = qq{$NODATA} if (!$total_count); + + if ($graph) { + my @small = (); + foreach my $d (sort keys %user_info) { + if ((($user_info{$d}{count} * 100) / ($total_count || 1)) > $pie_percentage_limit) { + $infos{$d} = $user_info{$d}{count} || 0; + } else { + $infos{"Sum queries per users < $pie_percentage_limit%"} += $user_info{$d}{count} || 0; + push(@small, $d); } - &flotr2_piegraph(15, 'userconnections_graph', 'Connections per user', %infos); } - print $fh "\n"; + if ($#small == 0) { + $infos{$small[0]} = $infos{"Sum queries per users < $pie_percentage_limit%"}; + delete $infos{"Sum queries per users < $pie_percentage_limit%"}; + } } + $drawn_graphs{queriesbyuser_graph} = &flotr2_piegraph($graphid++, 'queriesbyuser_graph', 'Queries per user', %infos); - # Show connection per host statistics - if (!$disable_connection && exists $connection_info{host}) { - print $fh qq{ -

    Connections per host ^

    - -"; + $total_count += $host_info{$d}{count}; + if ($main_host[1] < $host_info{$d}{count}) { + $main_host[0] = $d; + $main_host[1] = $host_info{$d}{count}; + } + foreach my $r (sort keys %{$host_info{$d}}) { + next if ($r eq 'count'); + $query_host_info .= ""; } - print $fh "
    - - - - - -}; - - my $total_count = 0; - my $c = 0; - foreach my $h (sort keys %{$connection_info{host}}) { - my $colb = $c % 2; - print $fh "\n"; - $total_count += $connection_info{host}{$h}; - $c++; + $main_user[1] = &comma_numbers($main_user[1]); + print $fh qq{ +
    +

    Queries by user

    +
    +

    Key values

    +
    +
      +
    • $main_user[0] Main user
      • +
    • $main_user[1] Requests
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{queriesbyuser_graph} +
    +
    +
    HostCount
    $h", &comma_numbers($connection_info{host}{$h}), - "
    + + + + + + + + + $query_user_info + +
    UserRequest typeCount
    + + + + + +}; + delete $drawn_graphs{queriesbyuser_graph}; + +} + +sub print_query_per_host +{ + my %infos = (); + my $total_count = 0; + my $query_host_info = ''; + my @main_host = ('unknown', 0); + foreach my $d (sort keys %host_info) { + $query_host_info .= "
    $dTotal" . + &comma_numbers($host_info{$d}{count}) . "
    $r" . + &comma_numbers($host_info{$d}{$r}) . "
    \n"; - if ($graph && $total_count) { - my %infos = (); - my @small = (); - foreach my $d (sort keys %{$connection_info{host}}) { - if ((($connection_info{host}{$d} * 100) / $total_count) > $pie_percentage_limit) { - $infos{$d} = $connection_info{host}{$d} || 0; - } else { - $infos{"Sum connections < $pie_percentage_limit%"} += $connection_info{host}{$d} || 0; - push(@small, $d); - } - } - if ($#small == 0) { - $infos{$small[0]} = $infos{"Sum connections < $pie_percentage_limit%"}; - delete $infos{"Sum connections < $pie_percentage_limit%"}; + } + $query_host_info = qq{$NODATA} if (!$total_count); + + if ($graph) { + my @small = (); + foreach my $d (sort keys %host_info) { + if ((($host_info{$d}{count} * 100) / ($total_count || 1)) > $pie_percentage_limit) { + $infos{$d} = $host_info{$d}{count} || 0; + } else { + $infos{"Sum queries per hosts < $pie_percentage_limit%"} += $host_info{$d}{count} || 0; + push(@small, $d); } - &flotr2_piegraph(16, 'hostconnections_graph', 'Connections per host', %infos); } - print $fh "\n"; + if ($#small == 0) { + $infos{$small[0]} = $infos{"Sum queries per hosts < $pie_percentage_limit%"}; + delete $infos{"Sum queries per hosts < $pie_percentage_limit%"}; + } } + $drawn_graphs{queriesbyhost_graph} = &flotr2_piegraph($graphid++, 'queriesbyhost_graph', 'Queries per host', %infos); - # Show lock wait detailed informations - if (!$disable_lock && scalar keys %lock_info > 0) { + $main_host[1] = &comma_numbers($main_host[1]); + print $fh qq{ +
    +

    Queries by host

    +
    +

    Key values

    +
    +
      +
    • $main_host[0] Main host
      • +
    • $main_host[1] Requests
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{queriesbyhost_graph} +
    +
    + + + + + + + + + + $query_host_info + +
    HostRequest typeCount
    +
    +
    +
    +
    +
    +}; + delete $drawn_graphs{queriesbyhost_graph}; - my @top_locked_queries; - foreach my $h (keys %normalyzed_info) { - if (exists($normalyzed_info{$h}{locks})) { - push (@top_locked_queries, [$h, $normalyzed_info{$h}{locks}{count}, $normalyzed_info{$h}{locks}{wait}, - $normalyzed_info{$h}{locks}{minwait}, $normalyzed_info{$h}{locks}{maxwait}]); - } +} + + +sub print_lock_queries_report +{ + my @top_locked_queries; + foreach my $h (keys %normalyzed_info) { + if (exists($normalyzed_info{$h}{locks})) { + push (@top_locked_queries, [$h, $normalyzed_info{$h}{locks}{count}, $normalyzed_info{$h}{locks}{wait}, + $normalyzed_info{$h}{locks}{minwait}, $normalyzed_info{$h}{locks}{maxwait}]); } + } - # Most frequent waiting queries (N) - @top_locked_queries = sort {$b->[2] <=> $a->[2]} @top_locked_queries; + # Most frequent waiting queries (N) + @top_locked_queries = sort {$b->[2] <=> $a->[2]} @top_locked_queries; + print $fh qq{ +
    +

    Most frequent waiting queries (N)

    +
    + + + + + + + + + + + + + +}; + + my $rank = 1; + for (my $i = 0 ; $i <= $#top_locked_queries ; $i++) { + my $count = &comma_numbers($top_locked_queries[$i]->[1]); + my $total_time = &convert_time($top_locked_queries[$i]->[2]); + my $min_time = &convert_time($top_locked_queries[$i]->[3]); + my $max_time = &convert_time($top_locked_queries[$i]->[4]); + my $avg_time = &convert_time($top_locked_queries[$i]->[4] / ($top_locked_queries[$i]->[1] || 1)); + my $query = &highlight_code($top_locked_queries[$i]->[0]); + my $k = $top_locked_queries[$i]->[0]; + my $example = qq{

    }; + $example = '' if (scalar keys %{$normalyzed_info{$k}{samples}} <= 1); print $fh qq{ -

    Most frequent waiting queries (N)^

    -
    RankCountTotal timeMin timeMax timeAvg durationQuery
    - - - - - - - + + + + + + + + \n"; - $idx++; + print $fh qq{ + +

    + + + + +}; } - print $fh "
    RankCountTotal wait time (s)Min/Max/Avg duration (s)Query
    $rank$count$total_time$min_time$max_time$avg_time +
    $query
    + $example + +
    +
    }; - my $idx = 1; - for (my $i = 0 ; $i <= $#top_locked_queries ; $i++) { - last if ($i > $end_top); - my $col = $i % 2; - print $fh "
    ", $i + 1, "", - $top_locked_queries[$i]->[1], "", &convert_time($top_locked_queries[$i]->[2]), - "", &convert_time($top_locked_queries[$i]->[3]), "/", &convert_time($top_locked_queries[$i]->[4]), "/", - &convert_time(($top_locked_queries[$i]->[4] / $top_locked_queries[$i]->[1])), - "
    ", - &highlight_code($top_locked_queries[$i]->[0]), "
    \n"; - my $k = $top_locked_queries[$i]->[0]; - if ($normalyzed_info{$k}{count} > 1) { - print $fh -"
    "; - my $j = 0; - foreach my $d (sort {$b <=> $a} keys %{$normalyzed_info{$k}{samples}}) { - my $colb = $j % 2; - my $db = " - database: $normalyzed_info{$k}{samples}{$d}{db}" if ($normalyzed_info{$k}{samples}{$d}{db}); - $db .= ", user: $normalyzed_info{$k}{samples}{$d}{user}" if ($normalyzed_info{$k}{samples}{$d}{user}); - $db .= ", remote: $normalyzed_info{$k}{samples}{$d}{remote}" if ($normalyzed_info{$k}{samples}{$d}{remote}); - $db .= ", app: $normalyzed_info{$k}{samples}{$d}{app}" if ($normalyzed_info{$k}{samples}{$d}{app}); - $db =~ s/^, / - /; - print $fh -"
    ", - &convert_time($d), " | ", &highlight_code($normalyzed_info{$k}{samples}{$d}{query}), "
    "; - $j++; - } - print $fh "
    "; + if ($normalyzed_info{$k}{count} > 1) { + foreach my $d (sort {$b <=> $a} keys %{$normalyzed_info{$k}{samples}}) { + $query = &highlight_code($normalyzed_info{$k}{samples}{$d}{query}); + my $details = "[ Date: $normalyzed_info{$k}{samples}{$d}{date}"; + $details .= " - Duration: " . &convert_time($d) if ($normalyzed_info{$k}{samples}{$d}{duration}); + $details .= " - Database: $normalyzed_info{$k}{samples}{$d}{db}" if ($normalyzed_info{$k}{samples}{$d}{db}); + $details .= " - User: $normalyzed_info{$k}{samples}{$d}{user}" if ($normalyzed_info{$k}{samples}{$d}{user}); + $details .= " - Remote: $normalyzed_info{$k}{samples}{$d}{remote}" if ($normalyzed_info{$k}{samples}{$d}{remote}); + $details .= " - Application: $normalyzed_info{$k}{samples}{$d}{app}" if ($normalyzed_info{$k}{samples}{$d}{app}); + $details .= " ]"; + print $fh qq{ +
    $query
    + 
    $details
    +}; + } - print $fh "
    \n"; - @top_locked_queries = (); + $rank++; + } + if ($#top_locked_queries == -1) { + print $fh qq{$NODATA}; + } + print $fh qq{ + + +
    +
    +}; - # Queries that waited the most - @top_locked_info = sort {$b->[1] <=> $a->[1]} @top_locked_info; + @top_locked_queries = (); + + # Queries that waited the most + @top_locked_info = sort {$b->[1] <=> $a->[1]} @top_locked_info; + print $fh qq{ +
    +

    Queries that waited the most

    +
    + + + + + + + + + +}; + + $rank = 1; + for (my $i = 0 ; $i <= $#top_locked_info ; $i++) { + my $query = &highlight_code($top_locked_info[$i]->[2]); + my $details = "[ Date: " . ($top_locked_info[$i]->[1] || ''); + $details .= " - Database: $top_locked_info[$i]->[3]" if ($top_locked_info[$i]->[3]); + $details .= " - User: $top_locked_info[$i]->[4]" if ($top_locked_info[$i]->[4]); + $details .= " - Remote: $top_locked_info[$i]->[5]" if ($top_locked_info[$i]->[5]); + $details .= " - Application: $top_locked_info[$i]->[6]" if ($top_locked_info[$i]->[6]); + $details .= " ]"; + my $time = &convert_time($top_locked_info[$i]->[0]); print $fh qq{ -

    Queries that waited the most^

    -
    RankWait timeQuery
    - - - - - + + + + + +}; + $rank++; + } + if ($#top_locked_info == -1) { + print $fh qq{}; + } + print $fh qq{ + +
    RankWait time (s)Query
    $rank$time +
    $query
    + 
    $details
    +
    $NODATA
    +
    +
    }; - for (my $i = 0 ; $i <= $#top_locked_info ; $i++) { - my $col = $i % 2; - my $ttl = $top_locked_info[$i]->[1] || ''; - my $db = " - database: $top_locked_info[$i]->[3]" if ($top_locked_info[$i]->[3]); - $db .= ", user: $top_locked_info[$i]->[4]" if ($top_locked_info[$i]->[4]); - $db .= ", remote: $top_locked_info[$i]->[5]" if ($top_locked_info[$i]->[5]); - $db .= ", app: $top_locked_info[$i]->[6]" if ($top_locked_info[$i]->[6]); - $db =~ s/^, / - /; - print $fh "", $i + 1, "", - &convert_time($top_locked_info[$i]->[0]), - "
    ", - &highlight_code($top_locked_info[$i]->[2]), "
    \n"; - } - print $fh "\n"; - } - # Show temporary files detailed informations - if (!$disable_temporary && scalar keys %tempfile_info > 0) { +} - my @top_temporary; - foreach my $h (keys %normalyzed_info) { - if (exists($normalyzed_info{$h}{tempfiles})) { - push (@top_temporary, [$h, $normalyzed_info{$h}{tempfiles}{count}, $normalyzed_info{$h}{tempfiles}{size}, - $normalyzed_info{$h}{tempfiles}{minsize}, $normalyzed_info{$h}{tempfiles}{maxsize}]); - } +sub print_tempfile_report +{ + my @top_temporary = (); + foreach my $h (keys %normalyzed_info) { + if (exists($normalyzed_info{$h}{tempfiles})) { + push (@top_temporary, [$h, $normalyzed_info{$h}{tempfiles}{count}, $normalyzed_info{$h}{tempfiles}{size}, + $normalyzed_info{$h}{tempfiles}{minsize}, $normalyzed_info{$h}{tempfiles}{maxsize}]); } + } - # Queries generating the most temporary files (N) + # Queries generating the most temporary files (N) + if ($#top_temporary >= 0) { @top_temporary = sort {$b->[1] <=> $a->[1]} @top_temporary; print $fh qq{ -

    Queries generating the most temporary files (N)^

    - - - - - - - - +
    +

    Queries generating the most temporary files (N)

    +
    +
    RankCountTotal sizeMin/Max/Avg sizeQuery
    + + + + + + + + + + + + }; - my $idx = 1; + my $rank = 1; for (my $i = 0 ; $i <= $#top_temporary ; $i++) { - last if ($i > $end_top); - my $col = $i % 2; - print $fh " + + + + + + + + +}; } - print $fh "\n"; - $idx++; + $rank++; } - print $fh "
    RankCountTotal sizeMin sizeMax sizeAvg sizeQuery
    ", $i + 1, "", - $top_temporary[$i]->[1], "", &comma_numbers($top_temporary[$i]->[2]), - "", &comma_numbers($top_temporary[$i]->[3]), - "/", &comma_numbers($top_temporary[$i]->[4]), "/", - &comma_numbers(sprintf("%.2f", $top_temporary[$i]->[2] / $top_temporary[$i]->[1])), - "
    ", - &highlight_code($top_temporary[$i]->[0]), "
    "; + my $count = &comma_numbers($top_temporary[$i]->[1]); + my $total_size = &pretty_print_size($top_temporary[$i]->[2]); + my $min_size = &pretty_print_size($top_temporary[$i]->[3]); + my $max_size = &pretty_print_size($top_temporary[$i]->[4]); + my $avg_size = &pretty_print_size($top_temporary[$i]->[2] / ($top_temporary[$i]->[1] || 1)); + my $query = &highlight_code($top_temporary[$i]->[0]); + my $example = qq{

    }; + $example = '' if ($count <= 1); + print $fh qq{ +
    $rank$count$total_size$min_size$max_size$avg_size +
    $query
    + $example + +
    +
    +}; my $k = $top_temporary[$i]->[0]; if ($normalyzed_info{$k}{count} > 1) { - print $fh "
    "; - my $i = 0; foreach my $d (sort {$b <=> $a} keys %{$normalyzed_info{$k}{samples}}) { - my $colb = $i % 2; - my $db = " - database: $normalyzed_info{$k}{samples}{$d}{db}" if ($normalyzed_info{$k}{samples}{$d}{db}); - $db .= ", user: $normalyzed_info{$k}{samples}{$d}{user}" if ($normalyzed_info{$k}{samples}{$d}{user}); - $db .= ", remote: $normalyzed_info{$k}{samples}{$d}{remote}" if ($normalyzed_info{$k}{samples}{$d}{remote}); - $db .= ", app: $normalyzed_info{$k}{samples}{$d}{app}" if ($normalyzed_info{$k}{samples}{$d}{app}); - $db =~ s/^, / - /; - print $fh "
    ", - &convert_time($d), " | ", &highlight_code($normalyzed_info{$k}{samples}{$d}{query}), "
    "; - $i++; + $query = &highlight_code($normalyzed_info{$k}{samples}{$d}{query}); + my $details = "Duration: " . &convert_time($d) . "
    "; + $details .= "Database: $normalyzed_info{$k}{samples}{$d}{db}
    " if ($normalyzed_info{$k}{samples}{$d}{db}); + $details .= "User: $normalyzed_info{$k}{samples}{$d}{user}
    " if ($normalyzed_info{$k}{samples}{$d}{user}); + $details .= "Remote: $normalyzed_info{$k}{samples}{$d}{remote}
    " if ($normalyzed_info{$k}{samples}{$d}{remote}); + $details .= "Application: $normalyzed_info{$k}{samples}{$d}{app}
    " if ($normalyzed_info{$k}{samples}{$d}{app}); + print $fh qq{ +
    $query
    + 
    $details
    +}; + } - print $fh "
    "; + print $fh qq{ +
    +

    +
    + +
    \n"; + print $fh qq{ + + + + +}; + @top_temporary = (); + } - # Top queries generating the largest temporary files - @top_tempfile_info = sort {$b->[1] <=> $a->[1]} @top_tempfile_info; + # Top queries generating the largest temporary files + if ($#top_tempfile_info >= 0) { + @top_tempfile_info = sort {$b->[0] <=> $a->[0]} @top_tempfile_info; + my $largest = &comma_numbers($top_temporary[0]->[0]); + print $fh qq{ +
    +

    Queries generating the largest temporary files

    +
    + + + + + + + + + +}; + my $rank = 1; + for (my $i = 0 ; $i <= $#top_tempfile_info ; $i++) { + my $size = &pretty_print_size($top_tempfile_info[$i]->[0]); + my $details = "[ Date: $top_tempfile_info[$i]->[1]"; + $details .= " - Database: $top_tempfile_info[$i]->[3]" if ($top_tempfile_info[$i]->[3]); + $details .= " - User: $top_tempfile_info[$i]->[4]" if ($top_tempfile_info[$i]->[4]); + $details .= " - Remote: $top_tempfile_info[$i]->[5]" if ($top_tempfile_info[$i]->[5]); + $details .= " - Application: $top_tempfile_info[$i]->[6]" if ($top_tempfile_info[$i]->[6]); + $details .= " ]"; + my $query = &highlight_code($top_tempfile_info[$i]->[2]); + print $fh qq{ + + + + + +}; + $rank++; + } + print $fh qq{ + +
    RankSizeQuery
    $rank$size +
    $query
    + 
    $details
    +
    +
    +
    +}; + @top_tempfile_info = (); + } + +} + +sub print_histogram_query_times +{ + my %data = (); + my $histogram_info = ''; + my $most_range = ''; + my $most_range_value = ''; + + for (my $i = 1; $i <= $#histogram_query_time; $i++) { + $histogram_info .= "$histogram_query_time[$i-1]-$histogram_query_time[$i]ms" . &comma_numbers($overall_stat{histogram}{query_time}{$histogram_query_time[$i-1]}) . + "" . sprintf("%0.2f", ($overall_stat{histogram}{query_time}{$histogram_query_time[$i-1]} * 100) / ($overall_stat{histogram}{total}||1)) . "%"; + $data{"$histogram_query_time[$i-1]-$histogram_query_time[$i]ms"} = $overall_stat{histogram}{query_time}{$histogram_query_time[$i-1]} if ($overall_stat{histogram}{query_time}{$histogram_query_time[$i-1]} > 0); + if ($overall_stat{histogram}{query_time}{$histogram_query_time[$i-1]} > $most_range_value) { + $most_range = "$histogram_query_time[$i-1]-$histogram_query_time[$i]ms"; + $most_range_value = $overall_stat{histogram}{query_time}{$histogram_query_time[$i-1]}; + } + } + if ($overall_stat{histogram}{total} > 0) { + $histogram_info .= " > $histogram_query_time[-1]ms" . &comma_numbers($overall_stat{histogram}{query_time}{'-1'}) . + "" . sprintf("%0.2f", ($overall_stat{histogram}{query_time}{'-1'} * 100) / ($overall_stat{histogram}{total}||1)) . "%"; + $data{"> $histogram_query_time[-1]ms"} = $overall_stat{histogram}{query_time}{"-1"} if ($overall_stat{histogram}{query_time}{"-1"} > 0); + if ($overall_stat{histogram}{query_time}{"-1"} > $most_range_value) { + $most_range = "> $histogram_query_time[-1]ms"; + $most_range_value = $overall_stat{histogram}{query_time}{"-1"}; + } + } else { + $histogram_info = qq{$NODATA}; + } + + $drawn_graphs{histogram_query_times_graph} = &flotr2_piegraph($graphid++, 'histogram_query_times_graph', 'Histogram of query times', %data); + + $most_range_value = &comma_numbers($most_range_value) if ($most_range_value); + + print $fh qq{ +

    Top Queries

    +
    +

    Histogram of query times

    +
    +

    Key values

    +
    +
      +
    • $most_range_value $most_range duration
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{histogram_query_times_graph} +
    +
    + + + + + + + + + + $histogram_info + +
    RangeCountPercentage
    +
    +
    +
    +
    +
    +}; + delete $drawn_graphs{histogram_query_times_graph}; +} + +sub print_slowest_individual_queries +{ + + print $fh qq{ +
    +

    Slowest individual queries

    +
    + + + + + + + + + +}; + + for (my $i = 0 ; $i <= $#top_slowest ; $i++) { + my $rank = $i + 1; + my $duration = &convert_time($top_slowest[$i]->[0]); + my $date = $top_slowest[$i]->[1] || ''; + my $details = "[ Date: " . ($top_slowest[$i]->[1] || ''); + $details .= " - Database: $top_slowest[$i]->[3]" if ($top_slowest[$i]->[3]); + $details .= " - User: $top_slowest[$i]->[4]" if ($top_slowest[$i]->[4]); + $details .= " - Remote: $top_slowest[$i]->[5]" if ($top_slowest[$i]->[5]); + $details .= " - Application: $top_slowest[$i]->[6]" if ($top_slowest[$i]->[6]); + $details .= " ]"; + my $query = &highlight_code($top_slowest[$i]->[2]); + print $fh qq{ + + + + + + + }; + } + if ($#top_slowest == -1) { + print $fh qq{}; + } + print $fh qq{ + +
    RankDurationQuery
    $rank$duration +
    $query
    + 
    $details
    +
    $NODATA
    +
    +
    +}; + +} + +sub print_time_consuming +{ + print $fh qq{ +
    +

    Time consuming queries

    +
    + + + + + + + + + + + + + +}; + my $rank = 1; + foreach my $k (sort {$normalyzed_info{$b}{duration} <=> $normalyzed_info{$a}{duration}} keys %normalyzed_info) { + next if (!$normalyzed_info{$k}{count}); + last if ($rank > $top); + $normalyzed_info{$k}{average} = $normalyzed_info{$k}{duration} / $normalyzed_info{$k}{count}; + my $duration = &convert_time($normalyzed_info{$k}{duration}); + my $count = &comma_numbers($normalyzed_info{$k}{count}); + my $min = &convert_time($normalyzed_info{$k}{min}); + my $max = &convert_time($normalyzed_info{$k}{max}); + my $avg = &convert_time($normalyzed_info{$k}{average}); + my $query = &highlight_code($k); + my $details = ''; + my %hourly_count = (); + my %hourly_duration = (); + my $days = 0; + foreach my $d (sort keys %{$normalyzed_info{$k}{chronos}}) { + $d =~ /^(\d{4})(\d{2})(\d{2})$/; + $days++; + my $zday = "$abbr_month{$2} $3"; + foreach my $h (sort keys %{$normalyzed_info{$k}{chronos}{$d}}) { + $normalyzed_info{$k}{chronos}{$d}{$h}{average} = + $normalyzed_info{$k}{chronos}{$d}{$h}{duration} / ($normalyzed_info{$k}{chronos}{$d}{$h}{count} || 1); + $hourly_count{"$h"} += $normalyzed_info{$k}{chronos}{$d}{$h}{count}; + $hourly_duration{"$h"} += $normalyzed_info{$k}{chronos}{$d}{$h}{duration}; + $details .= ""; + $zday = ""; + } + } + # Set graph dataset + my %graph_data = (); + foreach my $h ("00" .. "23") { + $graph_data{count} .= "[$h, " . (int($hourly_count{"$h"}/$days) || 0) . "],"; + $graph_data{duration} .= "[$h, " . (int($hourly_duration{"$h"} / ($hourly_count{"$h"} || 1)) || 0) . "],"; + } + $graph_data{count} =~ s/,$//; + $graph_data{duration} =~ s/,$//; + %hourly_count = (); + %hourly_duration = (); + + my $query_histo = + &flotr2_histograph($graphid++, 'timeconsuming_graph_'.$rank, $graph_data{count}, $graph_data{duration}); + + print $fh qq{ + + + + + + + + + +}; + $rank++; + } + + if (scalar keys %normalyzed_info == 0) { + print $fh qq{}; + } + print $fh qq{ + +
    RankTotal durationTimes executedMin durationMax durationAvg durationQuery
    $zday$h" . + &comma_numbers($normalyzed_info{$k}{chronos}{$d}{$h}{count}) . "" . + &convert_time($normalyzed_info{$k}{chronos}{$d}{$h}{duration}) . "" . + &convert_time($normalyzed_info{$k}{chronos}{$d}{$h}{average}) . "
    $rank$duration$count +

    Details

    +     		$min$max$avg +
    $query
    + +
    +

    Times Reported Time consuming queries #$rank

    + $query_histo + + + + + + + + + + + + $details + +
    DayHourCountDurationAvg duration
    +

    +
    +

    + +
    +
    +}; + + foreach my $d (sort {$b <=> $a} keys %{$normalyzed_info{$k}{samples}}) { + my $details = "[ Date: $normalyzed_info{$k}{samples}{$d}{date}"; + $details .= " - Duration: " . &convert_time($d); + $details .= " - Database: $normalyzed_info{$k}{samples}{$d}{details}" if ($normalyzed_info{$k}{samples}{$d}{details}); + $details .= " - User: $normalyzed_info{$k}{samples}{$d}{user}" if ($normalyzed_info{$k}{samples}{$d}{user}); + $details .= " - Remote: $normalyzed_info{$k}{samples}{$d}{remote}" if ($normalyzed_info{$k}{samples}{$d}{remote}); + $details .= " - Application: $normalyzed_info{$k}{samples}{$d}{app}" if ($normalyzed_info{$k}{samples}{$d}{app}); + $details .= " ]"; + $query = &highlight_code($normalyzed_info{$k}{samples}{$d}{query}); + print $fh qq{ +
    $query
    + 
    $details
    +}; + } + print $fh qq{ +
    +

    +
    + +
    $NODATA
    +
    +
    +}; + +} + +sub print_most_frequent +{ + print $fh qq{ +
    +

    Most frequent queries (N)

    +
    + + + + + + + + + + + + + +}; + my $rank = 1; + foreach my $k (sort {$normalyzed_info{$b}{count} <=> $normalyzed_info{$a}{count}} keys %normalyzed_info) { + next if (!$normalyzed_info{$k}{count}); + last if ($rank > $top); + $normalyzed_info{$k}{average} = $normalyzed_info{$k}{duration} / $normalyzed_info{$k}{count}; + my $duration = &convert_time($normalyzed_info{$k}{duration}); + my $count = &comma_numbers($normalyzed_info{$k}{count}); + my $min = &convert_time($normalyzed_info{$k}{min}); + my $max = &convert_time($normalyzed_info{$k}{max}); + my $avg = &convert_time($normalyzed_info{$k}{average}); + my $query = &highlight_code($k); + my %hourly_count = (); + my %hourly_duration = (); + my $days = 0; + my $details = ''; + foreach my $d (sort keys %{$normalyzed_info{$k}{chronos}}) { + $d =~ /^\d{4}(\d{2})(\d{2})$/; + $days++; + my $zday = "$abbr_month{$1} $2"; + foreach my $h (sort keys %{$normalyzed_info{$k}{chronos}{$d}}) { + $normalyzed_info{$k}{chronos}{$d}{$h}{average} = + $normalyzed_info{$k}{chronos}{$d}{$h}{duration} / $normalyzed_info{$k}{chronos}{$d}{$h}{count}; + $hourly_count{"$h"} += $normalyzed_info{$k}{chronos}{$d}{$h}{count}; + $hourly_duration{"$h"} += $normalyzed_info{$k}{chronos}{$d}{$h}{duration}; + $details .= ""; + $zday = ""; + } + } + # Set graph dataset + my %graph_data = (); + foreach my $h ("00" .. "23") { + $graph_data{count} .= "[$h, " . (int($hourly_count{"$h"}/$days) || 0) . "],"; + $graph_data{duration} .= "[$h, " . (int($hourly_duration{"$h"} / ($hourly_count{"$h"} || 1)) || 0) . "],"; + } + $graph_data{count} =~ s/,$//; + $graph_data{duration} =~ s/,$//; + %hourly_count = (); + %hourly_duration = (); + + my $query_histo = + &flotr2_histograph($graphid++, 'mostfrequent_graph_'.$rank, $graph_data{count}, $graph_data{duration}); + + print $fh qq{ + + + + + + + + + +}; + $rank++; + } + if (scalar keys %normalyzed_info == 0) { + print $fh qq{}; + } + print $fh qq{ + +
    RankTimes executedTotal durationMin durationMax durationAvg durationQuery
    $zday$h" . + &comma_numbers($normalyzed_info{$k}{chronos}{$d}{$h}{count}) . "" . + &convert_time($normalyzed_info{$k}{chronos}{$d}{$h}{duration}) . "" . + &convert_time($normalyzed_info{$k}{chronos}{$d}{$h}{average}) . "
    $rank$count +

    Details

    +     		$duration$min$max$avg +
    $query
    + +
    +

    Times Reported Time consuming queries #$rank

    + $query_histo + + + + + + + + + + + + $details + +
    DayHourCountDurationAvg duration
    +

    +
    +

    + +
    +
    +}; + + foreach my $d (sort {$b <=> $a} keys %{$normalyzed_info{$k}{samples}}) { + my $details = "[ Date: $normalyzed_info{$k}{samples}{$d}{date}"; + $details .= " - Duration: " . &convert_time($d); + $details .= " - Database: $normalyzed_info{$k}{samples}{$d}{details}" if ($normalyzed_info{$k}{samples}{$d}{details}); + $details .= " - User: $normalyzed_info{$k}{samples}{$d}{user}" if ($normalyzed_info{$k}{samples}{$d}{user}); + $details .= " - Remote: $normalyzed_info{$k}{samples}{$d}{remote}" if ($normalyzed_info{$k}{samples}{$d}{remote}); + $details .= " - Application: $normalyzed_info{$k}{samples}{$d}{app}" if ($normalyzed_info{$k}{samples}{$d}{app}); + $details .= " ]"; + $query = &highlight_code($normalyzed_info{$k}{samples}{$d}{query}); + print $fh qq{ +
    $query
    + 
    $details
    +}; + } + print $fh qq{ +
    +

    +
    + +
    $NODATA
    +
    +
    +}; + +} + + +sub print_slowest_queries +{ + print $fh qq{ +
    +

    Normalized slowest queries (N)

    +
    + + + + + + + + + + + + + +}; + my $rank = 1; + foreach my $k (sort {$normalyzed_info{$b}{average} <=> $normalyzed_info{$a}{average}} keys %normalyzed_info) { + next if (!$k || !$normalyzed_info{$k}{count}); + last if ($rank > $top); + $normalyzed_info{$k}{average} = $normalyzed_info{$k}{duration} / $normalyzed_info{$k}{count}; + my $duration = &convert_time($normalyzed_info{$k}{duration}); + my $count = &comma_numbers($normalyzed_info{$k}{count}); + my $min = &convert_time($normalyzed_info{$k}{min}); + my $max = &convert_time($normalyzed_info{$k}{max}); + my $avg = &convert_time($normalyzed_info{$k}{average}); + my $query = &highlight_code($k); + my $details = ''; + my %hourly_count = (); + my %hourly_duration = (); + my $days = 0; + foreach my $d (sort keys %{$normalyzed_info{$k}{chronos}}) { + my $c = 1; + $d =~ /^\d{4}(\d{2})(\d{2})$/; + $days++; + my $zday = "$abbr_month{$1} $2"; + foreach my $h (sort keys %{$normalyzed_info{$k}{chronos}{$d}}) { + $normalyzed_info{$k}{chronos}{$d}{$h}{average} = + $normalyzed_info{$k}{chronos}{$d}{$h}{duration} / $normalyzed_info{$k}{chronos}{$d}{$h}{count}; + $hourly_count{"$h"} += $normalyzed_info{$k}{chronos}{$d}{$h}{count}; + $hourly_duration{"$h"} += $normalyzed_info{$k}{chronos}{$d}{$h}{duration}; + $details .= ""; + $zday = ""; + } + } + # Set graph dataset + my %graph_data = (); + foreach my $h ("00" .. "23") { + $graph_data{count} .= "[$h, " . (int($hourly_count{"$h"}/$days) || 0) . "],"; + $graph_data{duration} .= "[$h, " . (int($hourly_duration{"$h"} / ($hourly_count{"$h"} || 1)) || 0) . "],"; + } + $graph_data{count} =~ s/,$//; + $graph_data{duration} =~ s/,$//; + %hourly_count = (); + %hourly_duration = (); + + my $query_histo = + &flotr2_histograph($graphid++, 'normalizedslowest_graph_'.$rank, $graph_data{count}, $graph_data{duration}); + + print $fh qq{ + + + + + + + + + +}; + $rank++; + } + if (scalar keys %normalyzed_info == 0) { + print $fh qq{}; + } + print $fh qq{ + +
    RankMin durationMax durationAvg durationTimes executedTotal durationQuery
    $zday$h" . + &comma_numbers($normalyzed_info{$k}{chronos}{$d}{$h}{count}) . "" . + &convert_time($normalyzed_info{$k}{chronos}{$d}{$h}{duration}) . "" . + &convert_time($normalyzed_info{$k}{chronos}{$d}{$h}{average}) . "
    $rank$min$max$avg$count +

    Details

    +     		$duration +
    $query
    + +
    +

    Times Reported Time consuming queries #$rank

    + $query_histo + + + + + + + + + + + + $details + +
    DayHourCountDurationAvg duration
    +

    +
    +

    + +
    +
    +}; + + foreach my $d (sort {$b <=> $a} keys %{$normalyzed_info{$k}{samples}}) { + my $details = "[ Date: $normalyzed_info{$k}{samples}{$d}{date}"; + $details .= " - Duration: " . &convert_time($d); + $details .= " - Database: $normalyzed_info{$k}{samples}{$d}{details}" if ($normalyzed_info{$k}{samples}{$d}{details}); + $details .= " - User: $normalyzed_info{$k}{samples}{$d}{user}" if ($normalyzed_info{$k}{samples}{$d}{user}); + $details .= " - Remote: $normalyzed_info{$k}{samples}{$d}{remote}" if ($normalyzed_info{$k}{samples}{$d}{remote}); + $details .= " - Application: $normalyzed_info{$k}{samples}{$d}{app}" if ($normalyzed_info{$k}{samples}{$d}{app}); + $details .= " ]"; + $query = &highlight_code($normalyzed_info{$k}{samples}{$d}{query}); + print $fh qq{ +
    $query
    + 
    $details
    +}; + } + print $fh qq{ +
    +

    +
    + +
    $NODATA
    +
    +
    +}; + +} + +sub dump_as_html +{ + + # Dump the html header + &html_header(); + + if (!$error_only) { + + # Overall statistics + print $fh qq{ +
  • +}; + &print_overall_statistics(); + + # Set graphs limits + $overall_stat{'first_log_ts'} =~ /(\d+)-(\d+)-(\d+) (\d+):(\d+):(\d+)/; + $t_min = timegm_nocheck(0, $5, $4, $3, $2 - 1, $1) * 1000; + $t_min -= ($avg_minutes * 60000); + + $overall_stat{'last_log_ts'} =~ /(\d+)-(\d+)-(\d+) (\d+):(\d+):(\d+)/; + $t_max = timegm_nocheck(59, $5, $4, $3, $2 - 1, $1) * 1000; + $t_max += ($avg_minutes * 60000); + + if (!$disable_hourly) { + # Build graphs based on hourly stat + &compute_query_graphs(); + + # Show global SQL traffic + &print_sql_traffic(); + + # Show hourly statistics + &print_general_activity(); + + } + + if (!$disable_connection) { + print $fh qq{ +
    • +
  • +

    Connections

    + +}; + + # Draw connections information + &print_established_connection() if (!$disable_hourly); + + # Show per database/user connections + &print_database_connection() if (exists $connection_info{database}); + + # Show per user connections + &print_user_connection() if (exists $connection_info{user}); + + # Show per client ip connections + &print_host_connection() if (exists $connection_info{host}); + } + + + # Show session per database statistics + if (!$disable_session) { + print $fh qq{ +
    • +
  • +

    Sessions

    - print $fh qq{ -

    Queries generating the largest temporary files^

    - - - - - - }; - for (my $i = 0 ; $i <= $#top_tempfile_info ; $i++) { - my $col = $i % 2; - my $ttl = $top_tempfile_info[$i]->[1] || ''; - my $db = " - database: $top_tempfile_info[$i]->[3]" if ($top_tempfile_info[$i]->[3]); - $db .= ", user: $top_tempfile_info[$i]->[4]" if ($top_tempfile_info[$i]->[4]); - $db .= ", remote: $top_tempfile_info[$i]->[5]" if ($top_tempfile_info[$i]->[5]); - $db .= ", app: $top_tempfile_info[$i]->[6]" if ($top_tempfile_info[$i]->[6]); - $db =~ s/^, / - /; - print $fh "\n"; + # Show number of simultaneous sessions + &print_simultaneous_session(); + # Show per database sessions + &print_database_session(); + # Show per user sessions + &print_user_session(); + # Show per host sessions + &print_host_session(); } - print $fh "
    RankSizeQuery
    ", $i + 1, "", - &comma_numbers($top_tempfile_info[$i]->[0]), - "
    ", - &highlight_code($top_tempfile_info[$i]->[2]), "
    \n"; - } - # Show top information - if (!$disable_query && ($#top_slowest >= 0)) { - print $fh qq{ -

    Slowest queries ^

    - - - - - - - -}; - for (my $i = 0 ; $i <= $#top_slowest ; $i++) { - my $col = $i % 2; - my $ttl = $top_slowest[$i]->[1] || ''; - my $db = " - database: $top_slowest[$i]->[3]" if ($top_slowest[$i]->[3]); - $db .= ", user: $top_slowest[$i]->[4]" if ($top_slowest[$i]->[4]); - $db .= ", remote: $top_slowest[$i]->[5]" if ($top_slowest[$i]->[5]); - $db .= ", app: $top_slowest[$i]->[6]" if ($top_slowest[$i]->[6]); - $db =~ s/^, / - /; - print $fh "\n"; + + # Display checkpoint and temporary files report + if (!$disable_checkpoint) { + print $fh qq{ + +
  • + }; + &print_checkpoint(); } - print $fh "
    • RankDuration (s)Query
    ", $i + 1, "", - &convert_time($top_slowest[$i]->[0]), - "
    ", - &highlight_code($top_slowest[$i]->[2]), "
    \n"; - print $fh qq{ -

    Queries that took up the most time (N) ^

    - - - - - - - - - + if (!$disable_temporary) { + print $fh qq{ + +
  • }; - my $idx = 1; - foreach my $k (sort {$normalyzed_info{$b}{duration} <=> $normalyzed_info{$a}{duration}} keys %normalyzed_info) { - next if (!$normalyzed_info{$k}{count}); - last if ($idx > $top); - my $q = $k; - if ($normalyzed_info{$k}{count} == 1) { - foreach my $d (sort {$b <=> $a} keys %{$normalyzed_info{$k}{samples}}) { - $q = $normalyzed_info{$k}{samples}{$d}{query}; - last; - } - } - $normalyzed_info{$k}{average} = $normalyzed_info{$k}{duration} / $normalyzed_info{$k}{count}; - my $col = $idx % 2; - print $fh "
    • "; - print $fh "\n"; - $idx++; + # Show information about queries generating temporary files + &print_tempfile_report(); } - print $fh "
    RankTotal durationTimes executedMin/Max/Avg duration (s)Query
    $idx", - &convert_time($normalyzed_info{$k}{duration}), - "
    ", - &comma_numbers($normalyzed_info{$k}{count}), -"
    "; - foreach my $d (sort keys %{$normalyzed_info{$k}{chronos}}) { - my $c = 1; - $d =~ /^\d{4}(\d{2})(\d{2})$/; - my $zday = "$abbr_month{$1} $2"; - foreach my $h (sort keys %{$normalyzed_info{$k}{chronos}{$d}}) { - $normalyzed_info{$k}{chronos}{$d}{$h}{average} = - $normalyzed_info{$k}{chronos}{$d}{$h}{duration} / $normalyzed_info{$k}{chronos}{$d}{$h}{count}; - my $colb = $c % 2; - $zday = " " if ($c > 1); - print $fh ""; - $c++; - } - } - print $fh "
    DayHourCountDurationAvg Duration
    $zday$h", - &comma_numbers($normalyzed_info{$k}{chronos}{$d}{$h}{count}), "", - &convert_time($normalyzed_info{$k}{chronos}{$d}{$h}{duration}), "", - &convert_time($normalyzed_info{$k}{chronos}{$d}{$h}{average}), "
    		", &convert_time($normalyzed_info{$k}{min}),"/", &convert_time($normalyzed_info{$k}{max}),"/", &convert_time($normalyzed_info{$k}{average}), - "
    ", - &highlight_code($q), "
    "; + # Show temporary files detailed information + &print_temporary_file(); - if ($normalyzed_info{$k}{count} > 1) { - print $fh -"
    "; - my $i = 0; - foreach my $d (sort {$b <=> $a} keys %{$normalyzed_info{$k}{samples}}) { - my $colb = $i % 2; - my $db = " - database: $normalyzed_info{$k}{samples}{$d}{db}" if ($normalyzed_info{$k}{samples}{$d}{db}); - $db .= ", user: $normalyzed_info{$k}{samples}{$d}{user}" if ($normalyzed_info{$k}{samples}{$d}{user}); - $db .= ", remote: $normalyzed_info{$k}{samples}{$d}{remote}" if ($normalyzed_info{$k}{samples}{$d}{remote}); - $db .= ", app: $normalyzed_info{$k}{samples}{$d}{app}" if ($normalyzed_info{$k}{samples}{$d}{app}); - $db =~ s/^, / - /; - print $fh -"
    ", - &convert_time($d), " | ", &highlight_code($normalyzed_info{$k}{samples}{$d}{query}), "
    "; - $i++; - } - print $fh "
    "; - } - print $fh "
    \n"; - } - if (!$disable_query && (scalar keys %normalyzed_info > 0)) { + if (!$disable_autovacuum) { + print $fh qq{ +
    • +
  • +}; + # Show detailed vacuum/analyse information + &print_vacuum(); - print $fh qq{ -

    Most frequent queries (N) ^

    - - - - - - - - + } + + if (!$disable_lock) { + print $fh qq{ + +
  • }; - my $idx = 1; - foreach my $k (sort {$normalyzed_info{$b}{count} <=> $normalyzed_info{$a}{count}} keys %normalyzed_info) { - next if (!$normalyzed_info{$k}{count}); - last if ($idx > $top); - my $q = $k; - if ($normalyzed_info{$k}{count} == 1) { - foreach my $d (sort {$b <=> $a} keys %{$normalyzed_info{$k}{samples}}) { - $q = $normalyzed_info{$k}{samples}{$d}{query}; - last; - } - } - my $col = $idx % 2; - print $fh -"
    • "; - print $fh "\n"; - $idx++; + # Show lock wait detailed information + &print_lock_queries_report(); } - print $fh "
    RankTimes executedTotal durationMin/Max/Avg duration (s)Query
    $idx
    ", - &comma_numbers($normalyzed_info{$k}{count}), -"
    "; - foreach my $d (sort keys %{$normalyzed_info{$k}{chronos}}) { - my $c = 1; - $d =~ /^\d{4}(\d{2})(\d{2})$/; - my $zday = "$abbr_month{$1} $2"; - foreach my $h (sort keys %{$normalyzed_info{$k}{chronos}{$d}}) { - $normalyzed_info{$k}{chronos}{$d}{$h}{average} = - $normalyzed_info{$k}{chronos}{$d}{$h}{duration} / $normalyzed_info{$k}{chronos}{$d}{$h}{count}; - my $colb = $c % 2; - $zday = " " if ($c > 1); - print $fh ""; - $c++; - } - } - print $fh "
    DayHourCountDurationAvg Duration
    $zday$h", - &comma_numbers($normalyzed_info{$k}{chronos}{$d}{$h}{count}), "", - &convert_time($normalyzed_info{$k}{chronos}{$d}{$h}{duration}), "", - &convert_time($normalyzed_info{$k}{chronos}{$d}{$h}{average}), "
    		", &convert_time($normalyzed_info{$k}{duration}), "",&convert_time($normalyzed_info{$k}{min}),"/",&convert_time($normalyzed_info{$k}{max}),"/", - &convert_time($normalyzed_info{$k}{average}), "
    ", - &highlight_code($q), "
    "; + # Lock stats per type + &print_lock_type(); - if ($normalyzed_info{$k}{count} > 1) { - print $fh -"
    "; - my $i = 0; - foreach my $d (sort {$b <=> $a} keys %{$normalyzed_info{$k}{samples}}) { - my $colb = $i % 2; - my $db = " - database: $normalyzed_info{$k}{samples}{$d}{db}" if ($normalyzed_info{$k}{samples}{$d}{db}); - $db .= ", user: $normalyzed_info{$k}{samples}{$d}{user}" if ($normalyzed_info{$k}{samples}{$d}{user}); - $db .= ", remote: $normalyzed_info{$k}{samples}{$d}{remote}" if ($normalyzed_info{$k}{samples}{$d}{remote}); - $db .= ", app: $normalyzed_info{$k}{samples}{$d}{app}" if ($normalyzed_info{$k}{samples}{$d}{app}); - $db =~ s/^, / - /; - print $fh -"
    ", - &convert_time($d), " | ", &highlight_code($normalyzed_info{$k}{samples}{$d}{query}), "
    "; - $i++; - } - print $fh "
    "; - } - print $fh "
    \n"; - } - if (!$disable_query && ($#top_slowest >= 0)) { - print $fh qq{ -

    Slowest queries (N) ^

    - - - - - - - - - + %per_minute_info = (); + + if (!$disable_query) { + print $fh qq{ + +
  • }; - my $idx = 1; - foreach my $k (sort {$normalyzed_info{$b}{average} <=> $normalyzed_info{$a}{average}} keys %normalyzed_info) { - next if (!$k || !$normalyzed_info{$k}{count}); - last if ($idx > $top); - my $q = $k; - if ($normalyzed_info{$k}{count} == 1) { - foreach my $d (sort {$b <=> $a} keys %{$normalyzed_info{$k}{samples}}) { - $q = $normalyzed_info{$k}{samples}{$d}{query}; - last; - } - } - my $col = $idx % 2; - print $fh "
    • "; - print $fh "\n"; - $idx++; - } - print $fh "
    RankMin/Max/Avg duration (s)Times executedTotal durationQuery
    $idx", - &convert_time($normalyzed_info{$k}{min}), "/", - &convert_time($normalyzed_info{$k}{max}), "/", - &convert_time($normalyzed_info{$k}{average}), - "
    ", - &comma_numbers($normalyzed_info{$k}{count}), -"
    "; - foreach my $d (sort keys %{$normalyzed_info{$k}{chronos}}) { - my $c = 1; - $d =~ /^\d{4}(\d{2})(\d{2})$/; - my $zday = "$abbr_month{$1} $2"; - foreach my $h (sort keys %{$normalyzed_info{$k}{chronos}{$d}}) { - $normalyzed_info{$k}{chronos}{$d}{$h}{average} = - $normalyzed_info{$k}{chronos}{$d}{$h}{duration} / $normalyzed_info{$k}{chronos}{$d}{$h}{count}; - my $colb = $c % 2; - $zday = " " if ($c > 1); - print $fh ""; - $c++; - } - } - print $fh "
    DayHourCountDurationAvg Duration
    $zday$h", - &comma_numbers($normalyzed_info{$k}{chronos}{$d}{$h}{count}), "", - &convert_time($normalyzed_info{$k}{chronos}{$d}{$h}{duration}), "", - &convert_time($normalyzed_info{$k}{chronos}{$d}{$h}{average}), "
    		", &convert_time($normalyzed_info{$k}{duration}), - "
    ", - &highlight_code($q), "
    "; - if ($normalyzed_info{$k}{count} > 1) { - print $fh -"
    "; - my $i = 0; - foreach my $d (sort {$b <=> $a} keys %{$normalyzed_info{$k}{samples}}) { - my $colb = $i % 2; - my $db = " - database: $normalyzed_info{$k}{samples}{$d}{db}" if ($normalyzed_info{$k}{samples}{$d}{db}); - $db .= ", user: $normalyzed_info{$k}{samples}{$d}{user}" if ($normalyzed_info{$k}{samples}{$d}{user}); - $db .= ", remote: $normalyzed_info{$k}{samples}{$d}{remote}" if ($normalyzed_info{$k}{samples}{$d}{remote}); - $db .= ", app: $normalyzed_info{$k}{samples}{$d}{app}" if ($normalyzed_info{$k}{samples}{$d}{app}); - $db =~ s/^, / - /; - print $fh -"
    ", - &convert_time($d), " | ", &highlight_code($normalyzed_info{$k}{samples}{$d}{query}), "
    "; - $i++; - } - print $fh "
    "; + # INSERT/DELETE/UPDATE/SELECT repartition + if (!$disable_type) { + &print_query_type(); + + # Show requests per database + &print_query_per_database(); + + # Show requests per user + &print_query_per_user(); + + # Show requests per host + &print_query_per_host(); + + # Show requests per application + &print_query_per_application(); } - print $fh "
    \n"; - } - if (!$disable_error) { - &show_error_as_html(); - } + print $fh qq{ +
    • +
  • +}; + # Show histogram for query times + &print_histogram_query_times(); - # Dump the html footer - &html_footer(); + # Show top information + &print_slowest_individual_queries(); -} + # Show queries that took up the most time + &print_time_consuming(); -sub dump_error_as_html -{ + # Show most frequent queries + &print_most_frequent(); - # Dump the html header - &html_header(); + # Print normalized slowest queries + &print_slowest_queries + } - # Global information - my $curdate = localtime(time); - my $fmt_nlines = &comma_numbers($nlines); - my $total_time = timestr($td); - $total_time =~ s/^([\.0-9]+) wallclock.*/$1/; - $total_time = &convert_time($total_time * 1000); - my $logfile_str = $log_files[0]; - if ($#log_files > 0) { - $logfile_str .= ', ..., ' . $log_files[-1]; } - print $fh qq{ -
    -
      -
    • Generated on $curdate
      • -
    • Log file: $logfile_str
      • -
    • Parsed $fmt_nlines log entries in $total_time
      • -
    • Log start from $overall_stat{'first_log_ts'} to $overall_stat{'last_log_ts'}
      • -
    -
    + + # Show errors report + if (!$disable_error) { + + if (!$error_only) { + print $fh qq{ +
    • +
  • }; - my $fmt_errors = &comma_numbers($overall_stat{'errors_number'}) || 0; - my $fmt_unique_error = &comma_numbers(scalar keys %{$overall_stat{'unique_normalized_errors'}}) || 0; - print $fh qq{ -
    -

    Overall statistics ^

    -
    -
    -
      -
    • Number of events: $fmt_errors
      • -
    • Number of unique normalized events: $fmt_unique_error
      • -
    -
    -
    + } else { + print $fh qq{ +
  • }; + } + # Show log level distribution + &print_log_level(); + + # Show Most Frequent Errors/Events + &show_error_as_html(); + } - &show_error_as_html(); # Dump the html footer &html_footer(); + } -sub show_error_as_html +sub escape_html { + $_[0] =~ s/<([\/a-zA-Z][\s\t\>]*)/\<$1/sg; - return if (scalar keys %error_info == 0); + return $_[0]; +} - print $fh qq{ -

    Most frequent events (N) ^

    - - - - - +sub print_log_level +{ + my %infos = (); - -}; - my $idx = 1; + # Some messages have seen their log level change during log parsing. + # Set the real log level count back foreach my $k (sort {$error_info{$b}{count} <=> $error_info{$a}{count}} keys %error_info) { next if (!$error_info{$k}{count}); - last if ($idx > $top); - my $col = $idx % 2; - print $fh -"\n"; if ($error_info{$k}{count} > 1) { - my $msg = $k; - $msg =~ s/HINT: (parameter "[^"]+" changed to)/LOG: $1/; - $msg =~ s/HINT: (database system was shut down)/LOG: $1/; - print $fh "\n"; - $idx++; } - print $fh "
    RankTimes reportedError
    $idx
    ", - &comma_numbers($error_info{$k}{count}), ""; - print $fh "
    "; - foreach my $d (sort keys %{$error_info{$k}{chronos}}) { - my $c = 1; - $d =~ /^\d{4}(\d{2})(\d{2})$/; - my $zday = "$abbr_month{$1} $2"; - foreach my $h (sort keys %{$error_info{$k}{chronos}{$d}}) { - my $colb = $c % 2; - $zday = " " if ($c > 1); - print $fh ""; - $c++; - } - } - print $fh "
    DayHourCount
    $zday$h", - &comma_numbers($error_info{$k}{chronos}{$d}{$h}{count}), "
    $msg
    "; - print $fh -"
    "; for (my $i = 0 ; $i <= $#{$error_info{$k}{date}} ; $i++) { - if ( ($error_info{$k}{error}[$i] =~ s/HINT: (parameter "[^"]+" changed to)/LOG: $1/) - || ($error_info{$k}{error}[$i] =~ s/HINT: (database system was shut down)/LOG: $1/)) + if ( ($error_info{$k}{error}[$i] =~ s/ERROR: (parameter "[^"]+" changed to)/LOG: $1/) + || ($error_info{$k}{error}[$i] =~ s/ERROR: (database system was shut down)/LOG: $1/) + || ($error_info{$k}{error}[$i] =~ s/ERROR: (database system was interrupted while in recovery)/LOG: $1/) + || ($error_info{$k}{error}[$i] =~ s/ERROR: (recovery has paused)/LOG: $1/)) { - $logs_type{HINT}--; + $logs_type{ERROR}--; $logs_type{LOG}++; } - my $c = $i % 2; - print $fh "
    $error_info{$k}{error}[$i]
    \n"; - print $fh "
    Detail: $error_info{$k}{detail}[$i]
    \n" - if ($error_info{$k}{detail}[$i]); - print $fh "
    Context: $error_info{$k}{context}[$i]
    \n" - if ($error_info{$k}{context}[$i]); - print $fh "
    Hint: $error_info{$k}{hint}[$i]
    \n" if ($error_info{$k}{hint}[$i]); - print $fh "
    Statement: $error_info{$k}{statement}[$i]
    \n" - if ($error_info{$k}{statement}[$i]); - print $fh "
    Database: $error_info{$k}{db}[$i]
    \n" if ($error_info{$k}{db}[$i]); } - print $fh "
    "; } else { - if ( ($error_info{$k}{error}[0] =~ s/HINT: (parameter "[^"]+" changed to)/LOG: $1/) - || ($error_info{$k}{error}[0] =~ s/HINT: (database system was shut down)/LOG: $1/)) + if ( ($error_info{$k}{error}[0] =~ s/ERROR: (parameter "[^"]+" changed to)/LOG: $1/) + || ($error_info{$k}{error}[0] =~ s/ERROR: (database system was shut down)/LOG: $1/) + || ($error_info{$k}{error}[0] =~ s/ERROR: (database system was interrupted while in recovery)/LOG: $1/) + || ($error_info{$k}{error}[0] =~ s/ERROR: (recovery has paused)/LOG: $1/)) { - $logs_type{HINT}--; + $logs_type{ERROR}--; $logs_type{LOG}++; } - print $fh "
    $error_info{$k}{error}[0]
    "; - print $fh "
    Detail: $error_info{$k}{detail}[0]
    \n" if ($error_info{$k}{detail}[0]); - print $fh "
    Context: $error_info{$k}{context}[0]
    \n" if ($error_info{$k}{context}[0]); - print $fh "
    Hint: $error_info{$k}{hint}[0]
    \n" if ($error_info{$k}{hint}[0]); - print $fh "
    Statement: $error_info{$k}{statement}[0]
    \n" - if ($error_info{$k}{statement}[0]); - print $fh "
    Database: $error_info{$k}{db}[0]
    \n" if ($error_info{$k}{db}[0]); } - print $fh "
    \n"; - - if (scalar keys %logs_type > 0) { - # Show log types - print $fh qq{ -

    Logs per type ^

    - -
    - - - - - - - }; + # Show log types + my $total_logs = 0; + foreach my $d (sort keys %logs_type) { + $total_logs += $logs_type{$d}; + } - my $total_logs = 0; + my $logtype_info = ''; + foreach my $d (sort keys %logs_type) { + next if (!$logs_type{$d}); + $logtype_info .= ""; + } + if ($graph) { + my @small = (); foreach my $d (sort keys %logs_type) { - $total_logs += $logs_type{$d}; + if ((($logs_type{$d} * 100) / ($total_logs || 1)) > $pie_percentage_limit) { + $infos{$d} = $logs_type{$d} || 0; + } else { + $infos{"Sum log types < $pie_percentage_limit%"} += $logs_type{$d} || 0; + push(@small, $d); + } } - my $c = 0; - - foreach my $d (sort keys %logs_type) { - next if (!$logs_type{$d}); - my $colb = $c % 2; - print $fh "\n"; - $c++; + if ($#small == 0) { + $infos{$small[0]} = $infos{"Sum log types < $pie_percentage_limit%"}; + delete $infos{"Sum log types < $pie_percentage_limit%"}; } + } + $drawn_graphs{logstype_graph} = &flotr2_piegraph($graphid++, 'logstype_graph', 'Logs per type', %infos); + if (!$total_logs) { + $logtype_info = qq{}; + } + $logs_type{ERROR} ||= 0; + $logs_type{FATAL} ||= 0; + $total_logs = &comma_numbers($total_logs); + print $fh qq{ +

    Events

    +
    +

    Log levels

    +
    +

    Key values

    +
    +
      +
    • $total_logs Log entries
      • +
    • $logs_type{ERROR} Number of ERROR entries
      • +
    • $logs_type{FATAL} Number of FATAL entries
      • +
    +
    +
    +
    +
    + +
    +
    + $drawn_graphs{logstype_graph} +
    +
    +
    TypeCountPercentage
    $d" . &comma_numbers($logs_type{$d}) . + "" . sprintf("%0.2f", ($logs_type{$d} * 100) / ($total_logs||1)) . "%
    $d", &comma_numbers($logs_type{$d}), - "", sprintf("%0.2f", ($logs_type{$d} * 100) / $total_logs), "%
    $NODATA
    + + + + + + + + + $logtype_info + +
    TypeCountPercentage
    + + + + + +}; + delete $drawn_graphs{logstype_graph}; - print $fh "
    \n"; - if ($graph && $total_logs) { - my %infos = (); - my @small = (); - foreach my $d (sort keys %logs_type) { - if ((($logs_type{$d} * 100) / $total_logs) > $pie_percentage_limit) { - $infos{$d} = $logs_type{$d} || 0; - } else { - $infos{"Sum log types < $pie_percentage_limit%"} += $logs_type{$d} || 0; - push(@small, $d); - } - } +} + +sub show_error_as_html +{ + + my $main_error = 0; + my $total = 0; + foreach my $k (sort {$error_info{$b}{count} <=> $error_info{$a}{count}} keys %error_info) { + next if (!$error_info{$k}{count}); + $main_error = &comma_numbers($error_info{$k}{count}) if (!$main_error); + $total += $error_info{$k}{count}; + } + $total = &comma_numbers($total); + + print $fh qq{ +
    +

    Most Frequent Errors/Events

    +
    +

    Key values

    +
    +
      +
    • $main_error Max number of times the same event was reported
      • +
    • $total Total events found
      • +
    +
    +
    +
    + + + + + + + + + +}; + my $rank = 1; + foreach my $k (sort {$error_info{$b}{count} <=> $error_info{$a}{count}} keys %error_info) { + next if (!$error_info{$k}{count}); + my $count = &comma_numbers($error_info{$k}{count}); + my $msg = $k; + $msg =~ s/ERROR: (parameter "[^"]+" changed to)/LOG: $1/; + $msg =~ s/ERROR: (database system was shut down)/LOG: $1/; + $msg =~ s/ERROR: (database system was interrupted while in recovery)/LOG: $1/; + $msg =~ s/ERROR: (recovery has paused)/LOG: $1/; + my $error_level_class = 'text-error'; + if ($msg =~ /^WARNING: /) { + $error_level_class = 'text-warning'; + } elsif ($msg =~ /^LOG: /) { + $error_level_class = 'text-success'; + } elsif ($msg =~ /^HINT: /) { + $error_level_class = 'text-info'; + } elsif ($msg =~ /^FATAL: /) { + $error_level_class = 'text-fatal'; + } elsif ($msg =~ /^PANIC: /) { + $error_level_class = 'text-panic'; + } + my $details = ''; + my %hourly_count = (); + my $days = 0; + foreach my $d (sort keys %{$error_info{$k}{chronos}}) { + my $c = 1; + $d =~ /^\d{4}(\d{2})(\d{2})$/; + $days++; + my $zday = "$abbr_month{$1} $2"; + foreach my $h (sort keys %{$error_info{$k}{chronos}{$d}}) { + $details .= ""; + $hourly_count{"$h"} += $error_info{$k}{chronos}{$d}{$h}{count}; + $zday = ""; + } + } + # Set graph dataset + my %graph_data = (); + foreach my $h ("00" .. "23") { + $graph_data{count} .= "[$h, " . (int($hourly_count{"$h"}/$days) || 0) . "],"; + } + $graph_data{count} =~ s/,$//; + %hourly_count = (); + + my $error_histo = + &flotr2_histograph($graphid++, 'error_graph_'.$rank, $graph_data{count}); - if ($#small == 0) { - $infos{$small[0]} = $infos{"Sum log types < $pie_percentage_limit%"}; - delete $infos{"Sum log types < $pie_percentage_limit%"}; + # Escape HTML code in error message + $msg = &escape_html($msg); + print $fh qq{ + + + +
    RankTimes reportedError
    $zday$h" . + &comma_numbers($error_info{$k}{chronos}{$d}{$h}{count}) . "
    $rank$count +

    Details

    +     		+ 
    $msg
    + +
    +

    Times Reported Most Frequent Error / Event #$rank

    + $error_histo + + + + + + + + + + $details + +
    DayHourCount
    +

    +
    +

    + +
    +
    +}; + + for (my $i = 0 ; $i <= $#{$error_info{$k}{date}} ; $i++) { + # Escape HTML code in error message + my $message = &escape_html($error_info{$k}{error}[$i]); + my $details = "Date: " . $error_info{$k}{date}[$i] . "\n"; + if ($error_info{$k}{detail}[$i]) { + $details .= "Detail: " . &escape_html($error_info{$k}{detail}[$i]) . "
    "; + } + if ($error_info{$k}{context}[$i]) { + $details .= "Context: " . &escape_html($error_info{$k}{context}[$i]) . "
    "; + } + if ($error_info{$k}{hint}[$i]) { + $details .= "Hint: " . &escape_html($error_info{$k}{hint}[$i]) . "
    "; + } + if ($error_info{$k}{statement}[$i]) { + $details .= "Statement: " . &escape_html($error_info{$k}{statement}[$i]) . "
    "; + } + if ($error_info{$k}{db}[$i]) { + $details .= "Database: " . $error_info{$k}{db}[$i] . "
    "; } - &flotr2_piegraph(17, 'logstype_graph', 'Logs per type', %infos); + print $fh qq{ +
    $message
    + 
    $details
    +}; } - print $fh "
    \n"; + print $fh qq{ + +

    +
    + + + +}; + $rank++; } + if (scalar keys %error_info == 0) { + print $fh qq{$NODATA}; + } + + print $fh qq{ + + +
    + +}; } @@ -4477,20 +7221,22 @@ { my $fd = shift; + my %stats = %{ fd_retrieve($fd) }; my %_overall_stat = %{$stats{overall_stat}}; + my %_overall_checkpoint = %{$stats{overall_checkpoint}}; my %_normalyzed_info = %{$stats{normalyzed_info}}; my %_error_info = %{$stats{error_info}}; my %_connection_info = %{$stats{connection_info}}; my %_database_info = %{$stats{database_info}}; my %_application_info = %{$stats{application_info}}; + my %_user_info = %{$stats{user_info}}; + my %_host_info = %{$stats{host_info}}; my %_checkpoint_info = %{$stats{checkpoint_info}}; - my %_restartpoint_info = %{$stats{restartpoint_info}}; my %_session_info = %{$stats{session_info}}; my %_tempfile_info = %{$stats{tempfile_info}}; my %_logs_type = %{$stats{logs_type}}; my %_lock_info = %{$stats{lock_info}}; - my %_per_hour_info = %{$stats{per_hour_info}}; my %_per_minute_info = %{$stats{per_minute_info}}; my @_top_slowest = @{$stats{top_slowest}}; my $_nlines = $stats{nlines}; @@ -4499,28 +7245,32 @@ my @_log_files = @{$stats{log_files}}; my %_autovacuum_info = %{$stats{autovacuum_info}}; my %_autoanalyze_info = %{$stats{autoanalyze_info}}; - - return if (!$_overall_stat{queries_number} && !$_overall_stat{'errors_number'}); + my @_top_locked_info = @{$stats{top_locked_info}}; + my @_top_tempfile_info = @{$stats{top_tempfile_info}}; ### overall_stat ### $overall_stat{queries_number} += $_overall_stat{queries_number}; - $overall_stat{'first_log_ts'} = $_overall_stat{'first_log_ts'} - if not $overall_stat{'first_log_ts'} - or $overall_stat{'first_log_ts'} gt $_overall_stat{'first_log_ts'}; + if ($_overall_stat{'first_log_ts'}) { + $overall_stat{'first_log_ts'} = $_overall_stat{'first_log_ts'} + if (!$overall_stat{'first_log_ts'} || + ($overall_stat{'first_log_ts'} gt $_overall_stat{'first_log_ts'})); + } $overall_stat{'last_log_ts'} = $_overall_stat{'last_log_ts'} if not $overall_stat{'last_log_ts'} or $overall_stat{'last_log_ts'} lt $_overall_stat{'last_log_ts'}; - $overall_stat{first_query_ts} = $_overall_stat{first_query_ts} - if not $overall_stat{first_query_ts} - or $overall_stat{first_query_ts} gt $_overall_stat{first_query_ts}; - - $overall_stat{last_query_ts} = $_overall_stat{last_query_ts} - if not $overall_stat{last_query_ts} - or $overall_stat{last_query_ts} lt $_overall_stat{last_query_ts}; + if ($_overall_stat{'first_query_ts'}) { + $overall_stat{'first_query_ts'} = $_overall_stat{'first_query_ts'} + if (!$overall_stat{'first_query_ts'} || + ($overall_stat{'first_query_ts'} gt $_overall_stat{'first_query_ts'})); + } + + $overall_stat{'last_query_ts'} = $_overall_stat{'last_query_ts'} + if not $overall_stat{'last_query_ts'} + or $overall_stat{'last_query_ts'} lt $_overall_stat{'last_query_ts'}; $overall_stat{errors_number} += $_overall_stat{errors_number}; $overall_stat{queries_duration} += $_overall_stat{queries_duration}; @@ -4534,21 +7284,40 @@ $overall_stat{SELECT} += $_overall_stat{SELECT} if exists $_overall_stat{SELECT}; - foreach my $k (keys %{$_overall_stat{query_peak}}) { - $overall_stat{query_peak}{$k} += $_overall_stat{query_peak}{$k}; - } - - # FIXME == $error_info ?? - foreach my $k (keys %{$_overall_stat{unique_normalized_errors}}) { - $overall_stat{unique_normalized_errors}{$k} += $_overall_stat{unique_normalized_errors}{$k}; + $overall_checkpoint{checkpoint_warning} += $_overall_checkpoint{checkpoint_warning}; + $overall_checkpoint{checkpoint_write} = $_overall_checkpoint{checkpoint_write} + if ($_overall_checkpoint{checkpoint_write} > $overall_checkpoint{checkpoint_write}); + $overall_checkpoint{checkpoint_sync} = $_overall_checkpoint{checkpoint_sync} + if ($_overall_checkpoint{checkpoint_sync} > $overall_checkpoint{checkpoint_sync}); + foreach my $k (keys %{$_overall_stat{peak}}) { + $overall_stat{peak}{$k}{query} += $_overall_stat{peak}{$k}{query}; + $overall_stat{peak}{$k}{select} += $_overall_stat{peak}{$k}{select}; + $overall_stat{peak}{$k}{write} += $_overall_stat{peak}{$k}{write}; + $overall_stat{peak}{$k}{connection} += $_overall_stat{peak}{$k}{connection}; + $overall_stat{peak}{$k}{session} += $_overall_stat{peak}{$k}{session}; + $overall_stat{peak}{$k}{tempfile_size} += $_overall_stat{peak}{$k}{tempfile_size}; + $overall_stat{peak}{$k}{tempfile_count} += $_overall_stat{peak}{$k}{tempfile_count}; + } + + foreach my $k (keys %{$_overall_stat{histogram}{query_time}}) { + $overall_stat{histogram}{query_time}{$k} += $_overall_stat{histogram}{query_time}{$k}; + } + $overall_stat{histogram}{total} += $_overall_stat{histogram}{total}; + + foreach my $k ('prepare', 'bind','execute') { + $overall_stat{$k} += $_overall_stat{$k}; + } + + foreach my $k (keys %{$_overall_checkpoint{peak}}) { + $overall_checkpoint{peak}{$k}{checkpoint_wbuffer} += $_overall_checkpoint{peak}{$k}{checkpoint_wbuffer}; + $overall_checkpoint{peak}{$k}{walfile_usage} += $_overall_checkpoint{peak}{$k}{walfile_usage}; + } + + ### Logs level ### + foreach my $l (qw(LOG WARNING ERROR FATAL PANIC DETAIL HINT STATEMENT CONTEXT)) { + $logs_type{$l} += $_logs_type{$l} if exists $_logs_type{$l}; } - - $logs_type{ERROR} += $_logs_type{ERROR} if exists $_logs_type{ERROR}; - $logs_type{LOG} += $_logs_type{LOG} if exists $_logs_type{LOG}; - $logs_type{DETAIL} += $_logs_type{DETAIL} if exists $_logs_type{DETAIL}; - $logs_type{STATEMENT} += $_logs_type{STATEMENT} if exists $_logs_type{STATEMENT}; - ### database_info ### foreach my $db (keys %_database_info) { @@ -4565,6 +7334,23 @@ } } + ### user_info ### + + foreach my $u (keys %_user_info) { + foreach my $k (keys %{ $_user_info{$u} }) { + $user_info{$u}{$k} += $_user_info{$u}{$k}; + } + } + + ### host_info ### + + foreach my $h (keys %_host_info) { + foreach my $k (keys %{ $_host_info{$h} }) { + $host_info{$h}{$k} += $_host_info{$h}{$k}; + } + } + + ### connection_info ### foreach my $db (keys %{ $_connection_info{database} }) { @@ -4590,71 +7376,39 @@ foreach my $day (keys %{ $_connection_info{chronos} }) { foreach my $hour (keys %{ $_connection_info{chronos}{$day} }) { - foreach my $db (keys %{ $_connection_info{chronos}{$day}{$hour}{database} }) { - $connection_info{chronos}{$day}{$hour}{database}{$db} += $_connection_info{chronos}{$day}{$hour}{database}{$db}; - } - - foreach my $db (keys %{ $_connection_info{chronos}{$day}{$hour}{database_user} }) { - foreach my $user (keys %{ $_connection_info{chronos}{$day}{$hour}{database_user}{$db} }) { - $connection_info{chronos}{$day}{$hour}{database_user}{$db}{$user} += - $_connection_info{chronos}{$day}{$hour}{database_user}{$db}{$user}; - } - } - - $connection_info{chronos}{$day}{$hour}{count} += $_connection_info{chronos}{$day}{$hour}{count}; - - foreach my $user (keys %{ $_connection_info{chronos}{$day}{$hour}{user} }) { - $connection_info{chronos}{$day}{$hour}{user}{$user} += - $_connection_info{chronos}{$day}{$hour}{user}{$user}; - } + $connection_info{chronos}{$day}{$hour}{count} += $_connection_info{chronos}{$day}{$hour}{count} - foreach my $host (keys %{ $_connection_info{chronos}{$day}{$hour}{host} }) { - $connection_info{chronos}{$day}{$hour}{host}{$host} += - $_connection_info{chronos}{$day}{$hour}{host}{$host}; - } +############################################################################### +# May be used in the future to display more detailed information on connection +# +# foreach my $db (keys %{ $_connection_info{chronos}{$day}{$hour}{database} }) { +# $connection_info{chronos}{$day}{$hour}{database}{$db} += $_connection_info{chronos}{$day}{$hour}{database}{$db}; +# } +# +# foreach my $db (keys %{ $_connection_info{chronos}{$day}{$hour}{database_user} }) { +# foreach my $user (keys %{ $_connection_info{chronos}{$day}{$hour}{database_user}{$db} }) { +# $connection_info{chronos}{$day}{$hour}{database_user}{$db}{$user} += +# $_connection_info{chronos}{$day}{$hour}{database_user}{$db}{$user}; +# } +# } +# +# foreach my $user (keys %{ $_connection_info{chronos}{$day}{$hour}{user} }) { +# $connection_info{chronos}{$day}{$hour}{user}{$user} += +# $_connection_info{chronos}{$day}{$hour}{user}{$user}; +# } +# +# foreach my $host (keys %{ $_connection_info{chronos}{$day}{$hour}{host} }) { +# $connection_info{chronos}{$day}{$hour}{host}{$host} += +# $_connection_info{chronos}{$day}{$hour}{host}{$host}; +# } +############################################################################### } } - ### log_files ### - - foreach my $f (@_log_files) { - push(@log_files, $f) if (!grep(m#^$f$#, @_log_files)); - } - - ### per_hour_info ### - - foreach my $day (keys %_per_hour_info) { - foreach my $hour (keys %{ $_per_hour_info{$day} }) { - $per_hour_info{$day}{$hour}{count} += $_per_hour_info{$day}{$hour}{count}; - $per_hour_info{$day}{$hour}{duration} += $_per_hour_info{$day}{$hour}{duration}; - # Set min / max duration for this query - if (!exists $per_hour_info{$day}{$hour}{min} || ($per_hour_info{$day}{$hour}{min} > $_per_hour_info{$day}{$hour}{min})) { - $per_hour_info{$day}{$hour}{min} = $_per_hour_info{$day}{$hour}{min}; - } - if (!exists $per_hour_info{$day}{$hour}{max} || ($per_hour_info{$day}{$hour}{max} < $_per_hour_info{$day}{$hour}{max})) { - $per_hour_info{$day}{$hour}{max} = $_per_hour_info{$day}{$hour}{max}; - } - - if (exists $_per_hour_info{$day}{$hour}{DELETE}) { - $per_hour_info{$day}{$hour}{DELETE}{count} += $_per_hour_info{$day}{$hour}{DELETE}{count}; - $per_hour_info{$day}{$hour}{DELETE}{duration} += $_per_hour_info{$day}{$hour}{DELETE}{duration}; - } - - if (exists $_per_hour_info{$day}{$hour}{SELECT}) { - $per_hour_info{$day}{$hour}{SELECT}{count} += $_per_hour_info{$day}{$hour}{SELECT}{count}; - $per_hour_info{$day}{$hour}{SELECT}{duration} += $_per_hour_info{$day}{$hour}{SELECT}{duration}; - } - - if (exists $_per_hour_info{$day}{$hour}{INSERT}) { - $per_hour_info{$day}{$hour}{INSERT}{count} += $_per_hour_info{$day}{$hour}{INSERT}{count}; - $per_hour_info{$day}{$hour}{INSERT}{duration} += $_per_hour_info{$day}{$hour}{INSERT}{duration}; - } + ### log_files ### - if (exists $_per_hour_info{$day}{$hour}{UPDATE}) { - $per_hour_info{$day}{$hour}{UPDATE}{count} += $_per_hour_info{$day}{$hour}{UPDATE}{count}; - $per_hour_info{$day}{$hour}{UPDATE}{duration} += $_per_hour_info{$day}{$hour}{UPDATE}{duration}; - } - } + foreach my $f (@_log_files) { + push(@log_files, $f) if (!grep(m#^$f$#, @_log_files)); } ### error_info ### @@ -4680,33 +7434,64 @@ ### per_minute_info ### - foreach my $day (keys %{ $_per_minute_info{connection} }) { - foreach my $hour (keys %{ $_per_minute_info{connection}{$day} }) { - foreach my $min (keys %{ $_per_minute_info{connection}{$day}{$hour} }) { - $per_minute_info{connection}{$day}{$hour}{$min}{count} += - $_per_minute_info{connection}{$day}{$hour}{$min}{count}; - - foreach my $sec (keys %{ $_per_minute_info{connection}{$day}{$hour}{$min}{second} }) { - $per_minute_info{connection}{$day}{$hour}{$min}{second}{$sec} += - $_per_minute_info{connection}{$day}{$hour}{$min}{second}{$sec}; + foreach my $day (keys %_per_minute_info) { + foreach my $hour (keys %{ $_per_minute_info{$day} }) { + foreach my $min (keys %{ $_per_minute_info{$day}{$hour} }) { + $per_minute_info{$day}{$hour}{$min}{connection}{count} += + ($_per_minute_info{$day}{$hour}{$min}{connection}{count} || 0); + $per_minute_info{$day}{$hour}{$min}{session}{count} += + ($_per_minute_info{$day}{$hour}{$min}{session}{count} || 0); + $per_minute_info{$day}{$hour}{$min}{query}{count} += + ($_per_minute_info{$day}{$hour}{$min}{query}{count} || 0); + $per_minute_info{$day}{$hour}{$min}{query}{duration} += $_per_minute_info{$day}{$hour}{$min}{query}{duration}; + + foreach my $sec (keys %{ $_per_minute_info{$day}{$hour}{$min}{connection}{second} }) { + $per_minute_info{$day}{$hour}{$min}{connection}{second}{$sec} += + ($_per_minute_info{$day}{$hour}{$min}{connection}{second}{$sec} || 0); + } + foreach my $sec (keys %{ $_per_minute_info{$day}{$hour}{$min}{session}{second} }) { + $per_minute_info{$day}{$hour}{$min}{session}{second}{$sec} += + ($_per_minute_info{$day}{$hour}{$min}{session}{second}{$sec} || 0); + } + foreach my $sec (keys %{ $_per_minute_info{$day}{$hour}{$min}{query}{second} }) { + $per_minute_info{$day}{$hour}{$min}{query}{second}{$sec} += + ($_per_minute_info{$day}{$hour}{$min}{query}{second}{$sec} || 0); + } + foreach my $action (@SQL_ACTION) { + if (exists $_per_minute_info{$day}{$hour}{$min}{$action}) { + $per_minute_info{$day}{$hour}{$min}{$action}{count} += $_per_minute_info{$day}{$hour}{$min}{$action}{count}; + $per_minute_info{$day}{$hour}{$min}{$action}{duration} += $_per_minute_info{$day}{$hour}{$min}{$action}{duration}; + foreach my $sec (keys %{ $_per_minute_info{$day}{$hour}{$min}{$action}{second} }) { + $per_minute_info{$day}{$hour}{$min}{$action}{second}{$sec} += + ($_per_minute_info{$day}{$hour}{$min}{$action}{second}{$sec} || 0); + } + } } - } - } - } - - foreach my $day (keys %{ $_per_minute_info{query} }) { - foreach my $hour (keys %{ $_per_minute_info{query}{$day} }) { - foreach my $min (keys %{ $_per_minute_info{query}{$day}{$hour} }) { - $per_minute_info{query}{$day}{$hour}{$min}{count} += - $_per_minute_info{query}{$day}{$hour}{$min}{count}; - - $per_minute_info{query}{$day}{$hour}{$min}{duration} += - $_per_minute_info{query}{$day}{$hour}{$min}{duration}; - - foreach my $sec (keys %{ $_per_minute_info{query}{$day}{$hour}{$min}{second} }) { - $per_minute_info{query}{$day}{$hour}{$min}{second}{$sec} += - $_per_minute_info{query}{$day}{$hour}{$min}{second}{$sec}; + foreach my $k ('prepare', 'bind','execute') { + if (exists $_per_minute_info{$day}{$hour}{$min}{$k}) { + $per_minute_info{$day}{$hour}{$min}{$k} += $_per_minute_info{$day}{$hour}{$min}{$k}; + } } + + $per_minute_info{$day}{$hour}{$min}{tempfile}{count} += $_per_minute_info{$day}{$hour}{$min}{tempfile}{count} + if defined $_per_minute_info{$day}{$hour}{$min}{tempfile}{count}; + $per_minute_info{$day}{$hour}{$min}{tempfile}{size} += $_per_minute_info{$day}{$hour}{$min}{tempfile}{size} + if defined $_per_minute_info{$day}{$hour}{$min}{tempfile}{size}; + + $per_minute_info{$day}{$hour}{$min}{checkpoint}{file_removed} += $_per_minute_info{$day}{$hour}{$min}{checkpoint}{file_removed}; + $per_minute_info{$day}{$hour}{$min}{checkpoint}{sync} += $_per_minute_info{$day}{$hour}{$min}{checkpoint}{sync}; + $per_minute_info{$day}{$hour}{$min}{checkpoint}{wbuffer} += $_per_minute_info{$day}{$hour}{$min}{checkpoint}{wbuffer}; + $per_minute_info{$day}{$hour}{$min}{checkpoint}{file_recycled} += $_per_minute_info{$day}{$hour}{$min}{checkpoint}{file_recycled}; + $per_minute_info{$day}{$hour}{$min}{checkpoint}{total} += $_per_minute_info{$day}{$hour}{$min}{checkpoint}{total}; + $per_minute_info{$day}{$hour}{$min}{checkpoint}{file_added} += $_per_minute_info{$day}{$hour}{$min}{checkpoint}{file_added}; + $per_minute_info{$day}{$hour}{$min}{checkpoint}{write} += $_per_minute_info{$day}{$hour}{$min}{checkpoint}{write}; + $per_minute_info{$day}{$hour}{$min}{autovacuum}{count} += $_per_minute_info{$day}{$hour}{$min}{autovacuum}{count}; + $per_minute_info{$day}{$hour}{$min}{autoanalyze}{count} += $_per_minute_info{$day}{$hour}{$min}{autoanalyze}{count}; + + $per_minute_info{$day}{$hour}{$min}{checkpoint}{sync_files} += $_per_minute_info{$day}{$hour}{$min}{checkpoint}{sync_files}; + $per_minute_info{$day}{$hour}{$min}{checkpoint}{sync_avg} += $_per_minute_info{$day}{$hour}{$min}{checkpoint}{sync_avg}; + $per_minute_info{$day}{$hour}{$min}{checkpoint}{sync_longest} = $_per_minute_info{$day}{$hour}{$min}{checkpoint}{sync_longest} + if ($_per_minute_info{$day}{$hour}{$min}{checkpoint}{sync_longest} > $per_minute_info{$day}{$hour}{$min}{checkpoint}{sync_longest}); } } } @@ -4836,25 +7621,28 @@ if defined $_tempfile_info{maxsize} and ( not defined $tempfile_info{maxsize} or $tempfile_info{maxsize} < $_tempfile_info{maxsize} ); - foreach my $day ( %{ $_tempfile_info{chronos} } ) { - foreach my $hour ( %{ $_tempfile_info{chronos}{$day} } ) { - - $tempfile_info{chronos}{$day}{$hour}{count} += - $_tempfile_info{chronos}{$day}{$hour}{count} - if defined $_tempfile_info{chronos}{$day}{$hour}{count}; - - $tempfile_info{chronos}{$day}{$hour}{size} += - $_tempfile_info{chronos}{$day}{$hour}{size} - if defined $_tempfile_info{chronos}{$day}{$hour}{size}; - } - } - ### top_slowest ### my @tmp_top_slowest = sort {$b->[0] <=> $a->[0]} (@top_slowest, @_top_slowest); @top_slowest = (); for (my $i = 0; $i <= $#tmp_top_slowest; $i++) { - last if ($i == $end_top); push(@top_slowest, $tmp_top_slowest[$i]); + last if ($i == $end_top); + } + + ### top_locked ### + my @tmp_top_locked_info = sort {$b->[0] <=> $a->[0]} (@top_locked_info, @_top_locked_info); + @top_locked_info = (); + for (my $i = 0; $i <= $#tmp_top_locked_info; $i++) { + push(@top_locked_info, $tmp_top_locked_info[$i]); + last if ($i == $end_top); + } + + ### top_tempfile ### + my @tmp_top_tempfile_info = sort {$b->[0] <=> $a->[0]} (@top_tempfile_info, @_top_tempfile_info); + @top_tempfile_info = (); + for (my $i = 0; $i <= $#tmp_top_tempfile_info; $i++) { + push(@top_tempfile_info, $tmp_top_tempfile_info[$i]); + last if ($i == $end_top); } ### checkpoint_info ### @@ -4866,34 +7654,7 @@ $checkpoint_info{file_added} += $_checkpoint_info{file_added}; $checkpoint_info{write} += $_checkpoint_info{write}; - foreach my $day (keys %{ $_checkpoint_info{chronos} }) { - foreach my $hour (keys %{ $_checkpoint_info{chronos}{$day} }) { - $checkpoint_info{chronos}{$day}{$hour}{file_removed} += $_checkpoint_info{chronos}{$day}{$hour}{file_removed}; - $checkpoint_info{chronos}{$day}{$hour}{sync} += $_checkpoint_info{chronos}{$day}{$hour}{sync}; - $checkpoint_info{chronos}{$day}{$hour}{wbuffer} += $_checkpoint_info{chronos}{$day}{$hour}{wbuffer}; - $checkpoint_info{chronos}{$day}{$hour}{file_recycled} += $_checkpoint_info{chronos}{$day}{$hour}{file_recycled}; - $checkpoint_info{chronos}{$day}{$hour}{total} += $_checkpoint_info{chronos}{$day}{$hour}{total}; - $checkpoint_info{chronos}{$day}{$hour}{file_added} += $_checkpoint_info{chronos}{$day}{$hour}{file_added}; - $checkpoint_info{chronos}{$day}{$hour}{write} += $_checkpoint_info{chronos}{$day}{$hour}{write}; - } - } - - ### restartpoint_info ### - $restartpoint_info{sync} += $_restartpoint_info{sync}; - $restartpoint_info{wbuffer} += $_restartpoint_info{wbuffer}; - $restartpoint_info{total} += $_restartpoint_info{total}; - $restartpoint_info{write} += $_restartpoint_info{write}; - - foreach my $day (keys %{ $_restartpoint_info{chronos} }) { - foreach my $hour (keys %{ $_restartpoint_info{chronos}{$day} }) { - $restartpoint_info{chronos}{$day}{$hour}{sync} += $_restartpoint_info{chronos}{$day}{$hour}{sync}; - $restartpoint_info{chronos}{$day}{$hour}{wbuffer} += $_restartpoint_info{chronos}{$day}{$hour}{wbuffer}; - $restartpoint_info{chronos}{$day}{$hour}{total} += $_restartpoint_info{chronos}{$day}{$hour}{total}; - $restartpoint_info{chronos}{$day}{$hour}{write} += $_restartpoint_info{chronos}{$day}{$hour}{write}; - } - } - - #### Autovacuum infos #### + #### Autovacuum info #### $autovacuum_info{count} += $_autovacuum_info{count}; @@ -4908,8 +7669,12 @@ $autovacuum_info{tables}{$table}{tuples}{removed} += $_autovacuum_info{tables}{$table}{tuples}{removed}; $autovacuum_info{tables}{$table}{pages}{removed} += $_autovacuum_info{tables}{$table}{pages}{removed}; } - - #### Autoanalyze infos #### + if ($_autovacuum_info{peak}{system_usage}{elapsed} > $autovacuum_info{peak}{system_usage}{elapsed}) { + $autovacuum_info{peak}{system_usage}{elapsed} = $_autovacuum_info{peak}{system_usage}{elapsed}; + $autovacuum_info{peak}{system_usage}{table} = $_autovacuum_info{peak}{system_usage}{table}; + $autovacuum_info{peak}{system_usage}{date} = $_autovacuum_info{peak}{system_usage}{date}; + } + #### Autoanalyze info #### $autoanalyze_info{count} += $_autoanalyze_info{count}; @@ -4921,6 +7686,11 @@ foreach my $table (keys %{ $_autoanalyze_info{tables} }) { $autoanalyze_info{tables}{$table}{analyzes} += $_autoanalyze_info{tables}{$table}{analyzes}; } + if ($_autoanalyze_info{peak}{system_usage}{elapsed} > $autoanalyze_info{peak}{system_usage}{elapsed}) { + $autoanalyze_info{peak}{system_usage}{elapsed} = $_autoanalyze_info{peak}{system_usage}{elapsed}; + $autoanalyze_info{peak}{system_usage}{table} = $_autoanalyze_info{peak}{system_usage}{table}; + $autoanalyze_info{peak}{system_usage}{date} = $_autoanalyze_info{peak}{system_usage}{date}; + } return; } @@ -4931,25 +7701,28 @@ store_fd({ 'overall_stat' => \%overall_stat, + 'overall_checkpoint' => \%overall_checkpoint, 'normalyzed_info' => \%normalyzed_info, 'error_info' => \%error_info, 'connection_info' => \%connection_info, 'database_info' => \%database_info, 'application_info' => \%application_info, + 'user_info' => \%user_info, + 'host_info' => \%host_info, 'checkpoint_info' => \%checkpoint_info, - 'restartpoint_info' => \%restartpoint_info, 'session_info' => \%session_info, 'tempfile_info' => \%tempfile_info, 'error_info' => \%error_info, 'logs_type' => \%logs_type, 'lock_info' => \%lock_info, - 'per_hour_info' => \%per_hour_info, 'per_minute_info' => \%per_minute_info, 'top_slowest' => \@top_slowest, 'nlines' => $nlines, 'log_files' => \@log_files, 'autovacuum_info' => \%autovacuum_info, - 'autoanalyze_info' => \%autoanalyze_info + 'autoanalyze_info' => \%autoanalyze_info, + 'top_tempfile_info' => \@top_tempfile_info, + 'top_locked_info' => \@top_locked_info, }, $lfh) || die ("Couldn't save binary data to «$outfile»!\n"); } @@ -4958,11 +7731,11 @@ { my $code = shift; - # Try to escape HTML code - $code =~ s/<([\/a-zA-Z])\b/\<$1/sg; + # Escape HTML code into SQL values + $code = &escape_html($code); - # Do not try to prettify queries longuer - # than 10KB this will take too much time + # Do not try to prettify queries longer + # than 10KB as this will take too much time return $code if (length($code) > 10240); # prettify SQL query @@ -4991,7 +7764,7 @@ } for (my $x = 0 ; $x <= $#KEYWORDS1 ; $x++) { $code =~ s/\b$KEYWORDS1[$x]\b/$KEYWORDS1[$x]<\/span>/igs; - $code =~ s/(?$KEYWORDS1[$x]<\/span>/igs; + $code =~ s/(?$KEYWORDS1[$x]<\/span>/igs; } for (my $x = 0 ; $x <= $#KEYWORDS2 ; $x++) { $code =~ s/(?$KEYWORDS2[$x]<\/span>/igs; @@ -5023,10 +7796,10 @@ sub compute_arg_list { - # Some command lines arguments can be used multiple time or be written - # as a coma separated list. + # Some command line arguments can be used multiple times or written + # as a comma-separated list. # For example: --dbuser=postgres --dbuser=joe or --dbuser=postgres,joe - # So we have to aggregate all the possible value + # So we have to aggregate all the possible values my @tmp = (); foreach my $v (@exclude_user) { push(@tmp, split(/,/, $v)); @@ -5061,26 +7834,38 @@ } @dbappname = (); push(@dbappname, @tmp); + + @tmp = (); + foreach my $v (@exclude_appname) { + push(@tmp, split(/,/, $v)); + } + @exclude_appname = (); + push(@exclude_appname, @tmp); + } sub validate_log_line { my ($t_pid) = @_; + # Look at particular cas of vacuum/analyze that have the database + # name inside the log message so that they could be associated + if ($prefix_vars{'t_query'} =~ / of table "([^\.]+)\.[^\.]+\.[^\.]+":/) { + $prefix_vars{'t_dbname'} = $1; + } + # Check user and/or database if require if ($#dbname >= 0) { - # Log line do not match the required dbname + # Log line does not match the required dbname if (!$prefix_vars{'t_dbname'} || !grep(/^$prefix_vars{'t_dbname'}$/i, @dbname)) { - delete $cur_info{$t_pid}; return 0; } } if ($#dbuser >= 0) { - # Log line do not match the required dbuser + # Log line does not match the required dbuser if (!$prefix_vars{'t_dbuser'} || !grep(/^$prefix_vars{'t_dbuser'}$/i, @dbuser)) { - delete $cur_info{$t_pid}; return 0; } } @@ -5089,7 +7874,6 @@ # Log line does not match the required dbclient $prefix_vars{'t_client'} ||= $prefix_vars{'t_hostport'}; if (!$prefix_vars{'t_client'} || !grep(/^$prefix_vars{'t_client'}$/i, @dbclient)) { - delete $cur_info{$t_pid}; return 0; } } @@ -5097,7 +7881,6 @@ # Log line does not match the required dbname if (!$prefix_vars{'t_appname'} || !grep(/^$prefix_vars{'t_appname'}$/i, @dbappname)) { - delete $cur_info{$t_pid}; return 0; } } @@ -5105,10 +7888,17 @@ # Log line matches the excluded dbuser if ($prefix_vars{'t_dbuser'} && grep(/^$prefix_vars{'t_dbuser'}$/i, @exclude_user)) { - delete $cur_info{$t_pid}; return 0; } } + if ($#exclude_appname >= 0) { + + # Log line matches the excluded appname + if ($prefix_vars{'t_appname'} && grep(/^$prefix_vars{'t_appname'}$/i, @exclude_appname)) { + return 0; + } + } + return 1; } @@ -5136,31 +7926,27 @@ my $t_pid = $prefix_vars{'t_pid'}; - # Force parameter change to be a hint message so that it can appear - # in the event/error/warning messages report part. + # Force some LOG messages to be ERROR messages so that they will appear + # in the event/error/warning messages report. if ($prefix_vars{'t_loglevel'} eq 'LOG') { if ($prefix_vars{'t_query'} =~ /parameter "[^"]+" changed to "[^"]+"/) { - $prefix_vars{'t_loglevel'} = 'HINT'; + $prefix_vars{'t_loglevel'} = 'ERROR'; } elsif ($prefix_vars{'t_query'} =~ /database system was shut down at /) { - $prefix_vars{'t_loglevel'} = 'HINT'; + $prefix_vars{'t_loglevel'} = 'ERROR'; + } elsif ($prefix_vars{'t_query'} =~ /database system was interrupted while in recovery/) { + $prefix_vars{'t_loglevel'} = 'ERROR'; + } elsif ($prefix_vars{'t_query'} =~ /recovery has paused/) { + $prefix_vars{'t_loglevel'} = 'ERROR'; } } - # Do not parse lines that are not an error like message - if ($error_only && ($prefix_vars{'t_loglevel'} !~ /(WARNING|ERROR|FATAL|PANIC|DETAIL|HINT|STATEMENT|CONTEXT)/)) { - if (exists $cur_info{$t_pid} && (!$prefix_vars{'t_session_line'} || ($prefix_vars{'t_session_line'} != $cur_info{$t_pid}{session}))) { - &store_queries($t_pid); - delete $cur_info{$t_pid}; - } + # Do not parse lines that are not an error message when error only report is requested + if ($error_only && ($prefix_vars{'t_loglevel'} !~ $full_error_regex)) { return; } - # Do not parse lines that are an error like message - if ($disable_error && ($prefix_vars{'t_loglevel'} =~ /WARNING|ERROR|FATAL|PANIC|HINT|CONTEXT|DETAIL|STATEMENT/)) { - if (exists $cur_info{$t_pid} && (!$prefix_vars{'t_session_line'} || ($prefix_vars{'t_session_line'} != $cur_info{$t_pid}{session}))) { - &store_queries($t_pid); - delete $cur_info{$t_pid}; - } + # Do not parse lines that are an error-like message when error reports are not wanted + if ($disable_error && ($prefix_vars{'t_loglevel'} =~ $full_error_regex)) { return; } @@ -5168,9 +7954,35 @@ $logs_type{$prefix_vars{'t_loglevel'}}++; # Replace syslog tabulation rewrite - $prefix_vars{'t_query'} =~ s/#011/\t/g if ($format =~ /syslog/); + if ($format =~ /syslog/) { + $prefix_vars{'t_query'} =~ s/#011/\t/g; + } + + # Reject lines generated by debug tool + if ( ($prefix_vars{'t_loglevel'} eq 'CONTEXT') && ($prefix_vars{'t_query'} =~ /SQL statement "/) ) { + return; + } + + # Stores the error's detail if previous line was an error + if ($cur_info{$t_pid}{loglevel} =~ $main_error_regex) { + # and current one is a detailed information + if ($prefix_vars{'t_loglevel'} =~ /(DETAIL|STATEMENT|CONTEXT|HINT)/) { + $cur_info{$t_pid}{"\L$1\E"} .= $prefix_vars{'t_query'}; + return; + } + } my $date_part = "$prefix_vars{'t_year'}$prefix_vars{'t_month'}$prefix_vars{'t_day'}"; + my $cur_last_log_timestamp = "$prefix_vars{'t_year'}-$prefix_vars{'t_month'}-$prefix_vars{'t_day'} " . + "$prefix_vars{t_hour}:$prefix_vars{t_min}:$prefix_vars{t_sec}"; + + # set current session workload + if (!$disable_session) { + my $sess_count = scalar keys %current_sessions; + $overall_stat{'peak'}{$cur_last_log_timestamp}{session} = $sess_count; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{"$prefix_vars{'t_min'}"}{session}{count} = $sess_count; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{"$prefix_vars{'t_min'}"}{session}{second}{$prefix_vars{'t_sec'}} = $sess_count; + } # Stores lock activity if (($prefix_vars{'t_loglevel'} eq 'LOG') && ($prefix_vars{'t_query'} =~ /acquired ([^\s]+) on ([^\s]+) .* after ([0-9\.]+) ms/)) @@ -5185,6 +7997,14 @@ # Store current lock information that will be used later # when we will parse the query responsible of the locks $cur_lock_info{$t_pid}{wait} = $3; + if ($format eq 'csv') { + $cur_lock_info{$t_pid}{query} = $prefix_vars{'t_statement'}; + $cur_lock_info{$t_pid}{timestamp} = $prefix_vars{'t_timestamp'}; + $cur_lock_info{$t_pid}{dbname} = $prefix_vars{'t_dbname'}; + $cur_lock_info{$t_pid}{dbuser} = $prefix_vars{'t_dbuser'}; + $cur_lock_info{$t_pid}{dbclient} = $prefix_vars{'t_client'}; + $cur_lock_info{$t_pid}{dbappname} = $prefix_vars{'t_appname'}; + } return; } @@ -5196,7 +8016,6 @@ $cur_lock_info{$t_pid}{dbuser} = $prefix_vars{'t_dbuser'}; $cur_lock_info{$t_pid}{dbclient} = $prefix_vars{'t_client'}; $cur_lock_info{$t_pid}{dbappname} = $prefix_vars{'t_appname'}; - $cur_lock_info{$t_pid}{timestamp} = $prefix_vars{'t_timestamp'}; return; } @@ -5205,41 +8024,61 @@ return if ($disable_temporary); $tempfile_info{count}++; $tempfile_info{size} += $1; - $tempfile_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{count}++; - $tempfile_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{size} += $1; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{tempfile}{count}++; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{tempfile}{size} += $1; $tempfile_info{maxsize} = $1 if ($tempfile_info{maxsize} < $1); # Store current temporary file information that will be used later # when we will parse the query responsible of the tempfile $cur_temp_info{$t_pid}{size} = $1; + $overall_stat{'peak'}{$cur_last_log_timestamp}{tempfile_size} += $1; + $overall_stat{'peak'}{$cur_last_log_timestamp}{tempfile_count}++; + if ($format eq 'csv') { + $cur_temp_info{$t_pid}{query} = $prefix_vars{'t_statement'}; + $cur_temp_info{$t_pid}{timestamp} = $prefix_vars{'t_timestamp'}; + $cur_temp_info{$t_pid}{dbname} = $prefix_vars{'t_dbname'}; + $cur_temp_info{$t_pid}{dbuser} = $prefix_vars{'t_dbuser'}; + $cur_temp_info{$t_pid}{dbclient} = $prefix_vars{'t_client'}; + $cur_temp_info{$t_pid}{dbappname} = $prefix_vars{'t_appname'}; + } return; } # Stores query related to last created temporary file - if (($prefix_vars{'t_loglevel'} eq 'STATEMENT') && exists $cur_temp_info{$t_pid}) { + if (($prefix_vars{'t_loglevel'} eq 'STATEMENT') && $cur_temp_info{$t_pid}{size}) { $cur_temp_info{$t_pid}{query} = $prefix_vars{'t_query'}; $cur_temp_info{$t_pid}{timestamp} = $prefix_vars{'t_timestamp'}; $cur_temp_info{$t_pid}{dbname} = $prefix_vars{'t_dbname'}; $cur_temp_info{$t_pid}{dbuser} = $prefix_vars{'t_dbuser'}; $cur_temp_info{$t_pid}{dbclient} = $prefix_vars{'t_client'}; $cur_temp_info{$t_pid}{dbappname} = $prefix_vars{'t_appname'}; - $cur_temp_info{$t_pid}{timestamp} = $prefix_vars{'t_timestamp'}; return; } # Stores pre-connection activity if (($prefix_vars{'t_loglevel'} eq 'LOG') && ($prefix_vars{'t_query'} =~ /connection received: host=([^\s]+) port=(\d+)/)) { return if ($disable_connection); + $current_sessions{$prefix_vars{'t_pid'}} = 1; $conn_received{$t_pid} = $1; return; } # Stores connection activity if ( ($prefix_vars{'t_loglevel'} eq 'LOG') - && ($prefix_vars{'t_query'} =~ /connection authorized: user=([^\s]+) database=([^\s]+)/)) + && ($prefix_vars{'t_query'} =~ /connection authorized: user=([^\s]+) /)) { return if ($disable_connection); + $current_sessions{$prefix_vars{'t_pid'}} = 1; my $usr = $1; - my $db = $2; + my $db = 'unknown'; + my $host = ''; + if ($prefix_vars{'t_query'} =~ / database=([^\s]+)/) { + $db = $1; + } elsif ($prefix_vars{'t_dbname'}) { + $db = $prefix_vars{'t_dbname'}; + } + if ($prefix_vars{'t_query'} =~ / host=([^\s]+)/) { + $host = $1; + } if ($extension eq 'tsung') { $tsung_session{$prefix_vars{'t_pid'}}{connection}{database} = $db; $tsung_session{$prefix_vars{'t_pid'}}{connection}{user} = $usr; @@ -5247,24 +8086,31 @@ return; } + $overall_stat{'peak'}{$cur_last_log_timestamp}{connection}++; + $connection_info{count}++; $connection_info{user}{$usr}++; $connection_info{database}{$db}++; $connection_info{database_user}{$db}{$usr}++; $connection_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{count}++; - $connection_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{user}{$usr}++; - $connection_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{database}{$db}++; - $connection_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{database_user}{$db}{$usr}++; +############################################################################### +# May be used in the future to display more detailed information on connection +# $connection_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{user}{$usr}++; +# $connection_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{database}{$db}++; +# $connection_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{database_user}{$db}{$usr}++; +############################################################################### if ($graph) { - $per_minute_info{connection}{$date_part}{$prefix_vars{'t_hour'}}{"$prefix_vars{'t_min'}"}{count}++; - $per_minute_info{connection}{$date_part}{$prefix_vars{'t_hour'}}{"$prefix_vars{'t_min'}"}{second} - {$prefix_vars{'t_sec'}}++; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{"$prefix_vars{'t_min'}"}{connection}{count}++; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{"$prefix_vars{'t_min'}"}{connection}{second}{$prefix_vars{'t_sec'}}++; } if (exists $conn_received{$t_pid}) { $connection_info{host}{$conn_received{$t_pid}}++; - $connection_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{host}{$conn_received{$t_pid}}++; + #$connection_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{host}{$conn_received{$t_pid}}++; delete $conn_received{$t_pid}; + } elsif ($host) { + $connection_info{host}{$host}++; + #$connection_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{host}{$host}++; } return; } @@ -5278,6 +8124,7 @@ if ($extension eq 'tsung') { $tsung_session{$prefix_vars{'t_pid'}}{disconnection}{date} = $prefix_vars{'t_timestamp'}; } + delete $current_sessions{$prefix_vars{'t_pid'}}; my $time = $1; my $usr = $2; my $db = $3; @@ -5317,9 +8164,18 @@ $autovacuum_info{tables}{$1}{vacuums} += 1; $autovacuum_info{tables}{$1}{idxscans} += $2; $autovacuum_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{count}++; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{autovacuum}{count}++; $cur_info{$t_pid}{vacuum} = $1; + $cur_info{$t_pid}{year} = $prefix_vars{'t_year'}; + $cur_info{$t_pid}{month} = $prefix_vars{'t_month'}; + $cur_info{$t_pid}{day} = $prefix_vars{'t_day'}; + $cur_info{$t_pid}{hour} = $prefix_vars{'t_hour'}; + $cur_info{$t_pid}{min} = $prefix_vars{'t_min'}; + $cur_info{$t_pid}{sec} = $prefix_vars{'t_sec'}; return; } + + # Store autoanalyze information if ( ($prefix_vars{'t_loglevel'} eq 'LOG') && ($prefix_vars{'t_query'} =~ @@ -5328,53 +8184,79 @@ ) { return if ($disable_autovacuum); + my $table = $1; $autoanalyze_info{count}++; - $autoanalyze_info{tables}{$1}{analyzes} += 1; + $autoanalyze_info{tables}{$table}{analyzes} += 1; $autoanalyze_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{count}++; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{autoanalyze}{count}++; + if ($prefix_vars{'t_query'} =~ m#system usage: CPU .* sec elapsed (.*) sec#) { + if ($1 > $autoanalyze_info{peak}{system_usage}{elapsed}) { + $autoanalyze_info{peak}{system_usage}{elapsed} = $1; + $autoanalyze_info{peak}{system_usage}{table} = $table; + $autoanalyze_info{peak}{system_usage}{date} = $cur_last_log_timestamp; + } + } } - # Store checkpoint information + # Store checkpoint or restartpoint information if ( ($prefix_vars{'t_loglevel'} eq 'LOG') && ($prefix_vars{'t_query'} =~ -/checkpoint complete: wrote (\d+) buffers \(([^\)]+)\); (\d+) transaction log file\(s\) added, (\d+) removed, (\d+) recycled; write=([0-9\.]+) s, sync=([0-9\.]+) s, total=([0-9\.]+) s/ +/point complete: wrote (\d+) buffers \(([^\)]+)\); (\d+) transaction log file\(s\) added, (\d+) removed, (\d+) recycled; write=([0-9\.]+) s, sync=([0-9\.]+) s, total=([0-9\.]+) s/ ) ) { + # Example: LOG: checkpoint complete: wrote 175 buffers (5.7%); 0 transaction log file(s) added, 1 removed, 2 recycled; write=17.437 s, sync=0.722 s, total=18.259 s; sync files=2, longest=0.708 s, average=0.361 s return if ($disable_checkpoint); + $checkpoint_info{wbuffer} += $1; #$checkpoint_info{percent_wbuffer} += $2; $checkpoint_info{file_added} += $3; $checkpoint_info{file_removed} += $4; $checkpoint_info{file_recycled} += $5; + $overall_checkpoint{'peak'}{$cur_last_log_timestamp}{walfile_usage} += ($3 + $5); $checkpoint_info{write} += $6; $checkpoint_info{sync} += $7; $checkpoint_info{total} += $8; - - $checkpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{wbuffer} += $1; - - #$checkpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{percent_wbuffer} += $2; - $checkpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{file_added} += $3; - $checkpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{file_removed} += $4; - $checkpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{file_recycled} += $5; - $checkpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{write} += $6; - $checkpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{sync} += $7; - $checkpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{total} += $8; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{wbuffer} += $1; + #$per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{percent_wbuffer} += $2; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{file_added} += $3; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{file_removed} += $4; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{file_recycled} += $5; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{write} += $6; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{sync} += $7; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{total} += $8; + + $overall_checkpoint{'peak'}{$cur_last_log_timestamp}{checkpoint_wbuffer} += $1; + if ($6 > $overall_checkpoint{checkpoint_write}) { + $overall_checkpoint{checkpoint_write} = $6; + $overall_checkpoint{checkpoint_sync} = $7; + } + + if ($prefix_vars{'t_query'} =~ /sync files=(\d+), longest=([0-9\.]+) s, average=([0-9\.]+) s/) { + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{sync_files} += $1; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{sync_longest} = $2 + if ($2 > $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{sync_longest}); + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{sync_avg} += $3; + } return; } + + # Store checkpoint warning information if ( ($prefix_vars{'t_loglevel'} eq 'LOG') && ($prefix_vars{'t_query'} =~ /checkpoints are occurring too frequently \((\d+) seconds apart\)/)) { return if ($disable_checkpoint); $checkpoint_info{warning}++; $checkpoint_info{warning_seconds} += $1; - $checkpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{warning}++; - $checkpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{warning_seconds} += $1; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{warning}++; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{warning_seconds} += $1; + $overall_checkpoint{checkpoint_warning}++; return; } - # Store restartpoint information + # Store old restartpoint information if ( ($prefix_vars{'t_loglevel'} eq 'LOG') && ($prefix_vars{'t_query'} =~ @@ -5382,32 +8264,31 @@ ) ) { + # Example: LOG: restartpoint complete: wrote 1568 buffers (0.3%); write=146.237 s, sync=0.251 s, total=146.489 s return if ($disable_checkpoint); - $restartpoint_info{wbuffer} += $1; - #$restartpoint_info{percent_wbuffer} += $2; - $restartpoint_info{write} += $3; - $restartpoint_info{sync} += $4; - $restartpoint_info{total} += $5; - - $restartpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{wbuffer} += $1; - - #$restartpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{percent_wbuffer} += $2; - $restartpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{write} += $3; - $restartpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{sync} += $4; - $restartpoint_info{chronos}{$date_part}{$prefix_vars{'t_hour'}}{total} += $5; - return; - } + $checkpoint_info{wbuffer} += $1; - # Store the detail of the error - if ($cur_info{$t_pid}{loglevel} =~ /WARNING|ERROR|FATAL|PANIC/) { - if ($prefix_vars{'t_loglevel'} =~ /(DETAIL|STATEMENT|CONTEXT|HINT)/) { - $cur_info{$t_pid}{"\L$1\E"} .= $prefix_vars{'t_query'}; - return; + #$checkpoint_info{percent_wbuffer} += $2; + $checkpoint_info{write} += $6; + $checkpoint_info{sync} += $7; + $checkpoint_info{total} += $8; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{wbuffer} += $1; + #$per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{percent_wbuffer} += $2; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{write} += $6; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{sync} += $7; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{checkpoint}{total} += $8; + + $overall_checkpoint{'peak'}{$cur_last_log_timestamp}{checkpoint_wbuffer} += $1; + if ($6 > $overall_checkpoint{checkpoint_write}) { + $overall_checkpoint{checkpoint_write} = $6; + $overall_checkpoint{checkpoint_sync} = $7; } + + return; } - # Process current query following context + # Look at bind parameters if any if ($cur_info{$t_pid}{query}) { # Remove obsolete connection storage @@ -5423,26 +8304,36 @@ return; } } - # When we are ready to overwrite the last storage, add it to the global stats - if ( ($prefix_vars{'t_loglevel'} =~ /LOG|FATAL|PANIC|ERROR|WARNING|HINT/) - && exists $cur_info{$t_pid} - && (($format eq 'csv') || (!$prefix_vars{'t_session_line'} || ($prefix_vars{'t_session_line'} != $cur_info{$t_pid}{session}))) - ) { + } + + # Apply bind parameters if any + if ($prefix_vars{'t_detail'} =~ /parameters: (.*)/) { + $cur_info{$t_pid}{parameters} = "$1"; + # go look at other params + } + + #### + # Registrer previous query storage into global statistics before starting to store current query + #### + if (exists $cur_info{$t_pid}{query}) { + # when switching to a new log message + if ( ($prefix_vars{'t_loglevel'} eq 'LOG') || ($format eq 'csv') || ($prefix_vars{'t_loglevel'} =~ $main_error_regex) ) { &store_queries($t_pid); delete $cur_info{$t_pid}; } } - # Registrer previous query storage into global statistics before starting to store current query - if (exists $cur_info{$t_pid} && (!$prefix_vars{'t_session_line'} || ($prefix_vars{'t_session_line'} != $cur_info{$t_pid}{session}))) { - &store_queries($t_pid); - delete $cur_info{$t_pid}; - } + #### + # Store current query information + #### # Log lines with duration only, generated by log_duration = on in postgresql.conf if ($prefix_vars{'t_query'} =~ s/duration: ([0-9\.]+) ms$//s) { $prefix_vars{'t_duration'} = $1; $prefix_vars{'t_query'} = ''; + my $k = &get_hist_inbound($1); + $overall_stat{histogram}{query_time}{$k}++; + $overall_stat{histogram}{total}++; &set_current_infos($t_pid); return; } @@ -5459,34 +8350,52 @@ if ($prefix_vars{'t_query'} =~ s/duration: ([0-9\.]+) ms (query|statement): //is) { $prefix_vars{'t_duration'} = $1; $t_action = $2; + my $k = &get_hist_inbound($1); + $overall_stat{histogram}{query_time}{$k}++; + $overall_stat{histogram}{total}++; + if (($t_action eq 'statement') && $prefix_vars{'t_query'} =~ /^(PREPARE|EXECUTE)\b/i) { + $overall_stat{lc($1)}++; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{lc($1)}++; + } # Log line with duration and statement from prepared queries - } elsif ($prefix_vars{'t_query'} =~ s/duration: ([0-9\.]+) ms (prepare|parse|bind|execute|execute from fetch)\s+[^:]+:\s//is) + } elsif ($prefix_vars{'t_query'} =~ s/duration: ([0-9\.]+) ms (prepare|parse|bind|execute from fetch|execute)\s+[^:]+:\s//is) { $prefix_vars{'t_duration'} = $1; - $t_action = $2; + $t_action = $2; + my $k = &get_hist_inbound($1); + $overall_stat{histogram}{query_time}{$k}++; + $overall_stat{histogram}{total}++; + $t_action =~ s/ from fetch//; + $t_action = 'prepare' if ($t_action eq 'parse'); + $overall_stat{$t_action}++; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{$t_action}++; # Skipping parse and bind logs return if ($t_action !~ /query|statement|execute/); # Log line without duration at all } elsif ($prefix_vars{'t_query'} =~ s/(query|statement): //is) { $t_action = $1; + if (($t_action eq 'statement') && $prefix_vars{'t_query'} =~ /^(PREPARE|EXECUTE)\b/i) { + $overall_stat{lc($1)}++; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{lc($1)}++; + } # Log line without duration at all from prepared queries - } elsif ($prefix_vars{'t_query'} =~ s/(prepare|parse|bind|execute|execute from fetch)\s+[^:]+:\s//is) + } elsif ($prefix_vars{'t_query'} =~ s/(prepare|parse|bind|execute from fetch|execute)\s+[^:]+:\s//is) { $t_action = $1; + $t_action =~ s/ from fetch//; + $t_action = 'prepare' if ($t_action eq 'parse'); + $overall_stat{$t_action}++; + $per_minute_info{$date_part}{$prefix_vars{'t_hour'}}{$prefix_vars{'t_min'}}{$t_action}++; # Skipping parse and bind logs return if ($t_action !~ /query|statement|execute/); - # Log line that should not be parse + # Log line that would not be parse } elsif ($prefix_vars{'t_loglevel'} eq 'LOG') { if ($prefix_vars{'t_query'} !~ -/incomplete startup packet|connection|receive|unexpected EOF|still waiting for [^\s]+Lock|checkpoint starting:|could not send data to client|parameter .*configuration file|autovacuum launcher|automatic (analyze|vacuum)|detected deadlock while waiting for|database system was shut down/ +/incomplete startup packet|connection|receive|unexpected EOF|still waiting for [^\s]+Lock|checkpoint starting:|could not send data to client|parameter .*configuration file|autovacuum launcher|automatic (analyze|vacuum)|detected deadlock while waiting for/ ) { &logmsg('DEBUG', "Unrecognized line: $prefix_vars{'t_loglevel'}: $prefix_vars{'t_query'} at line $nlines"); } - if (exists $cur_info{$t_pid} && (!$prefix_vars{'t_session_line'} || ($prefix_vars{'t_session_line'} != $cur_info{$t_pid}{session}))) { - &store_queries($t_pid); - delete $cur_info{$t_pid}; - } return; } @@ -5580,13 +8489,19 @@ sub store_queries { - my $t_pid = shift; + my $t_pid = shift; # Remove comments if required if ($remove_comment) { $cur_info{$t_pid}{query} =~ s/\/\*(.*?)\*\///gs; } + # Stores temporary files and locks information + &store_temporary_and_lock_infos($t_pid); + + return if (!exists $cur_info{$t_pid}); + return if (!$cur_info{$t_pid}{year}); + # Cleanup and normalize the current query $cur_info{$t_pid}{query} =~ s/^[\t\s\r\n]+//s; $cur_info{$t_pid}{query} =~ s/[\t\s\r\n;]+$//s; @@ -5650,7 +8565,7 @@ my $cur_hour_str = "$cur_info{$t_pid}{hour}"; # Store the collected information into global statistics - if ($cur_info{$t_pid}{loglevel} =~ /WARNING|ERROR|FATAL|PANIC|HINT/) { + if ($cur_info{$t_pid}{loglevel} =~ $main_error_regex) { # Add log level at beginning of the query and normalize it $cur_info{$t_pid}{query} = $cur_info{$t_pid}{loglevel} . ": " . $cur_info{$t_pid}{query}; @@ -5658,7 +8573,6 @@ # Stores total and normalized error count $overall_stat{'errors_number'}++; - $overall_stat{'unique_normalized_errors'}{"$normalized_error"}++; $error_info{$normalized_error}{count}++; # Stores normalized error count per time @@ -5687,80 +8601,43 @@ if (!$overall_stat{'last_query_ts'} || ($overall_stat{'last_query_ts'} lt $cur_last_log_timestamp)) { $overall_stat{'last_query_ts'} = $cur_last_log_timestamp; } - $overall_stat{'query_peak'}{$cur_last_log_timestamp}++; - $per_hour_info{"$cur_day_str"}{"$cur_hour_str"}{count}++; - if ($cur_info{$t_pid}{duration}) { - $per_hour_info{"$cur_day_str"}{"$cur_hour_str"}{duration} += $cur_info{$t_pid}{duration}; + $overall_stat{'peak'}{$cur_last_log_timestamp}{query}++; + + if ($graph) { + $per_minute_info{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{query}{count}++; + $per_minute_info{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{query}{second}{$cur_info{$t_pid}{sec}}++; + $per_minute_info{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{query}{duration} += $cur_info{$t_pid}{duration} if ($cur_info{$t_pid}{duration}); # Store min / max duration - if (!exists $per_hour_info{"$cur_day_str"}{"$cur_hour_str"}{min} || ($per_hour_info{"$cur_day_str"}{"$cur_hour_str"}{min} > $cur_info{$t_pid}{duration})) { - $per_hour_info{"$cur_day_str"}{"$cur_hour_str"}{min} = $cur_info{$t_pid}{duration}; + if (!exists $per_minute_info{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{min} || ($per_minute_info{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{min} > $cur_info{$t_pid}{duration})) { + $per_minute_info{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{min} = $cur_info{$t_pid}{duration}; } - if (!exists $per_hour_info{"$cur_day_str"}{"$cur_hour_str"}{max} || ($per_hour_info{"$cur_day_str"}{"$cur_hour_str"}{max} < $cur_info{$t_pid}{duration})) { - $per_hour_info{"$cur_day_str"}{"$cur_hour_str"}{max} = $cur_info{$t_pid}{duration}; + if (!exists $per_minute_info{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{max} || ($per_minute_info{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{max} < $cur_info{$t_pid}{duration})) { + $per_minute_info{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{max} = $cur_info{$t_pid}{duration}; } } - if ($graph) { - $per_minute_info{query}{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{count}++; - $per_minute_info{query}{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{second}{$cur_info{$t_pid}{sec}}++; - $per_minute_info{query}{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{duration} += $cur_info{$t_pid}{duration} if ($cur_info{$t_pid}{duration}); - } - # Counter per database and application name if ($cur_info{$t_pid}{dbname}) { $database_info{$cur_info{$t_pid}{dbname}}{count}++; + } else { + $database_info{'unknown'}{count}++; } if ($cur_info{$t_pid}{dbappname}) { $application_info{$cur_info{$t_pid}{dbappname}}{count}++; } else { - $application_info{others}{count}++; + $application_info{'unknown'}{count}++; } - - # Store normalized query temp file size if required - if (exists $cur_temp_info{$t_pid} && ($cur_temp_info{$t_pid} ne '') ) { - - # Add a semi-colon at end of the query - $cur_temp_info{$t_pid}{query} .= ';' if (substr($cur_temp_info{$t_pid}{query}, -1, 1) ne ';'); - - # Normalize query - my $normalized = &normalize_query($cur_temp_info{$t_pid}{query}); - - $normalyzed_info{$normalized}{tempfiles}{size} += $cur_temp_info{$t_pid}{size}; - $normalyzed_info{$normalized}{tempfiles}{count}++; - if ($normalyzed_info{$normalized}{tempfiles}{maxsize} < $cur_temp_info{$t_pid}{size}) { - $normalyzed_info{$normalized}{tempfiles}{maxsize} = $cur_temp_info{$t_pid}{size}; - } - if (!exists($normalyzed_info{$normalized}{tempfiles}{minsize}) - || $normalyzed_info{$normalized}{tempfiles}{minsize} > $cur_temp_info{$t_pid}{size}) { - $normalyzed_info{$normalized}{tempfiles}{minsize} = $cur_temp_info{$t_pid}{size}; - } - &set_top_tempfile_info($cur_temp_info{$t_pid}{query}, $cur_temp_info{$t_pid}{size}, $cur_temp_info{$t_pid}{timestamp}, $cur_temp_info{$t_pid}{dbname}, $cur_temp_info{$t_pid}{dbuser}, $cur_temp_info{$t_pid}{dbclient}, $cur_temp_info{$t_pid}{dbappname}); - delete $cur_temp_info{$t_pid}; + if ($cur_info{$t_pid}{dbuser}) { + $user_info{$cur_info{$t_pid}{dbuser}}{count}++; + } else { + $user_info{'unknown'}{count}++; } - - # Store normalized query that waited the most if required - if (exists $cur_lock_info{$t_pid}) { - - # Add a semi-colon at end of the query - $cur_lock_info{$t_pid}{query} .= ';' if (substr($cur_lock_info{$t_pid}{query}, -1, 1) ne ';'); - - # Normalize query - my $normalized = &normalize_query($cur_lock_info{$t_pid}{query}); - - $normalyzed_info{$normalized}{locks}{wait} += $cur_lock_info{$t_pid}{wait}; - $normalyzed_info{$normalized}{locks}{count}++; - if ($normalyzed_info{$normalized}{locks}{maxwait} < $cur_lock_info{$t_pid}{wait}) { - $normalyzed_info{$normalized}{locks}{maxwait} = $cur_lock_info{$t_pid}{wait}; - } - if (!exists($normalyzed_info{$normalized}{locks}{minwait}) - || $normalyzed_info{$normalized}{locks}{minwait} > $cur_lock_info{$t_pid}{wait}) { - $normalyzed_info{$normalized}{locks}{minwait} = $cur_lock_info{$t_pid}{wait}; - } - &set_top_locked_info($cur_lock_info{$t_pid}{query}, $cur_lock_info{$t_pid}{wait}, $cur_lock_info{$t_pid}{timestamp}, $cur_lock_info{$t_pid}{dbname}, $cur_lock_info{$t_pid}{dbuser}, $cur_lock_info{$t_pid}{dbclient}, $cur_lock_info{$t_pid}{dbappname}); - delete $cur_lock_info{$t_pid}; + if ($cur_info{$t_pid}{dbclient}) { + $host_info{$cur_info{$t_pid}{dbclient}}{count}++; + } else { + $host_info{'unknown'}{count}++; } - if ($cur_info{$t_pid}{query}) { # Add a semi-colon at end of the query $cur_info{$t_pid}{query} .= ';' if (substr($cur_info{$t_pid}{query}, -1, 1) ne ';'); @@ -5772,15 +8649,33 @@ if ($normalized =~ $act) { my $action = uc($1); $overall_stat{$action}++; - $per_hour_info{"$cur_day_str"}{"$cur_hour_str"}{$action}{count}++; - $per_hour_info{"$cur_day_str"}{"$cur_hour_str"}{$action}{duration} += $cur_info{$t_pid}{duration} if ($cur_info{$t_pid}{duration}); + if ($action eq 'SELECT') { + $overall_stat{'peak'}{$cur_last_log_timestamp}{select}++; + } else { + $overall_stat{'peak'}{$cur_last_log_timestamp}{write}++; + } + $per_minute_info{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{$action}{count}++; + $per_minute_info{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{$action}{second}{$cur_info{$t_pid}{sec}}++; + $per_minute_info{"$cur_day_str"}{"$cur_hour_str"}{$cur_info{$t_pid}{min}}{$action}{duration} += $cur_info{$t_pid}{duration} if ($cur_info{$t_pid}{duration}); if ($cur_info{$t_pid}{dbname}) { $database_info{$cur_info{$t_pid}{dbname}}{$action}++; + } else { + $database_info{'unknown'}{$action}++; } if ($cur_info{$t_pid}{dbappname}) { $application_info{$cur_info{$t_pid}{dbappname}}{$action}++; } else { - $application_info{others}{$action}++; + $application_info{'unknown'}{$action}++; + } + if ($cur_info{$t_pid}{dbuser}) { + $user_info{$cur_info{$t_pid}{dbuser}}{$action}++; + } else { + $user_info{'unknown'}{$action}++; + } + if ($cur_info{$t_pid}{dbclient}) { + $host_info{$cur_info{$t_pid}{dbclient}}{$action}++; + } else { + $host_info{'unknown'}{$action}++; } last; } @@ -5810,10 +8705,64 @@ $normalyzed_info{$normalized}{chronos}{"$cur_day_str"}{"$cur_hour_str"}{duration} += $cur_info{$t_pid}{duration}; # Store normalized query samples - &set_top_sample($normalized, $cur_info{$t_pid}{query}, $cur_info{$t_pid}{duration}, $overall_stat{'last_log_ts'},$cur_info{$t_pid}{dbname}, $cur_info{$t_pid}{dbuser}, $cur_info{$t_pid}{dbclient},$cur_info{$t_pid}{dbappname}); + &set_top_sample($normalized, $cur_info{$t_pid}{query}, $cur_info{$t_pid}{duration}, $cur_last_log_timestamp, $cur_info{$t_pid}{dbname}, $cur_info{$t_pid}{dbuser}, $cur_info{$t_pid}{dbclient},$cur_info{$t_pid}{dbappname}); } } } + +} + +sub store_temporary_and_lock_infos +{ + my $t_pid = shift; + + return if (!$t_pid); + + # Store normalized query temp file size if required + if (exists $cur_temp_info{$t_pid} && ($cur_temp_info{$t_pid}{query} ne '') && $cur_temp_info{$t_pid}{size}) { + + # Add a semi-colon at end of the query + $cur_temp_info{$t_pid}{query} .= ';' if (substr($cur_temp_info{$t_pid}{query}, -1, 1) ne ';'); + + # Normalize query + my $normalized = &normalize_query($cur_temp_info{$t_pid}{query}); + + $normalyzed_info{$normalized}{tempfiles}{size} += $cur_temp_info{$t_pid}{size}; + $normalyzed_info{$normalized}{tempfiles}{count}++; + + if ($normalyzed_info{$normalized}{tempfiles}{maxsize} < $cur_temp_info{$t_pid}{size}) { + $normalyzed_info{$normalized}{tempfiles}{maxsize} = $cur_temp_info{$t_pid}{size}; + } + if (!exists($normalyzed_info{$normalized}{tempfiles}{minsize}) + || $normalyzed_info{$normalized}{tempfiles}{minsize} > $cur_temp_info{$t_pid}{size}) { + $normalyzed_info{$normalized}{tempfiles}{minsize} = $cur_temp_info{$t_pid}{size}; + } + &set_top_tempfile_info($cur_temp_info{$t_pid}{query}, $cur_temp_info{$t_pid}{size}, $cur_temp_info{$t_pid}{timestamp}, $cur_temp_info{$t_pid}{dbname}, $cur_temp_info{$t_pid}{dbuser}, $cur_temp_info{$t_pid}{dbclient}, $cur_temp_info{$t_pid}{dbappname}); + delete $cur_temp_info{$t_pid}; + } + + # Store normalized query that waited the most if required + if (exists $cur_lock_info{$t_pid} && ($cur_lock_info{$t_pid}{query} ne '') && $cur_lock_info{$t_pid}{wait}) { + + # Add a semi-colon at end of the query + $cur_lock_info{$t_pid}{query} .= ';' if (substr($cur_lock_info{$t_pid}{query}, -1, 1) ne ';'); + + # Normalize query + my $normalized = &normalize_query($cur_lock_info{$t_pid}{query}); + + $normalyzed_info{$normalized}{locks}{wait} += $cur_lock_info{$t_pid}{wait}; + $normalyzed_info{$normalized}{locks}{count}++; + if ($normalyzed_info{$normalized}{locks}{maxwait} < $cur_lock_info{$t_pid}{wait}) { + $normalyzed_info{$normalized}{locks}{maxwait} = $cur_lock_info{$t_pid}{wait}; + } + if (!exists($normalyzed_info{$normalized}{locks}{minwait}) + || $normalyzed_info{$normalized}{locks}{minwait} > $cur_lock_info{$t_pid}{wait}) { + $normalyzed_info{$normalized}{locks}{minwait} = $cur_lock_info{$t_pid}{wait}; + } + &set_top_locked_info($cur_lock_info{$t_pid}{query}, $cur_lock_info{$t_pid}{wait}, $cur_lock_info{$t_pid}{timestamp}, $cur_lock_info{$t_pid}{dbname}, $cur_lock_info{$t_pid}{dbuser}, $cur_lock_info{$t_pid}{dbclient}, $cur_lock_info{$t_pid}{dbappname}); + delete $cur_lock_info{$t_pid}; + } + } # Normalize error messages @@ -5834,9 +8783,10 @@ $orig_query =~ s/\(.*\)/\(...\)/g; $orig_query =~ s/column .* does not exist/column "..." does not exist/; $orig_query =~ s/(database system was shut down at).*/$1 .../; + $orig_query =~ s/(relation) \d+ (deleted while still in use)/$1 ... $2/g; + $orig_query =~ s/[0-9A-F]{24}/.../g; # Remove WAL filename # Need more normalization stuff here - return $orig_query; } @@ -5846,16 +8796,15 @@ my $idx = shift; my @avgs = (); - for (my $i = 0 ; $i < 59 ; $i += $idx) { + for (my $i = 0 ; $i < 60 ; $i += $idx) { push(@avgs, sprintf("%02d", $i)); } - push(@avgs, 59); for (my $i = 0 ; $i <= $#avgs ; $i++) { if ($val == $avgs[$i]) { return "$avgs[$i]"; - } elsif ($avgs[$i] == $avgs[-1]) { - return "$avgs[$i-1]"; + } elsif ($i == $#avgs) { + return "$avgs[$i]"; } elsif (($val > $avgs[$i]) && ($val < $avgs[$i + 1])) { return "$avgs[$i]"; } @@ -5873,7 +8822,6 @@ my $fmt = ''; die "FATAL: can't open file $file, $!\n" unless(open(TESTFILE, $file)); - binmode(TESTFILE); my $fltf = ; close($fltf); # is file in binary format ? @@ -5899,7 +8847,7 @@ $ident_name{$1}++; } elsif ($line =~ - /^\d+-\d+-\d+T\d+:\d+:\d+(?:.[^\s]+)?\s[^\s]+\s([^\s\[]+)\[\d+\]:(?:\s\[[^\]]+\])?\s\[\d+\-\d+\].*?(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT):/ + /^\d+-\d+-\d+T\d+:\d+:\d+(?:.[^\s]+)?\s[^\s]+\s(?:[^\s]+\s)?([^\s\[]+)\[\d+\]:(?:\s\[[^\]]+\])?\s\[\d+\-\d+\].*?(LOG|WARNING|ERROR|FATAL|PANIC|DETAIL|STATEMENT|HINT|CONTEXT):/ ) { $fmt = 'syslog2'; @@ -5947,22 +8895,24 @@ $width ||= 25; $char ||= '='; my $num_width = length $total; + my $nchars = (($width - 1) * $got / $total); + $nchars = ($width - 1) if ($nchars >= $width); if ($extension eq 'tsung') { sprintf( "[%-${width}s] Parsed %${num_width}s bytes of %s (%.2f%%), queries: %d\r", - $char x (($width - 1) * $got / $total) . '>', + $char x $nchars . '>', $got, $total, 100 * $got / +$total, ($queries || $tsung_queries) ); - } elsif($format eq 'binary') { + } elsif ($format eq 'binary') { my $file = $_[-1]; sprintf( "Loaded %d queries and %d events from binary file %s...\r", - $overall_stat{'queries_number'}, $overall_stat{'errors_number'}, $queries + $overall_stat{'queries_number'}, $overall_stat{'errors_number'}, $file ); } else { sprintf( "[%-${width}s] Parsed %${num_width}s bytes of %s (%.2f%%), queries: %d, events: %d\r", - $char x (($width - 1) * $got / $total) . '>', + $char x $nchars . '>', $got, $total, 100 * $got / +$total, ($queries || $overall_stat{'queries_number'}), ($errors || $overall_stat{'errors_number'}) ); } @@ -5972,36 +8922,72 @@ { my ($buttonid, $divid, $data1, $data2, $data3, $title, $ytitle, $legend1, $legend2, $legend3, $ytitle2, $data4, $legend4) = @_; - $data1 = "var d1 = [$data1];" if ($data1); - $data2 = "var d2 = [$data2];" if ($data2); - $data3 = "var d3 = [$data3];" if ($data3); - $data4 = "var d4 = [$data4];" if ($data4); - - $legend1 = "{ data: d1, label: \"$legend1\" }," if ($legend1); - $legend2 = "{ data: d2, label: \"$legend2\" }," if ($legend2); - $legend3 = "{ data: d3, label: \"$legend3\" }," if ($legend3); - $legend4 = "{ data: d4, label: \"$legend4\",yaxis: 2 }," if ($legend4); - + if (!$data1) { + return qq{ +
    NO DATASET
    +}; + } + my $dateTracker_lblopts = ''; + if ($legend1) { + $dateTracker_lblopts .= "'$legend1',"; + $legend1 = "{ data: d1, label: \"$legend1\", color: \"#6e9dc9\", mouse:{track:true}},"; + } + if ($legend2) { + $dateTracker_lblopts .= "'$legend2',"; + $legend2 = "{ data: d2, label: \"$legend2\", color: \"#f4ab3a\", mouse:{track:true}},"; + } + if ($legend3) { + $dateTracker_lblopts .= "'$legend3',"; + $legend3 = "{ data: d3, label: \"$legend3\", color: \"#ac7fa8\", mouse:{track:true}},"; + } + if ($legend4) { + $dateTracker_lblopts .= "'$legend4',"; + $legend4 = "{ data: d4, label: \"$legend4\", color: \"#8dbd0f\",yaxis: 2},"; + } + $dateTracker_lblopts =~ s/,$//; + $dateTracker_lblopts = "[$dateTracker_lblopts]"; + + my $dateTracker_dataopts = ''; + if ($data1) { + $data1 = "var d1 = [$data1];"; + $dateTracker_dataopts .= "d1,"; + } + if ($data2) { + $data2 = "var d2 = [$data2];"; + $dateTracker_dataopts .= "d2,"; + } + if ($data3) { + $data3 = "var d3 = [$data3];"; + $dateTracker_dataopts .= "d3,"; + } + if ($data4) { + $data4 = "var d4 = [$data4];"; + $dateTracker_dataopts .= "d4,"; + } + $dateTracker_dataopts =~ s/,$//; + $dateTracker_dataopts = "[$dateTracker_dataopts]"; + my $yaxis2 = ''; if ($ytitle2) { - $yaxis2 = "y2axis: { title: \"$ytitle2\", min: 0, color: \"#4DA74D\" },"; + $yaxis2 = "y2axis: { mode: \"normal\", title: \"$ytitle2\", min: 0, color: \"#8dbd0f\" },"; } - my $min = $t_min; - my $max = $t_max; - if ($divid !~ /persecond/) { - $min = $t_min_hour; - $max = $t_max_hour; + my $type = ''; + if ($ytitle eq 'Size of files') { + $type = 'size'; + } elsif ($ytitle eq 'Duration') { + $type = 'duration'; } - print $fh < + + return < +EOF + +} + +sub flotr2_histograph +{ + my ($buttonid, $divid, $data1, $data2) = @_; + + if (!$data1) { + return qq{ +
    NO DATASET
    +}; + } + my $legend1 = 'Avg. queries'; + my $legend2 = ''; + my $dateTracker_lblopts = "'$legend1',"; + $legend1 = "{ data: d1, label: \"$legend1\", color: \"#6e9dc9\", mouse:{track:true}, bars: {show: true,shadowSize: 0}, },"; + if ($data2) { + $legend2 = 'Avg. duration'; + $dateTracker_lblopts .= "'$legend2'"; + $legend2 = "{ data: d2, label: \"$legend2\", color: \"#8dbd0f\", yaxis: 2},"; + } + $dateTracker_lblopts =~ s/,$//; + $dateTracker_lblopts = "[$dateTracker_lblopts]"; + + my $dateTracker_dataopts = ''; + if ($data1) { + $data1 = "var d1 = [$data1];"; + $dateTracker_dataopts .= "d1,"; + } + if ($data2) { + $data2 = "var d2 = [$data2];"; + $dateTracker_dataopts .= "d2"; + } + $dateTracker_dataopts =~ s/,$//; + $dateTracker_dataopts = "[$dateTracker_dataopts]"; + + my $yaxis2 = "y2axis: { mode: \"normal\", title: \"Duration\", min: 0, color: \"#8dbd0f\", tickFormatter: function(val){ return pretty_print_number(val,'duration') }, },"; + $yaxis2 = '' if (!$data2); + + return < + @@ -6189,11 +9339,12 @@ sub build_log_line_prefix_regex { my %regex_map = ( - '%a' => [('t_appname', '([0-9a-zA-Z\.\-\_\/\[\]]*)')], # application name + #'%a' => [('t_appname', '([0-9a-zA-Z\.\-\_\/\[\]]*)')], # application name + '%a' => [('t_appname', '(.*)')], # application name '%u' => [('t_dbuser', '([0-9a-zA-Z\_\[\]\-]*)')], # user name '%d' => [('t_dbname', '([0-9a-zA-Z\_\[\]\-]*)')], # database name - '%r' => [('t_hostport', '([a-zA-Z0-9\-\.]+|\[local\]|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})?[\(\d\)]*')], # remote host and port - '%h' => [('t_client', '([a-zA-Z0-9\-\.]+|\[local\]|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})?')], # remote host + '%r' => [('t_hostport', '([a-zA-Z0-9\-\.]+|\[local\]|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|[0-9a-fA-F:]+)?[\(\d\)]*')], # remote host and port + '%h' => [('t_client', '([a-zA-Z0-9\-\.]+|\[local\]|\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|[0-9a-fA-F:]+)?')], # remote host '%p' => [('t_pid', '(\d+)')], # process ID '%t' => [('t_timestamp', '(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})(?: [A-Z\d]{3,6})?')], # timestamp without milliseconds '%m' => [('t_mtimestamp', '(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\.\d+(?: [A-Z\d]{3,6})?')], # timestamp with milliseconds @@ -6300,7 +9451,7 @@ (?:[\w:@]+(?:\.(?:\w+|\*)?)*) # words, standard named placeholders, db.table.*, db.* | - (?: \$_\$ | \$\d+ | \${1,2} ) + (?: \$_\$ | \$\d+ | \${1,2}) # dollar expressions - eg $_$ $3 $$ | \n # newline @@ -6690,11 +9841,13 @@ # get file size my $totalsize = (stat("$logf"))[7] || 0; + my $iscompressed = 1; # Open a file handle if ($logf !~ /\.(gz|bz2|zip)/i) { open($lfile, $logf) || die "FATAL: cannot read log file $logf. $!\n"; $totalsize = 0 if ($lfile eq '-'); + $iscompressed = 0; } else { my $uncompress = $zcat; if (($logf =~ /\.bz2/i) && ($zcat =~ /^$zcat_cmd$/)) { @@ -6719,15 +9872,12 @@ $totalsize = `$cmd_file_size`; chomp($totalsize); } - if ($queue_size) { - $job_per_file = $queue_size; - $queue_size = 0; - } + $queue_size = 0; } # In list context returns the filehandle and the size of the file if (wantarray()) { - return ($lfile, $totalsize); + return ($lfile, $totalsize, $iscompressed); } # In scalar context return size only close($lfile); @@ -6755,18 +9905,20 @@ $cmd_file_size =~ s/\%f/$logf/g; $totalsize = `$cmd_file_size`; chomp($totalsize); - if ($queue_size) { - $job_per_file = $queue_size; - $queue_size = 0; - } + $queue_size = 0; } elsif ($logf =~ /\.bz2/i) { $totalsize = 0; + $queue_size = 0; } return (0, -1) if (!$totalsize); my @chunks = (0); my $i = 1; + if ($last_parsed && $saved_last_line{current_pos} && ($#given_log_files == 0)) { + $chunks[0] = $saved_last_line{current_pos}; + $i = $saved_last_line{current_pos}; + } while ($i < $queue_size) { push(@chunks, int(($totalsize/$queue_size) * $i)); $i++; @@ -6776,8 +9928,139 @@ return @chunks; } +# Return the week number of the year for a given date +sub get_week_number +{ + my ($year, $month, $day) = @_; + +# %U The week number of the current year as a decimal number, range 00 to 53, starting with the first +# Sunday as the first day of week 01. +# %V The ISO 8601 week number (see NOTES) of the current year as a decimal number, range 01 to 53, +# where week 1 is the first week that has at least 4 days in the new year. +# %W The week number of the current year as a decimal number, range 00 to 53, starting with the first +# Monday as the first day of week 01. + + # Check if the date is valide first + my $datefmt = POSIX::strftime("%F", 1, 1, 1, $day, $month - 1, $year - 1900); + if ($datefmt ne "$year-$month-$day") { + return -1; + } + my $weekNumber = POSIX::strftime("%W", 1, 1, 1, $day, $month - 1, $year - 1900); + + return sprintf("%02d", $weekNumber+1); +} + +# Returns day number of the week of a given days +sub get_day_of_week +{ + my ($year, $month, $day) = @_; + +# %u The day of the week as a decimal, range 1 to 7, Monday being 1. +# %w The day of the week as a decimal, range 0 to 6, Sunday being 0. + + my $weekDay = POSIX::strftime("%u", 1,1,1,$day,--$month,$year-1900); + + return $weekDay; +} + +# Returns all days following the week number +sub get_wdays_per_month +{ + my $wn = shift; + my ($year, $month) = split(/\-/, shift); + my @months = (); + my @retdays = (); + + $month ||= '01'; + push(@months, "$year$month"); + my $start_month = $month; + if ($month eq '01') { + unshift(@months, ($year - 1) . "12"); + } else { + unshift(@months, $year . sprintf("%02d", $month - 1)); + } + if ($month == 12) { + push(@months, ($year+1) . "01"); + } else { + push(@months, $year . sprintf("%02d", $month + 1)); + } + + foreach my $d (@months) { + $d =~ /^(\d{4})(\d{2})$/; + my $y = $1; + my $m = $2; + foreach my $day ("01" .. "31") { + # Check if the date is valide first + my $datefmt = POSIX::strftime("%F", 1, 1, 1, $day, $m - 1, $y - 1900); + if ($datefmt ne "$y-$m-$day") { + next; + } + my $weekNumber = POSIX::strftime("%W", 1, 1, 1, $day, $m - 1, $y - 1900); + if ( ($weekNumber == $wn) || ( ($weekNumber eq '00') && (($wn == 1) || ($wn >= 52)) ) ) { + push(@retdays, "$year-$m-$day"); + return @retdays if ($#retdays == 6); + } + next if ($weekNumber > $wn); + } + } + + return @retdays; +} + +# Returns all days following the week number +sub get_wdays_per_year +{ + my $y = shift; + my %result = (); + + foreach my $m ("01" .. "12") { + foreach my $day ("01" .. "31") { + # Check if the date is valide first + my $datefmt = POSIX::strftime("%F", 1, 1, 1, $day, $m - 1, $y - 1900); + if ($datefmt ne "$y-$m-$day") { + next; + } + my $weekNumber = POSIX::strftime("%W", 1, 1, 1, $day, $m - 1, $y - 1900); + push(@{$result{$weekNumber}}, "$y-$m-$day"); + } + } + + return %result; +} + + + __DATA__ + + + + + + + + + + diff -Nru pgbadger-3.3/README pgbadger-5.0/README --- pgbadger-3.3/README 2013-05-01 16:03:04.000000000 +0000 +++ pgbadger-5.0/README 2014-02-06 15:29:04.000000000 +0000 @@ -25,6 +25,8 @@ -G | --nograph : disable graphs on HTML output. Enable by default. -h | --help : show this message and exit. -i | --ident name : programname used as syslog ident. Default: postgres + -I | --incremental : use incremental mode, reports will be generated by + days in a separate directory, --outdir must be set. -j | --jobs number : number of jobs to run on parallel on each log file. Default is 1, run as single process. -J | --Jobs number : number of log file to parse in parallel. Default @@ -40,6 +42,7 @@ -o | --outfile filename: define the filename for output. Default depends on the output format: out.html, out.txt or out.tsung. To dump output to stdout use - as filename. + -O | --outdir path : directory where out file must be saved. -p | --prefix string : give here the value of your custom log_line_prefix defined in your postgresql.conf. Only use it if you aren't using one of the standard prefixes specified @@ -47,7 +50,7 @@ includes additional variables like client ip or application name. See examples below. -P | --no-prettify : disable SQL queries prettify formatter. - -q | --quiet : don't print anything to stdout, even not a progress bar. + -q | --quiet : don't print anything to stdout, not even a progress bar. -s | --sample number : number of query samples to store/display. Default: 3 -S | --select-only : use it if you want to report select queries only. -t | --top number : number of queries to store/display. Default: 20 @@ -82,6 +85,13 @@ --disable-temporary : do not generate temporary report. --disable-checkpoint : do not generate checkpoint report. --disable-autovacuum : do not generate autovacuum report. + --charset : used to set the HTML charset to be used. Default: utf-8. + --csv-separator : used to set the CSV field separator, default: , + --exclude-time regex : any timestamp matching the given regex will be + excluded from the report. Example: "2013-04-12 .*" + You can use this option multiple times. + --exclude-appname name : exclude entries for the specified application name + from report. Example: "pg_dump". Examples: @@ -119,29 +129,58 @@ This supposes that your log file and HTML report are also rotated every week. + Or better, use the auto-generated incremental reports: + + 0 4 * * * /usr/bin/pgbadger -I -q /var/log/postgresql/postgresql.log.1 \ + -O /var/www/pg_reports/ + + will generate a report per day and per week in the given output + directory. + + If you have a pg_dump at 23:00 and 13:00 each day during half an hour, + you can use pgbadger as follow to exclude these periods from the report: + + pgbadger --exclude-time "2013-09-.* (23|13):.*" postgresql.log + + This will help to not have all COPY order on top of slowest queries. You + can also use --exclude-appname "pg_dump" to solve this problem in a more + simple way. + DESCRIPTION - pgBadger is a PostgreSQL log analyzer built for speed with fully + pgBadger is a PostgreSQL log analyzer build for speed with fully detailed reports from your PostgreSQL log file. It's a single and small - Perl script that aims to replace and out-perform the old PHP script - pgFouine. + Perl script that outperform any other PostgreSQL log analyzer. - By the way, we would like to thank Guillaume Smet for all the work he - has done on this really nice tool. We've been using it a long time, it - is a really great tool! - - pgBadger is written in pure Perl language. It uses a Javascript library - to draw graphs so that you don't need additional Perl modules or any - other package to install. Furthermore, this library gives us additional - features, such as zooming. + It is written in pure Perl language and uses a javascript library + (flotr2) to draw graphs so that you don't need to install any additional + Perl modules or other packages. Furthermore, this library gives us more + features such as zooming. pgBadger also uses the Bootstrap javascript + library and the FontAwesome webfont for better design. Everything is + embedded. pgBadger is able to autodetect your log file format (syslog, stderr or - csvlog). It is designed to parse huge log files, as well as gzip, zip or - bzip2 compressed files. See a complete list of features below. + csvlog). It is designed to parse huge log files as well as gzip + compressed file. See a complete list of features below. + + All charts are zoomable and can be saved as PNG images. + + You can also limit pgBadger to only report errors or remove any part of + the report using command line options. + + pgBadger supports any custom format set into log_line_prefix of your + postgresql.conf file provide that you use the %t, %p and %l patterns. + + pgBadger allow parallel processing on a single log file and multiple + files through the use of the -j option and the number of CPUs as value. + + If you want to save system performance you can also use log_duration + instead of log_min_duration_statement to have reports on duration and + number of queries only. FEATURE pgBadger reports everything about your SQL queries: - Overall statistics. + Overall statistics The most frequent waiting queries. Queries that waited the most. Queries generating the most temporary files. @@ -150,15 +189,20 @@ Queries that took up the most time. The most frequent queries. The most frequent errors. + Histogram of query times. - The following reports are also available with hourly charts: + The following reports are also available with hourly charts divide by + periods of five minutes: + + SQL queries statistics. + Temporary file statistics. + Checkpoints statistics. + Autovacuum and autoanalyze statistics. + + There's also some pie reports of distribution about: - Hourly queries statistics. - Hourly temporary file statistics. - Hourly checkpoints statistics. - Hourly restartpoints statistics. Locks statistics. - Queries by type (select/insert/update/delete). + ueries by type (select/insert/update/delete). Distribution of queries type per database/application Sessions per database/user/client. Connections per database/user/client. @@ -167,6 +211,9 @@ All charts are zoomable and can be saved as PNG images. SQL queries reported are highlighted and beautified automatically. + You can also have incremental reports with one report per day and a + cumulative report per week. + REQUIREMENT pgBadger comes as a single Perl script - you do not need anything other than a modern Perl distribution. Charts are rendered using a Javascript @@ -200,6 +247,29 @@ Note that multiprocessing can not be used with compressed files or CSV files as well as under Windows platform. +INSTALLATION + Download the tarball from github and unpack the archive as follow: + + tar xzf pgbadger-4.x.tar.gz + cd pgbadger-4.x/ + perl Makefile.PL + make && sudo make install + + This will copy the Perl script pgbadger to /usr/local/bin/pgbadger by + default and the man page into /usr/local/share/man/man1/pgbadger.1. + Those are the default installation directories for 'site' install. + + If you want to install all under /usr/ location, use INSTALLDIRS='perl' + as an argument of Makefile.PL. The script will be installed into + /usr/bin/pgbadger and the manpage into /usr/share/man/man1/pgbadger.1. + + For example, to install everything just like Debian does, proceed as + follows: + + perl Makefile.PL INSTALLDIRS=vendor + + By default INSTALLDIRS is set to site. + POSTGRESQL CONFIGURATION You must enable and set some configuration directives in your postgresql.conf before starting. @@ -211,7 +281,8 @@ Here every statement will be logged, on busy server you may want to increase this value to only log queries with a higher duration time. Note that if you have log_statement set to 'all' nothing will be logged - with log_line_prefix. See next chapter for more information. + through log_min_duration_statement. See next chapter for more + information. With 'stderr' log format, log_line_prefix must be at least: @@ -241,6 +312,7 @@ log_disconnections = on log_lock_waits = on log_temp_files = 0 + log_autovacuum_min_duration = 0 Do not enable log_statement as their log format will not be parsed by pgBadger. @@ -267,7 +339,7 @@ have log_statement set to 'all' nothing will be logged with log_line_prefix. -Parallel processing +PARALLEL PROCESSING To enable parallel processing you just have to use the -j N option where N is the number of cores you want to use. @@ -322,37 +394,79 @@ are all named with the following template tmp_pgbadgerXXXX.bin so they can be easily identified. -INSTALLATION - Download the tarball from github and unpack the archive as follow: +INCREMENTAL REPORTS + pgBadger include an automatic incremental report mode using option -I or + --incremental. When running in this mode, pgBadger will generate one + report per day and a cumulative report per week. Output is first done in + binary format into the mandatory output directory (see option -O or + --outdir), then in HTML format for daily and weekly reports with a main + index file. - tar xzf pgbadger-3.x.tar.gz - cd pgbadger-3.x/ - perl Makefile.PL - make && sudo make install + The main index file will show a dropdown menu per week with a link to + the week report and links to daily reports of this week. - This will copy the Perl script pgbadger to /usr/local/bin/pgbadger by - default and the man page into /usr/local/share/man/man1/pgbadger.1. - Those are the default installation directories for 'site' install. + For example, if you run pgBadger as follow based on a daily rotated + file: - If you want to install all under /usr/ location, use INSTALLDIRS='perl' - as an argument of Makefile.PL. The script will be installed into - /usr/bin/pgbadger and the manpage into /usr/share/man/man1/pgbadger.1. + 0 4 * * * /usr/bin/pgbadger -I -q /var/log/postgresql/postgresql.log.1 \ + -O /var/www/pg_reports/ - For example, to install everything just like Debian does, proceed as - follows: + you will have all daily and weekly reports for the full running period. - perl Makefile.PL INSTALLDIRS=vendor + In this mode pgBagder will create an automatic incremental file into the + output directory, so you don't have to use the -l option unless you want + to change the path of that file. This mean that you can run pgBadger in + this mode each days on a log file rotated each week, it will not count + the log entries twice. - By default INSTALLDIRS is set to site. +BINARY FORMAT + Using the binary format it is possible to create custom incremental and + cumulative reports. For example, if you want to refresh a pgbadger + report each hour from a daily PostgreSQl log file, you can proceed by + running each hour the following commands: + + pgbadder --last-parsed .pgbadger_last_state_file -o sunday/hourX.bin /var/log/pgsql/postgresql-Sun.log + + to generate the incremental data files in binary format. And to generate + the fresh HTML report from that binary file: + + pgbadder sunday/*.bin + + Or an other example, if you have one log file per hour and you want a + reports to be rebuild each time the log file is switched. Proceed as + follow: + + pgbadger -o day1/hour01.bin /var/log/pgsql/pglog/postgresql-2012-03-23_10.log + pgbadger -o day1/hour02.bin /var/log/pgsql/pglog/postgresql-2012-03-23_11.log + pgbadger -o day1/hour03.bin /var/log/pgsql/pglog/postgresql-2012-03-23_12.log + ... + + When you want to refresh the HTML report, for example each time after a + new binary file is generated, just do the following: + + pgbadger -o day1_report.html day1/*.bin + + Adjust the commands following your needs. AUTHORS - pgBadger is an original work from Gilles Darold. It is maintained by the - good folk at Dalibo and everyone who wants to contribute. + pgBadger is an original work from Gilles Darold. + + The pgBadger logo is an original creation of Damien Clochard. + + The pgBadger v4.x design comes from the "Art is code" company. + + This web site is a work of Gilles Darold. + + pgBadger is maintained by Gilles Darold, the good folks at Dalibo, and + every one who wants to contribute. + + Many people have contributed to pgBadger, they are all quoted in the + Changelog file. LICENSE pgBadger is free software distributed under the PostgreSQL Licence. - Copyright (c) 2012-2013, Dalibo + Copyright (c) 2012-2014, Dalibo A modified version of the SQL::Beautify Perl Module is embedded in pgBadger with copyright (C) 2009 by Jonas Kramer and is published under