apache using large amounts of CPU

Automatically imported from Debian bug report #286604 http://bugs.debian.org/286604

Message-ID: <email address hidden>
Date: Tue, 21 Dec 2004 14:48:19 +1100 (EST)
From: PaulWB <email address hidden>
To: <email address hidden>
Subject: apache using large amounts of CPU

package: apache
version: 1.3.33-2
severity: critical

A module or package has been upgraded recently and is creating large
problems for apache

I attempted to downgrade php and apache to find out what was causing the
problem, this did not help, I have included a trace of the running child
apache process that is basically hammering my system.

In conclusion, something is severly CPU bound and basically rendering
apache useless and impacting on every other application until it times
out, it could be php related, however to get the process running.

I do have the same website appliction running on 3 separate servers and
all was working well, this one system died after I did an apt-get
dist-upgrade, the other two servers are running older versions of
apache/php.

regards
Paul.

Process 10487 attached - interrupt to quit
--- SIGALRM (Alarm clock) @ 0 (0) ---
sigreturn() = ? (mask now [])
--- SIGPROF (Profiling timer expired) @ 0 (0) ---
setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={240, 0}}, NULL) = 0
rt_sigaction(SIGPROF, {0xb7708830, [PROF], SA_RESTART}, {0xb7708830,
[PROF], SA_RESTART}, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0
mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE,
-1, 0) = 0xb7249000
munmap(0xb7249000, 749568) = 0
munmap(0xb7400000, 299008) = 0
mprotect(0xb7300000, 135168, PROT_READ|PROT_WRITE) = 0
open("/local/logs/php", O_WRONLY|O_APPEND|O_CREAT, 0666) = 7
fstat64(7, {st_mode=S_IFREG|0777, st_size=58064679, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0xb7448000
fstat64(7, {st_mode=S_IFREG|0777, st_size=58064679, ...}) = 0
_llseek(7, 58064679, [58064679], SEEK_SET) = 0
time([1103599287]) = 1103599287
write(7, "[21-Dec-2004 14:21:27] PHP Fatal"..., 142) = 142
close(7) = 0
munmap(0xb7448000, 4096) = 0
chdir("/") = 0
futex(0xb7e76620, FUTEX_WAIT, 2, NULL

Message-ID: <email address hidden>
Date: Tue, 21 Dec 2004 06:56:57 +0100
From: Fabio Massimo Di Nitto <email address hidden>
To: PaulWB <email address hidden>, <email address hidden>
Cc: <email address hidden>
Subject: Re: Bug#286604: apache using large amounts of CPU

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

severity 286604 important
reassign 286604 php4
thanks

PaulWB wrote:
|
|
| package: apache
| version: 1.3.33-2
| severity: critical
|
|
| A module or package has been upgraded recently and is creating large
| problems for apache

if a module has been upgraded that creates problems for apache, it is not
an apache fault.

|
| Process 10487 attached - interrupt to quit
| --- SIGALRM (Alarm clock) @ 0 (0) ---
| sigreturn() = ? (mask now [])
| --- SIGPROF (Profiling timer expired) @ 0 (0) ---
| setitimer(ITIMER_PROF, {it_interval={0, 0}, it_value={240, 0}}, NULL) = 0
| rt_sigaction(SIGPROF, {0xb7708830, [PROF], SA_RESTART}, {0xb7708830,
| [PROF], SA_RESTART}, 8) = 0
| rt_sigprocmask(SIG_UNBLOCK, [PROF], NULL, 8) = 0
| mmap2(NULL, 2097152, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE,
| -1, 0) = 0xb7249000
| munmap(0xb7249000, 749568) = 0
| munmap(0xb7400000, 299008) = 0
| mprotect(0xb7300000, 135168, PROT_READ|PROT_WRITE) = 0
| open("/local/logs/php", O_WRONLY|O_APPEND|O_CREAT, 0666) = 7
| fstat64(7, {st_mode=S_IFREG|0777, st_size=58064679, ...}) = 0
| mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
| 0) = 0xb7448000
| fstat64(7, {st_mode=S_IFREG|0777, st_size=58064679, ...}) = 0
| _llseek(7, 58064679, [58064679], SEEK_SET) = 0
| time([1103599287]) = 1103599287
| write(7, "[21-Dec-2004 14:21:27] PHP Fatal"..., 142) = 142

This call looks very suspicious.. why should it be Fatal?

| close(7) = 0
| munmap(0xb7448000, 4096) = 0
| chdir("/") = 0
| futex(0xb7e76620, FUTEX_WAIT, 2, NULL
|
|

Fabio

- --
Self-Service law:
The last available dish of the food you have decided to eat, will be
inevitably taken from the person in front of you.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBx7snhCzbekR3nhgRAvgyAKCd9fzoYhAHrqqVfgvsZkp2q/LwQgCffhfB
hNZlnREWiF3OfulzmG4NV54=
=pxhD
-----END PGP SIGNATURE-----

Message-ID: <email address hidden>
Date: Wed, 22 Dec 2004 19:52:19 +1100 (EST)
From: PaulWB <email address hidden>
To: <email address hidden>
Subject: update

I have run a php-apd trace and the problem appears to be narrowed down to
the str_replace function, str_replace was called by is function is_array

Message-ID: <email address hidden>
Date: Sun, 26 Dec 2004 12:34:30 +1100 (EST)
From: PaulWB <email address hidden>
To: <email address hidden>
Subject: 4.3.10 backout

I completely removed 4.3.10 from my system and all is back to normal,
something is very wrong with 4.3.10 and us admin people need more time to
know the implications of what has been impacted.

In conclusion 4.3.10 should never have been released into the mainstream
environment.

Message-Id: <1104392219.11411.45.camel@localhost>
Date: Thu, 30 Dec 2004 02:36:59 -0500
From: Andres Salomon <email address hidden>
To: <email address hidden>
Subject: Re: apache using large amounts of CPU

--=-KI/hpYgIg6X7Fqf0fpXx
Content-Type: text/plain
Content-Transfer-Encoding: quoted-printable

retitle 286604 php4: str_replace eating processor cycles
thanks

Hi,

Can you please provide a snippet of code that triggers this, so we can
reproduce and fix it? Also, specifically which debian php4 packages are
you seeing this with? What extensions are loaded?

--=20
Andres Salomon <email address hidden>

--=-KI/hpYgIg6X7Fqf0fpXx
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQBB07Ab78o9R9NraMQRAjjXAJ4vxJ0Uvd2nb286sgwgsEoMLtfPeQCgjKBM
34k9VyC4a1MAVW6CNXsdRgg=
=qf+W
-----END PGP SIGNATURE-----

--=-KI/hpYgIg6X7Fqf0fpXx--

Message-ID: <email address hidden>
Date: Fri, 31 Dec 2004 22:50:56 +1100 (EST)
From: PaulWB <email address hidden>
To: <email address hidden>
Subject: php 4.3.10 problems

I am attempting to locate the code that is the center of the problem,
alot of it is libraries and following the flow is a nightmare at the best
of times.

I have experienced this problem on Redhat Fedora core 2 and Debian
Unstable

I downgraded the following list of packages and the site worked fine

libapache2-mod-php4_4%3a4.3.9-2_i386.deb
libapache-mod-php4_4%3a4.3.9-2_i386.deb
php4_4%3a4.3.9-2_all.deb
php4-cgi_4%3a4.3.9-2_i386.deb
php4-cli_4%3a4.3.9-2_i386.deb
php4-common_4%3a4.3.9-2_i386.deb
php4-curl_4%3a4.3.9-2_i386.deb
php4-domxml_4%3a4.3.9-2_i386.deb
php4-gd_4%3a4.3.9-2_i386.deb
php4-imap_4%3a4.3.9-2_i386.deb
php4-mhash_4%3a4.3.9-2_i386.deb
php4-mysql_4%3a4.3.9-2_i386.deb
php4-pear_4%3a4.3.9-2_all.deb
php4-xslt_4%3a4.3.9-2_i386.deb

after the upgrade things break when these packages are loaded
libapache2-mod-php4_4%3a4.3.10-2_i386.deb
libapache-mod-php4_4%3a4.3.10-2_i386.deb
php4_4%3a4.3.10-2_all.deb
php4-cgi_4%3a4.3.10-2_i386.deb
php4-cli_4%3a4.3.10-2_i386.deb
php4-common_4%3a4.3.10-2_i386.deb
php4-curl_4%3a4.3.10-2_i386.deb
php4-domxml_4%3a4.3.10-2_i386.deb
php4-gd_4%3a4.3.10-2_i386.deb
php4-imap_4%3a4.3.10-2_i386.deb
php4-mhash_4%3a4.3.10-2_i386.deb
php4-mysql_4%3a4.3.10-2_i386.deb
php4-pear_4%3a4.3.10-2_all.deb
php4-xslt_4%3a4.3.10-2_i386.deb

Apache is
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Installed/Config-files/Unpacked/Failed-config/Half-installed
|/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err:
uppercase=bad)
||/ Name Version
Description
+++-=============================-=============================-==========================================================================
ii apache 1.3.33-2 Versatile,
high-performance HTTP server

Pear is

Installed packages:
===================
Package Version State
Archive_Tar 1.1 stable
Auth 1.2.3 stable
Auth_HTTP 2.0 stable
Auth_PrefManager 1.1.3 stable
Auth_SASL 1.0.1 stable
Benchmark 1.2.1 stable
Cache 1.5.4 stable
Cache_Lite 1.3.1 stable
Config 1.10.3 stable
Console_Getargs 1.2.1 stable
Console_Getopt 1.2 stable
Console_Table 1.0.1 stable
Contact_Vcard_Build 1.1 stable
Contact_Vcard_Parse 1.30 stable
Crypt_CBC 0.4 stable
Crypt_RC4 1.0.2 stable
Crypt_Xtea 1.0 stable
DB 1.6.2 stable
DBA 1.1 stable
DB_DataObject 1.7.2 stable
DB_NestedSet 1.2.4 stable
DB_Pager 0.7 stable
DB_QueryTool 0.11.1 stable
DB_ldap 1.1.0 stable
Date 1.4.3 stable
FSM 1.2.1 stable
File 1.0....

Message-ID: <email address hidden>
Date: Sat, 1 Jan 2005 17:22:52 +1100 (EST)
From: PaulWB <email address hidden>
To: <email address hidden>
Subject: 4.3.10

I have done a normal system update on the debian system and the site is
now working on that system, the redhat system is playing up but that
should not be hard to fix now that things are working on one system.

Thanks for your help,
regards
Paul.

Looks like a bug in a version which never entered Ubuntu

Message-ID: <email address hidden>
Date: Fri, 7 Jan 2005 14:54:25 +1100 (EST)
From: Paul <email address hidden>
To: <email address hidden>
Subject: still having problems

I am no closer to finding the cause of my php problems, I thought I had
it working so then I perform a norm system upgrade and it breaks again.

Message-ID: <email address hidden>
Date: Sat, 9 Apr 2005 12:45:17 +1000 (EST)
From: "Adam Conrad" <adconrad@0c3.net>
To: <email address hidden>
Subject: php4: str_replace eating processor cycles

Can you still reproduce this problem with the latest (4:4.3.10-10)
unstable php4 packages?

... Adam

Message-ID: <email address hidden>
Date: Sun, 10 Apr 2005 23:26:24 +1000 (EST)
From: Paul <email address hidden>
To: adconrad@0c3.net, <email address hidden>
cc: <email address hidden>, Debian BTS <email address hidden>
Subject: Re: Bug#286604: php4: str_replace eating processor cycles

Hi Adam

I had the php application on hold for ages, I removed that hold and
upgraded to the latest version 4.3.10-10 and it appears to work normal.

The application was hard to trace as a a large number of php scripts
interact with each other and the php functions.

In conclusion from what I can tell the problem is no longer occuring,
thanking you,

Paul.

On Sat, 9 Apr 2005, Adam Conrad wrote:

> Can you still reproduce this problem with the latest (4:4.3.10-10)
> unstable php4 packages?
>
> ... Adam
>
>
>
>

Message-ID: <email address hidden>
Date: Sun, 10 Apr 2005 23:55:16 +1000 (EST)
From: "Adam Conrad" <adconrad@0c3.net>
To: <email address hidden>
Subject: Re: Bug#286604: php4: str_replace eating processor cycles

Paul said:
>
> In conclusion from what I can tell the problem is no longer occuring,
> thanking you,

Closing, then...

... Adam