php: client.c:773: gearman_client_run_tasks: Assertion `client->task != ((void *)0)`failed.

Bug #518512 reported by RdeWilde
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Gearman
Fix Released
High
Eric Day

Bug Description

I'm using libgearman 0.11 with PECL PHP extension gearman 0.6.0 on CentOS 5.4 using Apache 2.2 and PHP 5.3.

While running a Gearman Worker I got this error:
php: client.c:773: gearman_client_run_tasks: Assertion `client->task != ((void *)0)`failed.

Context; I made this PHP script:
http://pastie.org/private/jzm9ox4kogpllz9onl7ma

When I start one worker, everything goes smooth. When I start an additional second worker, no problem. When I start the third worker, the first worker crashes with the error above. At the moment the third worker starts, it registers to the first worker (addNeighbour). This 'addNeighbour' will call the 'updateNeighbour' function on the first Worker itself. In the loop, the IF-statement will return false for the first time, so the else-part is executed. When it comes to #110 the crash occurs.

Little strange to me is that after the first worker crashed, I can start as many additional workers as I like... they won't crash.

This bug keeps me from using Gearman in PHP at the moment. Thanks in advance!

Related branches

Revision history for this message
RdeWilde (robert-de-wilde) wrote :
Revision history for this message
Eric Day (eday) wrote :

The problem is due to the following condition:

client_do - job X
timeout, don't process result
client_do - job Y
get result for job handle X
since client_do reuses data structures (only 1 active job at a time), context for job X is lost
cannot find job X, hit assert()

I think the best fix here is to remove the assert and simply ignore the result packets if it is no longer found in the active client task list.

Changed in gearmand:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Wilton (risenhoover) wrote :

I'm getting this same bug on a debian distribution, with gearmand-0.12 and php gearman-0.7.0.

papa:~# uname -a
Linux papa 2.6.26-2-amd64 #1 SMP Thu Feb 11 00:59:32 UTC 2010 x86_64 GNU/Linux

When I try to run a php script that starts a client, I get the following:

papa:~/src# php /var/www/joomla/getstocks.php

php: client.c:773: gearman_client_run_tasks: Assertion `client->task != ((void *)0)' failed.
Aborted

I tried modifying the source according to the above comments, but I haven't been able to get it to work. I might have done it incorrectly, but I searched the gearmand-0.12 codeline for a client.c and found one in libgearman. I then commented out two lines that had assertions:

//assert(client->task != NULL);

I then did make; make install, and then went to the php-gearman (0.7) project, recompiled and reinstalled all *that* code, and tried my test again. This time, I did *not* get the same assertion error.

The php script seems to be doing something now, but I'm not sure what. I'll post more once I learn more.

Revision history for this message
Wilton (risenhoover) wrote :

Now it just says "failed."

papa:~/src/gearman-0.7.0# php /var/www/joomla/getstocks.php

Failed

which means I'm getting a GEARMAN_WORK_FAIL from the script. Any thoughts?

Revision history for this message
Wilton (risenhoover) wrote :

I went back to the code and made another modification in the hopes that this might fix it. Just below one of the asserts I added an if/then block as such:

            //assert(client->task != NULL);

            if ( client->task != NULL ) {
              client->task->recv= &(client->con->packet);
            }

This, unfortunately, hasn't changed anything. I still get a failure when I try to execute my script. Here's a detailed debug log for what it's worth...

pauly@papa:~$ /usr/local/sbin/gearmand -vvvv
 INFO Starting up
DEBUG Initializing libevent for main thread
DEBUG Method for libevent: epoll
DEBUG Trying to listen on :::4730
 INFO Listening on :::4730 (6)
DEBUG Trying to listen on 0.0.0.0:4730
 INFO Creating wakeup pipe
DEBUG Creating 0 threads
 INFO Creating IO thread wakeup pipe
 INFO Adding event for listening socket (6)
 INFO Adding event for wakeup pipe
 INFO Entering main event loop
 INFO Accepted connection from ::%4294949696:46207
 INFO [ 0] ::%4294949696:46207 Connected
DEBUG [ 0] ::%4294949696:46207 Received SET_CLIENT_ID
DEBUG [ 0] ::%4294949696:46207 Received GRAB_JOB
DEBUG [ 0] ::%4294949696:46207 Sent NO_JOB
DEBUG [ 0] ::%4294949696:46207 Received CAN_DO
DEBUG [ 0] ::%4294949696:46207 Received CAN_DO
DEBUG [ 0] ::%4294949696:46207 Received CAN_DO
DEBUG [ 0] ::%4294949696:46207 Received CAN_DO
DEBUG [ 0] ::%4294949696:46207 Received PRE_SLEEP
 INFO Accepted connection from ::%4294949696:46208
 INFO [ 0] ::%4294949696:46208 Connected
DEBUG [ 0] ::%4294949696:46208 Received SUBMIT_JOB
DEBUG [ 0] ::%4294949696:46208 Sent JOB_CREATED
DEBUG [ 0] ::%4294949696:46207 Sent NOOP
DEBUG [ 0] ::%4294949696:46207 Received GRAB_JOB
DEBUG [ 0] ::%4294949696:46207 Sent JOB_ASSIGN
DEBUG [ 0] ::%4294949696:46207 Received WORK_EXCEPTION
DEBUG [ 0] ::%4294949696:46207 Received WORK_FAIL
DEBUG [ 0] ::%4294949696:46207 Received GRAB_JOB
DEBUG [ 0] ::%4294949696:46207 Sent NO_JOB
DEBUG [ 0] ::%4294949696:46208 Sent WORK_FAIL
 INFO [ 0] ::%4294949696:46208 Disconnected
DEBUG [ 0] ::%4294949696:46207 Received PRE_SLEEP

Revision history for this message
Wilton (risenhoover) wrote :

Ok now it works. The error I was getting before was an exception from my worker class. So to recap, this change makes it work.

//assert(client->task != NULL);

if ( client->task != NULL ) {
      client->task->recv= &(client->con->packet);
}

Revision history for this message
Brian Aker (brianaker) wrote :

Hi!

Are you planning on submitting this with a test case?

Thanks,
   -Brian

Revision history for this message
Eric Day (eday) wrote : Re: [Bug 518512] Re: php: client.c:773: gearman_client_run_tasks: Assertion `client->task != ((void *)0)`failed.

Hi Wilton,

The proper fix is slightly more involved than that check, you actually
need to skip processing of the packet entirely in the state machine
as well. If you don't get to it before then, I'll try to get it fixed
in the next couple days. Been on my todo for too long. :)

Thanks!
-Eric

On Wed, Mar 24, 2010 at 10:29:49PM -0000, Wilton wrote:
> Ok now it works. The error I was getting before was an exception from
> my worker class. So to recap, this change makes it work.
>
> //assert(client->task != NULL);
>
> if ( client->task != NULL ) {
> client->task->recv= &(client->con->packet);
> }
>
> --
> php: client.c:773: gearman_client_run_tasks: Assertion `client->task != ((void *)0)`failed.
> https://bugs.launchpad.net/bugs/518512
> You received this bug notification because you are a member of Gearman-
> developers, which is the registrant for Gearman C Server and Library.
>
> Status in Gearman C Server and Library: Confirmed
>
> Bug description:
> I'm using libgearman 0.11 with PECL PHP extension gearman 0.6.0 on CentOS 5.4 using Apache 2.2 and PHP 5.3.
>
> While running a Gearman Worker I got this error:
> php: client.c:773: gearman_client_run_tasks: Assertion `client->task != ((void *)0)`failed.
>
> Context; I made this PHP script:
> http://pastie.org/private/jzm9ox4kogpllz9onl7ma
>
> When I start one worker, everything goes smooth. When I start an additional second worker, no problem. When I start the third worker, the first worker crashes with the error above. At the moment the third worker starts, it registers to the first worker (addNeighbour). This 'addNeighbour' will call the 'updateNeighbour' function on the first Worker itself. In the loop, the IF-statement will return false for the first time, so the else-part is executed. When it comes to #110 the crash occurs.
>
> Little strange to me is that after the first worker crashed, I can start as many additional workers as I like... they won't crash.
>
> This bug keeps me from using Gearman in PHP at the moment. Thanks in advance!
>

Eric Day (eday)
Changed in gearmand:
assignee: nobody → Eric Day (eday)
milestone: none → 0.13
status: Confirmed → Fix Committed
Revision history for this message
Wilton (risenhoover) wrote : Re: [Bug 518512] Re: php: client.c:773: gearman_client_run_tasks: Assertion `client->task != ((void *)0)`failed.

Hi Brian,

I'm actually not really sure what the root cause of the problem is. If I
run the exact same php/gearman script with the exact same v. of gearman and
php-gearman, but on an ubuntu machine, it works fine. But it breaks on
debian. If I knew what the root cause was and was able to reproduce it
consistently across several platforms, I would definitely include a test
case.

W.

On Fri, Mar 26, 2010 at 9:37 AM, Brian Aker <email address hidden> wrote:

> Hi!
>
> Are you planning on submitting this with a test case?
>
> Thanks,
> -Brian
>
> --
> php: client.c:773: gearman_client_run_tasks: Assertion `client->task !=
> ((void *)0)`failed.
> https://bugs.launchpad.net/bugs/518512
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Gearman C Server and Library: Confirmed
>
> Bug description:
> I'm using libgearman 0.11 with PECL PHP extension gearman 0.6.0 on CentOS
> 5.4 using Apache 2.2 and PHP 5.3.
>
> While running a Gearman Worker I got this error:
> php: client.c:773: gearman_client_run_tasks: Assertion `client->task !=
> ((void *)0)`failed.
>
> Context; I made this PHP script:
> http://pastie.org/private/jzm9ox4kogpllz9onl7ma
>
> When I start one worker, everything goes smooth. When I start an additional
> second worker, no problem. When I start the third worker, the first worker
> crashes with the error above. At the moment the third worker starts, it
> registers to the first worker (addNeighbour). This 'addNeighbour' will call
> the 'updateNeighbour' function on the first Worker itself. In the loop, the
> IF-statement will return false for the first time, so the else-part is
> executed. When it comes to #110 the crash occurs.
>
> Little strange to me is that after the first worker crashed, I can start as
> many additional workers as I like... they won't crash.
>
> This bug keeps me from using Gearman in PHP at the moment. Thanks in
> advance!
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/gearmand/+bug/518512/+subscribe
>

--
Wilton Paul Risenhoover
UCLA Anderson School of Management
EMBA Class of 2010

follow me on twitter: twitter.com/risenhoover

Eric Day (eday)
Changed in gearmand:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.