Gearman::XS

Memory leak - GC

Bug #507781 reported by James M. Luedke on 2010-01-15

6

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Gearman::XS	Fix Released	Medium	Dennis Schön	Gearman::XS 0.8
Nominated for Trunk by Jeremy Zawodny

Bug Description

Client and worker stack grow and grow...

#!/usr/bin/perl -w

use strict;
use Gearman::XS qw(:constants);
use Gearman::XS::Worker;

my $gmw = new Gearman::XS::Worker;

$gmw->add_server("gmanserver");

$gmw->add_function(
"gmw_test", 0, sub {
my ($job, $options) = @_;

        print "workload: " , $job->workload(), "\n";
        print "handle: ", $job->handle(), "\n";
        print "unique: ", $job->unique(), "\n";
    }, ''
);

my $cnt=0;
while (1) {
    my $ret = $gmw->work();
    if ($ret != GEARMAN_SUCCESS) {
        print STDERR $gmw->error(), "\n";
        die "worker giving up (pid: $$)";
    }
    print "CNT: $cnt\n";
    $cnt++;
}

#!/usr/bin/perl -w
use strict;
use Data::Dumper;
use Gearman::XS qw(:constants);
use Gearman::XS::Client;

my $cnt = 0;
my $gmc = new Gearman::XS::Client;
$gmc->set_complete_fn(sub {
my $task = shift;
print $task->data() . "\n";
});
$gmc->add_server("gmanserver");

while(1) {
    $gmc->add_task("gmw_test", "test data", 0);
    $gmc->run_tasks();
    print "CNT: $cnt\n";
    $cnt++;
}

The code bellow leaks memory and leaves net ESTABLISHED connections around

#!/usr/bin/perl -w

use strict;
use Gearman::XS qw(:constants);
use Gearman::XS::Client;

my $cnt = 0;
sub gm_test {
    my $gmc = new Gearman::XS::Client;
    $gmc->set_complete_fn(sub {
        my $task = shift;
        print $task->data() . "\n";
    });
    $gmc->add_server("gmanserver");
    $gmc->add_task("gmw_test", "test data", 0);
    $gmc->run_tasks();
    print "CNT: $cnt\n";
    $cnt++;
}

while(1) {
gm_test();
}

netstat -a | grep gmanserver | wc -l
to confirm

Let me know if I can provide any more info or assist in any way.

-James

Related branches

lp:~roccoblues/gearmanxs/bug507781-memory-leak (Merged)

Revision history for this message

Dennis Schön (roccoblues) wrote on 2010-01-15:

#1

I can see the client stack growing with your example. The worker however doesn't leak memory. I'll investigate more.

Changed in gearmanxs:
status:	New → Confirmed
importance:	Undecided → Medium
assignee:	nobody → Dennis Schön (roccoblues)

Revision history for this message

James M. Luedke (jluedke) wrote on 2010-01-16:

#2

I noticed that this error is getting returned from $client->run_tasks
ERROR: gearman_wait:no active file descriptors

I also noticed that examples/reverse_client_cb.pl packaged with Gearman-XS does not work

-James

Revision history for this message

Dennis Schön (roccoblues) wrote on 2010-01-18:

#3

what problem do you have with reverse_client_cb.pl? Works for me here:

[dschoen@dennis ~/repos/gearmanxs/trunk/]$ ./examples/reverse_client_cb.pl blubb
Added: 00269838-FCFE-49FD-901E-FCDE8BAC346E
Added: 810ABA11-5C58-4C6F-8B0E-8D34EE7535D2
Added: 9DF49DA4-6CF4-40C3-AEFC-A4C753AD4465
Added: 40688563-81E7-4C20-880E-ED658BF15ACB
Added: 4D3122F1-667A-4762-A720-1A137A07EC78
Added: 3C3360E4-8F79-46EA-A6E5-8C36B81C8E75
Added: BBC260D2-8B20-4377-85DC-C200C53BB508
Added: B63C0B8F-945B-4801-8B94-5DB9F8E878C1
Added: 708762E8-6E74-4519-B2B0-72AE3EE258C9
Added: B4B97865-1036-424A-8355-37270087EFD1
Created: H:dennis.local:2
Created: H:dennis.local:3
Created: H:dennis.local:4
Created: H:dennis.local:5
Created: H:dennis.local:6
Created: H:dennis.local:7
Created: H:dennis.local:8
Created: H:dennis.local:9
Created: H:dennis.local:10
Created: H:dennis.local:11
Completed: H:dennis.local:2 bbulb
Completed: H:dennis.local:3 bbulb
Completed: H:dennis.local:4 bbulb
Completed: H:dennis.local:5 bbulb
Completed: H:dennis.local:6 bbulb
Completed: H:dennis.local:7 bbulb
Completed: H:dennis.local:8 bbulb
Completed: H:dennis.local:9 bbulb
Completed: H:dennis.local:10 bbulb
Completed: H:dennis.local:11 bbulb

Dennis Schön (roccoblues) on 2010-01-18

Changed in gearmanxs:
milestone:	none → 0.8
status:	Confirmed → Fix Committed

Revision history for this message

Dennis Schön (roccoblues) wrote on 2010-01-18:

#4

ok, I think I figured it out:

1. you're callback functions in set_complete_fn (client) and add_function (worker) explicitly need to return GEARMAN_SUCCESS. Check out examples/reverse_client_cb.pl for an example.

2. then there was still a memory leak which I hope to have fixed in lp:~roccoblues/gearmanxs/bug507781-memory-leak. Can you run your tests again with this branch?

Thanks!

Revision history for this message

Dennis Schön (roccoblues) wrote on 2010-01-18:

#5

of course the function passed to add_function() shouldn't return GEARMAN_SUCCESS but the actual result. I was to fast there. But everything else I wrote still stands. ;-)

Revision history for this message

Dennis Schön (roccoblues) wrote on 2010-01-18:

#6

checked with james on irc:
no memory leak with perl 5.8.8 but with v5.10.0 there's one.

Also there still seems to be a memory leak when calling set_complete_fn() multiple times.

Revision history for this message

Dennis Schön (roccoblues) wrote on 2010-01-19:

#7

ok the last memory leak in with perl 5.8.8 and the complete_fn() should be fixed now in the linked branch. Now I'll test with perl 5.10.

Revision history for this message

Dennis Schön (roccoblues) wrote on 2010-01-19:

#8

No more memory leaks with perl v5.8.8 and v5.10.1. If someone still has problems please open a new bugreport with testcase. Thanks Dennis

Changed in gearmanxs:
status:	Fix Committed → Won't Fix
status:	Won't Fix → Fix Committed

Revision history for this message

Jeremy Zawodny (jeremy-zawodny) wrote on 2010-02-01:

#9

Dennis, what's the timeline for a new CPAN release?

Revision history for this message

Dennis Schön (roccoblues) wrote on 2010-02-01: Re: [Bug 507781] Re: Memory leak - GC

#10

Hi Jeremy,

I wanted to wait for the next libgearman release. There are some
bugfixes in there as well. I'll ping Eric and Brian and check what
their timeline is.

On Feb 1, 2010, at 7:17 PM, Jeremy Zawodny wrote:

> Dennis, what's the timeline for a new CPAN release?
>
> --
> Memory leak - GC
> https://bugs.launchpad.net/bugs/507781
> You received this bug notification because you are a member of
> Gearman-
> developers, which is the registrant for Gearman::XS.
>
> Status in Gearman Perl Frontend: Fix Committed
>
> Bug description:
> Client and worker stack grow and grow...
>
> #!/usr/bin/perl -w
>
> use strict;
> use Gearman::XS qw(:constants);
> use Gearman::XS::Worker;
>
> my $gmw = new Gearman::XS::Worker;
>
> $gmw->add_server("gmanserver");
>
> $gmw->add_function(
> "gmw_test", 0, sub {
> my ($job, $options) = @_;
>
> print "workload: " , $job->workload(), "\n";
> print "handle: ", $job->handle(), "\n";
> print "unique: ", $job->unique(), "\n";
> }, ''
> );
>
> my $cnt=0;
> while (1) {
> my $ret = $gmw->work();
> if ($ret != GEARMAN_SUCCESS) {
> print STDERR $gmw->error(), "\n";
> die "worker giving up (pid: $$)";
> }
> print "CNT: $cnt\n";
> $cnt++;
> }
>
> #!/usr/bin/perl -w
> use strict;
> use Data::Dumper;
> use Gearman::XS qw(:constants);
> use Gearman::XS::Client;
>
> my $cnt = 0;
> my $gmc = new Gearman::XS::Client;
> $gmc->set_complete_fn(sub {
> my $task = shift;
> print $task->data() . "\n";
> });
> $gmc->add_server("gmanserver");
>
> while(1) {
> $gmc->add_task("gmw_test", "test data", 0);
> $gmc->run_tasks();
> print "CNT: $cnt\n";
> $cnt++;
> }
>
> The code bellow leaks memory and leaves net ESTABLISHED connections
> around
>
> #!/usr/bin/perl -w
>
> use strict;
> use Gearman::XS qw(:constants);
> use Gearman::XS::Client;
>
> my $cnt = 0;
> sub gm_test {
> my $gmc = new Gearman::XS::Client;
> $gmc->set_complete_fn(sub {
> my $task = shift;
> print $task->data() . "\n";
> });
> $gmc->add_server("gmanserver");
> $gmc->add_task("gmw_test", "test data", 0);
> $gmc->run_tasks();
> print "CNT: $cnt\n";
> $cnt++;
> }
>
> while(1) {
> gm_test();
> }
>
>
> netstat -a | grep gmanserver | wc -l
> to confirm
>
> Let me know if I can provide any more info or assist in any way.
>
> -James
>
>

Hi Jeremy,

I wanted to wait for the next libgearman release. There are some  
bugfixes in there as well. I'll ping Eric and Brian and check what  
their timeline is.

On Feb 1, 2010, at 7:17 PM, Jeremy Zawodny wrote:

> Dennis, what's the timeline for a new CPAN release?
>
> -- 
> Memory leak - GC
> https://bugs.launchpad.net/bugs/507781
> You received this bug notification because you are a member of  
> Gearman-
> developers, which is the registrant for Gearman::XS.
>
> Status in Gearman Perl Frontend: Fix Committed
>
> Bug description:
> Client and worker stack grow and grow...
>
> #!/usr/bin/perl -w
>
> use strict;
> use Gearman::XS qw(:constants);
> use Gearman::XS::Worker;
>
> my $gmw = new Gearman::XS::Worker;
>
> $gmw->add_server("gmanserver");
>
> $gmw->add_function(
>    "gmw_test", 0, sub {
>        my ($job, $options) = @_;
>
>        print "workload: " , $job->workload(), "\n";
>        print "handle: ", $job->handle(), "\n";
>        print "unique: ", $job->unique(), "\n";
>    }, ''
> );
>
> my $cnt=0;
> while (1) {
>    my $ret = $gmw->work();
>    if ($ret != GEARMAN_SUCCESS) {
>        print STDERR $gmw->error(), "\n";
>        die "worker giving up (pid: $$)";
>    }
>    print "CNT: $cnt\n";
>    $cnt++;
> }
>
> #!/usr/bin/perl -w
> use strict;
> use Data::Dumper;
> use Gearman::XS qw(:constants);
> use Gearman::XS::Client;
>
> my $cnt = 0;
> my $gmc = new Gearman::XS::Client;
> $gmc->set_complete_fn(sub {
>    my $task = shift;
>    print $task->data() . "\n";
> });
> $gmc->add_server("gmanserver");
>
> while(1) {
>    $gmc->add_task("gmw_test", "test data", 0);
>    $gmc->run_tasks();
>    print "CNT: $cnt\n";
>    $cnt++;
> }
>
> The code bellow leaks memory and leaves net ESTABLISHED connections  
> around
>
> #!/usr/bin/perl -w
>
> use strict;
> use Gearman::XS qw(:constants);
> use Gearman::XS::Client;
>
> my $cnt = 0;
> sub gm_test {
>    my $gmc = new Gearman::XS::Client;
>    $gmc->set_complete_fn(sub {
>        my $task = shift;
>        print $task->data() . "\n";
>    });
>    $gmc->add_server("gmanserver");
>    $gmc->add_task("gmw_test", "test data", 0);
>    $gmc->run_tasks();
>    print "CNT: $cnt\n";
>    $cnt++;
> }
>
> while(1) {
> gm_test();
> }
>
>
> netstat -a | grep gmanserver | wc -l
> to confirm
>
> Let me know if I can provide any more info or assist in any way.
>
> -James
>
>

Dennis Schön (roccoblues) on 2010-02-19

Changed in gearmanxs:
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.