Big samba memory leak fixed upstream

Bug #1814532 reported by Mikael Hartzell
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
samba (Ubuntu)
Fix Released
High
Unassigned
Bionic
Fix Released
High
Andreas Hasenack

Bug Description

[Impact]

There is a memory leak in vfswrap_getwd() that, depending on the server usage, can become severe and trigger an OOM killer.

[Test Case]
It's hard to come up with a test case for memory leaks, because they can take a while to manifest.

[Regression Potential]
Patch is sane and has been applied upstream. It's already present in cosmic and later. The code that frees the pointer is only reached if the pointer isn't NULL.

[Other Info]
None at this time.

[Original Description]

There is a big memory leak bug in Samba 4.1 - 4.7.6. Depending on the circumstances all memory of the Ubuntu server will be eaten by Samba sooner or later. Then Linux Oom - killer will kill Samba which will either restart or hang.

On our Ubuntu Server 14.04.5 I need to restart Samba 1 - 2 times a week. This bug probably affects also Ubuntu 18.04 if this fix has not yet been backported.

This bug has been fixed upstream in Samba 4.7.7. The fix is only two lines of code and the bug is caused by a single misplaced "if" when releasing memory.

This upstream commit fixes the bug: https://gitlab.com/samba-team/samba/commit/461a1172ff819692aa0a2dc5ce7fc5379c8a529e

This is the Samba bug report: https://bugzilla.samba.org/show_bug.cgi?id=13372

And here the author apologizes for the bug :) https://lists.samba.org/archive/samba-technical/2018-April/126937.html

Please backport this fix to Ubuntu 14.04 and 18.04, thanks :)

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: samba 2:4.3.11+dfsg-0ubuntu0.14.04.19
ProcVersionSignature: Ubuntu 3.13.0-164.214-generic 3.13.11-ckt39
Uname: Linux 3.13.0-164-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.29
Architecture: amd64
BothFailedConnect: Yes
Date: Mon Feb 4 11:34:06 2019
InstallationDate: Installed on 2014-09-25 (1592 days ago)
InstallationMedia: Ubuntu-Server 14.04.1 LTS "Trusty Tahr" - Release amd64 (20140722.3)
SambaServerRegression: Yes
SmbConfIncluded: No
SourcePackage: samba
UpgradeStatus: No upgrade log present (probably fresh install)
upstart.samba-ad-dc.override: manual

Related branches

Revision history for this message
Mikael Hartzell (mhartzel) wrote :
Changed in samba (Ubuntu):
importance: Undecided → High
Revision history for this message
Sebastien Bacher (seb128) wrote :

Thank you for your bug report, according to the upstream discussion the issue was create by that commit
https://github.com/samba-team/samba/commit/bd9285b1 and looks like a good candidate for a bionic SRU, it shouldn't apply/be an issue on trusty though which has 4.1/4.3 samba series

description: updated
Revision history for this message
Mikael Hartzell (mhartzel) wrote :

On top of the original Samba bug report (https://bugzilla.samba.org/show_bug.cgi?id=13372) it says this affects *Samba 4.1 and newer* or am I interpreting it incorrectly ?

At least I am experiencing the same behavior on Ubuntu 14.04.5 Server with Samba 4.3.11. Sad if it won't help us with our problem.

The offending code was submitted to Samba at 29th Jun 2017, you can find the commit here: https://gitlab.com/samba-team/samba/commit/bd9285b19741128bae501b721d9e63dd9a9bd833 If you search for the source file name: source3/modules/vfs_default.c on the page.

Maybe the date when the bug appeared sheds some light which releases really are affected.

Revision history for this message
Mikael Hartzell (mhartzel) wrote :

Ok, I see that you already found the date information :)

Revision history for this message
Sebastien Bacher (seb128) wrote :

"4.1 and newer" is only what their bugzilla has register as product, the list is
https://bugzilla.samba.org/describecomponents.cgi

they seem to use it to classify bugs that impact "not too ancien" samba versions, it doesn't mean 4.1 is impacted

Revision history for this message
Sebastien Bacher (seb128) wrote :

(so yeah, it's not likely to be the same issue you are having on trusty)

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Fixed in cosmic and later.

no longer affects: samba (Ubuntu Cosmic)
Changed in samba (Ubuntu):
status: New → Fix Released
Changed in samba (Ubuntu Bionic):
status: New → Triaged
importance: Undecided → High
tags: added: server-next
description: updated
description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

I'm not sure this is robust enough as a test case, that's why I'm adding it here instead of in the test caseh, but I ran smbd under valgrind before and after the update, and less bytes were lost for the same smbclient connection:

before:
==3977== LEAK SUMMARY:
==3977== definitely lost: 72 bytes in 7 blocks

After:
==5438== LEAK SUMMARY:
==5438== definitely lost: 24 bytes in 1 blocks

The client command was (with a suitable smb.conf and /pub directory):
smbclient //localhost/pub -U ubuntu%ubuntu -c "pwd;dir;cd dir1;dir;pwd;cd dir11; pwd; dir; cd /; cd dir2; pwd; dir; cd /"

Changed in samba (Ubuntu Bionic):
assignee: nobody → Andreas Hasenack (ahasenack)
status: Triaged → In Progress
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Mikael, or anyone else affected,

Accepted samba into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/samba/2:4.7.6+dfsg~ubuntu-0ubuntu2.7 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in samba (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

With the bionic packages:

 *** 2:4.7.6+dfsg~ubuntu-0ubuntu2.6 500
        500 http://br.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages

Running:
smbclient //localhost/pub -U ubuntu%ubuntu -c "pwd;dir;cd dir1;dir;pwd;cd dir11; pwd; dir; cd /; cd dir2; pwd; dir; cd /"

Server valgrind report:
==3887== LEAK SUMMARY:
==3887== definitely lost: 72 bytes in 7 blocks
==3887== indirectly lost: 0 bytes in 0 blocks
==3887== possibly lost: 125,212 bytes in 488 blocks
==3887== still reachable: 117,607 bytes in 901 blocks
==3887== suppressed: 0 bytes in 0 blocks

After update:
==7133== LEAK SUMMARY:
==7133== definitely lost: 24 bytes in 1 blocks
==7133== indirectly lost: 0 bytes in 0 blocks
==7133== possibly lost: 125,212 bytes in 488 blocks
==7133== still reachable: 117,607 bytes in 901 blocks
==7133== suppressed: 0 bytes in 0 blocks

I was hoping the original reported would have tried the updated packages, since he seemed quite affected by this memleak. But in the meantime, this is the verification I can do.

tags: added: verification-done-bionic
removed: verification-needed-bionic
Revision history for this message
Mathew Hodson (mhodson) wrote :

The original report was for Trusty, so the OP wouldn't be able to test if this Bionic package fixes their issue.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Do we intend to also fix this for trusty and xenial? Since I don't see any tasks set or uploads in the unapproved queues.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package samba - 2:4.7.6+dfsg~ubuntu-0ubuntu2.7

---------------
samba (2:4.7.6+dfsg~ubuntu-0ubuntu2.7) bionic; urgency=medium

  * d/p/memleak-fix-13372.patch: Fix memory leak in vfswrap_getwd().
    (LP: #1814532)

 -- Andreas Hasenack <email address hidden> Mon, 04 Feb 2019 17:37:51 -0200

Changed in samba (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for samba has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

No, we would need a new bug report for trusty.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.