pgsql fails to start due to shared buffer setting greater than kernel allows

This happens for me too in Intrepid on AMD64, except pgsql attempts to allocate 40MB while the shmax setting is still 32MB.

2008-11-02 00:31:09 GMT FATAL: could not create shared memory segment: Invalid argument
2008-11-02 00:31:09 GMT DETAIL: Failed system call was shmget(key=5432001, size=39288832, 03600).
2008-11-02 00:31:09 GMT HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 39288832 bytes), reduce PostgreSQL's shared_buffers parameter (currently 4096) and/or its max_connections parameter (currently 103).
        If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
        The PostgreSQL documentation contains more information about shared memory configuration.

Automatic upgrades on Ubuntu 8.10 i386 desktop failed with:

Setting up postgresql-8.3 (8.3.5-0ubuntu8.10) ...
 * Starting PostgreSQL 8.3 database server
 * The PostgreSQL server failed to start. Please check the log output:
2008-11-28 18:50:00 CET LOG: could not load root certificate file "root.crt": no SSL error reported
2008-11-28 18:50:00 CET DETAIL: Will not verify client certificates.
2008-11-28 18:50:00 CET FATAL: could not create shared memory segment: Invalid argument
2008-11-28 18:50:00 CET DETAIL: Failed system call was shmget(key=5432001, size=38207488, 03600).
2008-11-28 18:50:00 CET HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 38207488 bytes), reduce PostgreSQL's shared_buffers parameter (currently 4096) and/or its max_connections parameter (currently 103).
 If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
 The PostgreSQL documentation contains more information about shared memory configuration.
                                                                                                                                                 [fail]
invoke-rc.d: initscript postgresql-8.3, action "start" failed.
dpkg: error processing postgresql-8.3 (--configure):
 subprocess post-installation script returned error exit status 1
Errors were encountered while processing:
 postgresql-8.3
E: Sub-process /usr/bin/dpkg returned an error code (1)

I don't know if it is really the same issue, I haven't the "SHMAX" comment, but the rest seems similar on a installing postgres, I've got :
Paramétrage de postgresql-8.3 (8.3.5-0ubuntu8.10) ...
 * Starting PostgreSQL 8.3 database server * The PostgreSQL server failed to start. Please check the log output:
2009-01-10 22:21:03 CET LOG: n'a pas pu charger le fichier du certificat racine « root.crt » : aucun code d'erreur SSL rapporté
2009-01-10 22:21:03 CET DÉTAIL: Ne vérifiera pas les certificats du client.
2009-01-10 22:21:03 CET LOG: n'a pas pu résoudre le nom de l'hôte « localhost », service « 5432 » par l'adresse : Nom ou service inconnu
2009-01-10 22:21:03 CET ATTENTION: n'a pas pu créer le socket d'écoute pour « localhost »
2009-01-10 22:21:03 CET FATAL: n'a pas pu créer de socket TCP/IP
                                                                                                                                                      [fail]
invoke-rc.d: initscript postgresql-8.3, action "start" failed.
dpkg : erreur de traitement de postgresql-8.3 (--configure) :
 le sous-processus post-installation script a retourné une erreur de sortie d'état 1
Des erreurs ont été rencontrées pendant l'exécution :
 postgresql-8.3
E: Sub-process /usr/bin/dpkg returned an error code (1)

I can replicate in my install. Please upgrade to critical; it's hit a few production servers. Yikes!

I get exactly the same error with nearly the same parameters:
2009-02-07 08:34:38 CET FATAL: could not create shared memory segment: Invalid argument
2009-02-07 08:34:38 CET DETAIL: Failed system call was shmget(key=5432001, size=39288832, 03600).
2009-02-07 08:34:38 CET HINT: This error usually means that PostgreSQL's [...]

But... I only get this at boot time, while postgres' automatic startup.
If I do a '/etc/init.d/postgresql-8.3 start' later, it works without any error message!

System: ubuntu-server 8.04.2 (x86_64), kernel 2.6.24-23-server
PostgreSQL: 8.3.5-0ubuntu0.8.04
kernel.shmmax = 268435456
kernel.shmall = 2097152
kernel.shmmni = 4096

Did anyone of you tweak the default shm limits? I newer saw this problem on my boxes.

I guess not, since I get the very same settings under 9.04/i386 as Richard describes in the original description.

So I guess there are some other programs running on your boxes which use shared memory a lot. Can you please give me the output of

  sudo ipcs -m

and do

  sudo lsof > /tmp/lsof.txt

and attach /tmp/lsof.txt here?

This is a default setting in postgresql-8.x, and since the error message is very clear, this will not be fixed in the upstream code itself. We could give some more clever defaults in pg_createcluster (in postgresql-common), so I keep the -common task open and close the -8.3 task.

Yes. I just changed the shared_buffers to 24 megs and Postgresql now starts. So I guess that fixed it.

Thanks.

Ok here are the outputs of my ipcs and lsof.

And the lsof

Ah, seems that X.org is taking away tons of shared memory.

So, the failure message should make this clear enough, but I'll keep this open as a reminder to investigate the possibility of automatically configuring smaller SHM limits in postgresql.conf.

Like what aardvark told, I too have the same situation.

At boot time I get the error, but if I later start the service manually (/etc/init.d/postgresql-8.3 start) the postgres db starts without errors.

For me it seems that when the postgres service is started at boot time, the shm parameters are different then after logging in.

I rebooted my machine with some logging in the postgres startup script.

[at boot time]
$ ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status

$ sysctl -a | grep -i shm
kernel.shmmax = 33554432
kernel.shmall = 2097152
kernel.shmmni = 4096

[logged in later]
$ ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x00000000 0 root 700 16777216 1 dest
0x00000000 32769 root 700 16777216 1 dest
0x00000000 65538 root 700 16777216 1 dest

$ sysctl -a | grep -i shm
kernel.shmmax = 268435456
kernel.shmall = 2097152
kernel.shmmni = 4096

I increase kernel.shmmax to 40 mb and now postgresql starts.

Hit by the same problem. Reduced shared_buffers and postgresql started. Jaunty x86_64.

I started a discussion upstream.

Reduced shared_buffers and postgresql starts now without problems Ubuntu Intrepid updated today.

I thought about this problem a little, and also discussed it with upstream. Basically, we have four options:

(1) Ignore
    + no hidden magic
    - very inconvenient, package installation does not create default
      cluster sometimes, or the default cluster fails to start on
      system boot

   Best solution for admin control freaks.

(2) Bee shy about initdb's default setting and set it to 3/4 of the available memory.
    + no hidden magic
    + upstream compatible solution
    - suboptimal performance by default

(3) Change SHMALL/SHMMAX in postgresql's init script if necessary
    + Always works
    - Unexpected, works behind admin's back.

(4) Set a bigger SHMALL default in the kernel.
    + no hidden magic
    + no guesswork involved in scripts
    + no hidden magic

Upstream recommended to do (4). Quote: "I imagine BTW that it's SHMALL that's causing you issues. SHMMAX shouldn't have any effect that varies depending on what else is using shared memory."

I reassign this to the kernel guys to get their input.

This is still not breaking many boxes, and intrusive to solve either way, so taking off the Jaunty radar.

Kernel team,

do we change the default SHM* parameters in any way, or are we using current upstream's? If the latter, I'll go back to discuss it with upstream. If the former, what was the reason for this? Thanks!

(Assigning mainly to get your input; feel free to unassign/WONTFIX as you see fit).

I have set SHMALL to 1GB & it's ok now.
thx

2009/4/1 Martin Pitt <email address hidden>

> I thought about this problem a little, and also discussed it with
> upstream. Basically, we have four options:
>
> (1) Ignore
> + no hidden magic
> - very inconvenient, package installation does not create default
> cluster sometimes, or the default cluster fails to start on
> system boot
>
> Best solution for admin control freaks.
>
> (2) Bee shy about initdb's default setting and set it to 3/4 of the
> available memory.
> + no hidden magic
> + upstream compatible solution
> - suboptimal performance by default
>
> (3) Change SHMALL/SHMMAX in postgresql's init script if necessary
> + Always works
> - Unexpected, works behind admin's back.
>
> (4) Set a bigger SHMALL default in the kernel.
> + no hidden magic
> + no guesswork involved in scripts
> + no hidden magic
>
> Upstream recommended to do (4). Quote: "I imagine BTW that it's SHMALL
> that's causing you issues. SHMMAX shouldn't have any effect that varies
> depending on what else is using shared memory."
>
> I reassign this to the kernel guys to get their input.
>
> This is still not breaking many boxes, and intrusive to solve either
> way, so taking off the Jaunty radar.
>
> ** Package changed: postgresql-common (Ubuntu Jaunty) => linux (Ubuntu
> Jaunty)
>
> ** Changed in: linux (Ubuntu Jaunty)
> Status: Triaged => Won't Fix
>
> ** Changed in: linux (Ubuntu Jaunty)
> Assignee: Martin Pitt (pitti) => (unassigned)
>
> --
> pgsql fails to start due to shared buffer setting greater than kernel
> allows
> https://bugs.launchpad.net/bugs/264336
> You received this bug notification because you are a direct subscriber
> of a duplicate bug.
>
> Status in “linux” source package in Ubuntu: New
> Status in “postgresql-8.3” source package in Ubuntu: Won't Fix
> Status in linux in Ubuntu Jaunty: Won't Fix
> Status in postgresql-8.3 in Ubuntu Jaunty: Won't Fix
>
> Bug description:
> Binary package hint: postgresql-8.3
>
> Freshly installed pgsql 8.3 on hardy got me this error when restarting:
> 2008-09-03 09:16:39 EDT DETAIL: Failed system call was shmget(key=5432001,
> size=38207488, 03600).
> 2008-09-03 09:16:39 EDT HINT: This error usually means that PostgreSQL's
> request for a shared memory segment exceeded your kernel's SHMMAX parameter.
> You can either reduce the request size or reconfigure the kernel with
> larger SHMMAX. To reduce the request size (currently 38207488 bytes),
> reduce PostgreSQL's shared_buffers parameter (currently 4096) and/or its
> max_connections parameter (currently 103).
> If the request size is already small, it's possible that it is less
> than your kernel's SHMMIN parameter, in which case raising the request size
> or reconfiguring SHMMIN is called for.
> The PostgreSQL documentation contains more information about shared
> memory configuration.
>
> Checking the kernel settings:
> sudo sysctl -a | grep -i shm
> kernel.shmmax = 33554432
> kernel.shmall = 2097152
> kernel.shmmni = 4096
>
> The shared_buffers setting defaults to 32MB. I changed this down to 25MB to
> test and then the server would start righ...

I have added a file:

/etc/sysctl.d/90-postgres.conf

in which I have changed the kernel settings for shmmax to make postgres run correctly.

Isn't this why this folder was created, to make individual packages change the configurations according to their needs?

I do not get exactly why this would not affect all postgres users? Is the configuration of postgres dynamically created during install, based on the available memory in the system?
Because if I understand correctly it is a combination of parameters in the postgres config file which sum up to a required shared memory usage.

Anyway, I solved it by manually creating the above mentioned file.

Morten Minke [2009-04-01 11:03 -0000]:
> I have added a file:
>
> /etc/sysctl.d/90-postgres.conf
>
> in which I have changed the kernel settings for shmmax to make postgres
> run correctly.

That's fine.

> Isn't this why this folder was created, to make individual packages
> change the configurations according to their needs?

Not really. Just assume several packages would ship configuration
files there which would set different values. This is just a Russian
Roulette then. There should be one sensible system default, not
several packages trying to compete for settings.

> I do not get exactly why this would not affect all postgres users?
> Is the configuration of postgres dynamically created during install,
> based on the available memory in the system?

Yes, it is. initdb determines the current setting and sets up an
appropriate default in postgresql.conf. The problem occurs if
something major changes between installation and starting.

What we should do here, though, is to not make package installation fail if cluster startup fails. This involves creating an error handler in postgresql-common and telling postgresql-8.3's init script to use it.

sorry, we do not need a fancy error handler in postgresql-common, since a failed startup already logs the reason for the failure.

@Martin,

>do we change the default SHM* parameters in any way, or are we using current upstream's? If the latter,
> I'll go back to discuss it with upstream. If the former, what was the reason for this? Thanks!
> (Assigning mainly to get your input; feel free to unassign/WONTFIX as you see fit).

Just to confirm that we *don't* change the default SHM* parameters in include/linux/shm.h at all. I suspect this needs to be an upstream discussion.

@Martin, how do you want to take this bug forward?

Thanks Colin for confirming. I'll discuss it with the upstream PostgreSQL guys.

As pre request (https://bugs.launchpad.net/ubuntu/+source/postgresql-8.3/+bug/387682/comments/4)
$ ls -l /etc/rc2.d/S20*

gives

lrwxrwxrwx 1 root root 16 2009-05-06 11:26 /etc/rc2.d/S20apport -> ../init.d/apport
lrwxrwxrwx 1 root root 28 2009-05-06 12:24 /etc/rc2.d/S20dkms_autoinstaller -> ../init.d/dkms_autoinstaller
lrwxrwxrwx 1 root root 15 2009-05-06 13:06 /etc/rc2.d/S20exim4 -> ../init.d/exim4
lrwxrwxrwx 1 root root 22 2009-05-06 11:26 /etc/rc2.d/S20hotkey-setup -> ../init.d/hotkey-setup
lrwxrwxrwx 1 root root 13 2009-06-22 11:05 /etc/rc2.d/S20kvm -> ../init.d/kvm
lrwxrwxrwx 1 root root 21 2009-06-22 13:43 /etc/rc2.d/S20libvirt-bin -> ../init.d/libvirt-bin
lrwxrwxrwx 1 root root 23 2009-06-22 09:49 /etc/rc2.d/S20smartmontools -> ../init.d/smartmontools
lrwxrwxrwx 1 root root 17 2009-06-11 09:37 /etc/rc2.d/S20winbind -> ../init.d/winbind

I'll see if "sudo mv /etc/rc2.d/S19postgresql-8.3 /etc/rc2.d/S20postgresql-8.3" reliably fixes the problem next time I reboot.

By the way here is the output of ls -l /etc/rc2.d/S19*

I have vmware workstation installed!

I forgot the ouput of "ls -l /etc/rc2.d/S19*" (after I have "sudo mv /etc/rc2.d/S19postgresql-8.3 /etc/rc2.d/S20postgresql-8.3"):
lrwxrwxrwx 1 root root 15 2009-06-11 19:39 /etc/rc2.d/S19mysql -> ../init.d/mysql
lrwxrwxrwx 1 root root 16 2009-05-06 13:24 /etc/rc2.d/S19vmware -> ../init.d/vmware

As pre request (https://bugs.launchpad.net/ubuntu/+source/postgresql-8.3/+bug/387682/comments/4)
> You can confirm that "sudo mv /etc/rc2.d/S19postgresql-8.3 /etc/rc2.d/S20postgresql-8.3" reliably fixes the problem?

Yes, "sudo mv /etc/rc2.d/S19postgresql-8.3 /etc/rc2.d/S20postgresql-8.3" reliably fixes my problem.

Jarl

Now (probably after upgrading to kernel 2.6.28-14) postgres will no longer start, not even when I try to start it manually with sudo /etc/init.d/postgresql-8.3 start, it complains with

2009-08-06 10:55:59 CEST FATAL: could not create shared memory segment: Invalid argument
2009-08-06 10:55:59 CEST DETAIL: Failed system call was shmget(key=5432001, size=39288832, 03600).
2009-08-06 10:55:59 CEST HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 39288832 bytes), reduce PostgreSQL's shared_buffers parameter (currently 4096) and/or its max_connections parameter (currently 103).
        If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
        The PostgreSQL documentation contains more information about shared memory configuration.

@Morten Minke, Could you post the exact content of your /etc/sysctl.d/90-postgres.conf workaround? Please? As it seems that I am now also hit by this bug in a way that I cannot start postgresql at all...

@Martin, how do you want to take this bug forward? Dit the discussion with upstream PostgreSQL people lead to anything?

Jarl

I have now created /etc/sysctl.d/90-postgres.conf with the following content:
kernel.shmmax = 41943040

to increase it to 40MB, this seems to have solved the problem in my situation. Increasing kernel.shmall to 1GB did not make any difference.

I have this problem on jaunty. Postgres would not start when the server starts, but it will start when I do it manually. We have vmware server installed. I tried preventing vmware server from auto starting and also tried the S19 -> S20 workaround, but neither worked. We then followed this advice:

> I have now created /etc/sysctl.d/90-postgres.conf with the following content:
> kernel.shmmax = 41943040

and the problem seems to be resolved.

getting the same with 9.10 and postgres-8.4, too

Now the problem is back again on 9.10 and postgres-8.4.

I am willing to give complete ssh-access (with sudo previleges) to a server to however believe they have the capabilities to investigate and solve this issues.

Jarl

I had the same issue with Debian squeeze and PostgreSQL today. The solution was to add a line: kernel.shmmax = 41943040 to file /etc/sysctl.conf . Maybe it will help you.

Just had the same issue with 10.04 Lucid and Postgres-8.4. Never saw it before.

reopening the kernel task per discussion with Colin.

It does appear that these errors are triggered due to the per-segment limit being 32MB. People can work round this by either reducing the size of the segment in their application or by increasing the kernel limit using sysctl, this can be made permenant using files in /etc/sysctl.d. These should persist over an upgrade.

The default value in the kernel does not appear to have changed since Hardy at 32MB, so its not clear that we should change the default at the kernel level. It may be appropriate to ship a /etc/sysctl.d file to update this with postgres if its defaults require more than the default kernel limit. Else it may be a documentation issue for those configuring larger databases.

I have the same problem in Ubuntu 10.4.

I execute /etc/init.d/postgresql-8.4 start
and the error is:

 * The PostgreSQL server failed to start. Please check the log output:
2010-09-06 09:44:00 CEST FATAL: could not create shared memory segment: Argomento non valido
2010-09-06 09:44:00 CEST DETAIL: Failed system call was shmget(key=5432001, size=37879808, 03600).
2010-09-06 09:44:00 CEST HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded your kernel's SHMMAX parameter. You can either reduce the request size or reconfigure the kernel with larger SHMMAX. To reduce the request size (currently 37879808 bytes), reduce PostgreSQL's shared_buffers parameter (currently 4096) and/or its max_connections parameter (currently 103).
        If the request size is already small, it's possible that it is less than your kernel's SHMMIN parameter, in which case raising the request size or reconfiguring SHMMIN is called for.
        The PostgreSQL documentation contains more information about shared memory configuration.

At the moment I solved it with this guide:
http://dancingpenguinsoflight.com/2010/03/dealing-with-could-not-create-shared-memory-segment-from-postgres-on-ubuntu/

As previously indicated the default configuration is unchanged since hardy at least. The value can be simply changed in a persistant way, and application which rely on it being higher likely should document this and/or supply a sysctl configration to up it. Closing the kernel task.

Confirmed on Ubuntu 9.10 and postgresql-8.4.
workaround:
cd /etc/init.d
sudo update-rc.d -f postgresql-8.4 remove
sudo update-rc.d postgresql-8.4 defaults

This changed the scripts in /etc/rc*.d from S19 to S20.
Solution thus similar to https://bugs.launchpad.net/ubuntu/+source/postgresql-8.4/+bug/264336/comments/32

This is still not fixed. I just installed postgresql-9.1 on a brand new installation of Ubuntu 13.04 (64-bit) and received the following error when attempting to start the server:

     * Starting PostgreSQL 9.1 database server
     * The PostgreSQL server failed to start. Please check the log output:
    2013-08-26 11:40:41 PDT FATAL: could not create shared memory segment: Invalid
    argument
    2013-08-26 11:40:41 PDT DETAIL: Failed system call was shmget(key=5432001, size
    =41263104, 03600).
    2013-08-26 11:40:41 PDT HINT: This error usually means that PostgreSQL's reques
    t for a shared memory segment exceeded your kernel's SHMMAX parameter. You can
    either reduce the request size or reconfigure the kernel with larger SHMMAX. To
     reduce the request size (currently 41263104 bytes), reduce PostgreSQL's shared
    memory usage, perhaps by reducing shared_buffers or max_connections.
     If the request size is already small, it's possible that it is less than
     your kernel's SHMMIN parameter, in which case raising the request size or recon
    figuring SHMMIN is called for.
     The PostgreSQL documentation contains more information about shared memo
    ry configuration.
                                                                             [fail]

This just started happening to me in 12.04.

I have the same problem with latest LTS Ubuntu, and the proper solution is to set BOTH SHMMAX and SHMALL to the same value, the best is ¼ of total memory.

kernel.shmmax = xxx
kernel.shmall = xxx

Still, I would consider this an real bug, especially because the postgres error is misleading and incomplete.

On 3 March 2014 13:27, Sorin Sbârnea <email address hidden> wrote:

> I have the same problem with latest LTS Ubuntu, and the proper solution
> is to set BOTH SHMMAX and SHMALL to the same value, the best is ¼ of
> total memory.
>
> kernel.shmmax = xxx
> kernel.shmall = xxx
>
> Still, I would consider this an real bug, especially because the
> postgres error is misleading and incomplete.

SHMALL is measured in pages, whereas SHMMAX is measured in bytes. They
shouldn't be set to the same values as one another.

Regards

Thom

can anyone help me I am getting that message on ubuntu I installed Ubuntu on my nexus 6 phone
# service postgresql start
[....] Starting PostgreSQL 9.4 database server: main[....] The PostgreSQL server failed to start. Please check the log output: 2016-04-01 15:06:57 UTC [31454-1] FATAL: could not create shared memory segment: Function not implemented 2016-04-01 15:06:57 UTC [31454-2] DETAI[FAILiled system call was shmget(key=5432001, size=40, 03600). ... failed!
 failed!