Discussion:
Bug#793372: systemd breaks LXC cgroup memory limitations
Pablo Abelenda
2015-07-23 11:57:06 UTC
Permalink
Package: systemd
Version: 215-17+deb8u1
Severity: important

Dear Maintainer,

I have been checking on LXC for a while. I wanted to acquaint with the
containers management, specifically with the resources limitation
capabilities.

I have created, configured and destroyed several containers. The issue
comes when I try to limit the memory usage. I have added this option to
the grub defaults:

cgroup_enable=memory

This allowed me to start a container with options configured such as
this:

lxc.cgroup.memory.limit_in_bytes = 1024M
lxc.cgroup.memory.memsw.limit_in_bytes = 1200M

The thing is that when I try to use more than 1200 MB of memory, inside
the container, I actually can. I have used a very simply script I found
on someone page to test the memory allocation capabilities.

########################################
memory_allocation.py
########################################
#!/usr/bin/env python
import sys
import time

if len(sys.argv) != 2:
print "usage: fillmem <number-of-megabytes>"
sys.exit()

count = int(sys.argv[1])

megabyte = (0,) * (1024 * 1024 / 8)

data = megabyte * count

while True:
time.sleep(1)
########################################

I have tried to isolate the problem, focusing on some missing LXC or
cgroups configuration. Let me add the trace:

First, I have created a new control group, outside the LXC stuff:

cgcreate -g memory:/bmalloc1gb
echo $(( 1024 * 1024 * 1024 ))|tee /sys/fs/cgroup/memory/bmalloc1gb/memory.limit_in_bytes
echo $(( 1200 * 1024 * 1024 ))|tee /sys/fs/cgroup/memory/bmalloc1gb/memory.memsw.limit_in_bytes
cgexec -g memory:/bmalloc1gb bash
python memory_allocation.py 2000 [killed]

When I run the script on this environment, the memory allocation did not
went far from this 1200 MB, which is what I need.

Second, I have joined the LXC cgroup, to try the very same as above:

cgexec -g memory:lxc/test bash
python memory_allocation.py 2000 [killed]

This two tests made me point to LXC, but after some internet digging I
found this:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=777601

In this bug report, which is not exactly the same as this one, the
reporter states this:

"I also tested something which may be another problem (so another ticket)
but the memory and swap limitation doesn't work at all :-(...with or
without this bug. I can't tell if it's linked or not.
I tested cpu pinning and it works like a charm. The issue seems related
to memory and swap only."

After this income information, I have switched back both the host and
the container, to sysvinit. With the systems booted on sysvinit, the
memory limitation is working as it is expected.

Let me add the whole container configuration for your reference:

lxc.cgroup.cpuset.cpus = 0,1
lxc.cgroup.cpu.shares = 256
lxc.cgroup.blkio.weight = 500
lxc.cgroup.memory.limit_in_bytes = 1024M
lxc.cgroup.memory.memsw.limit_in_bytes = 1200M
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.rootfs = /dev/vg/test
lxc.start.auto = 1
lxc.include = /usr/share/lxc/config/debian.common.conf
lxc.mount = /var/lib/lxc/test/fstab
lxc.utsname = test
lxc.arch = amd64
lxc.autodev = 1

Thanks for your consideration.

Best regards.


-- Package-specific info:

-- System Information:
Debian Release: 8.1
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages systemd depends on:
ii acl 2.2.52-2
ii adduser 3.113+nmu3
ii initscripts 2.88dsf-59
ii libacl1 2.2.52-2
ii libaudit1 1:2.4-1+b1
ii libblkid1 2.25.2-6
ii libc6 2.19-18
ii libcap2 1:2.24-8
ii libcap2-bin 1:2.24-8
ii libcryptsetup4 2:1.6.6-5
ii libgcrypt20 1.6.3-2
ii libkmod2 18-3
ii liblzma5 5.1.1alpha+20120614-2+b3
ii libpam0g 1.1.8-3.1
ii libselinux1 2.3-2
ii libsystemd0 215-17+deb8u1
ii mount 2.25.2-6
ii sysv-rc 2.88dsf-59
ii udev 215-17+deb8u1
ii util-linux 2.25.2-6

Versions of packages systemd recommends:
ii dbus 1.8.18-0+deb8u1
ii libpam-systemd 215-17+deb8u1

Versions of packages systemd suggests:
pn systemd-ui <none>

-- no debconf information
Carlos Alberto Lopez Perez
2015-07-24 00:44:56 UTC
Permalink
Just CC'ing the Debian LXC maintainer and the upstream LXC mailing lists.

I guess they may find relevant this bug.

https://bugs.debian.org/793372

Regards.
Post by Pablo Abelenda
Package: systemd
Version: 215-17+deb8u1
Severity: important
Dear Maintainer,
I have been checking on LXC for a while. I wanted to acquaint with the
containers management, specifically with the resources limitation
capabilities.
I have created, configured and destroyed several containers. The issue
comes when I try to limit the memory usage. I have added this option to
cgroup_enable=memory
This allowed me to start a container with options configured such as
lxc.cgroup.memory.limit_in_bytes = 1024M
lxc.cgroup.memory.memsw.limit_in_bytes = 1200M
The thing is that when I try to use more than 1200 MB of memory, inside
the container, I actually can. I have used a very simply script I found
on someone page to test the memory allocation capabilities.
########################################
memory_allocation.py
########################################
#!/usr/bin/env python
import sys
import time
print "usage: fillmem <number-of-megabytes>"
sys.exit()
count = int(sys.argv[1])
megabyte = (0,) * (1024 * 1024 / 8)
data = megabyte * count
time.sleep(1)
########################################
I have tried to isolate the problem, focusing on some missing LXC or
cgcreate -g memory:/bmalloc1gb
echo $(( 1024 * 1024 * 1024 ))|tee /sys/fs/cgroup/memory/bmalloc1gb/memory.limit_in_bytes
echo $(( 1200 * 1024 * 1024 ))|tee /sys/fs/cgroup/memory/bmalloc1gb/memory.memsw.limit_in_bytes
cgexec -g memory:/bmalloc1gb bash
python memory_allocation.py 2000 [killed]
When I run the script on this environment, the memory allocation did not
went far from this 1200 MB, which is what I need.
cgexec -g memory:lxc/test bash
python memory_allocation.py 2000 [killed]
This two tests made me point to LXC, but after some internet digging I
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=777601
In this bug report, which is not exactly the same as this one, the
"I also tested something which may be another problem (so another ticket)
but the memory and swap limitation doesn't work at all :-(...with or
without this bug. I can't tell if it's linked or not.
I tested cpu pinning and it works like a charm. The issue seems related
to memory and swap only."
After this income information, I have switched back both the host and
the container, to sysvinit. With the systems booted on sysvinit, the
memory limitation is working as it is expected.
lxc.cgroup.cpuset.cpus = 0,1
lxc.cgroup.cpu.shares = 256
lxc.cgroup.blkio.weight = 500
lxc.cgroup.memory.limit_in_bytes = 1024M
lxc.cgroup.memory.memsw.limit_in_bytes = 1200M
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.rootfs = /dev/vg/test
lxc.start.auto = 1
lxc.include = /usr/share/lxc/config/debian.common.conf
lxc.mount = /var/lib/lxc/test/fstab
lxc.utsname = test
lxc.arch = amd64
lxc.autodev = 1
Thanks for your consideration.
Best regards.
Debian Release: 8.1
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 3.16.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
ii acl 2.2.52-2
ii adduser 3.113+nmu3
ii initscripts 2.88dsf-59
ii libacl1 2.2.52-2
ii libaudit1 1:2.4-1+b1
ii libblkid1 2.25.2-6
ii libc6 2.19-18
ii libcap2 1:2.24-8
ii libcap2-bin 1:2.24-8
ii libcryptsetup4 2:1.6.6-5
ii libgcrypt20 1.6.3-2
ii libkmod2 18-3
ii liblzma5 5.1.1alpha+20120614-2+b3
ii libpam0g 1.1.8-3.1
ii libselinux1 2.3-2
ii libsystemd0 215-17+deb8u1
ii mount 2.25.2-6
ii sysv-rc 2.88dsf-59
ii udev 215-17+deb8u1
ii util-linux 2.25.2-6
ii dbus 1.8.18-0+deb8u1
ii libpam-systemd 215-17+deb8u1
pn systemd-ui <none>
-- no debconf information
Carlos Alberto Lopez Perez
2016-01-20 22:43:52 UTC
Permalink
Post by Pablo Abelenda
After this income information, I have switched back both the host and
the container, to sysvinit. With the systems booted on sysvinit, the
memory limitation is working as it is expected.
I have been investigating this further and I'm not sure at this point if
the bug is on systemd or LXC.

I have posted my findings to the lxc-users mailing list:

http://thread.gmane.org/gmane.linux.kernel.containers.lxc.general/10825
Michael Biebl
2016-01-30 03:00:47 UTC
Permalink
On Wed, 20 Jan 2016 23:43:52 +0100 Carlos Alberto Lopez Perez
Post by Carlos Alberto Lopez Perez
Post by Pablo Abelenda
After this income information, I have switched back both the host and
the container, to sysvinit. With the systems booted on sysvinit, the
memory limitation is working as it is expected.
I have been investigating this further and I'm not sure at this point if
the bug is on systemd or LXC.
http://thread.gmane.org/gmane.linux.kernel.containers.lxc.general/10825
So, can you be more specific where the bug in systemd is?
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Michael Biebl
2016-12-15 21:57:31 UTC
Permalink
Control: tags -1 + moreinfo
Post by Michael Biebl
On Wed, 20 Jan 2016 23:43:52 +0100 Carlos Alberto Lopez Perez
Post by Carlos Alberto Lopez Perez
Post by Pablo Abelenda
After this income information, I have switched back both the host and
the container, to sysvinit. With the systems booted on sysvinit, the
memory limitation is working as it is expected.
I have been investigating this further and I'm not sure at this point if
the bug is on systemd or LXC.
http://thread.gmane.org/gmane.linux.kernel.containers.lxc.general/10825
So, can you be more specific where the bug in systemd is?
I guess this question is still open. So tagging the bug report accordingly.
--
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?
Debian Bug Tracking System
2016-12-15 22:00:03 UTC
Permalink
Post by Michael Biebl
tags -1 + moreinfo
Bug #793372 [systemd] systemd breaks LXC cgroup memory limitations
Added tag(s) moreinfo.
--
793372: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=793372
Debian Bug Tracking System
Contact ***@bugs.debian.org with problems
Debian Bug Tracking System
2017-02-13 04:21:06 UTC
Permalink
Your message dated Mon, 13 Feb 2017 05:19:44 +0100
with message-id <cab5fefa-3777-4ac2-0efa-***@debian.org>
and subject line Re: systemd breaks LXC cgroup memory limitations
has caused the Debian Bug report #793372,
regarding systemd breaks LXC cgroup memory limitations
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ***@bugs.debian.org
immediately.)
--
793372: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=793372
Debian Bug Tracking System
Contact ***@bugs.debian.org with problems
Loading...