[Bug][CLX]assertion failure with util_range_rw using libpmemlog, possible kernel DAX bug

Bug #1789146 reported by quanxian
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
intel
Fix Released
Medium
Unassigned
linux (Ubuntu)
Fix Released
Medium
Joseph Salisbury

Bug Description

Description:
On Mon, Apr 16, 2018 at 8:20 AM, Czurylo, Krzysztof
<email address hidden> wrote:
>
> I suspect the problem is caused by a bug in the kernel.
>
> I did a few experiments and it looks like the issue occurs only if the
> filesystem is mounted with "-o dax". I can reproduce is both for xfs
> and ext4, so it's not FS-specific, but rather DAX-specific. It also
> reproduces on an emulated PMEM - no need to use real AEP DIMMs.
>
> Using the latest kernel (4.16.0) does not help.
>
> What happens:
>
> In debug version of libpmemlog (but also libpmemblk), the entire pool
> is by default write-protected with mprotect(..., PROT_READ).
>
> When the program attempts to write some data to the pool (i.e.
> pmemlog_append, pmemblk_write, ...), the library unprotects the pages
> to be modified (usually just one or two pages) and once the data is
> stored, the pages are protected again.
>
> Inside the kernel, mprotect splits the memory region associated with
> the pool into 3 regions: the read-only head and tail + one r/w page in
> the middle.
>
> The problem is that after the last step, the memory region associated
> with the modified page is not merged with the adjacent regions having
> the same protection flags (ro) to form one big read-only region again.
> This leads to the situation where we have thousands of 4K memory
> mappings per process that are tracked by the kernel separately. When
> the number of maps exceeds the limit (default is 65536 - see:
> /proc/sys/vm/max_map_count), mprotect fails with ENOMEM, which aborts
> the program.

Commitid: e1fb4a0864958fac2fb1b23f9f4562a9f90e3e8f
dax: remove VM_MIXEDMAP for fsdax and device dax

Target Kernel: 4.19

Target Release: 18.10

CVE References

Revision history for this message
quanxian (quanxian-wang) wrote :

this patch is not in 4.18.

If possible, please cherry pick it into Ubuntu 18.10. Thanks

Revision history for this message
quanxian (quanxian-wang) wrote :

clx platform is not PV, just keep it private. Thanks

quanxian (quanxian-wang)
information type: Public → Private
Revision history for this message
quanxian (quanxian-wang) wrote :

need backporting

Changed in intel:
status: New → Triaged
importance: Undecided → Medium
Changed in ubuntu:
importance: Undecided → Medium
status: New → Triaged
affects: ubuntu → linux (Ubuntu)
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

I built a test kernel with commit e1fb4a0864958fac2fb1b23f9f4562a9f90e3e8f. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1789146

Can you test this kernel and see if it resolves this bug?

Note about installing test kernels:
• If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages.
• If the test kernel is 4.15(Bionic) or newer, you need to install the linux-modules, linux-modules-extra and linux-image-unsigned .deb packages.

Thanks in advance!

Changed in linux (Ubuntu):
assignee: nobody → Joseph Salisbury (jsalisbury)
status: Triaged → In Progress
Revision history for this message
quanxian (quanxian-wang) wrote :

I am finding the test case from upstream to have a try.

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :
Revision history for this message
quanxian (quanxian-wang) wrote :

hi, Joseph

Our test has tested in 4.16(no patch) and 4.19(with the patch), it works.

But with your build image, we don't find the bug is fixed. Would you like to double check if the patch has been integrated your kernel. Thanks

Quanxian

Changed in linux (Ubuntu):
status: In Progress → Fix Committed
information type: Private → Public
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

I built a 18.10 test kernel using the latest master-next branch of the repo. This branch has the patch applied.

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1789146

Can you test this kernel and see if it resolves this bug?

Revision history for this message
quanxian (quanxian-wang) wrote :

hi, Joseph

I have built a kernel by myself clone from below link. The version is 4.18.

git://kernel.ubuntu.com/kernel-ppa/mirror/ubuntu-cosmic.git
commit d4b160782ac74f5301651346495903a30cf752d3 (HEAD -> master, tag: Ubuntu-4.18.0-7.8, origin/master, origin/HEAD)
Author: Seth Forshee <email address hidden>
Date: Tue Aug 28 11:09:06 2018 -0500

    UBUNTU: Ubuntu-4.18.0-7.8

    Signed-off-by: Seth Forshee <email address hidden>

in this branch, there is no patch. Test failed.
I apply the new patch, the test works.

Therefore for 4.18, it will be fine with patch.

For your new kernel, I will have a try, will let you know the result.

Revision history for this message
quanxian (quanxian-wang) wrote :

we have verified your kernel package, it works. Thanks

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (60.2 KiB)

This bug was fixed in the package linux - 4.18.0-9.10

---------------
linux (4.18.0-9.10) cosmic; urgency=medium

  * linux: 4.18.0-9.10 -proposed tracker (LP: #1796346)

  * Cosmic update: v4.18.12 upstream stable release (LP: #1796139)
    - crypto: skcipher - Fix -Wstringop-truncation warnings
    - iio: adc: ina2xx: avoid kthread_stop() with stale task_struct
    - tsl2550: fix lux1_input error in low light
    - misc: ibmvmc: Use GFP_ATOMIC under spin lock
    - vmci: type promotion bug in qp_host_get_user_memory()
    - siox: don't create a thread without starting it
    - x86/numa_emulation: Fix emulated-to-physical node mapping
    - staging: rts5208: fix missing error check on call to rtsx_write_register
    - power: supply: axp288_charger: Fix initial constant_charge_current value
    - misc: sram: enable clock before registering regions
    - serial: sh-sci: Stop RX FIFO timer during port shutdown
    - uwb: hwa-rc: fix memory leak at probe
    - power: vexpress: fix corruption in notifier registration
    - iommu/amd: make sure TLB to be flushed before IOVA freed
    - Bluetooth: Add a new Realtek 8723DE ID 0bda:b009
    - USB: serial: kobil_sct: fix modem-status error handling
    - 6lowpan: iphc: reset mac_header after decompress to fix panic
    - iommu/msm: Don't call iommu_device_{,un}link from atomic context
    - s390/mm: correct allocate_pgste proc_handler callback
    - power: remove possible deadlock when unregistering power_supply
    - drm/amd/display/dc/dce: Fix multiple potential integer overflows
    - drm/amd/display: fix use of uninitialized memory
    - md-cluster: clear another node's suspend_area after the copy is finished
    - cxgb4: Fix the condition to check if the card is T5
    - RDMA/bnxt_re: Fix a couple off by one bugs
    - RDMA/i40w: Hold read semaphore while looking after VMA
    - RDMA/bnxt_re: Fix a bunch of off by one bugs in qplib_fp.c
    - IB/core: type promotion bug in rdma_rw_init_one_mr()
    - media: exynos4-is: Prevent NULL pointer dereference in __isp_video_try_fmt()
    - IB/mlx4: Test port number before querying type.
    - powerpc/kdump: Handle crashkernel memory reservation failure
    - media: fsl-viu: fix error handling in viu_of_probe()
    - vhost_net: Avoid tx vring kicks during busyloop
    - media: staging/imx: fill vb2_v4l2_buffer field entry
    - IB/mlx5: Fix GRE flow specification
    - include/rdma/opa_addr.h: Fix an endianness issue
    - x86/tsc: Add missing header to tsc_msr.c
    - ARM: hwmod: RTC: Don't assume lock/unlock will be called with irq enabled
    - x86/entry/64: Add two more instruction suffixes
    - ARM: dts: ls1021a: Add missing cooling device properties for CPUs
    - scsi: target/iscsi: Make iscsit_ta_authentication() respect the output
      buffer size
    - thermal: i.MX: Allow thermal probe to fail gracefully in case of bad
      calibration.
    - scsi: klist: Make it safe to use klists in atomic context
    - scsi: ibmvscsi: Improve strings handling
    - scsi: target: Avoid that EXTENDED COPY commands trigger lock inversion
    - usb: wusbcore: security: cast sizeof to int for comparison
    - ath10k: sdio: use same endpoint id for all packets...

Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
quanxian (quanxian-wang)
Changed in intel:
status: Triaged → Fix Released
Revision history for this message
Brad Figg (brad-figg) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-bionic
Andy Whitcroft (apw)
tags: added: kernel-fixup-verification-needed-bionic
removed: verification-needed-bionic
Brad Figg (brad-figg)
tags: added: verification-needed-bionic
Revision history for this message
Andy Whitcroft (apw) wrote :

This bug was erroneously marked for verification in bionic; verification is not required and verification-needed-bionic is being removed.

tags: removed: verification-needed-bionic
tags: added: verification-done-bionic
Brad Figg (brad-figg)
tags: added: cscc
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.