nday
CVE-2025-68310

s390/pci: Avoid deadlock between PCI error recovery and mlx5 crdump

Linux / Linux

Disclosed . nday has no fix record for it yet; that does not mean no patch exists. None of the four PoC sources lists it. Not in CISA KEV.

tags productLinux kernel product typeoperating system
why these tags

nday derives these tags by fixed rules from the facts below; NVD, the CNA and other sources do not assign them. Vocabulary and rules: ~/tags.

Linux kernel
affected products in the CVE record: Linux · Linux
operating system
affected products in the CVE record: Linux · Linuxthe product type of Linux kernel in nday's product registry
stage
DIS
disclosed
cvss
no score yet

Lifecycle

x axis UTC; dot = exact minute, wide bar = date only, hollow = unknown or inferred.
time unknown
DISdisclosed
2025-12-16T15:39Z
PATpatch
no record yet
ANAanalysis
no record yet
POCPoC
no record yet
ITWexploited
no record yet
KEVKEV
no record yet

Summary

In the Linux kernel, the following vulnerability has been resolved:

s390/pci: Avoid deadlock between PCI error recovery and mlx5 crdump

Do not block PCI config accesses through pci_cfg_access_lock() when executing the s390 variant of PCI error recovery: Acquire just device_lock() instead of pci_dev_lock() as powerpc's EEH and generig PCI AER processing do.

During error recovery testing a pair of tasks was reported to be hung:

mlx5_core 0000:00:00.1: mlx5_health_try_recover:338:(pid 5553): health recovery flow aborted, PCI reads still not working

INFO: task kmcheck:72 blocked for more than 122 seconds.
      Not tainted 5.14.0-570.12.1.bringup7.el9.s390x #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kmcheck         state:D stack:0     pid:72    tgid:72    ppid:2      flags:0x00000000
Call Trace:
 [<000000065256f030>] __schedule+0x2a0/0x590
 [<000000065256f356>] schedule+0x36/0xe0
 [<000000065256f572>] schedule_preempt_disabled+0x22/0x30
 [<0000000652570a94>] __mutex_lock.constprop.0+0x484/0x8a8
 [<000003ff800673a4>] mlx5_unload_one+0x34/0x58 [mlx5_core]
 [<000003ff8006745c>] mlx5_pci_err_detected+0x94/0x140 [mlx5_core]
 [<0000000652556c5a>] zpci_event_attempt_error_recovery+0xf2/0x398
 [<0000000651b9184a>] __zpci_event_error+0x23a/0x2c0
INFO: task kworker/u1664:6:1514 blocked for more than 122 seconds.
      Not tainted 5.14.0-570.12.1.bringup7.el9.s390x #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:kworker/u1664:6 state:D stack:0     pid:1514  tgid:1514  ppid:2      flags:0x00000000
Workqueue: mlx5_health0000:00:00.0 mlx5_fw_fatal_reporter_err_work [mlx5_core]
Call Trace:
 [<000000065256f030>] __schedule+0x2a0/0x590
 [<000000065256f356>] schedule+0x36/0xe0
 [<0000000652172e28>] pci_wait_cfg+0x80/0xe8
 [<0000000652172f94>] pci_cfg_access_lock+0x74/0x88
 [<000003ff800916b6>] mlx5_vsc_gw_lock+0x36/0x178 [mlx5_core]
 [<000003ff80098824>] mlx5_crdump_collect+0x34/0x1c8 [mlx5_core]
 [<000003ff80074b62>] mlx5_fw_fatal_reporter_dump+0x6a/0xe8 [mlx5_core]
 [<0000000652512242>] devlink_health_do_dump.part.0+0x82/0x168
 [<0000000652513212>] devlink_health_report+0x19a/0x230
 [<000003ff80075a12>] mlx5_fw_fatal_reporter_err_work+0xba/0x1b0 [mlx5_core]

No kernel log of the exact same error with an upstream kernel is available - but the very same deadlock situation can be constructed there, too:

- task: kmcheck mlx5_unload_one() tries to acquire devlink lock while the PCI error recovery code has set pdev->block_cfg_access by way of pci_cfg_access_lock() - task: kworker mlx5_crdump_collect() tries to set block_cfg_access through pci_cfg_access_lock() while devlink_health_report() had acquired the devlink lock.

A similar deadlock situation can be reproduced by requesting a crdump with > devlink health dump show pci/<BDF> reporter fw_fatal

while PCI error recovery is executed on the same <BDF> physical function by mlx5_core's pci_error_handlers. On s390 this can be injected with > zpcictl --reset-fw <BDF>

Tests with this patch failed to reproduce that second deadlock situation, the devlink command is rejected with "kernel answers: Permission denied" - and we get a kernel log message of:

mlx5_core 1ed0:00:00.1: mlx5_crdump_collect:50:(pid 254382): crdump: failed to lock vsc gw err -5

because the config read of VSC_SEMAPHORE is rejected by the underlying hardware.

Two prior attempts to address this issue have been discussed and ultimately rejected [see link], with the primary argument that s390's implementation of PCI error recovery is imposing restrictions that neither powerpc's EEH nor PCI AER handling need. Tests show that PCI error recovery on s390 is running to completion even without blocking access to PCI config space.

Original text of the CVE record (cvelist), shown as published.

Event log

+DISdisclosed

CVE record published by Linux.

src cvelist · nday , bulk import · raw
{
  "id": 391076,
  "type": "disclosed",
  "source": "cvelist",
  "url": "https://www.cve.org/CVERecord?id=CVE-2025-68310",
  "occurred_at": "2025-12-16T15:39:41.652Z",
  "observed_at": "2026-11-13T05:39:20.395751Z",
  "is_backfill": true,
  "dedupe_key": "",
  "detail": {
    "state": "PUBLISHED",
    "date_published": "2025-12-16T15:39:41.652Z",
    "assigner_short_name": "Linux"
  }
}

Report an error

report an error or omission

Something on this page wrong or missing? Tell the editors; if they accept it, this page's revision log says so publicly. Upstream data is not changed.

Researcher discussion

0 researchers mentioned this in the last 7 days

Posts on X by researchers on nday's list that mention this CVE, shown as posted. nday does not check them.

nday has not collected any post on X by listed researchers that mentions this CVE.

Discussion

0 comments

No comments yet.