summaryrefslogtreecommitdiff
path: root/drivers/accel/ivpu/ivpu_job.c
AgeCommit message (Collapse)Author
2025-06-16accel/ivpu: Add turbo flag to the DRM_IVPU_CMDQ_CREATE ioctlAndrzej Kacprowski
Introduce a new parameter to the DRM_IVPU_CMDQ_CREATE ioctl, enabling turbo mode for jobs submitted via the command queue. Turbo mode allows jobs to run at higher frequencies, potentially improving performance for demanding workloads. Also adds the IVPU_TEST_MODE_TURBO_DISABLE flag to allow test mode to explicitly disable turbo mode requested by the application. The IVPU_TEST_MODE_TURBO mode has been renamed to IVPU_TEST_MODE_TURBO_ENABLE for clarity and consistency. Signed-off-by: Andrzej Kacprowski <Andrzej.Kacprowski@intel.com> Signed-off-by: Maciej Falkowski <maciej.falkowski@linux.intel.com> Reviewed-by: Jeff Hugo <jeff.hugo@oss.qualcomm.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://lore.kernel.org/r/20250605162001.1237789-1-maciej.falkowski@linux.intel.com
2025-06-05accel/ivpu: Trigger device recovery on engine reset/resume failureKarol Wachowski
Trigger full device recovery when the driver fails to restore device state via engine reset and resume operations. This is necessary because, even if submissions from a faulty context are blocked, the NPU may still process previously submitted faulty jobs if the engine reset fails to abort them. Such jobs can continue to generate faults and occupy device resources. When engine reset is ineffective, the only way to recover is to perform a full device recovery. Fixes: dad945c27a42 ("accel/ivpu: Add handling of VPU_JSM_STATUS_MVNCI_CONTEXT_VIOLATION_HW") Cc: stable@vger.kernel.org # v6.15+ Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Reviewed-by: Lizhi Hou <lizhi.hou@amd.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://lore.kernel.org/r/20250528154253.500556-1-jacek.lawrynowicz@linux.intel.com
2025-05-28accel/ivpu: Reorder Doorbell Unregister and Command Queue DestructionKarol Wachowski
Refactor ivpu_cmdq_unregister() to ensure the doorbell is unregistered before destroying the command queue. The NPU firmware requires doorbells to be unregistered prior to command queue destruction. If doorbell remains registered when command queue destroy command is sent firmware will automatically unregister the doorbell, making subsequent unregister attempts no-operations (NOPs). Ensure compliance with firmware expectations by moving the doorbell unregister call ahead of the command queue destruction logic, thus preventing unnecessary NOP operation. Fixes: 465a3914b254 ("accel/ivpu: Add API for command queue create/destroy/submit") Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Reviewed-by: Jeff Hugo <jeff.hugo@oss.qualcomm.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://lore.kernel.org/r/20250515094124.255141-1-jacek.lawrynowicz@linux.intel.com
2025-04-30accel/ivpu: Correct mutex unlock order in job submissionKarol Wachowski
The mutex unlock for vdev->submitted_jobs_lock was incorrectly placed before unlocking file_priv->lock. Change order of unlocks to avoid potential race conditions. Fixes: 5bbccadaf33e ("accel/ivpu: Abort all jobs after command queue unregister") Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Reviewed-by: Jeff Hugo <jeff.hugo@oss.qualcomm.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://lore.kernel.org/r/20250425093656.2228168-1-jacek.lawrynowicz@linux.intel.com
2025-04-30accel/ivpu: Fix pm related deadlocks in cmdq ioctlsJacek Lawrynowicz
Fix deadlocks in ivpu_cmdq_create_ioctl() and ivpu_cmdq_destroy_ioctl() related to runtime suspend. Runtime suspend acquires file_priv->lock mutex by calling ivpu_cmdq_reset_all_contexts(). The same lock is acquired in the cmdq ioctls. If one of the cmdq ioctls is called while runtime suspend is in progress, it can lead to a deadlock. Call stacks from example deadlock below. Runtime suspend thread: [ 3443.179717] Call Trace: [ 3443.179724] __schedule+0x4b6/0x16b0 [ 3443.179732] ? __mod_timer+0x27d/0x3a0 [ 3443.179738] schedule+0x2f/0x140 [ 3443.179741] schedule_preempt_disabled+0x19/0x30 [ 3443.179743] __mutex_lock.constprop.0+0x335/0x7d0 [ 3443.179745] ? xas_find+0x1ed/0x260 [ 3443.179747] ? xa_find+0x8e/0xf0 [ 3443.179749] __mutex_lock_slowpath+0x13/0x20 [ 3443.179751] mutex_lock+0x41/0x60 [ 3443.179757] ivpu_cmdq_reset_all_contexts+0x82/0x150 [intel_vpu a9bd091a97f28f0235f161316b29f8234f437295] [ 3443.179786] ivpu_pm_runtime_suspend_cb+0x1f1/0x3f0 [intel_vpu a9bd091a97f28f0235f161316b29f8234f437295] [ 3443.179850] pci_pm_runtime_suspend+0x6e/0x1f0 [ 3443.179870] ? __pfx_pci_pm_runtime_suspend+0x10/0x10 [ 3443.179886] __rpm_callback+0x48/0x130 [ 3443.179899] rpm_callback+0x64/0x70 [ 3443.179911] rpm_suspend+0x12c/0x630 [ 3443.179922] ? __schedule+0x4be/0x16b0 [ 3443.179941] pm_runtime_work+0xca/0xf0 [ 3443.179955] process_one_work+0x188/0x3d0 [ 3443.179971] worker_thread+0x2b9/0x3c0 [ 3443.179984] kthread+0xfb/0x220 [ 3443.180001] ? __pfx_worker_thread+0x10/0x10 [ 3443.180013] ? __pfx_kthread+0x10/0x10 [ 3443.180029] ret_from_fork+0x47/0x70 [ 3443.180044] ? __pfx_kthread+0x10/0x10 [ 3443.180059] ret_from_fork_asm+0x1a/0x30 User space thread: [ 3443.180128] Call Trace: [ 3443.180138] __schedule+0x4b6/0x16b0 [ 3443.180159] schedule+0x2f/0x140 [ 3443.180163] rpm_resume+0x1a7/0x6a0 [ 3443.180165] ? __pfx_autoremove_wake_function+0x10/0x10 [ 3443.180169] __pm_runtime_resume+0x56/0x90 [ 3443.180171] ivpu_rpm_get+0x28/0xb0 [intel_vpu a9bd091a97f28f0235f161316b29f8234f437295] [ 3443.180181] ivpu_ipc_send_receive+0x6d/0x120 [intel_vpu a9bd091a97f28f0235f161316b29f8234f437295] [ 3443.180193] ? free_frozen_pages+0x395/0x670 [ 3443.180199] ? __free_pages+0xa7/0xc0 [ 3443.180202] ivpu_jsm_hws_destroy_cmdq+0x76/0xf0 [intel_vpu a9bd091a97f28f0235f161316b29f8234f437295] [ 3443.180213] ? locks_dispose_list+0x6c/0xa0 [ 3443.180219] ? kmem_cache_free+0x342/0x470 [ 3443.180222] ? vm_area_free+0x19/0x30 [ 3443.180225] ? xas_load+0x17/0xf0 [ 3443.180229] ? xa_load+0x72/0xb0 [ 3443.180230] ivpu_cmdq_unregister.isra.0+0xb1/0x100 [intel_vpu a9bd091a97f28f0235f161316b29f8234f437295] [ 3443.180241] ivpu_cmdq_destroy_ioctl+0x8d/0x130 [intel_vpu a9bd091a97f28f0235f161316b29f8234f437295] [ 3443.180251] ? __pfx_ivpu_cmdq_destroy_ioctl+0x10/0x10 [intel_vpu a9bd091a97f28f0235f161316b29f8234f437295] [ 3443.180260] drm_ioctl_kernel+0xb3/0x110 [ 3443.180265] drm_ioctl+0x2ca/0x580 [ 3443.180266] ? __pfx_ivpu_cmdq_destroy_ioctl+0x10/0x10 [intel_vpu a9bd091a97f28f0235f161316b29f8234f437295] [ 3443.180275] ? __fput+0x1ae/0x2f0 [ 3443.180279] ? kmem_cache_free+0x342/0x470 [ 3443.180282] __x64_sys_ioctl+0xa9/0xe0 [ 3443.180286] x64_sys_call+0x13b7/0x26f0 [ 3443.180289] do_syscall_64+0x62/0x180 [ 3443.180291] entry_SYSCALL_64_after_hwframe+0x71/0x79 Fixes: 465a3914b254 ("accel/ivpu: Add API for command queue create/destroy/submit") Reviewed-by: Jeff Hugo <jeff.hugo@oss.qualcomm.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://lore.kernel.org/r/20250425093341.2202895-1-jacek.lawrynowicz@linux.intel.com
2025-04-11accel/ivpu: Add cmdq_id to job related logsKarol Wachowski
Add tracking of command queue ID in JOB debug message to improve debugging capabilities. Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Signed-off-by: Maciej Falkowski <maciej.falkowski@linux.intel.com> Reviewed-by: Lizhi Hou <lizhi.hou@amd.com> Reviewed-by: Jeff Hugo <jeff.hugo@oss.qualcomm.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://lore.kernel.org/r/20250401155939.4049467-1-maciej.falkowski@linux.intel.com
2025-02-10accel/ivpu: Prevent runtime suspend during context abort workAndrzej Kacprowski
Increment the runtime PM counter when entering ivpu_context_abort_work_fn() to prevent the device from suspending while the function is executing. Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Signed-off-by: Andrzej Kacprowski <Andrzej.Kacprowski@intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250204084622.2422544-3-jacek.lawrynowicz@linux.intel.com
2025-02-03accel/ivpu: Fix missing MMU events if file_priv is unboundKarol Wachowski
Move the ivpu_mmu_discard_events() function to the common portion of the abort work function. This ensures it is called only once, even if there are no faulty contexts in context_xa, to guarantee that MMU events are discarded and new events are not missed. Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250129125636.1047413-4-jacek.lawrynowicz@linux.intel.com
2025-01-09accel/ivpu: Add handling of VPU_JSM_STATUS_MVNCI_CONTEXT_VIOLATION_HWKarol Wachowski
Mark as invalid context of a job that returned HW context violation error and queue work that aborts jobs from faulty context. Add engine reset to the context abort thread handler to not only abort currently executing jobs but also to ensure NPU invalid state recovery. Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Signed-off-by: Maciej Falkowski <maciej.falkowski@linux.intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250107173238.381120-13-maciej.falkowski@linux.intel.com
2025-01-09accel/ivpu: Fix locking order in ivpu_job_submitKarol Wachowski
Fix deadlock in job submission and abort handling. When a thread aborts currently executing jobs due to a fault, it first locks the global lock protecting submitted_jobs (#1). After the last job is destroyed, it proceeds to release the related context and locks file_priv (#2). Meanwhile, in the job submission thread, the file_priv lock (#2) is taken first, and then the submitted_jobs lock (#1) is obtained when a job is added to the submitted jobs list. CPU0 CPU1 ---- ---- (for example due to a fault) (jobs submissions keep coming) lock(&vdev->submitted_jobs_lock) #1 ivpu_jobs_abort_all() job_destroy() lock(&file_priv->lock) #2 lock(&vdev->submitted_jobs_lock) #1 file_priv_release() lock(&vdev->context_list_lock) lock(&file_priv->lock) #2 This order of locking causes a deadlock. To resolve this issue, change the order of locking in ivpu_job_submit(). Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Signed-off-by: Maciej Falkowski <maciej.falkowski@linux.intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250107173238.381120-12-maciej.falkowski@linux.intel.com
2025-01-09accel/ivpu: Fix locking order in ivpu_cmdq_destroy_ioctlKarol Wachowski
Fix deadlock caused by inversed locking order in ivpu_job_submit() and ivpu_cmdq_destroy_ioctl(). Both functions operate locking file_priv->lock and submitted_jobs_lock. Unlock file_priv->lock in ivpu_cmdq_destroy_ioctl() before calling ivpu_cmdq_abort_all_jobs() function which locks submitted_jobs_lock. That way locking order is maintained: 1) global submitted_jobs_lock first 2) per context file_priv->lock second Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Signed-off-by: Maciej Falkowski <maciej.falkowski@linux.intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250107173238.381120-11-maciej.falkowski@linux.intel.com
2025-01-09accel/ivpu: Set command queue management capability based on HWSKarol Wachowski
Control explicit command queue management capability bit based on scheduling mode. Capability will be available only when hardware scheduling mode is set. There is no point of allowing user space to create and destroy command queues with OS schedling mode because FW does not support all required functionalities for correct command queue management with OS scheduling. Return -ENODEV from command queue create/destroy/submit IOCTLs. Remove is_valid field from struct ivpu_job_cmdq Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Signed-off-by: Maciej Falkowski <maciej.falkowski@linux.intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250107173238.381120-10-maciej.falkowski@linux.intel.com
2025-01-09accel/ivpu: Move parts of MMU event IRQ handling to thread handlerKarol Wachowski
To prevent looping infinitely in MMU event handler we stop generating new events by removing 'R' (record) bit from context descriptor, but to ensure this change has effect KMD has to perform configuration invalidation followed by sync command. Because of that move parts of the interrupt handler that can take longer to a thread not to block in interrupt handler for too long. This includes: * disabling event queue for the time KMD updates MMU event queue consumer to ensure proper synchronization between MMU and KMD * removal of 'R' (record) bit from context descriptor to ensure no more faults are recorded until that context is destroyed Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Signed-off-by: Maciej Falkowski <maciej.falkowski@linux.intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250107173238.381120-8-maciej.falkowski@linux.intel.com
2025-01-09accel/ivpu: Use workqueue for IRQ handlingMaciej Falkowski
Convert IRQ bottom half from the thread handler into workqueue. This increases a stability in rare scenarios where driver on debugging/hardening kernels processes IRQ too slow and misses some interrupts due to it. Workqueue handler also gives a very minor performance increase. Signed-off-by: Maciej Falkowski <maciej.falkowski@linux.intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250107173238.381120-6-maciej.falkowski@linux.intel.com
2025-01-09accel/ivpu: Abort all jobs after command queue unregisterKarol Wachowski
With hardware scheduler it is not expected to receive JOB_DONE notifications from NPU FW for the jobs aborted due to command queue destroy JSM command. Remove jobs submitted to unregistered command queue from submitted_jobs_xa to avoid triggering a TDR in such case. Add explicit submitted_jobs_lock that protects access to list of submitted jobs which is now used to find jobs to abort. Move context abort procedure to separate work queue not to slow down handling of IPCs or DCT requests in case where job abort takes longer, especially when destruction of the last job of a specific context results in context release. Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Signed-off-by: Maciej Falkowski <maciej.falkowski@linux.intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250107173238.381120-4-maciej.falkowski@linux.intel.com
2025-01-09accel/ivpu: Add API for command queue create/destroy/submitKarol Wachowski
Implement support for explicit command queue management. To allow more flexible control over command queues add capabilities to create, destroy and submit jobs to specific command queues. Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Signed-off-by: Maciej Falkowski <maciej.falkowski@linux.intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250107173238.381120-3-maciej.falkowski@linux.intel.com
2025-01-09accel/ivpu: Separate DB ID and CMDQ ID allocations from CMDQ allocationKarol Wachowski
Move doorbell ID and command queue ID XArray allocations from command queue memory allocation function. This will allow ID allocations to be done without the need for actual memory allocation. Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Signed-off-by: Maciej Falkowski <maciej.falkowski@linux.intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20250107173238.381120-2-maciej.falkowski@linux.intel.com
2024-10-30accel/ivpu: Move secondary preemption buffer allocation to DMA rangeKarol Wachowski
Secondary preemption buffer is accessible by NPU's DMA and can be allocated with addresses above 4 GB. Move secondary preemption buffer allocation from SHAVE range which is much smaller (2GB) to DMA range. This allows to allocate more command queues with corresponding preemption buffers without running out of address range. Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20241017145817.121590-12-jacek.lawrynowicz@linux.intel.com
2024-10-30accel/ivpu: Don't allocate preemption buffers when MIP is disabledKarol Wachowski
Do not allocate preemption buffers when Mid Inference Preemption (MIP) is disabled through test mode. Rename IVPU_TEST_MODE_PREEMPTION_DISABLE to IVPU_TEST_MODE_MIP_DISABLE to better describe that this test mode only disables MIP - job level preemption will still occur. Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20241017145817.121590-9-jacek.lawrynowicz@linux.intel.com
2024-10-30accel/ivpu: Make command queue ID allocated on XArrayKarol Wachowski
Use XArray for dynamic command queue ID allocations instead of fixed ones. This is required by upcoming changes to UAPI that will allow to manage command queues by user space instead of having predefined number of queues in a context. Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20241017145817.121590-8-jacek.lawrynowicz@linux.intel.com
2024-10-30accel/ivpu: Use xa_alloc_cyclic() instead of custom functionKarol Wachowski
Remove custom ivpu_id_alloc() wrapper used for ID allocations and replace it with standard xa_alloc_cyclic() API. The idea behind ivpu_id_alloc() was to have monotonic IDs, so the driver is easier to debug because same IDs are not reused all over. The same can be achieved just by using appropriate Linux API. Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20241017145817.121590-7-jacek.lawrynowicz@linux.intel.com
2024-10-30accel/ivpu: Remove copy engine supportAndrzej Kacprowski
Copy engine was deprecated by the FW and is no longer supported. Compute engine includes all copy engine functionality and should be used instead. This change does not affect user space as the copy engine was never used outside of a couple of tests. Signed-off-by: Andrzej Kacprowski <Andrzej.Kacprowski@intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20241017145817.121590-4-jacek.lawrynowicz@linux.intel.com
2024-10-11accel/ivpu: Add tracing for IPC/PM/JOBJakub Pawlak
Add multiple trace points in couple of key places to enable tracing with ftrace. Signed-off-by: Jakub Pawlak <jakub.pawlak@intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240930195322.461209-31-jacek.lawrynowicz@linux.intel.com Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com>
2024-10-11accel/ivpu: Use whole user and shave ranges for preemption buffersKarol Wachowski
Do not restrict range for preemption buffers allocation just to the end of user and shave ranges, use them whole instead to avoid situation where end of range might be already allocated causing preemption buffers allocation to fail. Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240930195322.461209-27-jacek.lawrynowicz@linux.intel.com Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com>
2024-10-11accel/ivpu: Do not fail on cmdq if failed to allocate preemption buffersKarol Wachowski
Allow to proceed with job command queue creation even if preemption buffers failed to be allocated, print warning that preemption on such command queue will be disabled. Signed-off-by: Karol Wachowski <karol.wachowski@intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240930195322.461209-26-jacek.lawrynowicz@linux.intel.com Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com>
2024-10-11accel/ivpu: Add test_mode bit to force turboAndrzej Kacprowski
Add new test_mode BIT(9) that forces firmware to enable turbo burst mode. Signed-off-by: Andrzej Kacprowski <Andrzej.Kacprowski@intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240930195322.461209-20-jacek.lawrynowicz@linux.intel.com Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com>
2024-10-11accel/ivpu: Make DB_ID and JOB_ID allocations incrementalTomasz Rusinowicz
Save last used ID and use it to limit the possible values for the ID. This should decrease the rate at which the IDs are reused, which will make debugging easier. Signed-off-by: Tomasz Rusinowicz <tomasz.rusinowicz@intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240930195322.461209-19-jacek.lawrynowicz@linux.intel.com Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com>
2024-10-11accel/ivpu: Add auto selection logic for job schedulerJacek Lawrynowicz
Add ivpu_fw_sched_mode_select() function that can select scheduling mode based on HW and FW versions. This prepares for a switch to HWS on selected platforms. Reviewed-by: Karol Wachowski <karol.wachowski@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240930195322.461209-17-jacek.lawrynowicz@linux.intel.com Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com>
2024-10-11accel/ivpu: Update VPU FW API headersAndrzej Kacprowski
This commit bumps: - Boot API from 3.24.0 to 3.26.3 - JSM API from 3.16.0 to 3.25.0 Signed-off-by: Andrzej Kacprowski <Andrzej.Kacprowski@intel.com> Co-developed-by: Tomasz Rusinowicz <tomasz.rusinowicz@intel.com> Signed-off-by: Tomasz Rusinowicz <tomasz.rusinowicz@intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240930195322.461209-2-jacek.lawrynowicz@linux.intel.com Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com>
2024-06-14accel/ivpu: Abort jobs of faulty contextMaciej Falkowski
Abort all jobs that belong to contexts generating MMU faults in order to avoid flooding host with MMU IRQs. Jobs are cancelled with: - SSID_RELEASE command when OS scheduling is enabled - DESTROY_CMDQ command when HW scheduling is enabled Signed-off-by: Maciej Falkowski <maciej.falkowski@intel.com> Co-developed-by: Wachowski, Karol <karol.wachowski@intel.com> Signed-off-by: Wachowski, Karol <karol.wachowski@intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240611120433.1012423-3-jacek.lawrynowicz@linux.intel.com
2024-05-17accel/ivpu: Split IP and buttress codeWachowski, Karol
The NPU device consists of two parts: NPU buttress and NPU IP. Buttress is a platform specific part that integrates the NPU IP with the CPU. NPU IP is the platform agnostic part that does the inference. This separation enables support for multiple platforms using a single NPU IP, so for example NPU IP 37XX could be integrated into MTL and LNL platforms. Signed-off-by: Wachowski, Karol <karol.wachowski@intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240515113006.457472-3-jacek.lawrynowicz@linux.intel.com
2024-05-15accel/ivpu: Share NPU busy time in sysfsTomasz Rusinowicz
The driver tracks the time spent by NPU executing jobs and shares it through sysfs `npu_busy_time_us` file. It can be then used by user space applications to monitor device utilization. NPU is considered 'busy' starting with a first job submitted to firmware and ending when there is no more jobs pending/executing. Signed-off-by: Tomasz Rusinowicz <tomasz.rusinowicz@intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240513120431.3187212-13-jacek.lawrynowicz@linux.intel.com
2024-05-15accel/ivpu: Implement support for hardware schedulerWachowski, Karol
Add support for HWS (hardware scheduler). It is disabled by default. The sched_mode module param can be used to enable it. Each context has multiple command queues with different priorities and HWS enables priority based execution on the HW/FW side. The driver in HWS mode has to send a couple additional messages to initialize HWS and describe command queue priorities. Signed-off-by: Wachowski, Karol <karol.wachowski@intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240513120431.3187212-7-jacek.lawrynowicz@linux.intel.com
2024-05-15accel/ivpu: Implement support for preemption buffersWachowski, Karol
Allocate per-context preemption buffers that are required by HWS. There are two preemption buffers: * primary - allocated in user memory range (PIOVA accessible) * secondary - allocated in shave memory range Signed-off-by: Wachowski, Karol <karol.wachowski@intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240513120431.3187212-5-jacek.lawrynowicz@linux.intel.com
2024-05-15accel/ivpu: Create priority based command queuesWachowski, Karol
Create multiple command queues per engine with different priorities. The cmdqs are created on-demand and they support 4 priority levels. These priorities will later be used by the HWS (hardware scheduler). Signed-off-by: Wachowski, Karol <karol.wachowski@intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240513120431.3187212-4-jacek.lawrynowicz@linux.intel.com
2024-02-26Merge v6.8-rc6 into drm-nextDaniel Vetter
Thomas Zimmermann asked to backmerge -rc6 for drm-misc branches, there's a few same-area-changed conflicts (xe and amdgpu mostly) that are getting a bit too annoying. Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
2024-02-19accel/ivpu: Refactor BO creation functionsWachowski, Karol
Rename BO allocate/create functions, so the code is more consistent. There are now two matching buffer creation functions: - ivpu_bo_create_ioctl() - create a BO from user space - ivpu_bo_create() - create a BO from kernel space ivpu_bo_alloc() is now only used to allocate struct ivpu_bo which better matches its name. Signed-off-by: Wachowski, Karol <karol.wachowski@intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240214081305.290108-8-jacek.lawrynowicz@linux.intel.com
2024-02-19accel/ivpu: Use lazy allocation for doorbell IDsWachowski, Karol
Reserve/allocate and free doorbells for command queues when needed using xarray. This allows to avoid reserving a doorbell for a contexts that never issues a job. Signed-off-by: Wachowski, Karol <karol.wachowski@intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240214081305.290108-6-jacek.lawrynowicz@linux.intel.com
2024-02-06accel/ivpu: Add job status for jobs aborted by the driverGrzegorz Trzebiatowski
Add DRM_IVPU_JOB_STATUS_ABORTED to indicate that the job was aborted by the driver due to e.g. TDR or user context MMU faults. This will help UMD and tests distinguish if job was aborted by the FW or the driver. Signed-off-by: Grzegorz Trzebiatowski <grzegorz.trzebiatowski@intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240126122804.2169129-8-jacek.lawrynowicz@linux.intel.com
2024-01-25accel/ivpu: Improve recovery and reset supportJacek Lawrynowicz
- Synchronize job submission with reset/recovery using reset_lock - Always print recovery reason and call diagnose_failure() - Don't allow for autosupend during recovery - Prevent immediate autosuspend after reset/recovery - Prevent force_recovery for issuing TDR when device is suspended - Reset VPU instead triggering recovery after changing debugfs params Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Wachowski, Karol <karol.wachowski@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240122120945.1150728-4-jacek.lawrynowicz@linux.intel.com
2024-01-25accel/ivpu: Improve stability of ivpu_submit_ioctl()Jacek Lawrynowicz
- Wake up the device as late as possible - Remove job reference counting in order to simplify the code - Don't put jobs that are not fully submitted on submitted_jobs_xa in order to avoid potential races with reset/recovery Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Wachowski, Karol <karol.wachowski@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240122120945.1150728-3-jacek.lawrynowicz@linux.intel.com
2024-01-25accel/ivpu: Fix dev open/close races with unbindJacek Lawrynowicz
- Add context_list_lock to synchronize user context addition/removal - Use drm_dev_enter() to prevent unbinding the device during ivpu_open() and vpu address allocation Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Wachowski, Karol <karol.wachowski@intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240122120945.1150728-2-jacek.lawrynowicz@linux.intel.com
2024-01-22accel/ivpu: Deprecate DRM_IVPU_PARAM_CONTEXT_PRIORITY paramWachowski, Karol
DRM_IVPU_PARAM_CONTEXT_PRIORITY has been deprecated because it has been replaced with DRM_IVPU_JOB_PRIORITY levels set with submit IOCTL and was unused anyway. Signed-off-by: Wachowski, Karol <karol.wachowski@intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20240115134434.493839-10-jacek.lawrynowicz@linux.intel.com
2023-11-16accel/ivpu: Use threaded IRQ to handle JOB done messagesJacek Lawrynowicz
Remove job_done thread and replace it with generic callback based mechanism. Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231113170252.758137-6-jacek.lawrynowicz@linux.intel.com
2023-11-16accel/ivpu: Use dedicated work for job timeout detectionStanislaw Gruszka
Change to use work for timeout detection. Needed for thread_irq conversion. Signed-off-by: Stanislaw Gruszka <stanislaw.gruszka@linux.intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231113170252.758137-5-jacek.lawrynowicz@linux.intel.com
2023-11-16accel/ivpu: Do not use cons->aborted for job_done_threadStanislaw Gruszka
This allow to simplify ivpu_ipc_receive() as now we do not have to process all messages in aborted state - they will be freed in ivpu_ipc_consumer_del(). Signed-off-by: Stanislaw Gruszka <stanislaw.gruszka@linux.intel.com> Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231113170252.758137-4-jacek.lawrynowicz@linux.intel.com
2023-11-08accel/ivpu: Use GEM shmem helper for all buffersJacek Lawrynowicz
Use struct drm_gem_shmem_object as a base for struct ivpu_bo. This cuts by 50% the buffer management code. Signed-off-by: Jacek Lawrynowicz <jacek.lawrynowicz@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231031073156.1301669-5-stanislaw.gruszka@linux.intel.com
2023-10-31accel/ivpu: Abort pending rx ipc on resetStanislaw Gruszka
Waking up process, which wait for particular condition, will go to sleep again on wake_up() if the condition is not met. Add abort flag to wake up IPC receivers, which will finish with -ECANCELED error. This is only needed for reset, run time power management prevent to suspend VPU when there is pending IPC processing or pending job. Reviewed-by: Karol Wachowski <karol.wachowski@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Signed-off-by: Stanislaw Gruszka <stanislaw.gruszka@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231028155936.1183342-5-stanislaw.gruszka@linux.intel.com
2023-10-31accel/ivpu: Stop job_done_thread on suspendStanislaw Gruszka
Stop job_done thread when going to suspend. Use kthread_park() instead of kthread_stop() to avoid memory allocation and potential failure on resume. Use separate function as thread wake up condition. Use spin lock to assure rx_msg_list is properly protected against concurrent access. This avoid race condition when the rx_msg_list list is modified and read in ivpu_ipc_recive() at the same time. Reviewed-by: Karol Wachowski <karol.wachowski@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Signed-off-by: Stanislaw Gruszka <stanislaw.gruszka@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231028155936.1183342-4-stanislaw.gruszka@linux.intel.com
2023-10-30accel/ivpu: Change test_mode module param to bitmaskKarol Wachowski
Change meaning of test_mode module parameter from integer value to bitmask allowing setting different test features with corresponding bits. Signed-off-by: Karol Wachowski <karol.wachowski@linux.intel.com> Reviewed-by: Stanislaw Gruszka <stanislaw.gruszka@linux.intel.com> Reviewed-by: Jeffrey Hugo <quic_jhugo@quicinc.com> Signed-off-by: Stanislaw Gruszka <stanislaw.gruszka@linux.intel.com> Link: https://patchwork.freedesktop.org/patch/msgid/20231028133415.1169975-7-stanislaw.gruszka@linux.intel.com
generated by cgit v1.2.3 (git 2.39.1) at 2025-08-01 19:17:42 +0000