summaryrefslogtreecommitdiff
path: root/arch/loongarch/mm/tlb.c
AgeCommit message (Collapse)Author
2023-06-29LoongArch: Introduce hardware page table walkerHuacai Chen
Loongson-3A6000 and newer processors have hardware page table walker (PTW) support. PTW can handle all fastpaths of TLBI/TLBL/TLBS/TLBM exceptions by hardware, software only need to handle slowpaths (page faults). BTW, PTW doesn't append _PAGE_MODIFIED for page table entries, so we change pmd_dirty() and pte_dirty() to also check _PAGE_DIRTY for the "dirty" attribute. Signed-off-by: Liang Gao <gaoliang@loongson.cn> Signed-off-by: Jun Yi <yijun@loongson.cn> Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
2023-01-17LoongArch: Add generic ex-handler unwind in prologue unwinderJinyang He
When exception is triggered, code flow go handle_\exception in some cases. One of stackframe in this case as follows, high -> +-------+ | REGS | <- a pt_regs | | | | <- ex trigger | REGS | <- ex pt_regs <-+ | | | | | | low -> +-------+ ->unwind-+ When unwinder unwinds to handler_\exception it cannot go on prologue analysis. Because it is an asynchronous code flow, we should get the next frame PC from regs->csr_era rather than regs->regs[1]. At init time we copy the handlers to eentry and also copy them to NUMA-affine memory named pcpu_handlers if NUMA is enabled. Thus, unwinder cannot unwind normally. To solve this, we try to give some hints in handler_\exception and fixup unwinders in unwind_next_frame(). Reported-by: Qing Zhang <zhangqing@loongson.cn> Signed-off-by: Jinyang He <hejinyang@loongson.cn> Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
2022-10-12LoongArch: Flush TLB earlier at initializationHuacai Chen
Move local_flush_tlb_all() earlier (just after setup_ptwalker() and before page allocation). This can avoid stale TLB entries misguiding the later page allocation. Without this patch the second kernel of kexec/kdump fails to boot SMP. BTW, move output_pgtable_bits_defines() into tlb_init() since it has nothing to do with tlb handler setup. Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
2022-06-25LoongArch: Fix EENTRY/MERRENTRY setting in setup_tlb_handler()Huacai Chen
setup_tlb_handler() is expected to set per-cpu exception handlers, but it only set the TLBRENTRY successfully because of copy & paste errors, so fix it. Reviewed-by: WANG Xuerui <git@xen0n.name> Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
2022-06-25LoongArch: Fix sleeping in atomic context in setup_tlb_handler()Huacai Chen
Since setup_tlb_handler() is executed in atomic context, we should use GFP_ATOMIC instead of GFP_KERNEL to alloc pages. Otherwise we will get a "sleeping in atomic context" error: [ 0.013118] BUG: sleeping function called from invalid context at mm/page_alloc.c:5158 [ 0.013126] in_atomic(): 1, irqs_disabled(): 1, non_block: 0, pid: 0, name: swapper/1 [ 0.013131] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.19-rc3+ #1008 1a223086d14d07967cc427f15d52139422271360 [ 0.013136] Hardware name: Loongson Loongson-3A5000-7A1000-1w-V0.1-CRB/Loongson-LS3A5000-7A1000-1w-EVB-V1.21, BIOS Loongson-UDK2018-V2.0.04082-beta7 04/27 [ 0.013140] Stack : 90000000015fc990 9000000100493c18 9000000000df3370 9000000100490000 [ 0.013151] 9000000100493b50 0000000000000000 9000000100493b58 9000000001417ef0 [ 0.013160] 900000000199e54e 0000000000000040 9000000100493c18 90000000015f7a98 [ 0.013168] ffffffffffffffff 6de72f8b42179d1e 9000000100403b80 90000000015f7890 [ 0.013176] 0000000000000001 00000000fffff175 9000000000eb9860 9000000001530b4b [ 0.013184] 9000000000e99e60 0000000000000013 0000000006ecc000 0000000000000001 [ 0.013193] 90000000015f7a98 9000000001417ef0 0000000000000004 0000000000000000 [ 0.013201] 0000000000000cc0 0000000000000000 0000000000000001 90000000015fc990 [ 0.013209] 9000000000217e74 9000000001603b6b 9000000000208640 0000000000000000 [ 0.013217] 00000000000000b0 0000000000000004 0000000000000000 0000000000070000 [ 0.013225] ... [ 0.013229] Call Trace: [ 0.013230] [<9000000000208640>] show_stack+0x4c/0x14c [ 0.013240] [<9000000000df3370>] dump_stack_lvl+0x70/0xac [ 0.013246] [<9000000000270c8c>] ___might_sleep+0x104/0x124 [ 0.013253] [<9000000000477e84>] __alloc_pages+0x240/0x464 [ 0.013260] [<9000000000214214>] setup_tlb_handler+0x104/0x1e8 [ 0.013265] [<9000000000214324>] tlb_init+0x2c/0x3c [ 0.013270] [<9000000000208b74>] per_cpu_trap_init+0xec/0x108 [ 0.013275] [<9000000000202850>] cpu_probe+0x400/0x8a4 [ 0.013279] [<900000000020d160>] start_secondary+0x5c/0x3d4 Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
2022-06-03LoongArch: Add Non-Uniform Memory Access (NUMA) supportHuacai Chen
Add Non-Uniform Memory Access (NUMA) support for LoongArch. LoongArch has 48-bit physical address, but the HyperTransport I/O bus only support 40-bit address, so we need a custom phys_to_dma() and dma_to_phys() to extract the 4-bit node id (bit 44~47) from Loongson-3's 48-bit physical address space and embed it into 40-bit. In the 40-bit dma address, node id offset can be read from the LS7A_DMA_CFG register. Reviewed-by: WANG Xuerui <git@xen0n.name> Reviewed-by: Jiaxun Yang <jiaxun.yang@flygoat.com> Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>
2022-06-03LoongArch: Add memory managementHuacai Chen
Add memory management support for LoongArch, including: cache and tlb management, page fault handling and ioremap/mmap support. Reviewed-by: WANG Xuerui <git@xen0n.name> Reviewed-by: Jiaxun Yang <jiaxun.yang@flygoat.com> Signed-off-by: Huacai Chen <chenhuacai@loongson.cn>