You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

David S. Miller f36391d279 sparc64: Fix race in TLB batch processing. ... As reported by Dave Kleikamp, when we emit cross calls to do batched TLB flush processing we have a race because we do not synchronize on the sibling cpus completing the cross call. So meanwhile the TLB batch can be reset (tb->tlb_nr set to zero, etc.) and either flushes are missed or flushes will flush the wrong addresses. Fix this by using generic infrastructure to synchonize on the completion of the cross call. This first required getting the flush_tlb_pending() call out from switch_to() which operates with locks held and interrupts disabled. The problem is that smp_call_function_many() cannot be invoked with IRQs disabled and this is explicitly checked for with WARN_ON_ONCE(). We get the batch processing outside of locked IRQ disabled sections by using some ideas from the powerpc port. Namely, we only batch inside of arch_{enter,leave}_lazy_mmu_mode() calls. If we're not in such a region, we flush TLBs synchronously. 1) Get rid of xcall_flush_tlb_pending and per-cpu type implementations. 2) Do TLB batch cross calls instead via: smp_call_function_many() tlb_pending_func() __flush_tlb_pending() 3) Batch only in lazy mmu sequences: a) Add 'active' member to struct tlb_batch b) Define __HAVE_ARCH_ENTER_LAZY_MMU_MODE c) Set 'active' in arch_enter_lazy_mmu_mode() d) Run batch and clear 'active' in arch_leave_lazy_mmu_mode() e) Check 'active' in tlb_batch_add_one() and do a synchronous flush if it's clear. 4) Add infrastructure for synchronous TLB page flushes. a) Implement __flush_tlb_page and per-cpu variants, patch as needed. b) Likewise for xcall_flush_tlb_page. c) Implement smp_flush_tlb_page() to invoke the cross-call. d) Wire up global_flush_tlb_page() to the right routine based upon CONFIG_SMP 5) It turns out that singleton batches are very common, 2 out of every 3 batch flushes have only a single entry in them. The batch flush waiting is very expensive, both because of the poll on sibling cpu completeion, as well as because passing the tlb batch pointer to the sibling cpus invokes a shared memory dereference. Therefore, in flush_tlb_pending(), if there is only one entry in the batch perform a completely asynchronous global_flush_tlb_page() instead. Reported-by: Dave Kleikamp <dave.kleikamp@oracle.com> Signed-off-by: David S. Miller <davem@davemloft.net> Acked-by: Dave Kleikamp <dave.kleikamp@oracle.com>		12 years ago
..
Kbuild	sparc: use asm-generic version of types.h	12 years ago
agp.h	…
apb.h	…
asm-offsets.h	…
asm.h	…
asmmacro.h	sparc32: add support for run-time patching of leon/sun single instructions	13 years ago
atomic.h	…
atomic_32.h	Disintegrate asm/system.h for Sparc	13 years ago
atomic_64.h	sparc: Support atomic64_dec_if_positive properly.	12 years ago
auxio.h	…
auxio_32.h	Disintegrate asm/system.h for Sparc	13 years ago
auxio_64.h	…
backoff.h	sparc64: Improvde documentation and readability of atomic backoff code.	12 years ago
barrier.h	Disintegrate asm/system.h for Sparc	13 years ago
barrier_32.h	Disintegrate asm/system.h for Sparc	13 years ago
barrier_64.h	Disintegrate asm/system.h for Sparc	13 years ago
bbc.h	…
bitext.h	…
bitops.h	…
bitops_32.h	bitops: remove minix bitops from asm/bitops.h	14 years ago
bitops_64.h	sparc: Use popc when possible for ffs/__ffs/ffz.	14 years ago
btext.h	…
bug.h	Disintegrate asm/system.h for Sparc	13 years ago
bugs.h	…
cache.h	sparc32: Remove completely unused code from asm/cache.h	13 years ago
cacheflush.h	sparc32: use flushi when run-time patching in per_cpu_patch	13 years ago
cacheflush_32.h	sparc32: Move cache and TLB flushes over to method ops.	13 years ago
cacheflush_64.h	sparc32: use flushi when run-time patching in per_cpu_patch	13 years ago
cachetlb_32.h	sparc32: Move cache and TLB flushes over to method ops.	13 years ago
chafsr.h	…
checksum.h	…
checksum_32.h	…
checksum_64.h	…
chmctrl.h	…
clock.h	…
cmpxchg.h	Disintegrate asm/system.h for Sparc	13 years ago
cmpxchg_32.h	sparc32: remove runtime btfix support	13 years ago
cmpxchg_64.h	Disintegrate asm/system.h for Sparc	13 years ago
compat.h	sparc64: Make montmul/montsqr/mpmul usable in 32-bit threads.	12 years ago
compat_signal.h	sparc: switch to generic sigaltstack	12 years ago
contregs.h	sparc32: Remove sun4{,c} control reg definitions from contregs.h.	13 years ago
cpu_type.h	sparc32: Remove sun4 and sun4c from enum sparc_cpu.	13 years ago
cpudata.h	…
cpudata_32.h	sparc32: generic clockevent support	13 years ago
cpudata_64.h	sparc64: Use a seperate counter for timer interrupts and NMI checks, like x86.	15 years ago
current.h	…
dcr.h	…
dcu.h	…
delay.h	…
delay_32.h	…
delay_64.h	…
device.h	of: remove asm/of_device.h	15 years ago
dma-mapping.h	sparc: dma-mapping: support debug_dma_mapping_error	12 years ago
dma.h	sparc32: Convert mmu_* interfaces from btfixup to method ops.	13 years ago
ebus_dma.h	…
ecc.h	…
eeprom.h	…
elf.h	…
elf_32.h	default SET_PERSONALITY() in linux/elf.h	12 years ago
elf_64.h	sparc64: Add detection for features new in SPARC-T4.	13 years ago
estate.h	…
fb.h	sparc: Really fix "console=" for serial consoles.	15 years ago
fbio.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	13 years ago
fhc.h	…
floppy.h	…
floppy_32.h	sparc: Kill mmu_{un,}lockarea().	13 years ago
floppy_64.h	sparc64: Remove pointless assignment in floppy support.	13 years ago
fpumacro.h	…
ftrace.h	…
futex.h	…
futex_32.h	…
futex_64.h	Disintegrate asm/system.h for Sparc	13 years ago
gpio.h	gpiolib/arches: Centralise bolierplate asm/gpio.h	13 years ago
hardirq.h	…
hardirq_32.h	…
hardirq_64.h	…
head.h	…
head_32.h	sparc32: Kill unused defines from asm/head_32.h	13 years ago
head_64.h	…
highmem.h	sparc32: move kmap_init() to highmem.c	13 years ago
hugetlb.h	sparc64: Fix tsb_grow() in atomic context.	12 years ago
hvtramp.h	…
hw_irq.h	…
hypervisor.h	sparc64: Add hypervisor interfaces for SPARC-T4 perf counter access.	13 years ago
ide.h	…
idprom.h	…
intr_queue.h	…
io-unit.h	…
io.h	sparc: add {read,write}*_be routines	14 years ago
io_32.h	Disintegrate asm/system.h for Sparc	13 years ago
io_64.h	Disintegrate asm/system.h for Sparc	13 years ago
ioctls.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	13 years ago
iommu.h	…
iommu_32.h	…
iommu_64.h	…
irq.h	…
irq_32.h	sparc32: genirq support	14 years ago
irq_64.h	nmi watchdog: do not use cpp symbol in Kconfig	13 years ago
irqflags.h	…
irqflags_32.h	Disintegrate asm/system.h for Sparc	13 years ago
irqflags_64.h	sparc/irqs: Do not trace arch_local_{*,irq_*} functions	14 years ago
jump_label.h	static keys: Introduce 'struct static_key', static_key_true()/false() and static_key_slow_[inc|dec]()	13 years ago
kdebug.h	…
kdebug_32.h	…
kdebug_64.h	…
kgdb.h	…
kmap_types.h	…
kprobes.h	…
ldc.h	…
leon.h	sparc32,leon: drop leon_init()	13 years ago
leon_amba.h	sparc32,leon: always include leon_kernel in build	13 years ago
leon_pci.h	PCI: Make the struct pci_dev * argument of pci_fixup_irqs const.	14 years ago
linkage.h	…
lsu.h	…
machines.h	sparc32: Trivial removal of sun4c references in comments.	13 years ago
mbus.h	sparc32: Remove cypress cpu support.	13 years ago
mc146818rtc.h	…
mc146818rtc_32.h	…
mc146818rtc_64.h	…
mdesc.h	sparc64: Probe cpu page size support more portably.	13 years ago
memctrl.h	…
mman.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	13 years ago
mmu.h	…
mmu_32.h	sparc32: add irq + smp declarations to headers	14 years ago
mmu_64.h	sparc64: Support transparent huge pages.	13 years ago
mmu_context.h	…
mmu_context_32.h	sparc32: centralize all mmu context handling in srmmu.c	13 years ago
mmu_context_64.h	sparc64: Support transparent huge pages.	13 years ago
mmzone.h	Fix node_start/end_pfn() definition for mm/page_cgroup.c	14 years ago
msi.h	…
mxcc.h	…
nmi.h	…
ns87303.h	Disintegrate asm/system.h for Sparc	13 years ago
obio.h	sparc32: move smp ipi to method ops	13 years ago
openprom.h	sparc: Pass buffer pointer all the way down to prom_{get,put}char().	14 years ago
oplib.h	…
oplib_32.h	sparc: fix format string argument for prom_printf()	13 years ago
oplib_64.h	sparc: fix format string argument for prom_printf()	13 years ago
page.h	…
page_32.h	sparc32: drop sparc_unmapped_base	13 years ago
page_64.h	sparc64: Fix tsb_grow() in atomic context.	12 years ago
parport.h	SPARC: drivers: remove __dev* attributes.	12 years ago
pbm.h	…
pci.h	pci-dma: add linux/pci-dma.h to linux/pci.h	15 years ago
pci_32.h	PCI: collapse pcibios_resource_to_bus	13 years ago
pci_64.h	PCI: collapse pcibios_resource_to_bus	13 years ago
pcic.h	sparc32: added CONFIG_PCIC_PCI Kconfig setting	14 years ago
pcr.h	sparc64: Add PCR ops for SPARC-T4.	13 years ago
percpu.h	…
percpu_32.h	…
percpu_64.h	…
perf_event.h	perf, arch: Cleanup perf-pmu init vs lockup-detector	14 years ago
pgalloc.h	…
pgalloc_32.h	sparc32: drop quicklist	13 years ago
pgalloc_64.h	sparc64: Support transparent huge pages.	13 years ago
pgtable.h	…
pgtable_32.h	sparc32: drop unused BAD_PAGE stuff	13 years ago
pgtable_64.h	sparc64: Fix race in TLB batch processing.	12 years ago
pgtsrmmu.h	sparc32: srmmu_probe now knows about leon too	13 years ago
pil.h	…
processor.h	Disintegrate asm/system.h for Sparc	13 years ago
processor_32.h	sparc idle: rename pm_idle to sparc_idle	12 years ago
processor_64.h	Merge branch 'arch-microblaze' into no-rebases	12 years ago
prom.h	of/address: sparc: Declare of_iomap as an extern function for sparc again	12 years ago
psr.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	13 years ago
ptrace.h	Merge commit '517ffce4e1a03aea979fe3a18a3dd1761a24fafb' into arch-sparc	12 years ago
ross.h	…
rwsem.h	rwsem: Move duplicate function prototypes to linux/rwsem.h	14 years ago
sbi.h	…
scatterlist.h	remove needless ISA_DMA_THRESHOLD	15 years ago
scratchpad.h	…
seccomp.h	…
sections.h	sparc32: add support for run-time patching of leon/sun single instructions	13 years ago
setup.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	13 years ago
sfafsr.h	…
sfp-machine.h	…
sfp-machine_32.h	…
sfp-machine_64.h	…
shmparam.h	…
shmparam_32.h	sparc32: Remove sun4 and sun4c from enum sparc_cpu.	13 years ago
shmparam_64.h	…
sigcontext.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	13 years ago
siginfo.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	13 years ago
signal.h	consolidate kernel-side struct sigaction declarations	12 years ago
smp.h	…
smp_32.h	sparc:cleanup unused code in smp_32.h	12 years ago
smp_64.h	sparc64: Add global PMU register dumping via sysrq.	12 years ago
sparsemem.h	…
spinlock.h	…
spinlock_32.h	sparc32: unbreak arch_write_unlock()	14 years ago
spinlock_64.h	sparc64: remove unnecessary macros from spinlock_64.h	14 years ago
spinlock_types.h	…
spitfire.h	sparc64: correctly recognize SPARC64-X chips	12 years ago
stacktrace.h	…
starfire.h	…
string.h	…
string_32.h	sparc32: Remove inline strncmp "optimization" for constant counts.	13 years ago
string_64.h	…
sunbpp.h	…
swift.h	…
switch_to.h	Disintegrate asm/system.h for Sparc	13 years ago
switch_to_32.h	Disintegrate asm/system.h for Sparc	13 years ago
switch_to_64.h	sparc64: Fix race in TLB batch processing.	12 years ago
syscall.h	…
syscalls.h	sparc32: switch to generic sys_execve()	12 years ago
termbits.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	13 years ago
termios.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	13 years ago
thread_info.h	…
thread_info_32.h	sanitize tsk_is_polling()	13 years ago
thread_info_64.h	Merge commit '517ffce4e1a03aea979fe3a18a3dd1761a24fafb' into arch-sparc	12 years ago
timer.h	…
timer_32.h	sparc32: remove runtime btfix support	13 years ago
timer_64.h	SPARC: drivers: remove __dev* attributes.	12 years ago
timex.h	…
timex_32.h	sparc32: generic clockevent support	13 years ago
timex_64.h	…
tlb.h	…
tlb_32.h	…
tlb_64.h	sparc: mmu_gather rework	14 years ago
tlbflush.h	…
tlbflush_32.h	sparc32: Move cache and TLB flushes over to method ops.	13 years ago
tlbflush_64.h	sparc64: Fix race in TLB batch processing.	12 years ago
topology.h	…
topology_32.h	…
topology_64.h	sched/numa: Rewrite the CONFIG_NUMA sched domain support	13 years ago
trap_block.h	…
traps.h	UAPI: (Scripted) Disintegrate arch/sparc/include/asm	13 years ago
tsb.h	sparc64: Fix huge PMD to PTE translation for sun4u in TLB miss handler.	12 years ago
tsunami.h	…
ttable.h	sparc64: Make montmul/montsqr/mpmul usable in 32-bit threads.	12 years ago
turbosparc.h	…
uaccess.h	sparc: Fix user_addr_max() definition.	13 years ago
uaccess_32.h	sparc: use the new generic strnlen_user() function	13 years ago
uaccess_64.h	sparc64: clear syscall_noerror on the entry to syscall, not on the exit	13 years ago
unaligned.h	…
unistd.h	burying unused conditionals	12 years ago
upa.h	…
user.h	…
vaddrs.h	sparc32: drop fixmap.h	13 years ago
vga.h	BUG: headers with BUG/BUG_ON etc. need linux/bug.h	13 years ago
viking.h	…
vio.h	powerpc+sparc/vio: Modernize driver registration	13 years ago
visasm.h	…
winmacro.h	sparc32: Implement hard_smp_processor_id() via instruction patching.	13 years ago
xor.h	…
xor_32.h	…
xor_64.h	sparc64: Future proof Niagara cpu detection.	14 years ago