Start here. This is the direct spoken answer to practice first.
Overview
Both execute on the server; the choice is mainly about the caller and contract, not whether security is required.
I use a Server Action for a mutation owned by the Next.js UI, especially a form that benefits from progressive enhancement and can return updated UI and data in one round trip. I use a Route Handler when I need an explicit HTTP endpoint for external clients, webhooks, downloads, custom methods, or a stable API contract. Both are remotely invokable entry points and must validate input and authorize the operation.