Quick intro to scopes:
ClientId has a set of scope-patterns: ["abc*", "ab", "a"]
This clientId satisfies the following set of scopes:
  ["abcd", "abc", "abcfdsfdsfdsfdsfsdf", "ab", "a", ...] 
The clientId does NOT satisfy the following scopes:
  ["b", "abd", "ac", ...]
Hence, the scope-pattern "ab", only grants scope "ab"
but the scope-pattern "abc*", grants any scope starting with "abc"

My setup would be:

provisionerId: null-provisioner
workerType:    signing-worker

task-definition:
{
  ... // all the usual keys
  scopes:       ['signing-worker:sign-with:<name-of-key>'],
  payload: {
    fileToSign: "https://..."
    keyName:    "<name-of-key>"
  }
}

signing-worker (processing a task):
Required Scopes:
To create a task for this worker one needs the following scopes:
A) queue:create-task:null-provisioner/signing-worker
B) signing-worker:sign-with:<name-of-key>

(A) is required by queue to create the task.
(B) is required by queue to have the scope (B) in task.scopes
    and the worker refuses to sign if task.scopes doesn't
    contain the scope (B).

In this world view signing-worker is a trusted entity, that
does not execute any user generated code. It only operates
on data. And it has direct access to an HSM or keys to our
signing infrastructure.