Skip to content

JMA

Version 2.1.0

API version: 10.0.0

Changed

  • Job updates from workers are consumed from the new single-active-consumer queue Job.All.Updates.SingleActive, one message at a time, so the updates of a job are applied in the order the worker sent them however many instances run. With competing consumers on Job.All.Updates, two instances could apply a job’s “finished” before its error (the failed job ended up completed) or before its result. When a listener stops it finishes the message in processing before a standby takes over. Workers are unaffected: routing keys and message contracts are unchanged. Delivery stays at least once: after a redelivery or a failover an output’s metadata comment can appear twice; its name and tags are unaffected. With more than one instance, an output that receives more than one metadata update can be left with the older one after a failover; a step that writes each output once sends one (README, “Job update messages”; issue #182).
  • Operator-facing: at startup JMA unbinds Job.All.Updates and drains what is left in it before consuming the new queue. The drain acknowledges what it applied only once the queue is empty: an interrupted drain fails the startup and leaves everything it had applied in Job.All.Updates, and the restarted pod applies it again, in order. An update published in the instant of the switch reaches both queues; its copy is skipped by message id, and never applied after a newer update of its job (an update applied a second time appends its output metadata comment again, like any redelivery). Roll this release out with a single instance, scale out afterwards. Once Job.All.Updates is empty it can be deleted together with the job-all-updates-dead-letter broker policy. Rolling back to an earlier release needs a controlled hand-back of the queue (README, “Rolling back to a release without the single-active queue”): deleting Job.All.Updates.SingleActive without it loses the updates only it holds.

Fixed

  • A job update that is valid JSON but no usable message (null, or without header or content) is dead-lettered instead of staying unacknowledged. The same holds for any message whose handler fails unexpectedly, in every listener; with the job-update listener taking one message at a time, such a message would otherwise have stopped all job updates.
  • A job update whose command found the job store temporarily unavailable (a database timeout, lost connection or failover) was dead-lettered at once, and the job’s later updates were applied without it: a failed job’s “finished” without its error left the job completed. It is now retried in place for about three and a half minutes, which holds the job updates behind it until the store is back; only a longer outage dead-letters it.
  • Saving a job that has tags could fail as “resource already exists” when another save of the same job had just committed (every save replaces the job’s tag rows, and the stale save collided with the new ones before its version check). The job-update listener took that for an update already applied and dropped it, and API requests failed. It is now retried like any other concurrent modification.
  • With several instances, an instance whose broker connection recovered re-created the command queues of jobs another instance had already finished (the RabbitMQ client’s automatic recovery re-declares every queue a connection declared and did not delete itself). Job command queues are left out of that recovery; they are durable, so those of running jobs survive a reconnect anyway.