How to cancel script, update or plugin rollback when it’s stuck and is breaking ServiceNow

A useful roll back and delete recovery feature was released to ServiceNow. With each script execution you can track all the database changes with possibility to recover everything with Rollback Executor.

It sounds very useful and seems like it will fix all your oopsies. However, it has flaws, does not always recover everything and might take days or weeks to finish.

One resultant flaw is that RollbackExecutor is being counted an as ‘upgrade’ activity. While it’s running you will not be able to use preview and commit update set functionality – “Update set preview and commit are unavailable because the system is currently upgrading. Click here for the Upgrade Monitor”.

As well as scheduled jobs with upgrade_safe=false parameter will not be executed while rollback is running.

Usual way to stop any transaction is by following:

  1. Go to “Progress workers”, locate the running worker and change its state to cancelled.
  2. Go to “Active Transactions (All Nodes)” or “Active Transactions”, locate the transaction you want to kill, right click it and select kill.

This is what you must do, however, both steps are not enough to kill RollbackExecutor process. You can verify it’s still running by going to xmlstats.do and looking for background_progress_workers:

<background_progress_workers count="1" size="16">
  <running_background_progress_workers count="1">
    <background_progress_worker created_on="2019-08-14 20:38:40" executed_time="19:21:08.698" processor="glide.scheduler.worker.6" progress_worker_record="ebbefa45db97bb00c3e6294d0b96196d" queued_time="0:00:00.154" schedule_job="e7befa45db97bb00c3e6494d0b961970" total_duration="19:21:08.852">Executing Rollback</background_progress_worker>
  </running_background_progress_workers>
</background_progress_workers>

Luckily, we can figure out next steps based on the node logs:

The upgrade system is busy because the GlideSystem is Paused

GlideSession message was modified by sanitization. [message=Update set preview and commit are unavailable because the system is currently upgrading. <a href="$upgrade_client.do">Click here</a> for the Upgrade Monitor][sanitized=Update set preview and commit are unavailable because the system is currently upgrading. <a href="$upgrade_client.do" rel="nofollow">Click here</a> for the Upgrade Monitor]

Now we simply use undocumented GlideSystem API functions.

First, we can verify if GS is paused with gs.isPaused();

Then we simply resume GlideSystem with:

gs.resume();

Finally, our rollback activity is cancelled and we can use our update sets and scheduled jobs!