Continuous running Azure WebJobs, how to restart them quickly

We are running some web jobs in our current project which contain a continuous running singleton process.

We try to program the application a bit defensive, we don’t want to let it crash on warnings, etc. But once in a while, a real exception can occur due to external events or just pure evil. In that case, we stop the current instance so Azure will start a new instance.

Because Azure WebJobs behave just like regular console applications (see my previous blog) we can exit the instance quite elegant.

Here’s how Azure WebJobs are deployed. Just zip the release version of your console application and provide it in the Azure Portal. I set it to running continuous and setting the scale to single instance:

wj01

Note: single instance means that when the container web app scales up elastically, there is still one WebJob instance doing al the work.

I wrote a little example WebJob which simulates an exit situation. This can happen when the application crashes, when it’s just ready doing a job or when an exception is handled gracefully.

 
private static void Main(string[] args) 
{
  Console.WriteLine($"Application start as {DateTime.Now}"); 
  var i = 0; while (true) 
  { 
    Console.WriteLine("sleep");
    Thread.Sleep(1000); 
    i++; 

    if (i > 4) 
    { 
      Environment.Exit(42); 
    } 
  } 
}

This application will run for a while (actually, just five seconds) and then it will just shutdown. An exit code other than zero is returned to indicate the exit has occurred.

Note: This exit could have been a Return, also; because we are running code in the main method. But this Environment.Exit(42); works in events etc too.

But because WebJobs are resilient, just like windows services. After an (accidental) shutdown, these jobs will be restarted automatically.

But we have to wait for 60 seconds between stop and start, the restart time. Luckily, this waiting period can be manipulated. Look at this app setting:

<appSettings>    
  <!--WEBJOBS_RESTART_TIME Ignored. Override in portal!-->    
  <add key="WEBJOBS_RESTART_TIME" value="10" />  
</appSettings>

This should do the job. But… nothing happens. After a restart, the waiting period is still 60 seconds:

wj02

There is another, working solution, this app setting must be manipulated in the Azure portal:

wj03

Look at this output (after restarting the WebJob, the waiting period is altered):

wj04

Because the app.config value will be overruled by the portal value (this is great for administrators, btw.), let’s check if the application knows about the change too:

Console.WriteLine($"Application start as {DateTime.Now} - Restart time: {ConfigurationManager.AppSettings["WEBJOBS_RESTART_TIME"]}");

So deploy the WebJob again with this line:

wj05

Yes, this line of code reads the app setting. And here we see proof that the value given by administrator is shown. The value in the app.config is overruled.

With this little trick, the downtime of your restarting job is minimalized. In a production environment, the downtime should be zero due to the usage of elastic cloud (multiple instances of the same job) or the usage of temporary storage (using a queue or message bus).

Advertenties

One thought on “Continuous running Azure WebJobs, how to restart them quickly

Reacties zijn gesloten.