scheduled task silently died with ScheduledThreadPoolExecutor

We had a very annoying bug lately where a scheduled job silently stopped executing without our knowledge. After debugging and carefully reading Javadoc again, it turns out that the task throws an Exception thus suppressing the subsequent executions of the task.

Javadoc for ScheduledExecutorService.scheduleWithFixedDelay (scheduleAtFixedRate has the same issue):

ScheduledFuture<?> scheduleWithFixedDelay(Runnable command,
                                          long initialDelay,
                                          long delay,
                                          TimeUnit unit)

Creates and executes a periodic action that becomes enabled first after the given initial delay, and subsequently with the given delay between the termination of one execution and the commencement of the next. If any execution of the task encounters an exception, subsequent executions are suppressed. Otherwise, the task will only terminate via cancellation or termination of the executor.

So, the quick fix is to add a try-catch block around the task execution logic to catch Exception. I have created a simple demo app to show this issue and how the fix works with some analysis why it happens:

https://github.com/guozheng/scheduledjob

The lesson learned is that we need to read doc more carefully and read source code to understand the library behavior closely.