Stack Trace got eaten up in the log

During the production troubleshooting today, we found out in the log that NPE was logged occasionally, but without detailed stack trace. We've made sure that we always log the stack trace in our code. So, what was going on?

After searching back in our log history (we have a log aggregation service that keeps log history), we found there were NPEs with the full stack trace, then after some time, the stack trace was gone.

In this end, this turns out to be caused by a Hotspot JIT compiler optimization, when a particular exception keeps being thrown up to a threshold, the same exception stack trace will not be filled anymore due to performance impact of throwing exceptions.

The solution is to turn off this optimization by providing this JVM option:

 -XX:-OmitStackTraceInFastThrow

References:

1. Stackoverflow post on "NullPointerException in Java with no StackTrace"
2. A more detailed analysis with a demo app