Saturday, August 4, 2018

Stack Trace got eaten up in the log

During the production troubleshooting today, we found out in the log that NPE was logged occasionally, but without detailed stack trace. We've made sure that we always log the stack trace in our code. So, what was going on?

After searching back in our log history (we have a log aggregation service that keeps log history), we found there were NPEs with the full stack trace, then after some time, the stack trace was gone.

In this end, this turns out to be caused by a Hotspot JIT compiler optimization, when a particular exception keeps being thrown up to a threshold, the same exception stack trace will not be filled anymore due to performance impact of throwing exceptions.

The solution is to turn off this optimization by providing this JVM option:



1. Stackoverflow post on "NullPointerException in Java with no StackTrace"
2. A more detailed analysis with a demo app

Friday, August 3, 2018

scheduled task silently died with ScheduledThreadPoolExecutor

We had a very annoying bug lately where a scheduled job silently stopped executing without our knowledge. After debugging and carefully reading Javadoc again, it turns out that the task throws an Exception thus suppressing the subsequent executions of the task.

Javadoc for ScheduledExecutorService.scheduleWithFixedDelay (scheduleAtFixedRate has the same issue):

ScheduledFuture<?> scheduleWithFixedDelay(Runnable command,
                                          long initialDelay,
                                          long delay,
                                          TimeUnit unit)
Creates and executes a periodic action that becomes enabled first after the given initial delay, and subsequently with the given delay between the termination of one execution and the commencement of the next. If any execution of the task encounters an exception, subsequent executions are suppressed. Otherwise, the task will only terminate via cancellation or termination of the executor.

So, the quick fix is to add a try-catch block around the task execution logic to catch Exception. I have created a simple demo app to show this issue and how the fix works with some analysis why it happens:

The lesson learned is that we need to read doc more carefully and read source code to understand the library behavior closely.

Saturday, February 3, 2018

Interview Preparation for Tech Companies

For a candidate, to find a job in tech industry is not easy. From the hiring company's point of view, it is even more challenging to find qualifying candidates. In a very short process of phone interview and onsite interviews, it is very difficult to properly judge a candidate. Companies today mostly rely on short algorithmic or system design questions, here are some resources that I found quite useful and hopefully will be helpful to you as well.


Elements of Programming Interviews: Java version and Python version

Free Online Resources

5-star Comprehensive Tech Interview Handbook

Assorted Notes from

System Design Gitbook by soulmachine (Online, PDF and epub)

Leetcode Algorithmic Questions and Solutions Gitbook by soulmachine (Online, PDF and epub)

Top 10 Algorithms for Coding Interview from programmingcreek, also Leetcode grouped by type

Cracking the coding interview questions and answers by Hawstein (in C++, Chinese)

Top 10 algorithms in Interview Questions from GeeksforGeeks (a very good resource, also groups questions by type, e.g. dynamic programming, etc.)


Massive Technical Interviews Tips (very good coverage of system design problems)

Learn for Master
Leetcode questions organized by companies

There are also lots of video tutorials that gives more in-depth explanation how to solve problems. One good example is Huahua's Tech Road, it has the latest collection of Leetcode problems as well. If you search on Youtube, there are tons of similar resources as well.

Online Judge and Test Sites

- Leetcode
- Lintcode
- Interview Cake
- Hacker Rank
- TopCoder
- Kaggle (for data science and machine learning)

Tuesday, September 26, 2017

Probabilistic Data Structures and Stream Data Processing

The scale and the new way of stream processing has given rise to many interesting data structures and algorithms.

Here are some good resources that cover the topics like:

"Some Important Streaming Algorithms You Should Know About", covers several essential data structures and algorithms, from Ted Dunning

"Probabilistic Data Structures for Web Analytics and Data Mining", a really nice overview from Highly Scalable Blog:

A comprehensive list of the papers, presentations and talks by debasishg

Stanford CS369G: Algorithmic Techniques for Big Data

Stanford CS168: The Modern Algorithmic Toolbox

MIRI Seminar on Data Streams (Spring 2015 Edition)

Counting Items, Cardinality

- HashSet
- Linear Probabilistic Counter
- LogLog and HyperLogLog

Frequency Estimate, Top K

- Count Min Sketch
- Count Mean Min Sketch

Membership Query

- Bloom Filter
- Cuckoo Filter

Percentile and Quantile

- Q-digest
- t-digest

Thursday, December 8, 2016

Linksfest to Get Started with Apache Flink

Flink, another great data processing platform, has been a rising star this year. It is a high performance stream and batch data processing data platform, with fault-tolerant, scalable, distributed data stream computation at its core.

Here are several links and resources to get you started.

Company and Community
dataArtisans, company behind Flink
Google Trends comparing Flink, Spark and Storm (Spark is still way more popular)


Introduction to Apache Flink, book from Flink core developers, highly recommend to start your Flink journey with. It is a Free download from MapR.

Flink in Action (MEAP, available in Spring 2017), the first chapter (PDF) is Free and gives a good overview.

Quick start guide

Talks and Videos

Alibaba slides on Blink, their fork of Flink, Alibaba is one of the biggest online e-commerce site in China.

Performance and Benchmark


2016 Holiday Guide for Robot Toys

Holiday is just around the corner and it's time to order gifts from "Santa". This year, I decided to give my son something different, something other than candies, chocolates, pokemon cards, lego, etc.

Since he has been exposed to basic programming concepts through and Scratch. So..., how about a programmable Robot for this Christmas? Sounds good.

After doing some research and comparison, we ordered Ozobot Evo. To get started out of box, it supports a color-code based language for various actions, e.g. follow the black line and move forward, stop at the red color, rotating at the blue color and play color light and music, etc. You can also customize the action with a mobile app on the phone or tablet with an environment similar to Scratch.

When he grows up a bit more, we might introduce marty the robot to him. It looks and works more like the robots we know about. It also teaches some real mechanical dynamics for the kids.

Here are some notes I took during the research, hope it will be useful to you.

Ozobot Bit
only supports the colored line language
both 1.0 and 2.0 available on amazon (around $60)

Ozobot Evo
More advanced than Bit, supports the colored line language and a Scratch like visual programming language, can control the robot using a mobile app, supports social interactions with friends’ robots.
available on amazon (around $100):

available on amazon: $169.99

A playful companion, a robot that has personality, very cute!
available on amazon (around $300):

marty the robot
This is for more grown-up kids, more makebot-like robot, fully programable, start with Scratch, then move to Python. The way they dance together looks so funny ;-)
not available yet, currently on crowdsourcing

kids education robot and companion, not that programmable.
Founder from Shenzhen, China (around $230)

aido family robot
size of a toddler, family robot, assistant, voice control, helper, etc. Reminds me of Baymax in Big Hero 6 ;-)
available for pre-order (around $600): will ship in early 2017

Saturday, November 26, 2016

Java Concurrency Counters Benchmark

Java concurrency utilities have kept evolving and provides many different ways to achieve similar tasks. Recently, we had a task to implement a concurrent counter. This triggered my interest in comparing different ways and their performance under various read and write workload.

The end result is a simple concurrent counter implemented in various ways:
The benchmark is implemented using JMH, the standard way for reliable Java performance microbenchmark. You can find several really nice tutorials on JMH in the References section.

In my benchmark, there are write and read operations on the counter. The write takes 10ms and read takes 2ms. I set the number of read and write threads to simulate different mix of the workload scenarios using JMH group.

Both the source code and benchmark raw data, Excel sheets and visualizations can be found in the git repo: java-concurrency-counters-benchmark.

Here is a quick summary based on my experiment (I only set 2 rounds of warmups and 2 rounds of benchmark due to limited time):
  • AtomicLong and LongAdder has similar throughput. In read-heavy workloads, AtomicLong has better read and write throughput than LongAdder. In write-heavy workloads, LongAdder has slightly better write throughput.
  • Fair lock has lower throughput than regular lock in general, but not always.
  • Consider using ReentrantLock or ReentrantReadWriteLock if you need high read throughput and the concurrency level is high.
  • StampedLock provides very good write throughput in all the read-write mixes, if write throughput is important to you, you can try it. At the same time, if you need comparatively good read throughput, try optimistic read StampedLock. It has really good read throughput when concurrency level is high compared with regular StampedLock.

Special thanks and references: