Sunday, February 22, 2015

Insights into the success of Storm

Just read through Nathan Marz's post about the history of Storm. This is really a nice recap of how to successfully start, grow and maintain a great open source project. Technical excellence is important, but the marketing, growing the community and adoption is even more critical.

Here is his post "History of Apache Storm and Lessons Learned".

I am really looking forward to his new book on the lambda architecture: Big Data, principles and best practices of scalable realtime data systems.

Tuesday, December 30, 2014

Swift coding style guide

Found two coding styles for Swift:
Interesting notes:

1. In github style, it suggests that let should be used over var binding whenever possible. This is to explicitly show the intent that a value is supposed to or not supposed to change.

2. In raywenderlich style, when declaring protocol conformance, use separate extension instead of declaring all together. Also, do not forget to add // MARK comments.

class MyViewcontroller: UIViewController {
  // class stuff here

// MARK: - UITableViewDataSource
extension MyViewcontroller: UITableViewDataSource {
  // table view data source methods

// MARK: - UIScrollViewDelegate
extension MyViewcontroller: UIScrollViewDelegate {
  // scroll view delegate methods
Not Preferred:
class MyViewcontroller: UIViewController, UITableViewDataSource, UIScrollViewDelegate {
  // all methods

3. In raywenderlich style, when unwrapping the optional, shadow the original name instead of using a new name.

var subview: UIView?

// later on...
if let subview = subview {
  // do something with unwrapped subview
Not Preferred:
var optionalSubview: UIView?

if let unwrappedSubview = optionalSubview {
  // do something with unwrappedSubview

4. In github style, prefer implicit getters on read-only properties.

var myGreatProperty: Int {
    return 4

subscript(index: Int) -> T {
    return objects[index]
Not Preferred:
var myGreatProperty: Int {
    get {
        return 4

subscript(index: Int) -> T {
    get {
        return objects[index]

5. In github style guide, it is mentioned to prefer structs over classes. Since I am still quite new to Swift, I will need to learn a bit more to really understand it.

Saturday, November 29, 2014

book notes for Building Applications with iBeacon

I am always very interested in the location-based mobile applications, in particular indoor location apps. I've even explored a little bit about creating a mobile app that helps locating and assembling colleagues in a crowded cafeteria during lunch. At that time, what I learned is that indoor tracking is still an area being actively researched and explored, many solutions use wifi router capabilities specific to vendors such as Cisco.

Lately, I have quickly went through Matt Gast's book on iBeacon: Building Applications with iBeacon. This is a very concise book on the topic, only 80 pages. It gives a quick overview about iBeacon technology mainly focused on iOS.

Here are the book notes I took:


One of the earliest developers of beacon technology. Estimote beacons have fixed configuration parameters and, in particular, administrators cannot set the UUID.

RadBeacon is a $29 USB dongle that performs the transmission functions of an iBeacon. All you have to supply is USB power. Configuration of the beacon’s numbers is done through an app.

Kontakt sells an ARM-based iBeacon as well as tools for manag ing iBeacons and analyzing user interactions with them.

Also compatible with android

Gelo’s Beacons are waterproof and designed for both indoor and outdoor use, and the batteries can be replaced manually using simple tools.

  • Mac (Yosemite stops supporting Mac as iBeacon)
  • iOS devices
  • Raspberry Pi
  • Arduino: BLEduino, BLE Mini from RedBearLab
  • Nordic Semi nRF1822

iOS Apps

- A personal notification beacon for the iPhone. It comes with a beacon device and Geohopper for iOS. It sends notification based on your geo-location. It also integrates with web services for automated workflow.

(Developer Tools)

There are lots of developer util tooks that turns iOS devices into iBeacons. The list can go on and on, here are some apps I found in the app store:
  • Beacon Broadcaster
  • Locate Beacon (from Radius Networks)
  • Beacon Manager
  • My Beacon
  • Beacon Toolkit
  • Beacon Harvester: help you find and save iBeacons around you
  • Beacon Bits

One interesting use case for iBeacon is the treasure hunt games. There are several demoes during the mobile developer conferences, very interesting to connect mobile and real world in a playful way. For example, Beacon Scavenger Hunt (from Radius Networks) is a treasure hunt game to collect scavenger hunt’s badges.

Mac Apps

It was very bad that Yosemite does not support turning a Mac into an iBeacon. Maybe this is one of the bugs with Yosemite?

Mavericks as an iBeacon:

a nice thread on StackOverflow about turning your mac into iBeacon:

Use Cases

10 things to do with iBeacon:

Other Useful Resources

Apple's Official Doc on iBeacon:
Radius Networks Developer Guides: a website dedicated to topics on beacons with Bluetooth Low Energy tech (BLE)

Friday, October 31, 2014


I was working on a Storm project where the work nodes run in a tightly controlled grid environment. So, directly RDBMS access through traditional mechanism is prohibited since only certain ports are open for inbound and outbound traffic. So, I ended up writing a quick and dirty HTTP proxy that exposes certain MySQL database operations using REST API calls.

Then I just found this is actually quite a common use case and several people have written generic HTTP servers to expose REST APIs for database access.

One such project is jdbc-http-server, really handy. I will see if I can replace my own version with that.

Thursday, August 28, 2014

Friday, March 14, 2014

Twisting Maven pom.xml for your legacy code

Recently, I have been working on a legacy project which was not using the standard Maven pom conventions, source code and test code are located at separate paths, the directory structure is also not follow the standard.

So, now we are starting to add more unit tests and integrate that with CI pipeline. The code base is ~200MB including everything. Instead of restructuring the whole project layout, which could block the active development and introduce unpredictable bugs, we decided to twist the pom as best as we can. Here are some nice tips I learned (I am a Maven newbie).

-1. Refactor Your Code

Testing should not be an after-thought. It should be considered along the initial code design and implementation. Apply design patterns, use modular designs and other techniques so that writing tests  become possible in the first place.

0. Writing Unit Tests

There are many unit testing frameworks. Two of the popular ones we use in the company are JUnit and TestNG. Here is a nice StackOverflow comparison for them. Note that if you are using Eclipse, you will need to install TestNG plugin, while JUnit support is built-in. Other IDEs like IntelliJ and Netbeans support both too.

There are several mocking frameworks as well: Mockito, Powermock, EasyMock, jMock, just to name a few. We are using Mockito and Powermock. Here is a nice quick guide how to use them. If you are wondering why we need both, it's because Powermock addresses several features missing from Mockito, such as mocking static methods, etc.

Vogella has several short but useful guides on unit testing, highly recommend:

1. Maven Surefire Plugin with both JUnit and TestNG tests

This is the plugin that runs the unit tests and publish test results. The plugin is documented well on its website, so I am not going to repeat anything here. Quick summary:

  • It can easily include, exclude tests, skip tests (think twice before you do), etc.
  • It support JUnit, TestNG, plain POJO tests. The report format is compatible with JUnit output, so it can be easily integrated with CI tools.
  • It also supports parallel test runs.
  • etc.
Here is a list of all the configuration options for Surefire plugin, very useful.

Normally, you would pick a testing framework and stick to it. But in our case, we have both tests written in JUnit and TestNG under the same test directory. So, how to support that?

Luckily, Marcin has found a solution already to have JUnit and TestNG tests live happily together, see his post here for details. He also has a sample pom that you can use as boilerplate. Basically, you declare dependencies inside Surefire plugin:




2. Maven compiler plugin

Maven compiler plugin is used to compile your source code. Here are the lists of configuration options for its compiler:compile and compiler:testCompile goals:

But our problem is that it turns out the compiler plugin assumes the source code and test should live under the same root source code directory. In our case, this is not the case. I tried various options such as "testSource", "testIncludes" to specify the test path, but without luck.

Finally, I found a plugin build-helper-maven, which allows customized source and test directories. And it worked like a charm (borrowing from its website):

                <source>some directory</source>

3. Maven Clover Plugin for Code Coverage

To generate code coverage, you can use the Maven Clover plugin, note that Clover is free for non-commierical use, for commercial use, you will need to obtain a license and configure the plugin to point to the license file.

4. Maven FindBugs Plugin

FindBugs is a code analysis tool to find potential bugs in your code. It has nice IDE integration and it also integrates well with Maven. You can set it up as a step in CI so that if the bugs will fail the build. I actually scanned our code base and found several severe bugs (one is a switch statement without break, similar to Apple's recent SSL bug).

Please see the plugin website to set it up, quite straightforward.

5. "One Last Thing"

Another useful tip I found out is that when some of the plugin runs, e.g. test, code coverage, findbugs, etc. they require a fair amount of memory. Take the Surefire for example, depending on your configuration, it will fork separate JVM and threads to run the tests. I had JVM exited abruptly in several cases due to this reason.

Refer to the above configuration options to add customized JVM options. For example, for Maven compiler plugin:

<argLine>-Xmx2048m -XX:MaxPermSize=1024m</argLine>

For Maven compiler plugin:


Friday, December 27, 2013

hadoop shell commands auto-completion

For many users including myself, one of the nice features of BASH is its tab completion, which saves us so much typing. So, when I switched to the Hadoop Shell, it feels so inconvenient since there are many commands and options to remember.

Then I searched around and found one hadoop completion script from Facebook hadoop-20 github repo, but the script does not work for my hadoop installed using Homebrew.

So, I modified it to make it work. You can try it out from my hadoop-completion repo on github. All the installation instructions are there:

BTW, Bash-Completion includes a collection of similar auto-completion scripts, highly recommend to use it. You will find your life with git cli, ssh, etc. much easier.


- Programmable Bash Completion Buildins (for compgen and complete commands)

- Write your own Bash Completion Function (how to write a customized Bash completion script)

- Get Bash Completion for Mac OS X (a set of built-in scripts for commonly used tools, svn, make, gzip, ssh, git, etc., note that you will need to install git CLI using Homebrew to install the git completion scripts)