how Dropbox and Evernote file sync works

Spent some time trying to understand better how Dropbox and Evernote sync changes across different devices (mobile, desktop client, Web browser, etc.). Here are some of the interesting papers and articles that I found. Note that they might be outdated a little (you can see the timestamps).

Dropbox:

1. IMC'12 paper "Inside Dropbox: Understanding Personal Cloud Storage Services" by Idilio Drago, etc. (Nov 2012).

The paper studies the Dropbox architecture and traffic by intercepting and analyzing traffic data. It provides many insights about various control and data flow, service components, protocols, etc.

2. "Streaming File Synchronization" from Dropbox tech blog (July 2014).

This blog post gives an overview of file sync and presents a new stream-based sync mechanism that improves latency by upto 2X.

3. "Inside LAN Sync" from Dropbox tech blog (Oct 2015).

This blog post describes a new enhancement called LAN sync, which allows devices on the same network to share files without upload/download through Dropbox servers.

Evernote:

1. Core Concepts in dev API doc, especially the Data Model, which defines various data structures, familiar to you if you use Evernote a lot, like me.

2. Synchronization specification in dev doc: "Evernote Synchronization via EDAM" (Jan 2013)


If you are interested, you can also look into FUSE, which is a user space filesystem abstraction. You can build your own Dropbox and Evernote ;-)

order of event queue in Node.JS event loop

I was reading the book "Node.JS in Practice", in Chapter 2, Technique 14, it talks about the order of scheduling I/O events, setImmediate, setTimeout/setInterval and process.nextTick. Like in the following screenshot from the book:

However, it seem that the ordering is a bit different, from my test, it is more like: process.nextTick, setTimeout/setInterval, setImmediate.

Here is the sample test code and result (Node.JS v4.1.0):

setImmediate(function() {
    console.log("from setImmediate");
});

setTimeout(function() {
    console.log("from setTimeout");
}, 10);

process.nextTick(function() {
    console.log("from process.nextTick");
});

console.log("hello");

output:

hello

from process.nextTick

from setTimeout

from setImmediate

A retouch of JavaScript and Node

It has been more than two years since I last touched JavaScript and Node. My current projects are mostly centered around Java. But it was always fun to come back and refresh my memory and knowledge about it.

It almost feels like traveling through time machine, too many things have changed over the past two years. Node was forked into io.js and ES6 has been out and getting traction. And there are new frameworks out each week. Frontend engineers must have been having a tough life to keep up, and tons of fun as well.

Here I just keep some book notes for my reading through the book "Node.js in Action". The book itself turned out to be a bit outdated, but many basics are still valid. There is also another more recent Manning book "Node.js in Practice".

Difference between exports and module.exports
exports is a reference to module.exports, so as long as you do not reassign it to something different, they are the same.

Module search sequence
search for node_modules recursively from current directory, if not found, then search for environment variable NODE_MODULE or NODE_PATH.

Module require search sequence
Search for index.js, then inside package.json to find the "main" definition.

Flow control libraries

async: https://github.com/caolan/async

nimble: https://github.com/caolan/nimble

step: https://github.com/creationix/step

seq: https://github.com/substack/node-seq

See article for a discussion about how to handle async programming with JavaScript: How to survive asynchronous programming in JavaScript

unit test frameworks

nodeunit: https://github.com/caolan/nodeunit

mocha: http://mochajs.org

vows: http://vowsjs.org

should.js: assertion lib: https://github.com/tj/should.js/

functional test frameworks

tobi: https://github.com/LearnBoost/tobi

soda: https://github.com/LearnBoost/soda

Keep app running/restart

monit: https://mmonit.com/monit/

forever: https://github.com/foreverjs/forever

upstart: http://upstart.ubuntu.com/index.html

Node Web frameworks, so many of them

Express: http://expressjs.com

Hapi from walmartlabs: http://hapijs.com

A good intro about why Hapi was created: http://hueniverse.com/2012/12/20/hapi-a-prologue/

Kraken from paypal: http://krakenjs.com

Restify: http://mcavage.me/node-restify/

Koa, next gen Node Web framework using ES6: http://koajs.com

LoopBack: http://loopback.io

Sailsjs, MVC pattern similar to RoR: http://sailsjs.org

Meteor: https://www.meteor.com

Quite in depth comparison of Express, Koa and hapi: https://www.airpair.com/node.js/posts/nodejs-framework-comparison-express-koa-hapi

hapi vs Express: http://megagalactic.io/hapi-vs-express/

Express, hapi, Sails: http://blog.pluralsight.com/node-web-frameworks

Performance comparison of Express, hapi, and Restify: https://raygun.io/blog/2015/03/node-performance-hapi-express-js-restify/

Performance Testing with JMeter for REST Services - A Quick Start Guide

Started to pick up JMeter for a project that exposes a REST API. It is a quite versatile and popular performance and stress test tool. And people have built plugins to extend it.

Official guide: https://jmeter.apache.org/index.html
Extra plugins: https://github.com/undera/jmeter-plugins
Two books by Bayo Erinle: Performance Testing with JMeter 2.9 (2nd edition is coming out soon) and JMeter Cookbook

1. Install JMeter and Plugins

If you are on a Mac, using Homebrew is the easiest way, it installs both vanilla JMeter and extra plugins:

brew install jmeter --with-plugins
==> Downloading https://www.apache.org/dyn/closer.cgi?path=jmeter/binaries/apache-jmeter-2.13.tgz
==> Best Mirror http://mirrors.sonic.net/apache/jmeter/binaries/apache-jmeter-2.13.tgz
######################################################################## 100.0%
==> Downloading http://jmeter-plugins.org/downloads/file/JMeterPlugins-Standard-1.2.1.zip
######################################################################## 100.0%
==> Downloading http://jmeter-plugins.org/downloads/file/ServerAgent-2.2.1.zip
######################################################################## 100.0%
==> Downloading http://jmeter-plugins.org/downloads/file/JMeterPlugins-Extras-1.2.1.zip
######################################################################## 100.0%
==> Downloading http://jmeter-plugins.org/downloads/file/JMeterPlugins-ExtrasLibs-1.2.1.zip
######################################################################## 100.0%
==> Downloading http://jmeter-plugins.org/downloads/file/JMeterPlugins-WebDriver-1.2.1.zip
######################################################################## 100.0%
==> Downloading http://jmeter-plugins.org/downloads/file/JMeterPlugins-Hadoop-1.2.1.zip
######################################################################## 100.0%
🍺  /usr/local/Cellar/jmeter/2.13: 1926 files, 115M, built in 74 seconds

If you are on other platforms, do these:

1. download JMeter from the official site.
2. download and install extra plugins based on your use cases. These plugins are grouped into several packages, see the detailed plugins package content page to decide what you need. I only installed "Standard Set" and "Extra Set". Installing these plugin packages is simply unzipping them to the installation directory of JMeter, e.g. they install jars into install_dir/lib/ext, or install_dir/bin, etc., see plugin installation guide for more details and minor config changes. The download page has all the links for the plugin packages.
3. ServerAgent-X.X.X.zip contains the PerfMon Sever Agent that you need to run on the server under test. To run the agent, no special permissions are required. After the agent is running, you can use PerfMon Metrics Collector Listener to connect to the agent and monitor various metrics for CPU, Memory, Swap, Disk and Network I/O, etc. See the document for PerfMon Server Agent and Servers Performance Monitoring for more details.

2. Using JMeter and Plugins

JMeter runs in various modes, you can use with a GUI client or without, you can also set up remote test clients for distributed testing to simulate a more practical workload and traffic pattern.

Here is a basic test for HTTP GET request for a sample service API from geonames.org:

URL: http://api.geonames.org/citiesJSON?north=44.1&south=-9.9&east=-22.4&west=55.2&lang=de&username=demo

Here is the screencast (click on it), note those listeners starts with "jp@gc" are from the non-standard plugins we installed above.

3. Other Interesting Tools and Resources

gatling.io: another high performance open source load testing framework based on Scala, Akka and Netty. It has a DSL based on Scala. It also has nice integration with Jenkins.

yandex-tank: Load testing tool written in Python. For more details, check out its documents.

BlazeMeter: a hosted performance testing service, you can easily reuse your JMeter test scripts with it. It also provides integration with Jenkins CI/CD and supports mobile performance testing. Here is a quick screencast from its website:

Loadsophia: This is a service provided by BlazeMeter, it stores and visualizes the performance test results. The organic visualization in JMeter is quite limited and non-interactive. This service makes analyzing performance data intuitive and fun. It supports test results from tools like JMeter, Apache Benchmark and Yandex.Tank. You can see examples provided publicly by existing users here: http://loadosophia.org/examples/

flood.io: Cloud load testing tool, it supports JMeter and Gastling. Here is a sample report: https://flood.io/d384673f64e3a3

Insights into the success of Storm

Just read through Nathan Marz's post about the history of Storm. This is really a nice recap of how to successfully start, grow and maintain a great open source project. Technical excellence is important, but the marketing, growing the community and adoption is even more critical.

Here is his post "History of Apache Storm and Lessons Learned".

I am really looking forward to his new book on the lambda architecture: Big Data, principles and best practices of scalable realtime data systems.

Swift coding style guide

Found two coding styles for Swift:

• github style guide for Swift: https://github.com/github/swift-style-guide
• raywenderlich style guide for Swift: https://github.com/raywenderlich/swift-style-guide

Interesting notes:

1. In github style, it suggests that let should be used over var binding whenever possible. This is to explicitly show the intent that a value is supposed to or not supposed to change.

2. In raywenderlich style, when declaring protocol conformance, use separate extension instead of declaring all together. Also, do not forget to add // MARK comments.

Preferred:

class MyViewcontroller: UIViewController {
  // class stuff here
}

// MARK: - UITableViewDataSource
extension MyViewcontroller: UITableViewDataSource {
  // table view data source methods
}

// MARK: - UIScrollViewDelegate
extension MyViewcontroller: UIScrollViewDelegate {
  // scroll view delegate methods
}

Not Preferred:

class MyViewcontroller: UIViewController, UITableViewDataSource, UIScrollViewDelegate {
  // all methods
}

3. In raywenderlich style, when unwrapping the optional, shadow the original name instead of using a new name.

Preferred:

var subview: UIView?

// later on...
if let subview = subview {
  // do something with unwrapped subview
}

Not Preferred:

var optionalSubview: UIView?

if let unwrappedSubview = optionalSubview {
  // do something with unwrappedSubview
}

4. In github style, prefer implicit getters on read-only properties.

Preferred:

var myGreatProperty: Int {
    return 4
}

subscript(index: Int) -> T {
    return objects[index]
}

Not Preferred:

var myGreatProperty: Int {
    get {
        return 4
    }
}

subscript(index: Int) -> T {
    get {
        return objects[index]
    }
}

5. In github style guide, it is mentioned to prefer structs over classes. Since I am still quite new to Swift, I will need to learn a bit more to really understand it.

book notes for Building Applications with iBeacon

I am always very interested in the location-based mobile applications, in particular indoor location apps. I've even explored a little bit about creating a mobile app that helps locating and assembling colleagues in a crowded cafeteria during lunch. At that time, what I learned is that indoor tracking is still an area being actively researched and explored, many solutions use wifi router capabilities specific to vendors such as Cisco.

Lately, I have quickly went through Matt Gast's book on iBeacon: Building Applications with iBeacon. This is a very concise book on the topic, only 80 pages. It gives a quick overview about iBeacon technology mainly focused on iOS.

Here are the book notes I took:

Companies

Estimote

One of the earliest developers of beacon technology. Estimote beacons have fixed configuration parameters and, in particular, administrators cannot set the UUID.

RadBeacon

RadBeacon is a $29 USB dongle that performs the transmission functions of an iBeacon. All you have to supply is USB power. Configuration of the beacon’s numbers is done through an app.

Kontakt

Kontakt sells an ARM-based iBeacon as well as tools for manag‐ ing iBeacons and analyzing user interactions with them.

http://kontakt.io/our-technology/

Also compatible with android

Gelo

Gelo’s Beacons are waterproof and designed for both indoor and outdoor use, and the batteries can be replaced manually using simple tools.

GIMBAL: http://www.gimbal.com

---

Hardware

• Mac (Yosemite stops supporting Mac as iBeacon)
• iOS devices
• Raspberry Pi
• Arduino: BLEduino, BLE Mini from RedBearLab
• Nordic Semi nRF1822

---

iOS Apps

Geohopper: https://twocanoes.com/geohopper

- A personal notification beacon for the iPhone. It comes with a beacon device and Geohopper for iOS. It sends notification based on your geo-location. It also integrates with web services for automated workflow.

(Developer Tools)

There are lots of developer util tooks that turns iOS devices into iBeacons. The list can go on and on, here are some apps I found in the app store:

• Beacon Broadcaster
• Locate Beacon (from Radius Networks)
• Beacon Manager
• My Beacon
• Beacon Toolkit
• Beacon Harvester: help you find and save iBeacons around you
• Beacon Bits

(Games)

One interesting use case for iBeacon is the treasure hunt games. There are several demoes during the mobile developer conferences, very interesting to connect mobile and real world in a playful way. For example, Beacon Scavenger Hunt (from Radius Networks) is a treasure hunt game to collect scavenger hunt’s badges.

---

Mac Apps

It was very bad that Yosemite does not support turning a Mac into an iBeacon. Maybe this is one of the bugs with Yosemite?

MacBeacon: http://www.radiusnetworks.com/ibeacon/macbeacon/

http://beekn.net/2013/11/turn-your-mac-into-a-bluetooth-beacon/

Mavericks as an iBeacon: https://github.com/mttrb/BeaconOSX

MactsAsBeacon: https://github.com/timd/MactsAsBeacon

BeaconEmitter: https://github.com/lgaches/BeaconEmitter

a nice thread on StackOverflow about turning your mac into iBeacon: http://stackoverflow.com/questions/19410398/turn-macbook-into-ibeacon

---

Use Cases

10 things to do with iBeacon: http://blog.twocanoes.com/post/68861362715/10-awesome-things-you-can-do-today-with-ibeacons

---

Other Useful Resources

Apple's Official Doc on iBeacon: https://developer.apple.com/ibeacon/
Radius Networks Developer Guides: http://developer.radiusnetworks.com
beekn.net: a website dedicated to topics on beacons with Bluetooth Low Energy tech (BLE)