Wednesday, December 28, 2011

useful vim tips

Vim is definitely a text editor you need to learn progressively because it has an interesting learning curve. If you have not read "Seven Habits of Effective Text Editing" from Bram Moolenaar (main author of Vim), please do that first. There is also an updated version of this article that can be found here. Basically, there are three steps to improve your Vim skills through daily work:
  1. find repeated actions that are not efficient or quick enough
  2. find a quicker way (search online, ask around, read documentation and tutorials, etc.)
  3. practice the new tricks and make it a habbit
So, basically I am trying to take notes about some new tricks I learned over the past week. Hopefully, I will do the iterations and find even more efficient ways to do these in the future.


Quickly move around on a line.

0 (zero) moves the cursor to the beginning of a line (white spaces are considered as characters)
$ moves the cursor to the end of a line (white space are considered)
^ moves the cursor to the beginning of a line (white space ignored)
g_ moves to cursor to the end of a line (white space ignored)

For example (using + to stand for space here for illustration purpose):
++a line++
- -    - - 
0 ^    $ g_

w, W move the cursor forward to the beginning of a word, the difference between the two is W uses white space as separator. So, for me w is quite useful in code, W is useful in the textual sentences.

e, E move the cursor forward to the end of a word, same difference as in w and W.

b, B move the cursor backward to the beginning of a word, difference is again the separator.

ge, gE move the cursor backward to the end of a word.

*, # to quickly move forward and backward to the same word under the cursor. This is faster than search using / and press N and n.

o adds a new line under current line and start insert.
O adds a new line above current line and start insert.
s deletes the character under cursor and start insert.
S deletes the current line and start insert.
D deletes from cursor to the end of line.
C deletes from the cursor to the end of line and start insert.


Quickly move around in code.

% matches the parenthesis, bracket and curly brace. For example, put the cursor on a closing curly brace  and % takes the cursor to the matching opening brace.

[{ and ]} to match the opening and closing curly braces. Place the cursor inside a code block, using these two to quickly move to the beginning and end of the closes curly braces pair.

gd jumps to the local variable definition when the cursor is on a variable name.
gD jumps to the global variable definition.


Use . (dot) command to repeat the last change in normal mode. See Vim Twiki "repeat last change".

You can also do N. to repeat the change N times. For example, dd command delete the current line, use . to delete another line, use 5. to delete additional 5 lines.

There is also a similar repeat trick @: for last command made on command line. For example, on the command line, we do :!ls to list files in the current directory. Then, we can do @: to repeat it again.


Copy/cut and paste text in visual mode. See Vim Twiki "copy and paste using visual selection".
  • put cursor to the beginning of the desired text
  • enter visual mode: use v to begin character-based selection, V to begin line based selection, ctrl-v to begin vertical block based selection (move the cursor will select the whole columns of text)
  • put cursor to the end of the desired text
  • use d to cut, y to copy
  • put the cursor to the paste position
  • use p to paste after the desired position, P to paste before the desired position
On the Vim twiki, there is also the tip to overwrite the same amount of text using the copied/cut text using Nvp, NVp, N^Vp depending on the selection mode (character, line or block).


Disable auto-indentation, manual indentation.

Sometimes, I had one annoying problem when I copy and paste formatted source code from another application - Vim will reformat the content and mess up with the indentations. To prevent Vim from retabbing, use :set paste. After you are done, use :set nopaste to diable paste. You can also add a key binding using set pastetoggle in your .vimrc file. See Vim Twiki "Toggle auto-indenting for code paste" for more details.

Instead, you want manual indentation. In normal mode, == indents the current line. In line-based visual mode (V or shift-v), select several lines and = indents them all.


Jump in the change list, moving back and forth in changed positions in the current file. One common task is to jump back and forth between edit positions. See Vim Twiki "list changes to the current file".
  • `.  jumps to the last edit position
  • g; jumps back in the change list
  • g, jumps forward in the change list
  • :changes to show the change list
Interestingly, there is another (probably less useful) jump list that records the default last 100 positions a user makes a jump. A jump is one of search, substitute and mark. See Vim Twiki "jumping to previously visited locations".
  • ctrl-O to jump back in jump list
  • ctrl-I to jump forward in jump list
  • :jumps to show the jump list


Auto expansion, adding comments, etc.

One common case when coding is to add comments. So, how do you add something like these lines:

  • ///////////////////////////
  • /*------------------*/
  • /*************** <multiple lines of text> *************/
Ni/<esc> inserts N / (slash), quite useful for N repeating characters.

Another way is to use auto expansion. For example, you can add these into your .vimrc:

:ab #b /************************************************
:ab #e ************************************************/
:ab #l /*----------------------------------------------*/

Then, in the edit mode, you can type #b<enter> to automatically insert /*******.

Text objects.

Last, I found out about Vim text objects, it is so powerful and convenient that saved me a lot of time. For details, please read this great post "Vim Text Objects: The Definitive Guide" and Vim documentation on text objects.

Note that text object is more convenient than motion when you want to delete a chunk of text, you don't need to move cursor to particular position. For example, the difference between dw and diw is that when you want to use dw to delete a word, you must move the cursor to the beginning of that word first. For diw, you can place the cursor anywhere inside that word.

For example, I want to refactor code that has a string value hard coded to use a variable:

public class Hello {

    public static void main(String[] args) {
        System.out.println("hello, world");
        showMe();
    }

    public static void showMe() {
        System.out.println("hello, world");
    }
}

// refectored
public class Hello {

    public final static String hi = "hello, world";
    public static void main(String[] args) {
        System.out.println(hi);
        showMe();
    }

    public static void showMe() {
        System.out.println(hi);
    }
}

So, I can add a variable hi="hello world", then replace the string value "hello, world" on line 4 and line 9 using text object:

  1. move to line 4, a position before the string "hello, world", in normal mode, type ca", this will delete the string value including the quotes and change to insert mode.
  2. type the new variable name hi
  3. move to line 9, a position before the same string
  4. type . to repeat the replacement
  5. done!
One extra tip. Using text objects, you can delete/update the content inside the double quotes, but how do you delete the double quotes only? You can do this:


  1. move cursor to one of the double quotes
  2. use % to jump to the matching double quotes, delete it using x
  3. use Ctrl-o or two back-ticks (``) to jump back to the first double quote, delete it using x
  4. done!


Sunday, December 18, 2011

Web application security readings

Articles:

Slides:
Specifications:
  • CSP (Content Security Policy): a W3C draft spec that defines fine grained security policies for resource loading to mitigate the risk of injection attacks such as XSS. The policy per resource representation is defined in new header is Content-Security-Policy. Firefox and Chrome is experimenting in headers X-Content-Security-Policy and X-WebKit-CSP respectively. The spec also defines a report-only mode without browser enforcing the policy but sending violation reports to the server, which is helpful for gradually enforcing the right policy.
Books:

Tuesday, November 22, 2011

so many cookies

I was reading upon HTTP cookies and it is actually quite confusing. Because there are many different cookie specs and browsers not necessarily support all of them.

Original Netscape Cookie Spec: http://curl.haxx.se/rfc/cookie_spec.html

Very simple and straightforward format:

Set-Cookie: NAME=VALUE; expires=DATE;
path=PATH; domain=DOMAIN_NAME; secure

RFC 2109 (based on Netscape spec with minor tweaks): http://tools.ietf.org/html/rfc2109

The syntax for the Set-Cookie response header is

   set-cookie      =       "Set-Cookie:" cookies
   cookies         =       1#cookie
   cookie          =       NAME "=" VALUE *(";" cookie-av)
   NAME            =       attr
   VALUE           =       value
   cookie-av       =       "Comment" "=" value
                   |       "Domain" "=" value
                   |       "Max-Age" "=" value
                   |       "Path" "=" value
                   |       "Secure"
                   |       "Version" "=" 1*DIGIT

RFC 2965, obsoletes RFC 2109: http://tools.ietf.org/html/rfc2965

It adds more syntax items. It is very interesting to read 9.1 section about the compatibility with existing cookie implementation. Basically, it should overwrite an existing cookie value set by Set-Cookie if domain and path both matches.

The syntax for the Set-Cookie2 response
   header is

   set-cookie      =       "Set-Cookie2:" cookies
   cookies         =       1#cookie
   cookie          =       NAME "=" VALUE *(";" set-cookie-av)
   NAME            =       attr
   VALUE           =       value
   set-cookie-av   =       "Comment" "=" value
                   |       "CommentURL" "=" <"> http_URL <">
                   |       "Discard"
                   |       "Domain" "=" value
                   |       "Max-Age" "=" value
                   |       "Path" "=" value
                   |       "Port" [ "=" <"> portlist <"> ]
                   |       "Secure"
                   |       "Version" "=" 1*DIGIT
   portlist        =       1#portnum
   portnum         =       1*DIGIT

RFC 6265, obsoletes RFC 2965: http://tools.ietf.org/html/rfc6265

It admits the same confusion that I had, and deprecates the Cookie2 and Set-Cookie2 headers introduced in RFC 2965!

"Prior to this document, there were at least three descriptions of
   cookies: the so-called "Netscape cookie specification" [Netscape],
   RFC 2109 [RFC2109], and RFC 2965 [RFC2965].  However, none of these
   documents describe how the Cookie and Set-Cookie headers are actually
   used on the Internet (see [Kri2001] for historical context).  In
   relation to previous IETF specifications of HTTP state management
   mechanisms, this document requests the following actions:

   1.  Change the status of [RFC2109] to Historic (it has already been
       obsoleted by [RFC2965]).

   2.  Change the status of [RFC2965] to Historic.

   3.  Indicate that [RFC2965] has been obsoleted by this document.

   In particular, in moving RFC 2965 to Historic and obsoleting it, this
   document deprecates the use of the Cookie2 and Set-Cookie2 header
   fields."

The actual format is getting updated as well. Notice that now both Expires and Max-Age are supported with Max-Age overriding Expires if both exist. If not, the cookie is kept by user agent till the current session ends.

 set-cookie-header = "Set-Cookie:" SP set-cookie-string
 set-cookie-string = cookie-pair *( ";" SP cookie-av )
 cookie-pair       = cookie-name "=" cookie-value
 cookie-name       = token
 cookie-value      = *cookie-octet / ( DQUOTE *cookie-octet DQUOTE )
 cookie-octet      = %x21 / %x23-2B / %x2D-3A / %x3C-5B / %x5D-7E
                       ; US-ASCII characters excluding CTLs,
                       ; whitespace DQUOTE, comma, semicolon,
                       ; and backslash
 token             = <token, defined in [RFC2616], Section 2.2>

 cookie-av         = expires-av / max-age-av / domain-av /
                     path-av / secure-av / httponly-av /
                     extension-av
 expires-av        = "Expires=" sane-cookie-date
 sane-cookie-date  = <rfc1123-date, defined in [RFC2616], Section 3.3.1>
 max-age-av        = "Max-Age=" non-zero-digit *DIGIT
                       ; In practice, both expires-av and max-age-av
                       ; are limited to dates representable by the
                       ; user agent.
 non-zero-digit    = %x31-39
                       ; digits 1 through 9
 domain-av         = "Domain=" domain-value
 domain-value      = <subdomain>
                       ; defined in [RFC1034], Section 3.5, as
                       ; enhanced by [RFC1123], Section 2.1
 path-av           = "Path=" path-value
 path-value        = <any CHAR except CTLs or ";">
 secure-av         = "Secure"
 httponly-av       = "HttpOnly"
 extension-av      = <any CHAR except CTLs or ";">



To conclude, here is a paper providing a good overview and discussion about the cookie: "HTTP Cookie: Standards, Privacy and Politics".

Monday, November 14, 2011

SPAM: Please get a new EIN

Another spam I got today, it looks fairly serious ;-), maybe the "Dear Sirs" part sold itself. Anyways, here it is for fun:

Title: Please get a new EIN
From: Clementina Evans
To the ATTENTION of: Accounting Department
Dear Sirs,
please be informed that you need to get a new EIN in connection
with the probability of fraudulent actions on your behalf. We
are sorry for the inconveniences that might be caused to you
by this measure. Please use the following link for instructions:
http://onesourceprocess.com/7wmzwt/index.html

Yours sincerely,
Clementina Evans
Office of Professional Responsibility
Internal Revenue Service

Tuesday, November 8, 2011

SPAM: New EIN

Just found it interesting to publish some lastest spams I got each day here, hopefully this would be helpful to non-tech people. Please don't try the links inside ;-)

Title: New EIN
From: Robert Glass
Content:
To the ATTENTION of: Accounting Department
Dear Sirs,
we would like to notify you that you are required to receive
a new EIN in order to prevent fraud. We are sorry for any troubles
that might be caused to you by this measure. Please use the following
link for instructions:
http://web.knauf.es/5nse63/index.html

Yours sincerely,
Robert Glass
Office of Professional Responsibility
Internal Revenue Service 

Sunday, October 9, 2011

Unwanted Comcast features

After multiple occurrences of AT&T dropping connections over the past months, I finally decided to switch to Comcast (yes, I have heard many terrible stories about their billing department, sales fraud, etc.). Plus, I got a really good deal from my colleague ($19 per month for the first year, $35 per month for the second year for their performance package, 15MB upload and 5MB download.

Based on advice, I did not choose to rent their modem and bought my own from Amazon: Motorola SB6121 and it worked great. There is an older model SB6120 that is a bit cheaper, but my friend had problems. Please check with the technician and confirm that those models work in your area.

Today, they came in and installed it for me. The overall process worked fine except for three complaints:

1.  They kept calling for confirmation. In the past week, they called so many times for confirmation, 3 days before, yesterday, and this morning. I understand that some people might be forgetting things very quickly, but I am not that type of people. Plus, I have confirmed at the first call already and they are still calling as early as 7am in the morning! Since I did not get this morning's phone call, they left a message and said I would have to reschedule it, WTF. So, I ended up calling them again so they can send the technician in without rescheduling.

2. Some technicians are un-professional. I am quite busy with work and life, so my appointment is 8-10am, earliest I could get so I don't miss the work. For the first appointment, the technician waited outside for 1 hour before he came in (I saw his van and went out to tell him I am ready, but he kept sitting in his var for another half an hour). When he came in, I showed him where I want the setup and the pole to pull the signal from. He shrugged his shoulder and said "it will take at least 4-5 hours". I said I thought it'd be only 1-2 hours and I had no choice but to reschedule. Somehow from the look on his face, I felt that he did not want to do installation (bad mood, lazy, etc.) and I am fairly pissed. If you don't want to work, then don't come to waste your time and my time.

Today, another technician came in and he is much better. He finished the whole job in 1 hr and told me the previous technician "must be lazy".

3. The installation process forces you to install the desktop installer to activate. After the hardware is set up, I hooked up my wireless router and was directed to an activation page, which asked me to create a new Comcast user account. But the last step is to install Comcast desktop installer and there seems to be no way around according to the technician. (Update: according to my colleague, you don't need to install it, just restart the modem, shutdown router, connect router with modem, start the router. But I did not try this personally, not sure if it would work.)

The installer file is "ComcastDesktopInstaller_MMF3.pkg" in my case. I uploaded it online if you are interested. Luckily I unchecked "install browser toolbar" option during installation. People are suffering from this toolbar since it hijacks browser preferences and homepage. If you are the victim, please follow instructions on this page to get your previous setting restored.

After the network is up and running, I want to uninstall it but could not find much info. I did my best, and my conclusion is it might not do as much harm. It executed some Ruby scripts and installed some crap on the desktop, which you can delete easily.


Cleanup actions in summary:


1. Delete the two website shortcuts installed on my desktop.
2. Delete ~/.cim_install_log. 
3. I am not familiar with Ruby, but the scripts seems to just create Xfinity bookmarks for Safari, Firefox and Chrome. For details in the script, see the gist below.
4. Besides those, I did not seem to find other suspicious malware or unwanted junks.

Detailed checkups for the Comcast app:

1. Did a grep for "comcast" and found a log file in home directory: ~/.cim_install_log. It contains all the output during installation process.
2. Used AppCleaner to check installed applications, does not find any malicious app installed from Comcast installer.
3. Used pkgutil command to search for anything related to comcast in package receipt database. See pkgutil manpage for details how to use it. I could not find any new applications installed from this installer pkg.
4. Used Pacifist to check the content of the pkg file. Did not see much suspicious content except for several Ruby and shell scripts to run during installation.

Don't install Firefox Toolbar

By default, these website shortcuts are install on your desktop, remove them


Content of ComcastDesktopInstaller_MMF3.pkg






 Reference:

Saturday, September 10, 2011

Big list of things to get started with frontend development

Quickly ran though "Getting Good with JavaScript", this is a very short and basic introduction to JavaScript for someone new to the language. Deep and difficult concepts are avoided or explained in an easy to understand way for the reader to explore further by him/herself.

I found these two resources in the "Further Study" section very helpful and interesting, both are from Rey Bango's blog:

1. What to read to get up to speed in JavaScript
2. The big list of JavaScript, CSS, and HTML development tools, libraries, projects and books

Wednesday, August 31, 2011

display Apple keys in HTML

Just saw an interesting kbd tag supported on Stackoverflow and started wondering how to display Apple keys like command, shift, control, option, etc. on the Web page.

And it actually is very straightforward using HTML character entities. The concept is well explained in this Wikipedia article. There are two forms: numeric character reference and character entity reference:


numeric character reference: either in decimal form (&#dddd;) or in hexadecimal form (&#xhhhh;), e.g. "less than" character can be expressed as &#60; or &#x3c;

character entity reference: in the form of &name; the same "less than" character can be expressed as &lt;

Now, it is very clear how to display these special Apple keys with the help of entity tables such as this unicode table. Somehow I could not find the corresponding character entity reference table for the same keys.

These are the most common Apple keys you might use:

⌘ Command: &#x2318;
⇧ Shift: &#x21E7;
⌃ Control: &#x2303;
⌥ Option: &#x2325;
⎋ Esc: &#x238B;

If you use TextMate, there is a convenient HTML bundle feature that helps to insert these special characters in an HTML document. Create a new HTML document, then goto Bundles menu => HTML => Entities => Pick the character you want. You can also use the autocomplete, e.g. type "command" and press tab.

migrate from TextMate to Vim/MacVim

Choosing text editor is like choosing your religion. One popular/natural choice on Mac is TextMate. However, since I do lots of work on the server/dev linux box directly, I had to switch back and forth from Vim. Probably it is time to migrate to use Vim on Mac as well, especially given the fact that there are nice distro made for Mac specifically: MacVim and a customized MacVim packaged with many useful plugins called Janus.

Here are two good testimony from other migrants:

Here are several useful learning resources:
Plugins shipped with Janus:

Spend 5 minutes to go through the brief introduction for all the plugins that ships with Janus. Janus also includes a handy way to allow you to create/edit ~/.janus.rake file to add more plugins and you can run rake inside ~/.vim to update these plugins.

Sample ~/.janus.rake


vim_plugin_task "zencoding", "git://github.com/mattn/zencoding-vim.git"
vim_plugin_task "minibufexpl", "git://github.com/fholgado/minibufexpl.vim.git"

NERDTree: provides a file browser, use \n to show/hide.

Command-T: provides an efficient way to search for files in current working directory. Use ⌘+T to show/hide.

snipMate: provides the same Textmate Snippet feature. As you do in TextMate, you can simply type snippet keywords and press tab to expand to code snippets and use tab to jump through various locations in the snippets. You can find all snippet definitions for various languages in ~/.vim/snippets.

ctags: TagList plugin is shipped with Janus. Use :Tlist to show/hide it. But I like Tagbar better than TagList (see below for installation instructions).

Other handy plugins I installed:

Syntastic: syntax checker, you can also go to the jslint route. This Stackoverflow post covers the setup in details.

Tagbar: displays tags of the current file in a sidebar, sorta like TagList, but more comprehensive and orders tags under correct scope/context.

doctorjs: previously known as jstags, provides a set of static analysis tools for JavaScript. Combined with Tagbar, it provides a more detailed code structure navigation than TagList. See this Stackoverflow post for instructions how to set it up. Note that you need to install NodeJS as a prerequisite.

There are two issues to get doctorjs set up properly that is not covered in that Stackoverflow post:

  1. It seems that installing doctorjs needs sudo (sudo make install inside doctorjs git clone directory) to get around permission issue.
  2. I also got errors about underscore module and ctags NodeJS module cannot be found. They are located at /usr/local/lib/jsctags/ after doctorjs is installed. But node cannot locate them somehow. To fix it, I installed underscore using npm (npm install underscore -g), then I updated my NODE_PATH environment variable to include /usr/local/lib/jsctags/.




Friday, August 5, 2011

file system based on a single file and loop device

Worked on a task to test performance of read/write on various file system options on my RHEL5 box:
  • native ext3
  • virtual ext3 based on a zero-filled image file
  • virtual ext3 based on a sparse image file
  • virtual ext2 based on a sparse image file
The virtual file system is basically based on a big file connected as a loop device. This tutorial about setting up disk quota based on virtual file system has more details. For example, this is the basic steps to set up a virtual ext3 file system based on a sparse file:

dd if=/dev/null of=/tmp/vfs/sparse.image bs=1M seek=100
mkfs -t ext3 -q /tmp/vfs/sparse.image -F
mkdir /tmp/quota
sudo mount -t ext3 -o rw,loop /tmp/vfs/full.image /tmp/quota

The experiment is quite preliminary: basically, I copy over files in different sizes (1, 5, 10, 25, 50MB) into the mounted file system directory or the native file system directory to test write latency, then read the copied files to test read latency. The read/write latency is averaged over 10 iterations.

A couple of quick points from the experiments:

  • read latency is similar for all four cases.
  • write latency for native ext3 is around half of the virtual ext3 based either on a sparse file or a zero-filled file.
  • write latency for native ext3 is roughly the same as virtual ext2, maybe because ext2 gets rid of the journaling overhead.
  • creating a virtual file system based on a sparse file has many advantages over a zero-filled file: the creation time is way much faster 6-8 order of magnitude for the file sizes we tried, the bigger the file, the bigger the difference; the sparse file does not occupy physical disk space and grows when content is actually written to disc.
  • virtual ext3 has spikes for write, sometimes as big as 80x the average write latency, not exactly what caused it.
This is final the performance graph for comparison. I am using the average number here, notice that the high average latency for write on virtual ext3 is caused by the spikes mentioned above.


After removing the spikes, using virtual ext3 based on a sparse file or zero-filled file creates around 2X the latency for native ext2 or ext2 based on a sparse file, as show in the second graph:

Friday, July 22, 2011

Multi-language support for a website

I am not a frontend developer, just stumbled upon this problem helping out my friend to set up a website that is supposed to serve multi-language content based on the language setting of the user. This is actually an interesting problem (maybe already solved and I did not know).

There are several different solutions that I found through after doing some study:

- Server side detection based on Accept-Language request header

For example, apache mod_negotiation allows you to define a set of html pages in different languages (index.html.en, index.html.zh, etc.) and based on the user preferences from Accept-Language header, apache picks the right index.html language page to return.

You can also use mod_rewrite to perform the similar trick. Just set up the rewrite condition based on the value of the AL header and map to the correct language.

- Client side detection and action using JavaScript and navigator object.

In IE, navigator.userLanguage returns the operating system region and language setting, in Firefox, Opera, Safari and Chrome, navigator.language returns the browser application language setting. So, we can use them to get the user language and decide what language content to return.

For example, this code snippet (borrowed from an example on jsfiddle) defines multi-language content on the same page and use JavaScript to show or hide it based on the detected language.



The drawback of this approach is that a large part of the content is not needed and invisible. It is a waste of time to download.

Another approach is to use JavaScript to redirect to individual language content pages using window.location.href="redirected_page_url" or window.location.replace("redirected_page_url"). See Location object for more details on how to use them. The downside is this approach may not be search engine friendly (spiders may not be able to follow redirection).

One last note how to test multi-lang website. I tried in Firefox, simple change the language ranking in Preferences->Content Tab->Language Choose button and FF will start sending different Accept-Language headers based on the ranking. You can also play with the operating system language settings.

Monday, June 13, 2011

node.js testing frameworks

There are so many testing frameworks for node.js, start to evaluate some most mentioned/used ones. Criteria that I am looking for:


  1. easy to understand concept and constructs
  2. nice support for async calls and flow control
  3. easy integration with CI tools such as Hudson
  4. code coverage report
  5. mocks and stubs


Among those, I like Jasimine most, it is easy to understand and has rich feature set. The only feature missing is lack of code coverage report. jasimine-reporters supports output of JUnit format unit test results. jasimine-node brings Jasimine to node.js. You can check out "Testing Backbone applications with Jasimine and Sinon" for a practical example.

Sunday, June 5, 2011

dealing with multiple nesting callbacks in asynchronous code

If you are annoyed that you have to write multiple nesting asynchronous callback functions when you work with Node.js, you are not alone. Getting used to the asynchronous programming style and always keep in mind everything runs in the event loop is a bit tricky at the beginning.

There are many libraries and articles that addresses this nesting callback issue. This should help you to avoid those ugly pyramid shaped code now ;-)

InfoQ: how to survive asynchronous programming in JavaScript (includes a list of libraries with comparisons and Q&A from their developers)

Stackoverflow post about how to deal with the nesting callbacks (also includes a list of libraries)

Isaacs's presentation on the topic and his library Slide (he is the author of npm)

Tim Caswell (creationix)'s series on control flow in Node (II and III can be found on the right column, other articles by the same author)

CommonJS Promises is another way trying to address this issue by defining a standard interface for interacting with the result object of asynchronous actions. Roughly speaking, a promise object is a placeholder for the value for the asynchronous action. And you can treat is as normal object, assign values to another variable, pass it around, etc. Right now, FuturesJS implements this and it is not solely for server-side JS. dojo.Deferred is another similar effort. This SitePen blog post has some easier to understand coverage about the promises.

StratifiedJS extends the JavaScript language to achieve similar goals of allowing asynchronous control flow to be expressed in a synchronous way. Very neat work indeed. This is a short overview presentation about it.

Friday, May 13, 2011

get rid of annoying ._filename working with TextMate on a mounted NFS

Just had this annoying experience of working on an NFS mounted file system with TextMate. Basically, it tries to store metadata such as cursor position, etc. into metadata files. Since these mounted file system does not support extended attributes, so OSX instead writes to ._filename, which is really annoying since the directory is under version control and you see many of these metadata files showing up.

After poking around, I figured out how to turn this metadata feature off, yes, I lost a bit convenience feature, but no more cleanup of these ._filename anymore!

defaults write com.macromates.textmate OakDocumentDisableFSMetaData 1

References: TextMate Expert Preferences

Saturday, April 23, 2011

Hello, Cloud Foundry running node.js

Contrary to most people's view on the Amazon cloud outage incident, I believe this actually shows the power and outreach of the cloud computing. The technology needs some time to mature and become reliable, let's just be patient. It is very exciting to see even big sites like reddit, quora was affected, and even the cloud service provider Heroku was one of the victims.

There are many new players in the cloud computing field and Cloud Foundry from VMWare is definitely a very disruptive force to the current market. Currently, it supports Spring for Java apps, Rails and Sinatra for Ruby apps, JVM frameworks like Grails and Node.js! Best of all, the whole platform is open sourced.

CF provides a Micro Cloud for individual developers, for free now ;-) This is how to get a helloworld node.js app running:


1. Preparation

- Sign up for the free Micro Cloud for Cloud Foundry (http://www.cloudfoundry.com/).
- Wait a few days (mine took 9 days) and you will receive an approval email.
- Cloud Foundry uses a CLI tool called vmc for interactions with your Micro Cloud instance. Follow the instructions and use the credentials in the approval email to set up vmc (see "vmc getting started guide", really wish it is available not only in PDF format).

2. Write a helloworld node.js app

As of this writing, CF supports node.js-0.4.5 (use "vmc runtimes" to find out). The only thing that is different in the CF environment is that the system will assign a host and port to your application. CF adds several environment variables accessible through process.env.

For our helloworld, we are interested in VCAP_APP_HOST and VCAP_APP_PORT (there are also VMC_APP_HOST and VMC_APP_PORT, which seems still exist but will be deprecated).

So, you basically create a directory (I use the same appid as the directory name) and create an app.js for hello world like this:
var http = require('http'),
    port = Number(process.env.VCAP_APP_PORT || 3000),
    host = process.env.VCAP_APP_HOST || 'localhost';

http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Welcome to Cloud Foundry!\n' +
            host + ':' + port + '\n' +
            require('util').inspect(process.env, false, null));
}).listen(port, host);

console.log('Server running at http://' + host + ':' + port + '/');
This simple app just prints out "Welcome to Cloud Foundry!" and a bunch of environment variables ;-) Note that we put localhost and 3000 there so that it is easier to debug the app on a local node.js installation.

3. Deploy using vmc

The very first time you deploy the app, you run "vmc push <appid>". Note that you need to pick an id that is not being used, otherwise, you will get an error. It would be really nice for CF to provide a Web UI to check what appid is still available. You just follow the instructions and see the output. If everything goes well, you will see your app running at http://<appid>.cloudfoundry.com/.

To update your app, you need to run "vmc update <appid>" instead of "vmc push <appid>". You can also delete your app using "vmc delete <appid>".

vmc CLI provides many operations, "vmc -h" is your best friend to find and try them out.

4. Add npm module dependencies

In most cases, you will need to use dependent NPM modules such as Connect, Express, etc. With the current node.js 0.4.5 supported on CF, it is very straightforward.

- You create a package.json that defines your app/module with dependencies.
- Then you run "npm bundle", which creates a node_modules directory with all the required NPM modules.
- Finally, you do "vmc update <appid>" and you are done.

Here is a quick sample:

package.json
{
    "name": "helloworld-node",
    "version": "0.0.1",
    "dependencies":
        {
            "connect": "*"
        }
}

app.js
var connect = require('connect');

connect.createServer(function(req, res) {
    var body = 'hello, cloud foundry';
    res.setHeader('Content-Length', body.length);
    res.end(body);
}).listen(process.env.VCAP_APP_PORT || 3000);

console.log('connect server listening...');

dir structure: 
app.js, package.json, node_modules

5. To learn more

I will try to update this helloworld guide and add a bit about bundling dependent npm modules. To learn more about Cloud Foundry, read the CF blog and code, since it is open sourced, you can simply install it from github.

Updated: found a recent post about using MongoDB on cloudfoundry ;-)

Thursday, April 14, 2011

interesting node modules

There are so many node modules out there to the point that sometimes you can have several different ones that appear to solve the same problem.

node.js on GitHub has a list of classified node modules, but still it is quite difficult to pick the best one (in terms of node version compatibility, easy to use, still actively developed, good performance, etc.).

It would be really cool to have some sites that actually give reviews and code samples of using these node modules. If performance comparison can be provided, that is even better.

Inspired by "6 Must Have Node.js Modules", I am listing the modules that I found quite interesting and useful to the current work:

HTTP clients: 



XML to object conversion: 

  • node-xml2js, a fairly lightweight and fast xml to object conversion library
  • node-expat, a very high performance SAX lib, native binding to libexpat, it has a fork with object conversion support

Image processing: 

Sunday, March 20, 2011

resources to get started with node.js

There are quite some blogs and websites about "getting started with node.js". Here are some resources that I found most useful:

Saturday, March 19, 2011

set up Linksys WRT54GL as a WiFi repeater with Tomato

Since I got the new ASUS RT-N16, the old Linksys WRT54GL v1.1 router has been sitting there and collecting dust. There is only one phone line at home at one corner of the house, so I have been thinking of using the idle router as a WiFi repeater so that I can get decent WiFi coverage for each room.

Finally I found some time this weekend and studied about it. Many online tutorials cover how to do it using DD-WRT, another great customized firmware for WiFi routers. Since my RT-N16 already runs Tomato, I'd want to find a way to have Tomato on WRT54GL and work as the repeater as well.

After digging into the Web searches, finally I found Tomator author's FAQ for WDS, it is very straightforward. Basically, both both master and repeater routers need to set "Wireless Mode" into "AP and WDS". You also need to turn off dhcp and wan (turn WAN/Internet type into "disabled") on the repeater. Then you configure WDS to "Link with..." each other's MAC address (note this is the Wireless MAC address, not router's MAC address).

WDS stands for Wireless Distribution System, it enables interconnections between access points (AP). Wikipedia definition has more technical details. Ideally, using WDS we can chain more APs together, just make sure you don't keep any loop ;-)

The beautify of this is that it uses the same ssid, so when you move around, it switches to the router with the best signal strength automatically.

To test the signal strength, there is a nice free tool for Mac called AP Grapher, very handy indeed.

Saturday, March 5, 2011

wireshark no network interface issue

wireshark is one of the best tool to deep network analysis. But due to the permission issue on BPF devices on Mac OSX (by default only root can read and write BPF devices), you won't see network interfaces in wireshare on Mac ;-(

There are two ways to fix:

- run sudo chown <your_user_id> /dev/bpf* on command line before you start wireshark, you will need to do it again after reboot

- add a startup script to fix this permanently (see instructions that comes with wireshark, D&D Utilities/ChmodBPF directory in the installation package into /Library/StartupItems
. You can adjust the script for your needs as well, for details, see the documentation that comes with wireshare.

Saturday, February 26, 2011

node.js hosting in Amazon EC2

node.js has rapid growth and there are so many hosting services available. Ryan maintains a long list of providers on github already and new ones like NodeFu keeps coming out every day. I have personally tried two managed hosting services duostack and JSApp.US, both are very easy to set up and use. The only drawback is that they pose limitations on installed modules, etc.

Amazon is offering 1 year free trial for its cloud services, so called "AWS Free Usage Tier". I decided to give it a try and set up an EC2 instance for node.js.

The whole process is not quite user friendly (not as friendly as the shopping experience on Amazon). There are quite several choices for the image to choose. My first attempt with a SuSE 64bit image also got permission issues when I install gcc somehow. Finally, from the Web search results, I found Mike Leach's step-by-step guide "Installing node.js on Amazon EC2", which is really helpful. Jim Smith's blog post added a nginx as the front end, a nice addition since node.js is not multi-threading out of box.

Some minor differences when I followed Mike's steps:
  • instead of ssh to the vm using ec2-user@ec2-public-dns, I had to use ec2-user@<address you can find in AWS management console by right clicking on the instance and then clicking connect>, note that the instructions there told you to ssh as root@<address>, but it did not work for me, saying I had to use ec2-user@
  • instead of --without-ssl option to ./configure, I added openssl and openssl-devel
  • I did node and npm installation using the "node-and-npm-in-30-seconds.sh" script from this gist mentioned in the Joyeur post "Installing Node and npm"

Saturday, February 12, 2011

handy tutorials

Just found several handy tutorials/references that could quickly refresh your knowledge about the tools and get work done easily.


Again, reading a lot is not as useful as coding a lot. But if you don't use it for day-to-day work, you might easily forget many tricks. So, some quick references are handy in this situation to bring you quickly up to speed.

Saturday, February 5, 2011

what does "new" do in JavaScript

Today, a colleague asked about exactly what is the difference between using "new" and calling the function directly. Only thing I can think of is the meaning for "this" inside the function. With "new", this means the object being created, but without "new", this means the global object, either window or something else. And I am also not clear what exactly happens behind the scene.

So, did some search and found some good readings that explains this topic fairly clearly. Lots of related answers from Stack Overflow is also quite helpful:
Here is a good article that tests your understanding of "new" keyword ;-)

Now, whether or not you should use "new" is another question:

Using "new" to instantiate a class has both pros (great performance) and cons (can cause nightmare if you forget to use new). Many people including Douglas Crockford discourages the use of new. A module pattern is a more preferable way. John Resig also has a simple solution to avoid using new directly without much performance sacrifice.