NodeJS Misc

https://www.airpair.com/node.js/posts/utilities-for-everyday-node-development
npm init
package.json
Every Node module uses a package.json file to determine what modules need to be installed in order to function properly. Use npm init to generate your own.
This file also holds (other meta information)[https://www.npmjs.org/doc/files/package.json.html], such as version, descriptions, authors, license, and script hooks. After downloading / cloning a Node project, run npm install to download all the dependencies.
Make sure to .gitignore your node_modules folder.

When using npm in non-trivial projects, append --save, e.g. npm install <module> --save, in order to add the package information to package.json automagically.

npm offers two 'locations' for packages, local and global. Local packages are installed in a project's node_modules folder, using the command npm install <modulename>. Global packages are installed using the -g flag, e.g. npm install -g <modulename>, and -- if they are designed for it -- can be installed in your environment and accessed from the command line.

npm search
modules: async and lodash.

http://shapeshed.com/command-line-utilities-with-nodejs/
console.log(process.argv)
var args = process.argv.slice(2);
if (err) {
  process.exit(1);
} else {
  process.exit(0);
}

process.stdin.resume();
process.stdin.setEncoding('utf8');
process.stdin.on('data', function(data) {
  process.stdout.write(data);
});
echo 'foo' | ./yourscript

Practical Node.js: Building Real-World Scalable Web Apps
brew install node
brew install npm

node REPL
node program.js
node -e "console.log(new Date());"
NODE_ENV=production API_KEY=442CC1FE-4333-46CE-80EE-6705A1896832node server.js

Node.js was built on top of the Google Chrome V8 engine and its ECMAScript
== performs automatic typecasting whereas === does not.

JavaScript arrays are not real arrays; instead, they are objects with unique integer (usually 0 based) keys.

Think of buffers as extremely efficient data stores.
Node.js Globals
process
global
module.exports and exports

_ and $ are perfectly legitimate characters for literals (jQuery and Underscore libraries use them a lot). Private methods and attributes start with _ (and it does nothing by itself!).
node -e "console.log(process.pid)"

To export an object in Node.js, use exports.name = object;

exports.messages = messages;
var messages = require('./routes/messages.js');

__dirname is an absolute path to the file in which the global variable is called, whereas process.cwd is an absolute path to the process that runs the script.

var http = require('http');
var fs = require('fs');
var path = require('path');
fs.readFile(path.join(__dirname, '/data/customers.csv'), {encoding: 'utf-8'},function (err, data) {
  if (err)throw err;
  console.log(data);
});

fs.writeFile('message.txt', 'Hello World!',function (err) {
  if (err)throw err;
  console.log('Writing is done.');
});

fs.createReadStream('./data/customers.csv').pipe(process.stdout);
var http = require('http');
http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(1337, '127.0.0.1');
console.log('Server running at http://127.0.0.1:1337/');

debugger;
node debug program.js

Debugging with Node Inspector
npm install -g node-inspector
node --debug program.js

Watching for File Changes
npm install -g node-dev

Express.js

npm install -g express-generator@4.0.0
package.json
npm install express@4.1.2 --save
npm ls

express -c styl express-styl
npm install
DEBUG=my-application ./bin/www


Main file app.js: settings, inclusions of routes, and other important logic. This is where we run when we start the server.
Routes: all the logic related to pages and abstracted from app.js based on functional meaning, such as getting the data from the database and compiling the data into HTML
Node.js project file package.json: dependencies and other meta data

mkdir {public,public/css,public/img,public/js,db,views,views/includes,routes}

node app.js
node app
npm start
name the entry point index.js. In this case, we get the benefit of running the script with the $ node . command.
var app = express();

app.all('*', function(req, res) {
  res.render(
    'index',
     {msg: 'Welcome to the Practical Node.js!'}
  );
});

TDD and BDD for Node.js with Mocha
npm install –g mocha@1.16.2
NodeUnit, Jasmine, Vows
BDD is a specialized version of TDD that specifies what needs to be unit-tested from the perspective of business requirements.
describe('homepage', function(){
  it('should respond to GET',function(done){
    superagent
      .get(' http://localhost:'+port )
      .end(function(res){
        expect(res.status).to.equal(200);
        done();
    })
  })
var assert = require('assert');
describe('String#split', function(){
  it('should return an array', function(){
    assert(Array.isArray('a,b,c'.split(',')));
  });
})
npm install chai@1.8.1
var assert = require('chai').assert;

Template Engines: Jade and Handlebars
Any text at the beginning of a line—by default—is interpreted as an HTML tag.
Body
  div
    h1 Practical Node.js
    p The only book most people will ever need.
  div
    footer &copy; Apress
Variables/Locals
h1= title
p= body  

div(id="content", class="main")
app.get('/collections/:collectionName', function(req, res, next) {
  req.collection.find({},{
    limit:10, sort: [['_id',-1]]
  }).toArray(function(e, results){
    if (e) return next(e)
    res.send(results)
  })
})

Session-Based Authentication
var cookieParser = require('cookie-parser');
var session = require('express-session');
...
app.use(cookieParser());
app.use(session());

app.post('/login', function(req, res, next) {
  // This function checks for credentials
  // passed in the request's payload
  if (checkForCredentials(req)) {
    req.session.auth = true;
    res.redirect('/dashboard');  // Private resource
  } else {
    res.send(401); // Not authorized
  }
});

app.all('/api', authorize);
app.get('/api/articles', routes.article.list)

exports.login = function(req, res, next) {
  res.render('login');
};
exports.logout = function(req, res, next) {
  req.session.destroy();
  res.redirect('/');
};
exports.authenticate = function(req, res, next) {
  if (!req.body.email || !req.body.password)
    return res.render('login', {
      error: 'Please enter your email and password.'
    });
  req.collections.users.findOne({
    email: req.body.email,
    password: req.body.password
  }, function(error, user){
    if (error) return next(error);
    if (!user) return res.render('login', {
      error: 'Incorrect email&password combination.'
    });
    req.session.user = user;
    req.session.admin = user.admin;
    res.redirect('/admin');
  })
};

Building Node.js REST API Servers with Express.js and Hapi
Real-Time Apps with WebSocket, Socket.IO, and DerbyJS
WebSocket is a special communication “channel” between browsers (clients) and servers. It’s an HTML5 protocol. WebSocket’s connection is constant, in contrast to traditional HTTP requests

var ws = new WebSocket('ws://localhost:3000');
ws.onopen = function(event) {
ws.send('front-end message: ABC');
};
ws.onmessage = function(event) {
console.log('server message: ', event.data);
};

var WebSocketServer = require('ws').Server,
  wss = new WebSocketServer({port: 3000});

wss.on('connection', function(ws) {
    ws.send('XYZ');
    ws.on('message', function(message) {
        console.log('received: %s', message);
    });
});

var server = http.createServer(app);
var io = require('socket.io').listen(server);

io.sockets.on('connection', function (socket) {
  socket.on('messageChange', function (data) {
    console.log(data);
    socket.emit('receive',
      data.message.split('').reverse().join('')
    );
  });
});

app.set('port', process.env.PORT || 3000);
server.listen(app.get('port'), function(){
  console.log('Express server listening on port ' +  
    app.get('port')
  );
});

Deploying Node.js
Node makes a single thread more efficient by delegating many blocking operations to OS subsystems to process, bothering the main V8 thread only when there is data available for use.

"In event-driven programming, an application expresses interest in certain events and responds to them when they occur. The responsibility of gathering events from the operating system or monitoring other sources of events is handled by libuv, and the user can register callbacks to be invoked when an event occurs."

The event loop executes on a single thread but delegates I/O operations to libuv, which manages a thread pool that parallelizes these operations, notifying the event loop when results are available.

Making HTTP requests
var http = require('http');

http.request({
  host: 'www.google.com',
  method: 'GET',
  path: "/"
}, function(response) {
  response.setEncoding('utf8');
  response.on('readable', function() {
    console.log(response.read())
  });
}).end();

http.get("http://www.google.com/", function(response) {
  console.log("Status: " + response.statusCode);
}).on('error', function(err) {
  console.log("Error: " + err.message);
});

Redis
Also note that we must wait for the subscribe event to be fired on the subscriber client prior to publishing any messages. Redis does not hold a queue of published messages, which involves waiting for subscribers. A message for which there are no subscribers is simply dropped.
var redis = require("redis");

var publisher = redis.createClient();
var subscriber = redis.createClient();

subscriber.subscribe('channel5');

subscriber.on('message', function(channel, message) {
  console.log('channel: ', channel)
  console.log('message: ', message)
})

subscriber.on('subscribe', function() {
  publisher.publish('channel5', 'This is a message')
})

Logging
Morgan is an HTTP logger for the Express framework.
var morgan = require('morgan')
app.use(morgan('combined'))
app.use(morgan('combined', {
  stream : require('fs').createWriteStream('./out.log')
}));
skip: function(req, res) {
  return res.statusCode < 400;
}

Application/Examples
Real Time Chat With NodeJS, Socket.io and ExpressJS

NPMs

• express - Express.js, a Sinatra-inspired web development framework for Node.js, and the de-facto standard for the majority of Node.js applications out there today.
• connect - Connect is an extensible HTTP server framework for Node.js, providing a collection of high performance “plugins” known as middleware; serves as a base foundation for Express.
• socket.io and sockjs - Server-side component of the two most common websockets components out there today.
• Jade - One of the popular templating engines, inspired by HAML, a default in Express.js.
• mongo and mongojs - MongoDB wrappers to provide the API for MongoDB object databases in Node.js.
• redis - Redis client library.
• coffee-script - CoffeeScript compiler that allows developers to write their Node.js programs using Coffee.
• underscore (lodash, lazy) - The most popular utility library in JavaScript, packaged to be used with Node.js, as well as its two counterparts, which promise better performance by taking a slightly different implementation approach.
• forever - Probably the most common utility for ensuring that a given node script runs continuously. Keeps your Node.js process up in production in the face of any unexpected failures.

Instant Node Package Module

node.js can be used to:

Start an http/https server

Make http/https requests to other web servers

Start the TCP/UDP server or to make TCP/UDP requests

Read and write content on the filesystem

Spawn and manage new processes on operating system

The runtime of node.js is powered by Google's V8 JavaScript engine, whose task is to compile and run JavaScript code. 

evented I/O:

In node.js, during an I/O operation, whenever a chunk of data is read or written, an event is emitted. The programmer needs to register callbacks to react to these events, for example, to aggregate the chunks being read, or to determine whether the operation has been completed. This allows for making the I/O operations asynchronous, thus saving CPU cycles from being wasted. Other platforms make use of CPU time during I/O operations through multithreading. But an event-driven approach is much more efficient, because there is no thread spawning overhead and the memory footprint is much smaller.

event loop is constantly asking if there are there any more events to process. Whenever an event occurs, it gets registered into an event queue. The event loop executes the event handlers in the queue one by one. If there are no more events to process, the node.js runtime exits; of course, that is not true if you have started a server, in that case it will wait for events to occur.

To translate I/O operation into events, node.js uses libuv. libuv is a C++ library that provides infrastructure for evented I/O. The actual magic happens in the libraries written on top of libuv, node.js itself is just a thin JavaScript wrapper around these.

modularization and redistribution of code:

npm allows us to register our packages with a name so that we can import/export these packages using the registered name. Moreover, npm also integrates with the npm registry. The npm registry is a web service that hosts such packages.

creating your first npm module

module.exports = sum;

We can assign any entity to it, and we will be able to access that entity from outside that file. 

var someVar = require('./sum.js');

someVar(1,2)

Anything that we assigned to module.exports is returned by the require call.

Another use of require is to load the core modules or native modules

there is no leading slash or dot for loading a core module or any external module for that matter.

if two modules have the same name, the core module will always trump the external module.

require cache

Whenever a file or module is required, the result obtained is cached. Subsequent require calls to the same file/module will serve the results from the cache.

Node.js By Example
var fs = require('fs');
var fs = require('fs');
fs.writeFile('data.txt', 'Hello world!', function (err) {
   if(err) { throw err; }
   console.log('It is saved!');
});

fs.writeFileSync('data.txt', 'Hello world!');
fs.readFile('data.txt', function(err, data) {
   if (err) throw err;
   console.log(data.toString());
});

var events = require('events');
var eventEmitter = new events.EventEmitter();
var somethingHappen = function() {
   console.log('Something happen!');
}
eventEmitter
.on('something-happen', somethingHappen)
.emit('something-happen');

var util = require("util");
var events = require("events");
var Class = function() { };
util.inherits(Class, events.EventEmitter);
Class.prototype.ratePoints = 0;
Class.prototype.rate = function(points) {
   ratePoints = points;
   this.emit('rated');
};
Class.prototype.getPoints = function() {
   return ratePoints;
}
module.exports = Class;

var BookClass = require('./book.js');
var book = new BookClass();
book.on('rated', function() {
   console.log('Rated with ' + book.getPoints());
});
book.rate(10);

Managing child processes
var exec = require('child_process').exec;
exec('ls -l', function(error, stdout, stderr) {
    console.log('stdout: ' + stdout);
    console.log('stderr: ' + stderr);
    if (error !== null) {
        console.log('exec error: ' + error);
    }
});

var spawn = require('child_process').spawn;
var command = spawn('git', ['push', 'origin', 'master']);
command.stdout.on('data', function (data) {
   console.log('stdout: ' + data);
});
command.stderr.on('data', function (data) {
   console.log('stderr: ' + data);
});
command.on('close', function (code) {
   console.log('child process exited with code ' + code);
});

var spawn = require('child_process').spawn;
var command = spawn('git', ['push', 'origin', 'master']);
command.stdout.on('data', function (data) {
   console.log('stdout: ' + data);
});
command.stderr.on('data', function (data) {
   console.log('stderr: ' + data);
});
command.on('close', function (code) {
   console.log('child process exited with code ' + code);
});

The task runner and building system - grunt
npm install -g grunt-cli
Gruntfile.js
grunt-contrib-watch: looks out for file changes and runs our tasks
module.exports = function(grunt) {
   grunt.initConfig({
      concat: {
         javascript: {
            src: 'src/**/*.js',
            dest: 'build/scripts.js'
         }
      },
      uglify: {
         javascript: {
            files: {
               'build/scripts.min.js': '<%= concat.javascript.dest %>'
            }
         }
      },
      watch: {
         javascript: {
            files: ['<%= concat.javascript.src %>'],
            tasks: ['concat', 'uglify']
         }
      }
   });
   grunt.loadNpmTasks('grunt-contrib-concat');
   grunt.loadNpmTasks('grunt-contrib-uglify');
   grunt.loadNpmTasks('grunt-contrib-watch');
   grunt.registerTask('default', ['concat', 'uglify', 'watch']);
}

npm install grunt-contrib-watch grunt-contrib-uglify –save. The command will install the modules and will update the package.json file.

Gulp - gulpfile.js
Gulp is a streaming-based tool. It doesn't store anything on the disc when it is working. Grunt needs to create temporary files in order to pass data from one task to another, but Gulp keeps the data in the memory.
Gulp follows the code-over-configuration principle. In the gulpfile.js file, we write our tasks like a regular Node.js script.
var gulp = require('gulp');
var concat = require('gulp-concat');
var uglify = require('gulp-uglify');
var rename = require('gulp-rename');

gulp.task('js', function() {
   gulp.src('./src/**/*.js')
   .pipe(concat('scripts.js'))
   .pipe(gulp.dest('./build/'))
   .pipe(rename({suffix: '.min'}))
   .pipe(uglify())
   .pipe(gulp.dest('./build/'))
});
gulp.task('watchers', function() {
   gulp.watch('src/**/*.js', ['js']);
});
gulp.task('default', ['js', 'watchers']);

npm install mocha -g

Modules

var net = require('net');

require('node-router').

Examples

http://howtonode.org/hello-node

var net = require('net');

// Creates a new TCP server. The handler argument is automatically set as a listener for the 'connection' event

var server = net.createServer(function (socket) {

  // Every time someone connects, tell them hello and then close the connection.

  console.log("Connection from " + socket.remoteAddress);

  socket.end("Hello World\n");

});

// Fire up the server bound to port 7000 on localhost

server.listen(7000, "localhost");

var server = require('node-router').getServer();

// Configure our HTTP server to respond with Hello World the root request

server.get("/", function (request, response) {

  response.simpleText(200, "Hello World!");

});

When and why to use Nodejs
http://www.toptal.com/nodejs/why-the-hell-would-i-use-node-js
http://www.javaworld.com/article/2104480/java-web-development/why-use-node-js.html
to create real-time websites with push capability,

Node.js shines in real-time web applications employing push technology over websockets.
we finally have web applications with real-time, two-way connections, where both the client and server can initiate communication, allowing them to exchange data freely.

You definitely don’t want to use Node.js for CPU-intensive operations; in fact, using it for heavy computation will annul nearly all of its advantages. Where Node really shines is in building fast, scalable network applications, as it’s capable of handling a huge number of simultaneous connections with high throughput, which equates to high scalability.

 Node.js operates on a single-thread, using non-blocking I/O calls, allowing it to support tens of thousands of concurrent connections

 The technique used to avoid exceptions bubbling up to the surface is passing errors back to the caller as callback parameters (instead of throwing them, like in other environments).

How to install latest version of Node using Brew

Run commands below, in this order:

brew update
brew doctor
brew upgrade node

Now you have installed updated version of node, and it's probably not linked. If it's not, then just type: brew link node or brew link --override node

echo "node_modules" > .gitignore

var Http = require( 'http' );
 
var server = Http.createServer( );
server.listen( 8080, function( ) {
    console.log( 'Listening on port 8080' );
});

var count = 0;
function requestHandler( request, response ) {
    var message;
    count += 1;
    response.writeHead( 201, {
        'Content-Type': 'text/plain'
    });
    message = 'Visitor count: ' + count + ', path: ' + request.url;

    console.log( message );
    response.end( message );
}
var server = Http.createServer( requestHandler );
npm install router

router = new Router( );

server = Http.createServer( function( request, response ) {
    router( request, response, function( error ) {
        if( !error ) {
            response.writeHead( 404 );
        } else {
            //Handle errors
            console.log( error.message, error.stack );
            response.writeHead( 400 );
        }    
        response.end( '\n' );
    });
});
 
server.listen( 8080, function( ) {
    console.log( 'Listening on port 8080' );
});


var counter = 0,
    messages = { };
function createMessage( request, response ) {
    var id = counter += 1;
    console.log( 'Create message', id );
    response.writeHead( 201, {
        'Content-Type': 'text/plain'
    });
    response.end( 'Message ' + id );
}
router.post( '/message', createMessage );

router.use( function( request, response, next ) {
    console.log( 'middleware executed' );
    // Null as there were no errors
    // If there was an error then we could call `next( error );`
    next( null );
});
npm install body-parser

var BodyParser = require( 'body-parser' );
router.use( BodyParser.text( ) );

function createMessage( request, response ) {
    var id = counter += 1,
        message = request.body;
     
    console.log( 'Create message', id, message );
    messages[ id ] = message;
    response.writeHead( 201, {
        'Content-Type': 'text/plain',
        'Location': '/message/' + id
    });
    response.end( message );
}
function readMessage( request, response ) {
    var id = request.params.id,
        message = messages[ id ];
    console.log( 'Read message', id, message );
 
    response.writeHead( 200, {
        'Content-Type': 'text/plain'
    });
    response.end( message );
}
router.get( '/message/:id', readMessage );
function deleteMessage( request, response ) {
    var id = request.params.id;

    console.log( 'Delete message', id );

    messages[ id ] = undefined;

    response.writeHead( 204, { } );

    response.end( '' );
}

router.delete( '/message/:id', deleteMessage )
npm install express body-parser passport passport-local

NPM
Initialize NPM on a new project
https://nodesource.com/blog/your-first-nodejs-package/

module.exports = function(width, height) {  
  return width * height;
};

function(__dirname, __filename, module, exports, require) {  
// ...
}

Where __dirname and __filename are the current directory and filename of the module being loaded.

require() is a function that will synchronously load another package or module from the filesystem and return it. The package or module will be cached when first loaded, so subsequent calls to require() for that package or module in your Node.js application will not need to re-read the file.

module.exports provides a way for modules to expose functionality

https://www.codefellows.org/blog/create-a-node-js-project-from-scratch-with-node-sass

node index.js

var chalk = require( "chalk" );
console.log( chalk.blue( "I am blue text!" ) )
npm install --save chalk

https://code.visualstudio.com/Docs/runtimes/nodejs

npm install -g express-generator

To install all of the dependencies, execute: cd myExpressApp npm install




npm start
VS Code uses the TypeScript compiler to drive its JavaScript language service, which means we can take advantage of what the compiler can infer about your code.

The TypeScript Definition Manager (TSD) makes it easy to search for and install TypeScript definition files into your workspace. This tool will download the requested definition from the DefinitelyTyped repository. As we did with the express generator, we will install TSD globally using NPM so that you can use the tool in any application you create.

npm install -g tsd
tsd query node --action install
tsd query express --action install
Press F8 to navigate errors and warnings within a file.
http://www.ruanyifeng.com/blog/2016/01/npm-install.html

npm update命令怎么知道每个模块的最新版本呢？

答案是 npm 模块仓库提供了一个查询服务，叫做 registry 。以 npmjs.org 为例，它的查询服务网址是 https://registry.npmjs.org/ 。

这个网址后面跟上模块名，就会得到一个 JSON 对象，里面是该模块所有版本的信息。比如，访问 https://registry.npmjs.org/react，就会看到 react 模块所有版本的信息。

它跟下面命令的效果是一样的。

$ npm view react

# npm view 的别名
$ npm info react
$ npm show react
$ npm v react

registry 网址的模块名后面，还可以跟上版本号或者标签，用来查询某个具体版本的信息。比如， 访问 https://registry.npmjs.org/react/v0.14.6 ，就可以看到 React 的 0.14.6 版。

返回的 JSON 对象里面，有一个dist.tarball属性，是该版本压缩包的网址。

dist: {
  shasum: '2a57c2cf8747b483759ad8de0fa47fb0c5cf5c6a',
  tarball: 'http://registry.npmjs.org/react/-/react-0.14.6.tgz' 
},

到这个网址下载压缩包，在本地解压，就得到了模块的源码。npm install和npm update命令，都是通过这种方式安装模块的。

这个缓存目录，在 Linux 或 Mac 默认是用户主目录下的.npm目录，在 Windows 默认是%AppData%/npm-cache。通过配置命令，可以查看这个目录的具体位置。
$ npm config get cache
$ ls ~/.npm
# 或者
$ npm cache ls
npm cache ls react

发出npm install命令
npm 向 registry 查询模块压缩包的网址
下载压缩包，存放在~/.npm目录
解压压缩包到当前项目的node_modules目录

注意，一个模块安装以后，本地其实保存了两份。一份是~/.npm目录下的压缩包，另一份是node_modules目录下解压后的代码。
但是，运行npm install的时候，只会检查node_modules目录，而不会检查~/.npm目录。也就是说，如果一个模块在～/.npm下有压缩包，但是没有安装在node_modules目录中，npm 依然会从远程仓库下载一次新的压缩包。

这种行为固然可以保证总是取得最新的代码，但有时并不是我们想要的。最大的问题是，它会极大地影响安装速度。即使某个模块的压缩包就在缓存目录中，也要去远程仓库下载，这怎么可能不慢呢？
另外，有些场合没有网络（比如飞机上），但是你想安装的模块，明明就在缓存目录之中，这时也无法安装。

为了解决这些问题，npm 提供了一个--cache-min参数，用于从缓存目录安装模块。
--cache-min参数指定一个时间（单位为分钟），只有超过这个时间的模块，才会从 registry 下载。

$ npm install --cache-min 9999999 <package-name>
上面命令指定，只有超过999999分钟的模块，才从 registry 下载。实际上就是指定，所有模块都从缓存安装，这样就大大加快了下载速度。
它还有另一种写法。

$ npm install --cache-min Infinity <package-name>
但是，这并不等于离线模式，这时仍然需要网络连接。因为现在的--cache-min实现有一些问题。
（1）如果指定模块不在缓存目录，那么 npm 会连接 registry，下载最新版本。这没有问题，但是如果指定模块在缓存目录之中，npm 也会连接 registry，发出指定模块的 etag ，服务器返回状态码304，表示不需要重新下载压缩包。
（2）如果某个模块已经在缓存之中，但是版本低于要求，npm会直接报错，而不是去 registry 下载最新版本。
npm 团队知道存在这些问题，正在重写 cache。并且，将来会提供一个--offline参数，使得 npm 可以在离线情况下使用。
不过，这些改进没有日程表。所以，当前使用--cache-min改进安装速度，是有问题的。