Pipelining Bash Script’s output to Webapp using Socket.io

Yaydoc, an Automatic Documentation Generator by Fossasia, among other components, consists of a Web User Interface. This UI has a form that takes as its input certain information about a user’s project and generates documentations using this information in the backend with the help of a Bash Script. The caveat of executing such a Bash Script is that a user will have to wait for the processing to complete in order to get any output on the WebApp. This creates some problem as the user may not know if the process is executing properly. Furthermore, servers that are used to deploy such web applications have a limited time span within which it must send a response to a received GET or POST request. Since executing scripts may take some time, the process may lead to a Request Timeout.

We faced a similar problem with Yaydoc while deploying it to Heroku. Since Heroku has a timeout at 30 seconds, executing the Documentation Generation script lead to a Request Timeout as it takes more than 30 seconds for the execution. After doing a bit of research, we were introduced with Socket.io. Socket.IO is one of the most powerful Javascript frameworks which enables real-time bidirectional event-based communication.

At the client side, we define an “execute” event which emits the form data when the “Generate Docs” button is clicked. At the server side, we handle the event by executing a generator.executeScript(...) function with the socket and formData as its arguments.

/**
 * Client-side Event Handling
 */
$(function () {
  var socket = io();
  $(“#btnGenerate”).click(function () {
    var formData = getData();
    socket.emit(“execute”, formData);
  });
  ...
  ...
  ...
});

/**
 * Server-side Event Handling
 */
io.on(“connection”, function (socket) {
  socket.on(“execute”, function (formData) {
    generator.executeScript(socket, formData);
  });
});

Bash scripts are executed in NodeJS by creating child processes using the child_process module. This module provides four different methods for executing external applications. They are:

1. execFile
2. exec
3. spawn
4. fork

Out of these, the exec() and execFile() methods returns buffered data when the script executes successfully. We cannot use them as a solution because we need to continuously receive certain response from the server after execution of a limited number of commands in the script. Thus, we opt for spawn() which returns a stream based object every time the script produces some data. The spawn method is called in the executeScript method.

exports.executeScript = function (socket, formData) {
  ... 
  ...
  var process = spawn(“./generate.sh”, args);
  process.stdout.on(“data”, function (data) {
    socket.emit(“logs”, {data: data});
  });
  ...
  ...
};

The emitted logs are then received at the client-side for display in the web application.

/**
 * Client-side Event Handling
 */
$(function () {
  ...
  ...
  socket.on(“logs”, function (data) {
    $(“#messages”).append($(“<li>”).text(data.data));
  }
  ...
  ...
});

A minimal sample of this application can be found at: https://github.com/imujjwal96/socket-bashing