WebSockets Gateway and JS Library

Testing with Evergreen:

There is a lot of JS code in Evergreen (dojo libs and UI code) which assume synchronous communication, which does not work with Websockets, as noted above. Because of this, it's not possible to use the OpenSRF WebSocket JS libs as a drop-in replacement for Evergreen.

Testing for now has to happen a lower level, using OpenSRF JS directly or, for the adventurous.. The browser-based staff client prototype code assumes all async and I have confirmed that it works fine with WebSockets.

Another TODO:

Since we're using Apache, we can prevent new WS connections on a given server the same as we do with gateway/translator connections, via ldirector (etc.) ping files. However, we also need a way to prevent existing clients from sending new requests when an apache server wants to disconnect.

Related, we also need to add support for a timeout-based disconnects. Forcing clients to disconnect after a period of inactivity ensures Apache processes are given a chance to clear the Apache memory pool for the request and, more importantly, will allow the Apache process count to settle down after periods of high activity. In other words, if your browser is sitting idle for an hour, there is no reason it should hog an Apache process.

The client code is already capable to detecting a broken socket and reconnecting. Whats' more, the reconnect process is roughly equivalent to a single XMLHttpRequest, so it's not something the user should generally notice. So, we only need to the server pieces to set a timeout (via apache config) and send a disconnect to the client when the timeout is reached. With this, we can also support graceful shutdown by setting the timeout temporarily very low (say, via signal), which will cause a disconnect of all clients while they are idle, forcing them to reconnect to another server.

Pushed code to do the following:

* Added support for a configurable inactivity timeout. After each message, inbound and outbound, are processed, we set a last activity time. A separate thread wakes periodically to see if the last activity time occurred beyond the timeout interval. After a period of inactivity, we send a disconnect to the client. This allows the websocket handler to exit and return control to Apache.

* How often we wake to check for inactivity is also configurable.

* Using the inactivity logic, if we receive a SIGUSR1 -- the Apache graceful / reload signal -- a shortened inactivity timeout is applied so that the client can be disconnected as soon as all open conversations are completed.

* Apache configuration for websockets lives in /etc/apache2-websockets/envvars. I put them there because standard Apache configuration directives cannot be created, since our websocket module is not an Apache module proper, but a shared library loaded by an apache module. Also, for reasons that are not clear to me, SetEnv and SetEnvIf failed to pass the environment to our websocket module. Exporting them in envvars works fine, though, and it means we don't have to enable either of the env mods, which would add to the memory requirements. (TODO: add example to repo)

* Javascript client lib now performs the connect dance under the covers, no need to manually initialize.

* When a JS client is disconnected from the server, it will reconnect when needed -- when the next request is sent -- and not automatically, so that we are not unnecessarily hogging Apache processes.

Pushed settings info to installer (readme.websockets) docs.

General update:

The Apache gateway code is nominally done, but needs a whole lot of testing, poking, prodding, and sharp stick pointing.

The JS bits still need work to get the multi-tab (shared web worker) and non multi-tab versions in sync. Ditto testing.

Installer process and docs need more work.

* overhauled the apache config to use a single, stripped-down apache2.conf instead of wrestling with disabling mods, ports.conf, etc. Just copy the file into place and restart.

* Apache setup docs are considerably simpler now. It's practically scriptable at this point.

* removed (at least for now) the trailing-behind single-tab websocket JS client implementation. maintaining 2 different implementations will complicate things and we probably don't want multi-tab clients to have the ability to (trivially) open numerous WS connections, anyway. Single connection (via SharedWorkers) across all tabs is the future.

Regarding SharedWorkers and Firefox. Current stable FF (version 28) requires SharedWorkers to be turned on in about: config (search for "shared"). It will be officially supported (and on by default) in FF version 29.

I believe we're ready for some brave soul to test it out. Here's a simple HTML test.

-------------
<html>
  <head>
    <script src='/js/dojo/opensrf/JSON_v1.js'></script>
    <script src='/js/dojo/opensrf/opensrf.js'></script>
    <script>
      OpenSRF.Session.transport = OSRF_TRANSPORT_TYPE_WS_SHARED;
        var ses = new OpenSRF.ClientSession('open-ils.actor'); // Evergreen
        ses.request({
          method : 'opensrf.system.echo',
          params : ['hello', 'websockets'],
          onresponse : function(r) {
            alert('onresponse() => "' + r.recv().content() + '"');
          },
          oncomplete : function() {
            alert('oncomplete()');
          }
      }).send();
    </script>
  <head>
  <body><h1>If you get alerts, you win!</h1></body>
<html>
----------------

The only thing special about this test is the OSRF_TRANSPORT_TYPE_WS_SHARED bit (and that it's all async).

Note that we don't have to load the websocket JS, because it's loaded as a shared worker from within opensrf.js when activated.

To recap:

1. Install the OpenSRF WebSockets branch (collab/berick/websockets) like normal.
2. Follow the directions in README.websockets (as root)
3. Put the test script above on your server and see if it works IN CHROME (or Opera; see notes above about FF).
4. Profit

Just a quick note: the closing </head> and </html> tags in the sample HTML above are missing the '/'. Not sure it makes a difference but here's what I used:

-------------
<html>
  <head>
    <script src='/js/dojo/opensrf/JSON_v1.js'></script>
    <script src='/js/dojo/opensrf/opensrf.js'></script>
    <script>
      OpenSRF.Session.transport = OSRF_TRANSPORT_TYPE_WS_SHARED;
        var ses = new OpenSRF.ClientSession('open-ils.actor'); // Evergreen
        ses.request({
          method : 'opensrf.system.echo',
          params : ['hello', 'websockets'],
          onresponse : function(r) {
            alert('onresponse() => "' + r.recv().content() + '"');
          },
          oncomplete : function() {
            alert('oncomplete()');
          }
      }).send();
    </script>
  </head>
  <body><h1>If you get alerts, you win!</h1></body>
</html>
----------------

I've tested and pushed to master (with some squashing). Thanks, Bill!