Offline mode for the Web Client

Here is the branch:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/miker/lp-1706107-offline-mode

There is a second commit that is important for avoiding some classes of bugs, but is duplicated in the serials branch from which offline has been extracted.

I'm getting a bunch of failed karma unit tests

egOrg:
✗ should provide get by ID
✗ should provide get by node
✗ should provide ancestors
✗ should provide descendants
✗ should provide full path
✗ should provide root

egHomeControllers
✗ should focus the login controller

More detailed output is available at https://pastebin.com/

Err...that output would be at https://pastebin.com/GqjVVEkj

I found another issue that I thought I checked for when previously testing offline circ, but it's behaving differently now.

I successfully performed several transactions while offline. When back online, I logged into the client, and then went to the Offline Circulation by selecting this option from the Circulation Menu.

The offline checkout interface is retrieved along with the warning telling me that I am logged in and, if I proceed, I will be logged out. Of course, if I click Proceed, I will indeed be logged out and unable to upload transactions.

Previously, when I selected that option from the Circulation menu, it would bring me directly to the Session tab where I don't see that warning. I'm not sure why it doesn't bring me there now. The link from the menu item is https://mlnc1.noblenet.org/eg/staff/offline-interface/session, which I expect would bring me directly to that interface.

On your second point, indeed, it should be bringing you to the session interface. I haven't looked at that yet, but will.

The test failures are because the mock environment needs to load lovefield but aren't currently. I've pushed a commit to address that.

Thanks Mike! I haven't updated the branch yet with the fixes for the tests, but I did have problems when checking for more 404s. I'll outline my steps to make sure I tested it correctly.

1. I visited the required pages and successfully retrieved the offline interface.
2. I cleared the browser cached images and files by using More Tools -> Clear browsing data in Chrome.
3. In dev tools, I selected the option to disable the cache.
4. I performed a hard refresh in the offline interface, which, as expected, brings up the 'page cannot be found' page.
5. I then performed a standard refresh, but only got a blank page. I didn't even get the staff client menus.

I've attached the output from my Console.

Kathy,

What did you select from the list in Clear browsing data?

Thanks!

Cached images and files.

Kathy, do you see the same behavior on the pre-existing test server? I extracted the offline code from the serials branch, and there's a non-zero chance that teasing them apart left something in a bad way.

I took a look directly at the mlnc1 server, and it looks like there are three files referenced by the code that are not installed. One looks like an oversight from the status bar removal, and I'll push a fix for that in a moment, but the other two (ngToast.min.css and ngToast-animations.min.css) seem to have not been installed properly. The grunt build system does seem to put them in the right place on my server ... could your install have lacked some of the grunt commands, or encountered an error during the grunt copy phase?

Hi Mike,

I do not see the same behavior on the pre-existing test server. In fact, I was pleased to see if loaded properly while offline after not having visited it for a couple of weeks.

The only errors I recall seeing after running the grunt command were the karma unit tests I mentioned above. I can try reinstalling and get back to you, hopefully sometime later today.

Above-referenced update is pushed now.

And, I've added a live-test to make sure, as well as possible, that all the required offline assets are available.

Hi Mike,

I just built a VM with latest master and the latest updates to your branch.

When I run grunt all, I'm still getting the exact same karma unit test failures that I outlined in comment #2.

I see no other errors when I run the grunt all command. However, the ngToast.min.css and ngToast-animations.min.css files are not installed. In fact, there is no node_modules directory in /openils/var/web/js/ui/default/staff as I see on my other servers.

If I follow the same steps to build the web client on a VM without the offline branch, those files copy over to /openils/var/web/js/ui/default/staff/node_modules as expected.

I'm curious about the choice of UpUp instead of using Workbox (https://workboxjs.org/). Workbox development is led by Google--along with Mozilla, one of the main proponents of service workers--and is the more modular successor to the sw-helper and sw-precache libraries that I based my work on for my session at the Evergreen 2017 conference.

UpUp, by comparison, appears to be a single-person project and I've had no success finding a list of projects that use it or any real community around it, which potentially leaves our project exposed to the whims of a single dev. Compare the stats at https://www.npmjs.com/package/upup vs. https://www.npmjs.com/package/workbox-sw

Another concern is that this branch bundles the two minified UpUp JS files + license--but don't we generally prefer using npm to bundling these days (particularly for minified assets)? Both Workbox and UpUp are available via npm; with Workbox, the service worker would be generated at build time based on the routing config (similar to what's defined in UpUp's config).

Another concern is that UpUp is being given the entire service-worker scope, but is very limited in what kinds of different offline strategies it can support; for example, Workbox runtime caching supports both network-first and cache-first strategies (https://workboxjs.org/examples/workbox-runtime-caching/index.html), and also supports per-file cache invalidation, whereas UpUp appears to implement only a cache-first strategy and invalidates the entire set of cached files based only on the cache-version parameter (daily based on this branch). These sorts of limitations will impact what we can do with service workers for other parts of Evergreen.

Kathy,

Arg, that's very frustrating ... I'm going to try to enlist help from folks more experienced with karma than I am.

More as the story develops.

Dan,

I tried to explain the reason behind using UpUp in IRC; I'm sorry I failed at that.

My decision to use UpUp instead of Workbox (or any other massive framework) came down to complexity. For what we need to accomplish in offline mode, the simplest possible thing that could work is preferable, IMO. All we need is to make sure that the HTML, CSS, and JS are available in a local cache that is not subject to the whims of a user that wants to clear the normal browser cache at arbitrary times. Using UpUp for what we need was (for me) an order of magnitude simpler than Workbox, based on the examples. Put plainly, using UpUp is exactly one self-explanatory JS command in one place.

We can change to loading it via npm, but I included it directly for a specific reason. UpUp currently uses the stable ServiceWorker API, and it works. For an interface that will be tested just about the least in practice, and used the most by staff that are under stress, I judged it wiser that the code should be as stable and predictable as possible. Based on that logic I decided that pinning it was best. I'm open to debate on that.

Now, with all that said, I have to ask: do we as a project have plans to add some sort of public offline functionality? I'm honestly not sure what the point of that would be, unless we wanted to rewrite the TPAC as a modern "app", optimize interfaces with SW caching, and go back to the conceptual model we had with the JSPAC of fetching the data with APIs -- basically, extend what we've done with the staff client to the OPAC. If we were to do that then whatever full SW framework we use can be a drop-in replacement for UpUp. But, until then, the time investment isn't (IMO) worth it -- there's no practical benefit.

If someone wants to put in the time and testing to replace UpUp, I certainly won't stop them, but I'd consider that wasted tuits, personally.

Bill Erickson spotted my thinko in the karma fix I posted before. Branch updated with his fix.

Thanks, Bill!

Yes, my presentation clearly talked about separating out the "My Account" section of the public catalogue to support users being able to see the status of their holds, fines, reading history, bookbags, etc when disconnected, instead of requiring a separate native app on mobile devices. And I believe that proposal and the progress I had demonstrated was positively received.

There is more than just a service worker required to make this happen--a usable Web API would help--so it's unlikely to occur in the 3.0 cycle. But some of the building blocks are coming together in the 3.0 cycle. It would be a shame to have to reimplement the service worker in, say, the 3.1 cycle.

Dan,

I'm sorry I couldn't make it to your presentation, but I'm glad you had a positive response.

Since, as you intimated and I understand it to be true, Workbox could be used as a drop-in replacement for UpUp (after the attendant boilerplate and infrastructure addition, and whatever learning curve is involved), I suggest we proceed with my existing, pullrequested branch, and when you (or anyone with the time to dig through your presentation) have an enhancement for My Account ready to offer, that code can and should take over caching duties from UpUp.

IOW, please do feel free to rip UpUp out, root and branch, when you have something that can replace it ready for offer. If that's tomorrow (or any time before before Galen calls a halt), great! In fact, even if it's just a replacement for UpUp in offline context, I'll welcome it.

As for "reimplementing" (replacing) the service worker in 3.1, if I take your meaning, that's my loss, not yours, and I consider it a loss of three effective lines of code which you can effectively pretend are simply not there. I realize that I'm not giving you the infrastructure to build your My Account work on for free, but I simply didn't have the time to invest. So, really, I can't see how it's a big loss or impediment to either of us. I won't consider replacing UpUp a shame, I'll consider it progress. UpUp got me to working offline code in the time available for inclusion in 3.0, which was my goal. If something better supersedes it in the future (or immediately), that'll be great.

If you need my assistance getting offline caching into a new, more comprehensive framework, you'll have it. I suspect, though, that making sure a list of assets are available is something you'll already be working on, so adding a few files to the list probably won't be overly burdensome.

Also, thank you for your early push toward service workers for offline. It was very helpful!

Unfortunately, I had technical difficulties with my test environment for most of this week, and I am going to be unavailable to test this again until August 23. If somebody else has the tuits to look at this branch in the meantime, please do. Otherwise, it's on the top of my list for testing (along with bug 1698206) when I get back.

Thanks Mike! Given my limited time, it's not realistic for me to pull together a different branch at this point in the release cycle.

Let's agree to revisit for 3.1. At that point, we can break out separate service worker scopes and (likely aliased) asset subdirectories for /eg/staff, /eg/opac, and a new /eg/account or the like.

Perhaps we will also be able to bring web push notifications (https://developer.mozilla.org/en-US/docs/Web/API/Push_API) into play in the appropriate scopes--instead of having to check your account on your phone for those items that are overdue / on hold, why not have a notification on your phone or browser just tell you? But at this point I'm veering far away from the scope of this particular feature.

BTW I meant to say that if UpUp tests out for 3.0 it should certainly go forward!

Resolved conflicts in two files and rebased Mike's branch against current master.

New branch is available at http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/kmlussier/lp-1706107-offline-mode-rebased

Hi Mike,

I installed the above branch. Karma tests are passing, and I now appear to be getting all the goodies in the node_modules directory. However, I'm still having trouble with this branch. The most critical are #1 and #2.

1. I'm no longer getting any offline access after I clear my browser cache (More tools -> clear browsing data -> Cached images and files. Offline worked fine on my first attempt, but after I cleared the browser cache and disabled cache via the Chrome developer tools, I received a disconnected from the Internet error (the screen with the dinosaur game). In my previous tests before today, the browser was trying to load something, but was missing the required files. Now, the browser doesn't seem to be trying to load anything at all while offline once the browser cache is cleared.

2. When I am online and want to upload transactions, I'm unable to create a session. I hit the Create Session button, and I get the toast notification saying 'offline session creation failed.' I get the following message in my Console:
(unknown) GET https://mlnc1.noblenet.org/cgi-bin/offline/offline.pl?1503950675442&action=create&desc=aug-28-17-offline-testing&org=4&ws=BR1-masslnc&wc=1&ses=0ca06797d5a98db3ae43276e7b48f116 500 (Internal Server Error)
(anonymous)
(anonymous) @ angular.min.js:105
n @ angular.min.js:100
(anonymous) @ angular.min.js:98
(anonymous) @ angular.min.js:132
$eval @ angular.min.js:147
$digest @ angular.min.js:144
$apply @ angular.min.js:147
(anonymous) @ angular.min.js:281
dispatch @ jquery.min.js:3
r.handle @ jquery.min.js:3

3. I mentioned this in a previous comment a few weeks ago. The Offline Circulation entry in the Circ menu appears to be trying to direct me to the Sessions tab of the offline interface, but is, in fact, retrieving the Checkout tab by default. This is confusing for users since this tab also has a big alert informing them they should log out, which, of course, is not a good idea if you're trying to upload sessions.

4. Related to the above, when you are in offline mode and use the check out and check in entries from the standard online Circulation menu, it appears to be trying to retrieve those tabs in the offline interface. It is successful in retrieving those interfaces, but the tab stays selected at whatever tab you were using before you used the menu entry. Even if it worked as expected, I don't know that we want those menu entries to retrieve offline interfaces. When the system comes online again, if the user decides to use one of those menu entries to return to the live system, they will keep returning to that offline interface.

I took the following screencast to try to demonstrate the issues in #4. At the beginning of the screencast, we're offline and I'm showing how the tab selection doesn't change. When we go back online, those menu entries continue to direct the user to the offline interface. https://drive.google.com/file/d/0B74gDMUDwDXqeGdVaWN1dE9sLVU/view

5. I don't know if this is a problem, but after my first login to the web client, every time I return to the web client's home page, I see the following in my Console. Is this a problem? I saw it on the Equinox offline test server too:
...

I also get the following when I try running the live test:

Semicolon seems to be missing at 24-offline-all-assets.t line 5.
syntax error at 24-offline-all-assets.t line 5, near "= ="
Execution of 24-offline-all-assets.t aborted due to compilation errors.
# Looks like your test exited with 255 before it could output anything.

Thanks, Kathy!

Going through each of these:

1) This was because new global JS was added to base_js.tt2, but not added to the asset list for UpUp. Specifically, the moment.js files. Branch below adds those.

2) This is likely a local offline CGI script issue.

3) I've addressed this by being more heavy-handed about which tab you're dropped into based on logged-in-ness. If logged in, you'll be forced to the Session tab, and if logged out, the Checkout tab. I think we can refine this later, but it's a logical starting point and should reduce confusion. I also rearranged the tab order, which was necessary to make this work currently. Better minds than mine can hopefully help us improve the uib-tabset code in use here.

4) I think the whole tab use as it stands could be improved by more eyes on how it's being used. uib-tabset seems ... touchy. I don't have a great fix for that. As for the menu options staying in offline mode -- those are set at page render time, not dynamically, and that's because of the hotkey code we're using. Also, in order to have used the offline interfaces, I think the user would have to have logged out, so I suspect they'd need to log in again anyway, which means a new page load and "corrected" menu links. Am I missing something there?

5) I see it to, but it's not causing a problem. I think it's something to look into, but doesn't seem to be breaking anything.

And, here's the branch ... oops!

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/miker/lp-1706107-offline-mode-rebased-fixes

And, I've pushed another commit to the same branch to address the typos (and another problem) in the test.

Note: that test does /not/ spot the problem with moment.js I fixed above. It's specifically looking for references to resources that /do not/ exist and always give a 404. The canonical case is the woff2 font file from the bootstrap CSS distribution.

Thanks Mike!

I loaded the new branch. When I log into the web client, I'm getting 404s in the Console for ngToast.min.css and for ngToast-animations.min.css.

I'm still seeing the problems from #1 with the latest branch. Once I clear the browser cache (files and images), I get disconnected from network errors.

#2 was solved by loading the branch on a new VM where I did not see the problem.

I think we can take a look at the menus entries further down the road.

Oops! Ignore my last comment as I wrote it before Mike posted comment #29. I'll take another look.

I've loaded the latest commits from Mike's branch. I'm no longer getting the 404s for ngToast. When I run the live test, it tells me I'm not missing any assets.

I unregistered the service worker and cleared cached storage and then logged into the web client, retrieved a patron and retrieved the patron edit screen as recommended.

When I go offline, I initially can retrieve the offline interface through the Circulation menu. However, if I then refresh the page without even clearing the cache, I get the Disconnected from network errors.

I have a screencast showing this behavior at https://drive.google.com/file/d/0B74gDMUDwDXqd3BOd1dFUVpHRUU/view.

After working on this further with Mike, I discovered that a Chrome setting that I don't remember enabling was enabled and preventing the site from using the Service Workers whenever I had my dev tools opened. Once I disabled that setting, the offline branch worked as expected. All assets were loading, and I was able to retrieve the offline interface while disconnected from the Internet even after clearing my browser cache.

Merged to master for inclusion in 3.0. Thanks for your work on this Mike!