Real time data capture

These pages are intended as a living way of developing a specification for an on-competition-site data capture system as outlined in the paper presented to the 2012 CIMA Plenary.

Suggestions for improvement welcome!

Refer to the ABG system for a working demo of something which inspired the idea.

The app

• Core to the system is an app which works on common hand-held devices, primarily tablets, but should also be functional on smaller devices like phones. (the data terminal)
• It is proposed the initial work on terminals should be directed at Android devices since this is an open-source OS and the hardware is just as prevalent and generally much cheaper than the equivalent offerings from Apple, but there is no reason why the same thing can't be ported to iOS or Windows at a later time.
• The app should be a touch-screen centric application capable of:
  • Displaying means of collecting all the data as may be required to score all pilots in a task. (except, for the time being, logger derived data).
  • Interacting with USB connected hardware designed as alternative means of collecting data (eg button-box for stopwatch-like timing functions).
  • Sending that data in near-real time to a remote server which can further process it into a score or result.

Functional overview

The ABG system was disadvantaged by being hard-coded, which made it only usable in a very specific set of tasks without re-writing and re-installing the terminal software.

This system must be as 'universal' as possible, it is therefore proposed that the app is arranged in the form of a 'framework' consisting of:

• Overall graphic layout
• Universal settings, communications and other core services
• Many data-collection / graphical sub-components, ideally designed as 'add-ins' rather than fully compiled objects in the core source;
  • login component
  • Task location identification
  • Task name and description
  • Pilot start list
  • Active pilot display
  • Count-up timer
  • Count-down timer
  • Strike counter
  • Distance input
  • Lap input
  • Penalty selector
  • Button-box input translator
  • Etc.

Task configuration

This framework then makes it possible to collect all the data as may be required to score a task by applying a configuration script to the app which sets the app up to display all the required sub-components in the right place within the overall graphic layout, that those sub-components know how to interact with each other, and what data to collect.

It is not a dissimilar idea to a web page which is rendered within a software framework (the browser) according to a script (html). The difference is that there is only one script for a number of 'pages' and the components which go into building each page already exist as fully integrated entities within the app so security between the os and app (eg clock) and interaction between components is already configured rather than having to manage them dynamically from the script.

Configuration scripts are in xml to a precisely defined format which implies there should be an online task builder which can be used to interactively build suitable scripts, and a library where scripts known to work with a particular task can be saved for re-use.

Scripts are saved from the builder or library to each device to be loaded into the app ready for the start of a task.

Data-in

Some sub-components must be capable of collecting external data such as a start list. (Names, comp no., task start order). This implies there must be a standard data-structure (xml) with a detailed specification for each sub-component of this type, which in turn means there must be at least one simple method of generating it. It is proposed that the simplest way of initially achieving this is to produce an Excel add-in similar to the one already demoed with the WLC results, but other systems can be made available at a later time as necessary.

New discovery: A very good library of excel REST stuff at mcpher.com

Data-out

This too requires a standard data-structure (in xml) which is managed by the app in association with its communication protocols, but the sub-components must know what data they need to supply so the data is complete.
The app is useless unless the data it produces can be read by something. As with data-in, it is proposed an Excel add-in is initially produced which is capable of reading the supplied data and inserting it into the appropriate place in the scoring spreadsheet. Other systems can be made available at a later time as necessary.

Two variants of data-out.

• Full version: A full record of everything so it also contains logging info such as the current configuration, sequence of keypresses Etc. Continuously saved locally but sent to server only rarely.
• A 'reduced' version containing only key data for normal transmission to server.

Communications

The core app should be capable of communicating with:
1) Simple: A file system on the local subnet (classically using the wi-fi capability of the device)
2) Via http to a server on the local subnet (also via wifi)
3) Via http, wifi and a router on the local network to a remote internet based server.
4) Via data connection with a mobile network to a remote server. Could be file-based (eg dropbox) or http.

The choice of which system is used is dependent on the sophistication of the user. Variant 1 should require nothing more than a single computer connected to a wireless router in access-point mode which is within range of the terminals in use.

It is fundamental to the system that current communication state is known to the app and the operator, if communication is not currently possible to the configured data source then the app must cache data until comms are re-established. Equally, the operator must be made aware of current comms state so he can do something about it if comms are lost (eg move back into range of the wi-fi signal).

Syncronization

System should be capable of supporting several operators collecting task data simultaneously on different terminals. There are many occasions when two or more tasks may be occurring simultaneously (eg 2 parallel courses), or a task requires more than one source of data due to physical disposition of where the people operating the terminals must be (eg eco-laps). In both cases they are the same task and the data needs to be consolidated into one destination source so a current standing in task or score can be produced.

This is not the same thing as having a second operator collecting 'backup' data which can easily cause the dilemma: who is correct when there is a difference? It is not proposed the system should be 'dilemma-resolution capable' so backups, if collected, should be done manually in the traditional manner and corrections entered into the terminal if necessary.

Since the simplest variant of maintaining data-out data should require no special software on the receiving server, at its simplest it is thought this might be possible to achieve at at the file system level:

• Terminal A has data to send. First it retrieves the current data file on the server, adds its latest data to it and sends it back, maintaining a lock on the file during the whole process.
• Terminal B also wants to send data, but must wait until terminal A has unlocked the file before it can retrieve the file.... Etc.

Whilst this appears to be a relatively ineffecient synchronous way of doing it, it does mean every terminal has the ability to be constantly updated with the latest data from every terminal, and it should prevent conflicts of data. If all terminals are always looking for the newest file to update, timouts (ie unreleased locks) could be controlled by a terminal creating a new file on the receiving server after a reasonable delay, eg 30 sec. Other terminals then carry on updating that file.

Systems using http connection which is intrinsically asynchronous will need some level of software at the receiving end not just to do something with the data (eg write it to a file or db) but which can also manage the same sync protocol in an orderly manner.

summary of requirements for a complete initial system

• The app (android)
• Basic sub-components (as listed above, android)
• Configuration script structure (xml)
• Task builder (web)
• Task library (web)
• Data-in structure for start list component (xml)
• Excel add-in script for loading start list (excel)
• Data-out structure (full version, xml)
• Data-out structure (truncated version, xml)
• Excel add-in script for loading data into scoring spreadsheet. (excel)

Licensing

It is proposed CIMA invests in the the original work required to get a basic system up and running. It might therefore wish to retain the source code and even charge a licence fee. While there are quite a few components to the basic system, the overall design is intended to incorporate the core functionality but be expansible from the very beginning. (other OS's, other scoring systems Etc)

By far the best way to achieve this is to make the entire system open source from the very start which might encourage other developers to contribute to the success of the system without the complications of licensing Etc.

See opensource,org for a selection of common licences.