Control panels |

### Saving results, and telling participants what they are

with a description of the syntax of dSurvey data...

- Introduction
- Review of postCAP
  Specifying variables in `<general>`
  Specifying variables in `<message>`
  Specifying variables in `<reference>`
  Specifying variables in `<users>`
  Renaming
  Examples of <users> varlist

- Creating resultsData
  Using your own javascript function
  dSurvey functions to work with
- resultsData functions
- workData and submittedData functions
- dataObject functions (such as .math())
  Using other software

- Modifying userData and generalData

- Appendix 1: Syntax of resultsData database
- Appendix 1a: Syntax of workData database
  Structure of workData XML files
- Appendix 2: Structure of submitted data XML files
  Structure of submittedData database
  Structure of submitted data XML files
- Appendix 3: Retrieving/saving data
- Retrieving/saving submitted data
- Retrieving workData: from local file
- Creating resultsData, after reading a local file
- Retrieving workData/creating resultsData: from server archive Saving other data
- currentParticipants
- generaldata
- userData
- resultsData
- historyData
- Syntax of retrieved data files Structure of data objects used to store general, message, user, and reference data

- Appendix 4: Summary of <postCAP>
- Appendix 5: Useful functions
- Appendix 5a: Functions that work with `resultsData`
- Appendix 5b: Functions that work with `workData` and `submittedData`
- Appendix 5c: Functions that work with `dataObjects`

Introduction
It just might be possible that your survey will benefit from participants knowing the results of their choices.
For example, in a multi-round auction experiment, before a new round starts most people would like to know whether their most recent round's bids were accepted, and how much money they made (or lost).
Typically, these results are something that the administrator is responsible for generating (since results are often a function of the entries from all participants).

Thus, the administrator needs to be able to access information submitted by participants, manipulate this information to create results, and communicate these results back to all of the participants.

These actions are accomplished through the use of a dSurvey control panel. In particular

- dSurvey actions are used to retrieve, from the server, data submitted by participants
- This data is stored in control-panel internal databases: such as the `generalData`, `currentParticipants`, `userData`, and `resultsData` control panel databases
- Custom javascript functions, or external programs, can be used to manipulate these databases and create results
- The administrator communicates these results to clients via client accessible parameter files

This document addresses the above issues; with disuccsion of the structure of the control panel databases, documentation of javascript functions that can be used to manipulate and calculate, and more.
Review of postCAP
Starting at the final step, the `<postCAP>` subaction of the `<DO>` action (in control panel programs) is used to create client-accessible parameter files on the server. These parameter files are read by clients (using the params action in a dSurvey SPEC file).

The following briefly summaries `postCAP`. Appendix 4 contains further details.

<postCAP type="round" > <general> list of variables </general> <message> list of variables </message> <users> list of variables </users> <reference> list of variables </reference> </postCAP>

where: |
`<general> varlist </general>` | Save `general` variables (variables that are the same for all participants). As described below, `varlist` specifies which non-participant specific variables to include
`<message> varlist </message>` | List of participant-specific variables, which can include userData and currentParticipant data.
`<users> varlist </users>` | List of participant-specific variables, which can include the results. For each currently known participant (as determined using the most recently retrieved list of participants) a seperate `<user> .. </user>` record will be written to the client-accessible parameters file; with each record containing the participant-specific contents of the variables specified in varlist .
`<reference> varlist </reference>` | List of reference variables to include in the XML data file to be created. Each client will match his REFID to a record appearing in this `<reference>` section. For each reference record a seperate `<reference> .. </reference>` record will be written to the client-accessible parameters file; with each record containing the contents of the variables specified in varlist .
Shortcuts: # * : use all variables in the referenceData. If * appears, then ignore everything else! # $varname : See the above description

Creating resultsData
Let's summarize: the results, say of a round in an auction, needs to be sent to the participants. And these results are stored in an internal `resultsData` database. And you use `<postCAP />`, with `varname.R or *.R`, to create client-accessible files that contain these results. The clients can read read these files, and take appropriate action (say, move on to the next round).

Which means that administrator must first compute new results, and save them to the `resultsData` internal database. There are basically two ways of doing this:

1. Write a custom javascript program that will read the data submitted for this round, compute the necessary results (say, payouts and winners in an auction round), and directly write the results to `resultsData`.
2. Download data to your local hard drive, process it using your favorite software, save this processed data to an XML file, upload this file back into the control panel, and use it to populate the `resultsData` database.

Note: in either case, it is useful to understand the syntax & structure of resultsData and of submitted data

Using your own javascript function
If you can work with javascript, use of a custom function is the most fluid means of processing the submitted data to create `resultsData`. There are basically two steps:

1. Obtain, from the server, this round's participant-submitted data (you don't need to save it to the local hard drive). Or, if you are computing summaries across multiple rounds, you might want to retrieve a server archive
2. Call your javascript function. This function should use data in the submittedData control-panel database (and possibly the workData and resultsData control-panel databases). It should create new variables in the resultsData control-panel database. It is often convenient to use dataObject functions.

The following `<do>` actions in a control-panel program can be used:

- to read submitted data, and save it to the `submittedData` control-panel database: `<getSubmitted />`
- to read a results, history, or some other data archive on the server; and save it to the `workData` control-panel database: `<downloadData />`
- Unfortunately, currently there is no easy way to save prior-round variables from more than one dataset. For example: you can save to `workData` all rounds of `results`, or all rounds of `general variables`, but not both. In such cases, you can use dSurvey's dataObjects, in your own javascript functions, to combine and manipulate data.
- to call a custom javascript function: `<set />`

The following functions can be used in the custom function, to work with the resultsData database

`readResultsVar(clientid,avar,adefault)` | Read the `avar` variable for the `clientid` client. Return `adefault` if avar or clientid does not exist
`saveResultsVar(clientid,avar,aval)` | Save `aval` to `avar`, for client `clientid`.
`saveDOResultsVar(z,idVar,varlist,defval,noast)` | Save variables in entries in the z data-object
`aval=getVarInfo('varname.g','value')` | Read the `varname` general variable, and save its value to `aval`.
`aval=getVarInfo('varname.c','value')` | Read the `varname` control variable, and save its value to `aval`.
`aval=setControlVar('varname','value')` | Set the `varname` control variable to be `aval`. Your control-panel program (that calls your custom functio) can then read this value using a `<set&;gt`.

The following functions can be used to extract information from `workData` or `submittedData`

`extractRoundData` | extract a matrix containing a round's data (one row per client)
`extractClientData` | extract a matrix containing a client's data (one row per round)
`extractRoundList` | extract an array of round numbers
`extractClientList` | extract an array list of client IDs

`dataObjects` are special dSurvey tools that simplify manipulation of information on all participants in a session. The following functions are used to create, modify, and work with `dataObjects`.

`createDO(nlen) or `createDO(idsArray)` or `createDO(varray,vnames)`` | Create a dataObject
The following are methods of the `objects` created by `createDO`
`add(vv,vname)` | Add a `data` array (or several arrays) to a dataObject
`addRecord(vv,vname)` | Add a record to a dataObject
`addConstant(vv,vname,isarray)` | Add `constants` (value or array) to a dataObject
`combine(z2,defval,vlist)` | Combine the with values & entries from z2 dataObject; and return a new dataObject.
`display(varlist,constlist)` | Make a string containing an HTML contents of the values in the dataObject
`extract(getList,getlist,clist)` | Extract a new dataObject using `entries` specified in `getList`
`extractID(idList,vlist,clist)` | Extract a new dataObject using `entries` specified in `idList`
`extractRule(vv,arule,aparam,vlist,clist)` | Extract a new dataObject using the rules specified using `arule`
`get(vnames)` | Retrieve a `data` array (or several arrays) from dataObject
`math(anEquation,varlist,vallist)` | Save/update variables specified in anEquation, for records contained in zuse
`removeVars(vlist,iskeep)` | Remove (or keep) variables
`update(z2,vlist,adef,ahint)` | update the dataObject using data in z2
`updateRecord(idVal,vv,avars,defval,idVar)` | update one record in the dataObject using data in vv
`save(aname)` | Save the dataObject to global strorage (for later retrieval)
`z=retrieveDO(aname,exists)` | Retrieve a dataObject from global storage

The following statistical and matrix manipulation functions may be useful (they are an alternative to using the math method of a `dataObject`):

NormalP( x ) | pdf of a normal distribution
InvNormalP(x) | inverse normal (given a critical value)
...
sortNum(anArray) | numeric sort -- returns new array numerically sorted
indexSort(array,type) | sort an array, return index into array.
sortByIndex(theVals,theInds,ifrom,ito) | rearrange the "theVals" array using theInds index (i.e.; as returned by indexSort)
indexSort2(vec1,vec2,atype) | returns a 2d index matrix whose elements point to sorted rows of vec1 and vec2
...
maxArray(varray,amiss) | returns maximum of the varray array,
minArray(varray,amiss) | returns minimum of the varray array
sumArray(vv,amiss) | returns sum of the vv array, or sum of all columns in vv matrix.
meanArray(vv,amiss) | returns mean of the vv array, or mean of all columns in vv matrix.
medianArray(varray,amiss) | returns median value of the varray array,
varArray(varray,amiss) | returns variance value of the varray array
freqArray(varray,iclasses) | return an array of frequencies (# of varray elements that match an element in iclasses)
uniqueArray(vv,isort) | vector of unique values of this array. If isort=1, sorted
rnduArray(n,seed) | return an n element array of uniformly distributed random numbers
rndnArray(n,seed) | return an n element array of normally distributed random numbers
choseRandomArray(n,v,repOk) | choose n values randomly from array v
....
mtxMath(amtx,'mathEqu',defVal) | Return an array with values from math operations on the columns of amtx
mtxMath2(v1,v2,'mathEqu',defVal) | Return an array with values from math operations on the v1 and v2 arrays
....
selectRows(vals,cond,vcomp,asIndex) | select rows from vals; return new matrix or an index
...
extractCol(vv,icol) | extract the icol column from the vv matrix and return as an array.
extractRows(vv,rlist,useFlag) | extract rows from a matrix (i.e.; using an index returned by selectRows).
appendRows(amtx,bmtx) | append rows of matrix b to end of matrix a.
appendCols(amtx,bmtx) | append columns of matrix b to right of matrix a.
replaceCol(vv,icol,newCol) | replace the icol column in the vv matrix with newCol.
replaceRows(vv,rlist,newRows) | replace the rlist rows in vv with newRows
...
seqArray(i1,i2,istep) | returns an array of values starting at i1, to i2, by istep
initArray(nlen,defval) | initialize an nlen long array, with each element set to defval

Using other software
dSurvey saves information using XML files. In particular, from the control-panel the administrator can retrieve data submitted by participants, and save it as an XML file. These can contain all the data submitted to date, or they can contain data for just one round (say, the most recently completed round).

In either case, the XML file that dSurvey can be read by other software. This other software (for example, Excel), can then compute and then save results to a new XML file. This new XML file can be read by dSurvey, and used to populate the `resultsData` database.

There are basically three steps:

1. Obtain, from the server, this round's participant-submitted data. You must to save it to the local hard drive!
   You may also want to download other data, such as the `userData` (that describes each client), and the list of current participants.
2. Using your favorite software, read this XML data file (or files). Then process the data to determine the results.
3. Save the results to a new XML file, using the workData XML file format.
4. Retrieve the workData XML file (from your local hard drive) and use it create resultsData

Regardless of how you do it, once you create `resultsData`, you can update the client-accessible parameter files, and move on to the next step of the survey session.

Modifying userData and generalData
You can extract data from `resultsData` and use it to modify the `userData` and `generalData` control panel databases. This is done with `<fromResults>` (within a `<do>`).
Recollect that `userData` and `generalData` are read when `postCAP` is used to write `general` and `users` data to a client-accessible parameters file. Thus, you would use one or more `<fromResults>` just before a `<postCAP>`.

But first, by way of review ...

Example of modifying a `general` variable: `nowround.G=nowround.G + 1`

Syntax of `fromResults`

<fromResults> varname.g = rVar </fromResults> <fromResults> varname.u = rVar </fromResults>
where...

- `varname.g` : a `general` variable will be created The value is pulled from the `*` id record of the `resultsData` database, using the `rVar` variable.
- `varname.u` : a `user` variable will be created for all current clients (as specified in the `currentParticipants` control panel database).

Examples:
<fromResults> nAccept.G=numAccept </fromResults> <fromResults> profit.U=totalGain </fromResults>

`Technical note`: | the first example would create a `nAccept` general variable using the value of `resutlsData['*']['NUMACCEPT']` The second example would create a `profit` variable for all clients using the values of `resutlsData[cid]['TOTALGAIN']` , with `cid` ranging over all currently specified clientIDs
`Caution:` | As of this writing, you can not use equations to modify `resultsData`. That means that `fromResults` is a straight copy -- with no ability to use equations. However, you can modify a general variable after assigning it a value from `resultsData`.
`Missing values:` | If a variable is not defined for a particular clientId (or for the * clientID), a blank will be used (a value of '')

If you working with a custom javascript function, you can use:

readGeneralVar(aname,adefault) Return value of aname general variable; possibly an array element (x[1]). Use adefault if no such general variable saveGeneralVar(aname,aval,isarray) Save aval value of aname general variable. If isarray=1, then aval is an array readUserVar(cid,aname,adefault) Return value of aname user variable for the client with an id of cid Use adefault if no such client, or no such variable saveUserVar(aname,clientid,aval) Save aval to the aname user variable, for client clientid readResultsVar(cid,aname,adefault) Return the "round" these results are from saveResultsVar(clientid,aname,aval) Save aval to aname results variable, for client clientid

Appendices
Appendix 1: Syntax of resultsData database
Reminder: you can use several dSurvey functions to access resultsData. Thus, the following description is provided mostly for ambitious programmers (less ambitous progammers will find it easier to use the functions!)!
`resultsData` should be a javascript object, with the following structure:
resultsData['$IDS'] ; an array of IDS for which results are available resultsData['$ROUND']'; the current round. This is the value appended to names (as described above) resultsData['$VARS'] ; client-specific variables names -- note that each ID and round need not have all the variables resultsData[aid][avar] the current value of the avar variable for the aid participant resultsData['*']['$VARS'] non-client specific variable names. These can be complete different than the client-specific variable names resultsData['*'][avar] the current value of the avar non-client-specific variable
Thus, if the following results were stored, for round 3:
resultsData['$ROUND']=3 resultsData['$VARS']=['COST','profit']; // 2 element array (MUST Be UPPER CASE) resultsData['1001']['COST']=10.51 resultsData['1001']['PROFIT']=5.00 resultsData['1002']['COST']=4.25 resultsData['1002']['PROFIT']=1.00 resultsData['$VARS']=['MAXOFFER','NUMCHOSEN','MINOFFER']; // 3 element array resultsData['*']['MAXCOST']=10.51 resultsData['*']['MINCOST']=4.25 resultsData['*']['NUMCHOSEN']=1
and the `postCAP` specified ...
<general> *.R </general> <users> cost.r profit.r </users>
then the following would be written to the client-accessible parameters file:
<general> <MAXCOST.3>10.51</MAXCOST.3> <MINCOST.3>4.25</MINCOST.3> <NUMCHOSEN.3>1</NUMCHOSEN.3> </general> <users> <vars>COST.3,PROFIT.3 </vars> <vals> <user id="1001">10.51,5.00</user> <user id="1002">4.25,1.00</user> </vals> </users>
When the client read this file, it would use all the variables specified in `<general>`.

It would also search for an `id` that matched his client id. Thus, the client with an id of `1001` would:

- Find the row: `<user id="1001">10.51,5.00</user>`
- Set the `cost` variable, for round 3, to be equal to `10.51`
- Set the `profit` variable, for round 3, to be equal to `5.00`
- If, for some reason, a variable (such as `profit`) was not defined for the id, an empty value (a value of '') will be used.

Notes:

- you should not store arrays as client-specific `resultsData`
- you can store arrays as non-client-specific (*) `resultsData`
- you should never ssave data in `resultsData` using a name that includes `[` or `]` characters

Structure of workData internal database
The `workData` database is created when files are uploaded from the local hard drive, or when a server archive is retrieved.

Reminder: you can use several dSurvey functions to access workData data. Thus, the following description is provided mostly for ambitious programmers (less ambitous progammers will find it easier to use the functions!)!
The syntax of `workData` is an extension of `resultsData`: rather than just one round of data, `workData` can contain multiple rounds of data for both general and client-specific variables.
workData['ids'] : array of IDS (for whom at least one variable is specified) workData['rounds'] : array of ROUNDS (within which at least one variable is specified) workData['vars'] : array of variables that are specified at least once workData[around][aid][varname] : value of the varname variable in the around round for the aid client workData[around][aid]['varlist'] : list of variables defined for this round and client
Unlike `resultsData` `around` and `aid` can have a value of `*` .
`*` means all rounds, or all ids. Thus:
workData['*']['1001']['MAXWIN'] : the MAXWIN variable for client 1001 -- not round specific (applies to this client in all rounds) workData['2']['*']['CUTOFF'] : the CUTOFF variable for round 2 -- not client specific (applies to all clients) workData['*']['*']['DURATION'] : the DURATION variable for all clients in all rounds. This is equivalent to a GENERAL variable workData['3']['*']['varlist'] : an array containing the list of round-3-specific variables
Notes

- case matters (i.e.; use `ids` and not `IDS`. For `varname`, always use upper case.
- the `varlist` variable (it must be lowercase) specifies the variables defined for this round & client (where either round or client can be *).
- the `vars` variable lists all the variables defined at least once. That is, a variable listed in `vars` is not necessarily specifed for every round & client.

Structure of workData XML files
Once your external software has done its work (say, using submitted data to determine this round's results), it should write an XML file. We refer to this output (of your external software) as a `workData XML file`. dSurvey's control panel can then be instructed to read this `workData XML file`, and store its information in both the `workData` and resultsData internal databases. However, for this to succeed, your external software must use the proper syntax!

The syntax these `workData XML files` must have is essentially the same as submitted data XML files.

There are a few differences:

- `workData` can use `*` as either a round or client id. For example:
  <data> <round> * </round> .... </data>
- If `<id>` or `<round>` are not specified, a `*` is used. In other words, not specifying an `id` (or a `round`) in a `<data>` records means: use these values for all `ids` (or `rounds`).
- the `<ct>` and `<notFinal>` variables are not used.
  This means that when dSurvey reads a `workData` XML file, it will process records in the order of appearance. Thus, if a record has the same client and round id as an earlier, it will overwrite any pre-existing variables.

Appendix 2:Structure of submitted data

submitted data refers to data that participants have sent to the web-server. dSurvey offers several methods of retrieving all of this data, or just the data related to the current round. This data is be stored in an internal `submittedDat` database. Or, it can be saved to an XML file on on your local hard drive. It will be stored using a fairly simple XML syntax, a syntax that should be easy to read by other software (such as excel).

Structure of submittedData internal database
The `submittedData` dSurvey internal database is used to store data retrieved from the web-server. In particular, the information submitted to the web-server by participants. You can use <getSubmitted> to retrieve submitted data from the server (and save it to `submittedData`).
Reminder: you can use several dSurvey functions to access submitted data. Thus, the following description is provided mostly for ambitious programmers (less ambitous progammers will find it easier to use the functions!)!

`submittedData` has the following syntax. Note that `record` refers to a set of variables containing information submitted by one participant in a given round.

We start with the descriptive components of submittedData. Then, we describe the client & round specific components. Please note that the following is case sensitive.

submittedData['$records'] # of records stored submittedData['$idList'] An array listing client-ids that have at least one record in submittedData submittedData['$roundList'] An array listing the round numbers for which at least one record exists submittedData['$varList'] Array of variable names mentioned in any record. submittedData['$ids'] ; An object, reference by ids. Each element is an array of the round numbers records belonging to this id. Example: if submittedData['$ids']['121'] contains 1 2 3 5 6 7 then participant 121 participated in 6 rounds (1, 2, 3, 5, 6, 7) submittedData['$rounds'] ; An object, reference by round numbers. Each element is an array of the participant IDS belonging to this round. Example: if submittedData['$rounds'][6] contains 121 122 124 125 then 4 participants (121, 122, 124, and 125) participated in round 6.
The `submittedData['$ids']` and `submittedData['$rounds']` are useful tools -- they let you quickly extract all entries belonging to a single client, or all entries belonging to a single round. Note that:
- a given client does not have to have entries in all rounds
- a given round does not have to have entries for all clients
- each entry does not have to have values for all variables
This means that any variable specified in an entry will be listed in `submittedData['$vars']`, but not all entries will have values for all variables in `submittedData['$vars']`.

Now, the client & round specific components (that comprise a `record`).

submittedData[rNum][cId][varname] : value of the vaname variable for cId participant in round rNum. Special varnames: $notFinal : value of the "not final" attribute for the data. $ct : value of the "ct" attribute for the data $time : time of submission (in javascript milliseconds format) $cts : an array of all the values of the [ct] submissions for this id & round submittedData[rNum][cId][ct][varname] value of the varname variable for the ct submission of the cId client in round roundNum. There may be several submissions per round If no ct attribute in a data record, ct="1" is assigned. If ct is NOT a nunber, an error occurs For these [ct] records ... The $notFinal and $time varnames are also available but $ct and $cts are NOT available
Notes:

- For each client & round, there are two sets of data. First, there is a single final entry record. Second, there is a set (possibly of just one) intermediate records.

- Values for the final record are in `submittedData[roundNum][clientId][varName]`.
  That is: a final record consists of all the `submiteData[rNum][cId][[varname]` values for given a `rNum and Cid`.

- Values for the intermediate records are in `submittedData[roundNum][clientId][ct][varName]`
  That is: an intermediate final record consists of all the `submiteData[rNum][cId][ct][varname]` values for given a `rNum, cId, and ct`
  You can use the `submittedData[roundNum][clientId]['$cts']` to find the `ct` values of the intermediate records.

- The `$notFinal` variable indicates whether a final record (or an intermediate record) should be considered to be final.
- `submittedData[roundNum][clientId]['$notFinal']` equal to 1 implies that a future submission may be final -- in which case this future submission would overwrite the values for`submittedData[roundNum][clientId]`. In a sense, `$notFinal` of 0 protects this submittedData final record from being overwritten.
- `submittedData[roundNum][clientId][ct]['$notFinal']` equal to 0 implies that this `ct` record is being used as the final record (for this client in this round).
- There really should only be one record per client per round that has `$notFinal=0`. If, through some bad design, there is more than one; the first occuring record will be used as the final record.

Structure of submitted data XML files

The submitted data XML files are built from the `submitteData` database. Depending on how the file is created, a subset of the records in `submittedData` will be written to the XML file.

The following example outlines the structure of these XML files. Note that workData XML files have essentially the same structure.

<submittedData session="t94" time="1226757439456" > <data><round>1</round><id>1001</id> <time>1226357499881</time> <ct>1</ct> <notFinal>0</notFinal> <TICKETTYPE>/TICKETTYPE> <PRICE1>12.65</PRICE1> <OFFER_1_>15.65</OFFER_1_> <OFFER_2_>63</OFFER_2_> </data> ... <data><round>1</round><id>1012</id> <time>122635750399881</time> <ct>1</ct> <notFinal>0</notFinal> <TICKETTYPE>1</TICKETTYPE1> <PRICE1>12.65</PRICE1> <OFFER_1_>35</OFFER_1_> <OFFER_2_>22</OFFER_2_> </data> ... <data><round>14</round><id>1012</id> <time>12263575099881</time> <ct>1</ct> <notFinal>0</notFinal> <TICKETTYPE>3</TICKETTYPE> <PRICE1>13.00</PRICE1> <OFFER_1_>22.55</OFFER_1_> <OFFER_2_>12</OFFER_2_> </data> </submittedData>
Notes:

- As with any XML file, it must have a topmost node. In the above example we use `<submittedData>`, but the precise name doesn't matter.
- Note that for reference purposes, the topmost node has a `session` and `time` attributes. The `time` is expressed using javascript time (milliseconds since 00:00:00 GMT on 01-01-1970).
- Each `<data>` entry contains data for a single participant in a single round. This need not be unique: there can be several `<data>` entries for a single participant in a single round.
- Each node within a `<data>` contains the value of a variable specific to this participant in this round. Let's call them variable nodes.
- Five variable nodes are always included. Two are essential, the other three are less frequently used:
  1. `<round>` : The round
  2. `<id>` : The client ID
     the less essential three...
  3. `<time>` : The time (using javascript time) this information was submitted by the participant
  4. `<notFinal>` : See description above
  5. `<ct>` : See description above
- In the above example, the `TICKETTYPE` and `PRICE1` variable nodes are defined for each `<data>` .
- Each `data` entry can contain `different` variables. However, this is rarely the case; though one might see empty variable nodes used when no value is available for this variable in this round for this partcipant.
- Values specified in the variable nodes can contain any characters other than < and >
- Variable names that end in `_n_` are typically converted array variables -- so that a variable of `PRICE_1_` probably was submitted as `PRICE[1]`.

- Actually, the above is somewhat simplified -- the XML files will also contain a few useful comments.

Technical note: Data is saved on the server in a server-specific fashion. For some servers, a compressed, quasi-XML format is used; with each entry stored as a comma delimited list of escaped values (rather than seperate nodes for each variable).
In contrast: when a dSurvey control panel saves submitted data to a local file, it creates XML files that are configured for easy use by external software.

saving as CSV
Instead of saving data (to the local hard drive) as an XML file, you can save data as comma-seperated values (CSV). When saving as a a CSV
- Each line in the file contains an observation
- Values for each variable are seperated by commas
- Strings are enclosed within `"` quotes. Strings may contain commas!
- The top line typically contains a comma seperated list of variable names
Appendix 3: Retrieving/saving data
dSurvey uses a number of mechanisms for saving and retrieving data from the web-server, and from the local hard drive.

Unfortunately, due to javascript security protocols, some of these mechanisms are bit convoluted. They aren't real complicated, but they aren't totally transparent.

The next several sections discuss how-tos for various dSurvey internal databases, and the XML files they use.

Retrieving/saving submitted data
There are a few different ways of creating, and saving to your local hard drive, an XML file containing data submitted by participants in this survey session.

1. Using generic control-panel options. This is simple, but has no options -- you might get too much, you might get too little.
   1. Click on the Data button at the top of the control-panel
   2. Click on the button

      After you click this button, and once dSurvey completes its communications with the web server, the `submittedData` internal database will be populated. Thus, if you are working with a javascript function, you can skip the next two steps -- since the javascript function should work directly with `submittedData`.

      Note that a table will appear that display the values of all records in the `submittedData`.

      Reminder: the next two steps are only necessary if you are using external software to process the data. If you are using a javascript function, you can skip them ...

   3. Click on the Save as XML link.
   4. The specifics of the next step depends on the browser. Basically, a box will open that asks you whether or not to save the file, and what name to use. A default name of `submittedData.xml` should be displayed.

   This will create an XML file , using the name you chose. This XML file will contain:

   - Entries for all clients and all rounds.
   - Only the final record (for this client/round). That is, intermediate records will not be written.
   - For each record written, all variables will be specified; even if that client/round did not submit a value for that variable (in which case the value in the XML file will be a '').

2. Using `getSubmitted` subaction of the `<DO>` action (in your control-panel program).

   `getSubmitted` will download, from the server, the submitted data for this session: and optionally save it to the local hard drive as an XML file `getSubmitted` has the following syntax:

   <getSubmitted file="localFile.xml" rounds="roundlist" all="0/1" getCurrent="1" />
   where
   - file. Name of XML file.
   - If specified, the suggested name of the file to save the submitted data to.
   - If not specified, `submittedData.xml` is used.
   - if `file="0"` (or `file=""`) then do not save a file. The `submittedData` internal database will be populated, and nothing will be written to the local hard drive. This is often used when you are using javascript function to create `resultsData`

     Actually, due to javascript security a modest hack is used. Due to this hack, the actual use of `file` is browser dependent. Most of the time, a file save dialogue box will popup, with `localFile.xml` as the default choice.

     You should choose the desired name, and then hit the save (or whatever) button.

   - getCurrent If `getCurrent="1"`, get the current submitted-data archive from the server
   - rounds Optional. If not specified, then only the current round's records are saved.
     If specified, should be a space delimited list of round numbers.
     Shortcut: `rounds="*"` will save records from all rounds

   - all Optional. `all="0"` is the default -- it means only write records that are final.
     `finalOnly="1"` means write all records (write both the intermediate and final records).

     Note that if clients only engage in one submission per client/round, `all` will have no effect.

Due to XML limitiations ....

As noted above: when `submittedData` is saved to an XML file, array like names will be modified before being written to the XML file. Basically, `[n]` is converted to `_n_`.
For example, the `PRICE[2]` variable will saved as `<PRICE_2_> somevalue </PRICE_2_>`.

Retrieving workData: from local file
If you use external software (say, Excel) to compute results, this software must create an XML file that contains the results. And this file should be formatted as an workData XML file.

Assuming your external software does this properly, dSurvey must read this file and save its information to the `workData internal database`

There are two ways of doing this:

1. Use the `ReadFile` generic control-panel option
   - Click on the ReadFile option at the top of the control-panel
   - Use the button, and choose the `workData XML file` (that your external software created).
   - Click the button
   If you are curious, you can then examine the contents of `workData` by using the Data option (click the button).
2. Use the `<readLocal>` subaction of `<DO>`
   Syntax: <readLocal file="filename.xml" makeResults="1" checkRound="roundVar" useRound="rnum" roundRead="rndGot" />

Creating resultsData data, after reading a local file
Once dSurvey has populated the workData internal database, you can extract information from it to create `resultsData`. Note that `resultsData` will contain information for all ids specified in `workData`. This may not be the same as the current participants (as specified in `currentParticipants`).

There are two ways of doing this:

1. Use the `ReadFile` generic control-panel option. Click on the `Create resultsData?` checkbox before updating.

2. Use the <readLocal> subaction of `<DO>`. If you used `readLocal`, you can use the `makeResults` attribute tc extract data from the XML file (actually, from the workData internal database that will be filled with the data in the XML file). Note that `resultsData` will only contain data from one round. By default, this will be the maximum round in the XML file. However, you can specify a specific round (in the XML) file to use.

Retrieving workData: from server archive
To process several rounds of information, it is often convenient to upload each round's information to a server archive, and then (when needed) retrieve this archive to `workData`.

Note that you can use custom functions to manipulate `workData` and to save variables to `resultsData`.

Retrieving/saving other data
In addition to the submittedData, your external software might benefit from other data. This includes:

- currentParticipants

  and

- generalData
- userData
- resultsData
- historyData

Retrieving & saving: currentParticipants
The `currentParticipants` dSurvey internal database contains basic information on participants enrolled in the current session.

The structure of `currentParticipants` is:

currentParticipants['$IDS'] : array of IDS assigned to clients currentParticipants['$VARS'] : variables assigned to each client currentParticipants['$TIME'] : javascript time (milliseconds) currentParticipants last updated currentParticipants[aid][varname] : value of the varname variable for client aid
In all cases, `NAME`, `CONTACT`, and `NICKNAME`. In almost all cases, `REFID` will also be available.

There are a few ways of updating the `currentParticipants`.

1. <getParticipants>:

   The `getParticipants` subaction of the `<DO>` action (in your control-panel program) can be used download data on the current list of session participants, and save it to the `currentParticipants` dSurvey internal database.

   Syntax: `<getParticipants />`
2. The participants generic control-panel option: You can also use generic control panel options to create `currentParticipants`
   - Click on the participants button at the top of the control-panel

   - Click on the Update link (in the dropdown list)

#### Saving basic data on participants to an XML file

`getParticipants` can also be used to create an XML file, on your local hard drive, that contains basic information on session participants.

Syntax: `<getParticipants file="filename.xml" />` : where `filename.xml` is a suggested name to use for the local file.

The `file` attribute is used to supply a suggested file -- a dialogue box will open to ask you where to save the file.

The syntax of these particpants file is simple:

<participantList session="t94" time="1226812348118" > <user><id>1001</id> <NAME>A1</NAME> <CONTACT></CONTACT> <NICKNAME></CONTACT> <REFID></REFID> </user> </participantList>
where a seperate `user` is created for each participant.

Note: `getParticipants` will either retrieve the participant list from the server (and store it to the `currentParticipants` control panel database), or write the `currentParticipants` database to a local file. To do both, you need to specify `getParticipants` twice in a row:

1. `<getParticipants />` : retrieve the latest values from the server, and retain in `currentParticipants`
2. `<getParticipants file="filename.xml" />` : save `currentParticipants` to a local file.

generalData
dSurvey can save the current values of the `generalData` control panel database to either the web-server or to a file on the local hard drive.

1. When saved to the web-server, a record is added to a (possibly) pre-existing dataset on the webserver. This record will be identified by the current round number.

   At a later date, you can retrieve this dataset and write it to a file (on the local hard drive). And, assuming that `generalData` was saved-to-the-server at the end of each round, this retrieved dataset will contain a complete history of the `generalData`.

2. You can save the current round's values of the `generalData` to a file on the local drive. This file can be read, along with other files (such as submitted data) to compute results.

`uploadData` and `saveData` are used to do these actions.

1. To upload to the web server:
   <uploadData type="GENERAL" > varlist </uploadData>
2. To save to a local file:
   <saveData type="GENERAL file="filename.xml" > varlist </saveData>
   where `file="filename.xml"` is the suggested file name to use. For example, `file="myGeneral.xml"`.

For more details on `uploadData` and `saveData`, see controlPanel.htm. Note: if an array is saved, a seperate node will be stored for each element (with an index attribute).

#### Retrieving generalData from the server

You can write, to a local file, the database containing the `generalData` that you've saved to the server. This can be used as a round-by-round history of the session's `generalData`; which may come in handy when you analyze your data at some future date.

Note that this retrieved data does not effect the current values of the `generalData` control panel database. It is merely saved to a file.

To do this, use the `downloadData` action:

<downloadData type="GENERAL" file="localFile.xml" round="roundlist" />

- `file="localfile.xml"` is optional. A suggested name to use for the local file.
- `rounds="roundlist"` A list of round numbers to save.
  If `rounds` is not specified, or if `rounds="*"` is speified: than the entire dataset (stored on the web server) is used.
  If specified, should be a space delimited list of round numbers. Or variables (control or general) that contain a round number (such as nowround.g).

Note: as noted below, `downloadData` is also used to retreive the `userData`, `resultsData`, and `historyData` databases.
usersData
Saving `usersData`, like saving generalData, is accomplished with `<uploadData>` and `<saveData>`.
There are a few differences:

- `type="USERS` is specified.
- A record for each participant listed in `currentParticipants` is created, and saved (either to the server or to a file).
  Thus, more than one record is saved.
- The varlist can contain `* wildcarded` words. These will be expanded -- all variables names in `usersData` that match this wildcarded word will be used.
  For example: if `price[*]` is used, and a `price[1]` and `price[2]` variables are specified, than both `price[1]` and `price[2]` will be included in the list of variables to save.

To retrieve the server database that contains the saved `userData`, `downloadData` is used. The syntax is the same as as retrieving generalData, except `type="users"` should be used.
resultsData
Saving `resultsData`, like saving generalData, is accomplished with `<uploadData>` and `<saveData>`.
There are a few differences:

- `type="RESULTS"` is specified.
- A record for each participant listed in `currentParticipants` is created, and saved (either to the server or to a file).
  Thus, more than one record is saved.
- A record containing the current values for the * id (for both the current round and the * round) is also written

To retrieve the server database that contains the saved `resultsData`, `downloadData` is used. The syntax is the same as as retrieving generalData, except `type="results"` should be used.
historyData
Saving `historyData`, like saving generalData, is accomplished with `<uploadData>` and `<saveData>`.
There are a few differences:

- `type="HISTORY"` is specified.
- A record for each participant listed in `currentParticipants` is created, and saved (either to the server or to a file).
  Thus, more than one record is saved.
- `varlist` can specify variables from many of the control panel databases, using...
- To specify a generalData variable: `varname.g`. The value for this variable will be saved in eah record (thus, the value will be the same in each record)

- To specify a currentParticipants variable: `varname.c`. To specify all currentParticipants variables, use `*.c`
- To specify a usersData variable: `varname.u`. To specify all user variables, use `*.u`
- To specify a submittedData variable: `varname.s`. The most recent round is used; if, for a particular client, no information exists for this recent round blanks will be returned. To specify all submitted variables, use `*.s`
- To specify a resultsData variable: `varname.r`. If necessary, the `*` id version of this variable will be used (thus, like generalData variables, it is possible for the same value to be used in all records). To specify all results variables, use `*.r`

To retrieve the server database that contains the saved `historyData`, `downloadData` is used. The syntax is the same as as retrieving generalData, except `type="history"` should be used.

Note: one use of `historyData` is when a client has to reload the session. When this happens, store results are erased. You can use the `<recoverHistory >` action to restore this history -- by retrieving the `historyData` and reading the appropriate records.

Syntax of retrieved data files
The XML files created by `downloadData` (whether it is `generalData`, `usersData`, or `resultsData`) are similar...

For `generalData`

<generalData> <data> <round> roundNumber1 </round> <var1> avalue </var1> <arrayVar1 index="m" > avalue </arrayVar1> </data> .. <data> <round> roundNumberR </round> </data> </generalData>
Note that if an array is saved, a seperate node will be stored for each element (with an index attribute).

For `usersData`

<usersData> <data> <round> roundNumber1 </round> <id> idval </idd> <var1> avalue </var1> </data> .. <usersData> <data> <round> roundNumberR </round> <id> idval </idd> <var1> avalue </var1> </data> ..

For `resultsData`

<resultsData> <data> <round> roundNumber1 </round> <id> idval </idd> <var1> avalue </var1> </data> <data> <round> roundNumber1 </round> <id> idval2 </idd> <var1> avalue </var1> </data> <data> <round> roundNumber1 </round> <id> * </idd> <var1> avalue </var1> </data> .. <data> <round> roundNumberR </round> <id> idval </idd> <var1> avalue </var1> </data> ..
Note that each round of records will contain a record for the * id

For `historyData`

<historyData> <data> <round> roundNumber1 </round> <id> idval </idd> <var1& type="x" gt; avalue </var1> </data> .. <data> <round> roundNumberR </round> <id> idval </idd> <var1 type="x" > avalue </var1> </data> ..
Notes:

- the variables can be a mixture of userData, generalData, resultsData, and currentParticipants variables.
- The `type="x"` attribute in the <var > reflects the type:
  where `x` will be `U`, `G`, `R`, `S`, and `P`.
- Due to XML limitations, variable names that are array names (such as `PRICE[1]`) are written to the XML file with a conversion: `[n}` is converted to `_n_`.
- As of this version, the variable names are used as is: renaming is not supported.

Sturcture of data objects

#### Notes on how data dSurvey stores data in javascript `objects`:

dSurvey will store general, reference, and user data in global objects. Ambitious programmers can directly access these objects (say, in custom functions used within your control panel). The following describes the syntax of these variables.

- general variables are saved in the `generalVars` control-panel database. Hence, a `"costFactor"` variable would be saved as `generalVars['COSTFACTOR']` `generalVars['$VARS']` contains a list of all the general variables.

  Actually, a 3 element array is saved to generalVars['VARNAME']:

- `generalVars['VARNAME'][0]` : a value of `1` if this should be treated as an array
- `generalVars['VARNAME'][1]` : value. If an array, values are always seperated by a comma. Note that the value is always a string (not an array)
- `generalVars['VARNAME'][2]` : uncapitalized version of the varname You can use `readGeneralVar()` and `getVarInfo()` to work with generalVars.

- user variables are saved in the `userData` control-panel database. Hence, for userId of 35, a "income" variable would be saved as `userData['35']['INCOME']` (note the uppercase is always used in object references).
  `userData['$VARLOOKUP']` is an object used to identify if an variable exists: i.e.; `userData['$VARLOOKUP']['COST']=1` if `COST`variable exists.
  Otherwise, undefined
- `userData['$IDS']` contains a list of all the ids.
- `userData['$VARS']` is an array, with each element one of the variables
- uppercase is always used when specifying variables in userData

- reference variables are saved in the `refData` control-panel database. Hence, for refId of 12, a "group1" variable would be saved as `refData['12']['group1']`
- `refData['$VARLOOKUP']` is an objecct used to identify if an variable exists: i.e.; `refData['$VARLOOKUP']['COST']=1` if "COST" variable exists Otherwise, undefined
- `refData['$IDS']` contains a list of all the ids.
- `refData['$VARS']` is an array, with each element one of the variables
- uppercase is always used when specifying variables in refData

Appendix 4: Summary of `<postCAP >`
The following summarizes the most commonly used features in `<postCAP>` is used to create client-accessible parameter files. For further description of `postCAP`, see controlPanel.htm

Basic syntax:

<postCAP type="round" > <general> list of variables </general> <message> list of variables </message> <users> list of variables </users> <reference> list of variables </reference> </postCAP>
When `postCAP` is processed, client-accessible parameter files will be created. These can contain `general` and `client-specific` variables. The `client-specific` variables are written for all current participants. Thus, once a participant is removed (say, using the `remove a participant from a survey-session` option), no information will be included for him in future client-accessible parameter files.
Specifying variables in `<general >`
There are several kinds of variables that the `varlist` in `<general>` can refer to:

1. `generalData`: variables read from a parameters file (using getParams), or variables created/modified in by a input-fields specified in a <FORM>.
2. `resultsData`: Current results for this round.

To specify these variables, the following syntax is used:

1. `generalData`: `varname` :: Example: `numTickets`
2. `resultsData`: `varname.r` :: Example: `bidCutoff.r`

You can use the following shortcuts:

- `*` : use all variables in generalData
- `*.R` : use all the non-client-specific variables in resultsData
- `*.F` : use all variables changed (by input-fields in a `form` that have changed since the last postCAP)
- `$varname` : A space-delimited list of variables stored in the varname control variable

Notes

- if you specify a variable more than once (say, in a shortcut and explicitily), the extra reference will be ignored (thus, specifying a variable once is no different than specifying it a zillion times).

Specifying variables in `<message >`
There are several kinds of variables that the `varlist` in `<message>` can refer to:

1. `userData`: variables either read from a parameters file (using getParams). Typically, theare are variables assigned to a participant from a reference dataset.
2. `currentParticipants`: permanent variables, usually assigned when the participant joined. These include the `name`, `nickname`, and `contact`

To specify these variables, the following syntax is used:

1. `userData`: `varname.u` :: Example: `myType.u`
2. `currentParticipants`: `varname.p` :: Example: `nickname.p`

You can use the following shortcuts:

- `*` or `*.P` : use all variables in currentParticipants

Notes:

- In addition to variables specifed in a variable list, the value of `message_options` `control` variable is always written to the `<message>` section of CAP files.

Specifying variables in `<reference >`
`varlist` should contain a list of reference variables to include in the XML data file to be created.

Or, you can use `*` to use all variables in the referenceData. If * appears, then ignore everything else!
Specifying variables in `<users>`
There are several kinds of variables that the `varlist` in `<users>` can refer to:

1. `userData`: variables either read from a parameters file (using getParams). Typically, theare are variables assigned to a participant from a reference dataset.
2. `currentParticipants`: permanent variables, usually assigned when the participant joined. These include the `name`, `nickname`, and `contact`
3. `resultsData`: Current results for this participant.

To specify these variables, the following syntax is used:

1. `userData`: `varname.u` :: Example: `myType.u`
2. `currentParticipants`: `varname.p` :: Example: `nickname.p`

You can use the following shortcuts:

- `*` or `*.P` : use all variables in currentParticipants
- `$varname` : A space-delimited list of variables stored in the varname control variable
- `newname=varname.P` : Rename the variable (.P or .U)

Note

- if you specify a variable more than once (say, in a shortcut and explicitily), the extra reference will be ignored (thus, specifying a variable once is no different than specifying it a zillion times).
  Well... there is one exception. If you rename (as described below), you can:
  1. specify multiple output variables with the same value (i.e.; that use the same userData variable)
  2. use the value of a later variable (in the list). This may be useful if you use one of the * syntaxes

Renaming
Consider the default names given to variables saved in client accessible parameter files:

for variables from ... | the naming convention is ...
--- | ---
`userData` | the same variable name is used. Example: a `GROUP1` userData variable is saved as `GROUP1`
`currentParticipants` | a `_` is prepended to the varible name. Example: a `REFID` currentParticipants variable is saved as `_REFID`
`resultsData` | a `.round` is appended to the variable name, where round is the round the results apply to. Example: a `PAYMENT` for round `4` is saved as `PAYMENT.4`

You can override these defaults by using the following syntax:
nameToUse=varname
Where `varname` is as specified above, and `nameToUse` is the name to use in the parameter file

For example, you might want to save the variable as an array element. Thus,

GROUP[1]=GROUP1
would save the `GROUP1` userData as `GROUP[1]`

Note:

- When renaming resultsData, the appending of the round-number is suppressed. If you want the round number append a ".R" to the newname.

  Examples

  1. `PAYMENT[1].R=Profit1.r` (round number will be appended)
  2. `TOTPAYMENTS=AllProfits.r` (round number will not be appended)
- Renaming does not effect values, or variable names, stored in the control panel `userData`, `currentParticipants`, or `resultsData` databases. It merely sets the name used in the client accessible parameter files. Which means it changes the name used (by the clients) to store these values.

Using `*`
As a shortcut, you can use the `*` in varlist to match multiple variables.
Syntax:
<users >list of variables, some of which contain * </users>

Examples

allSum.R status[*].R earn[*].R cost*

-   Assume the following variables have been saved to `resultsData`: `status[1] status[2] earn[1] earn[2] allSum allLoss`
- Then:
  <users > allSum.R status[*].R earn[*].R </users>
- will have the same effect as specifying:
  <users> status[1].r status[2].r earn[1].r earn[2].r allSum.r </users>

- Assume the following variables have been saved to `resultsData`: `status[1] status[2] earn[1] earn[2] allSum allLoss`
  and `usersData` contains:`cost1 const2`
- Then:
  <users> allSum.R status[*].R earn[*].R cost* </users>
- will have the same effect as specifying:
  `<users> status[1].r status[2].r earn[1].r earn[2].r allSum.r cost1 cost2 </users>`

Note: * can not be used with renaming.
Examples of <users> varlist

- The following 2 do the same thing:
  * *.P *.R *.D
- In the following, note the use of "renaming":

  refid.r contact.r group1 group2 fee1 fee2 profit[1]=payout1.r profit[2]=payout2.r totAccepted.r
  Note that the renamed variables above will not the round number appended!
- If you wanted to append the round number, use the following

  refid.r contact.r group1 group2 fee1 fee2 profit[1].r=payout1.r profit[2].r=payout2.r totAccepted.r
- The following is similar, but uses *

  refid.r contact.r group* fee* profit[*].r totAccepted.r
- Use variable lists stored in a control variable

  $rVarsUse $uVarsRead

Technical notes
The participant-specific data will look like:
<users> <VARS>_NAME,_CONTACT,_NICKNAME,_REFID,GROUP[1],FEE1,GROUP[2],FEE2,GROUP[3],FEE3,SILLYNAME,PROFIT.3,ACCEPT.3</VARS> <VALS > <user id="39040336673509324">vc,,301,1,1,2,2,2,2,SHE,10.3,NO </user> ... </VALS> </users>
Notes that:

- non-standard XML is used: attributes and nodes are not used for each variable. This is done to shrink the file size, hence speed up throughput.
- Actually, each value (i.e.; the "301") in the above will be "escaped" (to avoid problems with commas, tabs, etc).
- currentUser variables are always prefaced with a _
- resultsData variables always end with a .round, where round is the round these results are from.

Appendix 5: Functions
Functions that work with resultsData
List of functions ...

`readResultsVar(clientid,avar,adefault)`
  Read the `avar` variable for the `clientid` client. Return `adefault` if avar or clientid does not exist

`saveResultsVar(clientid,avar,aval)`
  Save `aval` to `avar`, for client `clientid`. `avar` can look like array element, such as `STATUS[2]`.
  - These can then referenced in postCAP by using a syntax of `STATUS[*]`.
  - However, `readResultsVar(aclient,'STATUS',0)` will not return a `STATUS` array.
  - Thus, `saveResultsVar` (and `readResultsVar`) treats array element names as goofy looking names (though these goofy looking names are treated as actual array elements when a client reads a client-accessible parameter file).

  Special cases:

  - `saveResultsVar()` will clear (intialize) `resultsData`.
  - `saveResultsVar('','ROUND',iround)` will set the "ROUND" (in resultsData) to be `iround`
  - the `clientikd` can be `*`
  Shortcut:
  - If `clientID` is an array of IDs, then for each ID specified in `clientID`, set all of the variables (that are specified in `avar`) to the respective value of `aval`.
  - If `aval` is a constant, use this constant value.
  - Otherwise, `aval` must be an array with the same length as `clientId`.
  - Note that all the variables will recieve the same value (the `aval` value).
  - To specify different values for different variables requires seperate calls to `saveResultsVar`.
  - if `avar` has a `vname[*]` synatx, then save all elements of a `aval` array
    Example:
  - If `profit[1]=103` and `profit[2]=74`, then for client `aid`
  - `saveResultsVar(aid,'myCash[*]',profit);`
  - is the same as specifying:
    `saveResultsVar(aid,'myCash[1]',profit[1]);
    saveResultsVar(aid,'myCash[2]',profit[2]);`
  - Note that undefined elements (such as `profit[0]`) will be skipped

`saveDOResultsVar(z,idVar,varlist,defval,noast)`
  Save all records in the `z` dataObject to the `resultsData` control panel database.
  - `z`: a dataObject (created with `createDO`)
  - `idVar`: the variable containing the ID (to use when specifying the client id). If not specified, or `0`, use `ID`
  - `varlist`: space delimited list of variables to save. If not specified, save all the variables (except for the idVar). If `*`, save all the variables
  - `defval`: default value to use (if a variable is undefined)
  - `noast`: if `1`, then do not save values from an entry with an ID of `*`.
  Note that constants, will not be written to `resultsData`. However, if `noast` is not equal to 1, then values defined for the `*` ID will be save (as `general` results).

`aval=getVarInfo('varname.g','value')`
  Reat the `varname` general variable, and save its value to `aval`.
  To read a control variable, use `varname.c`.

Functions that work with workData and submittedData

`extractRoundData(whichData,iround,avars,defval,clist)`
  Extract data from workData or submittedData: return data on all clients who participated in a chosen round.
  - `whichData`: which dataset to extract from -- `w` or `s` (w for workData, s for submittedData).
  - `iround` : round number to extract data from.
  - `avars`: a variable name, or a space delimited list of variable names. The values of these variables will be returned in a matrix. If just one variable is specified, return values in an array, if more than one variable a matrix is returned (each column of the matrix corresponds to a variable).
  - `defval`: Use `defval` when a variable is missing : when it has not been defined for a client (in this round). Note that it is quite possible for a given client to have different variables specified in different rounds.

    If `defval` is not specified, then when a missing variable is encountered, the client is dropped even if some other variables are not missing.

  - `clist` Optional. If specified, return values for all clients listed in `clist`. If a client listed in `clist` did not participate in the `iround` round, return the `defval` value for all variables (for this client). Note that if you can specify a `clist` you must also specify a `defval`.

  Example:

  if 6 clients are enrolled and each of these clients has a `X1` variable in all the rounds, then
  `vv`
  in
  `vv= extractRoundData('s',3,'X1',0)`
  will have 6 elements (note that a `defval` of `0` is specified, but is not needed).
  Notes:
  - Special values of `avar`:
  - `$ID`: the client ID
  - `$ROUND`: the round number
  - `=aVal` : `aVal` is always returned. For example, `=3` will always yield a `3`.
  - You can use * as the `iround`; it is treated the same as a numbered round (the * round is used for generic values that apply to all rounds, for a given client).
  - Note that for `submittedData`, the "final" values are returned (intermediate records are ignored).
  - If there is no `iround` round, return an empty array (a javascript `[]`).

`extractClientData(whichData,iclient,avar,defval,rlist)`
  Extract data from workData or submittedData: return data on all rounds that a chosen client participated in.
  - `whichData`: which dataset to extract from -- `w` or `s` (w for workData, s for submittedData).
  - `iclient` : client ID to extract data from.
  - `avars`: a variable name, or a space delimited list of variable names. The values of these variables will be returned in a matrix. If just one variable is specified, return values in an array, if more than one variable a matrix is returned (each column of the matrix corresponds to a variable).
  - `defval`: Use `defval` when a variable is missing : when it has not been defined for a round (for this client). As noted above it is quite possible for a given client to have different variables specified in different rounds.

    If `defval` is not specified, then when a missing variable is encountered, the round is dropped even if some other variables are not missing.

  - `rlist` Optional. If specified, return values for all rounds listed in `rlist`. If the `iclient` client did not participate in a round listed in `rlist`, return the `defval` value for all variables (in this round). Note that if you specify a `rlist` you must also specify a `defval`.

  Example

  If 10 rounds have passed, and ID `1512` has provided a `X1` variable to all rounds, then
  `vv` in
  `vv= extractClientData('s','1512','X1')`
  will have 10 elements.
  Notes:
  Special values of `avar`:
  - `$ID`: the client ID
  - `$ROUND`: the round number
  - `=aVal` : `aVal` is always returned. For example, `=3` will always yield a `3`.
  - You can use * as the `iclient` (the * client ID is used for generic values that apply to all clients, for a given round).
  - For `submittedData`, the "final" values are returned (intermediate records are ignored).
  - If there is no `iclient` client, return an empty array (a javascript `[]`).

`extractRoundList(whichData,iclient,noast)`
  Return an array containing a list of rounds.
  - If `iclient` is not specified, or `iclient='*'`, return a list of all the rounds.
  - If `iclient` is specified, this a list of all the rounds that client `iclient` participated in.
  - If `noast='*'`, remove a `*` entry from a the array (the * entry is the "all rounds default" value)

`extractClientList(whichData,iround,noast)`
  Return an array containing a list of client IDs.
  - If `iround` is not specified, or `iround='*'`, this is a list of all the client IDs.
  - If `iround` is specified, this a list of all the client IDs who participated in round `iround`.
  - If `noast='*'`, remove a `*` entry from a the array (the * entry is the "all clients default" value)

Functions that work with dataObjects
A `dataObject` is similar to a multi-dimensional array (a matrix), with a lot more structure. Roughly speaking, a `dataObject` is just a javascript object,that has a number of methods that can be used to modify, subset, and combine information stored in the object. This information consists of arrays and constants. The arrays contain information on entries (you can think of an entry as a record describing one participant).

Typically one creates a dataObject, adds arrays that contain data on entries (let's call these data arrays) and adds constants (that are the same for all entries). You can then access this information using standard javascript notation.

For example, assuming you have created a `z` data-object, and have added the `price` data-array and the `maxCost` constant, you could compute a `totProfit` by using:
`for (var i=0;i<z.PRICES.length;i++) totprofit=totprofit+(z.PRICES[i]-z.MAXCOST)`
But the above example is actually clumsy: it is usually a lot easier to use methods associated with data-objects, along with the statistical functions, to perform this kind of repeated calculation.

#### Constructor and methods

Data-objects must be created with a special constructor function. Once created, data should be added, removed, and otherwise manipulated by using the methods listed below.

The constructor:

`zobject=createDO(nlen)` or
`zobject=createDO(idsArray)` or
`zobject=createDO(varray,nameList)` or
  Call this to create a `dataObject`

  The first two variants of `createDO` takes one variable: either the number of entries, or an array containing a list of entry ids.

  1. In the first case, this is the length of the `data arrays` that will be added using the `add` (and other) methods of `dataObjects`. Adding an array that has a length not equal to this length is an error. If you use this technique, entry ids will be generated and assigned to each entry.
  2. In the second case, the length of the `idsArray` is the length of the `data arrays` (with the same rules mentioned above). The entry ids will be the values specified in the `idsArray`.

  The third variant takes 2 arguments. This variant is basically a short cut that allows you to quickly create a data object that has one record, and to load values into this record.

  - `varray`: an array containing K values (0,1,...K-1)
  - `nameList`: a string containing K variable names in a space delimited list
  Example:
  `var vlist=[1,3,451]; var z=createDO(vlist,'V1 V2 V3')`

  After using `createDO`, you can use the `add`, `addConstant`, `extract`, etc methods (of the object created by `createDO`) to maninipulate it.

#### A note on entry ids

  As will be discussed below, dataObjects are designed to facilitate adding data to existing entries, extracting entries to form new data objects, and recombining data objects in a manner that combines variables that belong to the same entry. This all requires the use of an `entry Id` -- which is unique to each entry in the data object. Entries,in different data objects, with the same entry ID will be combined into a single entry when these data objects are combined (or when one data object is used to update a second data object).

  Note that these entries are retained when a new dataObject is created by extracting entries from an existing data object. This means that these descendents of a dataObject can be used to update values in the parent data object. If this is the kind of manipulation you will be doing (say, starting with a single large data object that may be updated in various ways), then use of the first method (the `nlen` method) is sufficient.

  However, if you will be combining data from different sources (say, from `results` and from `submitted` data), you will need to use the `idsArray` form, and specify the save `idsArray` for each of the several data objects you create (which you will want to combine into a larger data object).

  Examples: `z=createDO(8)`
  `zx=createDO([1251,552,661,212])`

#### More on entry ids

  When the data object is created, an `nentries` field is added. It contains the number of records currently stored in the dataobject.

  Example: `zx=createDO([1251,552,661,212])`
  `zx.nentries` will equal 4.

#### Variable names

  When you add a variable to a dataobject, the name used is case-insensitive. However, if you use a dot notation (i.e.; `z.VARNAME`) to extract an array from a data object, you must use a capitalized version of the variable name.

Methods of dataObjects created with `createDO`.

`add(vv,vname)`
  Add/replace a data array (or several data arrays) to a dataObject.
  This basically adds (or replaces) variables to existing records in a data object. When adding, you can think of it as a column concatentate. When replacing, think of this as a column overwrite.

  A data array is a javascript array that must have a length equal to the nlen used when creating the dataObject (or to the length of the `idsArray`). Note that data is added to entries in order: so the first row (row 0) of the data arrays is used for the first entry, etc.

  - `vv` : the array, matrix, or constant value to add
    - You can specify several data arrays at once if `vv` is a matrix (a multi-dimensional array ).
    - Or you can specify a constant (`vv` is a number or string)
    In either case, the number of rows (in the array or matrix) must be equal to the number of records in the data object.

    Short cut: if `vv` is a constant (say, `0`), use this constant value (for all entries for the arrays specified in `vname`). Note that you can also use `update` to replace (but not add) a data array with a constant value.

  - `vname`: name of the array, or a space delimited list of names.
    a Notes:
  - that if `vv` is a matrix, its columns correspond to the names in `vname`.
  - If a row (say, row m) of vv is not long enough (if it does not have enough columns), `undefined` is used as the value when necessary.
  - As noted above, if vv is a constant, its value is used for all values of the data arrays specified in `vname`.
  Example: `z.add(price,'MyPrices')` (where `z` is a dataObject created with `createDO`)

  Caution: Using simple methods to add data to a data-object, while not a javascript error, will mess up the internal accounting of the dataObject. So don't do it!
  For example:

  - `z.add(x1,'costs');` is okay
  - `z.COSTS[1]=z.costs[1]*3` is okay
  - `z.COSTS=x1` is not okay

`vv=z.get(vnames,errmess)`
  Retrieve a data array (or arrays) from a data object.
  - `vnames`: a space delimited list of variable names. These list what variables to retrieve. The variables listed in `vnames` are case insensitive.
  - `errmess`: optional. If specified, display this error message if non-existent variable is requested. If not specified, a generic error message is displayd
  - `vv` : an array, or a matrix (a multi-dimensional array) returned. The length of `vv` will equal `z.nentries` (the number of records in `z`). If one of the variables does not exist, an undefined is returned.

  Note that `get` is an alternative to retrieving an array using: `vv=z.VARNAME`.
  It has a few advantages:
  - You can use case insensitive names
  - If you include a name that does not exist (in the data object), an error message will be displayed (this can be suppressed by using `errmess=''`).
  - You can create matrices
`newz=z.math(anEquations,varList,valList)`
  Process equations that use dataObject variables. The equation is used to create new, or update existing, variables stored in a data object; where these variables can be `dataVariables` (nentries x 1 arrays), `scalarConstants` (single value), or `arrayConstants` (m x 1 arrays).

  The equation(s) can contain the usual dSurvey mathematical and logical operators, and generic functions.
  Moreover, it can also refer to variables and constants specific to the dataObject (i.e.; that are stored in `z`). In particular, it will process these a-row-at-a-time. Thus,
  `math` provides a convenient way of doing math on arrays!
  Such as arrays that contain information from all of a session's participants!

  Syntax: | `anEquations`: A string containing one (or more) equations to process. Each equation should have a syntax of: `avar= an equation` . You can specify one or more equations -- use semi-colons (`;`) to seperate equations. `varList`: optional. Used with `valList`. A space delimited list of constants to add to the dataObject (which can then be referred to in the equation). These will be removed when equation processing completes. `valList`: optional. Used with `varList`. A space delimited list of values, or an array. These are the values that will be used with the corresponding variables in `varList`.
  Note that the dataObject (`z`) will not be modified -- a new dataObject (`newz`) is created that contains the results of processing `anEquations`.

  #### Special commands in `anEquations`:

  Several special commands can be used to start `anEquation`: comment, select, if, by, keep, drop, and return

  - `COMMENT some stuff` or `// some stuff` : Ignore the equation

  - `SELECT anEquation` : Conditional dropping of rows
    `anEquation` should be a logical equation (that returns 0/1 values, say by using EQ and other logical operators).
    Every row that has a value of `0` will be dropped.
    In other words...

  - `SELECT` is used to select observations: simply start the equation with an `SELECT`, and follow this with a logical equation.
  - Subsequent equations that use statistical functions (such as `.SUM()`) will not include these dropped observations (see SELECT RESET below for more details).

  Example | constant costSumBefore = cost.sum() SELECT age gt 50 constant costSumAfter = cost.sum() The first costSumBefore constant will contain the sum of costs across all observations. Then, all participants with `age` le 50 are dropped, and `costSumAfter` is computed from these remaining participants.

  Note: You can also use the .IF(alist) function to select rows of a `dataVariable` (or `arrayConstant`) and save them to an `arrayConstant`.
  - `SELECT reset defVal` : restore all observations, combining with current values
    `SELECT reset` can only be used after one (or more) `SELECT anEquation` statements. `SELECT reset` will restore the dataObject from before the first `SELECT anEquation`, and combine the results from after this `If`. Thus, the resulting dataObject will have all its records, but with changes to the records that are currently selected (that is, the records that the preceding `SELECT` statement(s) have selected).

    `defVal` is used to assign values to variables in records that are not in the current set of records (the post `SELECT anequation` records).

    Basically, `SELECT reset` is an alternative to using `.math()` and `.combine()` in a sequence of calls.

    Example: | z=z.math('nn=_n(); SELECT nn le 3 ; goo=1 ; SELECT reset 9 ; SELECT nn ge 5 ; goo=2 ; SELECT reset '); is a shortcut for: z=z.math(' nn=_n() ') ; ztmp=z.math(' SELECT nn le 3 ; goo=1 '); z=z.combine(ztmp,9); ztmp=z.math(' SELECT nn ge 5 ; goo=2 '); z=z.combine(ztmp); In either case,assuming z.nentries=7, then nn = [0,1,2,3,4,5,6] goo = [1,1,1,1,9,2,2] z=z.math('nn=_n() ; SELECT nn le 3 ; goo=1 ; SELECT reset 9; SELECT nn ge 5; goo=2'); will yield: nn = [5,6] goo = [2,2] z=z.math('nn=_n() ; SELECT nn le 6 ; goo=1 ; SELECT nn ge 5 ; goo=2 ; SELECT reset ') ; will yield: nn = [0,1,2,3,4,5,6] goo = [undefined,undefined,undefined,undefined,undefined,2,2]

  - `IF logicalEquation THEN anEquation` : conditional execution of `anEquation`
    `logicalEquation` should be a logical equation that returns a single value (if it returns an array, only the first value is used). If the value is greater than 0, the `anEquation` is processed.
    Otherwise, `anEquation` is skipped.

    Hint: To process multiple equations after the `then`, seperate them with `~` characters!

  - `BY byVarName` :
    Create statistics on subsets of rows; where rows in a given subset have the value value of t `byVarName`.
    Syntax:
    BY varname ; equations using stat functions ; BY 0
    All the equations using statistics functions between the `BY byVarName` and the `BY 0` will be processed in subsets. That is: statistics will be computed seperately for subsets of the values. In particular, each subset is comprised of rows with the same value of the `byVarName` variable.

    For details, see the discussion under statistical functions
  - `KEEP list of variables` : keep these variables from the dataObject (drop all others)

  - `DROP list of variables` : drop these variables from the dataObject (keep all others)

  - `RETURN list of variables` : stop processing equations and create the dataObject
    - `RETURN ;` (with no list of variables) means return the newly created data object, with all of its variables.
      For example: `IF logicalEquation ELSE RETURN ;` allows you to selectively stop processing the equations (and return the dataObject without any further modifications).
    - `RETURN variable list` If a list of variables is specified, than do not return a dataObject!
      Instead, return a constant, or an array, or an array-of-arrays.
    - If only one variable is specified, and it is a scalarConstant, return a scalar
    - If only one variable is specified, and it is a dataVariable or an arrayConstant, return an array
    - If more than one variable is specified, return an array of arrays -- each row of the array contains the value of the corresponding variable specified in the list of variables. Each row may contain a scalar variable, or an array variable

    Examples of special commands: | comment this is for testing ; SELECT bid gt 100 ; IF income.sum() gt 100000 then topIncomes=income.min(100000) ~ const smallIncomes=0 ; KEEP var1 var2 var3 ; DROP var2 ; newbid=bid * weight ; return newbid ;

    #### Modifying a subset of records

    When specifying `newz=z.math(...)`, remember that `z` will not be modified -- a new dataObject (`newz` is created that contains the results of processing `anEquations`.

    As discussed above, you can use combinations of SELECT and SELECT reset to change just a subset of the records.

    However, as an alternative you can use a combination of calls to `.extract()` and `.math`. For example:
    zuse=z.extract(rowsToExtract); zuse2=zuse.math(anequations) z.update(zuse2) The above will use undefineds for new variables in records that do not appear in zuse2. To set the defaults to something else, say `0`, you can use. zuse=z.extract(rowsToExtract); zuse2=zuse.math(anequations) znew=z.combine(zuse2,0)
    Specifying a variable
      The variable specified in `avar` can be a data array (`dataArray`) or a constant. Constants can be scalar (`scalarConstant`), or an array of any size (`arrayConstant`).
      Reminder: the final result is that variables (whether arrays or constants) will be added to the `z` dataObject.
  - `varname= anEquation` : add a `dataArray` with the name of `varname`
  - `constant constname1= anEquation` : add a scalar constant with the name of `constname`
    or (a shortcut)... `constname1.C=anEquation`
  - `array arraynameX= anEquation` : add an `arrayConstant` with the name of `arraynameX`
    or (a shortcut)... `arraynameX.A=anEquation`

    If you just want to use a constant variables for temporary storage, start the name with an underscore (i.e.; `_tempConst`). When the equations are completed, these constants will be deleted. Note that you can also use `DROP` to remove data variables and constants.

    Examples.. | newPrice=anEquation constant _temp1 = anEquation oldPrice.c = anEquation constant totalProfit=anEquation array theMaxs= array(100,200,300,100)

Specifying the equation(s)
  Like other dSurvey equations, the equation (`anEquation`) can contain:
  - Numbers (`1,15.15, -135, etc.` )
  - Math operators (`+, *, etc.`)
  - Logical operators (`eq, gt, etc.`)
  - generic functions (`Math.max, Math.sqrt()`)
  - control variables (`newFactor.c, etc.`)
  - general variables (`cost_min.g, etc`)

  But of perhaps greater interest, it can also refer to variables contained in the dataObject, just by using the variable (array variable or constant's) name.

  Example | Assume that `z` has the following variables already defined: `cost`, `bid`, and `income`. And it has the following constant defined: `tax` profit=(bid - cost) ; newIncome=income - (profit * (1-tax) )
  Numeric example: | Constants: `tax=0.1` and 3 records with variables... Prior Variables New Variables Bid Cost Income profit newIncome 100 10 1000 70 947 120 40 500 80 428 220 120 1500 100 1410

  Even better, there are a number of statistical operations that can be readily used. These will return a single value. However, this single value (as with number, control variables, etc) will be expanded (it will be used in every row of the dataset). Or, you can use a `BY` variable to compute statistics on subsets of a variable, where the subsets are defined by the `BY` variable.

  Statistical functions are specified using: `varname.funcname(args)`. Note that many of the functions do not take any `args` The functions will operate on all values defined (across all rows) of the `varname` variable. Or, if a `BY` variable is specified, the function will operate on subsets of values, where subsets consist of rows with the same value of the `BY` variable.
  Statistical functions in `math` | `MAXVAL()`: minimum value. `MINVAL()`: maximum value `MEDIAN()`: median value `MAXVAL_ROW()`: row containing the minimum value. `MINVAL_ROW()`: row containing the maximum value `MEDIAN_ROW()`: row containing the median value `SUM()`: sum across all rows `MEAN()`: mean value `VAR()`: variance `SD()`: standard deviaion `MISS()`: number of missing values (note that missing values are ignored in these functions) `LENGTH()`: number or rows `NOTMISS()`: number of non-missing values `ORDER(nth)`: the `nth` largest value. Thus, `ORDER(0)` is the same as `MINVAL()` `ORDER_ROW(nth)`: row containing the `nth` largest value. Thus, `ORDER_ROW(0)` is the same as `MINVAL_ROW()` `#EQ(aval)`: number of values equal to aval `#NE(aval)`: number of values not equal to aval `#GT(aval)`: number of values greater than to aval `#GE(aval)`: number of values greater than or equal to aval `#LT(aval)`: number of values less than to aval `#LE(aval)`: number of values less than or equal to aval
  Using BY | The `BY byVarName ; .... ; BY 0` syntax is used to instruct the statistical functions to compute (and return) seperate values for subsets of rows -- with each row (within a subset) having the same value of a `byVarName`. Using `BY` `BY` will only effect the statistical functions (such as `MAXVAL`). It will not effect other operations, such as addition of two data variables, or the special functions. When `BY` is enabled, a vector is returned that will have the same length as the target variable. Example: in `myCost.mean()`, the statistical function is `mean` and the target variable is `myCost`. The statistical function is applied to subsets (of the values of the target variable). Each subset is created using rows that have the same value as the `byVarName` variable. The value computed from a subset is assigned to each row, of the newly created array, that match the rows of the `byVarName` variable. The BY_VALS() and BY_STATS() special functions can be used to extract subset-specific values. Example... Assume a data object with 8 rows; that contains the following data variables (with values)... myGroup= 1, 2, 1, 1, 2, 1, 1, 2 myCost= 25, 15, 30, 20, 10, 30, 25, 5 Then, BY myGroup ; myGroupAvgCost=myCost.mean() ; BY 0 ; Will create two subsets: G1 (myGroup=1): 25 30 20 30 25 G2 (myGroup=2): 15 10 5 The mean of each subset is computed, mean of G1= 26 mean of G2= 10 and these means are assigned to the appropriate rows of the newly created data variable: myGroupAvgCost=26, 10, 26, 26, 10, 26, 26, 10 Note that you can have several statistical functions between a `BY byVarName` and a `BY 0` -- each will use the same subset of rows. You can also include other equations -- although they won't be effected by the `BY`.
  Special functions: | There are a few functions which will create a new `dataArray` (or `arrayConstant`) that do not refer to existing variables. Note that in the following, `nentries` refers to the number of rows in the dataObject (the length of the `dataVariables`). _N : newVar= _n() : creates an array containing the the row number. Thus, the first row is given a value of 0, the second row gets 1, and so on. Thus, newVar will be nentries x 1. newVar= _n(start,end,by) : an array containing a sequence of values. Thus, newVar will be m x 1, where m can be any size. start : starting value end : end when sequence is greater than this value by : step size. 2nd value will be start + by. If not specified, by = 1. Example: array newVar = _n(0,2,4) then newVar = [0,2,4] Note: when using arguments (to create an m x 1 array), you will typically use array to create an arrayConstant. RANDOM : newVar= random(min,max,m,seed) : Generate uniformly distributed random numbers. If no arguments, return an nentries array with random values between 0 and 1 min and max are the minimum and maxium of the random numbers If not specified, min=0 and max=1. m is the number of values to generate. Thus, newVar will have length of m. If m is not specified, m=nentries. seed is optional -- specify a value to return the same set of random values If not specified, a different set of random values is returned Note: when using the m argument (to create an m x 1 array), you will typically use array to create an arrayConstant. NORMAL : newVar= normal(mean,sd,m,seed) : Generate normally distributed random numbers. mean is the mean of the random values, sd is the standard deviation. If not specified, mean=0, sd=1 m is the number of values to generate. Thus, newVar will have length of m. If m is not specified, m=nentries. seed is optional -- specify a value to return the same set of random values If not specified, a different set of random values is returned Note: when using the m argument (to create an m x 1 array), you will typically use array to create an arrayConstant. BY : anarray = by_vals() and by_stats() Returns a result of the most recent statistical function. This should ONLY be used inside of a BY byVarName; ... ; BY 0 * BY_VALS(): returns an array containing the unique values of byVarName, in the order these values appear * BY_STATS() returns an array containing the statistic for each subset. Note that the nth row returned by BY_STATS() corresponds to the nth row returned by BY_VALS(). Example: by myGroup ; groupVals.A=by_vals(); avgCosts=myCosts.mean(); groupAvgCosts.A=by_stats() by 0 ; Using the above example, these two array constants will equal: groupVals=1, 2 groupAvgCosts=26, 10 Note that everytime you call a statistical function, the value returned by by_stats() will change -- so you should use by_stats() immediately after the calling statistical function of interest.
  Creating `arrayConstant`s or `dataVariables` | A number of functions can be used to create an `arrayConstant`. Note that the length of an `arrayConstant` is not necessarily the number of records in the dataObject. ARRAY : array myArray=array(n0,n2,..,nN) -- create a N-1 element arrayConstant n0 can be an integer, a 'quoted strnig', the name of a control variable, the name of a scalarConstant, or the name of an arrayConstant. If you specify an arrayConstant, all the values in the arrayConstant will be inserted. Example: if cArray = [10,25,50] ipos = array(5,cArrray,100) then ipos = [5,10,25,50,100] FIX : array ivar=dvar.fix(ndec) -- Set number of significant decimal digits. If an element is non-numeric, it is not changed. dvar.fix() and dvar.fix(0) mean convert to an integer (drop decimal fraction) Example: if dvar = [15,5.34,20,12.125,'cat'] ipos = dvar.fix(1) then ipos = [15.0,5.3,12.1,'cat'] Note: since fix() returns an array of the same size as dvar, it is safe to use it to create new `dataVariables`. FREQ : array myCount = dvar.freq(alist) -- frequency counts of values in dvar A frequency count of every value in alist is computed. Values in dvar that are not in alist are ignored. As with the argument list in array(), alist is a comma delimited list that can contain a mixture of numbers, names of previously created arrayConstants, scalarConstants, and control variables. For example: dvar.freq([n1,n2,...]) -- only look at values in this array (i.e.; n1,n2). N1 etc must be numbers (not variables) dvar.freq[alist) -- alist is an array of values. For example, as created using .unique() dvar.freq() -- frequencies of all unique values in dvar, returned in order of these values. Note that array foo=dvar.freq() is the same as array uu=dvar.unique(1) ; array foo=dvar.freq(uu) IF : array ume=dvar.if(alist) -- alist should contain as many elements as in dvar. useme will contain only the values of dvar for whom the corresponding value in alist is greater than 0 Example: if dvar = [15,20,25,35] alist = [0,0,1,1] isin = dvar.IF(alist) then isin = [25,35] IN : array isin=dvar.in(alist) -- For each element in dvar, returns a pointer to alist. Note that the length of isin will be the length of dvar. If more than one match, returns the first one Example: if dvar = [15,20,25,35] alist = [10,20,30,35,40,45] isin = dvar.IN(alist) then isin = [-1,1,-1,3,-1,-1] Note: since in() returns an array of the same size as dvar, it is safe to use it to create new `dataVariables`. KEEP : array useme=dvar.keep(alist) -- alist specifies elements of dvar to keep alist should consist of row numbers, with 0 meaning the 0'th row of dvar. alist can be any size, and can contain repeats. The maximum value of alist must be less than the size of dvar. Note that SLICE is used to specify a range of elements to retain. Example: if dvar = [15,20,25,35] alist = [0,3] isin = dvar.KEEP(alist,1) then isin = [15,25,20] Example: array dvar = array(100,200,350,450,502,601,777,812,955) ; array chvar = array(10,11,12,13,2,1,16,5,18) ; array gt9 = chvar gt 9 ; array newch0 = chvar.if(gt9) ; array newD0 = dvar.if(gt9) ; array nth = _n(0,8) ; array ikeep = nth.if(gt9) ; array newCh = chvar.keep(ikeep) ; array newD = dvar.keep(ikeep) ; then gt9 = [1,1,1,1,0,0,1,0,1] nth = [1,2,3,4,5,6,7,8,9] ikeep = [0,1,2,3,6,8] newCh = [10,11,12,13,16,18] newCh0 = [10,11,12,13,16,18] newD = [100,200,350,450,777,955] newD0 = [100,200,350,450,777,955] MAX : array dnew=dvar.max(amax) -- return maximum of amax and the value or dnew=dvar.max(amax) Example: if dvar = [15,5,20,12,35] newVar = dvar.max(18) then newvar = [18,18,20,18,35] Note: since max() returns an array of the same size as dvar, it is safe to use it to create new `dataVariables`. Note : dvar.maxVal() returns a single value (the maximum value of dvar) -- though it can create an array (with values equal to this single value). In contrast, dvar.max(aval) returns a unique value for each element in dvar. MIN : array dnew=dvar.min(amin) -- return minimum of amin and the value or dnew=dvar.min(amin) Example: if dvar = [15,5,20,12,35] newVar = dvar.min(18) then newvar = [15,5,18,12,18] Note: since min() returns an array of the same size as dvar, it is safe to use it to create new `dataVariables`. Note : dvar.minVal() returns a single value (the minimum value of dvar) -- though it can create an array (with values equal to this single value). In contract, dvar.min(aval) returns a unique value for each element in dvar. REPLACE : array dnew=dvar.replace(oldval,newval) -- replace values Replace all occurences,in dvar, or oldval with newval. Note: since replace() returns an array of the same size as dvar, it is safe to use it to create new `dataVariables`. SLICE : array useme=dvar.slice(i1,i2) -- i1, i2 specify the range of elements to keep i1 points to the first element to keep: i1=0 means the first element in the array i2 points to the last. Thus, this is an "inclusive" retain. If i1 is greater than the size of the array, keep no elements (return an empty array). If i2 is greater than the size of the array, or if i2 is not specified, keep all elements from i1 to end. Note that KEEP is used to specify exactly which elements to retain. Example: array dvar = array(100,200,350,450,502,601,777,812,955) ; array newd = dvar.slice(3,5) ; array newd2= dvar.slice(7) ; then newd = [450,502,601] newd2 = [812,955] SORT : array ipos=dvar.sort() -- For every element in dvar, returns its rank Example: if dvar = [15,5,20,12,35] ipos = dvar.sort() then ipos = [2,0,3,1,4] Notes: * the smallest element will have a value of 0. * the length of ipos will be the length of dvar. * since sort() returns an array of the same size as dvar, it is safe to use it to create new `dataVariables`. UNIQUE : array astuff=dvar.unique(isrt) -- return an array containing unique values of dvar isrt=0 : in non-sorted order ; isrt1=1 : in sorted order Thus, UNIQUE can be used to return a sorted list (or unique values). You can use arrays, constructed using the above, in math. Furthermore, you can use all the statistical functions, and the above functions. on arrays you create! Note that when using arrays, you should be careful that the lengths of the various arrays you combine are equivalent. syntax. Example (count how many records have a prodCost equal to the the 1st, 3rd, and 5th largest prodCost): array a1=array(0,2,4) ; -- creates a 3 element arrayConstant array a1=_n(0,4,2) ; -- another way of creating the above array pUni=prodcost.unique(1) ; -- assuming that prodCost (a dataVariable) has m unique values, a sorted m element array is created array pCount=prodcost.freq(pUni); -- compute frequencies for all these m unique values array pKeep=pCount.keep(a1) ; -- keep the the elements of pCount specified in a1 In this example, if m<5, an error occurs! constant tot135=pkeep.sum() -- sum up these selected frequencies
  Extracting a single value: | You can extract the value of a particular row, for a particular variable, by using: varname[nth] This will extract the `nth` row's value of the varname `dataVariable` or `arrayConstant`. Note that this is in order of appearance, not in sort order. |
  Examples: | constant mincost=cost.minVal() ; isGtMin= cost gt cost_min.g ; constant c_of_v = cost.mean() / cost.sd() ; excessCost= cost - mincost ; incomeShare= income / income.sum() ; gt10k = income gt 10000 ; newcost=mincost + (excessCost * random(0.6,0.8) ) ; iBigBid = bid eq bid[0] ; array sameAsFirst = bid.if(iBigBid) ; // note that .sum() can be used on an arrayConstant ; constant nSameAsFirst = sameAsFirst.sum() |

Notes:

- Equations (and variables) are case insensitive!
- You can specify control variables as arguments in the functions.

`addRecord(vv,vname)`
  Add a record to a dataObject. This basically adds a record to existing records in a data object.

  - `vv` : an array of values.
  - `vnames`: a space delimited list of variable names. These list what variables to assign these values to.
  This will add one row to the dataobject, using values stored in vv. `vname` should have the same number of variables (as words in a string) as rows in vv.

  Notes:

  - this is a "row concatentate". `vv` is expected to hold the values of several variables for one record.
  - If a varible in `vname` does not exist in the dataobject, a new variable is added to all existing records (with value set to `undefined`).
  - If a variable already defined in the data object is not specified in `vname`, a value of `undefined` is used.
`addConstant(vv,vname,isarray)`
  Add constants (value or array) to a dataObject

  - `vv :` the values(s) to add
  - `vname :` the of the variable to add, or a space delimited list of names
  - `isarray :` if 1, then vv is an array variable. Vname should be a single name -- an array constant will be saved.
    If `isarray` is not `1`, then `vv` should be an array with the same length as the number of names in vname. Each row of vv can be a string, number, or an array.

    Examples: `z.addConstant(156,'weight') ; z.addConstant([10,20,30],'max3',1)`

`astring= z.display(varlist,constlist,fix2,colwidth)`
  Make a string containing an HTML contents of the values in the dataObject
  - `varlist` is optional -- it is a list of the variables to display. If not specified, display all of the variables.
    If `0`; skip. If `*`, all variables.
  - `constlist` is optional -- it is a list of the constants to display. If not specified, display all of the constants.
    If `0`; skip. If `*`, all variables.
  - `fix2` is optional -- if specified, display only 2 digits to the right of the decimal point.
  - `colWidth` is optional -- if specified, it is the maximum width of a column (strings will be wrapped if necessary)

  - `astring` will contain HTML (a table) containing the values in the dataObject. You can then use `writeDoStatus` to display this string.

`extract(getList,vList,cList)`
  Create a new dataObject by extracting entries specified in getList.

  - `getList`: an array that points to which entries in the `dataObject` to use in the new `dataObject`. Thus, if `getList=[0,2,4]` then the first,third, and fifth entries will be extracted (more precisely, data from the 0, 2, and 4 rows of the data-arrays will be extracted).
  - `vList`: Optional. Space delimited list of the names of the data arrays to use (or an array containing these names). If not specified, extract all `data arrays` added to the dataObject.

  - `cList`: Optional. Space delimited list of the names of the constants to use (or an array containing these names). If not specified, extract all the `constants` from the original dataObject and add them to the new dataObject.

    Note that the new dataObject will contain an `nentries` constant that will be equal to `getList.length`

    Example: `znew=z.extract([1,2,4,5])`

    Shortcut: `extract()` will clone the data object -- all the variables and all the constants in the original data object will be copied to the new dataObject. For example: `zCopy=z.extract()` : `zCopy` will be a clone of `z`.

`extractID(idList,vlist,clist)`
  Create a new dataObject by extracting entries specified in idList.

  - `idList`: an array containing entry-ids of the entires to extract to the new `dataObject`. Thus, if `getList=[12,533,4213]` then the entries with ids of 12, 533, and 4213 will be extracted.
  - `vlist`: Optional. Space delimited list of data arrays to use. If not specified, extract all `data arrays` added to the dataObject (using `add`).

  - `cList`: Optional. Space delimited list of the names of the constants to use (or an array containing these names). If not specified, extract all the `constants` from the original dataObject and add them to the new dataObject.

    Note that the new dataObject also contains an `nentries constant`, that is equal to `idList.length`

`extractRule(vv,arule,aparam,vlist,clist)` This is somewhat deprecated -- consider using the math method.
  Create a new dataObject by extacting entries using the `arule`.
  - vv: an `nentries` lengthed array (typically a data-array from the data object) to examine
  - values in the `vv` are are examined using the `arule` and the `aparam`.
  - `vlist` is the space delimited list of variables to include in the new dataObject. Optional: include all variables if not specified, or if ''.
  - `cList`: Optional. Space delimited list of the names of the constants to use (or an array containing these names). If not specified, extract all the `constants` from the original dataObject and add them to the new dataObject.

  The following `arule`s are supported:

  - `large`: extract `aparam` entries that have the largest (numerically) values of `vv` If `aparam` not specified, extract the largest.
  - `small`: extract `aparam` entries that have the smallest (numerically) values of `vv` If `aparam` not specified, extract the smallest
  - `nth`: extract the entry with the `aparam` smallest (numerically) value of `vv` If `aparam` not specified, extract the smallest. Note that `0` is the smallest.
  - `eq gt ge lt le ne`: extract all entries with a `vv` eq, or gt, or etc. to `aparam`

    Example:

    `znew=z.extractRule(z.PRICE,'small',5)`
    is a shortcut for:
    `ilist=sortIndex(z.PRICE);
    ilist2=extractRows(ilist,[0,4],2);
    znew=z.extract(ilist2);
    `

`removeVars(vlist,iskeep)`
  Remove dataVariables, arrayConstants, and scalarConstants from the dataObject.
  - `vlist`: Space delimited list of variables to remove.
  - `iskeep`: Optional. If `iskeep=1`, then the variables in `vlist` are kept -- all other variables are removed!.
  Examples: `z.removeVars('TEMP1 X3 ') ; z2.removeVars('INCOME PROFIT AGE SUM_INCOME ',1);`

`update(z2,vlist,adef,ahint)`
  Update the dataObject using data in z2.
  in the following, `z` refers to the dataObject; as in `z.update(z2)`
  - `z2`: The dataObject containing data to use (data from z2 replaces, or adds to, data in `).`
  - `vlist`: Optional. Space delimited list of variables to use. These can be `dataVariables`, `arrayConstant`, or `scalarConstants`. If not specified, use all the variables (dataVariables, arrayConstants, and scalarConstants) specified in `z2`.
  - `adef`: Optional. Default value to use. If `z2` contains a variable not currently in `z`, then `adef` will be used (for this variable) in all records of `z` that do not have a matching record in `z2`. If not specified, `undefined` is used.
  - `ahint`: Optional. A string that will be used in error messages (to help the programmer figure out what data object has a problem).

  Note that z2 must either be a descendant of the dataObject (say, it was created with `extract`), or have been created using an `idsArray` that matched the `idsArray` used to create the dataObject.

  Example:

  znew=z.extract([0,1,2,3]) -- extract first 4 entries in z znew=znew.math(' BID2 = BID* 1000'); -- create a new variable named BID2 z.update(znew,'BID2 ',0) -- add var BID2: first 4 obs use values in znew; 5 ... get BID2=0

  Shortcut:
    if `z2` is a string or a number, then use the value of `z2` for all variables (in vlist) and all entries (in the dataObject).

    Example:`z.update(5,'status')` will:

    - If a `status` `dataVariable` exists in `z`: all `z.nentries` elements of the `STATUS` dataVariable will be set equal to `5`
    - If a `status` `dataVariable` does not exist in `z`: a `status` scalarConstant will be created in `z` and set equal to `5`

  Note: Note that you can also use `add` to add, or replace, a data array with a constant value.

`astat=updateRecord(anid,vv,avars,defval,idVar)`
  Update a record in the dataObject using data in the vv array.
  - `anid`: The value of the entry-id to update. Specifically, the first record that has an `idVar` variable with a value of `anid`.
  - `vv`: an array containing the values to update
  - `avars`: the variables to update (using the values in `vv`).
    If `avars` is not specified, then update the variables in order of appearance (as stored in `z['$varlist']`).
  - `defval`: optional. A value to use if `vv` is too short.
  - `idVar`: optional -- the variable name that `anid` should match. If not specified, use the `entry id`.
  - `astat`: status. If success, `1`. If an error, `0 error message` .
  Notes:
  - If `idVar` is not specified, then the entry id is used.
  - If `idVar` equals `0`, then `idVal` is the record number (where `0` is the first record).
  - If a variable specified in `avars` does not exist in the dataObject, the corresponding value in `vv` is skipped.
  - If vv is too short (more elements in `avars` then in `vv`, use `defval`. If `defval` is not specified, use `undefined`.
    Hint: use `[]` for `vv` to set all the `avars` variables (for this record) to `defval`
  - If there is no `idVal` of `idVar` (or of the entry id), nothing is done.
`combine(z2,defval,alist,idrop)`

  Combine the values in the dataObject with values from a different z2 dataObject; and return a new dataObject. For purposes of exposition, let's call the dataObject (of which `combine` is called as a method) `z1`.

  Combining attempts to match the entry-id of each entry in z2 with an entry in z1. If a match is found, updated z1 values used in the new dataObject. If not found, the entry is added to the new dataObject as is.

  - `z2` the dataObject to combine with z1. Note that values in z2 take precedence over values from z1.
  - `defval` a default value to use when no value exists (say, to use for Z1 records that do not have a match in Z2, for variables in Z2 that ar not in Z1).
  - `alist` is optional. It should be a space delimited string containing the names of the variables (`dataVariables`, `scalarConstants`, and `arrayConstants`) to include in the new dataObject. If not specified, all the variables in Z1 and Z2 are used.
  - `idrop`: Optional. If equal to 1, then `alist` is used as a list of variables to drop (all others are retained).

    Example: `znew=z.combine(zuse2,0,'BUSHELS ACRES TOTALYIELD ')`

    As noted above, if an entry exists in both Z1 and Z2, then for all variables defined in Z2, the Z2 value is used. In a sense, Z2 updates Z1 (but the results are saved in the new dataObject; and Z1 does not change).
    By entry exists, we mean that the entry has the same value ofthe entry-id. This typically occurs when the Z1 and Z2 dataObjects have the same parent (say, both were created using `Z0.extract(...)`), or when two data objects were created with the same `idsArray`.

    `combine` is often useful when you want to subset entries in a dataObject, process each subset differently, and then recombine the subsets. It is also used when combining data from different sources (say, from `results` and from `submitted` data), where each source contains different information on the same set of entries.

    The following table specifies how entries and variables are added to the new dataObject.

    | Variable exists only in z1 | Variable exists in z1 and z2 | Variable exists only in z2
    --- | --- | --- | ---
    Entry only in z1 | Use value from Z1 entry | Use value from Z1 entry | Use defval
    Entry is in both z1 and z2 | Use value from Z1 entry | Use value from Z2 entry | Use value from Z2 entry
    Entry only in z2 | Use defval | Use value from Z2 entry | Use value from Z2 entry

    Note: all the constants in `z2` are added to `z1`: either by overwriting existing `z1` constants, or creating new ones.

`save(aname)`

  Save the dataObject to a global storage array. This can then be retrieved at a later time (say, when a different custom function is run) with `retrieve`.

  `aname` must be a single word string, which is used to identify the dataObject (that is, it will be used to retrieve this dataObject in `retrieve`.

`z=retrieveDO(aname,exists)`
  Retrieve a dataObject from global strorage, that was stored by a prior call to `saveDO`.

  `aname` must match the name used when the object was stored with `saveDo`.
  `exists` is optional. If specified, and equal to 1, then a 1 is returned if a dataObject with this name exists in storage. Otherwise, return 0.