Analytic Toolkit Docs
From AnalysisWiki
Analytic Toolkit for Knowledge Forum
(July 16, 2002 -- v4.0 )
(See the Analytic Toolkit Reference for detailed information on the computation of measures.)
Introduction
The Analytic Toolkit provides summary statistics on activity in a Knowledge Forum database. It shows how many notes are in the database, how connected they are, how many notes a user has created, which views a user is working in, what percentage of the notes have been read, whether build-ons, keywords, references and other knowledge building features are being used, who has read and written new notes during a particular time period, and social interactions. It is intended to be used by database managers, researchers, and participants in the database.
Mechanics
The Analytic Toolkit is Web-based, and accessed with a browser through a URL. The Knowledge Forum database to be analysed can be located anywhere, provided there is Internet access to it. The URL for the Toolkit can be obtained from Learning in Motion or OISE/UT.
The Toolkit uses only simple HTML, for the most part. It uses some JavaScript, if available, but will work without it. Most Internet browsers are therefore supported. It is written as a set of CGIs in Perl.
The Toolkit will work on databases running on Knowledge Forum version 2, 3, or 4.
The Sign-On Page
The sign-on page for the Analytic Toolkit asks for a username and password, which are the same ones used for Knowledge Forum itself.
The sign-on page may also ask for the Database name and Port number. This is the full name of the database to be analysed, and the port number it is running on. If it is running on a Mac or Windows machine, the port number is 27319. If it is running on a Linux machine, the port number is the one specified in the Knowledge Forum Remote Servers file.
There is an option at the bottom of the sign-on page called "Skip Update". This is not normally selected. Its use is described below under "Updating of Toolkit Information".
Reports
After you sign on, you are given a choice of reports. Each report calculates and presents a particular set of measures, as follows (detailed descriptions are given in the Reference section):
1) Database Overview gives summary statistics and group averages for a particular group's contributions to the database. It shows how many notes they've contributed, which views they've contributed to in the past month and past week, how much of the database users have read, and how many of their notes are linked to other notes. It is intended to provide evidence on the extent of knowledge-building activity in the database, and also on where the recent activity has been.
The next three reports give statistics for each user in the selected group. Means, medians, and standard deviations for the group are also given:
2) Basic Knowledge Building Measures are meant to be useful for assessing a user's contributions to the database from a knowledge building perspective. Is the user working with other users, or in isolation? To what extent is the user aware of other work in the database? Does a user's work span a variety of different views and problems, or is it concentrated in one area? The measures are: number of notes contributed, percentage of notes that are linked to other notes, percentage of notes that have been keyworded, number of views worked in, number of problems worked on, percentage of notes in the database that the user has read, and the number of times the user has revised a note.
3) Use of Features. These measures are intended to be useful in assessing whether users are using the knowledge-building features that are available in Knowledge Forum. When new features of Knowledge Forum are introduced to a group, which users have caught on to them? Is the use of these features correlated with advances in knowledge? The measures are: number of user's notes that are part of a build-on tree, number of keywords used, number of references in the user's notes, number of rise-above-it notes, number of views created by the user, and number of scaffold supports used. For KF2 databases, the number of collections and add tos are also shown. For KF3 and KF4 databases, the number of annotations that a user has made is shown.
4) Use of scaffold supports. This report gives a detailed breakdown of which scaffolds and supports each user has used. Are scaffolds being used well? Are new scaffolds being used?
5) Single User Report presents information on only one user, but it presents all of the measures in reports 2 to 4. It also presents as additional information the names of the notes that the user has contributed to each view, and the problems and keywords they have used.
The next four reports analyse reading and writing activity in the database during particular time periods:
6) Activity (Note Creation/Note Reading). Who has been active in the database during a given time period? The report shows reading and writing activity in the database for a specified time period. Which views are users finding useful, and adding to? Which group interventions or changes to the database were effective, and for which users?
7) Activity (Note Creation--Details). What types of notes has each user contributed to the database during a given time period? The report shows details of what each user has been doing--how many build-ons vs regular notes the user has contributed, group notes vs individual notes, and so on.
8) Activity Log for Single User. A detailed log of every time the user read or wrote a note in KF, during a selected time period. Patterns of use of individuals can be traced, and related, for example, to knowledge-building episodes seen in the notes.
9) Activity Log for Multiple Users Same information as for the single user, but for everyone in a group.
The next five reports all look at social interactions in the database. Who is working with whom? How is knowledge building affected by particular interactions? If there is more than one group working in the database, what are the interactions between groups? The first four reports show specific types of interactions, while the last one combines all of the KF interactions that could be considered as "linking".
10) Who's read whose notes?
11) Who's coauthored notes with whom?
12) Who's built on whom?
13) Who's referenced whom?
14) Who's linked to whom? (where linking includes building on, referencing, and use of rise-above-it notes, (and, for KF2 only, add tos and collections).)
The next two reports give information in terms of notes, rather than users:
15) Who Has Read Each Note. Shows which users have read each note in the database.
16) Note History. Shows the entire history of who has saved and read a particular note.
And finally:
17) Code Book. If coded names are requested for the Toolkit reports, this report shows which code goes with which user. (See Coded reports, below.)
Parameters
After you select a report and "Submit" it, the Toolkit requires parameters to be specified. There are three parameters for most of the reports:
Look at Group Each report is for the users in one or more groups, corresponding the groups already created in the KF client (under "Edit users..." (KF2) or "Users and Groups" (KF3 and KF4)).
Look at notes in all views, or only in the following selected views. The report can be restricted to part of the database by choosing particular views--only notes that are currently in those views will be reported on. All views in the database are listed as choices, including restricted views. A table that shows the number of notes from each group in each view is given to aid in selecting views. (A note is counted as being from a given group if it has at least one author from that group.)
Look at all notes or only notes created within a particular time period: The report can be restricted to a particular time period, such as the past week or month, by selecting beginning and end dates. Only notes that were created between these dates (inclusive) will be reported.
Other parameters that are used for some reports are:
Look at User, in the reports for single users, allows you to choose which user to report on. All users in KF are listed, even those who have been set to "inactive" in the KF client.
How extensive should the report be?, in the reports on user activity, affects the amount of information given for each read and save event. The number of events reported is not affected by these choices.
Show z-scores: Reports that give score-like measures for each user (reports 2, 3, 5, 6, and 7) will also display the z-score equivalent of each score if this box is selected.
Saving Reports
If a printed copy of a report is needed, the Web page can be printed directly from the browser. Reports 1-9 and 15-17 will fit well on standard paper. At present, there is no good way to print reports 10-14, however, as the large matrices do not fit on a page.
If an electronic copy of a report is needed, the browser's "Save" options will not preserve the table alignments with some browsers. The Toolkit therefore makes a text version of each report available, which uses tabs to align the table entries. Instructions for downloading the text version are presented at the bottom of the Web version of each report. Once saved, the text version can be opened in a text processor or a spreadsheet, and the tables will be tab-aligned. Again, reports 10-14 are a problem: The saved file will be suitable for reading into a spreadsheet, but it will be difficult to use it as a text file.
Some reports can also be saved as KF notes, back into the database that you are working in. Instructions are at the bottom of the report.
Coded Reports
User names can be replaced with code names in all of the Toolkit reports. This can be done in two ways. The first is for public presentations or written reports where confidentiality is required, and the second is for coded user access.
First, for users who are registered as editors or managers in Knowledge Forum 3 or 4 (or as teachers in Knowledge Forum 2), there is a choice box at the bottom of the report selection page that lets you substitute code names for the normal user names. If you select this option, all of the reports produced will show coded names in place of actual names. This is useful for public presentations or for written reports, where confidentiality is required.
Second, if users who are registered as writers in Knowledge Forum 3 or 4 (or students in Knowledge Forum 2) are given access to the ATK, an option is available that will let them see all of the results with coded names in place of real names. This lets writers (or students) look at Toolkit reports which show scores for everyone in the database without knowing who's who. These users can get their own code name by using the Code Book, which, with this option, will show only the code for the individual who is currently signed on.
Codes are not Secure:
Be aware that the coding of names in ATK does not prevent a user from discovering who's who, it only makes it a bit more difficult. For example, with a particular view selected, the Basic Knowledge Building Measures report will give results showing how many notes each coded author has written in that view. A curious user could easily count the number of notes each author has produced in that view by looking directly at Knowledge Forum, and then match these numbers with the ATK results to break the code.
For KF2 databases, use of the ATK, even with coded names, gives students access to the notes in restricted views. Restricted views should therefore be not be used in KF2 databases if students are given access to the ATK.
Updating of Toolkit Information
You do not need to worry about controlling the updating of Toolkit information. If you do not do anything special, updating will be handled automatically. However, in some cases you can save time by controlling it.
Most of the time taken by the Analytic Toolkit to prepare a report is spent in reading the necessary data from the Knowledge Forum server. A typical report on a large database can take two or three minutes to get the data from the server, but then only two or three seconds to compute the measures for the report. In order to speed things up, once information has been read from the KF server, the Toolkit saves it for future use. Since all of the Toolkit reports use the same data, after the first report on a database has been made, subsequent reports are computed mostly or entirely from the saved data, and are much faster.
Since the Toolkit works with saved data whenever possible, there is generally no updating of its information during a given session. If you need to update the data, however, you can do so by going back to the sign-on screen and signing on to the database again. This starts a new session, which forces the Toolkit to go to the KF server and update its information--unless the "Skip update" box is checked.
For some purposes, it may be unnecessary to update the data even at the beginning of a session--e.g., for demos, or for databases that have not changed since the last time you signed on. In this case, you can choose "Skip update" at the bottom of the sign-on page, and the Toolkit will not bother to read data from the KF server, but instead use the data it already has from the most recent session where updating was done.
Conclusion
The Analytic Toolkit is not the only way to do analyses on a Knowledge Forum database. KF itself provides overview tools, such as views and searches, which complement the reports that are available in the Toolkit. There is also a second Web-based analysis tool called the Vocabulary Analyser that provides information on the contents of notes.
There are many measures and variations of measures that could have been included in the present Toolkit but weren't. We welcome suggestions for additional measures that would improve the reports. Our long term plan is to incoporate the functionality of the Analytic Toolkit into Knowledge Forum itself, so that users can get overview information and statistics on the database while they are using Knowledge Forum.
