MOOCDB Research Guide¶

Step 1: Installing the required packages or downloading a VM¶

• Option 1: To run the software you may want to download a VM from Amazon cloud.

  This virtual machine image comes with all packages installed which are required to run the MOOCdb pipeline. To get a link to the image and request the ‘.pem’ file, please email kalyan@csail.mit.edu.

Important

• When instantiating this virtual machine on Amazon or locally, please provision the disk space (hard disk) such that there is atlease three times the size of the decrypted- uncompressed file size of the tracking logs.

• Option 2: Install all the packages on your local machine

  The following packages are required on a MOOCdb machine

  1. Install Unidecode package available here
  2. Install ijson package which can be found here
  3. Install python-setuptools
  4. Install pip using sudo easy_install pip
  5. Install pandas
  6. Download the code from MOOCdb github:
     • Openedx diagnosis
     • Openedx apipe
     • Openedx qpipe

Note

• Make sure your Pandas version is higher than 0.14.0. If it is below that you would have to update Pandas by running:

  pip install pandas –upgrade

• You may have to upgrade numpy and numexpr before upgrading pandas, if upgrading pandas gives you an error. The command to upgrade numpy and numexpr is the same:

  pip install numpy –upgrade

  pip install numexpr –upgrade

Step 2: Processing the tracking logs¶

If your course is through edX you would get the files shown below. The most important and perhaps most tedious is processing the tracking log files. Some of the files listed below in the table could be representative of what MIT delivers to us. But tracking_log.json is the largest file and contains the detailed clickstream events. These are the events which are recorded along with event type.

1. Unzip tracking log file:

   All raw data files in ‘data/raw/<course_name>’ have the same prefix in the format of ‘<course_name>__<creation date>‘, we will call the prefix ‘COURSE_PREFIX‘.

   From within the tracking log file folder, run the command:

   gzip -d COURSE_PREFIX__tracking_log.json.gz

   This will extract the tracking log file into .json format, ready to be piped.

2. If there are multiple log files, merge all the log files for a single course into one log file.

3. Run JSON to relation code (a.k.a apipe):

   This tutorial covers the transfer of JSON tracking log file to CSV files. The code is written by Andreas Paepcke from Stanford. JSON tracking log file is stored with other raw data files. We will call the raw data files as ‘raw data‘ and the output CSV as ‘intermediary CSV‘.

   Let us suppose that we want to pipe the course named ‘<course_name>‘. We assume that the raw data is stored in the folder:

   /.../<course_name>/log_data/

   Create a folder called intermeidary_csv under the folder named ‘<course_name>‘

   /.../<course_name>/intermediary_csv/

   Create another folder called moocdb_csv under the folder named ‘<course_name>‘

   /.../<course_name>/moocdb_csv/

4. Launch the piping:

   From within the import.openedx.json_to_relation folder, run command:

   bash scripts/transformGivenLogfiles.sh /.../<course_name>/intermediary_csv/

   /../<course_name>/log_data/COURSE_PREFIX__tracking_log.json

   As show in the command above, transfromGivenLogFiles.sh takes two arguments. First argument is the path to the destination folder, and second argument is the tracking log json file to pipe. ‘/.../‘ represents the path to the directory where the <course_name> folder is located on your machine. The command may run for a few hours and depends on the size of the raw json tracking log file.The output csv files will be in ‘/.../<course_name>/intermediary_csv‘. The following gives an example of the output csv files produced for link5_10x course:

link5_10x_trace_merged.2014-11-02T23_46_45.622627_28028.sql
link5_10x_trace_merged.2014-11-02T23_46_45.622627_28028.sql_ABExperimentTable.csv
link5_10x_trace_merged.2014-11-02T23_46_45.622627_28028.sql_AccountTable.csv
link5_10x_trace_merged.2014-11-02T23_46_45.622627_28028.sql_AnswerTable.csv
link5_10x_trace_merged.2014-11-02T23_46_45.622627_28028.sql_CorrectMapTable.csv
link5_10x_trace_merged.2014-11-02T23_46_45.622627_28028.sql_EdxTrackEventTable.csv
link5_10x_trace_merged.2014-11-02T23_46_45.622627_28028.sql_EventIpTable.csv
link5_10x_trace_merged.2014-11-02T23_46_45.622627_28028.sql_InputStateTable.csv
link5_10x_trace_merged.2014-11-02T23_46_45.622627_28028.sql_LoadInfoTable.csv
link5_10x_trace_merged.2014-11-02T23_46_45.622627_28028.sql_StateTable.csv

5. Run relation to MOOCdb (a.k.a qpipe):

   This tutorial covers the transfer of CSV files as output by Andreas Paepcke’s json_to_relation to MOOCdb CSV files. We will call the source CSV as ‘intermediary CSV‘ and the output CSV as ‘MOOCdb CSV‘.

   Let us suppose that we want to pipe to MOOCdb the course named ‘<course_name>’. We assume that the course’s log file has been processed by json_to_relation, and that the output files are stored in the folder :

   /.../<course_name>/intermediary_csv/

   We want the MOOCdb CSV to be written to folder

   /.../<course_name>/moocdb_csv/

   1. Edit import.openedx.qpipe/config.py

   2. QUOTECHAR: The quote character used in the intermediary CSV files. Most commonly a single quote (‘)

   3. TIMESTAMP_FORMAT: describes the timestamp pattern used in ‘*_EdxTrackEventTable.csv‘ intermediary CSV file. See python doc to understand syntax.

   4. COURSE_NAME: The name of the folder containing the intermediary CSV files. Here it is ‘<course_name>’.

   5. CSV_PREFIX: All the intermediary CSV file names in ‘/.../<course_name>/intermediary_csv/‘ folder share a common prefix that was generated when running JSON to relation. This prefix is also the name of the only ‘.sql’ file in the folder. For example, in the above case this prefix would be :

      link5_10x_trace_merged.2014-11-02T23_46_45.622627_28028.sql

   6. DOMAIN: the domain name of the course platform URL, most commonly they are https://www.edx.org or https://courses.edx.org.

      (No slash at the end of the domain name). To be sure, you can look at the URL’s appearing in ‘*_EdxTrackEventTable.csv‘ intermediary CSV file.

Note

The variables not mentioned in the tutorial must simply be left untouched.

6. Launch the piping:

   When the variables mentioned above have been correctly edited in config.py, the script is ready to launch. From within the import.openedx.qpipe folder, run the command:

   time python main.py

7. Delete log file:

   When the piping is done, if everything went well, go to the output directory ‘/.../<course_name>/moocdb_csv/‘ and delete the ‘log.org‘ file that takes a lot of space.

8. Load course into MySQL:

   Copy the file ‘/.../<course_name>/moocdb_csv/6002x_2013_spring/moocdb.sql‘ to ‘/.../<course_name>/moocdb_csv/‘ folder. Change directory to ‘/.../<course_name>/moocdb_csv/‘. Replace ‘6002x_spring_2013’ by ‘<course_name>’ in moocdb.sql file.

   Run the command:

   mysql -u root -p --local-infile=1 < moocdb.sql

   This creates a database named ‘<course_name>‘ in MySQL, and loads the CSV data into it.

Setting Up the Coursera-MOOCdb Transformation Scripts¶

MOOOCdb Core¶

This database is required by all transformation tasks, but no task will ever alter its data. Hence, you can import it once and use it many times. Call this database moocdb_core, and avoid using this name for any other databases. After creating this empty database, import the file .../core/moocdb_core.sql located inside the piping scripts directory.

The Schema¶

This database is also required for all transformations, and can be created once and used many times. Call this database moocdb_clean, and avoid using this name for any other databases. Unlike moocdb_core, this database has to be created on the same machine on which the output databases will be created. After you create moocdb_clean as an empty database, import the file .../core/schema.sql located inside the piping scripts directory. Using MOOCdb

You are now ready to transform a Coursera course to MOOCdb.

To do so, you have to supply a set of parameters telling the program where to find the input data, where to put the output data, and specifying other required information and options. To transform a course, you simply have a call a main function with a dictionary of these parameters. The scripts expect this dictionary to follow a very specific structure. To familiarize yourself with the parameter dictionary structure and how to call the main transformation function, please look at the run_coursera.py file located in the piping scripts directory. In this file, the main function is first imported from main.py. Second, the parameter dictionary is constructed. The variable holding it in the example is called vars. Finally, main(vars) is called. This should start the transformation. Below is an explanation of the parameters you need to specify. Before, we start, however, you will need to watch out for the following.

Coursera changed its export format at some point in the past. Previously, table columns referring to anonymized user IDs (such as in the lecture_submission_metadata) in the anonymized_general database were called anon_user_id. User IDs were anonymized in the anonymized_forum database as well and placed in columns called forum_user_id. If your course following this format, we will refer to its format as coursera_1.

More recently, Coursera changed the anon_user_id to session_user_id, and stopped anonymizing user IDs in the anonymized_forum database. If your course follows this format, we will refer to its format as coursera_2.

These naming changes will also be visible in the hash_mapping table directly. If your course does not strictly adhere to one of these 2 formats, this program might not be able to handle its data. Please request a new export of your course from Coursera. You should receive it in the second format regardless of when the course ran.

Setup¶

Follow the tutorial to set up for developing and upgrading MOOCviz.

Feature Factory Instructions¶

The following two github links will be useful reference:

• Feature Factory
• Feature Discovery

Fresh setup:¶

The ultimate guide for a fresh setup can be found here.

sudo apt-get update --fix-missing
sudo apt-get install python-pip python-dev build-essential
sudo pip install --upgrade pip
sudo apt-get install apache2 apache2.2-common apache2-mpm-prefork apache2-utils libexpat1
sudo apt-get install libapache2-mod-wsgi
sudo service apache2 restart
sudo pip install django celery django-celery django-kombu hoover django_extensions
sudo apt-get install python-mysqldb

sudo nano /etc/apache2/apache2.conf #add this
ServerAdmin kiarashplusplus@gmail.com
ServerName featurefactory.csail.mit.edu
WSGIScriptAlias / var/www/featurefactory.csail.mit.edu/index.wsgi

sudo mkdir /var/www/featurefactory.csail.mit.edu
sudo nano /var/www/featurefactory.csail.mit.edu/index.wsgi
import os
import sys

path = ‘/home/ubuntu/alfa’

If path not in sys.path then use the following command:

sys.path.append(path)
os.environ['DJANGO_SETTINGS_MODULE'] = 'alfa.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()
copy files to ~/alfa

To send email:

sudo apt-get install postfix

sudo service apache2 restart

Install Process - Installing Ruby on Rails¶

First step is to install ruby on your linux environment. The commands used here come from https://gorails.com/setup/ubuntu/14.10

In terminal, run these two commands:

sudo apt-get update
sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties

For this project I ran Ruby using RVM, so then run

sudo apt-get install libgdbm-dev libncurses5-dev automake libtool bison libffi-dev
curl -L https://get.rvm.io | bash -s stable
source ~/.rvm/scripts/rvm
echo "source ~/.rvm/scripts/rvm" >> ~/.bashrc
rvm install 2.1.5
rvm use 2.1.5 --default
ruby -v
echo "gem: --no-ri --no-rdoc" > ~/.gemrc

Install node.js

sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs

Install Rails

gem install rails

Install depedencies for pg gem **not in online instructions to install rails

sudo apt-get install ruby-dev libpq-dev

Server Setup

Time to set up the server. Move to the LabelMe-Text folder and run the following command:

bundle install
bundle exec rake db:migrate
bundle exec rake db:reset
bundle exec rails s

This starts a copy of the server running on the machine’s localhost. You can access it in your browser at:

localhost:3000/.

Making an Admin¶

• Now you want to make an admin user. Go to the website and sign up with the account info for your admin user.
• From the home page, click on “Create profile”
• Fill in the account details, check the box agreeing not to scrape the data from the website, and click “Create my account”
• Then go to terminal and type Ctrl+C to stop the server from running. Run the following commands:

bundle exec rails console.
a = User.find_by_email(“your@email.here”)
a.update_column(:admin, true)
exit

Leaving server running¶

If you are using a virtual machine and wish the process to keep running after you close the ‘ssh’ connection, we will use a program called screen to achieve this.

1. First we need to install screen.

   sudo apt-get install screen
   

2. Start the screen program

   screen
   

3. Navigate to the Label-Me Text folder and start the server again

   bundle exec rails s
   

4. While the server is running, detach from the current screen using the following command

   Ctrl + a, d
   

5. The server should now run even if you stop the ssh session. To return to the running server terminal, use the command:

   screen -r
   

Coming Soon¶