In this post I give an overview of WebLab, a web application for programming education, propose to use it to organize programming competitions, and call for student volunteers to organize a competition.
If you are a new computer science student, let me explain how the feedback cycle of programming courses used to be organized (and sometimes still is). As student you make a programming assignment and then you have to wait for a teaching assistant to manually grade your work. Typically, it takes at least a day, but often much longer, before you get feedback. And often there is no time to use that feedback to improve your work.
For teachers and assistants this is as frustrating as it is for you. Grading requires a lot of effort (with over 200 computer science students in our first year) and is extremely tedious. Moreover, it is quite hard to determine the correctness of a program just by reading code.
In addition, we have to make sure that all students can work with the same programming environment, which means installing software on lab machines and make sure that a distribution is available that works on a wide variety of student machines.
And finally, exams are typically conducted on paper, making it hard to test programming skills. An exam for a programming course should (at least) test whether you can write a correct solution for a programming problem in the scope of the course. Programming on paper is not what we train you for in the lab, so why test it in the exam?
To address these issues, Vlad Vergu and I started developing WebLab in early 2012. WebLab is a web application that supports all aspects of a programming course. Instead of installing a programming environment, editing a solution on your computer, and submitting it in a tool such as CPM, you can edit the solution for a WebLab assignment right there in the browser. While developing your solution, you can compile the solution, to make sure the program has no syntax or type errors, and then run it against a set of unit tests. This gives you direct feedback about the solution you are developing. By extending the set of unit tests you can express your increased understanding of the problem. When you are satisfied with your solution, you can run it against a set of secret specification tests. The only feedback you get from this test is the number of tests that succeeded and the total number of tests. While this is not a lot of information, it does give you an indication about how you are doing, and at a much earlier stage than waiting for an assistant to look at your code. After the submission deadline, grading is mostly based on this testing score. But some aspects of an assignment may be checked by a teaching assistant using a grading rubric. WebLab can also be used for exams. An exam is just a collection of regular WebLab assignments. Using personalized keys handed out on paper, we can make sure you are present in the room during the exam. This enables us to align the exam with the (programming) skills that you develop during the course.
WebLab also integrates the administration of courses. The grades for all your assignments are available in WebLab as soon grading is finalized (which can be on the day of the exam, when no additional manual grading is done). Overall, WebLab saves instructors and assistants a lot of time normally spent on grading and administration and the real-time statistics provides a good tool to monitor the progress of students in a course.
WebLab is now being used in several courses at TU Delft (Algorithms and Data Structures, Algorithms, Concepts of Programming Languages) and at TU Darmstadt (Concepts of Programming Languages, Type Systems of Programming Languages).
WebLab is itself an interesting software engineering project, since it combines a number of non-trivial requirements.
Speed: The aim of WebLab is to provide real-time feedback on submissions to programming assignments, reducing the feedback cycle from days or weeks to minutes. However, as soon as the feedback cycle gets that short, students expect really short response times. When testing is part of the development cycle, waiting for a minute to see results feels like an eternity.
Robustness: The application must be able to deal with erroneous programs. A key part of learning to program is that one (accidentally) writes non-terminating programs or programs that do otherwise not correspond to our expectations. The application should also not lose solutions written by students.
Scalability: The application should do all this not for a single student, but for hundreds (eventually thousands) of students simultaneously writing and executing programs.
Security: The application executes user-written code on the server. For regular web applications this is a scenario to be avoided at all costs, since it could allow users to use excessive amounts of resources (memory, CPU), disrupt the application (denial of service), or access and tamper with private data. Thus, the application must restrict what user code can do and how what resources it can use.
Quality of feedback: After testing student programs, the application should give useful feedback to students.
Satisfying all these requirements at the same time is non-trivial. It took us a number of iterations (and some painful failures) to get to the current implementation. And we have plenty of ideas for further improving it. Let me give an overview of the architecture of WebLab.
The front-end of WebLab takes care of the web-based user interface and maintains a database with all data for a course such as assignments, student enrollments, submissions, and grades. As it is a web application, it is not too hard to satisfy the requirements above even if does require care. Of course, the interaction design for such an application is not obvious and we are still fine tuning navigation and optimal use of the variety of screen resolutions the application is used with. One of the annoying issues with the front-end that we have recently addressed is data loss. We used to maintain only the latest version of a submission without support for undoing of changes. Editing accidents such as saving from an old open tab or pasting code in the wrong tab, could result in code loss. WebLab now maintains a revision history of all edits to a submission, so that you can always retrieve that clever piece of code that you accidentally deleted (as long as you saved it at some point).
The back-end of WebLab takes care of executing student code. This is where the requirements are harder to satisfy. The front-end stores submitted programs and tests in the database. On a request for a test of a submission, the front-end sends the solution code and the test code to the back-end. The back-end compiles and runs the code, and sends the results back to the front-end, which interprets the results and presents those in the console.
To enable rapid feedback (speed) on program executions, it is prohibitive to start a new process to run a compiler or a (Java) virtual machine for the execution of each solution. For scalability it would be problematic to allocate a running JVM for each user. For security we need to be able to restrict what a program can do; we should certainly not run a program in an unrestricted native process. For robustness we need to be able to control programs that attempt to use an excessive amount of CPU cycles, memory, or time.
We ended up with the following solution. A back-end is a Java Virtual Machine (JVM) that processes execution jobs. Each job is executed in a new thread using a separate class loader that restricts what the program can do. We can use this directly for the byte code generated from Java and Scala programs. C and JavaScript programs are first compiled so that they can be run on the JVM. A monitor kills execution tracks if they take too long. In order to cope with programs that crash a back-end anyway, we run a pool of back-ends that are restricted in the amount of memory and CPUs they can use.
In addition to use in traditional programming courses, WebLab has all the ingredients of a great tool for organizing programming competitions, provided we add support for rankings and leaderboards (which may be useful in (gamified) courses too). By frequently organizing programming competitions, you can test your programming skills and we can speed up the development iterations for WebLab, introducing new features for giving feedback and analyzing programs. These competitions would be different in format than the regular ACM competitions with a much shorter duration (say about 3 hours) and individual assignments.
To help us organize the competitions, we are looking for student volunteers. Are you a second year (or later) computer science student with good results in programming? Would you like see what setting programming assignments (and WebLab) looks from the instructor side? Do you have creative ideas for assignments that are feasible for your colleagues to make, yet challenging at the same time? Would you like to contribute to new ways of assessing programming assignments? Then, get in touch with me.