Jeffrey Pomerantz

Bear with me for a moment while I’m insufferable. Because…

Highest Distinction, baby! Hellz yeah!

So ok, seriously now. These are the levels of certificates for CS101, as articulated in the announcement email:

Certificate of Completion: you completed the class, and demonstrated that by either getting at least one question correct on the final exam, or solving at least 3 questions correctly on Homework 6.

Certificate of Accomplishment: you solved at least 3 questions correctly on the Final.

Certificate of Accomplishment with High Distinction: you solved at least 9 questions correctly on the Final.

Certificate of Accomplishment with Highest Distinction: you solved all 11 questions (including the 3 starred questions) correctly on the final, or you correctly answered over 80% of all the homework questions correctly and at least 6 questions correct on the final.

I did not get all 11 questions correct on the final; in fact, I got 88% correct (7 out of 8 questions) on the main part of the final, and 67% (2 of 3) on the starred questions. (The starred questions were the extra-challenging ones. Starred questions were a standard feature of the homeworks throughout the course.) But I did average an 85% on the homeworks. So there you go.

Ok, I was feeling all insufferably smug there for a while, until I actually ran those numbers, and realized that I’m a solid B+ student. Oh well.

Anyway… I don’t have any of my actual degrees hanging on my office wall. But I’m sure as hell going to hang the Udacity certificate.

It’s been several weeks since I posted anything about my Udacity CS101 course… mostly because over the past several weeks I’ve been spending every evening working on the course and not blogging about it. But now I’ve completed all 7 units, and the final exam, the grading robots are hard at work, and I’m waiting on my exam grade.

While I haven’t been blogging about CS101, I have been making notes. So now let me try to stitch my notes together into a semi-coherent post.

First of all, there’s been a hell of a lot written about Udacity, MITx, and disruptive innovation in higher ed generally, both in the higher ed press (The Chronicle, Inside Higher Ed, etc.) and the mainstream press (Wired, the NY Times, etc.). I won’t even try to sum it all up. But if you’re in higher ed and you’re not keeping up on these developments, well, all I can say is, you’re part of the problem.

There are two pieces that I will mention, however: One, Yvonne wrote a post on the CIT blog about her experience in MITx 6.002x: Circuits & Electronics, MITx: A view from the inside. Two, for my money, the most interesting piece on disruptive innovation in higher ed that I’ve read so far is Kevin Carey’s piece in The New Republic, The Higher Education Monopoly is Crumbling As We Speak.

Now on to CS101.

It was in about week 5 that I started to feel like a real student: the self-assessments were getting difficult, and I was actually concerned about getting the correct answers on the homeworks. Even though I enrolled in the course largely to experience their instructional design, I very quickly came to actually care about doing well in the course. I was also in it for the Python, of course, and so my geek pride demanded that I do well on that front. And, on that front, I have learned a good deal about Python. Evans, the instructor, kept it as simple as possible, and as a student I appreciate that, and as an instructor understand why he did so. But even so, we still learned a lot of Python, even if we only scratched the surface.

And now I’m a total Python convert. For the homeworks we often had to write code, and of course for the final exam. I tried as much as possible to use only the Python that we had learned in the course. Even though it was clear that not all students were doing that, as evidenced on the discussion boards, and even according to a comment that Evans and his TA Peter made in one of the office hours videos. But at certain points I thought it would be easier to just “cheat” and look up Python functions that we hadn’t learned in the course and use those. But honestly, I don’t really consider that cheating: if (part of) the point of the course was to learn Python, then teaching myself more Python surely isn’t cheating. But I’m telling you this as a way of explaining why I’m a Python convert: because every time I thought, “hm, maybe there’s a built-in function that will allow me to do X,” I went looking, and lo and behold, there is such a function! I read a comment in some discussion board post to the effect that “Python makes the difficult simple, the impossible feasible.” I totally buy it.

And in that vein… I found myself really enjoying programming again. It’s been yonks since I’ve done any programming of significance, anything more complicated than a one-line Unix script. I mean, who has the time? But I was really enjoying spending significant amounts of time in the evenings getting my head into coding again. I’d almost forgotten what that headspace felt like — where I used to live so much of the time, in a previous previous life — where I could lose time coding, and even after I stopped, some part of my mind was always churning on the algorithm I’m working on. I really enjoy that, and now I think I need to find reasons to do more development as part of my research, so I can have some of that back.

But anyway, on to the instructional design.

Each Unit has an associated set of notes, which is easily accessible throughout the entire Unit: below the video / Python interpreter block (they occupy the same real estate in the browser), there are 2 tabs, Instructor Comments and Supplementary Material. A link to the unit notes are always in the supplementary material. Anyway, I found it incredibly helpful to have these to refer to. For a while I had the notes from Units 1, 2, & 3 in 3 different browser tabs, just for reference. In I think Unit 4, they added a Python Reference document, which contains all of the functions that we’d covered up to that point. With each new Unit, they updated that Reference document. I kept that document open all the time, and that was incredibly helpful.

As the course progressed, I found that I backed up videos more. In the first few units, I watched videos once and moved on. But as the content got more challenging for me, I found that I had to re-watch bits. I rarely re-watched a whole video, though I did do that from time to time, but I frequently re-watched parts of videos. I found that these re-watchings fell into 2 categories:

1. Backing up a video to make sure I understood some point, because I tuned out momentarily or didn’t fully understand something; or
2. While I was working on a quiz or a Python exercise, re-watching part of some previous video that explained some salient point for completing the assessment.

I also found myself pausing videos to mentally replay steps to make sure I got something, and then backing the video up to watch that bit again.

Anyway, not that I was skeptical about the pedagogical value of short instructional videos before, but I’m really sold on the pedagogical value of short instructional videos now. I’ve read in several places that one of the most useful things about video instructional content is the ability to back up and replay something. It’s one thing to read that as an instructor; it’s quite another to experience it as a student.

On the subject of videos, however, I do have one issue. The video solutions to the homeworks were posted after their due date, which was usually a day or 3 after I finished them. By which time I’d usually started on the next unit. So the details of what I was thinking, and how I solved the homework problems from the previous unit were no longer fresh in my mind. So I found it a bit disorienting to go back and look at the homework video solutions. I found myself wishing they were available immediately.

Finally, I found that as the units progressed & got harder, I was looking at the discussion boards more frequently for help on the self-assessments and homework. I kept up with the course, but there were students who were clearly devoting way more time to the course than I was: by the time I got to the discussion boards for any given issue, there were always several threads on the topic. Yvonne is finding the same thing for her MITx course. On both the Udacity & MITx discussion boards, posters can earn badges for various things (Good Question, Good Answer, Civic Duty, Editor, etc.), which I think is a nice touch. To be honest, I was a complete free rider on the discussion boards: I read threads, but I posted no questions or answers. Maybe if I had more time to devote to the course I would have participated; I don’t know. I treated the discussion boards more or less the same way I treated the many sites on the Intertubes that contain Python Q&A and code snippets, and the Python Software Foundation’s documentation: as a resource to fulfill my information needs. I suppose this is very selfish of me, and very not community-minded (Wikipedian? Crowdsource-ish?). But there it is. As an instructor, I need to think about whether it’s important to try to draw out lurkers who could potentially be valuable contributors (I suppose I’m flattering myself that I would have been a valuable contributor). In a course on the scale of Udacity, with thousands of students, you only really need a smallish percentage of those to actively participate in discussions, to be useful to us lurkers. And I suppose, like everything else in human information-related behavior, there’s always going to be your basic long-tailed distribution of participation. Of course, to get students to do anything, you make it a graded assignment… not to be cynical, but it’s true. That was of course not the model for grading in CS101, and it might not be feasible on that scale anyway. (And there’s another question: is there a way to semi-automate evaluation of contributions to online discussions?) But for my teaching online, for small (compared to Udacity and MITx) courses, it is an issue. I suppose the question is: is it important to try to draw out lurkers? At what course size is it important, at what course size does it become unnecessary?

I wrote my last post when I was about a third of the way through Unit 1 in CS101, and before the homework for Unit 1 was posted (which happened last Thursday morning). Now that I’ve finished Unit 1, both content and homework, I thought it was a good time to take stock & post again.

First, I have an issue. The instructor, David Evans, made his videos using some technology I’m not familiar with, but it’s clearly some kind of smartboard- or tablet-and-smartpen combination. The way it appears in the video is like this: he writes on a white surface using this groovy pen, and what he writes shows up in various colors, just like on a smartboard. The weird part is that what he writes seems to be floating above the pen and his hand, rather than appearing below it. That is, not above in the Y axis on the plane of the screen, but hovering above on the Z axis, so that the writing appears to be closer to the viewer’s eye than the pen. It’s kind of distracting. Actually I’ve more or less gotten used to it, and I can tune it out now, but it was very distracting for the first few videos.

This of course is Rule #1 for the use of technology in teaching: Never let the technology overshadow the actual pedagogical purpose. And I wouldn’t go so far as to say that this tech overshadows the content. But it is distracting.

And this raises a question for me, for the videos I make for my courses: handwriting or Powerpoint? Evans has so far made all the videos for CS101 using his own handwriting. (Except the bits in the Python interpreter, of course.) The videos that I’ve seen from Sebastian Thrun’s AI course last semester were all done with handwriting. Khan Academy videos are all Sal Khan’s handwriting. (At least I assume it’s his handwriting.) Obviously there’s a trend here: making educational videos using handwriting. And I can see the advantage of that: it makes it feel personal, like the instructor is writing just for you, like you went to David Evans’ office hours and he’s jotting on his whiteboard while you talk. I get that.

But here’s my problem: my handwriting sucks, and these videos are edited heavily. Why edited? To fast-forward in time, so you don’t have to watch Evans form every single letter. Which brings me to Powerpoint. I made a few videos for my Digital Library course using Powerpoint: I wrote a script and created a slide deck to accompany it, more or less in tandem (the script usually slightly preceded the slides), then I used Powerpoint’s Record Slide Show feature to record the timing of the slides and my narration, and exported that YouTube. It was super-easy. Powerpoint is definitely the low-bar way of creating videos. And for getting time-consuming stuff done, I tend to prefer low-bar. Good enough is good enough. But I do fear that Calibri is a poor alternative to handwriting. Does using a font make the video feel less personal? Or can the voiceover compensate for a lack of handwriting? If anyone is reading this, I’d welcome your feedback on this weighty issue.

My second issue is that some of the videos feel slightly pedantic, especially on the quiz reviews. Evans proceeds through each quiz option in somewhat excruciating detail. Which is, of course, better than the opposite. And I understand why he does it this way: these materials are being prepared for 70,000 (or more) students. As in any course, the instructor has to make things as clear as possible, which often means going slower than more advanced students would prefer. I figure I’ll probably mind this less as the content gets more advanced, and I stop being an advanced student. And, I have to think, this is probably what half the students in my courses feel like a good bit of the time. I need to be careful of that in my classroom teaching. But for making videos, I think Evans makes the right call: better to err on the side of being slightly pedantic than to lose your audience within the span of a 3 minute video.

That’s one gripe and one sort-of gripe. Now the good stuff: I’m finding the course so compelling that I want to work on it all the time. I had a hard time this week stopping once I’d started. I even found myself wanting to work on the course during the day while I was at work, and the one time I actually succumbed to that temptation, it took a student walking into my office to make me veer off.

The downside of the course being so very compelling is that I found myself rushing through it: I would watch video after video, and spend some significant time working on the quizzes and Python exercises. On the one hand, this is good, because, well, education should be compelling. But on the other hand, by Wednesday I found myself concerned that I’d finish Unit 1 too quickly and lose the thread before Unit 2 was posted. In the end, that didn’t happen, as I should have known it would not: the homework was posted on Thursday, and that took me several hours, plus the simple fact of having a real job and a family slowed me down sufficiently.

But this did make me realize that this is no different than “traditional” classroom-based courses, where there are sometimes several days between class sessions. I teach a Monday/Wednesday course this semester, so my students have a 4 day gap between class sessions. Hopefully they’re doing work for the course during those 4 days, but I don’t really have a way to force that to happen. In an asynchronous course like CS101, there’s no way to force it either. I’m a big believer in project-based courses (both of the courses I’m teaching this semester are based around semester-long projects), so I have to assume that my students (most of them, anyway) have their head in the game during non-class days, otherwise they’d never get their project deliverables finished by the due dates. But CS101 has made made me appreciate the value of homework and other small self-assessments, which I tend not to use much in my courses. Something to reconsider for next semester.

And on the subject of self-assessments… the quizzes and Python exercises. These are automatically evaluated. It’s not clear what this looks like on the back end, though I imagine they’re fairly simple algorithms. It seems like it would be quite easy to automate evaluation of a multiple choice quiz. As for the Python, if the value of such-and-such variable (and Evans tells us what to name the important variable) equals the correct value, then the exercise is evaluated as correct. It’s not clear to me, at this stage, if the code that gets to that point is evaluated.

But my point is, it’s difficult to imagine assignments in Information Science that could be automatically evaluated. I know of some instructors in this field who use multiple choice exams in their courses, though I’m not one of them, and in fact I have a hard time even imagining what a good multiple choice question would look like in the courses I teach. Though maybe that’s a failure of imagination on my part. I can think of one or two assignments that I could use in my courses that could be automatically evaluated, and in fact I plan to set up one such assignment for the next time I teach my Digital Libraries course. (Assignment: Set up an OAI-compliant metadata repository. I’d have to create a harvester. If the harvester successfully harvests the student’s metadata records, then the student has successfully completed the assignment.) But the point is, I can only think of one or two assignments that could be automatically evaluated that I can use in my courses. I’ll probably think of more as time goes on. But I have a hard time imagining that I’ll ever be able to come up with enough such assignments to cover a whole course.

A lack of assignments that can be automatically evaluated means that my courses (any courses in ILS? any courses in the social sciences?) cannot scale to 70,000 or 160,000 students, or whatever. Without the ability to automate evaluation of all, and I mean all assignments, that scalability is just impossible. Because no automation = a human grading 70,000 assignments. And by “a human” I mean “me.” Now, I don’t expect that 70,000 students are suddenly going to rush out and take my Digital Libraries course online. (I should be so lucky.) But if I make my videos available, & I make whatever assessments I create available, there’s no reason why a student not enrolled in SILS shouldn’t be able to use them. And am I going to evaluate that student’s performance? No I am not. And so I feel that my teaching — and maybe the entire field of ILS, maybe the social sciences generally — hits a wall fairly quickly, in terms of the scalability of online courses. Some courses can probably be automated better than others. But fundamentally, there will probably always be some courses for which evaluation cannot be fully automated (or automated at all?). And so I feel a bit stuck. Again, maybe this is a failure of imagination on my part. If anyone is reading this, I’d like to hear your thoughts on this.

I’m really fascinated by the (fairly) recent boom in new models of online education: Khan Academy, MITx, Udacity, etc. And in fact I registered for, and have started Udacity’s CS 101, Building a Search Engine. Actually I already know how to build a simple search engine, though I’m sure I’ll learn more. Really I’m registered to see how they manage a course on that scale, how these models can impact my teaching, the future (or lack thereof) of the higher ed system, etc. I’m sure lots of others are registered for the same reason. Phil Edwards, for example, is registered for the first MITx course, 6.002x, Circuits and Electronics, for similar reasons.

I plan to report back on my experience with Udacity’s CS 101 here. I have not given myself a set of groundrules like Phil has… instead, I’ll just write. I’ve found that if I give myself too much structure, or try to write lengthy posts, that I never write anything on this blog. So I’ll just write as inspiration strikes. So here we go.

I’m about halfway through Unit 1 of CS 101. (For those of you enrolled, I just watched the video on Grace Hopper and took the first variables quiz.) And so far I’m not disappointed. Far from it, I’m fascinated and compelled. Here are my thoughts so far.

One. I haven’t learned much that’s new to me so far, except that Grace Hopper carries around nanosticks, and a bit about Python syntax. I leaned to write Hello World-style algorithms at age 11, and CS 101 is starting about at that same level. Which is good, since the goal of the course is to introduce CS concepts, and, though the instructor David Evans doesn’t say it in so many words, computational thinking. And that’s fine; I didn’t enroll in this course because it’s all new material to me… I enrolled in it because it isn’t all new to me, and I can therefore pay attention to the mechanisms of the course more closely. I am learning about Python, though, which is a language I’ve thought for a while that I should learn. So that’s good.

Two. The longest video so far has been 5 minutes. Most are 1-3 minutes. That feels about right to me… the 4 & 5 minute-long videos feel long to me. I know that sounds strange, but for whatever reason, some sort of time dilation happens when you watch videos online. Less is more. This is an especially important observation for me today, as I’ve given the students in my Digital Libraries course this video to watch as today’s “reading”… a video that weighs in at a whopping 45 minutes. When I said this to Yvonne about the 1-3 minutes thing, her comment was that most faculty would say that they can’t say anything in 3 minutes… to which the only possible response is, really? Try. David Evans has thus far done a very good job of chunking the content into 1-5 minute segments. Are those segments lacking in some way because they’re not longer? No.

This is more or less what I want to do with some of the content in my Digital Libraries course. For the course last summer, which was entirely online, I made a bunch of videos (very amateurishly), and collected a bunch more, into a YouTube playlist. I’m pretty happy with the content of the videos I made, and have plans for more, but I’m not happy with the videos themselves. They’re too long, for a start, weighing in at mostly 8-10 minutes. For another, they’re too sprawling; I try to cover too much in each video. I suppose this is an argument for more chunking. Clearly I should take some tips from the Udacity instructional designers.

Three. A word on assessments. Assessments of various types are embedded in the course site, in the same space that the video displays. Assessments take two forms: quizzes and writing Python code. The quizzes are single- or multiple-answer multiple choice questions using, radio buttons or checkboxes. These are not factored into the student’s final grade; they’re just self-assessments. While, as I’ve said, the content is so far not new to me, I’m still finding these useful for slowing things down & making me articulate what I know, even if only to myself. As for writing Python code: again, in the same space that the video displays, a Python interpreter appears and you have to write one simple program per assignment, run it, and when you’re satisfied with the result, submit it. It’s then automatically evaluated, which I assume means some algorithm checks that you got the right result. It’s not clear to me if the code itself is checked. Maybe that will become clear later as the assignments get more complex.

Anyway, embedding the assessments in the course site in the same space that the video displays is a neat trick. It makes the experience of the course very clean and seamless, since it’s all right in the same screen real estate. And I’m really curious to know how they embed quizzes into the videos. Yvonne tells me that there are several tools that can do this, including Camtasia Studio, which I have a license for. So I’ll have to experiment with that.

I’ll post this now, before I run out of steam. Stay tuned, gentle reader, for the further adventures of me in CS 101.