Instructor Data Analytics: the most boring conference presentation tile in the history of this conference, ever.

 Link to most recent conference slides:

https://docs.google.com/presentation/d/1ZQmZmQAo7k1Orh_GQuSGyv1T35SLdeBH_IpUOPXxktI/edit?usp=sharing

A list of quotes that I thought about including:

https://docs.google.com/document/d/1LtiKxja3ESCiSEWOsdzB-tb1SabW3hTPvNZujmD1ops/edit?usp=sharing

 

A bundle of bookmarks related to this presentation and related topics.  Not that curated:

https://pinboard.in/u:craniac/bundle:cw2023/

 

Here is the plain text of my slides.  I apologize that the images are missing and have not been replaced with captions.

Instructor Analytics in Learning Management Systems

Surveillance, Instructor Autonomy and Resistance

Canvas makes an announcement
So four years ago, the then-president of Instructure Dan Goldsmith announced that the company was going to become a data analytics company.   They also were moving into the corporate learning space with a product called Bridge, which was subsequently sold.  Instructure is always expanding, looking for untapped markets and buying up smaller companies.  Phil HIll  noted that Instructure was harvesting student data to train machine learning algorithms and Laura Gibbs has addressed some of the other implications.

[

Canvas employee responds
The fact is, our valuation is based on our revenue that comes from customers licensing our software, our tremendous growth rate, and our expansion beyond LMS. It wasn't based on student data, and we have zero plans to "monetize" student data. We've done nothing to show we are anything but good stewards of data :/ (2020)

Worker Analytics are not new.



Scientific management isn’t that scientific
This was the second coming of Taylorism, AKA "scientific management," an early 20th Century pseudoscience practiced by high-priced, unaccountable consultants who would fan out on factory floors in literal science cosplay, including lab-coats and clipboards, and loom over workers, watching their every movement, often going so far as to film them.

Then the consultants would go back to their workplace, soup the negatives, and produce completely arbitrary workplace rules about how workers must stand, hold their limbs and heads, and move – it was a kind of corporate anti-yoga whose asanas were designed to make workers look productive to their bosses, who often had no idea how to perform or evaluate their jobs. --Cory Doctorow, “Great Tayor’s Ghost”

https://pluralistic.net/2022/08/21/great-taylors-ghost/#solidarity-or-bust

Back in 2021, as we returned from the pandemic, our Dean summarized our student evaluation scores and sent us these dashboards:


Even good teachers can get the red light



Septiles!  
To address the fact that student ratings produce a skewed distribution of scores, I have been looking at the data using a number of different approaches. I think that dividing the data into septiles is promising because it makes patterns and relative positions easily visible versus looking at the raw scores. An explanation of septiles follows this letter.  --an administrator, used without permission

So to interpret this, the higher the number, the better.  The color coding (*cough* accessibility *cough*) represents where an instructor sits in relation to other instructors within the entire college.  So for an attribute scored from 1-5, you could receive an 85 percent approval, or a 4.25, but if everyone else in the college scored even marginally higher, you would still receive the red warning alert on your student evaluation dashboard.

Already feeling exhausted by the previous years of pandemic teaching, this dashboard was not well received by our department.

Limits of the Numerical

Managerial culture

Bureaucratization and Normative Influences
We argue that the causes of bureaucratization and rationalization have changed. The bureaucratization of the corporation and the state have been achieved. Organizations are still becoming more homogeneous, and bureaucracy remains the common organizational form. Today, however, structural change in organizations seems less and less driven by competition or by the need for efficiency. Instead, we will contend, bureaucratization and other forms of organizational change occur as the result of processes that make organizations more similar without necessarily making them more efficient. Bureaucratization and other forms of homogenization emerge, we argue, out of the structuration (Giddens, 1979) of organizational fields. This process, in turn, is effected largely by the state and the professions, which have become the great rationalizers of the second half of the twentieth century. For reasons that we will explain, highly structured organizational fields provide a context in which individual efforts to deal rationally with uncertainty and constraint often lead, in-the aggregate, to homogeneity in structure, culture, and output.  

DiMaggio, Paul J., and Walter W. Powell. “The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizational Fields.” American Sociological Review, vol. 48, no. 2, 1983, pp. 147–60. JSTOR, https://doi.org/10.2307/2095101. Accessed 22 June 2023.


Speculative fiction author and cultural critic Bruce Sterling once wrote that “​​Anything that can be done to a rat can be done to a human being. And we can do most anything to rats. This is a hard thing to think about, but it’s the truth. It won’t go away because we cover our eyes.”

Trigger warning: I am going to show a picture of a rat with a brain implant

Inasmuch as ye have done it unto the least of these

The Loneliness of the Long Distance Trucker
Read the truck driver excerpt

The future of online instruction


Canvas is a panopticon waiting to happen
According to a very recent conversation with a service desk level employee, the tools don’t currently exist to monitor instructor behavior in Canvas, rendering this entire presentation an exercise in fear mongering.

Moodle = Open Source = FULL SURVEILLANCE!
Well I don't know if the dead can see it, but anyone who can see student logs in our LMS can see instructor logs, too. We can filter logs by item, activity (view, submit, change completion, etc.), and role (student, instructor), or show logs for any individual participant, whether student or instructor. We can also set specific assignments to send grading reminders (just like notifications for work submitted), but I've never done that.
ETA: Sorry, by "anyone," I mean anyone with permissions to view the logs for a given course, which of course doesn't include students, only participants with role of instructor or administrator.
(/r/professors)

Hypothetical instructor analytics could measure many behaviors:

time to grade
engagement with class
interactions with students
Analysis of feedback
Emotional evaluation
Something something AI!

Pressure washing
A number is always presented as representing a given phenomenon. However, agents sometimes use numbers which are “spuriously precise,” in the sense that they provide a more exact representation than is—or could ever be—warranted by evidence. Agents deploy these numbers sometimes maliciously and opportunistically but sometimes also because there are strong pressures to have a number, any kind of number. Such pressures can come from a need for coordination, ease of communication, or for accountability.  -

--Newfield, Christopher. Limits of the Numerical The Abuses and Uses of Quantification

Forces at the state level shape administrative culture

A Gift From our Corporate Overlords: PERFORMANCE-BASED FUNDING

TIME TO COMPLETION OR GRADUATION

Utah, PDF circa 2001

Is PBF an intentional attack on higher education?

Surprisingly, Performance Based Funding in higher education doesn’t seem to work

Teaching is complex
using outcomes as a management tool is difficult because public services are delivered through complex organizations where tasks are not routine and are inherently difficult to define and measure.
--Nicholas Hillman, “Why Performance Based Funding Doesn’t Work”




Technology structures our desires and goals
Good teaching is a combination of art and skill and experience. I’m of the firm belief that no amount of data capture is going to be able to reproduce that. This is apparently a fringe belief in educational  technology circles.

When I’m in a classroom, I’m not interested in predicting what I think the student is going to do or what grade they might get. That is actually completely irrelevant to me, but prediction is central to how this technology operates.

--Chris Gilliard

PBF is great for simple and routine tasks
Performance-based funding regimes are most likely to work in non-complex situations where performance is easily measured, tasks are simple and routine, goals are unambiguous, employees have direct control over the production process, and there are not multiple people involved in producing the outcome.

“Why Performance-Based College Funding Doesn’t Work.” The Century Foundation, 25 May 2016, https://tcf.org/content/report/why-performance-based-college-funding-doesnt-work/.

performance-based funding/technology will invariably erase what it cannot measure.

Phil Agre’s Capture
When the presenter uses a screencap
Because they don’t want to type it all out.

I don’t mean to impose
In Phil Agre's model of capture, the final step is "Imposition" or taking the model that you have gathered, which is a reductive version of reality, at which point "the resulting grammar is then given a normative force." Instructors are then "induced to organize their actions so that they are readily 'parseable' in terms of the grammar"

Gradually, these “grammars of action” change our behavior

Agre, Philip E. “Surveillance and CaptureTwo Models of Privacy” Information Society 10(2): 101–127. April–June 1994.


It is not difficult to imagine these "grammars of action" emerging within an LMS.  For example, Canvas (and other LMS's) encourage USERS to create a bank of responses.  In addition, rubrics can be saved, reused and shared.  Entire courses can be duplicated and exchanged, and then have a normative effect on teaching.


Parseable behaviors

We Use Our Tools, and Our Tools Use Us
Using the Palm Pilot’s Graffiti alphabet as a crude metaphor, an LMS may subtly shape our teaching over time.  We tend to avoid activities that are difficult to fit into the LMS ecosystem, and eventually the preferences of the LMS start to feel normal and natural (Burke, Althusser).

Like religious supplicants, we respond with gratitude when an LMS adds a feature that grants us more agency, for example.

I see similar expressions of gratitude from our adjuncts when we loosen up the constraints of our course shells.

Note for myself
How is using (or imposing) a standardized course in an LMS any different from simply publishing a textbook, and using that book to teach and constrain the activity of new instructors?

Big Data Energy and religious fundamentalism

Invisible Labor
Not all of our work is visible.  Canvas will never see you reading for class, for example.  Canvas doesn't know if that's a new assignment you created over the summer or if you're recycling an older course.  Canvas doesn’t know when you wake up at 3:00 a.m. thinking about that day’s class.

Evading capture


Questions
1. How does capture shape our teaching?
2. To what degree is UDL complicitous with capture?
3. How to we effectively assess our teaching?
4. How do we resist and control this conversation?
5. What would an online teaching space look like that would not shape our behavior, and could it scale to support a writing program?
6. How do we address our own complicity in Writing Programs with spaces that intentionally shape instructor behavior?

Possible responses
It has become clear that we have to protect the adjunct instructors and students, not just because it’s the right thing to do, but because we end up helping everyone who falls under the panoptic eye, setting a precedent for responsible data management.

The language of holistic instructor assessment    
Like an academic who is entering the private sector, we need to take our teaching experiences and convert them into a language that is readable by both deans and machines.  

Compelling narratives rather than the strictly numerical

Women Shaping Policy
Author Kim Stanley Robinson, in a presentation on climate change at the University of Utah, recently described attending a climate summit at the United Nations, and contrasted the front-facing senior politicians, often older, powerful men, with the rooms full of lawyers, often women, hashing out policy in the back rooms, engaged in dialogue and getting it done.

Women Shaping Policy
At a recent campus workshop on faculty governance, our faculty senate president noted that it was often women faculty who were very good at, and committed to, slogging through the often tedious, detailed work of writing and revising policy for our school.

Our relationship to the LMS (and all policies) needs to be transparent.



Does your contract mention using an LMS?
Is there any self-governance on your campus?
Is FERPA invoked to prevent using some other LMS?

FERPA: It Cuts Both Ways
Institutions love to invoke FERPA to lock instructors into an LMS, but conveniently ignore egregious abuse by Education Technology Companies

The Master’s House
Privacy and the Convenient and Arbitrary Abandonment of FERPA is a Moral Insult

Generally, student privacy is protected under the Family Educational Rights and Privacy Act (FERPA), a federal privacy regulation for K-12 and higher education that affords students or their guardians some control over the disclosure of personally identifiable information in education records. However, ed tech companies can comply with FERPA without consent from the user under a loophole called the “school official exception,” which allows schools and districts to disclose this information to educational service providers. In these LMSs, student and instructor personal information, such as names and identification numbers, is frequently attached to classroom materials, as well as to user-generated content and assessments. Every assignment students submit and every video that instructors upload through the system is collected and attached to metadata like time, IP address, and even tracked cursor history in some instances. Additionally, there is no FERPA protection for end-users’ metadata and de-identified data, which can be easily recombined to individually identify students.

https://www.thenation.com/article/society/canvas-surveillance/

It’s not too late
“One of the arguments people like to make is that you can’t go back on particular technologies — you can’t put the genie back in the bottle,” Gilliard says. “That’s patently untrue. No one would look at asbestos and say, ‘Well, you can’t outlaw chemistry.’ But they look at facial recognition and say, ‘You can’t outlaw math.' ”

--Chris Gilliard

Find the right group.
In a truly acute crisis, survival is the only thing, and only the unsubtle benefits and downsides of being in a group or alone matter. But in a sustained, chronic crisis where you need to establish some sort of homeostatic equilibrium in relation to the adverse circumstances, who you’re navigating the crisis with matters.

The wrong group will slowly drain your will to persist, even if it has the right configuration of skills and resources to survive the situation. You will gradually lose sight of the point of the struggle, and your willingness to participate in it.

The right group will not only increase the odds of all individuals through a practical alchemy of capabilities, it will deeply intensify and enhance your sense of what is at stake in the struggle. For many, a time of crisis can be so rich with significance and meaning, when normalcy returns, it can seem impoverished. --Venkatesh Rao

Embrace the Heterogeneous
As George Madaus (1994) cautioned, institutions should not think about trying to homogenize the heterogeneous. In fact, teachers should embrace the heterogeneous because of its connection to multiple perspectives and critical thinking.  

--Vicki Hester

Compassionate Administration
surveillance does not equal care.
surveillance cannot guarantee safety.
surveillance will not alleviate anxiety.
surveillance cannot substitute for presence.
surveillance can undermine trust and responsibility.
--Audrey Watters

we should acknowledge that most people are forced to live inside someone else's imagination and one of the things we have to come to grips with is how the nightmares that many people are forced to endure are the underside of elite fantasies about efficiency, profit and social control.

Ruha Benjamin on "The New Jim Code? Race, Carceral Technoscience, and Liberatory Imagination" (2019)

Everything from this point on is me freaking out about capturing instructor data and using it to create artificial intelligences to erase our jobs

The Persistence of Captured Data
Agre points out that the information from tracking or capture usually ends up in a database. (1994)



Bureaucracies use data to create profiles, or scarecrows


*“I mean, I guess I technically read texts written by people who’ve passed all the time,” Ansuini said.

*“But it’s the fact that I looked up his email to send him a question and PULLED UP HIS MEMORIAM INSTEAD that just THREW ME OFF A LITTLE.""

Dead instructors don’t complain about enrollment caps.

Today’s horror is tomorrow’s startup