Homework Truthweaver

2023 — A web app that ranks homework pairs by how much wording they share

01 Problem

In November 2023 my homework was rejected because I had written it with ChatGPT. Classmates who had copied one another and run the copy through a paraphraser, a tool that rewords text, were not caught, and their work was accepted.

They were not caught because nobody checks. A teacher grading by hand reads one homework at a time, and spotting a copy means remembering every homework read before it. A class of 15 already makes 105 pairs to compare, and a paraphrased copy does not look like a copy at a glance.

A sad kitten labelled ChatGPT users under a red Rejected, beside a smiling kitten labelled Paraphraser users under a green Accepted

02 Solution

Homework Truthweaver does the comparing. A teacher gives it the homework of a class, and it measures how much wording every pair shares and raises a red flag on any pair at or above a threshold the teacher sets. A copy keeps much of the wording of its source even after a paraphraser, so it rises to the top and the reading starts there. The flag only says where to look, and the teacher still judges.

It was a web app, built in R with Shiny, a framework that turns R code into a web page, and hosted on shinyapps.io until it went offline. The teacher uploads the homework as PDFs, sets the threshold on a slider, and presses Conjure. Back come analytics of every pair and a Pair Analysis to compare any two.

For anyone without a class at hand, it came with 15 sample essays a chatbot wrote as rewrites of one another, and every screen in this notebook uses them. The rest of this notebook follows one press of Conjure, from the PDFs to those screens.

Two parchment documents under a High Similarity Detected banner, document B being a paraphrase of document A

The input form of the Shiny app, with the red flag threshold at 50% and the Conjure button

03 From PDF to text

A homework in a PDF looks like text to a person, but to a program it is a set of characters placed at positions on a page. Before two homeworks can be compared, each PDF has to give up its words. The app reads the text layer of every page with pdftools, a PDF library for R, joins the pages into one text, and names it after its file. Anything that is not text, like an image, counts for nothing.

folder_to_table <- function(folder_path) {
  convert_pdf_to_text <- function(pdf_path) {
    pdf_text_content <- pdf_text(pdf_path)

    extracted_text <- list()

    for (page in seq_along(pdf_text_content)) {
      text <- pdf_text_content[[page]]
      extracted_text[[page]] <- text
    }

    all_text <- paste(extracted_text, collapse = "\n")
  }

  get_file_name <- function(file_path) {
    file_path_sans_ext(basename(file_path))
  }

  pdf_files <- list.files(folder_path, pattern = ".pdf", full.names = TRUE)

  pdf_texts <- lapply(pdf_files, convert_pdf_to_text)

  table_data <- data.frame(
    text = unlist(pdf_texts)
  )

  rownames(table_data) <- get_file_name(pdf_files)

  return(table_data)
}

04 Text cleaning

Two homeworks can use the same word without a program noticing, since "Paints," and "painting" are different strings to it. Cleaning removes what does not matter, so that only real differences in wording are left.

Every text is lowercased and stripped of numbers, punctuation, and stopwords, the common words like "the" and "dan" that say nothing about the content. The stopwords cover English and Indonesian, since homework in an Indonesian class comes in either language. Every word is then cut to its stem, its root form, so "paints" and "painting" count as one term, and leftover runs of spaces collapse into one.

A stem is not always a real word, which is why the screens later in this notebook show "masterpiec" for "masterpiece".

clean_text <- function(text, as.corpus = T, lower = T, rm.number = T, rm.stopwords_english = T, rm.stopwords_bahasa = T, rm.punctuation = T, stem = T, rm.whitespace = T){
  text_corpus <- text %>% VectorSource() %>% VCorpus()

  if (lower){
    text_corpus <- tm_map(x = text_corpus,
                          FUN = content_transformer(tolower))
  }
  if (rm.number){
    text_corpus <- tm_map(x = text_corpus,
                          FUN = removeNumbers)
  }
  if (rm.stopwords_english){
    list_stop_words_english <- readLines("stop-words_english.txt", warn = FALSE, encoding = "UTF-8")

    text_corpus <- tm_map(x = text_corpus,
                          FUN = removeWords,
                          list_stop_words_english)
  }
  if (rm.stopwords_bahasa){
    list_stop_words_bahasa <- readLines("stop-words_bahasa.txt", warn = FALSE, encoding = "UTF-8")

    text_corpus <- tm_map(x = text_corpus,
                          FUN = removeWords,
                          list_stop_words_bahasa)
  }
  if (rm.punctuation){
    text_corpus <- tm_map(x = text_corpus,
                          FUN = removePunctuation)
  }
  if (stem){
    text_corpus <- tm_map(x = text_corpus,
                          FUN = stemDocument)
  }
  if (rm.whitespace){
    text_corpus <- tm_map(x = text_corpus,
                          FUN = stripWhitespace)
  }

  if (as.corpus){
    return(text_corpus)
  }
  else(
    return(sapply(text_corpus, as.character))
  )
}

df$text <- clean_text(df$text, F)

05 N-grams

A clean homework is still one long string, and two long strings are almost never equal, even when one is a copy of the other. To compare homeworks, the app has to cut each one into smaller pieces, and those pieces are called n-grams.

An n-gram is a run of n neighbouring words. The table below cuts one sentence into 1-grams, 2-grams, and 3-grams, also called unigrams, bigrams, and trigrams. The window moves one word at a time, so the seven words of the sentence give seven unigrams, six bigrams, and five trigrams.

The similarity score uses unigrams, which suit a paraphrased copy. One changed word breaks every trigram that contains it but leaves the other unigrams alone, so single words still show what a copy kept. Unigrams shorter than three letters are dropped at this point, so "ai" never counts toward the score. Bigrams and trigrams come back later, in the word clouds.

The sentence The Margherita pizza is not bad taste cut into seven 1-grams, six 2-grams, and five 3-grams

06 TF-IDF weighting

With every homework cut into unigrams, the plain idea is to count the ones two homeworks share. That fails in a classroom, because everyone writes on the same assignment. If the assignment is about artificial intelligence, every essay says "intelligence", and counting makes honest homeworks look copied.

TF-IDF gives each word a weight instead of a count. TF, the term frequency, is how often the word appears in a homework as a share of its words, so it grows the more that homework uses it. IDF, the inverse document frequency, shrinks the more homeworks use the word, so a word found in every homework weighs nothing and a word found in only a few weighs the most. The weight is the two multiplied, so it is high only for a word this homework uses a lot and few others use at all.

In the formulas below, $n_{t, d}$ counts word $t$ in homework $d$ , $N$ is the number of homeworks, and $N_{t}$ of them use $t$ .

{TF}_{t, d} = \frac{n_{t, d}}{\sum_{s} n_{s, d}}

{IDF}_{t} = \log_{2} \frac{N}{N_{t}}

w_{t, d} = {TF}_{t, d} {IDF}_{t}

07 Cosine similarity

TF-IDF turns each homework into a long list of weights, one per word in the class. A list of numbers like that is a vector, an arrow pointing somewhere in a space with one direction per word. Two homeworks that use the same words in the same proportions point the same way.

Cosine similarity measures the angle between two of those arrows. The cosine is 100% when they point the same way and 0% when they share no word. It looks at the direction and not the length, so a long homework and a short copy of it can still score high.

The app does this for every pair and keeps the results in a similarity matrix, a table with every homework on both its rows and its columns.

\cos (a, b) = \frac{\sum_{t} w_{t, a} w_{t, b}}{\sqrt{\sum_{t} w_{t, a}^{2}} \sqrt{\sum_{t} w_{t, b}^{2}}}

get_cosine_similarity_matrix <- function(df) {
  corpus <- VCorpus(VectorSource(df$text))

  dtm <- DocumentTermMatrix(corpus)
  rownames(dtm) <- rownames(df)
  dtm_tfidf <- weightTfIdf(dtm)

  dtm_matrix <- as.matrix(dtm_tfidf)

  similarities <- simil(dtm_matrix, method = "cosine")

  similarity_matrix <- as.matrix(similarities)

  similarity_matrix[is.na(similarity_matrix)] <- 1

  return(similarity_matrix)
}

08 Pair ranking and red flags

A matrix of numbers still does not tell a teacher where to start, so it becomes a ranking. The matrix holds every pair twice, once in each order, so the pairs are sorted from most similar down and one of each two is kept. Every pair at or above the threshold is flagged red and the rest green.

The right threshold depends on the class, so the analytics show how similar its pairs usually are. The median gives the typical pair, and a density chart shows how the pairs spread across similarity, with the threshold drawn across it. In the sample essays the typical pair shares 5%, nearly all pairs sit below 25%, and a thin tail stretches past 90%. A threshold of 23% sits at the edge of that crowd and flags 11 of the 105 pairs.

The analytics with the threshold at 23%, showing a median similarity of 5%, 11 red flag pairs, 13 red flag homeworks, and the density of pair similarities

get_pair_similarity_ranking <- function(similarity_matrix, red_flag_threshold){
  similarity_long <- as.data.frame(as.table(similarity_matrix))

  similarity_sorted <- similarity_long %>% arrange(desc(Freq))

  ranking <- similarity_sorted %>%
    filter(Var1 != Var2,
           row_number() %% 2 != 0) %>%
    mutate(Documents = paste(Var1, "&", Var2)) %>%
    rename(Document_1 = Var1,
           Document_2 = Var2,
           Similarity = Freq) %>%
    mutate(Similarity_Formatted = round(Similarity * 100) %>% paste0("%"),
           Flag = ifelse(Similarity >= red_flag_threshold, "RED", "GREEN")) %>%
    select(Documents, everything())

  return(ranking)
}

09 Homework ranking

A pair tells a teacher which two homeworks to read side by side, but grading happens one student at a time. The question there is whether this homework has a twin somewhere in the class. So each homework takes the score of its closest match and is flagged red when that match is. At 23%, 13 of the 15 sample essays are red.

The page shows both rankings next to each other, the 12 most similar pairs and the 12 most similar homeworks, as bar charts above searchable tables. Cosmos and Nova lead both at 92%.

The 12 most similar pairs and homeworks as bar charts above their tables, led by Cosmos and Nova at 92%

get_homework_similarity_ranking <- function(similarity_matrix, red_flag_threshold){
  similarity_long <- as.data.frame(as.table(similarity_matrix))

  ranking <- similarity_long %>%
    filter(Var1 != Var2) %>%
    rename(Highest_Similarity = Freq,
           Document = Var2,
           Other_Document = Var1) %>%
    group_by(Document) %>%
    top_n(1, wt = Highest_Similarity) %>%
    mutate(Flag = ifelse(Highest_Similarity >= red_flag_threshold, "RED", "GREEN"),
           Highest_Similarity_Formatted = paste0(round(Highest_Similarity * 100), "%")) %>%
    arrange(desc(Highest_Similarity)) %>%
    select(Document, Other_Document, Highest_Similarity, Highest_Similarity_Formatted, Flag)

  return(ranking)
}

10 Pair analysis

A score says two homeworks are alike but not why, and a teacher cannot accuse anyone on a number. Pair Analysis is where the teacher checks.

Given two homeworks, it shows their similarity and flag, then splits their words into three word clouds, where a word is drawn larger the more often it appears. One cloud holds the words only in the first, another holds the words both use, and the last holds the words only in the second. Aurora and Pulsar score 68%, and their shared cloud is led by "ai", "rorschach", and "inkblot", so both essays argue through the same metaphor. The clouds keep short words, so "ai" shows here even though it never counted toward the 68%.

Under the clouds, a teacher can switch them from unigrams to bigrams or trigrams, the longer n-grams from before. A cloud of single words can only say that both homeworks use "ink". It cannot say whether they use it in the same phrase, and a copied phrase is what a teacher is looking for. Words outside a cloud are removed before its n-grams are cut, though, so a bigram in the shared cloud can join two words that were not neighbours in either homework. It is a hint of where to read in the PDFs and not proof of a copied phrase.

Below the clouds both PDFs open side by side, each with its own flag and a table of the homeworks it matches at or above the threshold, so the checking ends with reading the homeworks themselves. Given only one homework, the app shows its flag and the homeworks it matches.

Pair Analysis, where Aurora and Pulsar are flagged 68% similar, with word clouds of the words unique to each and the words they share

Aurora and Pulsar side by side, each flagged as red flag work with the one similar homework it matches

get_intersect_and_distinct_words <- function(text_1, text_2) {

  text_1_tokens <- unlist(strsplit(tolower(text_1), "\\s+"))
  text_2_tokens <- unlist(strsplit(tolower(text_2), "\\s+"))

  combined_tokens <- c(text_1_tokens, text_2_tokens)

  intersect_tokens <- intersect(text_1_tokens, text_2_tokens)

  words <- list(
    "document_1_distinct" = text_1_tokens[!text_1_tokens %in% intersect_tokens] %>% paste(collapse = " ") %>% as.character(),
    "intersect" = combined_tokens[combined_tokens %in% intersect_tokens] %>% paste(collapse = " ") %>% as.character(),
    "document_2_distinct" = text_2_tokens[!text_2_tokens %in% intersect_tokens] %>% paste(collapse = " ") %>% as.character()
  )

  return(words)
}

get_wordcloud <- function(words, palette, n_gram = 1){
  words_list <- tokenize_ngrams(words, n = n_gram) %>% unlist()
  upper_scale <- ifelse(n_gram > 1, 2, 3)

  words <- data.frame(word = words_list) %>%
    count(word, sort = TRUE)

  words %>%
    with(
      wordcloud(
        words = word,
        random.order = FALSE,
        color = brewer.pal(9, palette),
        min.freq = 1,
        scale = c(upper_scale, 1),
        rot.per = 0,
        freq = n,
      )
    )
}

get_similar_works <- function(pair_similarity_ranking, document) {

  similar_works <- pair_similarity_ranking %>%
    filter(
      document == Document_1 | document == Document_2,
      Flag == "RED"
    ) %>%
    mutate(
      Document = gsub(paste(c(document, "&"), collapse = "|"), "", Documents),
      Document = trimws(Document)
    ) %>%
    select(Document, Similarity_Formatted) %>%
    rename(Similarity = Similarity_Formatted)

  return(similar_works)
}

11 System design

Every step above ran inside one Shiny program in R, deployed on shinyapps.io, a hosting service for Shiny apps. The teacher's browser sent the PDFs and the threshold. The server kept the PDFs where Pair Analysis could show them, read their text with pdftools, cleaned and weighted it with tm, measured every pair with proxy, and the page drew the results with plotly, DT, and wordcloud, cutting the n-grams of the clouds with tokenizers.

The uploaded files were cleared on every new upload and when the session ended, and nothing was kept in a database.

12 Limits

The app measures how much weighted vocabulary two homeworks share, not how much text was copied. A rewrite that keeps the argument and changes the words scores low, and changing the words is what a paraphraser does, so a thorough one can still slip under the threshold.

It also says nothing about who wrote a homework, so one written alone with a chatbot scores low against the rest of the class.

It reads only the text layer of a PDF, so images count for nothing and a scanned page without text scores zero against everything.

Homework Truthweaver

A teacher sees which pairs of homework share the most wording, flagged red at a threshold they choose, in one press