Since commentaries are linear lists of annotations on a foreign language source text, we argue this construction approaches the definition of a hypertext. Many commentaries try to provide full clarity on the unfamiliar historical, cultural, and linguistic contexts these texts were written in [5]. Scholars wrote commentaries on texts in many languages for millenia [12]. The first serious editions and commentaries appeared in Alexandria in the 3rd century BCE, and fragments of this work exist on ancient papyri [13]. While a well-developed model for page-based commentaries emerged more than 1,000 years ago, the commentary adopted its modern form by the 18th century [24].

One commentary item type is a claim about the frequency of a phenomenon in the source text. Often the claim is supported by a set of “parallels," or various primary source references for that phenomenon. Commentaries and commentators differ in focus and habit, and the number and quality of parallels were thoroughly surveyed and questioned by commentary critics [14, 18]. Parallels may be a form of or alongside a similar phenomenon we call “coverage creep." When writing a non-initial commentary on a text, convention compels the commentator to echo relevant insights from these commentaries and provide new insights. However, what was once scholars in the 1800s referencing an important manuscript is now the duty of modern commentators to cover centuries of analysis. Plus, a drop in references per item over the last century could be ascribed to lowering the cost of accessing all references supporting a commentator's claim [5]. This decrease in cost helps lower the barrier of entry to engaging with commentaries, as the study of Greco-Roman literature's audience broadened past the upper class. Lastly, in the late 90s/early 2000s, with the advent of computerized literature and hypertext, commentaries’ critical analysts called for varying radical commentary redesigns in a new digital medium [16, 26, 34]. Many hypertext reading environments still represent commentaries where their most radical design change is hyperlinking reference sets, as seen in environments like [17]. Born-digital commentaries show what commentaries could be, yet many digitized commentaries are mostly unaugmented and unrevised.

There are still shortcomings in commentaries that may be clearer to scholars of CL and accessibility, educators, students, and other unconventional and/or underrepresented types of language learners. First, humanist academics write mostly for their peers. [5] found that authors often support claims with parallels without accounting for lexical dispersion and other factors that may invalidate the claim. Next, claims can be vague or misleading. Charles Smith's 1913 commentary on Thucydides’ History of the Peloponnesian War said the verb λωφάω was “used by Thuc. with reference to sicknesses and grave misfortunes" [33]. Searching for all instances of λωφάω in the History, 2 instances reference the Athenian plague, and another refers to other misfortunes Athenians overcame. Given the note's misleading phrasing, readers may think this verb is used more than 3 times in the History. These claim analyses must also be accessibly explained. [5] reports that when academics write commentaries aiming to aid both newer and more advanced language learners, the content often underserves the former. The aid newer language learners need is often absent since commentators assume novices do not read this, leading to their exclusion of not just the novices from understanding the commentary.

Tompkinsia will also aid both hypertext completionists and those uninterested in these notebooks’ analyses. Treebanks have existed for decades, yet while many CL researchers analyzed many treebanks with their own bespoke workflows, such analyses are not often front-facing or novice friendly. This work could lower the barrier of entry into this kind of analysis to those familiar with Jupyter notebooks. Augmenting these commentaries with claim analysis may also keep the cursory reader from being misled by claims with notes summarizing their inconsistency or veracity.

Lastly, this work's scope matters regarding claim validation. Like the analytical work of Dan Tompkins [35], this design's namesake, the framework centers data-driven CL analysis of unambiguous claims (i.e., “Sentences ended with a genitive absolute are rare"). A claim like “Odysseus uses the gnomic aorist when in a rage" is more ambiguous and harder to find with treebanks. Finding aorists in Odysseus’ dialogue may be easy, yet checking results may be time consuming with no notation for what aorists are gnomic or what emotions are there. One could also use a keylist of anger words to further filter the results. Making an accurate keylist with rarer words takes time, but a valid instance could always be excluded. A third method may be training a BERT model to identify Odysseus’ anger, and evaluating its results. Tompkinsia is a presentation medium for checking claims made on a reliable, searchable, and mathematical foundation, and this design centers showing what can be found and analyzed in an unambiguous way.

There were n = 12 participants. 10 were undergraduates in a second-semester class reading ancient Greek literature, and 2 were experts who teach CL methods and are familiar with ancient Greek. All were either students or instructors at Tufts University. 1 participant was nonbinary, 5 were women, and 6 were men. Informed consent was obtained for all. Participants were alternatingly assigned to one of 2 experimental groups. Due to technical difficulties, we were unable to record the screen of 1 expert as they browsed.

4.2.1 Selecting A Hypertext. We chose lines 37-65 of Book 6 of Homer's Iliad as our source text, with excerpts from the Basel Commentary volume by Magdalene Stoevesandt. To choose between the Basel Commentary and Walter Leaf's 1886 Iliad commentary, a focus group of masters students in digital humanities was convened. Based on discussion with the focus group and the diversity of claims presented in the works, the Basel Commentary was chosen. To be familiar with the text, students were told to read the lines on their own before the user study. This study also features some of the History with two claims from Smith's commentary. The History is a text students may have greater difficulty with translating, but we included the text and claims so that participants did not need to translate any Thucydides, and emphasis was put on clearly explaining how the claims posed a challenge to the reader. That said, a common reason focus group members chose the Basel Commentary was because it was contemporary. Another reason they chose the Basel Commentary was its attention to the source text as a work of poetry. There are notes on the frequency of formulae and epithets, but there is still no universal consensus on frequency's relationship to the Homeric Formula [28]. Participants had up to 10 minutes of browsing time in Tompkinsia: enough time to read and re-read as needed in a time-bounded task setting.

4.2.2 Commentary Augmentation. We needed a portion of the Basel Commentary with some treebank-verifiable claims we could present analyses for while demanding just enough translation and review of the Iliad from participants. [5] already identified claims in this commentary; we chose from those, each roughly 20-30 lines presenting different premises that were easy to analyze and follow at any expertise level. Lines 37-65 did not start or stop abruptly, and corresponding notes included claims regarding a bow epithet, a two-word phrase, and a verse end or VE formula, which we analyzed in Tompkinsia. The first claim proposed that ἀγκύλοv, meaning curved or crooked, only modified the weapon word ἅρμα here and usually modified the bow word τόξov as a bow epithet [7]. The second proposed that σὺ δέ occurred mostly in Homer and Herodotus [7]. The last claimed that ἔπος ηὔδα was a VE formula always preceded by a participle [7]. Claims were translated as statements evaluatable by querying for and verifying instances of a phenomenon in the treebanks. Statements were then evaluated by their respective metrics, and these results were evaluated as a group to determine whether the base claim was valid. This completed base analysis was reviewed and adapted into a Python notebook where the process was explained step-by-step in an accessible way, with code users could run and tweak. These notebooks are available at [3].

We also removed items depending on references or a greater assumed knowledge of ancient Greek. Because our newer language learners had not read the Iliad, we removed items that would not make sense to a reader without exposure to ancient Greek poetry. We also removed notes that mainly pointed to where else the reader could see a list of occurrences or information explaining something the Basel commentary did not. Students often translate with many digital and analog aids, but we wanted to see if Tompkinsia could change reader perception of commentaries without interrupting the reading flow as pursuing references usually does.

4.2.3 Tompkinsia Design.

The Tompkinsia website depicting one view of this hypertext reading environment. The site consists of a white source text pane taking up the left two thirds of the screen, and the white commentary pane taking up the remaining third. These panes are outlined with a thin offwhite border. On the left is a section of the Iliad in black Greek text (6.45-658 specifically). There are line numbers for every multiple of 5 on the left-hand side of the text. In the commentary pane, there are 6 normal commentary notes in a cornflower blue, each one beginning with the relevant line numbers, followed by a bolded substring from the source text, which are in turn followed by notes in English text. The seventh note is a claim in midnight blue text, under which is a summary of what the notebook view finds. This summary is also this midnight blue text, enclosed in a beige rectangle with a rounded black border. Below that are two buttons: a cornflower blue button reading “Show Claim Analysis" and a gold button reading "Download Notebook".

The Tompkinsia website depicting one view of this hypertext reading environment. The site consists of a white source text pane taking up the left two thirds of the screen, and the white commentary pane taking up the remaining third. These panes are outlined with a thin offwhite border and above them both is a thin midnight blue header whose white heading reads “Tompkinsia." On the left is a representation of a Jupyter notebook assessing the claim in the commentary pane. This representation is enclosed in an iframe with a cornflower blue border that users can scroll around in. Inside the iframe, the notebook begins with a heading in large bolded black text. In thinner, body-size black text, the claim is repeated and interpreted. There is material in three slightly smaller black headings that break verification of the claim into steps. 0) contains just a list of requirements for running and tweaking the notebook, 1) investigates the presence of agkulon and arma, and 2) is the heading for an epithet investigation not present in this image. Right under 1), we can see some plain black text and a pale yellow box containing monospaced text meant to signal that it is a code cell. Below that, we have some more clarification of what we're trying to find, and then a larger code chunk for finding all possible instances of agkulon modifying arma. The table of results below consists of a white background, black borders, and black text, showing just one result: the instance the discussed note is attached to. The last thing following that and before 2) is an assessment of whether this verifies the first part of the claim.

In Fig. 1 the Tompkinsia reading environment shows the Iliad and the Basel commentary. The source text is in the left pane, and the Basel commentary is to the right. [6] shows that digital representations of commentaries in a multi-document hypertext space can seem like an overwhelming wall of text. Rather than reprise that design, we made claim comments with notebook analyses a navy blue and comments without notebook analyses a lighter blue. To view a static Jupyter notebook associated with a claim, clicking a blue button in the claim on the right replaces the contents of the left pane with the static notebook view, as Fig. 2 shows. Here, an iframe shows the notebook as Markdown, code, and output sections as common in the Jupyter paradigm, but code sections are static formatted text that do not run in browser. Pressing the blue button reverts the left pane to its prior contents, and the gold button next to it lets the user download the notebook to tweak and run themselves. Between the original comment and the buttons is a summary of the analysis, bordered to imply it is an addendum to the former. Tompkinsia is in action at GitHub [3].

This within-participants study had three phases: onboarding, control, and a test phase. It was also run with physical and Zoom attendance. During onboarding, participants attending physically were first reminded that their screen would be recorded at a certain point in the study, and there was an optional opportunity to have Socratic conversations with one researcher around claim verification recorded. Zoom participants were told that their whole meeting would be recorded. After completing the pre-experiment survey, all participants were given a basic overview of claim verification with Tompkinsia. After onboarding, to prevent unintentional priming, participants would proceed either to the control phase and then the test phase (Group A), or to the test phase and then the control phase (Group B). Participants were assigned to experimental groups alternatingly, and each group had one expert. In the control phase, participants completed a survey that presented a claim, asked participants to note their feelings on the claim, and then build a plan to verify the claim. Lastly in the test phase, participants completed a survey beginning with a different claim, and asked participants to note their feelings about the claim. Participants were told that they would build a verification plan, but first they would browse Tompkinsia for 10 minutes to explore analyses of claims in that same commentary that may use methods helpful for their plan. After browsing, participants completed the last survey and final part of the study, where they would build a verification plan for the claim introduced to them at the start of that phase.

Participants completed 4 surveys hosted on Google Forms. Participant identity was anonymized. The onboarding survey captured important demographic and experience information. The control survey required participants to note how they felt about a commentary claim, then build a plan to verify the claim. The test phase separated the control survey questions into an observation survey before participants’ time with Tompkinsia, and a plan-building survey after that. This subsection reviews this study's surveys, but they can be found at [4].

The pre-study survey captured important demographic and experience information. We asked for participants’ gender and the number of years of experience they had in Ancient Greek. We binned experience by years into no experience, 1-2 years, 3-4 years, 5-6 years, and more than 6 years. Next, there were 2 yes or no questions regarding prior use of 1) a commentary and 2) a hypertext reading environment like Perseus Digital Library. Familiarity with any of these may facilitate participants’ support of our hypotheses, as the germane load of learning the Tompkinsia system may be reduced from its similarity to these translation aids. The survey ended with 4 questions rating familiarity on a 5-point Likert scale with Python, Jupyter notebooks, quantitative textual analysis methods, and working with text in Python to assess the frequency of a textual phenomenon. Unfamiliarity did not disqualify participants, as we wanted to know whether we could reinforce critical thinking in those unfamiliar with such practices.

The control survey showed a claim from Smith's commentary on the History, which participants were not expected to know. The claim reads, “σημανῶ: as 2. 45. 8 and freq. in Thuc.; not common elsewhere in Attic prose" [33]. Participants were asked to guess what the reasoning behind the claim was, and then rated their acceptance of the claim, acceptance of its reasoning, and extent of understanding of the claim's reasoning all on a 5-point Likert scale. We wanted to measure both changes in claim perception and engagement. The rest of the survey guided participants through building a plan to verify the claim with 3 questions. A challenge, recognized here and in [6], is balancing both sufficiently preparing humanist students for engaging in the requisite math and logic, and ensuring that students feel empowered to engage with this content. We wanted this phase to assess student ability to apply what they learned in Tompkinsia, but we also wanted them to feel that overall language learner status or math and logic experience was not an emotional or intellectual blocker for building a plan. Rather than having 2 short answers where we asked what would be verified to assess the claim and what the verification plan was, we chose to include an intermediate question where participants could pick from 11 methods to use in their verification plan, including an “Other" option. Participants then explained how the methods they chose would fit into their plan. This yielded a frame of reference for folks conscious about their level of knowledge or their level of confidence in more mathematical or logical techniques. We still wanted to assess participants’ level of understanding with method choice, so 5 of our 10 defined methods were steps in an original plan to verify the claim and the other steps were nearly correct methods for claim investigation [4]. Lastly, participants could discuss any of these short answers with the researcher, who would co-build the plan with the participant in a Socratic way. The researcher would share knowledge to help the participant understand the goal and any methods they came across so far, but would deflect attempts at being used to devise the plan by asking the participant questions meant to guide them towards building the plan. Such conversations were recorded with informed consent.

The surveys in the Tompkinsia test phase are identical to the control phase, except that the first one shows the claim with perception and engagement questions, and the second one reprises the claim before asking the 3 questions about building a verification plan from the control phase. In this phase, the claim discussed is from the Basel Commentary and is present in Tompkinsia. It reads, “ἄξια: like other adjs. of evaluative content, used almost exclusively in character speech (exception: 23.885); see GRAZIOSI/HAUBOLD ad loc" [7]. The first survey ended with a summary of their task while browsing, and participants were verbally reminded of this task after survey submission. The second survey began with a reminder of the claim. The list of methods question in this plan-building section had only 6 non-other members, as the original plan for verification only had 3 steps [4]. Each claim is in this study is slightly different, but it participants could present a sound verification plan with methods from another analysis.

All participants had translated using a hypertext reading environment. The ratio of participants who had used a physical or digital commentary to those that did not is 3:1. 1 expert had more than 7 years of experience, and the ratio of participants in their first year of ancient Greek to those with 1-2 years of experience was also 3:1. Regarding Python experience, both experts reported peak familiarity (5), 1 participant responded with medium familiarity (3), 1 participant responded with some familiarity (2), and 8 participants reported total unfamiliarity (1). Likewise, regarding Jupyter experience, both experts reported total familiarity (5), 1 participant responded with some familiarity (2), and 9 participants reported total unfamiliarity (1). For both quantitative textual analysis and specific use case experience questions, both experts reported total familiarity (5), 2 participants responded with some familiarity (2), and 8 participants reported total unfamiliarity (1).

Two histograms depict both average remapped Likert scale scores rating their likelihood to accept the claim, their likelihood to accept the reasoning behind the claim, and their self-reported ability to follow said claim, with the left histogram representing Group A and the right histogram representing Group B. Each histogram consists of three translucent two-bar clusters, with a thinner black 95% confidence interval error bar accompanying each translucent bar representing an average score for the same distribution. In each cluster, the average bar on the left is azure, representing the average for a metric during the control phase, and the average bar on the right is orange, representing the average for a metric during the Tompkinsia-full phase. A legend reports these colors on the right side of the figure. In each histogram, the left cluster depicts Likert scale score distribution for rating their likelihood to accept the claim, labeled on the x-axis as Acceptance, the middle cluster depicts Likert scale score distribution for rating their likelihood to accept the reasoning behind the claim, labeled on the x-axis as Reasoning, and the right cluster depicts Likert scale score distribution for rating their self-reported ability to follow said claim, labeled on the x-axis as Following. What follows is a list of values reporting these average scores as well as the upper and lower bounds of each 95% confidence interval. Let's first look at the scores for Group A. Both average scores across phases for the acceptance metric are 1.6667, and they have matching 95% confidence ranges of 1 to 3. Looking at the reasoning metric, the control phase reports an average score of 3.2222 with a 95% confidence range of 1.8889 to 4.5556, whereas the Tompkinsia-full phase reports an average score of 3.6667 with a 95% confidence range of 2.0667 to 5. As for the following metric, the control phase reports an average score of 2.6667 with a 95% confidence range of 1.3333 to 4, whereas the Tompkinsia-full phase reports an average score of 3.6667 with a 95% confidence range of 2.3333 to 5. The scores for Group B are as follows. Looking at the acceptance metric, the control phase reports an average score of 3.1111 with a 95% confidence range of 1.6667 to 4.3361, whereas the Tompkinsia-full phase reports an average score of 4.4444 with a 95% confidence range of 3.7778 to 5. Turning to the reasoning metric, the control phase reports an average score of 2.2222 with a 95% confidence range of 1.2222 to 3.3333, whereas the Tompkinsia-full phase reports an average score of 2.5556 with a 95% confidence range of 1.5556 to 3.6667. As for the following metric, the control phase reports an average score of 3.3333 with a 95% confidence range of 2 to 4.6667, whereas the Tompkinsia-full phase reports an average score of 1.6667 with a 95% confidence range of 1 to 2.3333.

For each participant, we took their initial range on all perception and engagement Likert scales and remapped all values onto the original range of 1-5. Fig. 3 shows the averages of these scores split by group, with thin black 95% confidence error bars showing the relevant population. There are not significant insights for rating acceptance of the reasoning behind the claim, but there are some changes in group response otherwise. Most notably, the average score for participant acceptance of a claim dropped for Group B between phases. There is also a notable rise in score for participants’ extent of being able to understand and follow the claim's reasoning in Group B.

This thematic analysis was conducted with three specific subtopics in mind. To this end, we found our themes fit within one or more of these subtopics. This figure depicts a tri-circular sky blue Venn diagram on a white background where each circle is described by a subtopic in a sky blue and themes are in a navy blue, inhabiting at least one circle of this diagram. All circles overlap with each other, with one circle positioned in the center of the top of the figure, associated with identifying reasoning, a second circle positioned in the lower left, associated with identifying what to verify, and the remaining circle positioned in the lower right of the figure, associated with building a verification plan. This last circle is the only one with themes in it that do not intersect with other circles, and they are Hit Rate and Selection Size. In the overlap only between that last circle and identifying what to verify, the themes present are Filtering Method, Math Level, Search Type, and Number of Steps. Looking at the overlap only betwixt identifying what to verify and identifying reasoning, the only theme present in this space is Uncertainty. Last but not least, all circles share two themes: Phase and Analysis Method Type.

For all short answer questions across non-onboarding phases, 1 author ran a thematic analysis process [8] on each question. In each set, each answer was treated as a data extract and coded with at least 2 codes, which were grouped and iteratively reworked into coherent themes. During thematic analysis, it was found that codes, subthemes, and entire themes were applicable to more than 1 question. Fig. 4 shows these themes’ overlap. Regarding the last short answer, a theme set was developed to track both the complexity of methods chosen and how many different methods were in a plan.

While extracts were labeled by the individual and group IDs of a participant, one universal theme was the phase the extract was written during, noted with the codes control or test across all questions. Another universal theme was types of analysis methods in extracts. Word sense and spot checking were codes associated more with the Tompkinsia-full claim, with word sense often used to describe the meaning of ἄξια (“I would then narrow my search to whether those other uses of the word were in character speech") and spot checking describing reviewing the context of each instance of a phenomenon (“I would want to then look at all occurrences of the word in this subset."). Frequency recognition represented musings on a phenomenon's frequency (“I could then try to find the frequency of whether the word is almost exclusively in direct speech"), and learned method application coded examples of a participant applying methods seen in the onboarding or test phases. This theme has one more code: comparing frequencies (“Getting relative frequencies in and out of direct speech"). Codes in the theme of uncertainty varied from neutral, with guessy (“I would need to search this word…perhaps filtering by part of speech.") to doubting the self (“I am unsure") to not guessing with the I don't know code (“not enough information"). For the verification plan short answers, themes measuring answer quality included number of steps, search type, filtering method, and level of math used. For number of steps, answers to these questions were coded as a one step or multi step approach. Search type had two codes: simple search, for answers only stating what participants would search, and initial search, for answers where search was the first but not final step. It made sense to filter results by different criteria for each claim, so it follows that many control answers were coded filter by author whereas many test answers were coded filter by work. Level of math recognized advanced methods in verification answers (“This would allow me to set up a non-paramteric t-test") though some participants had less of this background, and those answers were coded as proto-math (“One would likely need to run calculation to assess whether the word itself is rare"). 2 themes were exclusive to one verification plan question: how many methods participants chose, and how many of those methods were correct, or hit rate. Selection size had 3 codes: one match covered those that chose only 1 method, hit and a miss covered those that chose 2 methods but only mentioned 1 in their final plan, and multi match covered those that chose more than 2 methods. Hit rate codes included full match, for participants that mentioned each method, unmentioned method, for participants that mentioned all but 1 chosen method in their answer, and unmentioned methods, for participants whose answer did not mention 2 or more chosen methods.

Members of each theme and code were examined to find how Tompkinsia affected critical thinking skills. In the question about what would be verified, half of Group A's answers were coded with simple search and one step, but most Group B members had their answers coded with multi step. 4 Group B members also had their answers coded with filter by work, as did 2 of 6 Group A members. In the verification plan question, defining codes of Group A's answers include initial search (5 in Group A, 3 in Group B), filtering by work (4 in Group A, 2 in Group B), and word sense (6 in Group A, 3 in Group B). 4 Group B members’ answers were coded with learned method application, and 1 Group A answer has the same code.

Table 1: Method Complexity Scores

This analysis can also tell us something about the complexity of methods chosen and how many methods the plan included. For each group, we collected data split by phase on the average number of codes per extract. In all phases Group A members had 8.5 codes on average. Yet in Group B, average code volume per extract began at 6.833 codes in the test phase and rose towards 8.6 codes per extract on average in the control phase. Likewise, we gave methods a complexity score as seen in Table 1 and for each extract we took the average complexity score across codes. We averaged those scores similarly to the prior metric, split by phase and group. Group A saw a rise in average complexity score from 1.717 in the control phase to 1.9317 in the test phase, and Group B's rise began at 1.533 in the test phase to 1.93 in the control phase.

2 authors coded screen recordings with a schema recording when Tompkinsia sessions started and ended, and when any Show/Hide Claim Analysis button was clicked. Note that 2 Group A members and 1 Group B member did not click on any of those buttons. On average, Group A members spent 1m 34s “scene setting" before clicking on a claim, and in Group B that time was 2m 30s. Group A members spent 1m 58s on average inside each claim, and in Group B this average time was 1 minute. Group B was also the only group where members would return to read the same claim, with the first claim having the highest average of clicks of claims at 1.16 clicks per claim, though the second claim was also revisited.