The Debate: Simple Prompts or Prompt Engineering?
There’s currently a debate about how helpful prompt engineering is—where you work hard to craft prompts to get the best results out of generative AI. Others think this is quickly becoming a non-issue for most users and we can talk to AIs—like ChatGPT, Claude 3 Opus, and Gemini Advanced—using “plain English.” I decided to test this.
The Project: Summarizing a 107-Page Pension File
When my niece was a freshman in college, I hired her to be my genealogy assistant. One of the projects I had her do was to summarize my great, great-grandfather’s pension file. The size? 107 pages! The result? 64 typed pages.
With such a large document, I chose to use Claude who has a bigger “context window” so can handle larger documents.
Experiment 1: Simple Prompt
My choice was to use a simple prompt: “Please summarize this pension file for Rheinhard (spelled various ways) Kaechle. Include any important details or dates.”
“Rhinort” and “Lana’s” marriage certificate which caused complications. From pension file.
Summary #1: Simple Prompt Results
Rheinhard Kaechle (name spelled various ways including Rheinhart, Reinhart, etc.) served as a Private in Company E of the 111th Ohio Infantry during the Civil War. He enlisted on August 4, 1862 and was honorably discharged on June 27, 1865.
He married Mary Magdalene Karbach (name also spelled various ways) on December 6, 1867 in Norwalk, Ohio.
Kaechle applied for an invalid pension on August 20, 1890 under the Act of June 27, 1890, claiming disabilities of rheumatism, catarrh, deafness and weak eyes. He was awarded a pension of $12 per month commencing August 20, 1890. This was later reduced to $8 per month from March 4, 1895.
Medical exams in the late 1890s also note heart disease and diabetes as disabilities. As of a September 20, 1899 exam, he was receiving $8 per month for rheumatism, catarrh, deafness and weak eyes.
Rheinhard Kaechle died on February 3, 1900 in Detroit, Michigan of diabetes. His widow Mary filed for a widow’s pension on February 12, 1900.
There were some discrepancies in the spelling of Mary’s maiden name between Karbach and Garabaugh on the marriage records, requiring affidavits in 1901 from friends and relatives to clarify her identity and marriage.
Mary was approved for a widow’s pension of $8 per month commencing February 12, 1900.
Using Word’s search function, I checked each fact. They were all correct! And I was happy with the results which gave me a great summary of the information found in this pension file.
Experiment 2: Engineered Prompt
But, deciding to test the difference between a simple prompt and an engineered prompt, today I gave Claude a prompt like what taught in Stephen Little’s first AI course with NGS.
Prompt: “You are an expert genealogist. Your goal is to summarize a pension file. Find attached a pension file. Write a narrative summary of the will which focuses on the dates and events mentioned in the pension file. The summary should be a dry recitation without editorialization. Include any meaningful information about specific events. The summary should be in a narrative form in an academic style (dry recitation without editorialization.)”
Reinhart’s death certificate. From pension file.
Summary #2: Prompt Engineering Results
Rheinhart Kaechle enlisted on August 4, 1862, and was honorably discharged on June 27, 1865. He married Mary Magdalene Karbach (also spelled Garabaugh due to clerical error) on December 6, 1867, in Norwalk, Ohio, by Justice of the Peace E. A. Pray.
On August 20, 1890, Rheinhart Kaechle filed a declaration for an invalid pension, citing disabilities including rheumatism, catarrh, deafness, and weak eyes. He was granted a pension of $12 per month under the Act of June 27, 1890, which was later reduced to $8 per month from March 4, 1895.
Rheinhart Kaechle passed away on February 3, 1900, in Detroit, Michigan. His widow, Mary M. Kaechle, filed a declaration for a widow’s pension on February 12, 1900. She provided evidence of their marriage and his death, as well as her financial situation. Mary was granted a widow’s pension of $8 per month, commencing February 12, 1900.
On June 20, 1912, Mary M. Kaechle remarried to John Sprenger in Detroit, Michigan. As a result, her widow’s pension was terminated. John Sprenger died on June 18, 1921, in Norwalk, Ohio.
Following the death of her second husband, Mary M. Sprenger filed a declaration for restoration of her widow’s pension on July 12, 1921. She provided evidence of her remarriage and the death of John Sprenger. Her pension was restored at a rate of $30 per month, commencing July 12, 1921, under the Act of May 1, 1920. This rate was later increased to $40 per month from June 4, 1928, under the Act of May 23, 1928.
Mary M. Sprenger passed away on December 16, 1938, as reported by her son, Charles Kaechle, in a letter dated December 17, 1938. Her pension payments were subsequently suspended.
Comparing the Summaries
Finally, I turned to ChatGPT and asked it to summarize the differences between the two in a bulleted list. Remember that Summary #1 used the simple prompt while Summary #2 used the engineered prompt. I added “winners” in brackets. Here’s ChatGPT’s analysis:
- Name Variations: Summary #1 lists various spellings of Rheinhard Kaechle’s name, while Summary #2 uses a single variation, “Rheinhart Kaechle.” [#1 winner]
- Marriage Official: Only Summary #2 mentions that Rheinhard Kaechle and Mary Magdalene Karbach were married by Justice of the Peace E. A. Pray. [#2 winner]
- Additional Disabilities: Summary #1 notes heart disease and diabetes as additional disabilities in the late 1890s, which is not mentioned in Summary #2. [#1 winner]
- Medical Exams: Summary #1 details a medical exam on September 20, 1899, and its outcome, not mentioned in Summary #2. [#1 winner]
- Maiden Name Discrepancy: Summary #1 discusses the discrepancies in Mary’s maiden name and the affidavits process in 1901, which Summary #2 does not. [#1 winner]
- John Sprenger’s Death Location: Summary #1 does not specify where John Sprenger died, while Summary #2 mentions he died in Norwalk, Ohio. [#2 winner]
- Pension Acts and Increases: Summary #2 provides specific Acts under which Mary’s pension was increased, while Summary #1 mentions the dates and amounts without specifying the Acts. [#2 winner]
- Death Reporting: Summary #1 states that Mary M. Sprenger’s death was reported by her son Charles Kaechle without a specific date, while Summary #2 mentions the reporting date as December 17, 1938. [#2 winner]
- Detail on Evidence and Declarations: Summary #2 provides more detail on the evidence and declarations provided by Mary for her pensions, which is not as detailed in Summary #1. [#2 winner]
I’m disappointed that the engineered prompt didn’t mention the various spellings of Rheinhart’s first name, heart disease or diabetes, or the discrepancies in Mary’s maiden name which was a huge issue in this case. I think the other points are minor. But, overall, I was happy with both summaries.
Your Experience with Prompts
How do you create prompts? Do you try to engineer them or use a more casual style? Have you ever tried to compare two results like this example? If so, I’d love to hear your results!
Unlock the Power of AI for Your Genealogy Journey
Are you ready to take your genealogy research to the next level? Join us for “AI Explorations in Genealogy & Beyond: A Course of Discovery & Application” and discover how AI can be your ultimate research partner. In this groundbreaking 8-week course, you’ll learn how to harness the power of AI to save time, boost accuracy, and uncover insights you never thought possible. Led by yours truly, your non-techie guide, you’ll explore a world of AI possibilities tailored specifically for genealogists like you.
Through live Zoom sessions, small group exploration, and a private Facebook group, you’ll have countless opportunities to ask questions, share your “aha!” moments, and collaborate with fellow learners. Whether you’re a seasoned genealogist or just starting out, this course will empower you with the skills and confidence to integrate AI into your research toolkit. You’ll not only enhance your genealogical work but also discover how AI can streamline your everyday tasks and open up a world of new possibilities.
Enroll now and embark on your AI adventure! Don’t miss this opportunity to be at the forefront of AI-powered genealogy. For just $425, you’ll gain access to 8 weeks of immersive learning, personalized guidance, and a supportive community of fellow explorers. Join us for “AI Explorations in Genealogy & Beyond” and let AI transform your research journey.
8 Comments
Leave your reply.