Over 10,600, OCR with Martin Herbach, Puzzazz Solving App, and Funny Typos

Great news—we've now litzed more than 10,600 puzzles!  Here's a recap of the week:  On Friday afternoon, Nancy Kavanaugh started us off by sending in 14 puzzles; that evening, Howard Barkin sent in another 14, putting us over 10,500!  The grand finale for the day was Mark Diehl sending in 14 more puzzles.  Then on Saturday, Denny Baker sent in 7 puzzles, which were followed by 7 more from Joe Cabrera.  Sunday morning, Joe sent in another 5, and that night, Mark sent in 35 more puzzles!  Early Monday morning, Jeffrey Krasnick sent in 7 puzzles, which were followed by 10 proofread puzzles from Todd Gross.  Monday evening, Howard sent in 7 more puzzles, and Tuesday night, Mark sent in another 21 puzzles, putting us over 10,600!  Then Thursday evening, Mark sent in another 5 puzzles—and a short while later, 5 more!  Awesome job, everyone!

As I've mentioned, I'm currently up at Stanford taking an artificial intelligence course.  Before it started, I had the pleasure of visiting with litzer Martin Herbach in Saratoga, which is about half an hour south of Stanford.  He and his wife, Elaine, are terrific cooks and made a delicious salmon meal, after which Martin demonstrated how he litzes puzzles using OCR technology.  Although I'd read his description of the process before, actually seeing it in action was fascinating!  The script that tidies up the file was particularly cool; I also hadn't realized that he litzes the grids manually.  Here are a couple of photos:

Martin Herbach demonstrating OCR litzing 

Clues litzed using OCR

Last week litzer/proofreader Todd Gross and I published a puzzle in The New York Times that many solvers had difficulty with, in part because of glitches in their solving apps.  [Spoiler alert for the 7/18/13 puzzle if you haven't already solved it.]  After the puzzle appeared, I received an e-mail from Roy Leban, who informed me that the Puzzazz app had supported this unusual puzzle properly:  The 23 was in the right place, the apostrophe worked correctly, and the diamond was highlighted in gray.  In addition, clicking on 23-Across highlighted the entire diamond and allowed solvers to write or type the whole title in order.  This software is a remarkable achievement and might just be the solving app of choice, not only for today's puzzles but for some of the trickier pre-Shortzian ones as well!  Here's a photo of what this puzzle looked like on an iPad:

Photo courtesy of Puzzazz

Thanks again for letting me know about this, Roy!

In other news, the 1981 proofreading has continued to progress—with just a few more months to go, our indefatigable proofreaders are working hard to finish up the year.  The funny typos list mushroomed when proofreaders began looking over puzzles from the first litzing contest!  In one of these contest puzzles, Todd Gross found one of the funniest typos of all time.  Here are some typical clue typos, followed by some "variety typos," followed by Todd's pièce de résistance:

• Entry:  TRACER
• Wrong:  Kind of ballet
• Right:  Kind of bullet
• Entry:  ASSET
• Wrong:  Brians or beauty
• Right:  Brains or beauty (though I'm sure litzer Brian Tyler would prefer the first one!)
• Entry:  NAIVE
• Wrong:  Ingenious
• Right:  Ingenuous
• Entry:  HOMER
• Wrong:  Slugger's request
• Right:  Slugger's quest
• Entry:  EGO
• Wrong:  Kind of trap
• Right:  Kind of trip
• Double typo:  Entry:  ELS
• Wrong:  Chi. loop loppers
• Right:  Chi. Loop loopers
• OCR mistake caught by Joe Cabrera:  Entry:  OLAN
• Wrong:  Pearl suck heroine
• Right:  Pearl Buck heroine
• Copyright Field
• Wrong:  © 1971, The New York Times.  Editor: Will Went.
• Right:  © 1971, The New York Times.  Editor: Will Weng.
• Copyright Field
• Wrong:  © 1969, The New York Times.  Editor: Margaret Farrah.
• Right:  © 1969, The New York Times.  Editor: Margaret Farrar.
• Ultimate typo:  Entry: DAMAGED
• Wrong:  Married
• Right:  Marred

Great typos—LOL!

Over 7,900, Man vs. Machine, in 1972, Daily-Puzzle Authors, Poll Results, and New Feature on Pre-Shortzian Constructors

We've now litzed more than 7,900 litzed puzzles—just a couple hundred more and we'll be at the halfway point!  On Sunday litzer Mark Diehl sent in 15 puzzles, which put us over 7,700, and then on Tuesday, Howard Barkin sent in 28 puzzles, putting us over 7,800, followed by another 21 puzzles this morning, which put us over 7,900!  And all the other packets from everyone have really helped push us along—thanks again, everybody!

Mark and Howard are still taking turns in the litzing contest's first-place position—as I write this, they're tied for the lead!  In a way, this has become a sort of battle between man (manual litzing) versus machine (OCR), not unlike the competition between human solvers and Dr. Fill at the ACPT!  Whatever the final result, it's all good for the project—Mark and Howard have each litzed more than 400 puzzles in the past 6 weeks!

I'm now sending out puzzles from 1972 to be litzed, so it won't be long before we're in the 1960s!  A couple of litzers have been saddened by the lack of authors for the daily puzzles now.  Although I'm hopeful that the authors of most puzzles will ultimately be identified, many will likely never have bylines.  The good news is that daily-puzzle authors will start reappearing in 1967, during the Margaret Farrar era.

The results of last week's poll are in!  The poll asked whether you thought last week's featured puzzle contained a rifle-related theme.  Most readers (55%) voted no, but the remaining respondents were evenly split between yes and unsure.  If you felt certain one way or the other (yes or no), I'd be interested in hearing why.  Please comment below or e-mail me.

I'm starting a new feature on pre-Shortzian constructors, which you'll find by clicking on the Pre-Shortzian Constructors tab above.  Many constructors had profiles written about them or were mentioned in articles in The New York Times and elsewhere, and I've decided to try to track down those publications and link to them from this site.  I'm starting with a few pre-Shortzian constructors whose last names began with L:  Jordan S. Lasher, William Lutwiniak, and Jack Luzzatto; I'll gradually add other constructors as I continue to find articles about them.  If you happen to know of additional links for a particular constructor, please let me know.

Today's featured pre-Shortzian puzzle was constructed by Evelyn B. Rosenthal.  In addition to being a prolific crossword constructor in the 1970s, Evelyn was also the high school math teacher of Crosswords LA director Elissa Grossman's father.  It's amazing how small the crossword puzzle world is!

The puzzle was originally published on November, 14, 1975, and was recently litzed by Alex Vratsanos.  This lovely Will Weng–edited construction features eleven theme entries for which the first part of the entry rhymes with the second (and, as Jim Horne just pointed out, the two parts—and, in one case, three—are also pronounced the same!).  The theme density is amazing—getting three theme entries to stack in each corner and the five other theme entries to interlock so elegantly must have been extremely challenging!  My favorite theme entries are DEAR DEER (clued as "Bambi, e.g."), ROAM ROME ("See the Eternal City"), and PLAIN PLANE ("Ordinary carpenter's tool").  EERY ERIE ("Weird lake") and TWO TO TWO ("1:58") feel weaker, but they're certainly better than nonthematic entries.  The nonthematic fill is a real mix on this one—I love the Scrabbly entries RAJPUT, ZWEI, PONZI, and DAFFY, but RYE OLD ("___-fashioned [cocktail]"), T MOORE ("Signature of an Irish poet"), and ENTEMPLE ("Set up a shrine") certainly aren't the best.  Warts and all, this is an outstanding pre-Shortzian puzzle and a stellar example of pre-Shortzian interlock!  The answer grid, with highlighted theme entries, can be seen below:

As we've continued to work through the Will Weng era, I've been litzing some of the Margaret Farrar–edited Sunday puzzles from 1942.  I find all the clues related to World War II (and that time period in general) fascinating, since they show how the United States felt about certain historical figures and also chronicle specific weeks in history.  The March 1, 1942, crossword alone, which was constructed by Charles Cross (an alias), contained at least eight topical clues:

• "German field marshal."  (LIST)
• "He thinks he can do business with Hitler."  (APPEASER)
• "Our ally in Russia."  (WINTER)
• "What the Axis is to U.S."  (THREAT)
• "Kind of battles we used to have."  (SHAM)
• "German battleship."  (GNEISENAU)
• "Warning lacking at Pearl Harbor."  (ALERT)
• "Hitler's 'merest utensil.'"  (DUCE)

The clues for APPEASER and WINTER are particularly amusing—even though our country took World War II very seriously, there was apparently still room for irony and humor in the press.  The clue for DUCE isn't particularly funny, but it's certainly historically significant in that it shows how Hitler felt about Mussolini.  I look forward to seeing what other war-related clues I come across as I continue litzing these puzzles!  In the meantime, below is a picture of the GNEISENAU:

Image courtesy of Wikipedia.

Labels, Over 7,600, Howard Barkin's OCR Method, Funny Typos, and New Poll

Important news:  I've added a new feature to the blog—labels—to make finding posts about particular subjects or people easier.  The labels are at the bottom of the column to your right.  If you want to read all the posts that include the label "XWord Info," for example, click on "XWord Info" in the labels field, and all those posts will appear.

I can hardly believe that we've now litzed more than 7,600 puzzles!  We're rapidly approaching our goal of the halfway point—8,113 litzed puzzles—by the end of February.  Awesome job, everybody!

Some of you have noticed the prodigious number of puzzles Howard Barkin has been litzing lately—nearly 400 since the beginning of January!  Howard has devised a very efficient way of litzing using optical character recognition (OCR) technology.  Back in September, I wrote an OCR post that described the methods of two other litzers—Joe Cabrera and Martin Herbach—who were using this technology to litz.  Now Howard has generously taken the time to present the details of his OCR method in response to some questions I asked him.  Read on:

You've litzed an amazing number of puzzles in a very short period of time using a combination of OCR and manual entry.  With this process, litzing a week of puzzles was taking you about 90 minutes when you started.  Has the process become even faster since then?

First, OCR = "optical character recognition," aka a program designed to analyze printed characters and convert to text, which can then be electronically edited.  (Whew.)

Now, with solving and litzing experience comes more efficiency.  Depending on the quality of the original files, a week of puzzles can be accurately scanned and litzed (converted electronically) in the range of 45–90 minutes.  That depends upon how readable the text is, how big the Sunday puzzle is, whether the baby wakes up, etc.

It's been fun to use my solving and technical experience to figure out the most efficient way to improve the process—remember, improvement is not about speed but about transferring these puzzles accurately.  That is the best way to honor the constructors and editors and ultimately complete the project.

You've mentioned using OCR to scan the PDF clue columns and Word macros to copy and paste individual clue text into Crossword Compiler.  Can you describe this process in detail in case other litzers want to give it a try?

I can try.  Again, this is about getting this project done efficiently and at the best possible quality.

Never sacrifice quality for speed.  Also, this is for PC-compatible use.  Mac users likely have similar apps available.  Use what works.

Have the following stuff (technical term . . .) available:

• A text editor with macro recording capability.
• Crossword Compiler or your favorite puzzle editor, of course.
• You'll need to have previously used your preferred OCR program to have scanned the clues of several PDF or graphic puzzle files to this point.  In each file, look carefully for the patterns in the scans that result in incorrect text.  Each scanner may result in different quirks.  This takes time and experience.  Be patient, grasshopper.

O.K.  Here we go:

1.  Find the common issues resulting from your OCR scans.  For the OCR app I use, each clue is often separated by extra blank lines, fill-in blanks may scan as "---", letter T as 'I' surrounded by single-quotes, etc.

2.  Once you see such patterns, record macros in your editor to find and replace each instance of the wrong scans with the right ones.  I then save these macros.

• Use the macro editor (in Word, this is the Visual Basic macro editor) to combine the macros into one master macro, and assign this macro to a keyboard shortcut key.

3.  Ensure that the patterns you are finding and replacing do not create "false positives"—so if you see that lowercase Ls are sometimes scanned as number 1s, don't simply replace all the lowercase Ls.  You'll create as many problems as you fix.  Care is the key word here in selecting which text to replace.  Wildcards can be helpful as well.

4.  To save extra time, create some additional macros for other frequent data-entry tasks.  I use other macros to enter fill-in blanks "___" where missing and to create end-of-clue ellipses in the New York Times style (. . . ").  It doesn't sound like much, but over the course of many puzzles it saves time (and your wrists!).

5.  After scanning all of your puzzle clues (this may take one or multiple scans, depending on the file), run your master cleanup macro on the file.

• At this point you have a cleaner text file with many of the common scan errors corrected.

6.  You still need to manually compare and edit each clue to match the actual clues.  Some manual tasks:

• Add those fill-in blanks, missing letters, words, ellipses, incorrect letters, etc.
• Style corrections:  missing spaces, ensuring that words that break in the middle to a new line with a dash (such as "dis- [new line] enchanted") are removed; the Times puzzles used dashes with impunity, so you have to make some judgment calls on these, but usually it is clear.
• I also use a handy list of Alt+<number> shortcut codes to type in special accented characters (é, à, etc).  In Windows, the "charmap" program lets you copy these as well.  This helps with those pesky foreign word and name entries.

7.  While editing, manually right-justify all clue numbers (or remove manually).  If you right-justify the numbers as shown below, you can then use the editor's block-select feature to highlight only the number columns on the left and delete them in one action:

   7
   8
   9
 10

8.  In the crossword editor, use the Import function and select the text file.  (CC8: Clue->Import.)

• If the file contains exactly one line per clue, all clues should import nicely into the clue review window.
• If not, an error message will display; then you must edit the text file again and compare to the clues.  Sometimes the initial scan will omit a clue, which you must add in manually.

9.  Review each clue one more time manually and edit as needed.  Sometimes a lowercase L will scan as the number 1, or zero as letter O, which is easy to miss.

10.  Save the file.  You're done!

This is actually a bit easier once you have a rhythm.  With an hour I have before sleeping, I can often finish a set of puzzles in this way, if the files are clean.  I have had to manually enter a full Sunday puzzle occasionally, though.  Not every file will scan cleanly.

Any questions on details here, feel free to ask.

Thanks so much, Howard—this is a very impressive system that has proved to be exceptionally accurate!  If anyone has questions or comments about Howard's OCR method, use the Comment feature below or send me an e-mail, which I'll forward along to Howard.

Speaking of finding mistakes, it's been a while since I've posted funny litzing typos, and my list is now several pages long!  So here are ten of the more humorous ones our proofreaders have caught lately:

• A clue for DIG was typed as "Verbal trust" instead of "Verbal thrust"
• A clue for ALFA should have been entered as "Esparto grass" rather than "Esperanto grass"
• A clue for OPA was accidentally typed as "W.W. III agency" instead of "W.W. II agency"
• A clue for TOE was supposed to read "Foot digit" instead of "Foot delight"
• A clue for ALAR should have read "Pteriod" but was accidentally entered as "Period"
• A clue for BIRDS OF A FEATHER, which was supposed to be "Group in a flock," was erroneously typed as "Group in a frock"
• A clue for ARNE was typed as "'Rue Britannia' composer" instead of "'Rule, Britannia' composer"
• The grid entry LET ONE'S HAIR DOWN was mistakenly entered as LET ONE'S TEAR DOWN
• A clue for ARE was accidentally entered as "Modem art" rather than "Modern art"
• Last, but certainly not least, a clue for FRED was supposed to read "An Allen who wrote 'Treadmill to Oblivion'" instead of "An Alien who wrote 'Treadmill to Oblivion'"

Today's featured Will Weng–edited puzzle was constructed by Shipp (probably Dorothea E.).  It was originally published on June 22, 1974, and was recently litzed by Mark Diehl.  The first thing that struck me about this puzzle was the shape of its grid, which reminded me of the crosshairs of a rifle.  I didn't think much of it until I noticed the entries SNIPERS and REVENGE in the grid!  I wonder if this was just a coincidence or if the shape/fill was intentional.  Will Weng mentioned somewhere that he published several puzzles constructed by prison inmates (shudder!). . . .  I've just posted a new poll, which asks you to vote on whether you think the puzzle has a rifle-related theme or not.  I'll post a recap next week!

Anyway, the rest of the fill is a mix of great entries and not-so-great ones.  HOBOKEN, LOZENGE, FANATIC, STAUNCH, and CADENCE are fantastic, but SCR (clued as "Movie scenario: Abbr."), API ("Nepal mountain"), and NPS ("Certifiers: Abbr.") are pretty bad.  Nevertheless, this is a remarkably clean "themeless" puzzle!  The solution (with highlighted "theme entries") can be seen below:

Today's featured pre-Shortzian entry, SESQUIDUPLICATE, originally appeared in the July 7, 1976, puzzle by Sidney Nelson, which was recently litzed by Nancy Kavanaugh.  Not surprisingly, according to the Ginsberg database, SESQUIDUPLICATE has yet to be reused in the Shortz era.  The clue for SESQUIDUPLICATE was "Having a ratio of 5 to 2."  This word is so obscure that it isn't even listed in Merriam-Webster's Online Dictionary!  Dictionary.com, however, notes that the word was listed in the 1996 and 1998 unabridged editions of Webster's.  The definition was "Twice and a half as great (as another thing); having the ratio of two and a half to one."  Sesquiduplicate sounds awesome, but I don't think I'll ever be able to use it in a sentence!  Below are some pictures of a sesquiduplicate ratio:

Images courtesy of Linux-Support.com and Wikipedia, respectively.

More Than One-Fourth of the Puzzles Litzed, Poll, and Litzing Methods Using OCR

I am thrilled to announce that, as of this week, we reached and passed the 4,000 mark and have now finished more than a quarter of the entire project (4,056 puzzles)!  This is amazing—I never imagined we'd be this far along in the project so early on.  We're also rapidly approaching 1982, with just six more months of puzzles to send out from 1983!

In other news, I've added a poll feature to the website.  The first poll asks how long it takes to litz a daily puzzle.  Several responses have already come in; if you'd like to respond before the poll expires next week, you can find the poll under the litzing thermometer to your right.  I'm planning to post a new poll every week in between my longer regular blog posts.

Litzers especially may be interested in the following:  Joe Cabrera and Martin Herbach, two of our litzers, have come up with innovative methods for litzing puzzles with optical character recognition (OCR) software!  Though the final puzzles need to be proofread for obvious mistakes (such as "A Tunrier" instead of "A Turner," for TED), the OCR methods seem to be quite fast and accurate.  Joe reports that it takes about an hour and fifteen minutes for him to litz an entire batch of puzzles, including the Sunday puzzle; Martin estimates that he takes just forty-five minutes (over several sittings) to litz a batch with his method!  I've summarized both methods below, paraphrasing slightly to make them less technical:

Joe Cabrera's Method

1.  Create new Across Lite text file templates based on the dates and authors' names.

2.  Pull each PDF into Photoshop (assuming everything was originally scanned at 300 dots per inch).  Rearrange all the columns [answer grid and clues] into one long one, with the answer grid on top. Delete the lines in the answer grid to leave just the letters.

3.  Run each Photoshop file through the online OCR reader to turn it into plain text.

4.  Run a script to remove all the clue numbers and use them to separate the clues so they're not all one big paragraph.  Also use the script to clean up extra spaces and garbage, format underscores and ellipses properly, make all quotes "dumb," and so on.  Proofread the clues.

5.  Clean up and proofread the answers.

6.  Cut and paste the proofread clues and answers into the Across Lite text files created earlier.

7.  Drag the text files into Across Lite and save them as standard .PUZ files.

Martin Herbach's Method

1.  Use an online OCR reader to convert from PDF to rich text format (RTF).

2.  Copy and paste the clues from the RTF file into a Notepad document.

3.  Manually fix missing numbers, broken lines, accented characters, ellipses, and fill-in-the-blank clues.  Save as a text file.

4.  Read the text file into Microsoft Excel, specifying "Space Delimited" and "No Text Indicator."

5.  Delete Column A (which consists of the clue numbers).  Save the Excel file.

6.  Upload the Excel file to Google Docs and then download the Excel file from Google Docs as a text file (because Excel screws up quoted strings and commas by inserting extra quotes, which Google Docs doesn't).

7.  Read the text file (actually a .tsv file for tab-separated values) into Microsoft Word.

8.  Replace all tabs with spaces.  Select the entire Word document and copy it.

9.  Paste the contents of the Word document into an Across Lite text template.

The only minor flaw in these methods (other than that litzers must be very tech-savvy!) is that the OCR service Joe and Martin use only allows fifteen file conversions per hour.  This isn't a huge problem, though, since each batch only contains seven puzzles.  Another very interesting idea Joe tested was speaking the answer grid into his smartphone, though he found that doing this actually took more time overall.  Nevertheless, the ingenious methods both Joe and Martin have come up with to automate the litzing process are very, very cool!

Today's featured pre-Shortzian New York Times puzzle was constructed by I. Judah Koolyk.  It was originally published on October 8, 1983, and was recently litzed by Mark Diehl.  This amazing daily puzzle contains twelve rebuses of STAR!  My favorite theme entries are **** RATINGS (yes, four rebus squares in a row!) and *** GENERAL.  Even today, stacked rebuses with this many rebus squares are extremely challenging to construct.  Not surprisingly, the nonthematic fill (consisting of fewer entries than usual) has a handful of undesirable entries (ESNES, ITEAS, etc.).  All things considered, though, this is an admirable construction that feels way ahead of its time!  The answer grid (with highlighted theme entries) can be seen below:

Today's featured pre-Shortzian entry is NEBEL.  According the the Ginsberg database, NEBEL has never been reused in a Shortz-era puzzle.  NEBEL originally appeared in the June 2, 1985, puzzle by Joy L. Wouk, which was recently litzed by Stephen Edward Anderson.  The original clue for NEBEL was "Ancient stringed instrument."  Webster defines a nebel as a variant of nabla, which it lists as "an ancient stringed instrument, probably like a Hebrew harp of 10 or 12 strings."  Not surprisingly, nebel comes from nēbhel, the Hebrew word for "harp."  Below is one interpretation of what a nebel might have looked like:

Image courtesy of the Potsdam Public Museum.