Labels, Over 7,600, Howard Barkin's OCR Method, Funny Typos, and New Poll

Important news:  I've added a new feature to the blog—labels—to make finding posts about particular subjects or people easier.  The labels are at the bottom of the column to your right.  If you want to read all the posts that include the label "XWord Info," for example, click on "XWord Info" in the labels field, and all those posts will appear.

I can hardly believe that we've now litzed more than 7,600 puzzles!  We're rapidly approaching our goal of the halfway point—8,113 litzed puzzles—by the end of February.  Awesome job, everybody!

Some of you have noticed the prodigious number of puzzles Howard Barkin has been litzing lately—nearly 400 since the beginning of January!  Howard has devised a very efficient way of litzing using optical character recognition (OCR) technology.  Back in September, I wrote an OCR post that described the methods of two other litzers—Joe Cabrera and Martin Herbach—who were using this technology to litz.  Now Howard has generously taken the time to present the details of his OCR method in response to some questions I asked him.  Read on:

You've litzed an amazing number of puzzles in a very short period of time using a combination of OCR and manual entry.  With this process, litzing a week of puzzles was taking you about 90 minutes when you started.  Has the process become even faster since then?

First, OCR = "optical character recognition," aka a program designed to analyze printed characters and convert to text, which can then be electronically edited.  (Whew.)

Now, with solving and litzing experience comes more efficiency.  Depending on the quality of the original files, a week of puzzles can be accurately scanned and litzed (converted electronically) in the range of 45–90 minutes.  That depends upon how readable the text is, how big the Sunday puzzle is, whether the baby wakes up, etc.

It's been fun to use my solving and technical experience to figure out the most efficient way to improve the process—remember, improvement is not about speed but about transferring these puzzles accurately.  That is the best way to honor the constructors and editors and ultimately complete the project.

You've mentioned using OCR to scan the PDF clue columns and Word macros to copy and paste individual clue text into Crossword Compiler.  Can you describe this process in detail in case other litzers want to give it a try?

I can try.  Again, this is about getting this project done efficiently and at the best possible quality.

Never sacrifice quality for speed.  Also, this is for PC-compatible use.  Mac users likely have similar apps available.  Use what works.

Have the following stuff (technical term . . .) available:

• A text editor with macro recording capability.
• Crossword Compiler or your favorite puzzle editor, of course.
• You'll need to have previously used your preferred OCR program to have scanned the clues of several PDF or graphic puzzle files to this point.  In each file, look carefully for the patterns in the scans that result in incorrect text.  Each scanner may result in different quirks.  This takes time and experience.  Be patient, grasshopper.

O.K.  Here we go:

1.  Find the common issues resulting from your OCR scans.  For the OCR app I use, each clue is often separated by extra blank lines, fill-in blanks may scan as "---", letter T as 'I' surrounded by single-quotes, etc.

2.  Once you see such patterns, record macros in your editor to find and replace each instance of the wrong scans with the right ones.  I then save these macros.

• Use the macro editor (in Word, this is the Visual Basic macro editor) to combine the macros into one master macro, and assign this macro to a keyboard shortcut key.

3.  Ensure that the patterns you are finding and replacing do not create "false positives"—so if you see that lowercase Ls are sometimes scanned as number 1s, don't simply replace all the lowercase Ls.  You'll create as many problems as you fix.  Care is the key word here in selecting which text to replace.  Wildcards can be helpful as well.

4.  To save extra time, create some additional macros for other frequent data-entry tasks.  I use other macros to enter fill-in blanks "___" where missing and to create end-of-clue ellipses in the New York Times style (. . . ").  It doesn't sound like much, but over the course of many puzzles it saves time (and your wrists!).

5.  After scanning all of your puzzle clues (this may take one or multiple scans, depending on the file), run your master cleanup macro on the file.

• At this point you have a cleaner text file with many of the common scan errors corrected.

6.  You still need to manually compare and edit each clue to match the actual clues.  Some manual tasks:

• Add those fill-in blanks, missing letters, words, ellipses, incorrect letters, etc.
• Style corrections:  missing spaces, ensuring that words that break in the middle to a new line with a dash (such as "dis- [new line] enchanted") are removed; the Times puzzles used dashes with impunity, so you have to make some judgment calls on these, but usually it is clear.
• I also use a handy list of Alt+<number> shortcut codes to type in special accented characters (é, à, etc).  In Windows, the "charmap" program lets you copy these as well.  This helps with those pesky foreign word and name entries.

7.  While editing, manually right-justify all clue numbers (or remove manually).  If you right-justify the numbers as shown below, you can then use the editor's block-select feature to highlight only the number columns on the left and delete them in one action:

   7
   8
   9
 10

8.  In the crossword editor, use the Import function and select the text file.  (CC8: Clue->Import.)

• If the file contains exactly one line per clue, all clues should import nicely into the clue review window.
• If not, an error message will display; then you must edit the text file again and compare to the clues.  Sometimes the initial scan will omit a clue, which you must add in manually.

9.  Review each clue one more time manually and edit as needed.  Sometimes a lowercase L will scan as the number 1, or zero as letter O, which is easy to miss.

10.  Save the file.  You're done!

This is actually a bit easier once you have a rhythm.  With an hour I have before sleeping, I can often finish a set of puzzles in this way, if the files are clean.  I have had to manually enter a full Sunday puzzle occasionally, though.  Not every file will scan cleanly.

Any questions on details here, feel free to ask.

Thanks so much, Howard—this is a very impressive system that has proved to be exceptionally accurate!  If anyone has questions or comments about Howard's OCR method, use the Comment feature below or send me an e-mail, which I'll forward along to Howard.

Speaking of finding mistakes, it's been a while since I've posted funny litzing typos, and my list is now several pages long!  So here are ten of the more humorous ones our proofreaders have caught lately:

• A clue for DIG was typed as "Verbal trust" instead of "Verbal thrust"
• A clue for ALFA should have been entered as "Esparto grass" rather than "Esperanto grass"
• A clue for OPA was accidentally typed as "W.W. III agency" instead of "W.W. II agency"
• A clue for TOE was supposed to read "Foot digit" instead of "Foot delight"
• A clue for ALAR should have read "Pteriod" but was accidentally entered as "Period"
• A clue for BIRDS OF A FEATHER, which was supposed to be "Group in a flock," was erroneously typed as "Group in a frock"
• A clue for ARNE was typed as "'Rue Britannia' composer" instead of "'Rule, Britannia' composer"
• The grid entry LET ONE'S HAIR DOWN was mistakenly entered as LET ONE'S TEAR DOWN
• A clue for ARE was accidentally entered as "Modem art" rather than "Modern art"
• Last, but certainly not least, a clue for FRED was supposed to read "An Allen who wrote 'Treadmill to Oblivion'" instead of "An Alien who wrote 'Treadmill to Oblivion'"

Today's featured Will Weng–edited puzzle was constructed by Shipp (probably Dorothea E.).  It was originally published on June 22, 1974, and was recently litzed by Mark Diehl.  The first thing that struck me about this puzzle was the shape of its grid, which reminded me of the crosshairs of a rifle.  I didn't think much of it until I noticed the entries SNIPERS and REVENGE in the grid!  I wonder if this was just a coincidence or if the shape/fill was intentional.  Will Weng mentioned somewhere that he published several puzzles constructed by prison inmates (shudder!). . . .  I've just posted a new poll, which asks you to vote on whether you think the puzzle has a rifle-related theme or not.  I'll post a recap next week!

Anyway, the rest of the fill is a mix of great entries and not-so-great ones.  HOBOKEN, LOZENGE, FANATIC, STAUNCH, and CADENCE are fantastic, but SCR (clued as "Movie scenario: Abbr."), API ("Nepal mountain"), and NPS ("Certifiers: Abbr.") are pretty bad.  Nevertheless, this is a remarkably clean "themeless" puzzle!  The solution (with highlighted "theme entries") can be seen below:

Today's featured pre-Shortzian entry, SESQUIDUPLICATE, originally appeared in the July 7, 1976, puzzle by Sidney Nelson, which was recently litzed by Nancy Kavanaugh.  Not surprisingly, according to the Ginsberg database, SESQUIDUPLICATE has yet to be reused in the Shortz era.  The clue for SESQUIDUPLICATE was "Having a ratio of 5 to 2."  This word is so obscure that it isn't even listed in Merriam-Webster's Online Dictionary!  Dictionary.com, however, notes that the word was listed in the 1996 and 1998 unabridged editions of Webster's.  The definition was "Twice and a half as great (as another thing); having the ratio of two and a half to one."  Sesquiduplicate sounds awesome, but I don't think I'll ever be able to use it in a sentence!  Below are some pictures of a sesquiduplicate ratio:

Images courtesy of Linux-Support.com and Wikipedia, respectively.

1986 Puzzles on XWord Info, Sending 1973 Puzzles, Over 7,400 at Halfway Point in Contest, Mark Diehl Hits 2,000, New Litzer of the Month Denny Baker, and David Astle's Call for Crossword Stories

Great news:  The 1986 puzzles are now up on XWord Info, and we've begun proofreading puzzles from 1985!  We're also now sending out puzzles from 1973 to be litzed—it won't be long before we're in the Margaret Farrar era!

We're also now at the halfway point in the litzing contest, and I'm delighted to report that we've litzed more than 7,400 puzzles!  At the beginning of January, we'd litzed 6,198 puzzles, and we're now at 7,441.  So in just one month, we've litzed 1,243 puzzles—a phenomenal achievement, surpassing even the first litzing contest in terms of number of puzzles litzed!  Our goal is the halfway point—8,113—by the end of February, so we only have 672 more to go in the next four weeks.  Awesome job, everybody!

Some of you have commented on what appears to be a neck-and-neck race between longtime litzing legend Mark Diehl and already-astounding Howard Barkin.  Mark put us over the 7,300 mark on Sunday and then over the 7,400 mark yesterday, and he has now litzed more than 2,000 puzzles—another all-time litzing record!  Congratulations, Mark!  And Howard, who just started litzing at the beginning of January, has already litzed more than 300 puzzles—that's approximately 10 per day.  Great job, Howard!  Here at Litzing Central, I never know what's going to come in or when:  Last night at 8:18, I received 23 litzed puzzles from Mark; just three minutes later, at 8:21, an e-mail with 19 litzed puzzles came in from Howard!  Then, a couple of hours later—at 10:06, to be precise—5 more puzzles arrived from Mark, putting Mark and Howard in a tie, with 308 litzed puzzles each during the contest!

The 616 puzzles from Mark and Howard comprise slightly less than half of the 1,243 that have been litzed so far this month, so all the other puzzles from everyone else are helping an enormous amount too!  Thanks so much, everybody—I think we'll be over the Halfway Hump in just a few weeks, if not sooner!

In other news, we have a new Litzer of the Month for February:  Denny Baker, who, as I've written before, is a constructor and was a litzer back in the day when Barry Haldiman was litzing his favorite puzzles!  To read more about Denny, click on his photo on the right or the Litzer of the Month tab above.

Also, I recently received an e-mail from Amy Reynaldo (Diary of a Crossword Fiend), who forwarded an e-mail she'd received from David Astle ("DA").  DA is putting together what sounds like a fascinating book that will celebrate the first 100 years of crossword puzzles.  The Great Clue Chase will contain 100 mini-chapters containing different stories linked to each of the 100 years.  DA is interested in "good stories, remarkable puzzles, flukes, backfires, Russian, Ninas, firsts and lasts" to fill out his timeline.  He comments that the "best stuff has a year attached—the further back the better often—as well as neat story."  He's not reproducing puzzles—just providing a clue or two and telling the story.  If you have any ideas or stories for this book—there are probably many about puzzles from the pre-Shortzian era!—contact DA through his web site.  Anyone who sends a "zinger" will receive a copy of DA's Puzzles and Words, a new book of word origins and connected word puzzles!

Today's featured pre-Shortzian puzzle, "Living It Up,"  was constructed by Frances Hansen.  It was originally published on December 29, 1974, and was recently litzed by Mark Diehl.  This is one of several Will Weng–era quote puzzles I've seen that doesn't work the same way as a modern quote puzzle.  Instead of dividing the quote by natural word breaks, the constructor broke the quote at every twenty-third letter; the quote simply continues from line to line in an acrosticlike fashion!  Nowadays this puzzle probably wouldn't fly because of the word-break issue, even though the constructor's original verse is very clever.  Interestingly, Maleska probably would have rejected this puzzle as well.  In all the many Maleska-edited quote puzzles I've looked at (some of which were constructed by Frances Hansen), I have yet to encounter a straight quote puzzle with this kind of word breakage (though I have seen several step quotes).

Back to the puzzle.  The ingenious verse reads:  A DEFIANT OLD MAID, PRAY FORGIVE HER, REMARKED WITH A HABIT OF A QUIVER: "TONIGHT I SHALL SMOKE AND DRINK TIL I CHOKE AND NUTS TO MY LUNGS AND MY LIVER."  It's amazing that the verse, in addition to being exactly 115 letters long, makes perfect sense (even with the word-break issue)!  Also, the fill, in addition to being pangrammatic, feels more polished and clean than the average Will Weng–edited Sunday.  I especially like the entries ROOF JOBS, RAFFISH, and TABASCO!  I can do without AANI (clued as "Dog-headed ape"), ADMIS ("African gazelles"), DHAVA ("East Indian gum tree"), NABO ("P. I. shrub: Var."), and TONDO ("Circular painting"); nevertheless, this is a remarkably small number of flat-out obscurities for a hand-constructed 23x.  Is anyone else surprised that Will Weng allowed the entry HOMICIDAL into the grid?  To me, this seems to violate the breakfast test. . . .

Even though it has its weaknesses, this crossword provides insight into what quote/verse puzzles looked like before the Maleska and Shortz eras.  I look forward to seeing how other types of themes evolve as we continue to litz through the Weng era and further back into the Farrar era!  This puzzle's answer grid (with highlighted theme entries) can be seen below:

Today's featured pre-Shortzian clue appeared in the November 28, 1950, puzzle by Thomas Meekin, which was edited by Margaret Farrar and recently litzed by yours truly.  The clue, which led to EXTRACURRICULAR, read: "Describing activities now lacking in N.Y.'s public schools."  Apparently in 1950 and 1951, New York teachers were dissatisfied with their salaries and decided to boycott extracurricular activities until their pay rates were increased!  I find this kind of historically significant clue fascinating—there's hardly any information about the boycott anymore.  Each pre-Shortzian puzzle (particularly those edited by Margaret Farrar) is a snapshot of major events occurring when it was published!  Below is a picture of children in a classroom from the 1949–50 school year:

Image courtesy of Artswipe.

We're in 1974, 7,200-Plus Litzed Puzzles, Contest Updates, and Missing PDFs

Just minutes after the last blog post was published, Howard Barkin sent 21 more puzzles, putting us over the 6,900 mark on the litzing thermometer and himself in first place in the litzing contest.  But not for long—by Sunday night, Mark Diehl had reclaimed first place and put us over the 7,000 mark!  (Coincidentally, just two days before, on January 18, Deb Amlen reported in Wordplay that the 7,000th daily puzzle edited by Will Shortz had just appeared!)  Since Sunday, though, we've blown 7,000 out of the water—the litzing thermometer now reads 7,203, and we're sending out packets from mid-1974.

As I write this, Howard Barkin is in first place with a total of 227 litzed puzzles since January 1!  Mark Diehl is at 212, but I have a feeling he'll regain his lead soon!  Jeffrey Krasnick continues to hold a solid third place, with a total of 121 litzed puzzles, and Denny Baker, with 77, isn't far behind.  The remaining litzers who've sent in puzzles in January have litzed 217 puzzles all told, so each and every puzzle definitely helps!  Thanks, everyone, for making this contest such a success so far!

In other news, Barry Haldiman recently volunteered to help track down missing and illegible puzzles.  Although ProQuest is supposed to have everything, it doesn't, and sometimes what it does have is impossible to read.  Barry has done an outstanding job of tracking down 21 puzzles and/or solutions so far on the microfilm at his library (which is a lot better than the microfilm here!) and will be working on finding the remaining problematic puzzles.  Below is a picture of an almost completely illegible solution from ProQuest—there's no way anyone could litz this puzzle's solution without solving it (or possibly hunting it down in a book, though the puzzles in books aren't always exactly the same as the originals):

Thanks so much for helping with this, Barry!

Today's featured puzzle is another Jordan S. Lasher masterpiece.  This Maleska-edited puzzle was originally published on April 1, 1977, and was recently litzed by Barry Haldiman.  The theme entries UPPER LEFT TO/BOTTOM RIGHT and READ THE DIAGONAL are clued as "Diagonal course" and "How to get this puzzle's absurd message," respectively.  The letters on the diagonal starting at the first square read WE FOOLED YOU HA HA.  I find it amazing how the diagonal message perfectly intersects READ THE DIAGONAL—it must have taken hours to get the wordings of both phrases perfect so that they could cross like this!  Even with both horizontal and diagonal constraints, the constructor still managed to include some lovely entries in the nonthematic fill, such as MAN TRAP, CURE-ALL, and NONSTOP.  I have yet to see an IYNX ("Wry-necked woodpecker") on its GENU ("Knee: Lat.") playing STH. ("Bridge seat: Abbr.") on the ALPH ("Xanadu's river"), but nevertheless, this is a phenomenal pre-Shortzian puzzle!  The answer grid, with highlighted theme entries, can be seen below:

Today's featured pre-Shortzian entry is FUZZY-GUZZY.  FUZZY-GUZZY originally appeared in the September 26, 1976, Will Weng–edited crossword by . . . you guessed it, Jordan S. Lasher!  According to the Ginsberg database, this bizarre entry has never been reused in a Shortz-era puzzle.  I'd be surprised if FUZZY-GUZZY showed up in any other pre-Shortzian puzzles because of its challenging letter pattern and relative obscurity (FUZZY WUZZY, just one letter different, is much more well-known)!  FUZZY-GUZZY was clued as "Balsamweed"; Webster defines fuzzy-guzzy as "a balsamweed (Gnaphalium Obtusifolium) with glandular villous stem."  Below are two pictures of fuzzy-guzzy:

Images courtesy of StillwaterWoods.net

Contest Standings, Almost at 6,900, More Publicity, Politically Incorrect Terms

We're making terrific progress in the second litzing contest!  Currently Mark Diehl is leading the pack with a total of 130 litzed puzzles, Howard Barkin is hot on his heels with 123 litzed puzzles, and Jeffrey Krasnick holds a solid third place with 87 litzed puzzles.  On Thursday, Jeffrey put us over 6,800 on the litzing thermometer, and we're now almost at 6,900—well on our way to the halfway-point goal of 8,113 litzed puzzles by the end of February.  Every litzed puzzle helps, so thanks and great job, everybody!

The Pre-Shortzian Puzzle Project received some more publicity recently, first in Tyler Hinman's article "Touch of Genius:  Puzzazz Brings Puzzles to Your Touchscreens" in Wired, in which he interviews Puzzazz founder Roy Leban, who mentions the Pre-Shortzian Puzzle Project.  Then, in Amy Reynaldo's Diary of a Crossword Fiend, the project was lauded by T Campbell for The Honorary Orcas award for best work in crossword scholarship.  Thanks, Roy and T!

Recently I've received numerous comments from litzers about politically incorrect terms that found their way into Weng-edited crosswords.  One litzer noted a puzzle that contained the entries MORON (clued as "One of low I.Q."), MANIC ("___-depressive"), and HOLO ("Prefix for caust")!  In the same batch, the litzer found a puzzle with its entire theme based on negativism, as well as a puzzle that contained the entry COOLIES.  Another litzer commented that he'd litzed Weng-edited puzzles with references to the Nazis, the KKK, and even fill-in-the-blank clues for racial slurs.  I took a look through Maleska's clues on XWord Info for potentially offensive terms and found that he bent over backwards to avoid cluing them like Weng did—he only clued MORON in reference to an Andalusian city, HOLO as the combining form for complete, and MANIC as a synonym for frenzied.  It should be interesting to see how Margaret Farrar approached clues for such entries—I'm a little leery, though, since I've already seen that she referred to the Japanese as Japs.  But that seems to have been common back then; in general, the clues and entries seem to have become more politically correct over time, no doubt reflecting changes in our society as a whole.

Today's featured pre-Shortzian puzzle was constructed by Jordan S. Lasher.  It was originally published on November 29, 1975, and was recently litzed by Todd Gross.  On the surface, this puzzle looks like a pretty typical Lasher opus—the grid is very open, the theme is concise, and the fill is better than average.  When I looked more closely, however, I realized that there was much more going on in this one—in addition to the theme entries WEIGHTLIFTER, PAUL ANDERSON (who, as Todd pointed out, once lifted more than three tons on his back!), and BARBELL, the blocks in the center of the grid are shaped like a weight!  This ingenious construction is a surprisingly early example of grid art in a standard 15 x 15!  The complexity of this puzzle's theme blows me away—Lasher seems to have been decades ahead of his time.  As I mentioned above, the nonthematic fill is admirable—I especially like the entries CHASSIS, HAIRCUT, RUN AMOK, and GATLING (a type of gun one of my teachers described to our class in lurid detail).  All in all, this is a groundbreaking pre-Shortzian puzzle—I hope to see many more Weng-edited crosswords that are this innovative!  The answer grid (with highlighted theme entries) can be seen below:

I was recently looking through some Margaret Farrar–edited puzzles from the 1950s and came across the historically significant clue "Cause of chaos in the entertainment world" in the December 9, 1950, puzzle.  The answer?  COLOR TELEVISION!  Wikipedia notes that back when color television was first being developed in 1950, there were many technical glitches and inaccurate color reproductions; also, at one point, just one hour of color television was shown per day!  On top of all this, the prototype color receivers were only available in the New York area.  I can definitely see why color television might have caused a lot of chaos back then.  Nowadays we'd laugh at a clue like this!  Below is a picture of an early color television set:

Image courtesy of CT-TV Vintage Television.