In my last post I promised some updates, so here’s a short report on the first two points:
- Improving the geotagging, so that we can find ancient place-name mentions more accurately in the input texts.
- Improving the georesolution, so we can plot more of them on the map, and in approximately the right places!
The geotagging work was outlined in an earlier post so I’ll just recap very briefly and bring us up to date. Geotagging is an NER (named entity recognition) process, and we actually recognise and categorise other classes (such as PERSON) as well as PLACE, to help discriminate between them. In the early stages we were getting good recall (we were correctly spotting over 80% of the Hestia “gold standard” place-name mentions) but very poor precision – only 40% of what we said were place-names actually were. The NER is done partly on linguistic clues (part of speech etc) and partly using lexicons – look-up lists of instances of the classes to be recognised, such as people or places. Most of our poor precision was because personal names, like “Priam”, were being classified as places; there are several places called Priam in the world, and they were in our lexicons.
What was needed was a new set of lexicons, tailored for ancient places and people. The place-names lexicon was easy – just base it on Pleiades+ – and the people lexicon was built from material available at http://www.eisenbraums.com, augmented by extracting marked-up personal names from the Hestia data. Except for the formal evaluation over the Hestia gold standard data, we also added the marked-up Hestia places into the mix, to be used over general input texts. There was a bit of fiddling and tweaking but basically this simple change did the trick, and brought our geotagging precision up to 87%, whilst maintaining acceptable recall, at 73%. (Hence an F-score of 79%.)
As for the georesolution step, the various ideas we explored were described in a previous post, so I’ll now just explain what worked best and why. We don’t have a way of formally evaluating the georesolution in terms of precision and recall, because we had no marked-up ancient text available with spatial co-ordinates defined. (And in any case, matching one point location against another can be a tricky business – see “Evaluation of Georeferencing“ for discussion.) But from simple visualisations of the data it was clear that if we used only Pleiades+ we missed quite a lot of places, because translators often use modern names (like “Egypt”) whilst Pleiades is all about ancient names (“Aegyptus”). If we tried to solve this by consulting Geonames as well then we were swamped with spurious modern places (like the “Priam” in Minnesota).
The neat solution that Leif came up with is to exploit the list of alternative names provided in Geonames. If we draw a blank on “Egypt” in Pleiades+ we can try it in Geonames and collect a set of alternatives that we then try on Pleiades+. One of them is indeed “Aegyptus”, so we score. There’s a risk that quite separate places may have alternative names in common, but overall this solution seems to work very well and it’s what we’ve used in the final results shown in the visualisations Eric and Nick have produced.
Of course the automatic geoparsing process will never be completely accurate, and its two-stage nature means that some errors are compounded. If a place-name is missed at the geotagging step there’s no opportunity to attempt georesolution for it. There are some compensations in the other direction however, and the GAP work is only making use of a subset of the Geoparser’s functionality. For example, “Phoenicia” is found by the geotagger but can’t be resolved because it’s not in Pleiades+ and the lookup for alternatives in Geonames also fails. (“Phoenice” is in Pleiades+ but is not listed as an alternative for “Phoenicia” in Geonames). However, because the Geoparser uses linguistic clues as well as simple lookups, and in fact outputs more information than GAP uses (such as extra entity classes, and relations between them), the Geoparser’s full output includes the information that Tyre (which is successfully located) is in Phoenicia. There is scope in future work for exploiting relationship information of this kind, which in this case would include a clue to where Phoenicia is.
Next time I’m here I’ll finish off on my end of things with a brief report about processing input texts on a large scale.