{"id":4175,"date":"2010-07-07T07:22:59","date_gmt":"2010-07-07T14:22:59","guid":{"rendered":"http:\/\/palblog.fxpal.com\/?p=4175"},"modified":"2010-07-07T07:21:52","modified_gmt":"2010-07-07T14:21:52","slug":"4175","status":"publish","type":"post","link":"https:\/\/blog.fxpal.net\/?p=4175","title":{"rendered":"Parsing patents, take 2"},"content":{"rendered":"

Working on parsing and indexing the patent collection that Google made available has been an interesting education in just how noisy allegedly clean data really is, and in the scale of the collection. I am by no means done; in fact, I’ve had to start over a couple of times. I have learned a few things so far, in addition to my earlier observations<\/a>.<\/p>\n

Data is messy<\/h3>\n

The data that the USPTO provided to Google consists of several formats. Older patents (per 2002) are stored in a text format that I described in an earlier post. The 2002 ZIP archive has two files, one with a v2.4 DTD, and one with a v2.5 DTD. The first file has an .sgm extension, the second .xml.\u00a0 The two files seem to contain the same data, however. In any case,the DTDs for 2002-2004 archives are a crazy set of mostly numbered deeply-nested elements such as SDOBI\/B100\/B110\/DNUM\/PDAT which leads to the patent number, and SDOBI\/B700\/B720\/B721\/PARTY-US\/NAM\/SNM\/STEXT\/PDAT which is an inventor’s last name.<\/p>\n

The DTD for the year 2005 (“us-patent-grant-v40-2004-12-02.dtd”) is a variant on the current one, with just a few annoying differences. For example, the name of the element that contains the “field of search” codes is different: the 2005 version uses “field-of-search” while later ones use “us-field-of-classification-search”. 2006 uses v41, butI have not figured out what the differences are between the v41 and v42 DTD are. In fact, I was not aware that the DTD for 2006 was different until quite recently.<\/p>\n

Finally, the years 2007-2010 share the v42 DTD (“us-patent-grant-v42-2006-08-23.dtd”).<\/p>\n\n\n\n\n\n\n\n\n\n
Years<\/strong><\/td>\nFormat<\/strong><\/td>\nDTD<\/strong><\/td>\n<\/tr>\n
-2001<\/td>\ntext<\/td>\nn\/a<\/td>\n<\/tr>\n
2002<\/td>\nSGML?<\/td>\nPUBLIC “-\/\/USPTO\/\/DTD ST.32 US PATENT GRANT V2.4 2000-09-20\/\/EN”<\/td>\n<\/tr>\n
2002-2004<\/td>\nXML<\/td>\nST32-US-Grant-025xml.dtd<\/td>\n<\/tr>\n
2005<\/td>\nXML<\/td>\nus-patent-grant-v40-2004-12-02.dtd<\/td>\n<\/tr>\n
2006<\/td>\nXML<\/td>\nus-patent-grant-v41-2005-08-25.dtd<\/td>\n<\/tr>\n
2007-2010<\/td>\nXML<\/td>\nus-patent-grant-v42-2006-08-23.dtd<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

These small differences are annoying because they make data extraction an uncertain process. Did I actually get all the differences, or will something crop up later that I hadn’t seen before? As things stand, for example, there are still two samples (from 2002 and from 2009) that failed to get parsed; one produced the following startling error from the SAX parser:
\n
\n[Fatal Error] :1:1: The parser has encountered more than \"64,000\" entity expansions in this document; this is the limit imposed by the application.
\n<\/code><\/p>\n

The upshot is that I guess I won’t know whether I’ve been able to parse everything until I actually run the entire dataset.<\/p>\n

Another set of hard-to-parse data are the classification codes. I am sure they are documented somewhere, but the sheer variety of them makes parsing and formatting them in a consistent and recognizable way a challenge. I still don’t know what to do with this one, for example: Patent 04795540 for class 204 has a subclass code that reads “243 R-247” in the data, but I could not figure out based on USPTO documentation<\/a> what that was or how to render it. Google shows it as “204\/243R” (search for patent:4795540<\/a>), the PDF file has it as “204\/243 R-247” and the USPTO summary has “204\/243 R-247\/” with the trailing slash, something that seems inconsistent with their convention of separating the class from the subclass with the slash.<\/p>\n

Even such simple things as country names for patent assignees are messy: some patents will add an X to the end of a country abbreviation, producing ‘JPX’ for Japan, ‘DEX’ for Germany, etc. Others use ‘JP’ and ‘DE’. For US patents, some specify the country name, others don’t.\u00a0 And by far the most surprising assignee for 62 of the 55K patents I have sampled is listed as “The United States of America as represented by the Secretary of the Army, Washington, DC”. I wonder what that means: I thought that Federal employees were not allowed to patent inventions, but it’s seemingly OK for others to assign their inventions to the government. Odd.<\/p>\n

And there is a lot of it<\/h3>\n

I’ve been trying to do test-driven development on this project to learn how and to cope with the evolving code given my understanding of what I am doing. Overall things have gone pretty well, and I have accumulated a bunch of tests for the various document types and field values. But just pulling out a few examples was inadequate, as digging deeper into the dataset produced more and more variations.<\/p>\n

I’ve taken to running my parser on a sample of documents from the past 22 years (one week’s worth per year) to get decent coverage, but we’re talking about processing 675 MB of zipped-up data, which takes a few hours on my reasonably-capable unix box.<\/p>\n

In short<\/h3>\n

This has been an interesting (if somewhat painful) learning experience. My initial optimism that I could have enough done to write up a submission for the PAIR workshop led me to take a few shortcuts that cost me quite a bit of time, and a chance at actually writing up the work. But I have certainly learned a lot about working with largish datasets. I expect there will be other unpleasant surprises in store, but at least I won’t be surprised when my code runs for hours and hours without getting anywhere.<\/p>\n

I would love to hear from anyone else who has looked at the data or tried parsing it.<\/p>\n","protected":false},"excerpt":{"rendered":"

Working on parsing and indexing the patent collection that Google made available has been an interesting education in just how noisy allegedly clean data really is, and in the scale of the collection. I am by no means done; in fact, I’ve had to start over a couple of times. I have learned a few […]<\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[123,213],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/4175"}],"collection":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4175"}],"version-history":[{"count":16,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/4175\/revisions"}],"predecessor-version":[{"id":4180,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=\/wp\/v2\/posts\/4175\/revisions\/4180"}],"wp:attachment":[{"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4175"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4175"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.fxpal.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4175"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}