Glad you sorted this. What was the issue and how did you resolve it?
All the following is just editorial so if you’re here for information go back to the search results or use the search button (top right)! I’m just another punter, not part of Companies House, not employed by the UK government so I’m just opining.
TL/DR: You get what you pay for and this is a free service. Expect issues from very large / old / public datasets. Just because something’s a part of a government doesn’t mean it is particularly important to the government.
I agree that this is not a perfect system. There are issues with everything from the overall role of Companies House and how it’s resourced down through the API to the documentation. (I find the API works well once you’ve mastered providing authorization / rate limiting and you understand the limitations of the data.)
Based on my own experience with several APIs both government and corporate (REST, SOAP etc.) I have found that this almost never “works” straight away. “Standards” mean something rather loose in the REST world, certainly. In several cases this has involved “reverse engineering” or hacking about as you’ve described. In a number of those cases a “specification” was provided and that also proved to have issues / not match the API.
So maybe I have lower expectations than you? In my experience large systems will generally have inconsistencies, missing information, undocumented “features” - or (in my limited experience with Microsoft / Google) will overwhelm you with complexity, jargon / concepts and tutorials and examples. Many of the latter turn out not to work as given e.g. possibly are out of date. So now we expect and budget for time spend “just making it work”. This goes double for very large datasets, triple for old datasets and through the roof if - as in Companies House - you effectively have the public inputting data!
As someone who has spent far too much time trying to get access to other APIs I do understand that’s extremely frustrating. I think the way Companies House provide your login and some of the settings around that are confusing. However as I wrote in reply to you elsewhere the majority of issues seem to be around people’s unfamiliarity with http Basic and / or how to make that work with their tool / language. Yes, Companies House could have picked a different authorisation / authentication method (e.g. OAuth with bearer token, or http Basic but using a username and password rather than empty password). However as long as it’s implemented correctly - which I believe it is - getting that working is down to the users.
I suppose they could maintain a series of articles on access from different software and systems but they clearly don’t have time to keep the rest of the documents up-to-date and that’s an endless treadmill anyway. Aside from taxes if you live in the UK, no-one’s paying to use this. (If you do want a more “professional” service you can pay - there is the XML Gateway however (a) it’s SOAP / XML which many people find more complex and (b) I believe that’s due to be phased out in favour of the REST API at some point).
I would like to see:
-
An example with walk through - using some very general tool, possibly with http header fragments shown - maybe from searching for a company, obtaining the company profile, listing the filings and then downloading a document. I’d guess that covers both some of the most commonly used endpoints and that illustrates all the features e.g. looking up text in the “enum constants”, “paging” (search, filing history) and (the most troublesome part) the change from their http basic to Amazon authentication (downloading).
-
Fixing some simple issues with documentation and the specification. Some of these have been known for years and are well-reported here.