I can access the CH streams with:
curl -v -u my-streaming-key: “https://stream.companieshouse.gov.uk/companies”
I try the same thing with on my PHP site, using a socket to open a connection:
$surl=“ssl://stream.companieshouse.gov.uk”;
$sock = fsockopen($surl, 443, $errno, $errstr, 3);
if (!$sock) die(“$errstr ($errno)\n”);
echo ‘sock opened
’;
which definitely opens a socket
echo ‘
’;
print_r(stream_get_meta_data($sock));
print_r(stream_get_transports());
print_r(stream_get_filters());
print_r(stream_get_wrappers());
echo ‘
’;
and I can read from it:
echo ‘
’;
while (!feof($sock)) {
$fs=fgets($sock, 100);
if ($fs!=‘’) echo $fs.‘
’;
}
echo ‘
’;
My problem is that ALL my requests get ‘Bad request’, e.g.,
$req=‘GET -u my-streaming-key: https://stream.companieshouse.gov.uk/companies HTTP/1.1’;
fwrite($sock, $req.‘\r\n’);
So how do I write a good request from here?
1 Like
Not a streaming API user (we just use the Public Data / Document APIs via PHP) but in your example you appear to be mixing curl commands with sending HTTP requests yourself:
$req=‘GET -u my-streaming-key: https://stream.companieshouse.gov.uk/companies HTTP/1.1’;
You should be duplicating what you see being sent when you connect using curl (you’ve added the -v flag for verbose so that should be displayed).
So should you not be using something like:
$streaming_key = "MYSTREAMINGKEYHERE";
$host = "stream.companieshouse.gov.uk";
$port = 443;
$path = "/companies";
$surl = “ssl://" . $host;
$sock = fsockopen($surl, $port, $errno, $errstr, 3);
// ...
// Assuming you're using HTTP 1.1
fputs($sock, "GET $path HTTP/1.1\r\n");
fputs($sock, "Host: $host\r\n");
// Have to build your own headers - format is username:password - here password is "":
fputs($sock, "Authorization: Basic " . base64_encode($streaming_key . ":") . "\r\n");
fputs($sock, "Connection: close\r\n\r\n");
Again - I have not tested any of that, nor is it necessarily complete!
For more general info see e.g. PHP’s documentation for fsockopen, CH documentation, @ebrian101 3rd party site documentation, the source of his demo (in js / node) of using the streaming API etc.
Good luck.
1 Like
I tried that and it just hangs - the usual response time is quite fast from a terminal.
The code looks like WebSocket server-side dealing with the HTTP client request, so I don’t think it’s the right approach.
https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers
1 Like
The code is just your code, but not mixing up a curl command and a stream of http headers… I did say I haven’t run it.
I’m not aware of someone’s existing work you can just copy-paste here or specific PHP Companies House stream library. There are PHP libraries for the Public Data API in existence.
If you would prefer to try a different interface there is the PHP library Guzzle - that provides a nicer way to issue a http request and then work with a stream of data - e.g. via a psr-7 httpmessagestreaminterface.
There are tons of options for this / ways to modify the behaviour.
There’s a built-in for http authorization: Request Options — Guzzle Documentation
(I believe the defaults for connect / stream timeouts are both zero e.g. infinite so that should be ok).
Again the following is just a sample of Guzzle’s interface. I have not run it and don’t know if it will (and you’d need to e.g. use Composer to get the required classes, then include a line requiring the autoloader etc.). Over to you to write your own code!
use GuzzleHttp\Client;
$username = "MYAPIKEY";
$password = "";
$client = new Client( [
'base_uri' => 'https://stream.companieshouse.gov.uk',
] );
$response = $client->request('GET', '/companies', [ 'auth' => [ $username, $password ], 'stream' => true, ] );
$body = $response->getBody();
I haven’t found an example that exactly matches what you want to do (keep reading from a long-running / persistent https connection) but I think you need the “stream” parameter to indicate this e.g. avoid closing the connection. Over to you to Google.
Certainly the body above you get back can be used as a stream. Guzzle’s PSR7 interface provides a way to read a line (up to a given maximum bytes) that might be useful for that.
You’ll still end up essentially reading bytes from a stream and you’ll also need something for managing disconnect / reconnect - because that’s what Companies House specifies.
Good luck.
1 Like
Connecting is not the issue - I need to transfer focus to /companies or any of the other streams, and I can’t find any command that works.
I’ve tried everything in
but nothing works.
I currently use the -o (-output) switch in curl.( curl - How To Use) to redirect the terminal output to a text file which can then be processed at a later date, but I want to do this from my online stream.
(Updated - I re-read the link in your post, I see what you were looking for now).
Short - I don’t have a detailed recipe for you - we don’t use the streaming API, just the Public Data API (via PHP). I could just see a particular error (sending part of a curl command in an http header - which obviously won’t work).
Use of curl to dump to a file makes sense - but you say that’s not quite what you need. Presumably you want something which is triggered when an update from CH comes in?
It would seem the issue you’re having is more to do with “how can I get sockets / streams working for long-running http connections in PHP”. If it helps, I think Companies House put it in terms of downloading a very large file (where you only get chunks of data intermittently) *.
There is certainly info out there on using PHP sockets and streams, and I still think your best bet would be using something a bit higher-level like the Guzzle libraries.
For a general example there’s the one that I already mentioned is in javascript (live demo here, source here) - but it does at least show e.g. how by constantly listening to a long-running http connection periodic updates (to the HTML DOM in that case) can be generated. Again - PHP stream via the socket functions / simple fopen / a higher-level interface like Guzzle / perhaps PHP’s own curl interface would seem to offer what you want?
Perhaps someone else has more detail? Good luck.
- For using the streaming API as CH describe you will need to have some long-running process going all the time e.g. open an http connection to their server, send the initial request to Companies House then don’t hang up (unless they instruct you to) and keep listening, processing the data you’ll intermittently receive (more detail here). The output from that could either be piped to a “sink” e.g. a DB, file, other stream - OR you could trigger something when you receive data from the stream as the example app I linked to.
1 Like
I got it to work!
Use PHP with curl,
$url=“https://stream.companieshouse.gov.uk/companies”;
$hdr=array(‘Authorization:Basic base 64 streaming key’);
$fout=fopen(‘chout.txt’,‘r+’);
where ‘chout.txt’ is the file to output the data to.
Then use curl
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_HTTPHEADER, $hdr);
curl_setopt($curl, CURLOPT_FILE,$fout);
curl_setopt($curl, CURLOPT_VERBOSE,true);
curl_exec($curl);
and the file fills up with data!