Streaming api for multiple endpoints

API Question
1

Hi,

For the same company, we usually fetch data from 6 different endpoints at the same time (company/officers/insolvency/persons-with-significant-control/charges/filing-history). Due to this we some times get rate limited when we have large traffic.

To solve this problem, we are planning on using the streaming api and are currently new to this. Say we maintain our own database and update it using the streaming api, given that there is a limit of 2 concurrent connections, how can we use the streaming api to fetch data from 6 different endpoints?

In a few forums I have come across using an offline connection and storing the timeouts. Would the below logic work? (Say, we run this task every 30 min)

for endpoint in endpoints:
     previous_timepoint = TIMEPOINTS[endpoint]
     data = get_streaming_data_using_single_http_request(endpoint, previous_timepoint)
     update_local_db(data)
     TIMEPOINTS[endpoint] = data["timepoint"]

Also, how do we get the full snapshot of the existing data till current timepoint before the 1st run of such a task?

Many thanks

2

That limit of 2 has been increased to 6. So you can connect to all streams.
You will be disconnected every 24hrs (at least) so you have to build in timepoint managment.

2 Likes
3

Hello, I have spotted that Connection limits section on Streaming API:
still mentions having max 2 connections per account. Please update this

4

Since the introduction of the persons-with-significant-control-statements endpoint, which now takes the total number of streaming endpoints to 8, we receive regular disconnections when we subscribe to the new endpoint.
Suggests that the previous limit was 7 - any chance this limit could be increased to 8 now @MArkWilliams?
Thanks

2 Likes
5

or even 9, if we count company-exemptions which is still stated as ‘in development’ but seems to be present on the server.

2 Likes