How many feeds can be downloaded and processed in parallel?
I have access to multiple incident and / or flow feeds. How many feeds can be downloaded and processed in parallel?
There is no ideal setup for downloading and processing the feeds as there are many parameters that can have an influence on the overall performance. The following are just a few parameters (there may be more):
- The bandwidth of the internet connection from your servers to your internet service provider (ISP);
- The potential limitation of network bandwidth between your ISP and TomTom;
- The latency between your system and TomTom servers deployed in (Western) Europe;
- The performance of your system in ingesting and processing the retrieved data.
Due to these limitations we can make the following general recommendations:
- It may be advisable to download and process only a few feeds in parallel, or to download the individual feeds with a certain time delay.
- Your client should support TCP window scaling. With this option, we can send more data to your client before an acknowledgement comes back from your client. This improves throughput, especially in low latency connections. Latency is determined with tools like PsPing (https://docs.microsoft.com/en-us/sysinternals/downloads/pstools). To test latency between your servers and
traffic.tomtom.com, use the following command:
psping traffic.tomtom.com:443Further examples of how to use PsPing are found on the download page. While this tool is only available for Windows, there are alternatives for other operating systems here: (https://alternativeto.net/software/psping/). Since ICMP is not supported by our service, the usual ping command will not work.
- Always use gzip compression by specifying the standard HTTP header
- Always use the If-Modified-Since header with the time stamp of the Last-Modified header for the last snapshot received in the same feed. This eliminates the possibility of downloading that you already have.
- Use free-flow (ff) / non-free-flow (nff) update mechanism for all flow feeds. This significantly reduces the total bandwidth consumption when compared to full-feed downloads. The same information is retrieve!