Analyzing 1.7 million Starlink speedtests - starlinkstatus.space

Geschrieben von Tysonpower am {{ "2024-07-26T08:22:00+00:00" | date "longDate" }}

We collected and analyzed over 1.7 million performance measurements from Starlink users and I got the chance to present the results at the IEEE ICC 2024 conference in Denver, CO, USA in June.

Thanks to all Starlink users that participated in the project so far as well as my colleagues at the Friedrich-Alexander-University Erlangen-Nürnberg that helped analyze the data and creating the scientific paper.

This might be the biggest detailed dataset of Starlink performance measurements so far! Each of the data points includes a speed test, latency tests to different servers around the world, and information about dishy provided by the gRPC API.

In 2021, I started the starlinkstatus.space project, including the website, backend and a client application to collect data from Starlink users. The project goal was to collect detailed and independent performance statistics of Starlink around the World, all made possible by the community. Since then we collected over 3 million data points from people that collected data for my project. After some filtering we got 1.663.114 data points left from over 300 different Users, out of 29 different countries that we then analyzed.

Data point structure

Each data point in our dataset contains the following data.

ICMP ping measures round trip times (RTTs) to:

• Cloudflare (next Datacenter)
• Frankfurt - Germany
• Sydney - Australia
• Miami - USA
• Seattle - USA
• Los Angeles - USA
• Toronto - Canada

RPC to the Starlink Terminal collects data like: hardware & software version, latency to point of presence, alerts, service type etc.

Ookla Speedtest CLI collects: up and download throughput, latency and latency during load

IP geolocation using ip-api.com: collects the ISP, country, state, approx. lat/lon etc.

What we analyzed

• Forward link goodput by continent (download)
• Return link goodput by continent (upload)
• Goodput by time of day (checking for slowdown during peak hours)
• Hardware versions used over time by continent
• Goodput by hardware version
• Latency around the world by continent
• Latency by time (checking improvements of the system over 3 years)
• Latency by time of day (checking for slowdown during peak hours)

Conclusion from our scientific paper

https://starlinkstatus.space provides crowdsourced performance measurements of the Starlink system and allows valuable insights into the overall system performance. The software is a combination of readily available tools: Ping traces, terminal status via RPCs, and the Ookla Speedtest CLI. Thanks to a large number of contributions from many different countries, meaningful results have been obtained.

Goodput and latencies are comparable with terrestrial Internet access. However, goodput in general varies in both directions, and it further depends on the continent and time of day, with North America showing signs of a higher subscription rate and lower performance. The performance seems to be not affected by the hardware version of the terminal. At the time of writing, a new Rev4 terminal (which is not actuated anymore) became available which did not yet show up in the measurements. RTTs are comparable with terrestrial cellular networks and show significant variance.

Future work

We will continue to collect data over starlinkstatus.space, and we are happy about anyone that participates, so if you got Starlink yourself and can set up our client on a Raspberry Pi or similar we would really appreciate it!

There is also a new Starlinkstatus version in development that collects even more detailed data about latency and other metrics. This will also include a new UI for all Users, with more and better statistics on the website.

Currently, we don't know when this new version will be ready, we will let you know when it's available.

The slides & graphs

Attached are the slides i used during my presentation in denver, these incude all graphs from the paper we wrote.

IEEE ICC 2024 - Starlinkstatus.space public slides

The paper is named "Crowdsourced starlink performance measurements from https://starlinkstatus.space" and it should be availiable in the future on https://ieeexplore.ieee.org

How I ended up at a scientific conference with this

This whole project is still just a hobby for me because I'm interested in the technology around Starlink! So how did I end up presenting a scientific paper at a big conference in the USA organized by the IEEE?

In 2023, I was approached by Jörg from the Friedrich-Alexander-University Erlangen-Nürnberg, he was interested in my work and the data I collected. He wanted to cooperate with me to analyze the data and write a paper about it. The result of this cooperation was a paper we then sent to some conferences to publish, but at first all of them rejected it. After some more refinements, we just tried it at the quite big IEEE ICC conference without any real hope that they accept it because there is so much competition at this conference.

In January, we then got a big surprise via email that the jury liked the paper, and it was in fact accepted! I didn't believe the email at first, to be honest, and asked Jörg if he got the same email from the conference organizers. And yes, it was all official, the paper was accepted and had to be presented in June by one of us in Denver, CO, USA! In the following weeks, we finalized the paper and reworked all statistics with the current dataset as well as talked about who will travel to the US to present it. We decided together that I would fly to Denver on behalf of the university and hold the presentation. I never went to the US before in my life, so it was a cool opportunity for me!

The 10-hour flight from Germany to Denver was a bit long, but the rest of my time in the US was a cool experience! I was quite nervous right before the presentation as it was my first time presenting in English, but it all worked out great! In my short holiday time after the conference, I was also lucky to visit LA for a few days and even see a SpaceX Falcon 9 successfully launch some more Starlink satellites into orbit. What an awesome Time!