[tor-relays] New Test for Measuring the Accuracy of Tor Relays' Advertised Bandwidths

[Georg Koppen]

Hello everyone!

Some of you might remember the speed test Rob had been running during
August 2019 to investigate the accuracy of the advertised bandwidth a
relay is reporting in its server descriptor.[1] Back then this involved,
for each relay, downloading several large data streams for a period of
about 20 seconds to get a better estimation of the relay's true capacity
(See: [1] and follow-up emails for the exact details of the experiment).

We plan to do this speed test again in about 1 week and relay operators
can opt-out of it (again) if they want to. We still believe that the
overhead of this speed test is in line with regular usage, but if anyone
doesn't want to participate that's fine. Let us know if so (by replying
to this thread or off-list, if preferred) and we remove your relay(s)
from the list to scan.

Thanks,
Georg

[1]
https://lists.torproject.org/pipermail/tor-relays/2019-July/017535.htmlff.

[Scott Bennett]

Post by Georg Koppen
Some of you might remember the speed test Rob had been running during
August 2019 to investigate the accuracy of the advertised bandwidth a
relay is reporting in its server descriptor.[1] Back then this involved,
for each relay, downloading several large data streams for a period of
about 20 seconds to get a better estimation of the relay's true capacity
(See: [1] and follow-up emails for the exact details of the experiment).
We plan to do this speed test again in about 1 week and relay operators
can opt-out of it (again) if they want to. We still believe that the
overhead of this speed test is in line with regular usage, but if anyone
doesn't want to participate that's fine. Let us know if so (by replying
to this thread or off-list, if preferred) and we remove your relay(s)
from the list to scan.
Thanks,
Georg
[1]
https://lists.torproject.org/pipermail/tor-relays/2019-July/017535.htmlff.

Okay, I read that, but I do not see where you offer any method to
account for other traffic that the relay being tested may be dealing with
at the time you are testing it. IOW, your testing procedure would be fine
if only the tor network, or at least each relay while it were being tested,
were entirely idle w.r.t. the traffic of other users. This is the same
defect that the authorities' testing procedures suffer from.
Do you have a solution to offer?


Scott Bennett, Comm. ASMELG, CFIAG
**********************************************************************
* Internet: bennett at sdf.org *xor* bennett at freeshell.org *
*--------------------------------------------------------------------*
* "A well regulated and disciplined militia, is at all times a good *
* objection to the introduction of that bane of all free governments *
* -- a standing army." *
* -- Gov. John Hancock, New York Journal, 28 January 1790 *
**********************************************************************

[Georg Koppen]

Post by Scott Bennett

Post by Georg Koppen
Some of you might remember the speed test Rob had been running during
August 2019 to investigate the accuracy of the advertised bandwidth a
relay is reporting in its server descriptor.[1] Back then this involved,
for each relay, downloading several large data streams for a period of
about 20 seconds to get a better estimation of the relay's true capacity
(See: [1] and follow-up emails for the exact details of the experiment).
We plan to do this speed test again in about 1 week and relay operators
can opt-out of it (again) if they want to. We still believe that the
overhead of this speed test is in line with regular usage, but if anyone
doesn't want to participate that's fine. Let us know if so (by replying
to this thread or off-list, if preferred) and we remove your relay(s)
from the list to scan.
Thanks,
Georg
[1]
https://lists.torproject.org/pipermail/tor-relays/2019-July/017535.htmlff.

Okay, I read that, but I do not see where you offer any method to
account for other traffic that the relay being tested may be dealing with
at the time you are testing it. IOW, your testing procedure would be fine
if only the tor network, or at least each relay while it were being tested,
were entirely idle w.r.t. the traffic of other users. This is the same
defect that the authorities' testing procedures suffer from.
Do you have a solution to offer?

I guess it depends on what your goal is as to whether calling it a
defect or not. For the test at hand, sure, we can't test as if no user
activity would be ongoing but that's not too important if we want to
test whether the advertised bandwidth significantly underestimates the
true capacity of Tor relays.

Georg

[Johnson, Aaron M CIV USN NRL (5543) Washington DC (USA)]

Post by Scott Bennett
Okay, I read that, but I do not see where you offer any method to
account for other traffic that the relay being tested may be dealing with
at the time you are testing it. IOW, your testing procedure would be fine
if only the tor network, or at least each relay while it were being tested,
were entirely idle w.r.t. the traffic of other users. This is the same
defect that the authorities' testing procedures suffer from.
Do you have a solution to offer?

I guess it depends on what your goal is as to whether calling it a
defect or not. For the test at hand, sure, we can't test as if no user
activity would be ongoing but that's not too important if we want to
test whether the advertised bandwidth significantly underestimates the
true capacity of Tor relays.

I do not believe there is a defect, or at least I don't understand the stated concern. If by "other users", you mean the Tor users other than the speedtest client, then there is no problem because the relay's advertised bandwidth takes into account all Tor traffic, regardless of user. If, however, by "other users" you mean users of other applications running on the same machine as the relay, then (1) if such other traffic is relatively constant, then the measurement is correct as we want to measure the maximum available capacity *for Tor users*; but (2) if such non-Tor traffic comes in infrequent bursts, then it is true that such a burst may occur during a speedtest measurement and inaccurately indicate low capacity, but given that such bursts are infrequent, this problem should occur infrequently among all Tor relays.

Best,
Aaron

[nusenu]

The effect of these measurements is visible in significant (temporary) "increase" of adv. bw
visible on metrics:

https://metrics.torproject.org/bandwidth.html?start=2021-04-20&end=2021-05-21

and on the OrNetStats per operator graphs, this is a good example:

https://nusenu.github.io/OrNetStats/emeraldonion.org.html
https://nusenu.github.io/OrNetStats/#proven-operator-domains

[Georg Koppen]

Post by nusenu
The effect of these measurements is visible in significant (temporary) "increase" of adv. bw
https://metrics.torproject.org/bandwidth.html?start=2021-04-20&end=2021-05-21
https://nusenu.github.io/OrNetStats/emeraldonion.org.html
https://nusenu.github.io/OrNetStats/#proven-operator-domains

For what it is worth: the experiment stopped at around 2021-06-05
00:00:00 UTC. The advertised bandwidth will decrease slowly and should
be without any influence from the experiment again later this week.

If you want to follow currently running experiments on the network (or
want to read up on past ones) we have created an entry, Network
Experiments, on our status page[1] showing whether experiments are
currently running or not.

Clicking there on "Network Experiments" should get you to the list[2] of
currently running and past experiments providing explanations as to how
long the experiments are already running/ran and what they are/were doing.

Let us know of any improvements to that Network Experiments part you
think we should make. We are still experimenting with it.

Georg

[1] https://status.torproject.org/
[2] https://status.torproject.org/affected/network-experiments/