4.1. Evaluation Metrics

To evaluate the performance of the candidate algorithms, the implementers must log enough information to visualize the following metrics at a fine enough time granularity:¶

1. Flow level:¶

   1. End-to-end delay for the congestion-controlled media flow(s). For example, end-to-end delay observed on the IP packet level and the video frame level.¶
   2. Variation in sending bit rate and throughput. Mainly observing the frequency and magnitude of oscillations.¶
   3. Packet losses observed at the receiving endpoint.¶
   4. Feedback message overhead.¶
   5. Convergence time. Time to reach steady state for the congestion-controlled media flow(s). Each occurrence of convergence during the test period needs to be presented.¶

2. Transport level:¶

   1. Bandwidth utilization.¶
   2. Queue length (milliseconds at specified path capacity).¶

4.2. Path Characteristics

Each path between a sender and receiver as described in Figure 1 has the following characteristics unless otherwise specified in the test case:¶

Path direction:

forward and backward.¶

Reference bottleneck capacity:

1 Mbps.¶

One-way propagation delay:

50 ms. Implementers are encouraged to run the experiment with additional propagation delays mentioned in [RFC8868].¶

Maximum end-to-end jitter:

30 ms. Jitter models are described in [RFC8868].¶

Bottleneck queue type:

"tail drop". Implementers are encouraged to run the experiment with other Active Queue Management (AQM) schemes, such as FQ-CoDel and PIE.¶

Bottleneck queue size:

300 ms.¶

Path loss ratio:

0%.¶

Examples of additional network parameters are discussed in [RFC8868].¶

For test cases involving time-varying bottleneck capacity, all capacity values are specified as a ratio with respect to a reference capacity value, so as to allow flexible scaling of capacity values along with media source rate range. There exist two different mechanisms for inducing path capacity variation: a) by explicitly modifying the value of physical link capacity, or b) by introducing background non-adaptive UDP traffic with time-varying traffic rate. Implementers are encouraged to run the experiments with both mechanisms for test cases specified in Section 5.1, Section 5.2, and Section 5.3.¶

4.3. Media Source

Unless otherwise specified, each test case will include one or more media sources as described below:¶

Media type:

Video¶

Media codec:

VBR¶

Media source behavior:

Adaptability:

Bit rate range:

150 Kbps - 1.5 Mbps. In real-life applications, the bit rate range can vary a lot depending on the provided service; for example, the maximum bit rate can be up to 4 Mbps. However, for running tests to evaluate the congestion control algorithms, it is more important to have a look at how they react to a certain amount of bandwidth change. Also it is possible that the media traffic generator used in a particular simulator or testbed is not capable of generating a higher bit rate. Hence, we have selected a suitable bit rate range typical of consumer-grade video conferencing applications in designing the test case. If a different bit rate range is used in the test cases, then the end-to-end path capacity values will also need to be scaled accordingly.¶

Frame resolution:

144p - 720p (or 1080p). This resolution range is selected based on the bit rate range. If a different bit rate range is used in the test cases, then a suitable frame resolution range also needs to be selected.¶

Frame rate:

10 fps - 30 fps. This frame rate range is selected based on the bit rate range. If a different bit rate range is used in the test cases, then the frame rate range also needs to be suitably adjusted.¶

Variation from target bit rate:

+/-5%. Unless otherwise specified in the test case(s), bit rate variation should be calculated over a one (1) second period of time.¶

Responsiveness to new bit rate request:

100 ms¶

Media content:

The media content should represent a typical video conversational scenario with head and shoulder movement. We recommend using the Foreman video sequence [xiph-seq].¶

Media startup behavior:

150 Kbps. It should be noted that applications can use smart ways to select an optimal startup bit rate value for a certain network condition. In such cases, the candidate proposals may show the effectiveness of such a smart approach as additional information for the evaluation process.¶

Media type:

Audio¶

Media codec:

CBR¶

Media bit rate:

20 Kbps¶

5.1. Variable Available Capacity with a Single Flow

In this test case, the minimum bottleneck-link capacity between the two endpoints varies over time. This test is designed to measure the responsiveness of the candidate algorithm. This test tries to address the requirements in [RFC8836], which requires the algorithm to adapt the flow(s) and provide lower end-to-end latency when there exists:¶

• an intermediate bottleneck¶
• change in available capacity (e.g., due to interface change, routing change, abrupt arrival/departure of background non-adaptive traffic)¶
• maximum media bit rate is greater than link capacity. In this case, when the application tries to ramp up to its maximum bit rate, since the link capacity is limited to a lower value, the congestion control scheme is expected to stabilize the sending bit rate close to the available bottleneck capacity.¶

It should be noted that the exact variation in available capacity due to any of the above depends on the underlying technologies. Hence, we describe a set of known factors, which may be extended to devise a more specific test case targeting certain behaviors in a certain network environment.¶

Expected behavior:

The candidate algorithm is expected to detect the path capacity constraint, converge to the bottleneck link's capacity, and adapt the flow to avoid unwanted media rate oscillation when the sending bit rate is approaching the bottleneck link's capacity. Such oscillations might occur when the media flow(s) attempts to reach its maximum bit rate but overshoots the usage of the available bottleneck capacity, then to rectify, it reduces the bit rate and starts to ramp up again.¶

Evaluation metrics:

As described in Section 4.1.¶

Testbed topology:

One media source S1 is connected to the corresponding R1. The media traffic is transported over the forward path and corresponding feedback/control traffic is transported over the backward path.¶

                             Forward -->
+---+       +-----+                               +-----+       +---+
|S1 |=======|  A  |------------------------------>|  B  |=======|R1 |
+---+       |     |<------------------------------|     |       +---+
            +-----+                               +-----+
                          <-- Backward

Testbed attributes:

Test duration:

100 s¶

Path characteristics:

as described in Section 4.2¶

Application-related:

Media Traffic:

Media type:

Video¶

Media direction:

forward¶

Number of media sources:

one (1)¶

Media timeline:

Start time:

0 s¶

End time:

99 s¶

Media type:

Audio¶

Media direction:

forward¶

Number of media sources:

one (1)¶

Media timeline:

Start time:

0 s¶

End time:

99 s¶

Competing traffic:

Number of sources:

zero (0)¶

Test-specific information:

One-way propagation delay:

[50 ms, 100 ms]. On the forward path direction.¶

This test uses bottleneck path capacity variation as listed in Table 1.¶

When using background non-adaptive UDP traffic to induce a time-varying bottleneck, the physical path capacity remains at 4 Mbps, and the UDP traffic source rate changes over time as (4 - (Y x r)), where r is the Reference bottleneck capacity in Mbps, and Y is the path capacity ratio specified in Table 1.¶

5.2. Variable Available Capacity with Multiple Flows

This test case is similar to Section 5.1. However, this test will also consider persistent network load due to competing traffic.¶

Expected behavior:

The candidate algorithm is expected to detect the variation in available capacity and adapt the media stream(s) accordingly. The flows stabilize around their maximum bit rate as the maximum link capacity is large enough to accommodate the flows. When the available capacity drops, the flows adapt by decreasing their sending bit rate, and when congestion disappears, the flows are again expected to ramp up.¶

Evaluation metrics:

As described in Section 4.1.¶

Testbed topology:

Two (2) media sources S1 and S2 are connected to their corresponding destinations R1 and R2. The media traffic is transported over the forward path and corresponding feedback/control traffic is transported over the backward path.¶

+---+                                                         +---+
|S1 |===== \                                         / =======|R1 |
+---+      \\             Forward -->               //        +---+
            \\                                     //
            +-----+                               +-----+
            |  A  |------------------------------>|  B  |
            |     |<------------------------------|     |
            +-----+                               +-----+
              //                                    \\
             //          <-- Backward                \\
+---+       //                                        \\       +---+
|S2 |====== /                                          \ ======|R2 |
+---+                                                          +---+

Testbed attributes:

Testbed attributes are similar to those described in Section 5.1, except for the test-specific capacity variation setup.¶

Test-specific information:

This test uses path capacity variation as listed in Table 2 with a corresponding end time of 125 seconds. The reference bottleneck capacity is 2 Mbps. When using background non-adaptive UDP traffic to induce time-varying bottleneck for congestion-controlled media flows, the physical path capacity is 4 Mbps, and the UDP traffic source rate changes over time as (4 - (Y x r)), where r is the Reference bottleneck capacity in Mbps, and Y is the path capacity ratio specified in Table 2.¶

5.3. Congested Feedback Link with Bi-directional Media Flows

Real-time interactive media uses RTP; hence it is assumed that RTCP, RTP header extension, or such would be used by the congestion control algorithm in the back channel. Due to the asymmetric nature of the link between communicating peers, it is possible for a participating peer to not receive such feedback information due to an impaired or congested back channel (even when the forward channel might not be impaired). This test case is designed to observe the candidate congestion control behavior in such an event.¶

Expected behavior:

It is expected that the candidate algorithms are able to cope with the lack of feedback information and to adapt to minimize the performance degradation of media flows in the forward channel.¶

It should be noted that for this test case, logs are compared with the reference case, i.e., when the backward channel has no impairments.¶

Evaluation metrics:

As described in Section 4.1.¶

Testbed topology:

One (1) media source S1 is connected to corresponding R1, but both endpoints are additionally receiving and sending data, respectively. The media traffic (S1->R1) is transported over the forward path, and the corresponding feedback/control traffic is transported over the backward path. Likewise, media traffic (S2->R2) is transported over the backward path, and the corresponding feedback/control traffic is transported over the forward path.¶

+---+                                                          +---+
|S1 |====== \                Forward -->              / =======|R1 |
+---+       \\                                       //        +---+
             \\                                     //
          +-----+                               +-----+
          |  A  |------------------------------>|  B  |
          |     |<------------------------------|     |
          +-----+                               +-----+
             //                                     \\
            //            <-- Backward               \\
+---+      //                                         \\       +---+
|R2 |===== /                                           \ ======|S2 |
+---+                                                          +---+

Testbed attributes:

Test duration:

100 s¶

Path characteristics:

Reference bottleneck capacity:

1 Mbps¶

Application-related:

Media source:

Media type:

Video¶

Media direction:

forward and backward¶

Number of media sources:

two (2)¶

Media timeline:

Start time:

0 s¶

End time:

99 s¶

Media type:

Audio¶

Media direction:

forward and backward¶

Number of media sources:

two (2)¶

Media timeline:

Start time:

0 s¶

End time:

99 s¶

Competing traffic:

Number of sources:

zero (0)¶

Test-specific information:

This test uses path capacity variations to create a congested feedback link. Table 3 lists the variation patterns applied to the forward path, and Table 4 lists the variation patterns applied to the backward path. When using background non-adaptive UDP traffic to induce a time-varying bottleneck for congestion-controlled media flows, the physical path capacity is 4 Mbps for both directions, and the UDP traffic source rate changes over time as (4-x) Mbps in each direction, where x is the bottleneck capacity specified in Table 4.¶

5.4. Competing Media Flows with the Same Congestion Control Algorithm

In this test case, more than one media flow share the bottleneck link, and each of them uses the same congestion control algorithm. This is a typical scenario where a real-time interactive application sends more than one media flow to the same destination, and these flows are multiplexed over the same port. In such a scenario, it is likely that the flows will be routed via the same path and need to share the available bandwidth amongst themselves. For the sake of simplicity, it is assumed that there are no other competing traffic sources in the bottleneck link and that there is sufficient capacity to accommodate all the flows individually. While this appears to be a variant of the test case defined in Section 5.2, it focuses on the capacity-sharing aspect of the candidate algorithm. The previous test case, on the other hand, measures adaptability, stability, and responsiveness of the candidate algorithm.¶

Expected behavior:

It is expected that the competing flows will converge to an optimum bit rate to accommodate all the flows with minimum possible latency and loss. Specifically, the test introduces three media flows at different time instances. When the second flow appears, there should still be room to accommodate another flow on the bottleneck link. Lastly, when the third flow appears, the bottleneck link should be saturated.¶

Evaluation metrics:

As described in Section 4.1.¶

Testbed topology:

Three media sources S1, S2, and S3 are connected to R1, R2, and R3, respectively. The media traffic is transported over the forward path, and the corresponding feedback/control traffic is transported over the backward path.¶

+---+                                                         +---+
|S1 |===== \                Forward -->              / =======|R1 |
+---+      \\                                       //        +---+
            \\                                     //
+---+       +-----+                               +-----+       +---+
|S2 |=======|  A  |------------------------------>|  B  |=======|R2 |
+---+       |     |<------------------------------|     |       +---+
            +-----+                               +-----+
            //          <-- Backward               \\
+---+      //                                       \\       +---+
|S3 |===== /                                         \ ======|R3 |
+---+                                                        +---+

Testbed attributes:

Test duration:

120 s¶

Path characteristics:

Reference bottleneck capacity:

3.5 Mbps¶

Path capacity ratio:

1.0¶

Application-related:

Media Source:

Media type:

Video¶

Media direction:

forward¶

Number of media sources:

three (3)¶

Media timeline:

New media flows are added sequentially, at short time intervals. See the test-specific setup below.¶

Media type:

Audio¶

Media direction:

forward¶

Number of media sources:

three (3)¶

Media timeline:

New media flows are added sequentially, at short time intervals. See the test-specific setup below.¶

Competing traffic:

Number of sources:

zero (0)¶

Test-specific information:

Table 5 defines the media timeline for both media types.¶