RTT Analysis
This tutorial is for showing how to check / analyze various factors that may affect RTT and help with RTT optimization
- Ask yourself on why you want to minimize RTT (In many cases, you may not need to minimize it)
- Importance of RTT would vary depending on Application.
- In some cases, Jitter (variance of RTT) would be more important than RTT
- Except for Voice, realtime video RTT is not so important for most of application
In many case, RAN side portion would be relatively small out of the whole RTT (end-to-end RTT), but people want to minimize it just by tweaking RAN parameters
If you want to estimate / compare Core network impact on end-to-end RTT, RAN side RTT is not so important. What's important about RAN side impact on the end-to-end RTT is the exact range of RAN side delay and variation of the delay. If you know of the exact delay of RAN side delay, you can just substract the value from the end-to-end RTT to estimate the core network impact on end-to-end RTT
NOTE : For the technical tips of RTT improvement with Amarisoft solution, refer to this wiki.
Table of Contents
Introduction
Round Trip Time (RTT) is a critical performance metric in modern communication networks, representing the elapsed time for a signal to travel from a source to a destination and back again. In cellular systems—such as LTE, 5G NR, and beyond—RTT is influenced by multiple architectural layers including the Radio Access Network (RAN), core network, and external data networks. RTT has far-reaching implications for user experience, service latency, and the efficiency of protocols such as TCP, particularly in latency-sensitive applications like VoIP, online gaming, and real-time video streaming. Understanding RTT entails dissecting the contributing delays at each segment of the network: radio interface transmission, scheduling, processing times, queuing in the core network, and external routing. This tutorial provides an in-depth exploration of the factors affecting RTT, focusing on methods for its measurement, analysis, and optimization. By examining architectural considerations, key concepts such as jitter (RTT variance), and the contextual importance of RTT for diverse applications, learners will gain a comprehensive foundation for diagnosing and improving latency performance in end-to-end network scenarios. The tutorial also discusses the relative impact of RAN versus core network delays, best practices for RTT optimization, and tools for precise delay analysis, equipping technical professionals with actionable insights to enhance network responsiveness and service quality.
-
Context and Scope
- This tutorial addresses RTT analysis and optimization in cellular and IP-based communication networks.
- Emphasis is placed on both RAN and core network segments, providing a high-level architectural overview.
- Key technical concepts such as delay sources, jitter, and their impact on application performance are discussed.
-
Relevance and Importance
- RTT directly affects end-user experience in many real-time and interactive applications.
- Understanding RTT is vital for diagnosing network bottlenecks and supporting service level agreements (SLAs).
- Optimization of RTT can contribute to better throughput, lower application latency, and improved reliability.
-
Learning Outcomes
- Ability to identify and analyze factors contributing to RTT across network segments.
- Knowledge of measurement methodologies and interpretation of RTT data.
- Skills to distinguish between RAN and core network impacts on RTT and actionable methods for optimization.
- Understanding of application-specific RTT requirements and the trade-offs between RTT and jitter.
-
Prerequisite Knowledge and Skills
- Familiarity with basic networking concepts such as latency, throughput, and protocol stack layers.
- Understanding of cellular network architecture (e.g., RAN, core network, user plane, control plane) is helpful.
- Some experience with network measurement tools or protocol analyzers is recommended for hands-on analysis.
Summary of the Tutorial
This tutorial presents a series of test procedures and methodologies for analyzing Round-Trip Time (RTT) and connectivity across WiFi, LTE, and NR/SA (5G Standalone) environments using a Callbox and various Device Under Test (DUT) configurations. The focus is on preparation, connectivity verification, logging configuration, and execution of ping-based RTT tests.
-
Test Setup and Preparation
- Use the provided SIM card for DUTs unless reconfiguration is needed (refer to the Configuration Guide for changes).
- Callbox and DUTs are placed in separate rooms, 3 meters apart, separated by a wall to challenge wireless connectivity.
- Various internet access methods for the Callbox are illustrated; Case 4 is used in this tutorial. For WiFi-based access (Cases 2 & 3), a set of nmcli Linux commands is provided for configuring WiFi connectivity from the command line.
- Before UE connection tests, ensure all network interfaces (including tun0, tun1, tun2, tun3) are up, and verify routing tables and internet connectivity (e.g., ping 8.8.8.8 for DNS, ping a URL for DNS resolution).
-
Logging Configuration
- ENB and MME logging properties are adjusted to increase maximum log size, ensuring that the full IP header is captured for detailed packet analysis.
-
Test 1: WiFi Reference Test
- This baseline test uses a WiFi network and PCs only, with no cellular protocol or mobile phone involved.
- Ping tests are performed between multiple points within the local WiFi network:
- Ping from Point (A) to Point (C)
- Ping from Point (A) to Point (B)
- Ping from Point (C) to Point (B)
- External connectivity and RTT are evaluated by pinging:
- Google DNS (to measure RTT from Callbox to US)
- A PC in France
- A PC in Korea
-
Test 2: LTE Ping Test
- Test is performed using enb.cfg with LTE and 5 MHz bandwidth.
- Ping packets (ICMP) are observed at the GTP-U layer in logs for higher-layer analysis.
- Exact PHY-layer transmission and reception times for ICMP packets are determined by inspecting PHY frame logs (requires large ENB log size).
- By comparing PHY frame timestamps for Echo Request and Echo Reply, and assuming negligible MAC~PDCP processing time, the RAN-induced delay can be isolated.
- End-to-end delay is also measured from the application layer using a speed test tool (preferably one that measures both delay and jitter).
- Comparing application-layer and RAN delays allows breakdown of total delay into RAN delay and Core/Internet delay.
-
Test 3: NR/SA (5G Standalone) Ping Test
- Test utilizes gnb-sa.cfg with CONFIG = 2; RTT may vary based on TDD UL/DL configuration pattern.
- Ping packet observation and timing analysis mirrors the LTE test:
- GTP-U layer logs for higher-layer packet visibility
- PHY frame inspection for transmission and reception timing
- Calculation of RAN delay using PHY frame timestamps
- Application-layer end-to-end delay is evaluated using a suitable speed test tool.
- Comparison of RAN and application-layer delays provides a breakdown of total delay across network segments.
The procedures emphasize step-by-step network setup, verification, and careful logging, followed by methodical ping-based RTT measurements in reference (WiFi) and cellular (LTE, NR/SA) scenarios. Analysis at multiple protocol layers enables comprehensive delay characterization.
Test Setup
Test setup for this tutorial is as shown below.
- SIM Card used in this tutorial is the one delivered with the system as it is.
- If you want to change the configuration, The tutorial Configuration Guide would help
Following shows the location of the Callbox and the DUTs. Callbox and DUTs are in different rooms separated by a wall at a distance of about 3 m. (NOTE : Usually other test equipment does not provide stable connectivity in this kind of situation)
Three PCs share the local WiFi router. (A) is a Windows laptop at 10.0.0.104. (B) is a Fedora laptop with graphics at 10.0.0.162. (C) is the Callbox PC at 10.0.0.185, running Fedora in command line mode.
The remote targets are (D) www.google.com in the US, (E) www.amarisoft.com in France and (F) www.google.co.kr in Korea. I use these points in Test 1 to get reference RTT values without any cellular link.
The wireless connectivity among Callbox and all the DUTs used in this tutorial is as shown below.
The Ethernet port on the Callbox PC is where the Internet connection goes in. The antennas are connected only to the first SDR card (RF 1 / sdr 0). The UE connects to the eNB over the air through these antennas.
There are several many ways to provide internet access to your CallBox. Followings are some of them I can think of. In this tutorial, I used the Case 4.
Case 1 is a LAN cable from the Callbox Ethernet port to a wall socket with Internet access. It is the easiest way if you have such a socket. Case 2 uses embedded WiFi, which some Callbox models do not have.
Case 3 uses a WiFi dongle. Make sure it supports Linux, since some dongles do not work with Fedora or Ubuntu. This is not recommended with a fixed license, because a new NIC changes the host id that the license is mapped to. A floating or USB license has no such problem.
Case 4 connects a LAN cable between the Ethernet port of a WiFi extender and the Callbox Ethernet port. The WiFi extender has to be set up with your WiFi access point first.
If you decided to use Case 2 and Case 3, following tips would be helpful.
Since Amari Callbox is running command line mode of Linux (Fedora or Ubuntu) NOT on GUI , it is not straightforward to enable / connect WiFi.
Followings are some of the command line command you would need to get WiFi connectivity. You may search these commands on internet for the detailed usage.
# nmcli dev status
# nmcli radio wifi
# nmcli device wifi rescan
# nmcli dev wifi list
# nmcli dev wifi connect network-ssid password "network-password"
Configuration
You can use any configuration that will assign at least one IP to UE with internet apn. For example, you can use LTE attach configuration or SA or NSA configuration.
Check Up before trying UE connection
Before you trying internet connection with UE, you need to check several basic things shown in this section and make it sure everything works as shown here, otherwise Internet connection from UE may not work.
Make it sure that the network interface with internet connectivity is up and tun0,1,2,3 are up.
In my setup the LAN port eno1 got 10.0.0.185 from the WiFi access point. You may see a different IP depending on your Internet service provider. tun0 has 192.168.2.1, tun1 has 192.168.3.1, tun2 has 192.168.4.1 and tun3 has 192.168.5.1, all with netmask 255.255.255.0.
tun0, tun1, tun2 and tun3 are network interfaces created by the Callbox. Their IP addresses are configured in /root/mme/config/mme.cfg. eno1 and the tun interfaces do not need to be in the same subnet. NAT (Network Address Translation) forwards the packets between them.
Check up the routing table on CallBox PC. It would usually be as follows. (NOTE : The gateway Iface name may be different on your system depending on your system configuration. But you should see tun0, tun1, tun2, tun3 as shown below)
The first line (A) is the default route. Destination 0.0.0.0 goes to gateway 10.0.0.1 on eno1 with flags UG, so 10.0.0.1 is the gateway. The other lines (B) are the local subnets. 10.0.0.xxx goes through eno1, and 192.168.2.xxx to 192.168.5.xxx go through tun0 to tun3.
A packet to any other address goes through the default gateway on eno1. Linux adds the routes for tun0 to tun3 when the mme-ifup script configures the interfaces. The NAT is set up by the mme/lte_init.sh script. A packet from the UE to an address outside the tunX subnet is then forwarded to the default gateway with NAT.
Make it sure that the call box can reach a server over the internet. I am checking the connectivity as follows
Try ping to 8.8.8.8. This is google DNS and this is configured as DNS by default in mme.cfg
In this run every reply comes back with ttl=117. The time is between 11.0 ms and 21.6 ms. If you get no reply here, check the routing table and the eno1 connection before going further.
Try ping to a url to check the DNS is working
www.google.com resolves to yyz10s14-in-x04.1e100.net at the IPv6 address 2607:f8b0:400b:80f::2004. The replies take 10.4 ms to 13.1 ms. The name is resolved, so the DNS works.
[Optional] you may find the IP address to a specific url and try ping to the server with both in direct IP and url
nslookup www.google.com uses the DNS server 64.71.255.204 in this setup. It returns the IPv4 address 142.251.41.68 and the IPv6 address 2607:f8b0:400b:807::2004. You can then ping the IP address directly and compare it with the ping by url.
Setting Logging Property
For the detailed analysis, I set the (ENB) logging properties as follows. Increased Max size to capture the full IP header.
This is the Configure ENB dialog for the eNB at 10.0.0.185, port 9001, with Log buffer count 8192. Every layer from PHY to LPPa has Filter debug, Level debug and Max size 128. Payload is checked for all layers.
For the same purpose, set the (MME) logging properties as follows. Increased Max size to capture the full IP header.
This is the Configure MME dialog for the MME at 10.0.0.185, port 9000, with Log buffer count 8192. NAS, IP, S1AP, NGAP, GTPU and the other layers are set to Filter debug, Level debug and Max size 128. The IP and GTPU layers are the ones used to find the ping packets in the tests below.
Test 1 : WiFi
This Test is NOT a main purpose of the tutorial. This is just to show a comparative setup (a reference) to other tests. This Test is done with WiFi and PC. No cellular protocol and mobile phone is involved
As a starting reference, I ran a speed test from a PC on the WiFi network. In this run the download is 397.21 Mb/s and the upload is 20.69 Mb/s. The ping is 31 ms and the jitter is 2 ms. The provider is Rogers Cable and the test server is at Vint Hill.
ping from the point (A) to point (C). This is to check the connectivity within Local WiFi Network.
ping 10.0.0.185 -n 15 sends 15 packets from the Windows PC (A). All 15 replies come back with 0% loss. The RTT is Minimum 4 ms, Maximum 13 ms and Average 7 ms.
ping from the point (A) to point (B). This is also to check the connectivity within Local WiFi Network.
ping 10.0.0.162 -n 15 gives Minimum 19 ms, Maximum 116 ms and Average 70 ms. Even in the local WiFi network with the same radio condition, the RTT can be very different like this. The only difference from the previous test is the target PC.
ping from the point (C) to point (B). This is also to check the connectivity within Local WiFi Network.
ping 10.0.0.162 -c 15 runs from the Callbox PC (C). All 15 packets come back. rtt min/avg/max/mdev is 19.652/69.611/201.612/45.209 ms. This is close to the result from (A) to (B), so the large RTT follows PC (B).
ping from the point (C) to Google. This is to check the connectivity and RTT from the callbox to Google DNS located in US
ping www.google.com -c 15 goes to the IPv6 address 2607:f8b0:400b:80f::2004. rtt min/avg/max/mdev is 11.188/12.641/14.526/0.953 ms with 0% packet loss. The mdev of 0.953 ms is much smaller than inside the local WiFi network.
ping from the point (C) to a PC located in France to check Connectivity and RTT.
ping 94.23.206.73 -c 15 goes to the PC in France. rtt min/avg/max/mdev is 96.815/99.455/101.297/1.001 ms with 0% packet loss.
ping from the point (C) to a PC located in Korea to check Connectivity and RTT.
ping 222.231.2.40 -c 15 goes to the PC in Korea. rtt min/avg/max/mdev is 212.133/216.553/223.059/2.941 ms with 0% packet loss. This is about twice the RTT to France.
Test 2 : LTE Ping
In this test, I ran the test with enb.cfg file (LTE, 5 Mhz Bandwidth)
I sent ping 192.168.3.2 -c 15 from the Callbox PC to the UE. 192.168.3.2 is the UE IP on the tun1 subnet. All 15 packets come back, and rtt min/avg/max/mdev is 19.947/31.050/61.602/9.996 ms. The first reply takes 61.6 ms and the later ones go down to about 20 ms.
The question is how much of the 31.050 ms average comes from the RAN protocol stack. The log analysis below answers this.
You can check on ping packet (ICMP packet) at higher layer (GTPU) log
I searched the log for 192.168.3.1 > 192.168.3.2. The GTPU line shows G-PDU TEID=0x1c1aa08c SDU_len=84 IP/ICMP 192.168.3.1 > 192.168.3.2 at 12:21:48.920. This is the echo request going down to the UE. The echo reply comes back as G-PDU TEID=0xc064d626 IP/ICMP 192.168.3.2 > 192.168.3.1. Between them you can follow the packet through PDCP, RLC, MAC and PHY.
You can check the exact timing when the ICMP packet is transmitted at PHY layer by checking the contents of PHY frame. This is for the packet transmitted from the callbox. This is why I set the large value to (ENB) log property at the beginning.
The PDSCH at SFN 3.35.3 carries the echo request, with harq=1, tb_len=101 and mod=2. In its payload, c0 a8 03 01 is the server IP 192.168.3.1 and c0 a8 03 02 is the UE IP 192.168.3.2. The next byte 08 is the ICMP type for Echo Request.
You can check the exact timing when the ICMP packet is recieved at PHY layer by checking the contents of PHY frame. This is for the packet received by the callbox.
If you assume that the processing time in MAC~PDCP is negligibly short, you can check the delay time caused by RAN by comparing the PHY frame timing for Echo Request and Echo Reply.
The PUSCH at SFN 3.36.8 carries the echo reply, with tb_len=157, mod=6 and crc=OK. In its payload the UE IP c0 a8 03 02 is the source and the server IP c0 a8 03 01 is the destination. The ICMP type is 00, which is Echo Reply.
From SFN 3.35.3 to 3.36.8 is 1 frame and 5 subframes, which is 15 ms. Assuming the processing time in MAC~PDCP is negligible, this is the delay from the RAN protocol stack. It can be as short as 8 ms if the UE does not need to send SR for the UL grant. It can be 12 ms if the UE needs to send SR.
You can also check on the end to end delay from an Application layer using a speed test tool. Not all speed test tool shows delay time (like Ping delay). It would be good to find the speed test tool that shows the delay (and jitter).
If you compare the delay at the application layer and the delay by RAN as shown above, you can break down the end to end delay into RAN delay and Core/Internet delay.
On the LTE phone the speed test gives ping 51 ms and jitter 4 ms, with download 43.98 Mb/s and upload 11.49 Mb/s. The provider is Rogers Cable, the same as in Test 1.
This delay is large compared to the 15 ms RAN delay from the log. In this case a RAN parameter change would not improve much in terms of the percentage of the total end-to-end delay.
Test 3 : NR/SA Ping
In this test, I used gnb-sa.cfg with CONFIG = 2. The RTT may vary depending on TDD UL/DL Configuration pattern.
I sent ping 192.168.3.2 -c 15 from the Callbox PC to the UE again, this time over NR SA. All 15 packets come back, and rtt min/avg/max/mdev is 13.922/23.558/43.535/7.912 ms. The average is lower than the 31.050 ms of the LTE test.
The question is again how much of the 23.558 ms average comes from the RAN protocol stack. The log analysis below answers this.
You can check on ping packet (ICMP packet) at higher layer (GTPU) log
The IP line at 14:38:33.322 shows IP/ICMP 192.168.3.1 > 192.168.3.2, the echo request to the UE. It goes down as a GTPU G-PDU with TEID=0x2e0f84d9 and QFI=1. The echo reply comes back as a G-PDU with TEID=0x8f1d583f at 14:38:33.336, then as IP/ICMP 192.168.3.2 > 192.168.3.1.
You can check the exact timing when the ICMP packet is transmitted at PHY layer by checking the contents of PHY frame. This is for the packet transmitted from the callbox. This is why I set the large value to (ENB) log property at the beginning.
The PDSCH at SFN 11.544.1 carries the echo request, with k1=7, nl=2, tb_len=241 and mod=8. In its payload, c0 a8 03 01 is the server IP and c0 a8 03 02 is the UE IP. The ICMP type 08 marks it as Echo Request.
You can check the exact timing when the ICMP packet is recieved at PHY layer by checking the contents of PHY frame. This is for the packet received by the callbox.
If you assume that the processing time in MAC~PDCP is negligibly short, you can check the delay time caused by RAN by comparing the PHY frame timing for Echo Request and Echo Reply.
The PUSCH at SFN 11.544.19 carries the echo reply, with tb_len=141, mod=4 and crc=OK. In its payload the UE IP is the source and the server IP is the destination. From slot 11.544.1 to 11.544.19 is 0 frame and 18 slots, which is 9 ms.
Assuming the processing time in MAC~PDCP is negligible, this is the delay from the RAN protocol stack. How much it can be reduced depends on the minimum k1 and k2 that the UE and the RAN support. In my experience, high end UE modems support k1 and k2 = 1 in FR1. The delay can then be as short as 2~3 ms. Even then, the TDD UL/DL pattern and the MAC/PHY scheduling of the RAN affect whether you reach it.
You can also check on the end to end delay from an Application layer using a speed test tool. Not all speed test tool shows delay time (like Ping delay). It would be good to find the speed test tool that shows the delay (and jitter).
If you compare the delay at the application layer and the delay by RAN as shown above, you can break down the end to end delay into RAN delay and Core/Internet delay.
On the NR phone the speed test gives ping 50 ms and jitter 8 ms, with download 63.09 Mb/s and upload 1.17 Mb/s. The test server is in Washington and the provider is Rogers Cable.
This delay is large compared to the 9 ms RAN delay from the log. So here too, a RAN parameter change would not improve much in terms of the percentage of the total end-to-end delay.
