SDR Check Up

The purpose of this tutorial is to shows how to check basic status and information about sdr cards installed on the callbox PC. This will help you to find root cause or troubleshoot for the sdr card related problems that you may face.  

SDR stands for Software Defined Radio. The SDR card is the hardware card placed in PCI slots of the Callbox or UEsim. Each of the cards has multiple RF port (SMA type) where you connect antenna or coex cables. There are not so many cases where you need to check on SDR. But there are a few cases where it is worth checking SDR status/functionality.

Whenever you think you have any issues with sdr card (e.g, UE does not seem to receive any signal from Callbox), first suggestion is always to tweak the power of RF ports using tx_gain, rx_gain, t spl

If you end up with failure with the teaking the power tuning,  it is not easy to confirm whether you have any issues with sdr power issue or timing/sync issue.  so more practical advise would be ' do some sdr check up as in this tutorial whenever you have some problem and have done all other software related troubleshooting)

Table of Contents

• SDR Check Up
• Test Setup
• Key Command Line
• Check Up #1 : Errors / Warnings
• Check Up #2 : Check SDR Status with sdr_util
• Check SDR Status
• ./sdr_util -c all version   
• ./sdr_util -c all sync_state   
• ./sdr_util -c all gps_state   
• ./sdr_util -c all clock_tune
• ./sdr_util -c all temp
• Sync with GPS
• Check Up #3 : Check SDR power with sdr_spectrum
• Check Up #4 : Running LTE on different sdr cards
• Running on sdr 0
• Running on sdr 1
• Running one cell(one NR Cell) on 2 x sdr 20
• Check Up #5 : Clock Calibration
• Check Up #6 : Checkup for underflow/overflow
• Check Up #7 : Hardware Check-Up
• Check if each individual sdr card works
• Check if Multiple cards works
• Additional Information
• Max Power

Test Setup

Test setup for this tutorial is as shown below.  I would suggest you to connect antenna (or RF cables) to as many sdr cards as possible as per your needs.

Key Command Line

Followings are the list of the important command used in this tutorial.

• sdr_util
• ./sdr_util -c all version
• ./sdr_util -c all sync_state
• ./sdr_util -c all gps_state
• ./sdr_util clock_tune
• ./sdr_util -c all temp
• sdr_test
• sdr_spectrum

Check Up #1 : Errors / Warnings

When you have explicit errors or warnings about sdr cards. Some example of this kind of errors are as follows : ( NOTE : The full set of error/warning list is much longer than this, but I just tried to put some of the errors that most commonly happen. If you want to get the full list of error/warning, check out this file)

In most cases when you have these kinds of error, you are not allowed to proceed any further. So the first step is to find the root cause of the error / warning and remove the error / warning.

• "*** Can't initialize phy ***"
• "Clock of device 'DDD' is not master - please check cables"
• "Clock of device 'DDD' is not slave - please check cables"
• Clock source of SDR device DDD should be set as master for the first one and slave for the others
• "Error: Timeout while waiting for the PPS"
• Timeout waiting for a valid SYNC signal (PPS).
• "Error: GPS time not available for sync"
• Sync is set as GPS but no signal received.
• "Can't init thread"
• Fatal system error
• "ERROR: OverTemp on 'DDD':  FPGA:FFF°  RFIC:RRR°"
• Temperature is above threshold for device DDD
• Low level internal error: retry

Check Up #2 : Check SDR Status with sdr_util

If you don't have any specific warnings or errors and are still suspicious whether a sdr_card is functioning OK or not.

Check SDR Status

You may check on some basic status of sdr card using sdr_util.

Go to the directory /root/trx_sdr as shown below

NOTE : Make it sure that lte service is not running before you run any of the commands shown in this test  (just do { service lte stop })

Check how many SDR cards are installed and basic information of each cards. run ./sdr_util -c all version

NOTE : If any of SDR is now shown in the output, you would need to do hardware checkup as described here.

./sdr_util -c all version   

: returns Basic informations of all sdr cards detected from the system

./sdr_util -c all sync_state   

: check if each of the sdr cards are synched and prints sync_src, clock_source etc

NOTE : Usually the card with sync_source = 0 functions as sync master that provides sync signal to other cards with sync_source != 0. The sync ports on every sdr cards are cable-connected and by default sdr 0 is configured to be sync_source(sync master, internal) and other sdr cards are configured to be sync_sinc (sync slave, external). If you are not using sdr0 and using other cards only, the cards that you are using is configured to be sync source (internal). Even in this case, sdr0 always remain as (internal).

./sdr_util -c all gps_state   

: Check gps status of all sdr cards (You will get the following result { GPS not locked } if GPS port on SDR card is not connected to GPS antenna)

If your SDR card is connected with GPS Antenna, check if the GPS signal is properly detected.

If sdr 0 is connected with GPS antenna, you can check the gps_state only for sdr 0 by running ./sdr_util -c 0 gps_state

If the GPS signal is properly detected, you will get the message 'GPS locked(active)' as shown below.

If the GPS signal is not detected, you will get the message 'GPS not locked'.

./sdr_util -c all clock_tune

This gives you the current drift applied. If it is 0, it is likely the factory process was not applied yet on this board. (NOTE : If a sdr is not used as a sync master or as a standalone, it may be OK if it is not clock-tuned by the factory).

./sdr_util -c all temp

Check temperature of all sdr cards. (NOTE : If you are using very old release, sometimes you may have false alarm of high temperature. Try reboot the PC and check the temperature again when you see high temp alarm before you conclude that the sdr has temperature issue).

Sync with GPS

If you have GPS antenna connected to GPS input port on SDR card, you may improve the performance (possibly reduce BLER) by synchronizing the SDR card with gps clock.

Check Up #3 : Check SDR power with sdr_spectrum

If you don't find any outstanding issues in previous checkup, next step I want to suggest is to let the sdr transmit a known signal and check out the power coming out of the RF port of the sdr.

You can generate a predefined reference signal from a SDR card and check out the power from the RF port. If you have any spectrum analyzer, you can use it. If you don't have any spectrum analyer, you can use another SDR card installed on your system as a spectrum analyzer.

Check out this tutorial for the details : sdr_test-generated OFDM signal on sdr 0 and sdr_spectrum on sdr 1

Check Up #4 : Running LTE on different sdr cards

This is mainly for troubleshooting when you have multiple SDR cards and UE is not attaching to the cell even after you did all other procedure / checkups (e.g, the procedure described in lteenb.pdf  - 6.5).

The idea is to run simple LTE(or NR) requiring only one SDR card on different SDR card one by one.

Running on sdr 0

In this test, I run an LTE cell on sdr0 and then run it on sdr1. If UE can attach to both cards, you can say both SDR cards are functioning OK. I used the configuration file enb.default.cfg as the baseline and modified it for this test as follows.

NOTE : You can change enb.default.cfg file for this test, but I strongly recommend you to copy the enb.default.cfg to another name (e.g, enb-sdr0.cfg) and modify the copied file.

I have added the following configurations.

Restart the lte service (service lte restart) and launch screen (screen -r) and check rf_info in (enb) window. It will show you the sdr information as shown below.

Now run trace command and get UE attached to the cell as shown below. If UE successfully attaches and you see some traffic, it would be a good indicator that the sdr is functioning properly (NOTE : even if UE successfully attaches and gives you some traffic, there still be some possibility of sdr card problems like problem with a certain bandwidth, frequency etc, but at least you can say there is no crtical problems with the sdr card)

Running on sdr 1

In this test, I run an LTE cell on sdr0 and then run it on sdr1. If UE can attach to both cards, you can say both SDR cards are functioning OK. I used the configuration file enb.default.cfg as the baseline and modified it for this test as follows.

NOTE : You can change enb.default.cfg file for this test, but I strongly recommend you to copy the enb.default.cfg to another name (e.g, enb-sdr0.cfg) and modify the copied file.

I have added the following configurations.

Restart the lte service (service lte restart) and launch screen (screen -r) and check rf_info in (enb) window. It will show you the sdr information as shown below.

Now run trace command and get UE attached to the cell as shown below. If UE successfully attaches and you see some traffic, it would be a good indicator that the sdr is functioning properly (NOTE : even if UE successfully attaches and gives you some traffic, there still be some possibility of sdr card problems like problem with a certain bandwidth, frequency etc, but at least you can say there is no crtical problems with the sdr card)

Running one cell(one NR Cell) on 2 x sdr 20

In some cases, an sdr would work fine but you may see some issues when you combine multiple sdr cards to run one cell (e.g, achieving wider bandwidth than single sdr bandwidth). For this operation, refer to this tutorial.

Check Up #5 : Clock Calibration

If all of the check up mentioned above (probably except Check Up #4) did not show any conspicuous (outstanding) problem and still an SDR does not work, you may think of the possibility of Clock Calibration. Clock Calibration in this context mean to tune up an SDR card clock to GPS clock.

Regarding how to do clock calibration, refer to this wiki.

Amarisoft deliver SDR card with factory calibration. So it is less likely for you to have clock calibration issue, but you may consider trying this calibration in some situation exampled below.

• It has been used over long time (months/years) and the internal oscillator might have drifted
• You want to synchronize our enb/gnb with a other test equipment
• If you are using SDR card on UEsim and notice the CFO is too high (NOTE : if abs(CFO) > SCS (in Hz), it is strongly suggested to try clock calibration. this criteria is based on assumption that there is no clock issue on gNB/eNB side)  

Check Up #6 : Checkup for underflow/overflow

When you have 'underflow / overflow' even after all the previous checkup has passed, I would suggest you to check out this wiki.

Check Up #7 : Hardware Check-Up

There are some cases where most of the basic checkup (e.g, checkup 1, 2, 6) passed but still fails to get SDR working for the complete call connection. In this case, you would need to check on the possible hardware issues. However, there is no simple and clear way to identify the root cause of hardware issues. It is good time for you to get a chance to open up the Chassis of the PC and play with sdr cards and slots. Since there is no determistic ways to trace out the root cause of the hardware issues, I will try to consolidate various trials based on cases that I personally came across.

Check if each individual sdr card works

First, take out all the sdr card. You would get the empty PCI slots as shown below (NOTE : The types of PCI slots would vary depending on the product that you have. In my case, I have a Callbox Classic and the PCI slots looks as below).

Pick any one PCI slot and plug in only one of your sdr cards and check if it works. And then check if the card works with no problem. For this checking, I would try followings

• i) Perform Checkup #2
• ii) Perform Checkup #6
• iii) Run lte service with any single cell configuration (e.g, enb.default.cfg or gnb-sa.cfg) and check if the initial attach with UE complete successfully
• iv) If there is no problem with step i),ii),iii), pull out the sdr cards, plug in another sdr cards and repeat the step i), ii), iii)
• v) Identify all the sdr cards that is working OK.

Check if Multiple cards works

If you have multiple sdr cards that is confirmed working at previous step, check if those cards works OK in multi sdr configuration.

Plug in two sdr cards in two PCI slots and connect Sync cable as shown below (NOTE : The way the sync cable is connected varies with the type of sdr card (SDR50 or SDR100). The image shown below is the case of SDR50 cards plugged into a Callbox Classic).

NOTE : When you connect Sync cable, make it sure that it is tighly connected to socket. If it is not tightly connected or there is any issues with the cable itself, you would see various problems (e.g, second card not detected, underflow/overflow, second cell detection failure etc).

Make it sure that you get green LED lit on each of the cards.

Now check out if the sdr cards work with multi card configuration as follows :

• i) Perform Checkup #2
• ii) Perform Checkup #6
• iii) Run lte service with any two cell configuration (e.g, enb-2cell-ho.cfg) and check if the initial attach with UE complete successfully

NOTE : If you have any problems with this setting, it is highly likely that the problem would be caused by the Sync cable itself or Sync cable connection. It is less likely that the problem is from the sdr card itself because all the sdr cards in this test is proven working at previous step (single card checkup).

Additional Information

Max Power

the max power would vary depending on frequency, bandwidth and tx_gain. The measurement condition for the documentation is a little different between SDR 50 and SDR100. You need to refer to different document depending on the type of SDR card you are using.

• For SDR 100, refer to this document 
• For SDR 50, refer to this document  

NOTE : Please pay attention to measurement condition mentioned in the document.