Update Feb 23, 2019: You might also be interested in my initial take on FW v2.0.0, User Space Bootstrap in EdgeRouter Firmware 2.0.
In my previous post, we looked at the switch block in MediaTek's MT7621A SoC used in Edgerouter X. To recap, between the CPU and the switch is aggregate 2Gbit/s throughput. The switch block has five GbE ports as well as one optional RGMII interface for the additional SFP port in the Edgerouter X SFP.
While the information discussed there is true for MT7621A, my tests reveal a catch in Ubiquiti's implementation. This post intends to clarify further the switch block implemented in Edgerouter X and Edgerouter X SFP.
Throughput to and from the CPU
There are two 1Gbit/s links between MIPS 1004Kc CPU and the switch block. In common usage, one would make use of both links to achieve an aggregate 2Gbit/s throughput. So 1Gbit/s full-duplex routing between two network interfaces is made possible. For example, a symmetric gigabit WAN is well and fully served as such.
The catch in MT7621A is that the RGMII interface has to fan out from the switch block and that takes 1Gbit/s aggregate throughput away from the rest of the switch block. The 1Gbit/s link is dedicated to the CPU and the RGMII interface, leaving only 1Gbit/s link between the CPU and the rest of the switch block of five GbE ports.
Edgerouter X SFP comes with one SFP port. There are two implications here. First the SFP is only capable of 1Gbit/s aggregate throughput. In other words, maximum 500Mbit/s full-duplex. Other combinations 800Mbit/s TX and 200Mbit/s RX is fine as well as 400Mbit/s TX and 600Mbit/s RX just examples. The split between TX and RX is dynamically adjusted based on traffic flow but aggregate sum will always be 1Gbit/s or less.
The other implication is this 1Gbit/s aggregate limit also applies to the CPU and the five GbE ports. For example, if you have a symmetric gigabit WAN, you might get close to 1Gbit/s download or upload but not both simultaneously. Same applies to routing between two networks on LAN. The limit is 1Gbit/s aggregate to and from the CPU which routing needs to go (regardless HWNAT enabled or not).
Evidence from My Tests
Any tests between a gigabit WAN and LAN or between two network interfaces on LAN can prove the above assertion. As we see below, the routing throughput is ~500Mbit/s full-duplex between
eth1 is part of
eth2 is a separate network.
Worth noting that switching between two ports of
switch0 can achieve 1Gbit/s full-duplex. I only tested two ports at the same time. I don't know what's the maximum switching capacity. I would wager a second pair of ports will be switching at 1Gbit/s full-duplex too.
Why such limit in Edgerouter X?
For Edgerouter X SFP, I can understand the split of 2Gbit/s to and from CPU due to the additional SFP port and the design restriction in MT7621A. For Edgerouter X, I see no reason the 2Gbit/s links to and from CPU cannot be dedicated to the switch block i.e. the five GbE ports.
That's a good question for Ubiquiti. They may have artificially imposed the same limit for consistent performance across ER-X models with and without a SFP port. It may be also simply an oversight in their firmware build.
Let's hope with enough ER-X owners educated Ubiquiti will unleash the full capacity in ER-X with a future firmware update. If it's an oversight, shall be easy to resolve. If it's a marketing decision, then will be a little uphill struggle.