IP Packet Overhead
Richard Hay
1 Introduction
What
does it cost for transport? This
question can be applied to moving goods and delivering services across
distances. King Hussein of
Let
me tell a true story about the first time I realized lower layer overhead
matters and makes a difference in performance results. The most compelling reason to understand the
overhead costs for transporting IP packets is the impact on performance. And performance becomes more important as more
voice, video, and data services use IP for content & service delivery. The world is going IP baby. IP over everything. These are the mantras of the Internet
Age. Adopt a common packet switched
& routed topology and global intercommunication is facilitated. Point to point dedicated connections are
replaced with multi-user networks that determine delivery on a per packet
basis. And that ties in to the story.
Once
upon a time I was a test engineer for a Tier 1 IP Service provider. There was a test process for evaluating the
performance of a POS (Packet over SONET) OC-48 interface on a high end backbone
router. As we will discuss later on,
OC-48 has a line rate of 2.488 Gbps. But after you remove the section, line, and
path overhead associated with SONET, 2.39616 Mbps of payload remains as useable
for transporting data (like PPP, HDLC, or Frame-Relay encapsulated IP
packets). To saturate the POS OC-48
link, the lab had an IP packet generator/analyzer (a packet blaster) with 6
Gigabit Ethernet test interfaces that linked into 6 GigE
interfaces on two routers interconnected with a POS OC-48 link. So there were three Gigabit Interfaces on
either side of the POS OC-48 allowing for up to three Gbps
of Ethernet IP packets to be sent across the POS link bidirectional. Refer to Figure I
for a graphic of the test topology.
Figure
I. POS OC-48
Test Topology
When
I ran the tests I had some of the oddest results I had ever encountered. Typically IP performance is a function of
packet distribution. The smaller the
size of the IP packets the more demanding routing them becomes to achieve the
same level of performance. A routing
engine has to dedicate the same processing lookup for a 40byte TCP SYN packet
as for a 1500 byte HTTP data packet but the 1500 byte packet carried a lot more
data with it. If there are increased
numbers of smaller packets are on a network, the more demanding it is for the
routers to perform at the same data rates.
Here is an example. Say the
network load is 15 Mbps (15,000,000 bits per second). To create that load using 1500 byte IP
packets using 802.3 100baseTX Ethernet, you would need to send 1219.116 packets
per second (we will discuss how we derived this rate later in the text). To create the same 15 Mbps condition using 40
byte IP packets (which with be transported over Ethernet using 64 byte Ethernet
Frames) would 22,321.429 packets per second.
That is more than 18 times as many packets to create the same bits on
the wire condition. So smaller packets
are more demanding than big packets.
They are shorter and often less latent, but their small size makes it
possible for there to be many more packets that need to be transported. And typically, the results reflect that and
fixed size tests using small packets tend to experience failure more often than
large size packet tests. But this test
yielded results that were counter-intuitive and the opposite of the expected
result. The tests were passing at 100%
utilization for 64 and 128 byte packet tests (controlled tests based on RFC
2544 that send fixed size packet tests using the sizes: 64, 128, 256, 512,
1024, 1280, and 1518), but dropping packets below line rate for larger
packets. So on
the face of it the router was doing better with massive volumes of smaller
packets while the bigger packets were causing problems. Another thing that should be pointed out is
that 100% line rate utilization equated to three Gbps
of IP Ethernet traffic. How could three Gbps of IP traffic traverse a link that has a maximum
payload capacity of 2.39616 Gbps? Every packet size should have experienced
loss at below line rate at rates consistent with the maximum capacity of the
OC-48 link. Three Gbps
is greater than 2.39616 Gbps. Except when it isn’t. What caused this anomaly? The answer was found in analyzing the
differences in lower layer overhead between Ethernet and POS.
As
this book will explain, a 46 byte IP packet inside of a 64 byte Ethernet frame
actually utilizes at least 84 bytes on the Ethernet wire. But only the 46 byte IP packet is routed onto
a POS link for transport. Now to
traverse the POS connection the 46 byte IP packet was encapsulated using HDLC
which places a 4 byte header on the packet plus a 2 byte checksum (it can also
use a 4 byte checksum but it was 2 bytes in this case) and a 1 byte flag to
delimit packets. So a 46 byte IP packet
on the POS link required 53 bytes to transport it in this scenario. So of the 84 bytes used on the Gigabit
Ethernet wire per 46 byte IP packet, only 53 bytes made it to the POS wire when
the IP packet was de-encapsulated and re-encapsulated using HDLC over
SONET. When you do the math and divide
53/84 only 63% of the load of the Gigabit Ethernet wire was translated to the POS
wire at this packet size. The result was
less than 2 Gbps of load was placed on the POS OC-48
from three Gigabit Ethernet links operating at 100% load. And that is less than the payload throughput
capability of OC-48 at 2.39616 Gbps. That is why the test succeeded. Because Ethernet is far from the most
efficient networking technology ever invented.
It is just one of the cheapest and easiest. Now the 128 byte packets tests created a
condition where the 148 bytes used on the Ethernet wire translated to 117 bytes
on the POS wire and 117/148 is closer to 79% of the traffic from the Ethernet
makes it to the POS, but that still creates 2.3716 Gbps
of load which is still less than the 2.3916 Gbps of
payload capacity OC-48 has available. As
the packet sizes get larger the fixed per packet overhead incurred by Ethernet
and the efficientcy of POS becomes less dramatic and
the three Gbps of Ethernet traffic is able to oversubscribe
the POS OC-48 and packet loss occurs because the interface is being asked to
transport more packets than is physically possible. That was the reason for the counter-intuitive
results. POS is more efficient than
Ethernet. And that is something that can
only be understood by analyzing physical and data-link overhead.
How
does anyone come into a situation where knowing the per packet layer1 &
layer2 overhead conditions for various technologies even matters you may
ask? The answer is found in IP test labs
all over the world. Anyone who has ever
used a packet blaster (a device that is designed to perform high end packet
generation and analysis) knows that the results are given in packet rates, not
bps. This is where calculating per
packet overhead including lower layer overhead comes into practical
application. Also, most test engineers
generate packets using IP Ethernet interfaces and often are testing interfaces
using a WAN technology (ATM, Frame-Relay, SONET, etc…). In that scenario it is key
to understand the differences between the IP packets on the interfaces you are
sending to and from, and the ones that are being traversed in the network
system under test.
1.1 Packets/Sec & Bits/Sec
IP
performance can be measured in a variety of ways. Packets per second (pps) is one
metric of network performance. So too is
bits per second (bps). Of course, bps is far more common in
marketing materials from service providers because it sounds impressive. One is reminded of the Doctor in Back to the
Future™ as he exclaimed ‘One point one
GIGAWATTS!!!’ as if that was more power than could be imagined (apologies to
the international audience and we will keep the American pop cultural references
to a minimum). So too is 10 Gbps. 10 billion
zeros and ones sent every second and processed by the receiver. That is a lot of zeros and ones to process
every second. It is impressive. But how many Ethernet Packets can be sent
over 10 Gbps.
What is the maximum number of packets that can be sent? Of course it is variable and a function of
the size of the packets. As one might
expect, smaller packets can achieve higher rates while bigger packets fill the
pipe more quickly. I am using 10 Gigabit
Ethernet as an example but the principles apply to every physical layer
technology discussed in this book.
How are pps and bps related? Well there is a mathematical equation that
allows anyone to calculate how many packets can be sent using a specified bps
and how many bits on the wire will be created by a given pps. Here is the equation:
pps x packet size (bytes) x 8
= bps
Now there
are some caveats and assumptions in this mathematical relationship. The number one key to this equation is that
all the packets have to be the same size for this equation to work. If the packet size is uniform the number of
packets can be multiplied by that size and multiplied by 8 to convert to bits
and will reflect a bit rate instead of a packet rate. Also the bps that is calculated reflect the number of bits on the wire that are in frames
(not all bits on the wire are always in frames). An additional equation multiplying the bits
that are not in frames per packet times the packet rate times 8 to convert to
bits. This can be considered layer 1
overhead bps and this can be added to the frame bps to reflect the total bits
on the wire.
The pps/bps equation will be referred to for each physical
layer medium discussed but there are subtle calculations that will modify it
for each technology.
1.2 OSI & TCP/IP Layer Models
Figure II.
OSI & TCP/IP Layer Models
Both
of these models have logical similarities, and the main functional difference
is that the TCP/IP model aggregates layer 5,6, and 7 into
a single “Application’ layer. For the
purpose of this reference, the OSI model will be used as the model of choice
for reasons of familiarity. When people
refer to ‘Layer 7’ it is clear the model is the OSI model and the layer is the
‘Application’ layer. When you refer to
‘Layer 5’ the distinction is not as clear and not automatically assumed to be
the ‘Application’ layer. You know you
have arrived as an accepted model to describe the flow of data through a
network when there are jokes and rules of thumb that refer to your model. One of the ‘rules of thumb’ is ‘If you don’t have layer 1, you have
bupkis’ or something along that line. This is a way of saying physical layer is
paramount and often networking problems can be traced to the cables. One of the common jokes is references to
layers 8, 9, & 10 as being the economic, political, & religious layers
of networking. Economic factors play
into network decision making, politics can green light or 86 a project, and as
for religion, the reference is about zealot topics (MAC vs. Windows vs. Linux).
1.3
Organization
This
paper addresses how IP packets are encapsulated and transported using various common
Layer 1 & 2 technologies. The
chapters are designed to focus on a specific technology or set of standards and
describe how that technology applies itself to the transport of IP packets. Vendor implementations are very diverse and
sometimes may have capabilities beyond the standard feature set for a given
physical layer technology.
IEEE 802.3 Ethernet
– Clearly this chapter focuses on the transport of IP packets over standard
Ethernet networks, one of the world’s most popular networking technologies. People use CAT5 UTP with RJ-45 mod plugs
without even thinking about the operation of the interface they plug it into. This chapter covers the world’s most deployed
networking technology and how it is used to transport the world’s most popular
Layer 3 protocol. IP
over Ethernet.
ATM
– Don’t throw tomatoes. ATM is used to
send IP packets all the time. And to use
existing networks and SONET rings, ATM is established and continues to deliver
services reliably (most notably cell phone service).
Frame-Relay, PPP, &
HDLC – Frame-Relay networks are still found in
the deployed networks frequently and the encapsulation has similarities with
PPP & HDLC. The signaling is
completely different for each of these, but the encapsulations and overhead
incurred per packet are similar.
SONET and POS
– The world’s most popular long haul optical fiber technology
Chapter
8: Legacy Technologies, FDDI, ISDN, Token-Ring, X.25 – FDDI was one of the
first fiber technologies, ISDN is still used all over the world, Token-Ring is
primarily a LAN technology developed by IBM to compete with Ethernet, and X.25
was one of the first modern telecom technologies developed.
Chapter
9: Tunneling / Encryption, GRE, L2TP, MPLS, VLAN, IPSec, SSL
Chapter
10: Layer 3/4 Overhead, IP, IPv6, UDP, RTP, TCP
2 IEEE 802.3
Ethernet
Table
1.1 – Common Ethernet Speeds and Feeds
|
Name |
Connector |
Speed |
Distance |
|
10BASE-2 |
AUI |
10 Mbps |
500m |
|
10BASE-5 |
BNC |
10 Mbps |
200m |
|
10BASE-T |
RJ-45 |
10 Mbps |
100m |
|
100BASE-TX |
RJ-45 |
100 Mbps |
100m |
|
100BASE-FX |
ST, SC,
LC |
100 Mbps |
2000m |
|
1000BASE-T |
RJ-45 |
1 Gbps |
100m |
|
1000BASE-X |
ST, SC,
LC |
1 Gbps |
2000m |
|
10GBASE-X |
ST, SC,
LC |
10 Gbps |
2000m |
Ethernet
won. Ethernet interfaces are the number
one installed network interface on PCs, servers, and Unix Workstations. Ethernet has progressed from 10base2,
10base5, 10baseT with AUI, BNC, and RJ-45 connectors to 100baseTX and
100baseFX, 1000baseX, and 10000baseX.
For network administrators who supported BNC Ethernet networks, it is
easy to understand why they have been preempted by CAT5. For anyone who has not had the pleasure of
troubleshooting the ‘Christmas-light Topology’ that is a BNC Ethernet network,
the issue was that the network was a string of connected systems with 50Ω ohm terminators on each end. When there is an un-terminated break in the
string, the noise on the wire makes it impossible for any communication. So clever support technicians would carry
around a pocket of BNC T connectors so when the call came in that the
secretaries cannot print (an indication that the network is down), they would
validate the outage, and then ask who had moved offices to locate the most
likely source of the disconnect. The
likelihood was that the PC that was removed was taken with the BNC T connector
leaving two disconnected ends, and inserting a connector would resolve the
outage. CAT5 with RJ-45 modular
connectors links uses a star topology so if a station is disconnected from the
concentrator, hub, or switch, it only impacts that one station and none of the
others. That is a more fault tolerant
network architecture. That is why BNC
Ethernet LANs are not widely used anymore.
And it demonstrates how far Ethernet has come as a technology.
Now one thing that is interesting about IEEE 802.3
Ethernet is that the whether it is running over BNC, Fiber, at 10 Mbps, or 1 Gbps, the construction of the Ethernet frames is the
same. And while there are multiple
formats for the Ethernet header (802.2, 802.3, etc…), only Ethernet II is used to
encapsulate IP Packets. Refer to figure
1.1 for a graphic representation of the Ethernet II frame format. This is the format used to transport IP
packets over Ethernet networks.
Figure 1.1 – Ethernet II Frame Format
The numbers reflect the number of bytes used for
each field. Feel free to search the
Internet and most sites and standards documents will show this frame format
diagram for Ethernet II without the ‘Preamble’ and ‘IFG (Inter-Frame Gap)’
referred to. However these fields are
bits on the wire that exist between every Ethernet packet. The minimum Inter-frame gap is 96 bit times
and the preamble is 8 bytes. That is 12
bytes plus 8 bytes per frame that is on the wire at Layer 1, but not part of
the Ethernet frames. Every frame incurs
these bits and must have them to be transmitted, but they are not in the frames. So they don’t count. But they are counted as part of the 100
million bits per second sent by 100BASE-TX.
Also they are part of Gigabit Ethernet and 10 Gigabit Ethernet.
Here is a sample captured Ethernet II IP Packet
decoded. It is a Server to Client HTTP
200 OK packet.
Frame 1 (214 bytes on wire,
214 bytes captured)
Arrival Time:
Packet Length: 214 bytes
Capture Length: 214 bytes
Protocols in frame: eth:ip:tcp:http:data
Ethernet II, Src: fe:ff:
Destination: Xerox_00:
Source: fe:ff:
Type: IP (0x0800)
Internet Protocol,
Src: 216.239.59.99
(216.239.59.99), Dst:
145.254.160.237 (145.254.160.237)
Transmission Control
Protocol,
Hypertext Transfer Protocol
Data (160 bytes)
2.1
CSMA/CD
Carrier Sense Multiple Access Collision Detect is
the mechanism employed by IEEE 802.3 Ethernet interfaces to share a common
medium (like in wireless and half-duplex coax wire). Stations defer for a quiet period and then
start sending frames in bit serial form.
If a collision occurs, both stations backoff
and wait a random period before retrying.
In full-duplex mode, stations can send and receive without collisions as
there is a dedicated pipe per direction.
The CSMA/CD algorithms, collisions, and backoff
can impact Ethernet performance as well as per packet overhead. This is mainly seen in wireless 802.11x
(a/b/g/n) networks today as they are shared media and the access point or
infrastructure node that links to a wired network uses a half duplex mechanism
per radio channel. Some wireless vendors
employ proprietary implementations that associate and connect client radios
with infrastructure access points using two channels and then use one as the
‘upstream’ and one as ‘downstream’ to create a pseudo full-duplex mode of
operation. These implantations are not
covered under the 802.11x standards but they are used to improve the performance
of the wireless clients.
3 ATM
3.1
Asynchronous Transfer Mode
ATM will be around for decades. While other networking technologies may
provide more efficiency for IP Transport (there is a “cell tax” for using ATM for
transporting multiprotocol PDU’s
using ATM cells), ATM continues to be popular.
However, ATM continues to be a highly popular choice by telecommunication
companies and service providers because of the ability to provision Peak Cell
Rates (PCR) to specific Virtual Circuits (VCs) and provide bandwidth
guarantees. And for Voice and Video
Services, and specifically cell phones, the service quality of ATM makes up for
the cost of using it to transport IP packets.
In short, ATM continues to be the WAN technology
of choice for telecommunication companies looking to build out new service
areas. Now a major function of ATM
backbones is carrying IP packets. Having
worked for major telecommunications service providers, it was a commonly known
fact that IP transport represented 80% of the costs, but only 20% of the
revenues. ATM may only use 20% of the
bandwidth for Voice and Video, but as converse to the data, clearly that
portion would represent 80% of the revenues.
That is not to say ATM does not have engineering
and provisioning challenges that make IP backbones very tempting. Especially as IP services for Voice, Video,
and Data become more functional, fault tolerant, and usable, IP networks become
more practical and offer advantages. The
traditional issue with ATM networks is that VCs are unidirectional and for a
fully functional network, every ATM switch needs a VC to every other
switch. And often these switches are
configured with static VCs. This leads
to a provisioning nightmare as the number of network nodes grows. For each switch added to a network, the
number of VCs required to be configured equates to “N * (N-1)” VCs. Also, the VCs pointing at the new switch from
the other switches need to be added. This
approaches a tedious level of configuration and one mistake can leave a network
node disconnected from the network. And
a node failure is a major problem, especially if a node is acting as an interconnect. The
advantage of IP is dynamic routing protocols that allow routing to occur and
the network can ‘self-heal’ and find alternate paths to a destination if there
is one when a node fails.
And the biggest problem with IP Networks is the
default behavior of IP routers to use First-In,
First-Out (FIFO) queuing and, in the event of congestion, IP is best
effort. In other words, there is no
guaranteed delivery, and packets can be dropped. Often engineers fall back on the mechanisms of
higher layer protocols like TCP to resend dropped packets. If the sender does not receive an
acknowledgement packet (ACK), it resends the packet. Now this is fine for TCP based applications
like HTTP or email (SMTP). However,
Voice and Video are loss sensitive and resending lost frames does nothing to
make up for a gap in service (caused by lost packets). So Voice and Video and streaming media
typically use UDP and sometimes RTP for sending data. If Voice and Video frames are lost, service
quality degrades. So the challenges
facing IP deployments are how to deal with congestion and maintain service
quality for Voice and Video. Many
Service Providers deal with congestion by throwing bandwidth at the problem and
over-provisioning so there is no congestion.
New services like tunneling and MPLS seek to provide end-to-end service
quality and some of the advantages of ATM to the world of IP Networks. Part of this is to use different queuing
algorithms to preserve Voice and Video quality when congestion does occur. This is a real challenge when legacy IP
routing gear is considered and just one congested link using FIFO can trash
Voice and Video services. Period. So ATM
endures.
IP is growing and the ‘IP over everything’ mantra
will continue to gather momentum. But it
does not stand alone in the world of networking protocols.
ATM Signaling
Discussion of SVCs.
ATM cell header
VPI
– Virtual Path Identifier
VCI
– Virtual Channel Identifier
ATM Adaptation Layer 5 (AAL5)
AAL5
is the format ATM uses for sending IP packets.
ATM uses fixed size cells and IP packets are almost always bigger that
the 48 bytes available in a single cell.
So most IP packets require multiple cells to be
transported. And the process of
slicing the packets into cells and then reassembling them on the other side of
an ATM link is called, appropriately, Segmentation and Reassembly (SAR).
Figure 2.1
3.2
Case Study: ATM / Ethernet Comparison
The graph that was created in this case study was
designed to display the differences in bandwidth usage between Ethernet and
ATM. The blue and the pink data points
represent results from the same tests.
The pink display the bps load on the ATM interface, and the blue
represents the load on the GigEthernet interface in
the same test iterations. In this case,
the ATM AAL5 encapsulation used was VCMUX.
Here is an example of a real world test of an ATM
OC-3 being tested using GigEthernet ports to saturate
the interfaces:
Figure 2.2
Note that the ATM Interface usually is at or near
100% performance of the 149,760,000 bps that is it’s maximun capacity. So
the ATM load is usually in the high 148 Mbps to the low 149 Mbps range. Whereas the Ethernet load
ranges from 110 Mbps to 171 Mbps.
That is quite a sweeping difference.
And it has to do with how the IP packets are inserted into ATM cells
using the AAL5 PDUs described earlier. The packet sizes selected where designed to
hit the sweetest and the meanest spots for fitting packets in ATM Cells.
Take the example of the 106 byte and 107 byte IP Ethernet
packets (Refer to Table 2.2). A 106 byte
Ethernet packet will have the 18 byte IEEE Ethernet header removed when it
arrives at the networking device. That
leaves a 88 byte IP payload to be placed on the ATM
wire. As this example used VCMUX to
create AAL5 PDUs, only an 8 byte trailer is added to
the 88 byte IP payload, creating a 96 byte PDU.
By design, 96 bytes fits perfectly into 2 ATM cells without a single
byte wasted in cell padding. So even
though the ATM SAR mechanism wasn’t quite able to segment and reassemble
packets at this size
at 100% theoretical capacity (143.8 Mbps of 149.76 Mbps
possible), a massive 171 Mbps load was carried on the Ethernet wire in the same
instance.
In this rare instance, ATM was far more efficient
than Ethernet at carrying the same number of packets. ATM used over 27 Mbps less bandwidth to
transfer the same number of 106 byte IP Ethernet packets.
Ah, but then there is the 107 byte IP Ethernet
packet. In the next case a single byte
is added to the size of the packet, the whole comparison is flipped 180
degrees. With the 107 byte IP Ethernet
packet, the same 18 bytes of Ethernet overhead is discarded and this time an 89
byte IP payload remains. The same 8 byte
VCMux trailer is added creating a 97 byte PDU. Also by design, this barely misses being able
to fit into two ATM cells and will require three ATM cells to be
transported. Note that more data was
actually able to be placed on the ATM wire (149.64 Mbps). That is 99.92% of theoretical line rate of
149.76 Mbps (OC-3 without SONET Overhead).
So the ATM interface carried about as much as it theoretically could
carry. But the load on the Ethernet wire
is 119.52 Mbps in this instance. So in
this case the Ethernet required 30 Mbps less than the ATM interface to carry
the same number of packets. Naturally this is because the 107 byte IP Ethernet
packet was 1 byte too big for two cells so all but 1 byte of the 3rd
cell was wasted (52 or the 53 bytes).
That is quite a role reversal. To go from ATM being 27
Mbps more efficient than Ethernet to Ethernet being 30 Mbps more efficient than
ATM with a difference of a single byte in the IP payload size. Now the 52 bytes that is wasted by adding the
extra cell between the most efficient and least efficient IP payload sizes is
constant, and as the packets get larger this extra 52 bytes incurred on ATM
makes less and less of a difference. Certainly the difference in performance
between a 106 byte IP Ethernet packets (171 Mbps) and 107 byte IP Ethernet
packets (119.54 Mbps) is more dramatic than the difference between 1498 and
1499 byte IP Ethernet packets which are also selected to perfectly fit into 31
cells, and then to just miss 31 and require a 32nd cell to be
transmitted. The 1498 byte IP Ethernet
performance was 138.5 Mbps and the 1499 byte IP packet achieved 134 mbps. In both cases the ATM load was right around
149.76 Mbps. But the difference in
Ethernet performance is less than 5 Mbps (compared to the 50 Mbps difference at
the smaller packet size).
Obviously IP packets have variable length payloads
and they are created by applications that rarely consider how well that packet
will fit into ATM cells. But it is fair
to say in looking at the comparison between the Ethernet load and the ATM load
in the same test instances (Figure 1.2) that Ethernet tends to be more
efficient than ATM (unless you are using some kind of VOIP application that
creates nothing but 202 byte IP Ethernet packets). That is fairly unlikely.
Table 2.2 Sample
Data points of Table that created the XY plot displayed in Figure 2.2
|
ATM AAL5 VCMUX
Encapsulation |
|
|
|
||||
|
|
|
|
|
PPS x Packet Size x 8
= bps |
|
||
|
Ethernet IP Packet
Size |
# ATM Cells |
PPS |
Cells |
ATM bps |
Line Rate |
% Line Rate Observed |
Ethernet bps |
|
64 |
2 |
163690 |
327380 |
138809120 |
149760000 |
92.68771368 |
109,999,680 |
|
106 |
2 |
169643 |
339286 |
143857264 |
149760000 |
96.05853632 |
171,000,144 |
|
107 |
3 |
117642 |
352926 |
149640624 |
149760000 |
99.92028846 |
119,524,272 |
|
154 |
3 |
117098 |
351294 |
148948656 |
149760000 |
99.45823718 |
163,000,416 |
|
155 |
4 |
87857 |
351428 |
149005472 |
149760000 |
99.49617521 |
122,999,800 |
|
202 |
4 |
88401 |
|||||