[afnog] emergency packet drops

ibtisam jamal ibty.jamal at gmail.com
Wed Apr 11 11:36:48 UTC 2012


On Wed, Apr 11, 2012 at 1:25 PM, ibtisam jamal <ibty.jamal at gmail.com> wrote:

> Hi ,
>
> Please see below.Kernel ,cfeb and pfe .Please help understand what they
> mean.link issues on ospf are explainable.
>

Apr 10 22:46:35  DSSR01 /kernel: RT_PFE: NH IPC op 30 (GET NH BULK
STATS) failed, err 1 (Unknown) peer_class 0, peer_index 0 peer_type 1
Apr 10 22:46:35  DSSR01 cfeb NH: invalid nh (2506) for bulk statistics
Apr 10 22:46:35  DSSR01 /kernel: RT_PFE: NH IPC op 30 (GET NH BULK
STATS) failed, err 1 (Unknown) peer_class 0, peer_index 0 peer_type 1
Apr 10 22:46:35  DSSR01 cfeb NH: invalid nh (1263) for bulk statistics
Apr 10 22:46:47  DSSR01 cfeb NH: invalid nh (1463) for bulk statistics
Apr 10 22:46:47  DSSR01 /kernel: RT_PFE: NH IPC op 30 (GET NH BULK
STATS) failed, err 1 (Unknown) peer_class 0, peer_index 0 peer_type 1
Apr 10 22:46:57  DSSR01 rpd[1009]: RPD_RSVP_NBRDOWN: RSVP neighbor
10.87.2.86 down on interface ge-1/2/3.222, neighbor seq number change
Apr 10 22:46:58  DSSR01 /kernel: RT_PFE: NH IPC op 30 (GET NH BULK
STATS) failed, err 1 (Unknown) peer_class 0, peer_index 0 peer_type 1
Apr 10 22:46:58  DSSR01 cfeb NH: invalid nh (1205) for bulk statistics

 show chassis routing-engine
Routing Engine status:
  Slot 0:
    Current state                  Master
    Election priority              Master (default)
    Temperature                 29 degrees C / 84 degrees F
    CPU temperature             33 degrees C / 91 degrees F
    DRAM                      1536 MB
    Memory utilization          28 percent
    CPU utilization:
      User                       2 percent
      Background                 0 percent
      Kernel                     6 percent
      Interrupt                  2 percent
      Idle                      91 percent
    Model                          RE-850
    Serial ID                      9009054782
    Start time                     2012-04-11 00:38:34 EAT
    Uptime                         13 hours, 43 minutes, 37 seconds
    Last reboot reason             Router rebooted after a normal shutdown.
    Load averages:                 1 minute   5 minute  15 minute
                                       0.06       0.12       0.08


 show chassis hardware
Hardware inventory:
Item             Version  Part number  Serial number     Description
Chassis                                A7768             M10i
Midplane         REV 06   710-008920   DC0582            M10i Midplane
Power Supply 0   Rev 04   740-008985   TA50329           DC Power Supply
Power Supply 1   Rev 04   740-008985   TB52644           DC Power Supply
HCM 0            REV 04   710-010580   DC0794            M10i HCM
HCM 1            REV 04   710-010580   DC0674            M10i HCM
Routing Engine 0 REV 08   740-011202   9009054782        RE-850
CFEB 0           REV 09   750-010465   DE0210            Internet Processor II
FPC 0                                                    E-FPC
  PIC 0          REV 15   750-012838   AACY5014          4x 1GE(LAN), IQ2
    Xcvr 0                NON-JNPR     PCE4K0Y           SFP-T
  PIC 1          REV 08   750-003873   CJ4213            4x E1, RJ48
  PIC 2          REV 17   750-007626   PY7711            10x CHE1 IQ
  PIC 3          REV 12   750-002992   RD9666            4x F/E, 100 BASE-TX
FPC 1                                                    E-FPC
  PIC 0          REV 15   750-012838   AACY5016          4x 1GE(LAN), IQ2
    Xcvr 0       REV 02   740-011613   PJC2N4W           SFP-SX
    Xcvr 1       REV 02   740-011613   PJC2MPN           SFP-SX
    Xcvr 2       REV 02   740-011613   PJ95B7L           SFP-SX
  PIC 1          REV 11   750-005723   RE2070            2x OC-3 ATM-II IQ, SMIR
  PIC 2          REV 15   750-012838   EC7896            4x 1GE(LAN), IQ2
    Xcvr 1       REV 02   740-011613   PJC2N1X           SFP-SX
    Xcvr 3       REV 02   740-013111   9111048           SFP-T
  PIC 3          REV 16   750-002992   AAAF1194          4x F/E, 100 BASE-TX
Fan Tray 1                                               Rear Right Fan Tray


 show pfe statistics traffic
Packet Forwarding Engine traffic statistics:
    Input  packets:          13621598548               432011 pps
    Output packets:          13606096805               431679 pps
Packet Forwarding Engine local traffic statistics:
    Local packets input                 :             20255431
    Local packets output                :              9922079
    Software input control plane drops  :                    0
    Software input high drops           :                    0
    Software input medium drops         :                    0
    Software input low drops            :                    0
    Software output drops               :                    0
    Hardware input drops                :                    0
Packet Forwarding Engine local protocol statistics:
    HDLC keepalives            :                    0
    ATM OAM                    :              2169664
    Frame Relay LMI            :                    0
    PPP LCP/NCP                :                    0
    OSPF hello                 :               314090
    OSPF3 hello                :                    0
    RSVP hello                 :                40872
    LDP hello                  :                    0
    BFD                        :              1056530
    IS-IS IIH                  :                    0
    LACP                       :                    0
    ARP                        :             11747630
    ETHER OAM                  :                    0
    Unknown                    :                  253
Packet Forwarding Engine hardware discard statistics:
    Timeout                    :                    0
    Truncated key              :                    0
    Bits to test               :                    0
    Data error                 :                    0
    Stack underflow            :                    0
    Stack overflow             :                    0
    Normal discard             :              4009442
    Extended discard           :                    0
    Invalid interface          :                    0
    Info cell drops            :                    0
    Fabric drops               :                    0
Packet Forwarding Engine Input IPv4 Header Checksum Error and Output
MTU Error statistics:
    Input Checksum             :          21474936985
    Output MTU                 :                    0

 show chassis cfeb
CFEB status:
Slot 0 information:
  State                                 Master
  Intake temperature                 27 degrees C / 80 degrees F
  Exhaust temperature                35 degrees C / 95 degrees F
  CPU utilization                    15 percent
  Interrupt utilization               5 percent
  Heap utilization                   15 percent
  Buffer utilization                 25 percent
  Total CPU DRAM                    128 MB
  Internet Processor II                 Version 2, Foundry IBM, Part number 164
  Start time:                           2012-04-11 00:41:42 EAT
  Uptime:                               13 hours, 43 minutes, 47 seconds
Slot 1 information:
  State                                 Empty
 show route summary | match bgp
                 BGP:     17 routes,      8 active
                 BGP:      2 routes,      1 active
                 BGP:      4 routes,      4 active
                 BGP:      8 routes,      3 active
                 BGP:     10 routes,      4 active
                 BGP:    331 routes,    211 active
                 BGP:     64 routes,     30 active
                 BGP:     52 routes,     30 active


The bgp routes are not as many and cfeb heap utilization is not high .
But when doing a p2p ping from all the ports on different PIC's we
have packet loss but other router connected to it is not having these
losses.

> ping 10.87.8.190 rapid count 1000 size 1472
PING 10.87.8.190 (10.87.8.190): 1472 data bytes
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!..!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!.!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!
--- 10.87.8.190 ping statistics ---
1000 packets transmitted, 956 packets received, 4% packet loss
round-trip min/avg/max/stddev = 12.889/15.294/69.974/6.946 ms

 ping rapid count 10.87.8.99
                                 ^
invalid trailing data '.87.8.99' for numeric value: '10.87.8.99' at '10.87.8.99'
 ping 10.87.8.99 rapid count 1000 size 1472
PING 10.87.8.99 (10.87.8.99): 1472 data bytes
!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!.!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!.!!!!!!!!!
--- 10.87.8.99 ping statistics ---
1000 packets transmitted, 939 packets received, 6% packet loss
round-trip min/avg/max/stddev = 7.646/9.993/88.558/8.079 ms






>
> Regards,
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://afnog.org/pipermail/afnog/attachments/20120411/998d5124/attachment-0001.html>


More information about the afnog mailing list