speed-capabilities: Added Speed Capabilities test case

  From: Daniel Kirichok <dkirichok@iol.unh.edu>

Added test case for checking speed capabilities.

Signed-off-by: Daniel Kirichok <dkirichok@iol.unh.edu>
---
 conf/speed_capabilities.cfg                 |   4 +
 test_plans/speed_capabilities_test_plan.rst |  84 ++++++++++++++
 tests/TestSuite_speed_capabilities.py       | 120 ++++++++++++++++++++
 3 files changed, 208 insertions(+)
 create mode 100644 conf/speed_capabilities.cfg
 create mode 100644 test_plans/speed_capabilities_test_plan.rst
 create mode 100644 tests/TestSuite_speed_capabilities.py
  

Hi Daniel,

29.06.2020 23:28, dkirichok@iol.unh.edu пишет:
> From: Daniel Kirichok <dkirichok@iol.unh.edu>
>
> Added test case for checking speed capabilities.
>
> Signed-off-by: Daniel Kirichok <dkirichok@iol.unh.edu>
> ---
>   conf/speed_capabilities.cfg                 |   4 +

I would prefer separate file for every supported device. i.e.  
conf/i40e_speed_capabilities.cfg, conf/mlx5_speed_capabilities.cfg etc.

and one test setup config with device:capabilities mapping. e.g.

conf/speed_capabilities_setup.cfg

     eth0 = i40e

     eth1 = mlx5

>   test_plans/speed_capabilities_test_plan.rst |  84 ++++++++++++++
>   tests/TestSuite_speed_capabilities.py       | 120 ++++++++++++++++++++
>   3 files changed, 208 insertions(+)
>   create mode 100644 conf/speed_capabilities.cfg
>   create mode 100644 test_plans/speed_capabilities_test_plan.rst
>   create mode 100644 tests/TestSuite_speed_capabilities.py
>
> diff --git a/conf/speed_capabilities.cfg b/conf/speed_capabilities.cfg
> new file mode 100644
> index 0000000..c22f531
> --- /dev/null
> +++ b/conf/speed_capabilities.cfg
> @@ -0,0 +1,4 @@
> +[suite]
> +# The format for entering expected speeds is {'<INTERFACE>':'<EXPECTED SPEED>',...}
speed capability is not a value of speed. It's set of values. format 
should be something like that: {'<INTERFACE>':['<EXPECTED SPEED1>', 
'<EXPECTED SPEED2>'], ...}
> +# Unit for speed must be in G or M
> +expected_speeds={'interface1': '10G', 'interface2': '100M'}
> diff --git a/test_plans/speed_capabilities_test_plan.rst b/test_plans/speed_capabilities_test_plan.rst
> new file mode 100644
> index 0000000..2821612
> --- /dev/null
> +++ b/test_plans/speed_capabilities_test_plan.rst
> @@ -0,0 +1,84 @@
> +.. # BSD LICENSE
> +    #
> +    # Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
> +    # Copyright © 2018[, 2019] The University of New Hampshire. All rights reserved.
> +    # All rights reserved.
> +    #
> +    # Redistribution and use in source and binary forms, with or without
> +    # modification, are permitted provided that the following conditions
> +    # are met:
> +    #
> +    #   * Redistributions of source code must retain the above copyright
> +    #     notice, this list of conditions and the following disclaimer.
> +    #   * Redistributions in binary form must reproduce the above copyright
> +    #     notice, this list of conditions and the following disclaimer in
> +    #     the documentation and/or other materials provided with the
> +    #     distribution.
> +    #   * Neither the name of Intel Corporation nor the names of its
> +    #     contributors may be used to endorse or promote products derived
> +    #     from this software without specific prior written permission.
> +    #
> +    # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
> +    # "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
> +    # LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> +    # A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
> +    # OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
> +    # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
> +    # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
> +    # DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
> +    # THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> +    # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
> +    # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
> +
> +=======================
> +Speed Capabilities Test
> +=======================
> +
> +The Speed Capabilities Test checks the speeds that the device lists and compares
> +it to a user-defined set of expected speeds set in the ``speed_capabilities.cfg``.
> +
> +Prerequisites
> +=============
> +
> +If using vfio the kernel must be >= 3.6+ and VT-d must be enabled in bios.When
> +using vfio, use the following commands to load the vfio driver and bind it
> +to the device under test::
> +
> +   modprobe vfio
> +   modprobe vfio-pci
> +   usertools/dpdk-devbind.py --bind=vfio-pci device_bus_id
> +
> +Assuming that ports ``0`` and ``1`` of the test target are directly connected
> +to the traffic generator, launch the ``testpmd`` application with the following
> +arguments::
> +
> +  ./build/app/testpmd -- -i --portmask=0x3
> +
> +Start packet forwarding in the ``testpmd`` application with the ``start``
> +command. Then, for each port on the target make the Traffic Generator
> +transmit a packet to the port of arbitrary size less than the MTU of
> +the target port, checking that the same amount of frames and bytes
> +are received back by the Traffic Generator from the port.
> +
> +Ensure that the ``speed_capabilities.cfg`` file correctly reflects the speeds
> +the device is expected to run at.
> +
> +
> +Test Case : Speed Capabilities Test
> +===================================
> +
> +1. Use testpmd to retrieve the speed information that the interface is linked at::
> +
> +      testpmd> show port info <PORT_ID>

I'm afraid that testpmd don't show speed_capa value. It shows only 
link_speed value. to get speed_capa values you should write new 
ethtool-like application for dpdk interfaces.

Note that speed_capa is bitmap value and it's part of rte_eth_dev_info 
struct.

  269  * Device supported speeds bitmap flags
  270  */
  271 #define ETH_LINK_SPEED_AUTONEG  (0 <<  0)  /**< Autonegotiate (all 
speeds) */
  272 #define ETH_LINK_SPEED_FIXED    (1 <<  0)  /**< Disable autoneg 
(fixed speed) */
  273 #define ETH_LINK_SPEED_10M_HD   (1 <<  1)  /**< 10 Mbps half-duplex */
  274 #define ETH_LINK_SPEED_10M      (1 <<  2)  /**< 10 Mbps full-duplex */
  275 #define ETH_LINK_SPEED_100M_HD  (1 <<  3)  /**< 100 Mbps 
half-duplex */
  276 #define ETH_LINK_SPEED_100M     (1 <<  4)  /**< 100 Mbps 
full-duplex */
  277 #define ETH_LINK_SPEED_1G       (1 <<  5)  /**< 1 Gbps */
  278 #define ETH_LINK_SPEED_2_5G     (1 <<  6)  /**< 2.5 Gbps */
  279 #define ETH_LINK_SPEED_5G       (1 <<  7)  /**< 5 Gbps */
  280 #define ETH_LINK_SPEED_10G      (1 <<  8)  /**< 10 Gbps */
  281 #define ETH_LINK_SPEED_20G      (1 <<  9)  /**< 20 Gbps */
  282 #define ETH_LINK_SPEED_25G      (1 << 10)  /**< 25 Gbps */
  283 #define ETH_LINK_SPEED_40G      (1 << 11)  /**< 40 Gbps */
  284 #define ETH_LINK_SPEED_50G      (1 << 12)  /**< 50 Gbps */
  285 #define ETH_LINK_SPEED_56G      (1 << 13)  /**< 56 Gbps */
  286 #define ETH_LINK_SPEED_100G     (1 << 14)  /**< 100 Gbps */
  287 #define ETH_LINK_SPEED_200G     (1 << 15)  /**< 200 Gbps */

but link_speed is speed values. it's part of rte_eth_link struct.

  289 /**
  290  * Ethernet numeric link speeds in Mbps
  291  */
  292 #define ETH_SPEED_NUM_NONE         0 /**< Not defined */
  293 #define ETH_SPEED_NUM_10M         10 /**<  10 Mbps */
  294 #define ETH_SPEED_NUM_100M       100 /**< 100 Mbps */
  295 #define ETH_SPEED_NUM_1G        1000 /**<   1 Gbps */
  296 #define ETH_SPEED_NUM_2_5G      2500 /**< 2.5 Gbps */
  297 #define ETH_SPEED_NUM_5G        5000 /**<   5 Gbps */
  298 #define ETH_SPEED_NUM_10G      10000 /**<  10 Gbps */
  299 #define ETH_SPEED_NUM_20G      20000 /**<  20 Gbps */
  300 #define ETH_SPEED_NUM_25G      25000 /**<  25 Gbps */
  301 #define ETH_SPEED_NUM_40G      40000 /**<  40 Gbps */
  302 #define ETH_SPEED_NUM_50G      50000 /**<  50 Gbps */
  303 #define ETH_SPEED_NUM_56G      56000 /**<  56 Gbps */
  304 #define ETH_SPEED_NUM_100G    100000 /**< 100 Gbps */
  305 #define ETH_SPEED_NUM_200G    200000 /**< 200 Gbps */
  306 #define ETH_SPEED_NUM_UNKNOWN UINT32_MAX /**< Unknown */

Best regards,

Ivan

> +
> +2. Compares the speed value against the rate in the speed_capabilities.cfg file.
> +   The supported options for expected speeds are 1G, 10G, 25G, 40G, and 100G. Each interface must have an expected speed associated with it.
> +   Example file below::
> +
> +      [suite]
> +      expected_speeds={'interface1': '10G', 'interface2': '100M'}
> +
> +
> +3. Verifies that the speeds matches accordingly.
> +
> +4. Repeats the verification for each NIC and interface found on the system.
> diff --git a/tests/TestSuite_speed_capabilities.py b/tests/TestSuite_speed_capabilities.py
> new file mode 100644
> index 0000000..25c45e4
> --- /dev/null
> +++ b/tests/TestSuite_speed_capabilities.py
> @@ -0,0 +1,120 @@
> +# BSD LICENSE
> +#
> +# Copyright(c) 2010-2014 Intel Corporation. All rights reserved.
> +# Copyright © 2018[, 2019] The University of New Hampshire. All rights reserved.
> +# All rights reserved.
> +#
> +# Redistribution and use in source and binary forms, with or without
> +# modification, are permitted provided that the following conditions
> +# are met:
> +#
> +#   * Redistributions of source code must retain the above copyright
> +#     notice, this list of conditions and the following disclaimer.
> +#   * Redistributions in binary form must reproduce the above copyright
> +#     notice, this list of conditions and the following disclaimer in
> +#     the documentation and/or other materials provided with the
> +#     distribution.
> +#   * Neither the name of Intel Corporation nor the names of its
> +#     contributors may be used to endorse or promote products derived
> +#     from this software without specific prior written permission.
> +#
> +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
> +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
> +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
> +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
> +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
> +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
> +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
> +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
> +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
> +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
> +
> +"""
> +DPDK Test suite.
> +"""
> +import utils
> +from pmd_output import PmdOutput
> +from test_case import TestCase
> +
> +
> +class TestSpeedCapabilities(TestCase):
> +    def set_up_all(self):
> +        """
> +        Run at the start of each test suite.
> +        """
> +        self.ports = self.dut.get_ports()
> +        self.ports_socket = self.dut.get_numa_id(self.ports[0])
> +
> +        for port in self.ports:
> +            self.tester.send_expect(f"ifconfig {self.tester.get_interface(self.tester.get_local_port(port))} mtu 5000"
> +                                    , "# ")
> +
> +        self.pmdout = PmdOutput(self.dut)
> +        self.vm_env_done = False
> +
> +    def test_speed_capabilities(self):
> +        self.pmdout.start_testpmd("Default")
> +
> +        cfg_content = self.get_suite_cfg()
> +        expected_speeds = cfg_content.get('expected_speeds')
> +
> +        detected_interfaces = []
> +
> +        for port in self.ports:
> +            interface_name = self.tester.get_interface(self.tester.get_local_port(port))
> +
> +            # Gives the speed in Mb/s
> +            interface_speed = self.pmdout.get_port_link_speed(port)
> +
> +            self.verify(interface_name in expected_speeds, f"The interface {interface_name} does not have an expected "
> +                                                           f"speed associated with it.")
> +
> +            detected_interfaces.append(interface_name)
> +
> +            expected_speed = expected_speeds[interface_name]
> +
> +            # Isolates the unit (Either M or G)
> +            expected_speed_unit = ''.join(i for i in expected_speed if not i.isdigit())
> +
> +            # Removes the unit from the speed
> +            expected_speed = ''.join(i for i in expected_speed if i.isdigit())
> +
> +            self.verify(len(interface_speed) > 0,
> +                        f"A valid speed could not be read for the interface {interface_name}.")
> +
> +            # Converts Gb/s to Mb/s for consistent comparison
> +            if expected_speed_unit == "G":
> +                expected_speed += "000"
> +
> +            self.verify(interface_speed == expected_speed,
> +                        f"Detected speed: {interface_speed} Mb/s for the interface {interface_name}, "
> +                        f"but expected speed: {expected_speed} Mb/s")
> +
> +        for key, value in expected_speeds.items():
> +            self.verify(key in detected_interfaces, f"The interface {key} expected the speed {value} in "
> +                                                    "speed_capabilities.cfg file, but it did not detect that interface.")
> +
> +    def tear_down(self):
> +        """
> +        Run after each test case.
> +        """
> +        self.dut.kill_all()
> +        self.pmdout.start_testpmd("Default", "--portmask=%s --port-topology=loop" % utils.create_mask(self.ports),
> +                                  socket=self.ports_socket)
> +        ports_num = len(self.ports)
> +        # link up test, to avoid failing further tests if link was down
> +        for i in range(ports_num):
> +            # sometimes output text messing up testpmd prompt so trimmed prompt
> +            self.dut.send_expect("set link-up port %d" % i, ">")
> +        # start ports, to avoid failing further tests if ports are stopped
> +        self.dut.send_expect("port start all", "testpmd> ", 100)
> +        self.dut.send_expect("quit", "# ")
> +
> +    def tear_down_all(self):
> +        """
> +        Run after each test suite.
> +        """
> +        if self.vm_env_done:
> +            self.destroy_vm_env()
> +        self.dut.kill_all()
  

> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ivan Dyukov
> Sent: Tuesday, June 30, 2020 9:03 AM

[...]

> > +Test Case : Speed Capabilities Test
> > +===================================
> > +
> > +1. Use testpmd to retrieve the speed information that the interface
> is linked at::
> > +
> > +      testpmd> show port info <PORT_ID>
> 
> I'm afraid that testpmd don't show speed_capa value. It shows only
> link_speed value. to get speed_capa values you should write new
> ethtool-like application for dpdk interfaces.

That would probably need to be a secondary process, hooking up to the testpmd process used for testing. Not the simplest solution.

If testpmd is the primary tool used for testing PMDs, it should support all features being tested. In other words: Testpmd needs to be updated, so it also shows speed_capa.

I also considered ethtool, but it is only listed under Other Example Applications, and has no maintainers. So that does not seem to have a promising future.

CC: testpmd Maintainers.
  

Inline

On Tue, Jun 30, 2020 at 3:27 AM Morten Brørup <mb@smartsharesystems.com>
wrote:

> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ivan Dyukov
> > Sent: Tuesday, June 30, 2020 9:03 AM
>
> [...]
>
> > > +Test Case : Speed Capabilities Test
> > > +===================================
> > > +
> > > +1. Use testpmd to retrieve the speed information that the interface
> > is linked at::
> > > +
> > > +      testpmd> show port info <PORT_ID>
> >
> > I'm afraid that testpmd don't show speed_capa value. It shows only
> > link_speed value. to get speed_capa values you should write new
> > ethtool-like application for dpdk interfaces.
>
> That would probably need to be a secondary process, hooking up to the
> testpmd process used for testing. Not the simplest solution.
>
> If testpmd is the primary tool used for testing PMDs, it should support
> all features being tested. In other words: Testpmd needs to be updated, so
> it also shows speed_capa.
>
> I also considered ethtool, but it is only listed under Other Example
> Applications, and has no maintainers. So that does not seem to have a
> promising future.
>
> CC: testpmd Maintainers.
>
> Dan > I have filed a bug report on bugzilla regarding testpmd being
updated for this. I will also look into incorporating the suggestion Ivan
made about the naming convention and organization for the config files.
  

Hi all,

I just wanted to follow up on the status of the patch. I was wondering if
there were plans to merge the patch before testpmd was updated with the
speed_capa change.

Thanks,
Dan

On Tue, Jun 30, 2020 at 3:59 PM Daniel Kirichok <dkirichok@iol.unh.edu>
wrote:

> Inline
>
> On Tue, Jun 30, 2020 at 3:27 AM Morten Brørup <mb@smartsharesystems.com>
> wrote:
>
>> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ivan Dyukov
>> > Sent: Tuesday, June 30, 2020 9:03 AM
>>
>> [...]
>>
>> > > +Test Case : Speed Capabilities Test
>> > > +===================================
>> > > +
>> > > +1. Use testpmd to retrieve the speed information that the interface
>> > is linked at::
>> > > +
>> > > +      testpmd> show port info <PORT_ID>
>> >
>> > I'm afraid that testpmd don't show speed_capa value. It shows only
>> > link_speed value. to get speed_capa values you should write new
>> > ethtool-like application for dpdk interfaces.
>>
>> That would probably need to be a secondary process, hooking up to the
>> testpmd process used for testing. Not the simplest solution.
>>
>> If testpmd is the primary tool used for testing PMDs, it should support
>> all features being tested. In other words: Testpmd needs to be updated, so
>> it also shows speed_capa.
>>
>> I also considered ethtool, but it is only listed under Other Example
>> Applications, and has no maintainers. So that does not seem to have a
>> promising future.
>>
>> CC: testpmd Maintainers.
>>
>> Dan > I have filed a bug report on bugzilla regarding testpmd being
> updated for this. I will also look into incorporating the suggestion Ivan
> made about the naming convention and organization for the config files.
>
>
> --
>
> Dan Kirichok
>
> UNH InterOperability Laboratory
>
> 21 Madbury Rd, Suite 100, Durham, NH 03824
>
> dkirichok@iol.unh.edu
>
> www.iol.unh.edu
>
>
>
>