@@ -187,6 +187,7 @@ The following are the test plans for the DPDK DTS automated test system.
power_branch_ratio_test_plan
power_empty_poll_test_plan
power_pbf_test_plan
+ power_pmd
power_pstate_test_plan
power_telemetry_test_plan
vmdq_test_plan
new file mode 100644
@@ -0,0 +1,204 @@
+.. SPDX-License-Identifier: BSD-3-Clause
+ Copyright(c) 2010-2020 Intel Corporation
+
+==============================
+PMD power management test plan
+==============================
+
+This test plan intends to test the PMD power management functionality by
+measuring total power consumption on the system with `turbostat`.
+
+Preparation Work
+================
+
+1. Tests require that your hardware does support WAITPKG instruction set
+ - to check if CPU supports WAITPKG, run the following command::
+
+ cat /proc/cpuinfo | grep waitpkg
+
+ if the output is not empty, the platform supports WAITPKG instructions.
+2. Ensure Hardware P-States (HWP) is enabled in the BIOS
+3. Ensure Intel(R) VT-d is enabled in the BIOS
+4. Ensure the kernel command-line includes "intel_iommu=on iommu=pt" parameters
+5. Ensure the VFIO PCI driver is available and loaded
+
+ - To check if VFIO PCI module is loaded, run the following command::
+
+ lsmod | grep vfio
+
+ The output should contain at least the following drivers: `vfio-pci`,
+ `vfio` and `vfio_iommu_type1`
+6. Ensure the `turbostat` tool is present on your platform, and reports average
+ core frequency as well as platform power consumption
+
+ - This can be checked by issuing the following command::
+
+ turbostat -l
+
+ and checking if the list of available columns has `PkgWatt`, `Core` (or
+ `CPU`), and `Bzy_MHz` columns
+7. Check if MSR driver is built-in or is loaded
+
+ - To check if the module is built-in, run the following command::
+
+ cat /usr/lib/modules/$(uname -r)/modules.builtin | grep msr.ko
+
+ if the output is not empty, the MSR module is built into the kernel, and
+ no further action is necessary.
+ - To check if the module is loaded, run the following command::
+
+ lsmod | grep msr
+
+ Non-empty output will indicate that MSR kernel module is loaded.
+ - If the MSR driver is not loaded but is available as a module, the
+ following command will load it::
+
+ modprobe msr
+
+8. At least one physical NIC port is required, and should be bound to the VFIO
+ PCI driver
+
+ - Use `dpdk-devbind.py` to bind physical NIC ports to VFIO PCI driver::
+
+ dpdk-devbind.py -b vfio-pci <PCI address 1> <PCI address 2> ...
+
+ - In a situation where number of logical CPU cores on a platform exceeds the
+ maximum number of queues a physical NIC port can have, use more NIC ports
+ in the same way (i.e. bind them to VFIO PCI driver)
+
+
+Test Case 1 : Test PMD power management in pstate mode
+======================================================
+Step 1. Reset all pstate min/max frequency assignments for all cores
+
+- This can be done by overwriting lowest and highest frequencies in the CPU
+ scaling driver::
+
+ # assume min/max frequencies are 1.2GHz/3.6GHz respectively
+ for d in /sys/bus/cpu/devices/cpu*/cpufreq/
+ do
+ cat $d/cpuinfo_min_freq > $d/scaling_min_freq
+ cat $d/cpuinfo_max_freq > $d/scaling_max_freq
+ done
+
+Step 2. Launch l3fwd-power and enable "scale" PMD power management mode, and
+assign each forwarding lcore (including main lcore) exactly one queue::
+
+ ./examples/dpdk-l3fwd-power -l 0-... -- --pmd-mgmt=scale -P -p 0x1 --config="(0,0,0),(0,1,1),(0,2,2),..."
+ Example:
+ ./examples/dpdk-l3fwd-power -l 0-63 -- --pmd-mgmt=scale -P -p 0x1 --config="(0,0,1),(0,1,2),(0,2,3),(0,3,4),(0,4,5),(0,5,6),(0,6,7),(0,7,8),(0,8,9),(0,9,10),(0,10,11),(0,11,12),(0,12,13),(0,13,14),(0,14,15),(0,15,16),(0,16,17),(0,17,18),(0,18,19),(0,19,20),(0,20,21),(0,21,22),(0,22,23),(0,23,24),(0,24,25),(0,25,26),(0,26,27),(0,27,28),(0,28,29),(0,29,30),(0,30,31),(0,31,32),(0,32,33),(0,33,34),(0,34,35),(0,35,36),(0,36,37),(0,37,38),(0,38,39),(0,39,40),(0,40,41),(0,41,42),(0,42,43),(0,43,44),(0,44,45),(0,45,46),(0,46,47),(0,47,48),(0,48,49),(0,49,50),(0,50,51),(0,51,52),(0,52,53),(0,53,54),(0,54,55),(0,55,56),(0,56,57),(0,57,58),(0,58,59),(0,59,60),(0,60,61),(0,61,62),(0,62,63)"
+
+- The DPDK core mask must include all cores (e.g. for a machine with 64 cores, the
+ parameter should be set to `-l 0-63`)
+- The port mask must include all physical ports that will be used by the test
+- The `--config` parameter must reflect the number of queues used by each port.
+ For example, if the maximum number of queues is 32 while the platform has 64
+ cores, the test will require two physical ports, and the first 32 cores will
+ use port 0 (`--config="(0,0,0),(0,1,1),(0,2,2)..."`) while the next 32 cores
+ will use port 1 (`--config="(1,0,32),(1,1,33),(1,2,34)..."`)
+
+Step 3. Ensure all cores operate at lowest frequency available. To find lowest
+frequency, read the `cpuinfo_min_freq` value for core 0 from sysfs::
+
+ # find lowest frequency available, in KHz
+ cat /sys/bus/cpu/devices/cpu0/cpufreq/cpuinfo_min_freq
+ 1200000
+
+Then, *while running `dpdk-l3fwd-power` as described in Step 2*, also run
+`turbostat` and ensure that all forwarding cores are running at lowest frequency
+available for those cores::
+
+ # assume min frequency is 1.2GHz
+ turbostat -i 1 -n 1 -s "Core,Bzy_MHz"
+ CPU Bzy_MHz
+ - 1200
+ 0 1200
+ 1 1200
+ 2 1200
+ 3 1200
+ 4 1200
+ 5 1200
+ 6 1200
+ 7 1200
+ ...
+
+Step 4. Repeat Step 1 to reset the pstate scaling settings.
+
+Pass Criteria: average frequency on all cores is roughly equal to minimum
+frequency (there is some variance to be expected, values within 100MHz are
+acceptable)
+
+
+Test Case 2 : Test PMD power management in pause mode with WAITPKG
+=====================================================================
+Requirement: this test requires that the platform *must* support WAITPKG instruction set
+
+Step 1. Launch l3fwd-power in "baseline" mode, and assign each forwarding lcore
+(not including main lcore) exactly one queue::
+
+ ./examples/dpdk-l3fwd-power -l 0-... -- --pmd-mgmt=baseline -P -p 0x1 --config="(0,0,1),(0,1,2),(0,2,3),..."
+ Example:
+ ./examples/dpdk-l3fwd-power -l 0-63 -- --pmd-mgmt=baseline -P -p 0x1 --config="(0,0,1),(0,1,2),(0,2,3),(0,3,4),(0,4,5),(0,5,6),(0,6,7),(0,7,8),(0,8,9),(0,9,10),(0,10,11),(0,11,12),(0,12,13),(0,13,14),(0,14,15),(0,15,16),(0,16,17),(0,17,18),(0,18,19),(0,19,20),(0,20,21),(0,21,22),(0,22,23),(0,23,24),(0,24,25),(0,25,26),(0,26,27),(0,27,28),(0,28,29),(0,29,30),(0,30,31),(0,31,32),(0,32,33),(0,33,34),(0,34,35),(0,35,36),(0,36,37),(0,37,38),(0,38,39),(0,39,40),(0,40,41),(0,41,42),(0,42,43),(0,43,44),(0,44,45),(0,45,46),(0,46,47),(0,47,48),(0,48,49),(0,49,50),(0,50,51),(0,51,52),(0,52,53),(0,53,54),(0,54,55),(0,55,56),(0,56,57),(0,57,58),(0,58,59),(0,59,60),(0,60,61),(0,61,62),(0,62,63)"
+
+- Note that lcore 0 is used by telemetry, so the `--config` parameter will skip
+ the lcore 0 and start from lcore 1
+- See notes for Test Case 1 for more information about how to correctly set up
+ the command-line parameters
+
+Step 2. While Step 1 is in progress, also run `turbostat` and make note of the
+power consumption to establish a baseline against which further measurements
+will be compared to::
+
+ turbostat -i 1 -n 1 -s "PkgWatt"
+
+- The PkgWatt value will be per-socket, as well as aggregate per platform::
+
+ ...
+ PkgWatt
+ 16.31 # aggregate
+ 8.83 # socket 0
+ 7.48 # socket 1
+ ...
+
+ The value that should be noted is the topmost value (aggregate power usage
+ across all sockets).
+
+Step 3. Relaunch l3fwd-power and enable "pause" PMD power management mode, and
+assign each forwarding lcore (including main lcore) exactly one queue::
+
+ ./examples/dpdk-l3fwd-power -l 0-... -- --pmd-mgmt=pause -P -p 0x1 --config="(0,0,0),(0,1,1),(0,2,2),..."
+ Example:
+ ./examples/dpdk-l3fwd-power -l 0-63 -- --pmd-mgmt=pause -P -p 0x1 --config="(0,0,1),(0,1,2),(0,2,3),(0,3,4),(0,4,5),(0,5,6),(0,6,7),(0,7,8),(0,8,9),(0,9,10),(0,10,11),(0,11,12),(0,12,13),(0,13,14),(0,14,15),(0,15,16),(0,16,17),(0,17,18),(0,18,19),(0,19,20),(0,20,21),(0,21,22),(0,22,23),(0,23,24),(0,24,25),(0,25,26),(0,26,27),(0,27,28),(0,28,29),(0,29,30),(0,30,31),(0,31,32),(0,32,33),(0,33,34),(0,34,35),(0,35,36),(0,36,37),(0,37,38),(0,38,39),(0,39,40),(0,40,41),(0,41,42),(0,42,43),(0,43,44),(0,44,45),(0,45,46),(0,46,47),(0,47,48),(0,48,49),(0,49,50),(0,50,51),(0,51,52),(0,52,53),(0,53,54),(0,54,55),(0,55,56),(0,56,57),(0,57,58),(0,58,59),(0,59,60),(0,60,61),(0,61,62),(0,62,63)"
+
+- See notes for Test Case 1 Step 2 for more information about how to correctly
+ set up the command-line parameters
+
+Step 4. While Step 3 is in progress, repeat Step 2 to measure power consumption
+with "pause" PMD power management mode.
+
+Pass Criteria: PkgWatt number has measurably (e.g. >5%) decreased from the
+baseline.
+
+Test Case 3 : Test PMD power management in monitor mode
+=======================================================
+Requirement: this test requires that the platform *must* support WAITPKG instruction set
+
+Step 1. Repeat Step 1 of Test Case 2 to run l3fwd-power in "baseline" mode.
+Step 2. While Step 1 is in progress, repeat Step 2 of Test Case 2 to measure
+power usage baseline.
+
+Step 3. Relaunch l3fwd-power and enable "monitor" PMD power management mode, and
+assign each forwarding lcore (including main lcore) exactly one queue::
+
+ ./examples/dpdk-l3fwd-power -l 0-... -- --pmd-mgmt=monitor -P -p 0x1 --config="(0,0,0),(0,1,1),(0,2,2),..."
+ Example:
+ ./examples/dpdk-l3fwd-power -l 0-63 -- --pmd-mgmt=monitor -P -p 0x1 --config="(0,0,1),(0,1,2),(0,2,3),(0,3,4),(0,4,5),(0,5,6),(0,6,7),(0,7,8),(0,8,9),(0,9,10),(0,10,11),(0,11,12),(0,12,13),(0,13,14),(0,14,15),(0,15,16),(0,16,17),(0,17,18),(0,18,19),(0,19,20),(0,20,21),(0,21,22),(0,22,23),(0,23,24),(0,24,25),(0,25,26),(0,26,27),(0,27,28),(0,28,29),(0,29,30),(0,30,31),(0,31,32),(0,32,33),(0,33,34),(0,34,35),(0,35,36),(0,36,37),(0,37,38),(0,38,39),(0,39,40),(0,40,41),(0,41,42),(0,42,43),(0,43,44),(0,44,45),(0,45,46),(0,46,47),(0,47,48),(0,48,49),(0,49,50),(0,50,51),(0,51,52),(0,52,53),(0,53,54),(0,54,55),(0,55,56),(0,56,57),(0,57,58),(0,58,59),(0,59,60),(0,60,61),(0,61,62),(0,62,63)"
+
+- See notes for Test Case 1 Step 2 for more information about how to correctly
+ set up the command-line parameters
+
+Step 4. While Step 3 is in progress, repeat Step 2 of Test Case 2 to measure
+power consumption with PMD power management.
+
+Pass Criteria: PkgWatt number has measurably (e.g. >5%) decreased from the
+baseline.
new file mode 100644
@@ -0,0 +1,211 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2021-2022 Intel Corporation
+#
+
+"""
+DPDK Test suite.
+PMD power management test plan.
+"""
+
+import re
+import time
+from framework.test_case import TestCase
+from framework.utils import create_mask as dts_create_mask
+
+
+class TestPowerPMD(TestCase):
+
+ def configure_vfio_with_no_iommu(self):
+ # configure VFIO to be used without IOMMU
+ # http://doc.dpdk.org/guides/linux_gsg/linux_drivers.html?#vfio-no-iommu-mode
+ self.dut.send_expect('modprobe vfio-pci', '# ')
+ self.dut.send_expect('echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode', '# ')
+
+ def config_pql_gen(self, lcores, offset):
+ # generate config string
+ # --config(port,queue,lcore)[,(port,queue,lcore)]
+ # determines which queues from which ports are mapped to which cores.
+ arg = ','.join(
+ (
+ f'(0,{i},{i + offset})' for i in range(lcores - offset)
+ )
+ )
+ return f'--config="{arg}"'
+
+ def verify_waitpkg_support(self):
+ out = self.dut.send_expect('grep waitpkg /proc/cpuinfo', '# ', 10)
+ self.verify('waitpkg' in out, 'This test requires that the platform MUST support WAITPKG instruction set.')
+ self.logger.info('The "waitpkg" capability is NOT available on the platform - OK')
+
+ def reset_pstate(self):
+ # Reset all pstate min/max frequency assignments for all cores:
+ # This can be done by overwriting lowest and highest frequencies in the CPU
+ # scaling driver:
+ # for d in /sys/bus/cpu/devices/cpu*/cpufreq/
+ # do
+ # cat $d/cpuinfo_min_freq > $d/scaling_min_freq
+ # cat $d/cpuinfo_max_freq > $d/scaling_max_freq
+ # done
+ self.logger.info('reseting all pstate min/max frequency ..')
+ cmd = 'for d in /sys/bus/cpu/devices/cpu*/cpufreq/; do'
+ cmd = cmd + ' cat ${d}/cpuinfo_min_freq > ${d}/scaling_min_freq;'
+ cmd = cmd + ' cat ${d}/cpuinfo_max_freq > ${d}/scaling_max_freq; done'
+ self.dut.send_expect(cmd, '# ')
+
+ def start_l3fwd(self, mode):
+ if mode in ['baseline', 'scale', 'pause', 'monitor']:
+ l3fwd_cmd = f'{self.l3fwd_path} -l 0-{self.max_lcore} -- --pmd-mgmt={mode} -P -p 0x1 {self.config_0}'
+ else:
+ l3fwd_cmd = f'{self.l3fwd_path} -l 0-{self.max_lcore} -- --{mode} -P -p 0x1 {self.config_1}'
+
+ self.l3fwd_is_on = True
+ self.l3fwd_session.send_expect(l3fwd_cmd, f'L3FWD_POWER: -- lcoreid={self.max_lcore} ', 20)
+ time.sleep(3)
+
+ def stop_l3fwd(self):
+ if self.l3fwd_is_on:
+ self.l3fwd_session.send_expect('^c', '# ', 20)
+ self.l3fwd_is_on = False
+
+ def get_pkgwatt(self):
+ out = self.dut.send_expect('turbostat --show PkgWatt --Summary --num_iterations 1 2>/dev/null | tail -1', '# ')
+ return float(out)
+
+ def get_max_and_ref_freq(self):
+ cpu0_min_freq = self.dut.send_expect('cat /sys/bus/cpu/devices/cpu0/cpufreq/cpuinfo_min_freq', '# ')
+ cpu_min_mhz = int(cpu0_min_freq) // 1000
+ out = self.dut.send_expect('turbostat --quiet --show "Core,Bzy_MHz" --num_iterations 1', '# ')
+ nums = re.findall(r'\n\d+\s+(\d+)', out)
+ min_mhz, max_mhz = min(nums), max(nums)
+
+ self.logger.info(f'the lowest "cpuinfo_min_freq" is {cpu_min_mhz} MHz')
+ self.logger.info(f'cores are running at average frequency between {min_mhz} MHz and {max_mhz} MHz')
+ return int(max_mhz), cpu_min_mhz
+
+ def verify_turbostat(self):
+ # turbostat tool must be installed on the DUT
+ # it also must report average core frequency and platform power consumption
+ # "turbostat -l" must have "PkgWatt", "Core" (or "CPU"), and "Bzy_MHz" columns
+ out = self.dut.send_expect('turbostat -h', '# ')
+ self.verify('Usage:' in out, '"turbostat" tool was not found. Please install "linux-tools-common" and restart the test.')
+ self.logger.info('The "turbostat" tool is available.')
+
+ # check if turbostat reports "PkgWatt", "Core" and "Bzy_MHz" columns
+ out = self.dut.send_expect('turbostat --list', '# ')
+ for column in ['PkgWatt', 'Core', 'Bzy_MHz']:
+ self.verify(f',{column},' in f',{out},', f'"turbostat" does not report "{column}" column. Please check the system configuration.')
+ self.logger.info(f'The "turbostat" reports "{column}" column - OK.')
+
+ def verify_pkgwatt_change(self, mode):
+ # Test Case 2, 3 and 4: Test PMD power management in "pause"/"monitor" mode
+ #
+ # Step 1. Launch l3fwd-power in "baseline" mode
+ # Step 2. While Step 1 is in progress, run turbostat to determine power consumption
+ # Step 3. Relaunch l3fwd-power and enable "pause"/"monitor" PMD power management mode
+ # Step 4. While Step 3 is in progress, measure the power consumption again
+ # Pass Criteria: PkgWatt number has measurably (e.g. >5%) decreased from the baseline.
+ self.start_l3fwd('baseline')
+ pkgwatt_baseline = self.get_pkgwatt()
+ self.stop_l3fwd()
+
+ self.start_l3fwd(mode)
+ pkgwatt_current = self.get_pkgwatt()
+
+ change = pkgwatt_current / pkgwatt_baseline * 100
+ summ = f'PkgWatt: {change:.2f}% change from {pkgwatt_baseline} W in "baseline" mode to {pkgwatt_current} W in "{mode}" mode.'
+ self.logger.info(summ)
+ self.logger.info('Pass Criteria: PkgWatt number has measurably (e.g. >5%) decreased from the baseline.')
+ self.verify(change <= 95, 'PkgWatt did not drop at least 5% below the baseline value.')
+
+ def set_up_all(self):
+ """
+ Run once at the start of test suite.
+ """
+ self.verify_turbostat()
+ self.configure_vfio_with_no_iommu()
+
+ # build the 'dpdk-l3fwd-power' tool
+ out = self.dut.build_dpdk_apps('examples/l3fwd-power')
+ self.verify('Error' not in out, ' "dpdk-l3fwd-power" build error')
+ self.l3fwd_path=self.dut.apps_name['l3fwd-power']
+ self.logger.debug("l3fwd-power tool path: {}".format(self.l3fwd_path))
+ self.l3fwd_is_on = False
+ self.l3fwd_session = self.dut.new_session("l3fwd")
+
+ # determine number of logical cores
+ self.max_lcore = len(self.dut.cores)
+ self.lcores = self.max_lcore + 1
+ self.logger.debug(f'lcores: {self.lcores}')
+
+ # generate two config strings, one including the main lcore 0 and one without
+ # --config(port,queue,lcore)[,(port,queue,lcore)]
+ # determines which queues from which ports are mapped to which cores.
+ self.config_0 = self.config_pql_gen(self.lcores, 0)
+ self.config_1 = self.config_pql_gen(self.lcores, 1)
+ self.logger.debug('reff cfg: --config="(0,0,0),(0,1,1),(0,2,2),..."')
+ self.logger.debug(f'config_0: {self.config_0}')
+ self.logger.debug('reff cfg: --config="(0,0,1),(0,1,2),(0,2,3),..."')
+ self.logger.debug(f'config_1: {self.config_1}')
+
+ def set_up(self):
+ """
+ Run before each test case.
+ """
+ self.reset_pstate()
+
+ def tear_down(self):
+ """
+ Run after each test case.
+ """
+ self.stop_l3fwd()
+ self.dut.kill_all()
+
+ def tear_down_all(self):
+ """
+ Run after each test suite.
+ """
+ self.reset_pstate()
+ self.dut.close_session("l3fwd")
+
+ def test_scale_mode(self):
+ # Test Case 1: Test PMD power management in scale mode
+ # Step 1. Reset all pstate min/max frequency assignments for all cores
+ # Step 2. Launch l3fwd-power and enable "scale" PMD power management mode
+ # Step 3. Ensure all cores operate at lowest frequency available.
+ # To find lowest frequency, read the cpuinfo_min_freq value for core 0 from sysfs:
+ # # find lowest frequency available, in KHz
+ # Ex: cat /sys/bus/cpu/devices/cpu0/cpufreq/cpuinfo_min_freq
+ # 1200000
+ # Then, while running `dpdk-l3fwd-power` as described in Step 2, also run
+ # turbostat and ensure that all forwarding cores are running at lowest frequency
+ # available for those cores:
+ # # assume min frequency is 1.2GHz
+ # turbostat -i 1 -n 1 -s "Core,Bzy_MHz"
+ # CPU Bzy_Mhz
+ # - 1200
+ # 0 1200
+ # 1 1200
+ # 2 1200
+ # 3 1200
+ # ...
+ # Step 4. Repeat Step 1 to reset the pstate scaling settings.
+ # Pass Criteria: average frequency on all cores is roughly equal to minimum
+ # frequency (there is some variance to be expected, values within 100MHz are acceptable)
+ self.start_l3fwd('scale')
+ max_mhz, ref_mhz = self.get_max_and_ref_freq()
+ self.logger.info('Pass Criteria: average frequency on all cores is roughly equal to minimum frequency')
+ self.logger.info(' (there is some variance to be expected, values within 100 MHz are acceptable)')
+ err_msg = f'found frequency {max_mhz} MHz which is over 100 MHz higher than aceptable minimum.'
+ self.verify(abs(max_mhz - ref_mhz) <= 100, err_msg)
+
+ def test_pause_mode_waitpkg(self):
+ # Test Case 3: Test PMD power management in "pause" mode with WAITPKG
+ # Requirement: this test requires that the platform must support WAITPKG instruction set
+ self.verify_waitpkg_support()
+ self.verify_pkgwatt_change('pause')
+
+ def test_monitor_mode_waitpkg(self):
+ # Test Case 4: Test PMD power management in "monitor" mode
+ # Requirement: this test requires that the platform must support WAITPKG instruction set
+ self.verify_waitpkg_support()
+ self.verify_pkgwatt_change('monitor')