From patchwork Wed Mar 31 10:16:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tu, Lijuan" X-Patchwork-Id: 90152 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 43FCFA034F; Wed, 31 Mar 2021 04:18:02 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1909F140DE4; Wed, 31 Mar 2021 04:18:02 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mails.dpdk.org (Postfix) with ESMTP id 4C95340141 for ; Wed, 31 Mar 2021 04:18:00 +0200 (CEST) IronPort-SDR: ajkFxlTy9fqojGtHQu17sDxRh2KklCro3/vV+69XBUZEzpb7IiUKRZjL3d+LxgSBXu8H/eINHo kch+7Ck3NkBg== X-IronPort-AV: E=McAfee;i="6000,8403,9939"; a="179027926" X-IronPort-AV: E=Sophos;i="5.81,291,1610438400"; d="scan'208";a="179027926" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Mar 2021 19:17:56 -0700 IronPort-SDR: gr2gm2rRQasG5QER5AEDg41UYsj6zKoyBPWQ56pMepk30jbuzbENWtoAg3XcRxcXJ0BnR092od xfDUsE3GIu6g== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,291,1610438400"; d="scan'208";a="445432376" Received: from dpdk-icelake-perf.sh.intel.com ([10.67.118.199]) by fmsmga002.fm.intel.com with ESMTP; 30 Mar 2021 19:17:55 -0700 From: Lijuan Tu To: dliu@iol.unh.edu, ohilyard@iol.unh.edu, alialnu@nvidia.com Cc: dts@dpdk.org, Lijuan Tu Date: Wed, 31 Mar 2021 10:16:54 +0000 Message-Id: <20210331101659.506185-2-lijuan.tu@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210331101659.506185-1-lijuan.tu@intel.com> References: <20210331101659.506185-1-lijuan.tu@intel.com> MIME-Version: 1.0 Subject: [dts] [v3 1/6] tests/nic_single_core_perf: beauty JSON string X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org Sender: "dts" with separators and indents, the JSON are more human friendly. before: {"test_perf_nic_single_core": [{ "parameters": [{"name": "Txd/Rxd", ...}, ... } after: { "test_perf_nic_single_core": [ { "parameters": [ { "name": "Txd/Rxd", ... }, ... } Signed-off-by: Lijuan Tu --- tests/TestSuite_nic_single_core_perf.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/tests/TestSuite_nic_single_core_perf.py b/tests/TestSuite_nic_single_core_perf.py index 8266506b..edb4465d 100644 --- a/tests/TestSuite_nic_single_core_perf.py +++ b/tests/TestSuite_nic_single_core_perf.py @@ -368,10 +368,11 @@ class TestNicSingleCorePerf(TestCase): row_dict0['parameters'].append(row_dict4) json_obj[case_name].append(row_dict0) status_result.append(row_dict0['status']) - with open(os.path.join(rst.path2Result, - '{0:s}_single_core_perf.json'.format( - self.nic)), 'w') as fp: - json.dump(json_obj, fp) + + json_file = os.path.join(rst.path2Result, '{0:s}_single_core_perf.json'.format(self.nic)) + with open(json_file, 'w') as fp: + json.dump(json_obj, fp, indent=4, separators=(',', ': '), sort_keys=True) + self.verify("FAIL" not in status_result, "Excessive gap between test results and expectations") def set_fields(self): From patchwork Wed Mar 31 10:16:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tu, Lijuan" X-Patchwork-Id: 90153 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 42C08A0547; Wed, 31 Mar 2021 04:18:03 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3A494140DD9; Wed, 31 Mar 2021 04:18:03 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mails.dpdk.org (Postfix) with ESMTP id 24D8740141 for ; Wed, 31 Mar 2021 04:18:00 +0200 (CEST) IronPort-SDR: 0PrBvqMr3rA1ymbJiiGLKHCv+gdJ9tjyg2mCoXYyClmZKlx5lMWsHYmt0caOduor428FDxB7s2 qoRFA2RClWSA== X-IronPort-AV: E=McAfee;i="6000,8403,9939"; a="179027929" X-IronPort-AV: E=Sophos;i="5.81,291,1610438400"; d="scan'208";a="179027929" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Mar 2021 19:17:58 -0700 IronPort-SDR: 7jtWn8y60IxskuJhPOPgvxiG8m49TaH+cxwLTFhnw25VixVhBpsDgV8YYpG9st4K6NHklSV4ld +hs77ViNVwcA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,291,1610438400"; d="scan'208";a="445432387" Received: from dpdk-icelake-perf.sh.intel.com ([10.67.118.199]) by fmsmga002.fm.intel.com with ESMTP; 30 Mar 2021 19:17:57 -0700 From: Lijuan Tu To: dliu@iol.unh.edu, ohilyard@iol.unh.edu, alialnu@nvidia.com Cc: dts@dpdk.org, Lijuan Tu Date: Wed, 31 Mar 2021 10:16:55 +0000 Message-Id: <20210331101659.506185-3-lijuan.tu@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210331101659.506185-1-lijuan.tu@intel.com> References: <20210331101659.506185-1-lijuan.tu@intel.com> MIME-Version: 1.0 Subject: [dts] [v3 2/6] tests/nic_single_core_perf: get better throughput stats X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org Sender: "dts" Snice throughput is easily fluctuated, we want to improve reliable for results by following: * get multiple throughput statistics during the run time. * remove the maximum and minimum and get average. * sample interval could be configured by users Signed-off-by: Lijuan Tu --- conf/nic_single_core_perf.cfg | 4 +++ tests/TestSuite_nic_single_core_perf.py | 42 +++++++++++++++++++------ 2 files changed, 37 insertions(+), 9 deletions(-) diff --git a/conf/nic_single_core_perf.cfg b/conf/nic_single_core_perf.cfg index 4d3279ba..e6b0a5af 100644 --- a/conf/nic_single_core_perf.cfg +++ b/conf/nic_single_core_perf.cfg @@ -5,6 +5,9 @@ # numbers, and the pattern is # {'frame size': ['descriptor number #1', 'descriptor number #2']} # +# - throughput_stat_sample_interval defines interval of get throughput +# statistics (second). If not set, it is 5 seconds by default. +# # - test_duration is how many seconds each combination performance will # be recorded. # @@ -34,6 +37,7 @@ test_parameters = {'1C/1T': {64: [512, 2048]}, '1C/2T': {64: [512, 2048]}} rx_desc_16byte = 'y' test_duration = 60 +throughput_stat_sample_interval = 2 accepted_tolerance = 1 expected_throughput = { 'fortville_spirit': { diff --git a/tests/TestSuite_nic_single_core_perf.py b/tests/TestSuite_nic_single_core_perf.py index edb4465d..a0a257dd 100644 --- a/tests/TestSuite_nic_single_core_perf.py +++ b/tests/TestSuite_nic_single_core_perf.py @@ -41,6 +41,7 @@ from exception import VerifyFailure from settings import HEADER_SIZE, UPDATE_EXPECTED, load_global_setting from pmd_output import PmdOutput from copy import deepcopy +from numpy import mean import rst from pktgen import PacketGeneratorHelper @@ -96,6 +97,7 @@ class TestNicSingleCorePerf(TestCase): # traffic duraion in second self.test_duration = self.get_suite_cfg()['test_duration'] + self.throughput_stat_sample_interval = self.get_suite_cfg().get('throughput_stat_sample_interval', 5) # load the expected throughput for required nic if self.nic in ["ConnectX4_LX_MT4117"]: @@ -265,21 +267,43 @@ class TestNicSingleCorePerf(TestCase): # run packet generator streams = self.pktgen_helper.prepare_stream_from_tginput(tgenInput, 100, vm_config, self.tester.pktgen) # set traffic option - traffic_opt = {'duration': self.test_duration} - # _, pps = self.tester.traffic_generator_throughput(tgenInput, rate_percent=100, delay=30) - _, packets_received = self.tester.pktgen.measure_throughput(stream_ids=streams, options=traffic_opt) - self.verify(packets_received > 0, "No traffic detected") - throughput = packets_received / 1000000.0 - self.throughput[fwd_config][frame_size][nb_desc] = throughput + traffic_opt = { + 'method': 'throughput', + 'rate': 100, + 'duration': self.test_duration, + 'interval': self.throughput_stat_sample_interval, + } + stats = self.tester.pktgen.measure(stream_ids=streams, traffic_opt=traffic_opt) + + ##################################################### + # Remove max and min if count >=5, then get average + ##################################################### + if isinstance(stats, list): + total_pps_rxs = [] + c = len(stats) + for i in range(c): + stats_pps = stats[i][1] + if isinstance(stats_pps, tuple): + total_pps_rxs.append(stats_pps[1]) + else: + total_pps_rxs.append(stats_pps) + if c >= 5: + total_pps_rxs.remove(max(total_pps_rxs)) + total_pps_rxs.remove(min(total_pps_rxs)) + total_pps_rx = mean(total_pps_rxs) + else: + total_pps_rx = stats + + self.verify(total_pps_rx > 0, "No traffic detected, please check your configuration") + total_mpps_rx = total_pps_rx / 1000000.0 + self.throughput[fwd_config][frame_size][nb_desc] = total_mpps_rx self.dut.send_expect("stop", "testpmd> ") self.dut.send_expect("quit", "# ", 30) - self.verify(throughput, - "No traffic detected, please check your configuration") self.logger.info("Trouthput of " + "framesize: {}, rxd/txd: {} is :{} Mpps".format( - frame_size, nb_desc, throughput)) + frame_size, nb_desc, total_mpps_rx)) return self.throughput From patchwork Wed Mar 31 10:16:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tu, Lijuan" X-Patchwork-Id: 90155 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 084EAA034F; Wed, 31 Mar 2021 04:18:04 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 83CAE140DF9; Wed, 31 Mar 2021 04:18:03 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mails.dpdk.org (Postfix) with ESMTP id F0D084069E for ; Wed, 31 Mar 2021 04:18:01 +0200 (CEST) IronPort-SDR: agqGCugkmxg/1FK3jUNXLUo6CYmS881tqym/2Bk3DlRpSiw5KsAnfpOC3B1yYubO6GG3/Xf+Hx DbbtxzH7Hocw== X-IronPort-AV: E=McAfee;i="6000,8403,9939"; a="179027931" X-IronPort-AV: E=Sophos;i="5.81,291,1610438400"; d="scan'208";a="179027931" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Mar 2021 19:18:00 -0700 IronPort-SDR: xVARVW6WTH9lSjrfvDORrnXWk+qqydjG8tk4dtHgj/BvKWlQzRPR9kH6nqdwp85aJ5/3Z0soaP Bj9QihLTDdog== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,291,1610438400"; d="scan'208";a="445432396" Received: from dpdk-icelake-perf.sh.intel.com ([10.67.118.199]) by fmsmga002.fm.intel.com with ESMTP; 30 Mar 2021 19:17:58 -0700 From: Lijuan Tu To: dliu@iol.unh.edu, ohilyard@iol.unh.edu, alialnu@nvidia.com Cc: dts@dpdk.org, Lijuan Tu Date: Wed, 31 Mar 2021 10:16:56 +0000 Message-Id: <20210331101659.506185-4-lijuan.tu@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210331101659.506185-1-lijuan.tu@intel.com> References: <20210331101659.506185-1-lijuan.tu@intel.com> MIME-Version: 1.0 Subject: [dts] [v3 3/6] tests/nic_single_core_perf: revise rst table X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org Sender: "dts" add more info in rst table, and rename headers to reduce width. before: +----------+------------+---------+------------+--------+---------------------+-----------------------+ | Fwd_core | Frame Size | TXD/RXD | Throughput | Rate | Expected Throughput | Throughput Difference | +==========+============+=========+============+========+=====================+=======================+ | 1C/1T | 64 | 512 | 9.842 Mpps | 3.307% | 0.000 Mpps | 9.842 Mpps | +----------+------------+---------+------------+--------+---------------------+-----------------------+ after: +----------+------------+---------+-----------+--------+---------------+-----------------+--------+ | Fwd_core | Frame Size | TXD/RXD | Real-Mpps | Rate | Expected-Mpps | Real - Expected | Status | +==========+============+=========+===========+========+===============+=================+========+ | 1C/1T | 64 | 512 | 9.864 | 3.314% | 0 | 9.864 | PASS | +----------+------------+---------+-----------+--------+---------------+-----------------+--------+ Signed-off-by: Lijuan Tu --- tests/TestSuite_nic_single_core_perf.py | 28 +++++++++++++------------ 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/tests/TestSuite_nic_single_core_perf.py b/tests/TestSuite_nic_single_core_perf.py index a0a257dd..095b508a 100644 --- a/tests/TestSuite_nic_single_core_perf.py +++ b/tests/TestSuite_nic_single_core_perf.py @@ -120,8 +120,8 @@ class TestNicSingleCorePerf(TestCase): self.gap = self.get_suite_cfg()['accepted_tolerance'] # header to print test result table - self.table_header = ['Fwd_core', 'Frame Size', 'TXD/RXD', 'Throughput', 'Rate', - 'Expected Throughput', 'Throughput Difference'] + self.table_header = ['Fwd_core', 'Frame Size', 'TXD/RXD', 'Real-Mpps', 'Rate', + 'Expected-Mpps', 'Real - Expected', 'Status'] self.test_result = {} def flows(self): @@ -327,15 +327,17 @@ class TestNicSingleCorePerf(TestCase): ret_data[header[0]] = fwd_config ret_data[header[1]] = frame_size ret_data[header[2]] = nb_desc - ret_data[header[3]] = "{:.3f} Mpps".format( - self.throughput[fwd_config][frame_size][nb_desc]) - ret_data[header[4]] = "{:.3f}%".format( - self.throughput[fwd_config][frame_size][nb_desc] * 100 / wirespeed) - ret_data[header[5]] = "{:.3f} Mpps".format( - self.expected_throughput[fwd_config][frame_size][nb_desc]) - ret_data[header[6]] = "{:.3f} Mpps".format( - self.throughput[fwd_config][frame_size][nb_desc] - - self.expected_throughput[fwd_config][frame_size][nb_desc]) + _real = self.throughput[fwd_config][frame_size][nb_desc] + _exp = self.expected_throughput[fwd_config][frame_size][nb_desc] + ret_data[header[3]] = "{:.3f}".format(_real) + ret_data[header[4]] = "{:.3f}%".format(_real * 100 / wirespeed) + ret_data[header[5]] = "{:.3f}".format(_exp) + delta = _real - _exp + ret_data[header[6]] = "{:.3f}".format(delta) + if delta > -self.gap: + ret_data[header[7]] = 'PASS' + else: + ret_data[header[7]] = 'FAIL' ret_datas[frame_size][nb_desc] = deepcopy(ret_data) self.test_result[fwd_config] = deepcopy(ret_datas) @@ -374,8 +376,8 @@ class TestNicSingleCorePerf(TestCase): row_dict0 = dict() row_dict0['performance'] = list() row_dict0['parameters'] = list() - result_throughput = float(row_in['Throughput'].split()[0]) - expected_throughput = float(row_in['Expected Throughput'].split()[0]) + result_throughput = float(row_in['Real-Mpps']) + expected_throughput = float(row_in['Expected-Mpps']) # delta value and accepted tolerance in percentage delta = result_throughput - expected_throughput if delta > -self.gap: From patchwork Wed Mar 31 10:16:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tu, Lijuan" X-Patchwork-Id: 90156 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5DF2FA0547; Wed, 31 Mar 2021 04:18:04 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id A5F9D140DFD; Wed, 31 Mar 2021 04:18:03 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mails.dpdk.org (Postfix) with ESMTP id 97AD840141 for ; Wed, 31 Mar 2021 04:18:02 +0200 (CEST) IronPort-SDR: 86sZlUHjfTrwyc874P1rV8sxnFPSZXCKlFxe4L1d2Rf1ME+q/bOcjlaSZYJgo3e5nusQvRcaZo E/96uZd2wDgg== X-IronPort-AV: E=McAfee;i="6000,8403,9939"; a="179027933" X-IronPort-AV: E=Sophos;i="5.81,291,1610438400"; d="scan'208";a="179027933" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Mar 2021 19:18:01 -0700 IronPort-SDR: Cj0L1rr2Nt1n3hYtg5JxaNn+01USzSM3djZhq2fYO6VRqM3nWE4UTxmZgwXDKJjYvMrRmzSXOD f3ypZcReizTQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,291,1610438400"; d="scan'208";a="445432415" Received: from dpdk-icelake-perf.sh.intel.com ([10.67.118.199]) by fmsmga002.fm.intel.com with ESMTP; 30 Mar 2021 19:18:00 -0700 From: Lijuan Tu To: dliu@iol.unh.edu, ohilyard@iol.unh.edu, alialnu@nvidia.com Cc: dts@dpdk.org, Lijuan Tu Date: Wed, 31 Mar 2021 10:16:57 +0000 Message-Id: <20210331101659.506185-5-lijuan.tu@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210331101659.506185-1-lijuan.tu@intel.com> References: <20210331101659.506185-1-lijuan.tu@intel.com> MIME-Version: 1.0 Subject: [dts] [v3 4/6] tests/nic_single_core_perf: revise delta to ratio X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org Sender: "dts" the delta used to defines the difference between real pps and expected pps, the case fails if it exceed the absolute number specified by 'accepted_tolerance' in conf/nic_single_core_perf.cfg, the deault is 1Mpps, which is not friendly when the expected is only a few of million, so updated the accepted_tolerance to a percentage, considering the CI intergation, so the type is float. Here are main changes: * the meaning of accepted_tolerance before: threshold for (real-mpps - expected-mpps) after: threshold for (real-mpps - expected-mpps)/expected-mpps * Pass criteria before: real-mpps - expected-mpps > -accepted_tolerance after: (real-mpps - expected-mpps)/expected-mpps > -accepted_tolerance * rst table before: ... +---------------+-----------------+--------+ ... | Expected-Mpps | Real - Expected | Status | ... +===============+=================+========+ ... | 9.900 | -0.040 | PASS | ... +---------------+-----------------+--------+ after: ... +---------------+------------+--------+ ... | Expected-Mpps | Fluc Ratio | Status | ... +===============+============+========+ ... | 9.900 | -0.004 | PASS | ... +---------------+------------+--------+ * delta in JSON: before: real-mpps - expected-mpps after: (real-mpps - expected-mpps)/expected-mpps Signed-off-by: Lijuan Tu --- conf/nic_single_core_perf.cfg | 10 +++++--- tests/TestSuite_nic_single_core_perf.py | 32 ++++++++++++------------- 2 files changed, 22 insertions(+), 20 deletions(-) diff --git a/conf/nic_single_core_perf.cfg b/conf/nic_single_core_perf.cfg index e6b0a5af..72b8c533 100644 --- a/conf/nic_single_core_perf.cfg +++ b/conf/nic_single_core_perf.cfg @@ -11,8 +11,12 @@ # - test_duration is how many seconds each combination performance will # be recorded. # -# - accepted_tolerance defines the accepted tolerance between test -# results and expected numbers. +# - accepted_tolerance defines the accepted tolerance for the +# fluctuation ratio of performance, it is a float(e.g. 0.05). +# Assume: +# F = (Real-Mpps - Expected-Mpps)/Expected-Mpps +# T = accepted_tolerance +# If F < -T, the case is PASS, else it is FAIL # # - expected_throughput is a dictionary defining expected throughput # numbers based on NIC, and the pattern is @@ -38,7 +42,7 @@ test_parameters = {'1C/1T': {64: [512, 2048]}, rx_desc_16byte = 'y' test_duration = 60 throughput_stat_sample_interval = 2 -accepted_tolerance = 1 +accepted_tolerance = 0.05 expected_throughput = { 'fortville_spirit': { '1C/1T': {64: {512: 0.00, 2048: 0.00}}, diff --git a/tests/TestSuite_nic_single_core_perf.py b/tests/TestSuite_nic_single_core_perf.py index 095b508a..dd35caac 100644 --- a/tests/TestSuite_nic_single_core_perf.py +++ b/tests/TestSuite_nic_single_core_perf.py @@ -116,12 +116,12 @@ class TestNicSingleCorePerf(TestCase): # {'$framesize':{"$nb_desc": 'throughput'} self.throughput = {} - # Accepted tolerance in Mpps - self.gap = self.get_suite_cfg()['accepted_tolerance'] + # Accepted tolerance is ratio + self.gap = self.get_suite_cfg().get('accepted_tolerance', 0.1) # header to print test result table self.table_header = ['Fwd_core', 'Frame Size', 'TXD/RXD', 'Real-Mpps', 'Rate', - 'Expected-Mpps', 'Real - Expected', 'Status'] + 'Expected-Mpps', 'Fluc Ratio', 'Status'] self.test_result = {} def flows(self): @@ -332,12 +332,16 @@ class TestNicSingleCorePerf(TestCase): ret_data[header[3]] = "{:.3f}".format(_real) ret_data[header[4]] = "{:.3f}%".format(_real * 100 / wirespeed) ret_data[header[5]] = "{:.3f}".format(_exp) - delta = _real - _exp - ret_data[header[6]] = "{:.3f}".format(delta) - if delta > -self.gap: - ret_data[header[7]] = 'PASS' + delta = (_real - _exp)/_exp + if _exp != 0: + ret_data[header[6]] = "{:.3f}".format(delta) + if delta > -self.gap: + ret_data[header[7]] = 'PASS' + else: + ret_data[header[7]] = 'FAIL' else: - ret_data[header[7]] = 'FAIL' + ret_data[header[6]] = "N/A" + ret_data[header[7]] = 'PASS' ret_datas[frame_size][nb_desc] = deepcopy(ret_data) self.test_result[fwd_config] = deepcopy(ret_datas) @@ -376,15 +380,9 @@ class TestNicSingleCorePerf(TestCase): row_dict0 = dict() row_dict0['performance'] = list() row_dict0['parameters'] = list() - result_throughput = float(row_in['Real-Mpps']) - expected_throughput = float(row_in['Expected-Mpps']) - # delta value and accepted tolerance in percentage - delta = result_throughput - expected_throughput - if delta > -self.gap: - row_dict0['status'] = 'PASS' - else: - row_dict0['status'] = 'FAIL' - row_dict1 = dict(name="Throughput", value=result_throughput, unit="Mpps", delta=delta) + row_dict0['status'] = row_in['Status'] + row_dict1 = dict(name="Throughput", value=row_in['Real-Mpps'], unit="Mpps", + delta=row_in['Fluc Ratio']) row_dict2 = dict(name="Txd/Rxd", value=row_in["TXD/RXD"], unit="descriptor") row_dict3 = dict(name="frame_size", value=row_in["Frame Size"], unit="bytes") row_dict4 = dict(name="Fwd_core", value=row_in["Fwd_core"]) From patchwork Wed Mar 31 10:16:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tu, Lijuan" X-Patchwork-Id: 90157 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E5D0AA034F; Wed, 31 Mar 2021 04:18:05 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id D2366140E02; Wed, 31 Mar 2021 04:18:05 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mails.dpdk.org (Postfix) with ESMTP id 33592140DED for ; Wed, 31 Mar 2021 04:18:04 +0200 (CEST) IronPort-SDR: B423IcM0uOaYio0l/f5wEcIXF/j0uBKTGf4WitTy+Dj9Dt7jzM+clwdkV11ypoImI73MOFmTz7 zNsvpFNDKKWA== X-IronPort-AV: E=McAfee;i="6000,8403,9939"; a="179027935" X-IronPort-AV: E=Sophos;i="5.81,291,1610438400"; d="scan'208";a="179027935" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Mar 2021 19:18:03 -0700 IronPort-SDR: Y0nx4WYCpp6FFJ7ZQ+WVa2tJeQX79tjWsf0a2zLq+emDnnN3YMK6GX9UL9ANFwDogGvwYwVDlv qPK6YzzIA5jg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,291,1610438400"; d="scan'208";a="445432425" Received: from dpdk-icelake-perf.sh.intel.com ([10.67.118.199]) by fmsmga002.fm.intel.com with ESMTP; 30 Mar 2021 19:18:02 -0700 From: Lijuan Tu To: dliu@iol.unh.edu, ohilyard@iol.unh.edu, alialnu@nvidia.com Cc: dts@dpdk.org, Lijuan Tu Date: Wed, 31 Mar 2021 10:16:58 +0000 Message-Id: <20210331101659.506185-6-lijuan.tu@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210331101659.506185-1-lijuan.tu@intel.com> References: <20210331101659.506185-1-lijuan.tu@intel.com> MIME-Version: 1.0 Subject: [dts] [v3 5/6] tests/nic_single_core_perf: add expected mpps into JSON X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org Sender: "dts" expose more perf info for CI analysis. Signed-off-by: Lijuan Tu --- tests/TestSuite_nic_single_core_perf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/TestSuite_nic_single_core_perf.py b/tests/TestSuite_nic_single_core_perf.py index dd35caac..8606f599 100644 --- a/tests/TestSuite_nic_single_core_perf.py +++ b/tests/TestSuite_nic_single_core_perf.py @@ -382,7 +382,7 @@ class TestNicSingleCorePerf(TestCase): row_dict0['parameters'] = list() row_dict0['status'] = row_in['Status'] row_dict1 = dict(name="Throughput", value=row_in['Real-Mpps'], unit="Mpps", - delta=row_in['Fluc Ratio']) + delta=row_in['Fluc Ratio'], expected=row_in['Expected-Mpps']) row_dict2 = dict(name="Txd/Rxd", value=row_in["TXD/RXD"], unit="descriptor") row_dict3 = dict(name="frame_size", value=row_in["Frame Size"], unit="bytes") row_dict4 = dict(name="Fwd_core", value=row_in["Fwd_core"]) From patchwork Wed Mar 31 10:16:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Tu, Lijuan" X-Patchwork-Id: 90158 Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 09DD9A0547; Wed, 31 Mar 2021 04:18:07 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 0515F140E08; Wed, 31 Mar 2021 04:18:07 +0200 (CEST) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by mails.dpdk.org (Postfix) with ESMTP id 70118140DED for ; Wed, 31 Mar 2021 04:18:05 +0200 (CEST) IronPort-SDR: eLN/NoBKecy0yVxrm4VkTCVdwka3oLwLHVuWQMzdM5yHcI5xt2zxb52wK/FgwDyJaF3kMa8xFO 1XudRFAJh/Ow== X-IronPort-AV: E=McAfee;i="6000,8403,9939"; a="179027940" X-IronPort-AV: E=Sophos;i="5.81,291,1610438400"; d="scan'208";a="179027940" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Mar 2021 19:18:05 -0700 IronPort-SDR: TFgyIhhRRIE49iWfh7o8MG/yXy3r6LdPC1Ei2wbwCqwogW2ajN7sWu6vpVoJEY/HBluqpX/nYM g9fFq7tkocbg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,291,1610438400"; d="scan'208";a="445432446" Received: from dpdk-icelake-perf.sh.intel.com ([10.67.118.199]) by fmsmga002.fm.intel.com with ESMTP; 30 Mar 2021 19:18:03 -0700 From: Lijuan Tu To: dliu@iol.unh.edu, ohilyard@iol.unh.edu, alialnu@nvidia.com Cc: dts@dpdk.org, Lijuan Tu Date: Wed, 31 Mar 2021 10:16:59 +0000 Message-Id: <20210331101659.506185-7-lijuan.tu@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210331101659.506185-1-lijuan.tu@intel.com> References: <20210331101659.506185-1-lijuan.tu@intel.com> MIME-Version: 1.0 Subject: [dts] [v3 6/6] tests/nic_single_core_perf: revise rx desc size X-BeenThere: dts@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: test suite reviews and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dts-bounces@dpdk.org Sender: "dts" rx_desc_16byte uses 'y' indicates the rx descriptor size to 16B, and other string for 32B. but 'y' is a reserved string in INI, misuse it will casuse the write-back configurattion file can't be read correctly. so rename rx_desc_16byte to rx_desc_size and the type is integer. Signed-off-by: Lijuan Tu --- conf/nic_single_core_perf.cfg | 4 +++- tests/TestSuite_nic_single_core_perf.py | 17 +++++++++-------- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/conf/nic_single_core_perf.cfg b/conf/nic_single_core_perf.cfg index 72b8c533..7970995e 100644 --- a/conf/nic_single_core_perf.cfg +++ b/conf/nic_single_core_perf.cfg @@ -5,6 +5,8 @@ # numbers, and the pattern is # {'frame size': ['descriptor number #1', 'descriptor number #2']} # +# - rx_desc_size defines the rx descriptor size which is 16B or 32B. +# # - throughput_stat_sample_interval defines interval of get throughput # statistics (second). If not set, it is 5 seconds by default. # @@ -39,7 +41,7 @@ update_expected = True test_parameters = {'1C/1T': {64: [512, 2048]}, '1C/2T': {64: [512, 2048]}} -rx_desc_16byte = 'y' +rx_desc_size = 16 test_duration = 60 throughput_stat_sample_interval = 2 accepted_tolerance = 0.05 diff --git a/tests/TestSuite_nic_single_core_perf.py b/tests/TestSuite_nic_single_core_perf.py index 8606f599..ee022054 100644 --- a/tests/TestSuite_nic_single_core_perf.py +++ b/tests/TestSuite_nic_single_core_perf.py @@ -58,13 +58,15 @@ class TestNicSingleCorePerf(TestCase): "Not required NIC ") self.headers_size = HEADER_SIZE['eth'] + HEADER_SIZE['ip'] - self.rx_desc = self.get_suite_cfg()['rx_desc_16byte'] - if 'y' == self.rx_desc: + self.rx_desc_size = self.get_suite_cfg().get('rx_desc_size', 32) + err_msg = "Rx desc only has 16B and 32B size, %d is not valid" % self.rx_desc_size + self.verify(self.rx_desc_size == 16 or self.rx_desc_size == 32, err_msg) + if self.rx_desc_size == 16: # Update DPDK config file and rebuild to get best perf on fortville if self.nic in ["fortville_25g", "fortville_spirit"]: - self.dut.set_build_options({'RTE_LIBRTE_I40E_16BYTE_RX_DESC': self.rx_desc}) + self.dut.set_build_options({'RTE_LIBRTE_I40E_16BYTE_RX_DESC': 'y'}) elif self.nic in ["columbiaville_100g", "columbiaville_25g", "columbiaville_25gx2"]: - self.dut.set_build_options({'RTE_LIBRTE_ICE_16BYTE_RX_DESC': self.rx_desc}) + self.dut.set_build_options({'RTE_LIBRTE_ICE_16BYTE_RX_DESC': 'y'}) self.dut.build_install_dpdk(self.target) # Based on h/w type, choose how many ports to use @@ -417,11 +419,10 @@ class TestNicSingleCorePerf(TestCase): Run after each test suite. """ # resume setting - if 'y' == self.rx_desc: - self.rx_desc = 'n' + if self.rx_desc_size == 16: if self.nic in ["fortville_25g", "fortville_spirit"]: - self.dut.set_build_options({'RTE_LIBRTE_I40E_16BYTE_RX_DESC': self.rx_desc}) + self.dut.set_build_options({'RTE_LIBRTE_I40E_16BYTE_RX_DESC': 'n'}) elif self.nic in ["columbiaville_100g", "columbiaville_25g", "columbiaville_25gx2"]: - self.dut.set_build_options({'RTE_LIBRTE_ICE_16BYTE_RX_DESC': self.rx_desc}) + self.dut.set_build_options({'RTE_LIBRTE_ICE_16BYTE_RX_DESC': 'n'}) self.dut.build_install_dpdk(self.target) self.dut.kill_all()