From patchwork Thu Jan 25 19:15:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Mattias_R=C3=B6nnblom?= X-Patchwork-Id: 136169 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id DCBA2439C5; Thu, 25 Jan 2024 20:23:42 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C018B402D7; Thu, 25 Jan 2024 20:23:40 +0100 (CET) Received: from EUR02-AM0-obe.outbound.protection.outlook.com (mail-am0eur02on2069.outbound.protection.outlook.com [40.107.247.69]) by mails.dpdk.org (Postfix) with ESMTP id 3D210402AB for ; Thu, 25 Jan 2024 20:23:40 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ab8yaCSKuy7cFFdzbSpJeJwLWRYb8r1Irb2/IBf5OwR/mUpWH93YU/vz0NLvCcjPI1AeIH7M0ls/9sSGuExKyrj/soz8rcDtQDyGU+c5JKU41K7Pq/iM1/sEhOGbDQl4gr0SWddIEDI6x75VxMsOYezv4lRAysVej3OrXDvdDUsm9cjwq4lBzTe7VGZDd6+JnNaIsRJLKqo1nyc0sVDThmB0ChsT0fhidqSAsZyNSpfdCMVQxCaEEcSjKimxKDClXi8+17JzdqMdh55URy13BcZiowDNJC3UE9YNyTunvNcTdC/mT0X1qKscVo2g4l6BAJXbZeCnGUn9uUiJjfS9sQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LgaKm/qB+RePPRVuEhXPIbG4QDfi58UwZLx/QN3aIig=; b=Elb/TcnaHSozf2pjFX2Aiuy1lc8PUOdxpfm5GcQRZdjMsJZHrq6Bnt8a7JCgtb0E6rJhE+WpWupdZAcdlbuyMBX7DCskmyPI6GWwYpr/D/vF+NDN4u+q1hlDcz31B/Biw56uwq0lZAuW5l2b1BlsafVddcTMK3eEvs0tg6uL5/JGjyEUC3Vd4heyNNgwPnrgj8XCYMPTCVh7Y9zxAcqcA8nfuUCzwfEeoy0p31i6uvTGCnAIjISFhy9ds4LgE42NbXPJKgka1zMa0tq43PWbmNuiN9kzwhljOu+IEGzzDFFxKB0KHAhjWIS3h5NAknRVSxou50JhZ07hyTiceyWy9w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 192.176.1.74) smtp.rcpttodomain=dpdk.org smtp.mailfrom=ericsson.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=ericsson.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ericsson.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LgaKm/qB+RePPRVuEhXPIbG4QDfi58UwZLx/QN3aIig=; b=R2LifXFRNc3GcIsvACwxiopwr/IcZBvS/m8/dxRVp6nzJKFziGDwUgpLLXTZAkFbo8gpGRPTo4YRhVi5finlA5VZsA08G2KnFP45XFpmPaKSbls9m7pCBYjDB2LxWB7JWDfPUKvjwiERcZRoHupqWjY6CgqxeHo2krRCwGau010x7WiJhH1u810qcuLrq9Bj0RkPTzVWqg5pjCpWa0q7qCmvYhIkoqBDHTB+3sVl5aiAd1ZmlQMGMIH5d/L3fIVsybdovmbtpxetZpRyMn9ptSETF332BE574UsVAhvWiZK0IY+19T0u/vdKSwd4I7s8ZPXbyl2xnzwoRJ3AZEtY1Q== Received: from AM6P194CA0074.EURP194.PROD.OUTLOOK.COM (2603:10a6:209:8f::15) by AS2PR07MB9325.eurprd07.prod.outlook.com (2603:10a6:20b:60b::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.22; Thu, 25 Jan 2024 19:23:37 +0000 Received: from AMS1EPF00000042.eurprd04.prod.outlook.com (2603:10a6:209:8f:cafe::5a) by AM6P194CA0074.outlook.office365.com (2603:10a6:209:8f::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.26 via Frontend Transport; Thu, 25 Jan 2024 19:23:36 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 192.176.1.74) smtp.mailfrom=ericsson.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=ericsson.com; Received-SPF: Pass (protection.outlook.com: domain of ericsson.com designates 192.176.1.74 as permitted sender) receiver=protection.outlook.com; client-ip=192.176.1.74; helo=oa.msg.ericsson.com; pr=C Received: from oa.msg.ericsson.com (192.176.1.74) by AMS1EPF00000042.mail.protection.outlook.com (10.167.16.39) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7228.16 via Frontend Transport; Thu, 25 Jan 2024 19:23:36 +0000 Received: from seliicinfr00050.seli.gic.ericsson.se (153.88.142.248) by smtp-central.internal.ericsson.com (100.87.178.61) with Microsoft SMTP Server id 15.2.1258.12; Thu, 25 Jan 2024 20:23:36 +0100 Received: from breslau.. (seliicwb00002.seli.gic.ericsson.se [10.156.25.100]) by seliicinfr00050.seli.gic.ericsson.se (Postfix) with ESMTP id 4DD8B1C006A; Thu, 25 Jan 2024 20:23:36 +0100 (CET) From: =?utf-8?q?Mattias_R=C3=B6nnblom?= To: CC: , , Stefan Sundkvist , =?utf-8?q?Mattias_R=C3=B6?= =?utf-8?q?nnblom?= Subject: [RFC] service: extend service function call statistics Date: Thu, 25 Jan 2024 20:15:29 +0100 Message-ID: <20240125191529.388570-1-mattias.ronnblom@ericsson.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: AMS1EPF00000042:EE_|AS2PR07MB9325:EE_ X-MS-Office365-Filtering-Correlation-Id: 4cd117c8-9ed3-4b95-01f9-08dc1ddb210d X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6w5+fuK3Cn2WAniARcE6KWbuzB+nyPvfqstcj8vUVLW5PirkiklTLqLbXiizABSZFDYmkJ6dCMv2T9+POz1fz6FGPG/D27XOD4AOSZtNfluaMGzXiiejWA9XTxAxRI5CwcNnJ8NT6SCoDVyJXozTqq3YUnuyYCWpD+2a+juLnIM32h0T382xor3/FkUyBlWe3Lbigu4i31kU8qZCRMUKgCsYKVRCwa+1mWaaW7V8bJmv6Uzh2TPgsgLvoOHvsuH/Km3nPdoHzlXlJ/uK1PruqFnglr17SYkKHck6GVRiTFY/XuogNYTvFKkMTdyF0D8lOZkyvE76jJ4pjC5Sdj9xEELlTShog+ChadAeWnjazBMd64d/qbgAafa/R1QkGc6mbTN9qVmUrQcMIBmJ2DLIAsIrSHu/O5zaZioiZZc2Jhy32Ip17dDf/ly3S8PnbPMWptLao3QZEWnBhLRSdqh0OURswkB0pjqCf53fzZYkS24Ho37fDQkhjV1Zcfa0hLMek3ah2+5zNhfiRt6XgKt9EC9ADHqMWqZgfIgzzV7oUuEniUAllFiDKVdPsiaKyNQY69A8bil3CM4mjBrS8uk4+cbj9RWxG2a5L0Ztk6g+RcHbvdtUPNNLGtL1lfGDKO80Kk1kEkzlfXzJznYZzVaxALq4MNRy5FcR6YzuMvO/zaS4orVrCc6CrjVrWJsjv/qiOmUHL2k8e7RfHwwbBye7BVbfEAdmhv6pTQ22XABnwd+NDK/CIZ9jsh7m8YotKXAT X-Forefront-Antispam-Report: CIP:192.176.1.74; CTRY:SE; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:oa.msg.ericsson.com; PTR:office365.se.ericsson.net; CAT:NONE; SFS:(13230031)(4636009)(39860400002)(136003)(376002)(346002)(396003)(230922051799003)(451199024)(64100799003)(1800799012)(186009)(82310400011)(40470700004)(36840700001)(46966006)(82740400003)(30864003)(36860700001)(2906002)(36756003)(41300700001)(82960400001)(86362001)(7636003)(356005)(26005)(70586007)(2616005)(336012)(70206006)(6266002)(316002)(66574015)(1076003)(54906003)(47076005)(107886003)(6666004)(6916009)(83380400001)(478600001)(5660300002)(8676002)(4326008)(8936002)(40460700003)(40480700001); DIR:OUT; SFP:1101; X-OriginatorOrg: ericsson.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2024 19:23:36.8842 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 4cd117c8-9ed3-4b95-01f9-08dc1ddb210d X-MS-Exchange-CrossTenant-Id: 92e84ceb-fbfd-47ab-be52-080c6b87953f X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=92e84ceb-fbfd-47ab-be52-080c6b87953f; Ip=[192.176.1.74]; Helo=[oa.msg.ericsson.com] X-MS-Exchange-CrossTenant-AuthSource: AMS1EPF00000042.eurprd04.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: AS2PR07MB9325 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Add two new per-service counters. RTE_SERVICE_ATTR_IDLE_CALL_COUNT tracks the number of service function invocations where no work was performed. RTE_SERVICE_ATTR_ERROR_CALL_COUNT tracks the number invocations resulting in an error. The semantics of RTE_SERVICE_ATTR_CALL_COUNT remains the same (i.e., counting all invocations, regardless of return value). The new statistics may be useful for both debugging and profiling (e.g., calculate the average per-call processing latency for non-idle service calls). Service core tests are extended to cover the new counters, and coverage for RTE_SERVICE_ATTR_CALL_COUNT is improved. The documentation for the CYCLES attributes are updated to reflect their actual semantics. Signed-off-by: Mattias Rönnblom Acked-by: Morten Brørup Acked-by: Harry van Haaren --- app/test/test_service_cores.c | 70 ++++++++++++++++++++++++++--------- lib/eal/common/rte_service.c | 70 ++++++++++++++++++++++++++++------- lib/eal/include/rte_service.h | 24 ++++++++++-- 3 files changed, 131 insertions(+), 33 deletions(-) diff --git a/app/test/test_service_cores.c b/app/test/test_service_cores.c index c12d52d8f1..13a01b195f 100644 --- a/app/test/test_service_cores.c +++ b/app/test/test_service_cores.c @@ -3,11 +3,12 @@ */ #include +#include #include -#include #include +#include #include -#include +#include #include #include @@ -16,8 +17,10 @@ /* used as the service core ID */ static uint32_t slcore_id; -/* used as timestamp to detect if a service core is running */ -static uint64_t service_tick; +/* track service call count */ +static uint64_t service_calls; +static uint64_t service_idle_calls; +static uint64_t service_error_calls; /* used as a flag to check if a function was run */ static uint32_t service_remote_launch_flag; @@ -46,9 +49,21 @@ testsuite_teardown(void) static int32_t dummy_cb(void *args) { RTE_SET_USED(args); - service_tick++; + + service_calls++; + + switch (rte_rand_max(3)) { + case 0: + return 0; + case 1: + service_idle_calls++; + return -EAGAIN; + default: + service_error_calls++; + return -ENOENT; + } + rte_delay_ms(SERVICE_DELAY); - return 0; } static int32_t dummy_mt_unsafe_cb(void *args) @@ -121,6 +136,10 @@ unregister_all(void) rte_service_lcore_reset_all(); rte_eal_mp_wait_lcore(); + service_calls = 0; + service_idle_calls = 0; + service_error_calls = 0; + return TEST_SUCCESS; } @@ -295,12 +314,19 @@ service_attr_get(void) "Valid attr_get() call didn't return success"); TEST_ASSERT_EQUAL(0, attr_value, "attr_get() call didn't set correct cycles (zero)"); - /* check correct call count */ + /* check correct call counts */ const int attr_calls = RTE_SERVICE_ATTR_CALL_COUNT; TEST_ASSERT_EQUAL(0, rte_service_attr_get(id, attr_calls, &attr_value), "Valid attr_get() call didn't return success"); - TEST_ASSERT_EQUAL(0, attr_value, - "attr_get() call didn't get call count (zero)"); + TEST_ASSERT_EQUAL(0, attr_value, "Call count was not zero"); + const int attr_idle_calls = RTE_SERVICE_ATTR_IDLE_CALL_COUNT; + TEST_ASSERT_EQUAL(0, rte_service_attr_get(id, attr_idle_calls, &attr_value), + "Valid attr_get() call didn't return success"); + TEST_ASSERT_EQUAL(0, attr_value, "Idle call count was not zero"); + const int attr_error_calls = RTE_SERVICE_ATTR_ERROR_CALL_COUNT; + TEST_ASSERT_EQUAL(0, rte_service_attr_get(id, attr_error_calls, &attr_value), + "Valid attr_get() call didn't return success"); + TEST_ASSERT_EQUAL(0, attr_value, "Error call count was not zero"); /* Call service to increment cycle count */ TEST_ASSERT_EQUAL(0, rte_service_lcore_add(slcore_id), @@ -331,8 +357,13 @@ service_attr_get(void) TEST_ASSERT_EQUAL(0, rte_service_attr_get(id, attr_calls, &attr_value), "Valid attr_get() call didn't return success"); - TEST_ASSERT_EQUAL(1, (attr_value > 0), - "attr_get() call didn't get call count (zero)"); + TEST_ASSERT_EQUAL(service_calls, attr_value, "Unexpected call count"); + TEST_ASSERT_EQUAL(0, rte_service_attr_get(id, attr_idle_calls, &attr_value), + "Valid attr_get() call didn't return success"); + TEST_ASSERT_EQUAL(service_idle_calls, attr_value, "Unexpected idle call count"); + TEST_ASSERT_EQUAL(0, rte_service_attr_get(id, attr_error_calls, &attr_value), + "Valid attr_get() call didn't return success"); + TEST_ASSERT_EQUAL(service_error_calls, attr_value, "Unexpected error call count"); TEST_ASSERT_EQUAL(0, rte_service_attr_reset_all(id), "Valid attr_reset_all() return success"); @@ -341,11 +372,16 @@ service_attr_get(void) "Valid attr_get() call didn't return success"); TEST_ASSERT_EQUAL(0, attr_value, "attr_get() call didn't set correct cycles (zero)"); - /* ensure call count > zero */ + /* ensure call counts are zero */ TEST_ASSERT_EQUAL(0, rte_service_attr_get(id, attr_calls, &attr_value), "Valid attr_get() call didn't return success"); - TEST_ASSERT_EQUAL(0, (attr_value > 0), - "attr_get() call didn't get call count (zero)"); + TEST_ASSERT_EQUAL(0, attr_value, "Call count was not reset"); + TEST_ASSERT_EQUAL(0, rte_service_attr_get(id, attr_idle_calls, &attr_value), + "Valid attr_get() call didn't return success"); + TEST_ASSERT_EQUAL(0, attr_value, "Idle call count was not reset"); + TEST_ASSERT_EQUAL(0, rte_service_attr_get(id, attr_error_calls, &attr_value), + "Valid attr_get() call didn't return success"); + TEST_ASSERT_EQUAL(0, attr_value, "Error call count was not reset"); return unregister_all(); } @@ -533,10 +569,10 @@ service_lcore_en_dis_able(void) static int service_lcore_running_check(void) { - uint64_t tick = service_tick; + uint64_t calls = service_calls; rte_delay_ms(SERVICE_DELAY * 100); - /* if (tick != service_tick) we know the lcore as polled the service */ - return tick != service_tick; + bool service_polled = calls != service_calls; + return service_polled; } static int diff --git a/lib/eal/common/rte_service.c b/lib/eal/common/rte_service.c index d959c91459..9438d35c50 100644 --- a/lib/eal/common/rte_service.c +++ b/lib/eal/common/rte_service.c @@ -57,6 +57,8 @@ struct rte_service_spec_impl { struct service_stats { RTE_ATOMIC(uint64_t) calls; + RTE_ATOMIC(uint64_t) idle_calls; + RTE_ATOMIC(uint64_t) error_calls; RTE_ATOMIC(uint64_t) cycles; }; @@ -369,6 +371,16 @@ rte_service_runstate_get(uint32_t id) } +static void +service_counter_add(uint64_t *counter, uint64_t value) +{ + /* The lcore service worker thread is the only writer, and + * thus only a non-atomic load and an atomic store is needed, + * and not the more expensive atomic add. + */ + rte_atomic_store_explicit(counter, *counter + value, rte_memory_order_relaxed); +} + static inline void service_runner_do_callback(struct rte_service_spec_impl *s, struct core_state *cs, uint32_t service_idx) @@ -380,27 +392,23 @@ service_runner_do_callback(struct rte_service_spec_impl *s, uint64_t start = rte_rdtsc(); int rc = s->spec.callback(userdata); - /* The lcore service worker thread is the only writer, - * and thus only a non-atomic load and an atomic store - * is needed, and not the more expensive atomic - * add. - */ struct service_stats *service_stats = &cs->service_stats[service_idx]; + service_counter_add(&service_stats->calls, 1); + + if (rc == -EAGAIN) + service_counter_add(&service_stats->idle_calls, 1); + else if (rc != 0) + service_counter_add(&service_stats->error_calls, 1); + if (likely(rc != -EAGAIN)) { uint64_t end = rte_rdtsc(); uint64_t cycles = end - start; - rte_atomic_store_explicit(&cs->cycles, cs->cycles + cycles, - rte_memory_order_relaxed); - rte_atomic_store_explicit(&service_stats->cycles, - service_stats->cycles + cycles, - rte_memory_order_relaxed); + service_counter_add(&cs->cycles, cycles); + service_counter_add(&service_stats->cycles, cycles); } - - rte_atomic_store_explicit(&service_stats->calls, - service_stats->calls + 1, rte_memory_order_relaxed); } else { s->spec.callback(userdata); } @@ -867,6 +875,24 @@ lcore_attr_get_service_calls(uint32_t service_id, unsigned int lcore) rte_memory_order_relaxed); } +static uint64_t +lcore_attr_get_service_idle_calls(uint32_t service_id, unsigned int lcore) +{ + struct core_state *cs = &lcore_states[lcore]; + + return rte_atomic_load_explicit(&cs->service_stats[service_id].idle_calls, + rte_memory_order_relaxed); +} + +static uint64_t +lcore_attr_get_service_error_calls(uint32_t service_id, unsigned int lcore) +{ + struct core_state *cs = &lcore_states[lcore]; + + return rte_atomic_load_explicit(&cs->service_stats[service_id].error_calls, + rte_memory_order_relaxed); +} + static uint64_t lcore_attr_get_service_cycles(uint32_t service_id, unsigned int lcore) { @@ -899,6 +925,18 @@ attr_get_service_calls(uint32_t service_id) return attr_get(service_id, lcore_attr_get_service_calls); } +static uint64_t +attr_get_service_idle_calls(uint32_t service_id) +{ + return attr_get(service_id, lcore_attr_get_service_idle_calls); +} + +static uint64_t +attr_get_service_error_calls(uint32_t service_id) +{ + return attr_get(service_id, lcore_attr_get_service_error_calls); +} + static uint64_t attr_get_service_cycles(uint32_t service_id) { @@ -918,6 +956,12 @@ rte_service_attr_get(uint32_t id, uint32_t attr_id, uint64_t *attr_value) case RTE_SERVICE_ATTR_CALL_COUNT: *attr_value = attr_get_service_calls(id); return 0; + case RTE_SERVICE_ATTR_IDLE_CALL_COUNT: + *attr_value = attr_get_service_idle_calls(id); + return 0; + case RTE_SERVICE_ATTR_ERROR_CALL_COUNT: + *attr_value = attr_get_service_error_calls(id); + return 0; case RTE_SERVICE_ATTR_CYCLES: *attr_value = attr_get_service_cycles(id); return 0; diff --git a/lib/eal/include/rte_service.h b/lib/eal/include/rte_service.h index e49a7a877e..89057df585 100644 --- a/lib/eal/include/rte_service.h +++ b/lib/eal/include/rte_service.h @@ -374,15 +374,32 @@ int32_t rte_service_lcore_count_services(uint32_t lcore); int32_t rte_service_dump(FILE *f, uint32_t id); /** - * Returns the number of cycles that this service has consumed + * Returns the number of cycles that this service has consumed. Only + * cycles spent in non-idle calls (i.e., calls not returning -EAGAIN) + * count. */ #define RTE_SERVICE_ATTR_CYCLES 0 /** - * Returns the count of invocations of this service function + * Returns the total number of invocations of this service function + * (regardless of return value). */ #define RTE_SERVICE_ATTR_CALL_COUNT 1 +/** + * Returns the number of invocations of this service function where the + * service reported having not performed any useful work (i.e., + * returned -EAGAIN). + */ +#define RTE_SERVICE_ATTR_IDLE_CALL_COUNT 2 + +/** + * Returns the number of invocations of this service function where the + * service reported an error (i.e., the return value was neither 0 nor + * -EAGAIN). + */ +#define RTE_SERVICE_ATTR_ERROR_CALL_COUNT 3 + /** * Get an attribute from a service. * @@ -408,7 +425,8 @@ int32_t rte_service_attr_reset_all(uint32_t id); /** * Returns the total number of cycles that the lcore has spent on - * running services. + * running services. Only non-idle calls (i.e., calls not returning + * -EAGAIN) count toward this total. */ #define RTE_SERVICE_LCORE_ATTR_CYCLES 1