From patchwork Mon Jul 9 10:44:56 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Yipeng1" X-Patchwork-Id: 42642 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 7508F5B12; Mon, 9 Jul 2018 19:52:26 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 80CF82C16 for ; Mon, 9 Jul 2018 19:52:22 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 Jul 2018 10:52:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,330,1526367600"; d="scan'208";a="63369584" Received: from skx-yipeng.jf.intel.com ([10.54.81.175]) by FMSMGA003.fm.intel.com with ESMTP; 09 Jul 2018 10:52:10 -0700 From: Yipeng Wang To: pablo.de.lara.guarch@intel.com Cc: dev@dpdk.org, yipeng1.wang@intel.com, bruce.richardson@intel.com, honnappa.nagarahalli@arm.com, vguvva@caviumnetworks.com, brijesh.s.singh@gmail.com Date: Mon, 9 Jul 2018 03:44:56 -0700 Message-Id: <1531133103-437316-2-git-send-email-yipeng1.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> References: <1528455078-328182-1-git-send-email-yipeng1.wang@intel.com> <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> Subject: [dpdk-dev] [PATCH v4 1/8] hash: fix multiwriter lock memory allocation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" When malloc for multiwriter_lock, the align should be RTE_CACHE_LINE_SIZE rather than LCORE_CACHE_SIZE. Also there should be check to verify the success of rte_malloc. Fixes: be856325cba3 ("hash: add scalable multi-writer insertion with Intel TSX") Cc: stable@dpdk.org Signed-off-by: Yipeng Wang Acked-by: Pablo de Lara --- lib/librte_hash/rte_cuckoo_hash.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c index a07543a..80dcf41 100644 --- a/lib/librte_hash/rte_cuckoo_hash.c +++ b/lib/librte_hash/rte_cuckoo_hash.c @@ -281,7 +281,10 @@ rte_hash_create(const struct rte_hash_parameters *params) h->add_key = ADD_KEY_MULTIWRITER; h->multiwriter_lock = rte_malloc(NULL, sizeof(rte_spinlock_t), - LCORE_CACHE_SIZE); + RTE_CACHE_LINE_SIZE); + if (h->multiwriter_lock == NULL) + goto err_unlock; + rte_spinlock_init(h->multiwriter_lock); } } else From patchwork Mon Jul 9 10:44:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Yipeng1" X-Patchwork-Id: 42645 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 8EC675F51; Mon, 9 Jul 2018 19:52:32 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id DA7C2324B for ; Mon, 9 Jul 2018 19:52:23 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 Jul 2018 10:52:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,330,1526367600"; d="scan'208";a="63369593" Received: from skx-yipeng.jf.intel.com ([10.54.81.175]) by FMSMGA003.fm.intel.com with ESMTP; 09 Jul 2018 10:52:10 -0700 From: Yipeng Wang To: pablo.de.lara.guarch@intel.com Cc: dev@dpdk.org, yipeng1.wang@intel.com, bruce.richardson@intel.com, honnappa.nagarahalli@arm.com, vguvva@caviumnetworks.com, brijesh.s.singh@gmail.com Date: Mon, 9 Jul 2018 03:44:57 -0700 Message-Id: <1531133103-437316-3-git-send-email-yipeng1.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> References: <1528455078-328182-1-git-send-email-yipeng1.wang@intel.com> <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> Subject: [dpdk-dev] [PATCH v4 2/8] hash: fix a multi-writer race condition X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Current multi-writer implementation uses Intel TSX to protect the cuckoo path moving but not the cuckoo path searching. After searching, we need to verify again if the same empty slot still exists at the begining of the TSX region. Otherwise another writer could occupy the empty slot before the TSX region. Current code does not verify. Fixes: be856325cba3 ("hash: add scalable multi-writer insertion with Intel TSX") Cc: stable@dpdk.org Signed-off-by: Yipeng Wang Acked-by: Pablo de Lara --- lib/librte_hash/rte_cuckoo_hash_x86.h | 3 +++ 1 file changed, 3 insertions(+) diff --git a/lib/librte_hash/rte_cuckoo_hash_x86.h b/lib/librte_hash/rte_cuckoo_hash_x86.h index 2c5b017..981d7bd 100644 --- a/lib/librte_hash/rte_cuckoo_hash_x86.h +++ b/lib/librte_hash/rte_cuckoo_hash_x86.h @@ -66,6 +66,9 @@ rte_hash_cuckoo_move_insert_mw_tm(const struct rte_hash *h, while (try < RTE_HASH_TSX_MAX_RETRY) { status = rte_xbegin(); if (likely(status == RTE_XBEGIN_STARTED)) { + /* In case empty slot was gone before entering TSX */ + if (curr_bkt->key_idx[curr_slot] != EMPTY_SLOT) + rte_xabort(RTE_XABORT_CUCKOO_PATH_INVALIDED); while (likely(curr_node->prev != NULL)) { prev_node = curr_node->prev; prev_bkt = prev_node->bkt; From patchwork Mon Jul 9 10:44:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Yipeng1" X-Patchwork-Id: 42643 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id C9ABC5F2B; Mon, 9 Jul 2018 19:52:28 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 2167D2C16 for ; Mon, 9 Jul 2018 19:52:22 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 Jul 2018 10:52:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,330,1526367600"; d="scan'208";a="63369594" Received: from skx-yipeng.jf.intel.com ([10.54.81.175]) by FMSMGA003.fm.intel.com with ESMTP; 09 Jul 2018 10:52:11 -0700 From: Yipeng Wang To: pablo.de.lara.guarch@intel.com Cc: dev@dpdk.org, yipeng1.wang@intel.com, bruce.richardson@intel.com, honnappa.nagarahalli@arm.com, vguvva@caviumnetworks.com, brijesh.s.singh@gmail.com Date: Mon, 9 Jul 2018 03:44:58 -0700 Message-Id: <1531133103-437316-4-git-send-email-yipeng1.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> References: <1528455078-328182-1-git-send-email-yipeng1.wang@intel.com> <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> Subject: [dpdk-dev] [PATCH v4 3/8] hash: fix key slot size accuracy X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit calculates the needed key slot size more accurately. The previous local cache fix requires the free slot ring to be larger than actually needed. The calculation of the value is inaccurate. Fixes: 5915699153d7 ("hash: fix scaling by reducing contention") Cc: stable@dpdk.org Signed-off-by: Yipeng Wang Acked-by: Pablo de Lara --- lib/librte_hash/rte_cuckoo_hash.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c index 80dcf41..11602af 100644 --- a/lib/librte_hash/rte_cuckoo_hash.c +++ b/lib/librte_hash/rte_cuckoo_hash.c @@ -126,13 +126,13 @@ rte_hash_create(const struct rte_hash_parameters *params) * except for the first cache */ num_key_slots = params->entries + (RTE_MAX_LCORE - 1) * - LCORE_CACHE_SIZE + 1; + (LCORE_CACHE_SIZE - 1) + 1; else num_key_slots = params->entries + 1; snprintf(ring_name, sizeof(ring_name), "HT_%s", params->name); /* Create ring (Dummy slot index is not enqueued) */ - r = rte_ring_create(ring_name, rte_align32pow2(num_key_slots - 1), + r = rte_ring_create(ring_name, rte_align32pow2(num_key_slots), params->socket_id, 0); if (r == NULL) { RTE_LOG(ERR, HASH, "memory allocation failed\n"); @@ -291,7 +291,7 @@ rte_hash_create(const struct rte_hash_parameters *params) h->add_key = ADD_KEY_SINGLEWRITER; /* Populate free slots ring. Entry zero is reserved for key misses. */ - for (i = 1; i < params->entries + 1; i++) + for (i = 1; i < num_key_slots; i++) rte_ring_sp_enqueue(r, (void *)((uintptr_t) i)); te->data = (void *) h; @@ -373,7 +373,7 @@ void rte_hash_reset(struct rte_hash *h) { void *ptr; - unsigned i; + uint32_t tot_ring_cnt, i; if (h == NULL) return; @@ -386,7 +386,13 @@ rte_hash_reset(struct rte_hash *h) rte_pause(); /* Repopulate the free slots ring. Entry zero is reserved for key misses */ - for (i = 1; i < h->entries + 1; i++) + if (h->hw_trans_mem_support) + tot_ring_cnt = h->entries + (RTE_MAX_LCORE - 1) * + (LCORE_CACHE_SIZE - 1); + else + tot_ring_cnt = h->entries; + + for (i = 1; i < tot_ring_cnt + 1; i++) rte_ring_sp_enqueue(h->free_slots, (void *)((uintptr_t) i)); if (h->hw_trans_mem_support) { From patchwork Mon Jul 9 10:44:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Yipeng1" X-Patchwork-Id: 42647 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 5D6C11B056; Mon, 9 Jul 2018 19:52:35 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 33ADC378B for ; Mon, 9 Jul 2018 19:52:24 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 Jul 2018 10:52:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,330,1526367600"; d="scan'208";a="63369596" Received: from skx-yipeng.jf.intel.com ([10.54.81.175]) by FMSMGA003.fm.intel.com with ESMTP; 09 Jul 2018 10:52:11 -0700 From: Yipeng Wang To: pablo.de.lara.guarch@intel.com Cc: dev@dpdk.org, yipeng1.wang@intel.com, bruce.richardson@intel.com, honnappa.nagarahalli@arm.com, vguvva@caviumnetworks.com, brijesh.s.singh@gmail.com Date: Mon, 9 Jul 2018 03:44:59 -0700 Message-Id: <1531133103-437316-5-git-send-email-yipeng1.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> References: <1528455078-328182-1-git-send-email-yipeng1.wang@intel.com> <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> Subject: [dpdk-dev] [PATCH v4 4/8] hash: make duplicated code into functions X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commit refactors the hash table lookup/add/del code to remove some code duplication. Processing on primary bucket can also apply to secondary bucket with same code. Signed-off-by: Yipeng Wang Acked-by: Pablo de Lara --- lib/librte_hash/rte_cuckoo_hash.c | 182 +++++++++++++++++++------------------- 1 file changed, 89 insertions(+), 93 deletions(-) diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c index 11602af..b812f33 100644 --- a/lib/librte_hash/rte_cuckoo_hash.c +++ b/lib/librte_hash/rte_cuckoo_hash.c @@ -485,6 +485,33 @@ enqueue_slot_back(const struct rte_hash *h, rte_ring_sp_enqueue(h->free_slots, slot_id); } +/* Search a key from bucket and update its data */ +static inline int32_t +search_and_update(const struct rte_hash *h, void *data, const void *key, + struct rte_hash_bucket *bkt, hash_sig_t sig, hash_sig_t alt_hash) +{ + int i; + struct rte_hash_key *k, *keys = h->key_store; + + for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { + if (bkt->sig_current[i] == sig && + bkt->sig_alt[i] == alt_hash) { + k = (struct rte_hash_key *) ((char *)keys + + bkt->key_idx[i] * h->key_entry_size); + if (rte_hash_cmp_eq(key, k->key, h) == 0) { + /* Update data */ + k->pdata = data; + /* + * Return index where key is stored, + * subtracting the first dummy index + */ + return bkt->key_idx[i] - 1; + } + } + } + return -1; +} + static inline int32_t __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t sig, void *data) @@ -493,7 +520,7 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, uint32_t prim_bucket_idx, sec_bucket_idx; unsigned i; struct rte_hash_bucket *prim_bkt, *sec_bkt; - struct rte_hash_key *new_k, *k, *keys = h->key_store; + struct rte_hash_key *new_k, *keys = h->key_store; void *slot_id = NULL; uint32_t new_idx; int ret; @@ -547,46 +574,14 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, new_idx = (uint32_t)((uintptr_t) slot_id); /* Check if key is already inserted in primary location */ - for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { - if (prim_bkt->sig_current[i] == sig && - prim_bkt->sig_alt[i] == alt_hash) { - k = (struct rte_hash_key *) ((char *)keys + - prim_bkt->key_idx[i] * h->key_entry_size); - if (rte_hash_cmp_eq(key, k->key, h) == 0) { - /* Enqueue index of free slot back in the ring. */ - enqueue_slot_back(h, cached_free_slots, slot_id); - /* Update data */ - k->pdata = data; - /* - * Return index where key is stored, - * subtracting the first dummy index - */ - ret = prim_bkt->key_idx[i] - 1; - goto failure; - } - } - } + ret = search_and_update(h, data, key, prim_bkt, sig, alt_hash); + if (ret != -1) + goto failure; /* Check if key is already inserted in secondary location */ - for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { - if (sec_bkt->sig_alt[i] == sig && - sec_bkt->sig_current[i] == alt_hash) { - k = (struct rte_hash_key *) ((char *)keys + - sec_bkt->key_idx[i] * h->key_entry_size); - if (rte_hash_cmp_eq(key, k->key, h) == 0) { - /* Enqueue index of free slot back in the ring. */ - enqueue_slot_back(h, cached_free_slots, slot_id); - /* Update data */ - k->pdata = data; - /* - * Return index where key is stored, - * subtracting the first dummy index - */ - ret = sec_bkt->key_idx[i] - 1; - goto failure; - } - } - } + ret = search_and_update(h, data, key, sec_bkt, alt_hash, sig); + if (ret != -1) + goto failure; /* Copy key */ rte_memcpy(new_k->key, key, h->key_len); @@ -699,20 +694,15 @@ rte_hash_add_key_data(const struct rte_hash *h, const void *key, void *data) else return ret; } + +/* Search one bucket to find the match key */ static inline int32_t -__rte_hash_lookup_with_hash(const struct rte_hash *h, const void *key, - hash_sig_t sig, void **data) +search_one_bucket(const struct rte_hash *h, const void *key, hash_sig_t sig, + void **data, const struct rte_hash_bucket *bkt) { - uint32_t bucket_idx; - hash_sig_t alt_hash; - unsigned i; - struct rte_hash_bucket *bkt; + int i; struct rte_hash_key *k, *keys = h->key_store; - bucket_idx = sig & h->bucket_bitmask; - bkt = &h->buckets[bucket_idx]; - - /* Check if key is in primary location */ for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { if (bkt->sig_current[i] == sig && bkt->key_idx[i] != EMPTY_SLOT) { @@ -729,6 +719,26 @@ __rte_hash_lookup_with_hash(const struct rte_hash *h, const void *key, } } } + return -1; +} + +static inline int32_t +__rte_hash_lookup_with_hash(const struct rte_hash *h, const void *key, + hash_sig_t sig, void **data) +{ + uint32_t bucket_idx; + hash_sig_t alt_hash; + struct rte_hash_bucket *bkt; + int ret; + + bucket_idx = sig & h->bucket_bitmask; + bkt = &h->buckets[bucket_idx]; + + + /* Check if key is in primary location */ + ret = search_one_bucket(h, key, sig, data, bkt); + if (ret != -1) + return ret; /* Calculate secondary hash */ alt_hash = rte_hash_secondary_hash(sig); @@ -736,22 +746,9 @@ __rte_hash_lookup_with_hash(const struct rte_hash *h, const void *key, bkt = &h->buckets[bucket_idx]; /* Check if key is in secondary location */ - for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { - if (bkt->sig_current[i] == alt_hash && - bkt->sig_alt[i] == sig) { - k = (struct rte_hash_key *) ((char *)keys + - bkt->key_idx[i] * h->key_entry_size); - if (rte_hash_cmp_eq(key, k->key, h) == 0) { - if (data != NULL) - *data = k->pdata; - /* - * Return index where key is stored, - * subtracting the first dummy index - */ - return bkt->key_idx[i] - 1; - } - } - } + ret = search_one_bucket(h, key, alt_hash, data, bkt); + if (ret != -1) + return ret; return -ENOENT; } @@ -815,20 +812,15 @@ remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i) } } +/* Search one bucket and remove the matched key */ static inline int32_t -__rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key, - hash_sig_t sig) +search_and_remove(const struct rte_hash *h, const void *key, + struct rte_hash_bucket *bkt, hash_sig_t sig) { - uint32_t bucket_idx; - hash_sig_t alt_hash; - unsigned i; - struct rte_hash_bucket *bkt; struct rte_hash_key *k, *keys = h->key_store; + unsigned int i; int32_t ret; - bucket_idx = sig & h->bucket_bitmask; - bkt = &h->buckets[bucket_idx]; - /* Check if key is in primary location */ for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { if (bkt->sig_current[i] == sig && @@ -848,31 +840,35 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key, } } } + return -1; +} + +static inline int32_t +__rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key, + hash_sig_t sig) +{ + uint32_t bucket_idx; + hash_sig_t alt_hash; + struct rte_hash_bucket *bkt; + int32_t ret; + + bucket_idx = sig & h->bucket_bitmask; + bkt = &h->buckets[bucket_idx]; + + /* look for key in primary bucket */ + ret = search_and_remove(h, key, bkt, sig); + if (ret != -1) + return ret; /* Calculate secondary hash */ alt_hash = rte_hash_secondary_hash(sig); bucket_idx = alt_hash & h->bucket_bitmask; bkt = &h->buckets[bucket_idx]; - /* Check if key is in secondary location */ - for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { - if (bkt->sig_current[i] == alt_hash && - bkt->key_idx[i] != EMPTY_SLOT) { - k = (struct rte_hash_key *) ((char *)keys + - bkt->key_idx[i] * h->key_entry_size); - if (rte_hash_cmp_eq(key, k->key, h) == 0) { - remove_entry(h, bkt, i); - - /* - * Return index where key is stored, - * subtracting the first dummy index - */ - ret = bkt->key_idx[i] - 1; - bkt->key_idx[i] = EMPTY_SLOT; - return ret; - } - } - } + /* look for key in secondary bucket */ + ret = search_and_remove(h, key, bkt, alt_hash); + if (ret != -1) + return ret; return -ENOENT; } From patchwork Mon Jul 9 10:45:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Yipeng1" X-Patchwork-Id: 42646 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id F00C11B027; Mon, 9 Jul 2018 19:52:33 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id D7EEB2C16 for ; Mon, 9 Jul 2018 19:52:23 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 Jul 2018 10:52:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,330,1526367600"; d="scan'208";a="63369598" Received: from skx-yipeng.jf.intel.com ([10.54.81.175]) by FMSMGA003.fm.intel.com with ESMTP; 09 Jul 2018 10:52:12 -0700 From: Yipeng Wang To: pablo.de.lara.guarch@intel.com Cc: dev@dpdk.org, yipeng1.wang@intel.com, bruce.richardson@intel.com, honnappa.nagarahalli@arm.com, vguvva@caviumnetworks.com, brijesh.s.singh@gmail.com Date: Mon, 9 Jul 2018 03:45:00 -0700 Message-Id: <1531133103-437316-6-git-send-email-yipeng1.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> References: <1528455078-328182-1-git-send-email-yipeng1.wang@intel.com> <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> Subject: [dpdk-dev] [PATCH v4 5/8] hash: add read and write concurrency support X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" The existing implementation of librte_hash does not support read-write concurrency. This commit implements read-write safety using rte_rwlock and rte_rwlock TM version if hardware transactional memory is available. Both multi-writer and read-write concurrency is protected by rte_rwlock now. The x86 specific header file is removed since the x86 specific RTM function is not called directly by rte hash now. Signed-off-by: Yipeng Wang Acked-by: Pablo de Lara --- lib/librte_hash/meson.build | 1 - lib/librte_hash/rte_cuckoo_hash.c | 520 ++++++++++++++++++++++------------ lib/librte_hash/rte_cuckoo_hash.h | 18 +- lib/librte_hash/rte_cuckoo_hash_x86.h | 167 ----------- lib/librte_hash/rte_hash.h | 3 + 5 files changed, 348 insertions(+), 361 deletions(-) delete mode 100644 lib/librte_hash/rte_cuckoo_hash_x86.h diff --git a/lib/librte_hash/meson.build b/lib/librte_hash/meson.build index e139e1d..efc06ed 100644 --- a/lib/librte_hash/meson.build +++ b/lib/librte_hash/meson.build @@ -6,7 +6,6 @@ headers = files('rte_cmp_arm64.h', 'rte_cmp_x86.h', 'rte_crc_arm64.h', 'rte_cuckoo_hash.h', - 'rte_cuckoo_hash_x86.h', 'rte_fbk_hash.h', 'rte_hash_crc.h', 'rte_hash.h', diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c index b812f33..35631cc 100644 --- a/lib/librte_hash/rte_cuckoo_hash.c +++ b/lib/librte_hash/rte_cuckoo_hash.c @@ -31,9 +31,6 @@ #include "rte_hash.h" #include "rte_cuckoo_hash.h" -#if defined(RTE_ARCH_X86) -#include "rte_cuckoo_hash_x86.h" -#endif TAILQ_HEAD(rte_hash_list, rte_tailq_entry); @@ -93,8 +90,10 @@ rte_hash_create(const struct rte_hash_parameters *params) void *buckets = NULL; char ring_name[RTE_RING_NAMESIZE]; unsigned num_key_slots; - unsigned hw_trans_mem_support = 0; unsigned i; + unsigned int hw_trans_mem_support = 0, multi_writer_support = 0; + unsigned int readwrite_concur_support = 0; + rte_hash_function default_hash_func = (rte_hash_function)rte_jhash; hash_list = RTE_TAILQ_CAST(rte_hash_tailq.head, rte_hash_list); @@ -118,8 +117,16 @@ rte_hash_create(const struct rte_hash_parameters *params) if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_TRANS_MEM_SUPPORT) hw_trans_mem_support = 1; + if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD) + multi_writer_support = 1; + + if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY) { + readwrite_concur_support = 1; + multi_writer_support = 1; + } + /* Store all keys and leave the first entry as a dummy entry for lookup_bulk */ - if (hw_trans_mem_support) + if (multi_writer_support) /* * Increase number of slots by total number of indices * that can be stored in the lcore caches @@ -233,7 +240,7 @@ rte_hash_create(const struct rte_hash_parameters *params) h->cmp_jump_table_idx = KEY_OTHER_BYTES; #endif - if (hw_trans_mem_support) { + if (multi_writer_support) { h->local_free_slots = rte_zmalloc_socket(NULL, sizeof(struct lcore_cache) * RTE_MAX_LCORE, RTE_CACHE_LINE_SIZE, params->socket_id); @@ -261,6 +268,8 @@ rte_hash_create(const struct rte_hash_parameters *params) h->key_store = k; h->free_slots = r; h->hw_trans_mem_support = hw_trans_mem_support; + h->multi_writer_support = multi_writer_support; + h->readwrite_concur_support = readwrite_concur_support; #if defined(RTE_ARCH_X86) if (rte_cpu_get_flag_enabled(RTE_CPUFLAG_AVX2)) @@ -271,24 +280,17 @@ rte_hash_create(const struct rte_hash_parameters *params) #endif h->sig_cmp_fn = RTE_HASH_COMPARE_SCALAR; - /* Turn on multi-writer only with explicit flat from user and TM + /* Turn on multi-writer only with explicit flag from user and TM * support. */ - if (params->extra_flag & RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD) { - if (h->hw_trans_mem_support) { - h->add_key = ADD_KEY_MULTIWRITER_TM; - } else { - h->add_key = ADD_KEY_MULTIWRITER; - h->multiwriter_lock = rte_malloc(NULL, - sizeof(rte_spinlock_t), - RTE_CACHE_LINE_SIZE); - if (h->multiwriter_lock == NULL) - goto err_unlock; - - rte_spinlock_init(h->multiwriter_lock); - } - } else - h->add_key = ADD_KEY_SINGLEWRITER; + if (h->multi_writer_support) { + h->readwrite_lock = rte_malloc(NULL, sizeof(rte_rwlock_t), + RTE_CACHE_LINE_SIZE); + if (h->readwrite_lock == NULL) + goto err_unlock; + + rte_rwlock_init(h->readwrite_lock); + } /* Populate free slots ring. Entry zero is reserved for key misses. */ for (i = 1; i < num_key_slots; i++) @@ -338,11 +340,10 @@ rte_hash_free(struct rte_hash *h) rte_rwlock_write_unlock(RTE_EAL_TAILQ_RWLOCK); - if (h->hw_trans_mem_support) + if (h->multi_writer_support) { rte_free(h->local_free_slots); - - if (h->add_key == ADD_KEY_MULTIWRITER) - rte_free(h->multiwriter_lock); + rte_free(h->readwrite_lock); + } rte_ring_free(h->free_slots); rte_free(h->key_store); rte_free(h->buckets); @@ -369,6 +370,44 @@ rte_hash_secondary_hash(const hash_sig_t primary_hash) return primary_hash ^ ((tag + 1) * alt_bits_xor); } +/* Read write locks implemented using rte_rwlock */ +static inline void +__hash_rw_writer_lock(const struct rte_hash *h) +{ + if (h->multi_writer_support && h->hw_trans_mem_support) + rte_rwlock_write_lock_tm(h->readwrite_lock); + else if (h->multi_writer_support) + rte_rwlock_write_lock(h->readwrite_lock); +} + + +static inline void +__hash_rw_reader_lock(const struct rte_hash *h) +{ + if (h->readwrite_concur_support && h->hw_trans_mem_support) + rte_rwlock_read_lock_tm(h->readwrite_lock); + else if (h->readwrite_concur_support) + rte_rwlock_read_lock(h->readwrite_lock); +} + +static inline void +__hash_rw_writer_unlock(const struct rte_hash *h) +{ + if (h->multi_writer_support && h->hw_trans_mem_support) + rte_rwlock_write_unlock_tm(h->readwrite_lock); + else if (h->multi_writer_support) + rte_rwlock_write_unlock(h->readwrite_lock); +} + +static inline void +__hash_rw_reader_unlock(const struct rte_hash *h) +{ + if (h->readwrite_concur_support && h->hw_trans_mem_support) + rte_rwlock_read_unlock_tm(h->readwrite_lock); + else if (h->readwrite_concur_support) + rte_rwlock_read_unlock(h->readwrite_lock); +} + void rte_hash_reset(struct rte_hash *h) { @@ -378,6 +417,7 @@ rte_hash_reset(struct rte_hash *h) if (h == NULL) return; + __hash_rw_writer_lock(h); memset(h->buckets, 0, h->num_buckets * sizeof(struct rte_hash_bucket)); memset(h->key_store, 0, h->key_entry_size * (h->entries + 1)); @@ -386,7 +426,7 @@ rte_hash_reset(struct rte_hash *h) rte_pause(); /* Repopulate the free slots ring. Entry zero is reserved for key misses */ - if (h->hw_trans_mem_support) + if (h->multi_writer_support) tot_ring_cnt = h->entries + (RTE_MAX_LCORE - 1) * (LCORE_CACHE_SIZE - 1); else @@ -395,77 +435,12 @@ rte_hash_reset(struct rte_hash *h) for (i = 1; i < tot_ring_cnt + 1; i++) rte_ring_sp_enqueue(h->free_slots, (void *)((uintptr_t) i)); - if (h->hw_trans_mem_support) { + if (h->multi_writer_support) { /* Reset local caches per lcore */ for (i = 0; i < RTE_MAX_LCORE; i++) h->local_free_slots[i].len = 0; } -} - -/* Search for an entry that can be pushed to its alternative location */ -static inline int -make_space_bucket(const struct rte_hash *h, struct rte_hash_bucket *bkt, - unsigned int *nr_pushes) -{ - unsigned i, j; - int ret; - uint32_t next_bucket_idx; - struct rte_hash_bucket *next_bkt[RTE_HASH_BUCKET_ENTRIES]; - - /* - * Push existing item (search for bucket with space in - * alternative locations) to its alternative location - */ - for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { - /* Search for space in alternative locations */ - next_bucket_idx = bkt->sig_alt[i] & h->bucket_bitmask; - next_bkt[i] = &h->buckets[next_bucket_idx]; - for (j = 0; j < RTE_HASH_BUCKET_ENTRIES; j++) { - if (next_bkt[i]->key_idx[j] == EMPTY_SLOT) - break; - } - - if (j != RTE_HASH_BUCKET_ENTRIES) - break; - } - - /* Alternative location has spare room (end of recursive function) */ - if (i != RTE_HASH_BUCKET_ENTRIES) { - next_bkt[i]->sig_alt[j] = bkt->sig_current[i]; - next_bkt[i]->sig_current[j] = bkt->sig_alt[i]; - next_bkt[i]->key_idx[j] = bkt->key_idx[i]; - return i; - } - - /* Pick entry that has not been pushed yet */ - for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) - if (bkt->flag[i] == 0) - break; - - /* All entries have been pushed, so entry cannot be added */ - if (i == RTE_HASH_BUCKET_ENTRIES || ++(*nr_pushes) > RTE_HASH_MAX_PUSHES) - return -ENOSPC; - - /* Set flag to indicate that this entry is going to be pushed */ - bkt->flag[i] = 1; - - /* Need room in alternative bucket to insert the pushed entry */ - ret = make_space_bucket(h, next_bkt[i], nr_pushes); - /* - * After recursive function. - * Clear flags and insert the pushed entry - * in its alternative location if successful, - * or return error - */ - bkt->flag[i] = 0; - if (ret >= 0) { - next_bkt[i]->sig_alt[ret] = bkt->sig_current[i]; - next_bkt[i]->sig_current[ret] = bkt->sig_alt[i]; - next_bkt[i]->key_idx[ret] = bkt->key_idx[i]; - return i; - } else - return ret; - + __hash_rw_writer_unlock(h); } /* @@ -478,7 +453,7 @@ enqueue_slot_back(const struct rte_hash *h, struct lcore_cache *cached_free_slots, void *slot_id) { - if (h->hw_trans_mem_support) { + if (h->multi_writer_support) { cached_free_slots->objs[cached_free_slots->len] = slot_id; cached_free_slots->len++; } else @@ -512,13 +487,207 @@ search_and_update(const struct rte_hash *h, void *data, const void *key, return -1; } +/* Only tries to insert at one bucket (@prim_bkt) without trying to push + * buckets around. + * return 1 if matching existing key, return 0 if succeeds, return -1 for no + * empty entry. + */ +static inline int32_t +rte_hash_cuckoo_insert_mw(const struct rte_hash *h, + struct rte_hash_bucket *prim_bkt, + struct rte_hash_bucket *sec_bkt, + const struct rte_hash_key *key, void *data, + hash_sig_t sig, hash_sig_t alt_hash, uint32_t new_idx, + int32_t *ret_val) +{ + unsigned int i; + struct rte_hash_bucket *cur_bkt = prim_bkt; + int32_t ret; + + __hash_rw_writer_lock(h); + /* Check if key was inserted after last check but before this + * protected region in case of inserting duplicated keys. + */ + ret = search_and_update(h, data, key, cur_bkt, sig, alt_hash); + if (ret != -1) { + __hash_rw_writer_unlock(h); + *ret_val = ret; + return 1; + } + ret = search_and_update(h, data, key, sec_bkt, alt_hash, sig); + if (ret != -1) { + __hash_rw_writer_unlock(h); + *ret_val = ret; + return 1; + } + + /* Insert new entry if there is room in the primary + * bucket. + */ + for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { + /* Check if slot is available */ + if (likely(prim_bkt->key_idx[i] == EMPTY_SLOT)) { + prim_bkt->sig_current[i] = sig; + prim_bkt->sig_alt[i] = alt_hash; + prim_bkt->key_idx[i] = new_idx; + break; + } + } + __hash_rw_writer_unlock(h); + + if (i != RTE_HASH_BUCKET_ENTRIES) + return 0; + + /* no empty entry */ + return -1; +} + +/* Shift buckets along provided cuckoo_path (@leaf and @leaf_slot) and fill + * the path head with new entry (sig, alt_hash, new_idx) + * return 1 if matched key found, return -1 if cuckoo path invalided and fail, + * return 0 if succeeds. + */ +static inline int +rte_hash_cuckoo_move_insert_mw(const struct rte_hash *h, + struct rte_hash_bucket *bkt, + struct rte_hash_bucket *alt_bkt, + const struct rte_hash_key *key, void *data, + struct queue_node *leaf, uint32_t leaf_slot, + hash_sig_t sig, hash_sig_t alt_hash, uint32_t new_idx, + int32_t *ret_val) +{ + uint32_t prev_alt_bkt_idx; + struct rte_hash_bucket *cur_bkt = bkt; + struct queue_node *prev_node, *curr_node = leaf; + struct rte_hash_bucket *prev_bkt, *curr_bkt = leaf->bkt; + uint32_t prev_slot, curr_slot = leaf_slot; + int32_t ret; + + __hash_rw_writer_lock(h); + + /* In case empty slot was gone before entering protected region */ + if (curr_bkt->key_idx[curr_slot] != EMPTY_SLOT) { + __hash_rw_writer_unlock(h); + return -1; + } + + /* Check if key was inserted after last check but before this + * protected region. + */ + ret = search_and_update(h, data, key, cur_bkt, sig, alt_hash); + if (ret != -1) { + __hash_rw_writer_unlock(h); + *ret_val = ret; + return 1; + } + + ret = search_and_update(h, data, key, alt_bkt, alt_hash, sig); + if (ret != -1) { + __hash_rw_writer_unlock(h); + *ret_val = ret; + return 1; + } + + while (likely(curr_node->prev != NULL)) { + prev_node = curr_node->prev; + prev_bkt = prev_node->bkt; + prev_slot = curr_node->prev_slot; + + prev_alt_bkt_idx = + prev_bkt->sig_alt[prev_slot] & h->bucket_bitmask; + + if (unlikely(&h->buckets[prev_alt_bkt_idx] + != curr_bkt)) { + /* revert it to empty, otherwise duplicated keys */ + curr_bkt->key_idx[curr_slot] = EMPTY_SLOT; + __hash_rw_writer_unlock(h); + return -1; + } + + /* Need to swap current/alt sig to allow later + * Cuckoo insert to move elements back to its + * primary bucket if available + */ + curr_bkt->sig_alt[curr_slot] = + prev_bkt->sig_current[prev_slot]; + curr_bkt->sig_current[curr_slot] = + prev_bkt->sig_alt[prev_slot]; + curr_bkt->key_idx[curr_slot] = + prev_bkt->key_idx[prev_slot]; + + curr_slot = prev_slot; + curr_node = prev_node; + curr_bkt = curr_node->bkt; + } + + curr_bkt->sig_current[curr_slot] = sig; + curr_bkt->sig_alt[curr_slot] = alt_hash; + curr_bkt->key_idx[curr_slot] = new_idx; + + __hash_rw_writer_unlock(h); + + return 0; + +} + +/* + * Make space for new key, using bfs Cuckoo Search and Multi-Writer safe + * Cuckoo + */ +static inline int +rte_hash_cuckoo_make_space_mw(const struct rte_hash *h, + struct rte_hash_bucket *bkt, + struct rte_hash_bucket *sec_bkt, + const struct rte_hash_key *key, void *data, + hash_sig_t sig, hash_sig_t alt_hash, + uint32_t new_idx, int32_t *ret_val) +{ + unsigned int i; + struct queue_node queue[RTE_HASH_BFS_QUEUE_MAX_LEN]; + struct queue_node *tail, *head; + struct rte_hash_bucket *curr_bkt, *alt_bkt; + + tail = queue; + head = queue + 1; + tail->bkt = bkt; + tail->prev = NULL; + tail->prev_slot = -1; + + /* Cuckoo bfs Search */ + while (likely(tail != head && head < + queue + RTE_HASH_BFS_QUEUE_MAX_LEN - + RTE_HASH_BUCKET_ENTRIES)) { + curr_bkt = tail->bkt; + for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { + if (curr_bkt->key_idx[i] == EMPTY_SLOT) { + int32_t ret = rte_hash_cuckoo_move_insert_mw(h, + bkt, sec_bkt, key, data, + tail, i, sig, alt_hash, + new_idx, ret_val); + if (likely(ret != -1)) + return ret; + } + + /* Enqueue new node and keep prev node info */ + alt_bkt = &(h->buckets[curr_bkt->sig_alt[i] + & h->bucket_bitmask]); + head->bkt = alt_bkt; + head->prev = tail; + head->prev_slot = i; + head++; + } + tail++; + } + + return -ENOSPC; +} + static inline int32_t __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, hash_sig_t sig, void *data) { hash_sig_t alt_hash; uint32_t prim_bucket_idx, sec_bucket_idx; - unsigned i; struct rte_hash_bucket *prim_bkt, *sec_bkt; struct rte_hash_key *new_k, *keys = h->key_store; void *slot_id = NULL; @@ -527,10 +696,7 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, unsigned n_slots; unsigned lcore_id; struct lcore_cache *cached_free_slots = NULL; - unsigned int nr_pushes = 0; - - if (h->add_key == ADD_KEY_MULTIWRITER) - rte_spinlock_lock(h->multiwriter_lock); + int32_t ret_val; prim_bucket_idx = sig & h->bucket_bitmask; prim_bkt = &h->buckets[prim_bucket_idx]; @@ -541,8 +707,24 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, sec_bkt = &h->buckets[sec_bucket_idx]; rte_prefetch0(sec_bkt); - /* Get a new slot for storing the new key */ - if (h->hw_trans_mem_support) { + /* Check if key is already inserted in primary location */ + __hash_rw_writer_lock(h); + ret = search_and_update(h, data, key, prim_bkt, sig, alt_hash); + if (ret != -1) { + __hash_rw_writer_unlock(h); + return ret; + } + + /* Check if key is already inserted in secondary location */ + ret = search_and_update(h, data, key, sec_bkt, alt_hash, sig); + if (ret != -1) { + __hash_rw_writer_unlock(h); + return ret; + } + __hash_rw_writer_unlock(h); + + /* Did not find a match, so get a new slot for storing the new key */ + if (h->multi_writer_support) { lcore_id = rte_lcore_id(); cached_free_slots = &h->local_free_slots[lcore_id]; /* Try to get a free slot from the local cache */ @@ -552,8 +734,7 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, cached_free_slots->objs, LCORE_CACHE_SIZE, NULL); if (n_slots == 0) { - ret = -ENOSPC; - goto failure; + return -ENOSPC; } cached_free_slots->len += n_slots; @@ -564,92 +745,50 @@ __rte_hash_add_key_with_hash(const struct rte_hash *h, const void *key, slot_id = cached_free_slots->objs[cached_free_slots->len]; } else { if (rte_ring_sc_dequeue(h->free_slots, &slot_id) != 0) { - ret = -ENOSPC; - goto failure; + return -ENOSPC; } } new_k = RTE_PTR_ADD(keys, (uintptr_t)slot_id * h->key_entry_size); - rte_prefetch0(new_k); new_idx = (uint32_t)((uintptr_t) slot_id); - - /* Check if key is already inserted in primary location */ - ret = search_and_update(h, data, key, prim_bkt, sig, alt_hash); - if (ret != -1) - goto failure; - - /* Check if key is already inserted in secondary location */ - ret = search_and_update(h, data, key, sec_bkt, alt_hash, sig); - if (ret != -1) - goto failure; - /* Copy key */ rte_memcpy(new_k->key, key, h->key_len); new_k->pdata = data; -#if defined(RTE_ARCH_X86) /* currently only x86 support HTM */ - if (h->add_key == ADD_KEY_MULTIWRITER_TM) { - ret = rte_hash_cuckoo_insert_mw_tm(prim_bkt, - sig, alt_hash, new_idx); - if (ret >= 0) - return new_idx - 1; - /* Primary bucket full, need to make space for new entry */ - ret = rte_hash_cuckoo_make_space_mw_tm(h, prim_bkt, sig, - alt_hash, new_idx); + /* Find an empty slot and insert */ + ret = rte_hash_cuckoo_insert_mw(h, prim_bkt, sec_bkt, key, data, + sig, alt_hash, new_idx, &ret_val); + if (ret == 0) + return new_idx - 1; + else if (ret == 1) { + enqueue_slot_back(h, cached_free_slots, slot_id); + return ret_val; + } - if (ret >= 0) - return new_idx - 1; + /* Primary bucket full, need to make space for new entry */ + ret = rte_hash_cuckoo_make_space_mw(h, prim_bkt, sec_bkt, key, data, + sig, alt_hash, new_idx, &ret_val); + if (ret == 0) + return new_idx - 1; + else if (ret == 1) { + enqueue_slot_back(h, cached_free_slots, slot_id); + return ret_val; + } - /* Also search secondary bucket to get better occupancy */ - ret = rte_hash_cuckoo_make_space_mw_tm(h, sec_bkt, sig, - alt_hash, new_idx); + /* Also search secondary bucket to get better occupancy */ + ret = rte_hash_cuckoo_make_space_mw(h, sec_bkt, prim_bkt, key, data, + alt_hash, sig, new_idx, &ret_val); - if (ret >= 0) - return new_idx - 1; + if (ret == 0) + return new_idx - 1; + else if (ret == 1) { + enqueue_slot_back(h, cached_free_slots, slot_id); + return ret_val; } else { -#endif - for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { - /* Check if slot is available */ - if (likely(prim_bkt->key_idx[i] == EMPTY_SLOT)) { - prim_bkt->sig_current[i] = sig; - prim_bkt->sig_alt[i] = alt_hash; - prim_bkt->key_idx[i] = new_idx; - break; - } - } - - if (i != RTE_HASH_BUCKET_ENTRIES) { - if (h->add_key == ADD_KEY_MULTIWRITER) - rte_spinlock_unlock(h->multiwriter_lock); - return new_idx - 1; - } - - /* Primary bucket full, need to make space for new entry - * After recursive function. - * Insert the new entry in the position of the pushed entry - * if successful or return error and - * store the new slot back in the ring - */ - ret = make_space_bucket(h, prim_bkt, &nr_pushes); - if (ret >= 0) { - prim_bkt->sig_current[ret] = sig; - prim_bkt->sig_alt[ret] = alt_hash; - prim_bkt->key_idx[ret] = new_idx; - if (h->add_key == ADD_KEY_MULTIWRITER) - rte_spinlock_unlock(h->multiwriter_lock); - return new_idx - 1; - } -#if defined(RTE_ARCH_X86) + enqueue_slot_back(h, cached_free_slots, slot_id); + return ret; } -#endif - /* Error in addition, store new slot back in the ring and return error */ - enqueue_slot_back(h, cached_free_slots, (void *)((uintptr_t) new_idx)); - -failure: - if (h->add_key == ADD_KEY_MULTIWRITER) - rte_spinlock_unlock(h->multiwriter_lock); - return ret; } int32_t @@ -734,12 +873,14 @@ __rte_hash_lookup_with_hash(const struct rte_hash *h, const void *key, bucket_idx = sig & h->bucket_bitmask; bkt = &h->buckets[bucket_idx]; + __hash_rw_reader_lock(h); /* Check if key is in primary location */ ret = search_one_bucket(h, key, sig, data, bkt); - if (ret != -1) + if (ret != -1) { + __hash_rw_reader_unlock(h); return ret; - + } /* Calculate secondary hash */ alt_hash = rte_hash_secondary_hash(sig); bucket_idx = alt_hash & h->bucket_bitmask; @@ -747,9 +888,11 @@ __rte_hash_lookup_with_hash(const struct rte_hash *h, const void *key, /* Check if key is in secondary location */ ret = search_one_bucket(h, key, alt_hash, data, bkt); - if (ret != -1) + if (ret != -1) { + __hash_rw_reader_unlock(h); return ret; - + } + __hash_rw_reader_unlock(h); return -ENOENT; } @@ -791,7 +934,7 @@ remove_entry(const struct rte_hash *h, struct rte_hash_bucket *bkt, unsigned i) bkt->sig_current[i] = NULL_SIGNATURE; bkt->sig_alt[i] = NULL_SIGNATURE; - if (h->hw_trans_mem_support) { + if (h->multi_writer_support) { lcore_id = rte_lcore_id(); cached_free_slots = &h->local_free_slots[lcore_id]; /* Cache full, need to free it. */ @@ -855,10 +998,13 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key, bucket_idx = sig & h->bucket_bitmask; bkt = &h->buckets[bucket_idx]; + __hash_rw_writer_lock(h); /* look for key in primary bucket */ ret = search_and_remove(h, key, bkt, sig); - if (ret != -1) + if (ret != -1) { + __hash_rw_writer_unlock(h); return ret; + } /* Calculate secondary hash */ alt_hash = rte_hash_secondary_hash(sig); @@ -867,9 +1013,12 @@ __rte_hash_del_key_with_hash(const struct rte_hash *h, const void *key, /* look for key in secondary bucket */ ret = search_and_remove(h, key, bkt, alt_hash); - if (ret != -1) + if (ret != -1) { + __hash_rw_writer_unlock(h); return ret; + } + __hash_rw_writer_unlock(h); return -ENOENT; } @@ -1011,6 +1160,7 @@ __rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys, rte_prefetch0(secondary_bkt[i]); } + __hash_rw_reader_lock(h); /* Compare signatures and prefetch key slot of first hit */ for (i = 0; i < num_keys; i++) { compare_signatures(&prim_hitmask[i], &sec_hitmask[i], @@ -1093,6 +1243,8 @@ __rte_hash_lookup_bulk(const struct rte_hash *h, const void **keys, continue; } + __hash_rw_reader_unlock(h); + if (hit_mask != NULL) *hit_mask = hits; } @@ -1151,7 +1303,7 @@ rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32 bucket_idx = *next / RTE_HASH_BUCKET_ENTRIES; idx = *next % RTE_HASH_BUCKET_ENTRIES; } - + __hash_rw_reader_lock(h); /* Get position of entry in key table */ position = h->buckets[bucket_idx].key_idx[idx]; next_key = (struct rte_hash_key *) ((char *)h->key_store + @@ -1160,6 +1312,8 @@ rte_hash_iterate(const struct rte_hash *h, const void **key, void **data, uint32 *key = next_key->key; *data = next_key->pdata; + __hash_rw_reader_unlock(h); + /* Increment iterator */ (*next)++; diff --git a/lib/librte_hash/rte_cuckoo_hash.h b/lib/librte_hash/rte_cuckoo_hash.h index 7a54e55..db4d1a0 100644 --- a/lib/librte_hash/rte_cuckoo_hash.h +++ b/lib/librte_hash/rte_cuckoo_hash.h @@ -88,11 +88,6 @@ const rte_hash_cmp_eq_t cmp_jump_table[NUM_KEY_CMP_CASES] = { #endif -enum add_key_case { - ADD_KEY_SINGLEWRITER = 0, - ADD_KEY_MULTIWRITER, - ADD_KEY_MULTIWRITER_TM, -}; /** Number of items per bucket. */ #define RTE_HASH_BUCKET_ENTRIES 8 @@ -155,18 +150,20 @@ struct rte_hash { struct rte_ring *free_slots; /**< Ring that stores all indexes of the free slots in the key table */ - uint8_t hw_trans_mem_support; - /**< Hardware transactional memory support */ + struct lcore_cache *local_free_slots; /**< Local cache per lcore, storing some indexes of the free slots */ - enum add_key_case add_key; /**< Multi-writer hash add behavior */ - - rte_spinlock_t *multiwriter_lock; /**< Multi-writer spinlock for w/o TM */ /* Fields used in lookup */ uint32_t key_len __rte_cache_aligned; /**< Length of hash key. */ + uint8_t hw_trans_mem_support; + /**< If hardware transactional memory is used. */ + uint8_t multi_writer_support; + /**< If multi-writer support is enabled. */ + uint8_t readwrite_concur_support; + /**< If read-write concurrency support is enabled */ rte_hash_function hash_func; /**< Function used to calculate hash. */ uint32_t hash_func_init_val; /**< Init value used by hash_func. */ rte_hash_cmp_eq_t rte_hash_custom_cmp_eq; @@ -184,6 +181,7 @@ struct rte_hash { /**< Table with buckets storing all the hash values and key indexes * to the key table. */ + rte_rwlock_t *readwrite_lock; /**< Read-write lock thread-safety. */ } __rte_cache_aligned; struct queue_node { diff --git a/lib/librte_hash/rte_cuckoo_hash_x86.h b/lib/librte_hash/rte_cuckoo_hash_x86.h deleted file mode 100644 index 981d7bd..0000000 --- a/lib/librte_hash/rte_cuckoo_hash_x86.h +++ /dev/null @@ -1,167 +0,0 @@ -/* SPDX-License-Identifier: BSD-3-Clause - * Copyright(c) 2016 Intel Corporation - */ - -/* rte_cuckoo_hash_x86.h - * This file holds all x86 specific Cuckoo Hash functions - */ - -/* Only tries to insert at one bucket (@prim_bkt) without trying to push - * buckets around - */ -static inline unsigned -rte_hash_cuckoo_insert_mw_tm(struct rte_hash_bucket *prim_bkt, - hash_sig_t sig, hash_sig_t alt_hash, uint32_t new_idx) -{ - unsigned i, status; - unsigned try = 0; - - while (try < RTE_HASH_TSX_MAX_RETRY) { - status = rte_xbegin(); - if (likely(status == RTE_XBEGIN_STARTED)) { - /* Insert new entry if there is room in the primary - * bucket. - */ - for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { - /* Check if slot is available */ - if (likely(prim_bkt->key_idx[i] == EMPTY_SLOT)) { - prim_bkt->sig_current[i] = sig; - prim_bkt->sig_alt[i] = alt_hash; - prim_bkt->key_idx[i] = new_idx; - break; - } - } - rte_xend(); - - if (i != RTE_HASH_BUCKET_ENTRIES) - return 0; - - break; /* break off try loop if transaction commits */ - } else { - /* If we abort we give up this cuckoo path. */ - try++; - rte_pause(); - } - } - - return -1; -} - -/* Shift buckets along provided cuckoo_path (@leaf and @leaf_slot) and fill - * the path head with new entry (sig, alt_hash, new_idx) - */ -static inline int -rte_hash_cuckoo_move_insert_mw_tm(const struct rte_hash *h, - struct queue_node *leaf, uint32_t leaf_slot, - hash_sig_t sig, hash_sig_t alt_hash, uint32_t new_idx) -{ - unsigned try = 0; - unsigned status; - uint32_t prev_alt_bkt_idx; - - struct queue_node *prev_node, *curr_node = leaf; - struct rte_hash_bucket *prev_bkt, *curr_bkt = leaf->bkt; - uint32_t prev_slot, curr_slot = leaf_slot; - - while (try < RTE_HASH_TSX_MAX_RETRY) { - status = rte_xbegin(); - if (likely(status == RTE_XBEGIN_STARTED)) { - /* In case empty slot was gone before entering TSX */ - if (curr_bkt->key_idx[curr_slot] != EMPTY_SLOT) - rte_xabort(RTE_XABORT_CUCKOO_PATH_INVALIDED); - while (likely(curr_node->prev != NULL)) { - prev_node = curr_node->prev; - prev_bkt = prev_node->bkt; - prev_slot = curr_node->prev_slot; - - prev_alt_bkt_idx - = prev_bkt->sig_alt[prev_slot] - & h->bucket_bitmask; - - if (unlikely(&h->buckets[prev_alt_bkt_idx] - != curr_bkt)) { - rte_xabort(RTE_XABORT_CUCKOO_PATH_INVALIDED); - } - - /* Need to swap current/alt sig to allow later - * Cuckoo insert to move elements back to its - * primary bucket if available - */ - curr_bkt->sig_alt[curr_slot] = - prev_bkt->sig_current[prev_slot]; - curr_bkt->sig_current[curr_slot] = - prev_bkt->sig_alt[prev_slot]; - curr_bkt->key_idx[curr_slot] - = prev_bkt->key_idx[prev_slot]; - - curr_slot = prev_slot; - curr_node = prev_node; - curr_bkt = curr_node->bkt; - } - - curr_bkt->sig_current[curr_slot] = sig; - curr_bkt->sig_alt[curr_slot] = alt_hash; - curr_bkt->key_idx[curr_slot] = new_idx; - - rte_xend(); - - return 0; - } - - /* If we abort we give up this cuckoo path, since most likely it's - * no longer valid as TSX detected data conflict - */ - try++; - rte_pause(); - } - - return -1; -} - -/* - * Make space for new key, using bfs Cuckoo Search and Multi-Writer safe - * Cuckoo - */ -static inline int -rte_hash_cuckoo_make_space_mw_tm(const struct rte_hash *h, - struct rte_hash_bucket *bkt, - hash_sig_t sig, hash_sig_t alt_hash, - uint32_t new_idx) -{ - unsigned i; - struct queue_node queue[RTE_HASH_BFS_QUEUE_MAX_LEN]; - struct queue_node *tail, *head; - struct rte_hash_bucket *curr_bkt, *alt_bkt; - - tail = queue; - head = queue + 1; - tail->bkt = bkt; - tail->prev = NULL; - tail->prev_slot = -1; - - /* Cuckoo bfs Search */ - while (likely(tail != head && head < - queue + RTE_HASH_BFS_QUEUE_MAX_LEN - - RTE_HASH_BUCKET_ENTRIES)) { - curr_bkt = tail->bkt; - for (i = 0; i < RTE_HASH_BUCKET_ENTRIES; i++) { - if (curr_bkt->key_idx[i] == EMPTY_SLOT) { - if (likely(rte_hash_cuckoo_move_insert_mw_tm(h, - tail, i, sig, - alt_hash, new_idx) == 0)) - return 0; - } - - /* Enqueue new node and keep prev node info */ - alt_bkt = &(h->buckets[curr_bkt->sig_alt[i] - & h->bucket_bitmask]); - head->bkt = alt_bkt; - head->prev = tail; - head->prev_slot = i; - head++; - } - tail++; - } - - return -ENOSPC; -} diff --git a/lib/librte_hash/rte_hash.h b/lib/librte_hash/rte_hash.h index f71ca9f..ecb49e4 100644 --- a/lib/librte_hash/rte_hash.h +++ b/lib/librte_hash/rte_hash.h @@ -34,6 +34,9 @@ extern "C" { /** Default behavior of insertion, single writer/multi writer */ #define RTE_HASH_EXTRA_FLAGS_MULTI_WRITER_ADD 0x02 +/** Flag to support reader writer concurrency */ +#define RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY 0x04 + /** Signature of key that is stored internally. */ typedef uint32_t hash_sig_t; From patchwork Mon Jul 9 10:45:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Yipeng1" X-Patchwork-Id: 42644 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id B09F25F30; Mon, 9 Jul 2018 19:52:30 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 2918A324B for ; Mon, 9 Jul 2018 19:52:22 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 Jul 2018 10:52:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,330,1526367600"; d="scan'208";a="63369601" Received: from skx-yipeng.jf.intel.com ([10.54.81.175]) by FMSMGA003.fm.intel.com with ESMTP; 09 Jul 2018 10:52:12 -0700 From: Yipeng Wang To: pablo.de.lara.guarch@intel.com Cc: dev@dpdk.org, yipeng1.wang@intel.com, bruce.richardson@intel.com, honnappa.nagarahalli@arm.com, vguvva@caviumnetworks.com, brijesh.s.singh@gmail.com Date: Mon, 9 Jul 2018 03:45:01 -0700 Message-Id: <1531133103-437316-7-git-send-email-yipeng1.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> References: <1528455078-328182-1-git-send-email-yipeng1.wang@intel.com> <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> Subject: [dpdk-dev] [PATCH v4 6/8] test: add tests in hash table perf test X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" New code is added to support read-write concurrency for rte_hash. Due to the newly added code in critial path, the perf test is modified to show any performance impact. It is still a single-thread test. Signed-off-by: Yipeng Wang Acked-by: Pablo de Lara --- test/test/test_hash_perf.c | 36 +++++++++++++++++++++++++----------- 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/test/test/test_hash_perf.c b/test/test/test_hash_perf.c index a81d0c7..33dcb9f 100644 --- a/test/test/test_hash_perf.c +++ b/test/test/test_hash_perf.c @@ -76,7 +76,8 @@ static struct rte_hash_parameters ut_params = { }; static int -create_table(unsigned with_data, unsigned table_index) +create_table(unsigned int with_data, unsigned int table_index, + unsigned int with_locks) { char name[RTE_HASH_NAMESIZE]; @@ -86,6 +87,14 @@ create_table(unsigned with_data, unsigned table_index) else sprintf(name, "test_hash%d", hashtest_key_lens[table_index]); + + if (with_locks) + ut_params.extra_flag = + RTE_HASH_EXTRA_FLAGS_TRANS_MEM_SUPPORT + | RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY; + else + ut_params.extra_flag = 0; + ut_params.name = name; ut_params.key_len = hashtest_key_lens[table_index]; ut_params.socket_id = rte_socket_id(); @@ -459,7 +468,7 @@ reset_table(unsigned table_index) } static int -run_all_tbl_perf_tests(unsigned with_pushes) +run_all_tbl_perf_tests(unsigned int with_pushes, unsigned int with_locks) { unsigned i, j, with_data, with_hash; @@ -468,7 +477,7 @@ run_all_tbl_perf_tests(unsigned with_pushes) for (with_data = 0; with_data <= 1; with_data++) { for (i = 0; i < NUM_KEYSIZES; i++) { - if (create_table(with_data, i) < 0) + if (create_table(with_data, i, with_locks) < 0) return -1; if (get_input_keys(with_pushes, i) < 0) @@ -611,15 +620,20 @@ fbk_hash_perf_test(void) static int test_hash_perf(void) { - unsigned with_pushes; - - for (with_pushes = 0; with_pushes <= 1; with_pushes++) { - if (with_pushes == 0) - printf("\nALL ELEMENTS IN PRIMARY LOCATION\n"); + unsigned int with_pushes, with_locks; + for (with_locks = 0; with_locks <= 1; with_locks++) { + if (with_locks) + printf("\nWith locks in the code\n"); else - printf("\nELEMENTS IN PRIMARY OR SECONDARY LOCATION\n"); - if (run_all_tbl_perf_tests(with_pushes) < 0) - return -1; + printf("\nWithout locks in the code\n"); + for (with_pushes = 0; with_pushes <= 1; with_pushes++) { + if (with_pushes == 0) + printf("\nALL ELEMENTS IN PRIMARY LOCATION\n"); + else + printf("\nELEMENTS IN PRIMARY OR SECONDARY LOCATION\n"); + if (run_all_tbl_perf_tests(with_pushes, with_locks) < 0) + return -1; + } } if (fbk_hash_perf_test() < 0) return -1; From patchwork Mon Jul 9 10:45:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Yipeng1" X-Patchwork-Id: 42648 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 9A6791B064; Mon, 9 Jul 2018 19:52:36 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 766A058CB for ; Mon, 9 Jul 2018 19:52:24 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 Jul 2018 10:52:20 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,330,1526367600"; d="scan'208";a="63369602" Received: from skx-yipeng.jf.intel.com ([10.54.81.175]) by FMSMGA003.fm.intel.com with ESMTP; 09 Jul 2018 10:52:12 -0700 From: Yipeng Wang To: pablo.de.lara.guarch@intel.com Cc: dev@dpdk.org, yipeng1.wang@intel.com, bruce.richardson@intel.com, honnappa.nagarahalli@arm.com, vguvva@caviumnetworks.com, brijesh.s.singh@gmail.com Date: Mon, 9 Jul 2018 03:45:02 -0700 Message-Id: <1531133103-437316-8-git-send-email-yipeng1.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> References: <1528455078-328182-1-git-send-email-yipeng1.wang@intel.com> <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> Subject: [dpdk-dev] [PATCH v4 7/8] test: add test case for read write concurrency X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This commits add a new test case for testing read/write concurrency. Signed-off-by: Yipeng Wang Acked-by: Pablo de Lara --- test/test/Makefile | 1 + test/test/test_hash_readwrite.c | 637 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 638 insertions(+) create mode 100644 test/test/test_hash_readwrite.c diff --git a/test/test/Makefile b/test/test/Makefile index eccc8ef..6ce66c9 100644 --- a/test/test/Makefile +++ b/test/test/Makefile @@ -113,6 +113,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_perf.c SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_functions.c SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_scaling.c SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_multiwriter.c +SRCS-$(CONFIG_RTE_LIBRTE_HASH) += test_hash_readwrite.c SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm.c SRCS-$(CONFIG_RTE_LIBRTE_LPM) += test_lpm_perf.c diff --git a/test/test/test_hash_readwrite.c b/test/test/test_hash_readwrite.c new file mode 100644 index 0000000..55ae33d --- /dev/null +++ b/test/test/test_hash_readwrite.c @@ -0,0 +1,637 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2018 Intel Corporation + */ + +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "test.h" + +#define RTE_RWTEST_FAIL 0 + +#define TOTAL_ENTRY (16*1024*1024) +#define TOTAL_INSERT (15*1024*1024) + +#define NUM_TEST 3 +unsigned int core_cnt[NUM_TEST] = {2, 4, 8}; + +struct perf { + uint32_t single_read; + uint32_t single_write; + uint32_t read_only[NUM_TEST]; + uint32_t write_only[NUM_TEST]; + uint32_t read_write_r[NUM_TEST]; + uint32_t read_write_w[NUM_TEST]; +}; + +static struct perf htm_results, non_htm_results; + +struct { + uint32_t *keys; + uint32_t *found; + uint32_t num_insert; + uint32_t rounded_tot_insert; + struct rte_hash *h; +} tbl_rw_test_param; + +static rte_atomic64_t gcycles; +static rte_atomic64_t ginsertions; + +static rte_atomic64_t gread_cycles; +static rte_atomic64_t gwrite_cycles; + +static rte_atomic64_t greads; +static rte_atomic64_t gwrites; + +static int +test_hash_readwrite_worker(__attribute__((unused)) void *arg) +{ + uint64_t i, offset; + uint32_t lcore_id = rte_lcore_id(); + uint64_t begin, cycles; + int ret; + + offset = (lcore_id - rte_get_master_lcore()) + * tbl_rw_test_param.num_insert; + + printf("Core #%d inserting and reading %d: %'"PRId64" - %'"PRId64"\n", + lcore_id, tbl_rw_test_param.num_insert, + offset, offset + tbl_rw_test_param.num_insert); + + begin = rte_rdtsc_precise(); + + for (i = offset; i < offset + tbl_rw_test_param.num_insert; i++) { + + if (rte_hash_lookup(tbl_rw_test_param.h, + tbl_rw_test_param.keys + i) > 0) + break; + + ret = rte_hash_add_key(tbl_rw_test_param.h, + tbl_rw_test_param.keys + i); + if (ret < 0) + break; + + if (rte_hash_lookup(tbl_rw_test_param.h, + tbl_rw_test_param.keys + i) != ret) + break; + } + + cycles = rte_rdtsc_precise() - begin; + rte_atomic64_add(&gcycles, cycles); + rte_atomic64_add(&ginsertions, i - offset); + + for (; i < offset + tbl_rw_test_param.num_insert; i++) + tbl_rw_test_param.keys[i] = RTE_RWTEST_FAIL; + + return 0; +} + +static int +init_params(int use_htm, int use_jhash) +{ + unsigned int i; + + uint32_t *keys = NULL; + uint32_t *found = NULL; + struct rte_hash *handle; + + struct rte_hash_parameters hash_params = { + .entries = TOTAL_ENTRY, + .key_len = sizeof(uint32_t), + .hash_func_init_val = 0, + .socket_id = rte_socket_id(), + }; + if (use_jhash) + hash_params.hash_func = rte_jhash; + else + hash_params.hash_func = rte_hash_crc; + + if (use_htm) + hash_params.extra_flag = + RTE_HASH_EXTRA_FLAGS_TRANS_MEM_SUPPORT | + RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY; + else + hash_params.extra_flag = + RTE_HASH_EXTRA_FLAGS_RW_CONCURRENCY; + + hash_params.name = "tests"; + + handle = rte_hash_create(&hash_params); + if (handle == NULL) { + printf("hash creation failed"); + return -1; + } + + tbl_rw_test_param.h = handle; + keys = rte_malloc(NULL, sizeof(uint32_t) * TOTAL_ENTRY, 0); + + if (keys == NULL) { + printf("RTE_MALLOC failed\n"); + goto err; + } + + found = rte_zmalloc(NULL, sizeof(uint32_t) * TOTAL_ENTRY, 0); + if (found == NULL) { + printf("RTE_ZMALLOC failed\n"); + goto err; + } + + tbl_rw_test_param.keys = keys; + tbl_rw_test_param.found = found; + + for (i = 0; i < TOTAL_ENTRY; i++) + keys[i] = i; + + return 0; + +err: + rte_free(keys); + rte_hash_free(handle); + + return -1; +} + +static int +test_hash_readwrite_functional(int use_htm) +{ + unsigned int i; + const void *next_key; + void *next_data; + uint32_t iter = 0; + + uint32_t duplicated_keys = 0; + uint32_t lost_keys = 0; + int use_jhash = 1; + + rte_atomic64_init(&gcycles); + rte_atomic64_clear(&gcycles); + + rte_atomic64_init(&ginsertions); + rte_atomic64_clear(&ginsertions); + + if (init_params(use_htm, use_jhash) != 0) + goto err; + + tbl_rw_test_param.num_insert = + TOTAL_INSERT / rte_lcore_count(); + + tbl_rw_test_param.rounded_tot_insert = + tbl_rw_test_param.num_insert + * rte_lcore_count(); + + printf("++++++++Start function tests:+++++++++\n"); + + /* Fire all threads. */ + rte_eal_mp_remote_launch(test_hash_readwrite_worker, + NULL, CALL_MASTER); + rte_eal_mp_wait_lcore(); + + while (rte_hash_iterate(tbl_rw_test_param.h, &next_key, + &next_data, &iter) >= 0) { + /* Search for the key in the list of keys added .*/ + i = *(const uint32_t *)next_key; + tbl_rw_test_param.found[i]++; + } + + for (i = 0; i < tbl_rw_test_param.rounded_tot_insert; i++) { + if (tbl_rw_test_param.keys[i] != RTE_RWTEST_FAIL) { + if (tbl_rw_test_param.found[i] > 1) { + duplicated_keys++; + break; + } + if (tbl_rw_test_param.found[i] == 0) { + lost_keys++; + printf("key %d is lost\n", i); + break; + } + } + } + + if (duplicated_keys > 0) { + printf("%d key duplicated\n", duplicated_keys); + goto err_free; + } + + if (lost_keys > 0) { + printf("%d key lost\n", lost_keys); + goto err_free; + } + + printf("No key corrupted during read-write test.\n"); + + unsigned long long int cycles_per_insertion = + rte_atomic64_read(&gcycles) / + rte_atomic64_read(&ginsertions); + + printf("cycles per insertion and lookup: %llu\n", cycles_per_insertion); + + rte_free(tbl_rw_test_param.found); + rte_free(tbl_rw_test_param.keys); + rte_hash_free(tbl_rw_test_param.h); + printf("+++++++++Complete function tests+++++++++\n"); + return 0; + +err_free: + rte_free(tbl_rw_test_param.found); + rte_free(tbl_rw_test_param.keys); + rte_hash_free(tbl_rw_test_param.h); +err: + return -1; +} + +static int +test_rw_reader(__attribute__((unused)) void *arg) +{ + uint64_t i; + uint64_t begin, cycles; + uint64_t read_cnt = (uint64_t)((uintptr_t)arg); + + begin = rte_rdtsc_precise(); + for (i = 0; i < read_cnt; i++) { + void *data; + rte_hash_lookup_data(tbl_rw_test_param.h, + tbl_rw_test_param.keys + i, + &data); + if (i != (uint64_t)(uintptr_t)data) { + printf("lookup find wrong value %"PRIu64"," + "%"PRIu64"\n", i, + (uint64_t)(uintptr_t)data); + break; + } + } + + cycles = rte_rdtsc_precise() - begin; + rte_atomic64_add(&gread_cycles, cycles); + rte_atomic64_add(&greads, i); + return 0; +} + +static int +test_rw_writer(__attribute__((unused)) void *arg) +{ + uint64_t i; + uint32_t lcore_id = rte_lcore_id(); + uint64_t begin, cycles; + int ret; + uint64_t start_coreid = (uint64_t)(uintptr_t)arg; + uint64_t offset; + + offset = TOTAL_INSERT / 2 + (lcore_id - start_coreid) + * tbl_rw_test_param.num_insert; + begin = rte_rdtsc_precise(); + for (i = offset; i < offset + tbl_rw_test_param.num_insert; i++) { + ret = rte_hash_add_key_data(tbl_rw_test_param.h, + tbl_rw_test_param.keys + i, + (void *)((uintptr_t)i)); + if (ret < 0) { + printf("writer failed %"PRIu64"\n", i); + break; + } + } + + cycles = rte_rdtsc_precise() - begin; + rte_atomic64_add(&gwrite_cycles, cycles); + rte_atomic64_add(&gwrites, tbl_rw_test_param.num_insert); + return 0; +} + +static int +test_hash_readwrite_perf(struct perf *perf_results, int use_htm, + int reader_faster) +{ + unsigned int n; + int ret; + int start_coreid; + uint64_t i, read_cnt; + + const void *next_key; + void *next_data; + uint32_t iter = 0; + int use_jhash = 0; + + uint32_t duplicated_keys = 0; + uint32_t lost_keys = 0; + + uint64_t start = 0, end = 0; + + rte_atomic64_init(&greads); + rte_atomic64_init(&gwrites); + rte_atomic64_clear(&gwrites); + rte_atomic64_clear(&greads); + + rte_atomic64_init(&gread_cycles); + rte_atomic64_clear(&gread_cycles); + rte_atomic64_init(&gwrite_cycles); + rte_atomic64_clear(&gwrite_cycles); + + if (init_params(use_htm, use_jhash) != 0) + goto err; + + /* + * Do a readers finish faster or writers finish faster test. + * When readers finish faster, we timing the readers, and when writers + * finish faster, we timing the writers. + * Divided by 10 or 2 is just experimental values to vary the workload + * of readers. + */ + if (reader_faster) { + printf("++++++Start perf test: reader++++++++\n"); + read_cnt = TOTAL_INSERT / 10; + } else { + printf("++++++Start perf test: writer++++++++\n"); + read_cnt = TOTAL_INSERT / 2; + } + + /* We first test single thread performance */ + start = rte_rdtsc_precise(); + /* Insert half of the keys */ + for (i = 0; i < TOTAL_INSERT / 2; i++) { + ret = rte_hash_add_key_data(tbl_rw_test_param.h, + tbl_rw_test_param.keys + i, + (void *)((uintptr_t)i)); + if (ret < 0) { + printf("Failed to insert half of keys\n"); + goto err_free; + } + } + end = rte_rdtsc_precise() - start; + perf_results->single_write = end / i; + + start = rte_rdtsc_precise(); + + for (i = 0; i < read_cnt; i++) { + void *data; + rte_hash_lookup_data(tbl_rw_test_param.h, + tbl_rw_test_param.keys + i, + &data); + if (i != (uint64_t)(uintptr_t)data) { + printf("lookup find wrong value" + " %"PRIu64",%"PRIu64"\n", i, + (uint64_t)(uintptr_t)data); + break; + } + } + end = rte_rdtsc_precise() - start; + perf_results->single_read = end / i; + + for (n = 0; n < NUM_TEST; n++) { + unsigned int tot_lcore = rte_lcore_count(); + if (tot_lcore < core_cnt[n] * 2 + 1) + goto finish; + + rte_atomic64_clear(&greads); + rte_atomic64_clear(&gread_cycles); + rte_atomic64_clear(&gwrites); + rte_atomic64_clear(&gwrite_cycles); + + rte_hash_reset(tbl_rw_test_param.h); + + tbl_rw_test_param.num_insert = TOTAL_INSERT / 2 / core_cnt[n]; + tbl_rw_test_param.rounded_tot_insert = TOTAL_INSERT / 2 + + tbl_rw_test_param.num_insert * + core_cnt[n]; + + for (i = 0; i < TOTAL_INSERT / 2; i++) { + ret = rte_hash_add_key_data(tbl_rw_test_param.h, + tbl_rw_test_param.keys + i, + (void *)((uintptr_t)i)); + if (ret < 0) { + printf("Failed to insert half of keys\n"); + goto err_free; + } + } + + /* Then test multiple thread case but only all reads or + * all writes + */ + + /* Test only reader cases */ + for (i = 1; i <= core_cnt[n]; i++) + rte_eal_remote_launch(test_rw_reader, + (void *)(uintptr_t)read_cnt, i); + + rte_eal_mp_wait_lcore(); + + start_coreid = i; + /* Test only writer cases */ + for (; i <= core_cnt[n] * 2; i++) + rte_eal_remote_launch(test_rw_writer, + (void *)((uintptr_t)start_coreid), i); + + rte_eal_mp_wait_lcore(); + + if (reader_faster) { + unsigned long long int cycles_per_insertion = + rte_atomic64_read(&gread_cycles) / + rte_atomic64_read(&greads); + perf_results->read_only[n] = cycles_per_insertion; + printf("Reader only: cycles per lookup: %llu\n", + cycles_per_insertion); + } + + else { + unsigned long long int cycles_per_insertion = + rte_atomic64_read(&gwrite_cycles) / + rte_atomic64_read(&gwrites); + perf_results->write_only[n] = cycles_per_insertion; + printf("Writer only: cycles per writes: %llu\n", + cycles_per_insertion); + } + + rte_atomic64_clear(&greads); + rte_atomic64_clear(&gread_cycles); + rte_atomic64_clear(&gwrites); + rte_atomic64_clear(&gwrite_cycles); + + rte_hash_reset(tbl_rw_test_param.h); + + for (i = 0; i < TOTAL_INSERT / 2; i++) { + ret = rte_hash_add_key_data(tbl_rw_test_param.h, + tbl_rw_test_param.keys + i, + (void *)((uintptr_t)i)); + if (ret < 0) { + printf("Failed to insert half of keys\n"); + goto err_free; + } + } + + start_coreid = core_cnt[n] + 1; + + if (reader_faster) { + for (i = core_cnt[n] + 1; i <= core_cnt[n] * 2; i++) + rte_eal_remote_launch(test_rw_writer, + (void *)((uintptr_t)start_coreid), i); + for (i = 1; i <= core_cnt[n]; i++) + rte_eal_remote_launch(test_rw_reader, + (void *)(uintptr_t)read_cnt, i); + } else { + for (i = 1; i <= core_cnt[n]; i++) + rte_eal_remote_launch(test_rw_reader, + (void *)(uintptr_t)read_cnt, i); + for (; i <= core_cnt[n] * 2; i++) + rte_eal_remote_launch(test_rw_writer, + (void *)((uintptr_t)start_coreid), i); + } + + rte_eal_mp_wait_lcore(); + + while (rte_hash_iterate(tbl_rw_test_param.h, + &next_key, &next_data, &iter) >= 0) { + /* Search for the key in the list of keys added .*/ + i = *(const uint32_t *)next_key; + tbl_rw_test_param.found[i]++; + } + + for (i = 0; i < tbl_rw_test_param.rounded_tot_insert; i++) { + if (tbl_rw_test_param.keys[i] != RTE_RWTEST_FAIL) { + if (tbl_rw_test_param.found[i] > 1) { + duplicated_keys++; + break; + } + if (tbl_rw_test_param.found[i] == 0) { + lost_keys++; + printf("key %"PRIu64" is lost\n", i); + break; + } + } + } + + if (duplicated_keys > 0) { + printf("%d key duplicated\n", duplicated_keys); + goto err_free; + } + + if (lost_keys > 0) { + printf("%d key lost\n", lost_keys); + goto err_free; + } + + printf("No key corrupted during read-write test.\n"); + + if (reader_faster) { + unsigned long long int cycles_per_insertion = + rte_atomic64_read(&gread_cycles) / + rte_atomic64_read(&greads); + perf_results->read_write_r[n] = cycles_per_insertion; + printf("Read-write cycles per lookup: %llu\n", + cycles_per_insertion); + } + + else { + unsigned long long int cycles_per_insertion = + rte_atomic64_read(&gwrite_cycles) / + rte_atomic64_read(&gwrites); + perf_results->read_write_w[n] = cycles_per_insertion; + printf("Read-write cycles per writes: %llu\n", + cycles_per_insertion); + } + } + +finish: + rte_free(tbl_rw_test_param.found); + rte_free(tbl_rw_test_param.keys); + rte_hash_free(tbl_rw_test_param.h); + return 0; + +err_free: + rte_free(tbl_rw_test_param.found); + rte_free(tbl_rw_test_param.keys); + rte_hash_free(tbl_rw_test_param.h); + +err: + return -1; +} + +static int +test_hash_readwrite_main(void) +{ + /* + * Variables used to choose different tests. + * use_htm indicates if hardware transactional memory should be used. + * reader_faster indicates if the reader threads should finish earlier + * than writer threads. This is to timing either reader threads or + * writer threads for performance numbers. + */ + int use_htm, reader_faster; + + if (rte_lcore_count() == 1) { + printf("More than one lcore is required " + "to do read write test\n"); + return 0; + } + + setlocale(LC_NUMERIC, ""); + + if (rte_tm_supported()) { + printf("Hardware transactional memory (lock elision) " + "is supported\n"); + + printf("Test read-write with Hardware transactional memory\n"); + + use_htm = 1; + if (test_hash_readwrite_functional(use_htm) < 0) + return -1; + + reader_faster = 1; + if (test_hash_readwrite_perf(&htm_results, use_htm, + reader_faster) < 0) + return -1; + + reader_faster = 0; + if (test_hash_readwrite_perf(&htm_results, use_htm, + reader_faster) < 0) + return -1; + } else { + printf("Hardware transactional memory (lock elision) " + "is NOT supported\n"); + } + + printf("Test read-write without Hardware transactional memory\n"); + use_htm = 0; + if (test_hash_readwrite_functional(use_htm) < 0) + return -1; + reader_faster = 1; + if (test_hash_readwrite_perf(&non_htm_results, use_htm, + reader_faster) < 0) + return -1; + reader_faster = 0; + if (test_hash_readwrite_perf(&non_htm_results, use_htm, + reader_faster) < 0) + return -1; + + printf("Results summary:\n"); + + int i; + + printf("single read: %u\n", htm_results.single_read); + printf("single write: %u\n", htm_results.single_write); + for (i = 0; i < NUM_TEST; i++) { + printf("core_cnt: %u\n", core_cnt[i]); + printf("HTM:\n"); + printf("read only: %u\n", htm_results.read_only[i]); + printf("write only: %u\n", htm_results.write_only[i]); + printf("read-write read: %u\n", htm_results.read_write_r[i]); + printf("read-write write: %u\n", htm_results.read_write_w[i]); + + printf("non HTM:\n"); + printf("read only: %u\n", non_htm_results.read_only[i]); + printf("write only: %u\n", non_htm_results.write_only[i]); + printf("read-write read: %u\n", + non_htm_results.read_write_r[i]); + printf("read-write write: %u\n", + non_htm_results.read_write_w[i]); + } + + return 0; +} + +REGISTER_TEST_COMMAND(hash_readwrite_autotest, test_hash_readwrite_main); From patchwork Mon Jul 9 10:45:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Wang, Yipeng1" X-Patchwork-Id: 42649 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BDAEE1B152; Mon, 9 Jul 2018 19:52:37 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id C00365B12 for ; Mon, 9 Jul 2018 19:52:24 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 09 Jul 2018 10:52:20 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,330,1526367600"; d="scan'208";a="63369604" Received: from skx-yipeng.jf.intel.com ([10.54.81.175]) by FMSMGA003.fm.intel.com with ESMTP; 09 Jul 2018 10:52:12 -0700 From: Yipeng Wang To: pablo.de.lara.guarch@intel.com Cc: dev@dpdk.org, yipeng1.wang@intel.com, bruce.richardson@intel.com, honnappa.nagarahalli@arm.com, vguvva@caviumnetworks.com, brijesh.s.singh@gmail.com Date: Mon, 9 Jul 2018 03:45:03 -0700 Message-Id: <1531133103-437316-9-git-send-email-yipeng1.wang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> References: <1528455078-328182-1-git-send-email-yipeng1.wang@intel.com> <1531133103-437316-1-git-send-email-yipeng1.wang@intel.com> Subject: [dpdk-dev] [PATCH v4 8/8] hash: add new API function to query the key count X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add a new function, rte_hash_count, to return the number of keys that are currently stored in the hash table. Corresponding test functions are added into hash_test and hash_multiwriter test. Signed-off-by: Yipeng Wang Acked-by: Pablo de Lara --- lib/librte_hash/rte_cuckoo_hash.c | 24 ++++++++++++++++++++++++ lib/librte_hash/rte_hash.h | 11 +++++++++++ lib/librte_hash/rte_hash_version.map | 8 ++++++++ test/test/test_hash.c | 12 ++++++++++++ test/test/test_hash_multiwriter.c | 8 ++++++++ 5 files changed, 63 insertions(+) diff --git a/lib/librte_hash/rte_cuckoo_hash.c b/lib/librte_hash/rte_cuckoo_hash.c index 35631cc..bb67ade 100644 --- a/lib/librte_hash/rte_cuckoo_hash.c +++ b/lib/librte_hash/rte_cuckoo_hash.c @@ -370,6 +370,30 @@ rte_hash_secondary_hash(const hash_sig_t primary_hash) return primary_hash ^ ((tag + 1) * alt_bits_xor); } +int32_t +rte_hash_count(const struct rte_hash *h) +{ + uint32_t tot_ring_cnt, cached_cnt = 0; + uint32_t i, ret; + + if (h == NULL) + return -EINVAL; + + if (h->multi_writer_support) { + tot_ring_cnt = h->entries + (RTE_MAX_LCORE - 1) * + (LCORE_CACHE_SIZE - 1); + for (i = 0; i < RTE_MAX_LCORE; i++) + cached_cnt += h->local_free_slots[i].len; + + ret = tot_ring_cnt - rte_ring_count(h->free_slots) - + cached_cnt; + } else { + tot_ring_cnt = h->entries; + ret = tot_ring_cnt - rte_ring_count(h->free_slots); + } + return ret; +} + /* Read write locks implemented using rte_rwlock */ static inline void __hash_rw_writer_lock(const struct rte_hash *h) diff --git a/lib/librte_hash/rte_hash.h b/lib/librte_hash/rte_hash.h index ecb49e4..1f1a276 100644 --- a/lib/librte_hash/rte_hash.h +++ b/lib/librte_hash/rte_hash.h @@ -127,6 +127,17 @@ void rte_hash_reset(struct rte_hash *h); /** + * Return the number of keys in the hash table + * @param h + * Hash table to query from + * @return + * - -EINVAL if parameters are invalid + * - A value indicating how many keys were inserted in the table. + */ +int32_t +rte_hash_count(const struct rte_hash *h); + +/** * Add a key-value pair to an existing hash table. * This operation is not multi-thread safe * and should only be called from one thread. diff --git a/lib/librte_hash/rte_hash_version.map b/lib/librte_hash/rte_hash_version.map index 52a2576..e216ac8 100644 --- a/lib/librte_hash/rte_hash_version.map +++ b/lib/librte_hash/rte_hash_version.map @@ -45,3 +45,11 @@ DPDK_16.07 { rte_hash_get_key_with_position; } DPDK_2.2; + + +DPDK_18.08 { + global: + + rte_hash_count; + +} DPDK_16.07; diff --git a/test/test/test_hash.c b/test/test/test_hash.c index edf41f5..b3db9fd 100644 --- a/test/test/test_hash.c +++ b/test/test/test_hash.c @@ -1103,6 +1103,7 @@ static int test_average_table_utilization(void) unsigned i, j; unsigned added_keys, average_keys_added = 0; int ret; + unsigned int cnt; printf("\n# Running test to determine average utilization" "\n before adding elements begins to fail\n"); @@ -1121,13 +1122,24 @@ static int test_average_table_utilization(void) for (i = 0; i < ut_params.key_len; i++) simple_key[i] = rte_rand() % 255; ret = rte_hash_add_key(handle, simple_key); + if (ret < 0) + break; } + if (ret != -ENOSPC) { printf("Unexpected error when adding keys\n"); rte_hash_free(handle); return -1; } + cnt = rte_hash_count(handle); + if (cnt != added_keys) { + printf("rte_hash_count returned wrong value %u, %u," + "%u\n", j, added_keys, cnt); + rte_hash_free(handle); + return -1; + } + average_keys_added += added_keys; /* Reset the table */ diff --git a/test/test/test_hash_multiwriter.c b/test/test/test_hash_multiwriter.c index ef5fce3..f182f40 100644 --- a/test/test/test_hash_multiwriter.c +++ b/test/test/test_hash_multiwriter.c @@ -116,6 +116,7 @@ test_hash_multiwriter(void) uint32_t duplicated_keys = 0; uint32_t lost_keys = 0; + uint32_t count; snprintf(name, 32, "test%u", calledCount++); hash_params.name = name; @@ -163,6 +164,13 @@ test_hash_multiwriter(void) NULL, CALL_MASTER); rte_eal_mp_wait_lcore(); + count = rte_hash_count(handle); + if (count != rounded_nb_total_tsx_insertion) { + printf("rte_hash_count returned wrong value %u, %d\n", + rounded_nb_total_tsx_insertion, count); + goto err3; + } + while (rte_hash_iterate(handle, &next_key, &next_data, &iter) >= 0) { /* Search for the key in the list of keys added .*/ i = *(const uint32_t *)next_key;