From patchwork Thu Feb 22 18:40:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Boyer X-Patchwork-Id: 137044 X-Patchwork-Delegate: gakhil@marvell.com Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 06A4743B7F; Thu, 22 Feb 2024 19:41:05 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9EDDB40EE4; Thu, 22 Feb 2024 19:40:51 +0100 (CET) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) by mails.dpdk.org (Postfix) with ESMTP id 65B6340F16 for ; Thu, 22 Feb 2024 19:40:49 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Xa2+KUNLHCphTLtz+q9TGEuIQjp1BzuFZalX8m9d5dRUMIdNm6aWkNVL1eYJkHtXwtHK9yNsCtJkqvMHWSiYZ3HUEYCsFK+HV/SYGP3gJKPiNe4Gl4uYjNeH4916iX+LzCRbGZ1q6P7DNUuztZVgbv4ACg6XyQTVxU/2gPLbQawtcXe+xWQG2EvfVFgIYzuqbZyO0h0kcIFxq58R4MQNMlRxS3KJp+vUHg+so8vvnVVgjV70aubW8o0GXg88mvUeRFKqQk7HiBCm0x4LcuLyF1keEbr8xUilYyyfcfo18nhQ5bJLqN1FyFzoRrdNMYT/ix1v3eUzM8EvcUp1P3lf5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5tuxfGOCmzxtMvRAKeSvvj1afCNp5+sqIDToNq7IBTE=; b=MCUW6fc1UGp792xk7xlmVBwtVxb1vV4/xbt1KJSkkU03Kb2xALadVmzrbP/iU++zLOm5zKd8MWENFbKR33bfBn5+b0e3yWMSN1u6iW+/749tjVKqnWKCtyRrq7JGCvGCoj8axbAHsKROKFVau0OW2VFolVuoDjrKgWDaPQtmoa+g6xj3+gUq9yO0QEtVrISHM4Adx6VtL+nGgj3tK8p06fz2GbxrlOzbPx9tH/JaCAZRLFf+7TU4MWr+pUL28piO7Bfgg2MID4QxcqKqhCoPEoCLeGhU2RtlAjVpO+nAOvfHrgAZ/NWkI47hB5U9IGodIl0jKgIdP5ig6QzbziSQSw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=dpdk.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5tuxfGOCmzxtMvRAKeSvvj1afCNp5+sqIDToNq7IBTE=; b=JvdXU8BC5T5D3O6G1ca0M/s9g8Ztp0dTQoXTYvF01X7LAma6ibpqSHYfb4ZS4iQRkQwZbg/WzDe8jKKFqnL4x9gC27/btABF3pGyGF2y7cEhhKS/q/NzAXsNUpR/i2fzYdciWQxfNOTU2P02lzYNp5d2FWyN0q0haeL5vOFvi3s= Received: from BN9PR03CA0606.namprd03.prod.outlook.com (2603:10b6:408:106::11) by IA0PR12MB8421.namprd12.prod.outlook.com (2603:10b6:208:40f::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7316.14; Thu, 22 Feb 2024 18:40:43 +0000 Received: from BN3PEPF0000B06B.namprd21.prod.outlook.com (2603:10b6:408:106:cafe::4c) by BN9PR03CA0606.outlook.office365.com (2603:10b6:408:106::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.43 via Frontend Transport; Thu, 22 Feb 2024 18:40:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN3PEPF0000B06B.mail.protection.outlook.com (10.167.243.70) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.7339.0 via Frontend Transport; Thu, 22 Feb 2024 18:40:41 +0000 Received: from driver-dev1.pensando.io (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Thu, 22 Feb 2024 12:40:39 -0600 From: Andrew Boyer To: CC: Akhil Goyal , Andrew Boyer Subject: [RFC 2/5] crypto/ionic: add device and admin command handlers Date: Thu, 22 Feb 2024 10:40:16 -0800 Message-ID: <20240222184019.15301-3-andrew.boyer@amd.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240222184019.15301-1-andrew.boyer@amd.com> References: <20240222184019.15301-1-andrew.boyer@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB04.amd.com (10.181.40.145) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN3PEPF0000B06B:EE_|IA0PR12MB8421:EE_ X-MS-Office365-Filtering-Correlation-Id: 7ef3a6ec-5083-4ce8-1044-08dc33d5c5ac X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: d6n0izD10QwJ6ryTKHF04j7XK07LQx2m+CadPM7M9RhS76tsLRcJu1Zt0T1EwuX+arAqYFApOqSOQMGejHvncwPyAtSElHlVMqFM6eIisMe5vAGg7o4cw/hC2sazynYxRn5YZ5yuNlZLkD8tdzw1Nc+IFEPTc+8frGj24DbG89Xj10osJw0cezw6/a6U3AKbT1EdohAIcmumTdAN87vot9iNo8hW0Eknek/IGrWUaLe6xPhgvwIDtXg0UvgXzjWV317YhBjMpB+gFIf/VjJJocUjnj4pE+2cXMZiT7BB8FISA1ZohMW6RcuiyAzw7iEfo8hlxH23rLIn2aP1kDchfpb9fbNTG3ZDdM0myoqVFhH9jD83UYfDkeCXHw6lSdYlrY2be3I8cclxnLkLq96gRFd5fBpA5lD1P+OKbLyMbMUeHEH7oTxETq69woXyOf8Cneu9+74O1QddLTRcxByrXecHcc6Zz8iI/cimNLMjFOh2ixr1PQENVtwzRJunL+OEbiz2GTeswvzVEfdX2FZ7Wt4lnJYQkO4iDapaYgNSRBvNN6iTDRMtpgklr+SPl5E+CQdITmXFxpY6o0+z4WrnaWapRRw4EEGsNYS7VhO9wziWu42rpmoLINoMw0xGpeKQt38tIVPa/TC8r0sHH/ejxzkdpzxT7eKKBGgsOu9uVeg= X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:SATLEXMB04.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFS:(13230031)(36860700004)(46966006)(40470700004); DIR:OUT; SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Feb 2024 18:40:41.6737 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7ef3a6ec-5083-4ce8-1044-08dc33d5c5ac X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN3PEPF0000B06B.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8421 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This defines the handlers used for device (register-based) and admin (adminq-based) commands. Signed-off-by: Andrew Boyer --- drivers/crypto/ionic/ionic_crypto.h | 209 ++++++++ drivers/crypto/ionic/ionic_crypto_cmds.c | 651 +++++++++++++++++++++++ drivers/crypto/ionic/ionic_crypto_main.c | 40 ++ drivers/crypto/ionic/meson.build | 12 + drivers/crypto/meson.build | 1 + 5 files changed, 913 insertions(+) create mode 100644 drivers/crypto/ionic/ionic_crypto.h create mode 100644 drivers/crypto/ionic/ionic_crypto_cmds.c create mode 100644 drivers/crypto/ionic/ionic_crypto_main.c create mode 100644 drivers/crypto/ionic/meson.build diff --git a/drivers/crypto/ionic/ionic_crypto.h b/drivers/crypto/ionic/ionic_crypto.h new file mode 100644 index 0000000000..8b774704e6 --- /dev/null +++ b/drivers/crypto/ionic/ionic_crypto.h @@ -0,0 +1,209 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021-2024 Advanced Micro Devices, Inc. + */ + +#ifndef _IONIC_CRYPTO_H_ +#define _IONIC_CRYPTO_H_ + +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +#include "ionic_common.h" +#include "ionic_crypto_if.h" +#include "ionic_regs.h" + +#define IOCPT_ADMINQ_LENGTH 16 /* must be a power of two */ + +#define IOCPT_CRYPTOQ_WAIT 10 /* 1s */ + +extern int iocpt_logtype; + +#define IOCPT_PRINT(level, fmt, args...) rte_log(RTE_LOG_ ## level, \ + iocpt_logtype, "%s(): " fmt "\n", __func__, ##args) + +#define IOCPT_PRINT_CALL() IOCPT_PRINT(DEBUG, " >>") + +struct iocpt_qtype_info { + uint8_t version; + uint8_t supported; + uint64_t features; + uint16_t desc_sz; + uint16_t comp_sz; + uint16_t sg_desc_sz; + uint16_t max_sg_elems; + uint16_t sg_desc_stride; +}; + +#define IOCPT_Q_F_INITED BIT(0) +#define IOCPT_Q_F_DEFERRED BIT(1) +#define IOCPT_Q_F_SG BIT(2) + +#define Q_NEXT_TO_POST(_q, _n) (((_q)->head_idx + (_n)) & ((_q)->size_mask)) +#define Q_NEXT_TO_SRVC(_q, _n) (((_q)->tail_idx + (_n)) & ((_q)->size_mask)) + +#define IOCPT_INFO_SZ(_q) ((_q)->num_segs * sizeof(void *)) +#define IOCPT_INFO_IDX(_q, _i) ((_i) * (_q)->num_segs) +#define IOCPT_INFO_PTR(_q, _i) (&(_q)->info[IOCPT_INFO_IDX((_q), _i)]) + +struct iocpt_queue { + uint16_t num_descs; + uint16_t num_segs; + uint16_t head_idx; + uint16_t tail_idx; + uint16_t size_mask; + uint8_t type; + uint8_t hw_type; + void *base; + void *sg_base; + struct ionic_doorbell __iomem *db; + void **info; + + uint32_t index; + uint32_t hw_index; + rte_iova_t base_pa; + rte_iova_t sg_base_pa; +}; + +struct iocpt_cq { + uint16_t tail_idx; + uint16_t num_descs; + uint16_t size_mask; + bool done_color; + void *base; + rte_iova_t base_pa; +}; + +#define IOCPT_COMMON_FIELDS \ + struct iocpt_queue q; \ + struct iocpt_cq cq; \ + struct iocpt_dev *dev; \ + const struct rte_memzone *base_z; \ + void *base; \ + rte_iova_t base_pa + +struct iocpt_common_q { + IOCPT_COMMON_FIELDS; +}; + +struct iocpt_admin_q { + IOCPT_COMMON_FIELDS; + + uint16_t flags; +}; + +#define IOCPT_DEV_F_INITED BIT(0) +#define IOCPT_DEV_F_UP BIT(1) +#define IOCPT_DEV_F_FW_RESET BIT(2) + +/* Combined dev / LIF object */ +struct iocpt_dev { + const char *name; + char fw_version[IOCPT_FWVERS_BUFLEN]; + struct iocpt_identity ident; + + void *bus_dev; + struct rte_cryptodev *crypto_dev; + + union iocpt_dev_info_regs __iomem *dev_info; + union iocpt_dev_cmd_regs __iomem *dev_cmd; + + struct ionic_doorbell __iomem *db_pages; + struct ionic_intr __iomem *intr_ctrl; + + uint32_t max_qps; + uint32_t max_sessions; + uint16_t state; + uint8_t driver_id; + uint8_t socket_id; + + rte_spinlock_t adminq_lock; + rte_spinlock_t adminq_service_lock; + + struct iocpt_admin_q *adminq; + + uint64_t features; + uint32_t hw_features; + + uint32_t info_sz; + struct iocpt_lif_info *info; + rte_iova_t info_pa; + const struct rte_memzone *info_z; + + struct iocpt_qtype_info qtype_info[IOCPT_QTYPE_MAX]; + uint8_t qtype_ver[IOCPT_QTYPE_MAX]; + + struct rte_cryptodev_stats stats_base; +}; + +/** iocpt_admin_ctx - Admin command context. + * @pending_work: Flag that indicates a completion. + * @cmd: Admin command (64B) to be copied to the queue. + * @comp: Admin completion (16B) copied from the queue. + */ +struct iocpt_admin_ctx { + bool pending_work; + union iocpt_adminq_cmd cmd; + union iocpt_adminq_comp comp; +}; + +int iocpt_dev_identify(struct iocpt_dev *dev); +int iocpt_dev_init(struct iocpt_dev *dev, rte_iova_t info_pa); +int iocpt_dev_adminq_init(struct iocpt_dev *dev); +void iocpt_dev_reset(struct iocpt_dev *dev); + +int iocpt_adminq_post_wait(struct iocpt_dev *dev, struct iocpt_admin_ctx *ctx); + +struct ionic_doorbell __iomem *iocpt_db_map(struct iocpt_dev *dev, + struct iocpt_queue *q); + +typedef bool (*iocpt_cq_cb)(struct iocpt_cq *cq, uint16_t cq_desc_index, + void *cb_arg); +uint32_t iocpt_cq_service(struct iocpt_cq *cq, uint32_t work_to_do, + iocpt_cq_cb cb, void *cb_arg); + +static inline uint16_t +iocpt_q_space_avail(struct iocpt_queue *q) +{ + uint16_t avail = q->tail_idx; + + if (q->head_idx >= avail) + avail += q->num_descs - q->head_idx - 1; + else + avail -= q->head_idx + 1; + + return avail; +} + +static inline void +iocpt_q_flush(struct iocpt_queue *q) +{ + uint64_t val = IONIC_DBELL_QID(q->hw_index) | q->head_idx; + +#if defined(RTE_LIBRTE_IONIC_PMD_BARRIER_ERRATA) + /* On some devices the standard 'dmb' barrier is insufficient */ + asm volatile("dsb st" : : : "memory"); + rte_write64_relaxed(rte_cpu_to_le_64(val), q->db); +#else + rte_write64(rte_cpu_to_le_64(val), q->db); +#endif +} + +static inline bool +iocpt_is_embedded(void) +{ +#if defined(RTE_LIBRTE_IONIC_PMD_EMBEDDED) + return true; +#else + return false; +#endif +} + +#endif /* _IONIC_CRYPTO_H_ */ diff --git a/drivers/crypto/ionic/ionic_crypto_cmds.c b/drivers/crypto/ionic/ionic_crypto_cmds.c new file mode 100644 index 0000000000..ebaf8fff71 --- /dev/null +++ b/drivers/crypto/ionic/ionic_crypto_cmds.c @@ -0,0 +1,651 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021-2024 Advanced Micro Devices, Inc. + */ + +#include + +#include +#include + +#include "ionic_crypto.h" + +/* queuetype support level */ +static const uint8_t iocpt_qtype_vers[IOCPT_QTYPE_MAX] = { + [IOCPT_QTYPE_ADMINQ] = 0, /* 0 = Base version */ + [IOCPT_QTYPE_NOTIFYQ] = 0, /* 0 = Base version */ + [IOCPT_QTYPE_CRYPTOQ] = 0, /* 0 = Base version */ +}; + +static const char * +iocpt_error_to_str(enum iocpt_status_code code) +{ + switch (code) { + case IOCPT_RC_SUCCESS: + return "IOCPT_RC_SUCCESS"; + case IOCPT_RC_EVERSION: + return "IOCPT_RC_EVERSION"; + case IOCPT_RC_EOPCODE: + return "IOCPT_RC_EOPCODE"; + case IOCPT_RC_EIO: + return "IOCPT_RC_EIO"; + case IOCPT_RC_EPERM: + return "IOCPT_RC_EPERM"; + case IOCPT_RC_EQID: + return "IOCPT_RC_EQID"; + case IOCPT_RC_EQTYPE: + return "IOCPT_RC_EQTYPE"; + case IOCPT_RC_ENOENT: + return "IOCPT_RC_ENOENT"; + case IOCPT_RC_EINTR: + return "IOCPT_RC_EINTR"; + case IOCPT_RC_EAGAIN: + return "IOCPT_RC_EAGAIN"; + case IOCPT_RC_ENOMEM: + return "IOCPT_RC_ENOMEM"; + case IOCPT_RC_EFAULT: + return "IOCPT_RC_EFAULT"; + case IOCPT_RC_EBUSY: + return "IOCPT_RC_EBUSY"; + case IOCPT_RC_EEXIST: + return "IOCPT_RC_EEXIST"; + case IOCPT_RC_EINVAL: + return "IOCPT_RC_EINVAL"; + case IOCPT_RC_ENOSPC: + return "IOCPT_RC_ENOSPC"; + case IOCPT_RC_ERANGE: + return "IOCPT_RC_ERANGE"; + case IOCPT_RC_BAD_ADDR: + return "IOCPT_RC_BAD_ADDR"; + case IOCPT_RC_DEV_CMD: + return "IOCPT_RC_DEV_CMD"; + case IOCPT_RC_ERROR: + return "IOCPT_RC_ERROR"; + default: + return "IOCPT_RC_UNKNOWN"; + } +} + +static const char * +iocpt_opcode_to_str(enum iocpt_cmd_opcode opcode) +{ + switch (opcode) { + case IOCPT_CMD_NOP: + return "IOCPT_CMD_NOP"; + case IOCPT_CMD_IDENTIFY: + return "IOCPT_CMD_IDENTIFY"; + case IOCPT_CMD_RESET: + return "IOCPT_CMD_RESET"; + case IOCPT_CMD_LIF_IDENTIFY: + return "IOCPT_CMD_LIF_IDENTIFY"; + case IOCPT_CMD_LIF_INIT: + return "IOCPT_CMD_LIF_INIT"; + case IOCPT_CMD_LIF_RESET: + return "IOCPT_CMD_LIF_RESET"; + case IOCPT_CMD_LIF_GETATTR: + return "IOCPT_CMD_LIF_GETATTR"; + case IOCPT_CMD_LIF_SETATTR: + return "IOCPT_CMD_LIF_SETATTR"; + case IOCPT_CMD_Q_IDENTIFY: + return "IOCPT_CMD_Q_IDENTIFY"; + case IOCPT_CMD_Q_INIT: + return "IOCPT_CMD_Q_INIT"; + case IOCPT_CMD_Q_CONTROL: + return "IOCPT_CMD_Q_CONTROL"; + case IOCPT_CMD_SESS_CONTROL: + return "IOCPT_CMD_SESS_CONTROL"; + default: + return "DEVCMD_UNKNOWN"; + } +} + +/* Dev_cmd Interface */ + +static void +iocpt_dev_cmd_go(struct iocpt_dev *dev, union iocpt_dev_cmd *cmd) +{ + uint32_t cmd_size = RTE_DIM(cmd->words); + uint32_t i; + + IOCPT_PRINT(DEBUG, "Sending %s (%d) via dev_cmd", + iocpt_opcode_to_str(cmd->cmd.opcode), cmd->cmd.opcode); + + for (i = 0; i < cmd_size; i++) + iowrite32(cmd->words[i], &dev->dev_cmd->cmd.words[i]); + + iowrite32(0, &dev->dev_cmd->done); + iowrite32(1, &dev->dev_cmd->doorbell); +} + +static int +iocpt_dev_cmd_wait(struct iocpt_dev *dev, unsigned long max_wait) +{ + unsigned long step_usec = IONIC_DEVCMD_CHECK_PERIOD_US; + unsigned long max_wait_usec = max_wait * 1000000L; + unsigned long elapsed_usec = 0; + int done; + + /* Wait for dev cmd to complete.. but no more than max_wait sec */ + + do { + done = ioread32(&dev->dev_cmd->done) & IONIC_DEV_CMD_DONE; + if (done != 0) { + IOCPT_PRINT(DEBUG, "DEVCMD %d done took %lu usecs", + ioread8(&dev->dev_cmd->cmd.cmd.opcode), + elapsed_usec); + return 0; + } + + rte_delay_us_block(step_usec); + + elapsed_usec += step_usec; + } while (elapsed_usec < max_wait_usec); + + IOCPT_PRINT(ERR, "DEVCMD %d timeout after %lu usecs", + ioread8(&dev->dev_cmd->cmd.cmd.opcode), elapsed_usec); + + return -ETIMEDOUT; +} + +static void +iocpt_dev_cmd_comp(struct iocpt_dev *dev, void *mem) +{ + union iocpt_dev_cmd_comp *comp = mem; + uint32_t comp_size = RTE_DIM(comp->words); + uint32_t i; + + for (i = 0; i < comp_size; i++) + comp->words[i] = ioread32(&dev->dev_cmd->comp.words[i]); +} + + +static int +iocpt_dev_cmd_wait_check(struct iocpt_dev *dev, unsigned long max_wait) +{ + uint8_t status; + int err; + + err = iocpt_dev_cmd_wait(dev, max_wait); + if (err == 0) { + status = ioread8(&dev->dev_cmd->comp.comp.status); + if (status == IOCPT_RC_EAGAIN) + err = -EAGAIN; + else if (status != 0) + err = -EIO; + } + + IOCPT_PRINT(DEBUG, "dev_cmd returned %d", err); + return err; +} + +/* Dev_cmds */ + +static void +iocpt_dev_cmd_reset(struct iocpt_dev *dev) +{ + union iocpt_dev_cmd cmd = { + .reset.opcode = IOCPT_CMD_RESET, + }; + + iocpt_dev_cmd_go(dev, &cmd); +} + +static void +iocpt_dev_cmd_lif_identify(struct iocpt_dev *dev, uint8_t ver) +{ + union iocpt_dev_cmd cmd = { + .lif_identify.opcode = IOCPT_CMD_LIF_IDENTIFY, + .lif_identify.type = IOCPT_LIF_TYPE_DEFAULT, + .lif_identify.ver = ver, + }; + + iocpt_dev_cmd_go(dev, &cmd); +} + +static void +iocpt_dev_cmd_lif_init(struct iocpt_dev *dev, rte_iova_t info_pa) +{ + union iocpt_dev_cmd cmd = { + .lif_init.opcode = IOCPT_CMD_LIF_INIT, + .lif_init.type = IOCPT_LIF_TYPE_DEFAULT, + .lif_init.info_pa = info_pa, + }; + + iocpt_dev_cmd_go(dev, &cmd); +} + +static void +iocpt_dev_cmd_lif_reset(struct iocpt_dev *dev) +{ + union iocpt_dev_cmd cmd = { + .lif_reset.opcode = IOCPT_CMD_LIF_RESET, + }; + + iocpt_dev_cmd_go(dev, &cmd); +} + +static void +iocpt_dev_cmd_queue_identify(struct iocpt_dev *dev, + uint8_t qtype, uint8_t qver) +{ + union iocpt_dev_cmd cmd = { + .q_identify.opcode = IOCPT_CMD_Q_IDENTIFY, + .q_identify.type = qtype, + .q_identify.ver = qver, + }; + + iocpt_dev_cmd_go(dev, &cmd); +} + +static void +iocpt_dev_cmd_adminq_init(struct iocpt_dev *dev) +{ + struct iocpt_queue *q = &dev->adminq->q; + struct iocpt_cq *cq = &dev->adminq->cq; + + union iocpt_dev_cmd cmd = { + .q_init.opcode = IOCPT_CMD_Q_INIT, + .q_init.type = q->type, + .q_init.ver = dev->qtype_info[q->type].version, + .q_init.index = rte_cpu_to_le_32(q->index), + .q_init.flags = rte_cpu_to_le_16(IOCPT_QINIT_F_ENA), + .q_init.intr_index = rte_cpu_to_le_16(IONIC_INTR_NONE), + .q_init.ring_size = rte_log2_u32(q->num_descs), + .q_init.ring_base = rte_cpu_to_le_64(q->base_pa), + .q_init.cq_ring_base = rte_cpu_to_le_64(cq->base_pa), + }; + + IOCPT_PRINT(DEBUG, "adminq.q_init.ver %u", cmd.q_init.ver); + + iocpt_dev_cmd_go(dev, &cmd); +} + +/* Dev_cmd consumers */ + +static void +iocpt_queue_identify(struct iocpt_dev *dev) +{ + union iocpt_q_identity *q_ident = &dev->ident.q; + uint32_t q_words = RTE_DIM(q_ident->words); + uint32_t cmd_words = RTE_DIM(dev->dev_cmd->data); + uint32_t i, nwords, qtype; + int err; + + for (qtype = 0; qtype < RTE_DIM(iocpt_qtype_vers); qtype++) { + struct iocpt_qtype_info *qti = &dev->qtype_info[qtype]; + + /* Filter out the types this driver knows about */ + switch (qtype) { + case IOCPT_QTYPE_ADMINQ: + case IOCPT_QTYPE_NOTIFYQ: + case IOCPT_QTYPE_CRYPTOQ: + break; + default: + continue; + } + + memset(qti, 0, sizeof(*qti)); + + if (iocpt_is_embedded()) { + /* When embedded, FW will always match the driver */ + qti->version = iocpt_qtype_vers[qtype]; + continue; + } + + /* On the host, query the FW for info */ + iocpt_dev_cmd_queue_identify(dev, + qtype, iocpt_qtype_vers[qtype]); + err = iocpt_dev_cmd_wait_check(dev, IONIC_DEVCMD_TIMEOUT); + if (err == -EINVAL) { + IOCPT_PRINT(ERR, "qtype %d not supported\n", qtype); + continue; + } else if (err == -EIO) { + IOCPT_PRINT(ERR, "q_ident failed, older FW\n"); + return; + } else if (err != 0) { + IOCPT_PRINT(ERR, "q_ident failed, qtype %d: %d\n", + qtype, err); + return; + } + + nwords = RTE_MIN(q_words, cmd_words); + for (i = 0; i < nwords; i++) + q_ident->words[i] = ioread32(&dev->dev_cmd->data[i]); + + qti->version = q_ident->version; + qti->supported = q_ident->supported; + qti->features = rte_le_to_cpu_64(q_ident->features); + qti->desc_sz = rte_le_to_cpu_16(q_ident->desc_sz); + qti->comp_sz = rte_le_to_cpu_16(q_ident->comp_sz); + qti->sg_desc_sz = rte_le_to_cpu_16(q_ident->sg_desc_sz); + qti->max_sg_elems = rte_le_to_cpu_16(q_ident->max_sg_elems); + qti->sg_desc_stride = + rte_le_to_cpu_16(q_ident->sg_desc_stride); + + IOCPT_PRINT(DEBUG, " qtype[%d].version = %d", + qtype, qti->version); + IOCPT_PRINT(DEBUG, " qtype[%d].supported = %#x", + qtype, qti->supported); + IOCPT_PRINT(DEBUG, " qtype[%d].features = %#jx", + qtype, qti->features); + IOCPT_PRINT(DEBUG, " qtype[%d].desc_sz = %d", + qtype, qti->desc_sz); + IOCPT_PRINT(DEBUG, " qtype[%d].comp_sz = %d", + qtype, qti->comp_sz); + IOCPT_PRINT(DEBUG, " qtype[%d].sg_desc_sz = %d", + qtype, qti->sg_desc_sz); + IOCPT_PRINT(DEBUG, " qtype[%d].max_sg_elems = %d", + qtype, qti->max_sg_elems); + IOCPT_PRINT(DEBUG, " qtype[%d].sg_desc_stride = %d", + qtype, qti->sg_desc_stride); + } +} + +int +iocpt_dev_identify(struct iocpt_dev *dev) +{ + union iocpt_lif_identity *ident = &dev->ident.lif; + union iocpt_lif_config *cfg = &ident->config; + uint64_t features; + uint32_t cmd_size = RTE_DIM(dev->dev_cmd->data); + uint32_t dev_size = RTE_DIM(ident->words); + uint32_t i, nwords; + int err; + + memset(ident, 0, sizeof(*ident)); + + iocpt_dev_cmd_lif_identify(dev, IOCPT_IDENTITY_VERSION_1); + err = iocpt_dev_cmd_wait_check(dev, IONIC_DEVCMD_TIMEOUT); + if (err != 0) + return err; + + nwords = RTE_MIN(dev_size, cmd_size); + for (i = 0; i < nwords; i++) + ident->words[i] = ioread32(&dev->dev_cmd->data[i]); + + dev->max_qps = + rte_le_to_cpu_32(cfg->queue_count[IOCPT_QTYPE_CRYPTOQ]); + dev->max_sessions = + rte_le_to_cpu_32(ident->max_nb_sessions); + + features = rte_le_to_cpu_64(ident->features); + dev->features = RTE_CRYPTODEV_FF_HW_ACCELERATED; + if (features & IOCPT_HW_SYM) + dev->features |= RTE_CRYPTODEV_FF_SYMMETRIC_CRYPTO; + if (features & IOCPT_HW_ASYM) + dev->features |= RTE_CRYPTODEV_FF_ASYMMETRIC_CRYPTO; + if (features & IOCPT_HW_CHAIN) + dev->features |= RTE_CRYPTODEV_FF_SYM_OPERATION_CHAINING; + if (features & IOCPT_HW_IP) + dev->features |= RTE_CRYPTODEV_FF_IN_PLACE_SGL; + if (features & IOCPT_HW_OOP) { + dev->features |= RTE_CRYPTODEV_FF_OOP_SGL_IN_SGL_OUT; + dev->features |= RTE_CRYPTODEV_FF_OOP_SGL_IN_LB_OUT; + dev->features |= RTE_CRYPTODEV_FF_OOP_LB_IN_LB_OUT; + dev->features |= RTE_CRYPTODEV_FF_OOP_LB_IN_SGL_OUT; + } + + IOCPT_PRINT(INFO, "crypto.features %#jx", + rte_le_to_cpu_64(ident->features)); + IOCPT_PRINT(INFO, "crypto.features_active %#jx", + rte_le_to_cpu_64(cfg->features)); + IOCPT_PRINT(INFO, "crypto.queue_count[IOCPT_QTYPE_ADMINQ] %#x", + rte_le_to_cpu_32(cfg->queue_count[IOCPT_QTYPE_ADMINQ])); + IOCPT_PRINT(INFO, "crypto.queue_count[IOCPT_QTYPE_NOTIFYQ] %#x", + rte_le_to_cpu_32(cfg->queue_count[IOCPT_QTYPE_NOTIFYQ])); + IOCPT_PRINT(INFO, "crypto.queue_count[IOCPT_QTYPE_CRYPTOQ] %#x", + rte_le_to_cpu_32(cfg->queue_count[IOCPT_QTYPE_CRYPTOQ])); + IOCPT_PRINT(INFO, "crypto.max_sessions %u", + rte_le_to_cpu_32(ident->max_nb_sessions)); + + iocpt_queue_identify(dev); + + return 0; +} + +int +iocpt_dev_init(struct iocpt_dev *dev, rte_iova_t info_pa) +{ + uint32_t retries = 5; + int err; + +retry_lif_init: + iocpt_dev_cmd_lif_init(dev, info_pa); + + err = iocpt_dev_cmd_wait_check(dev, IONIC_DEVCMD_TIMEOUT); + if (err == -EAGAIN && retries > 0) { + retries--; + rte_delay_us_block(IONIC_DEVCMD_RETRY_WAIT_US); + goto retry_lif_init; + } + + return err; +} + +void +iocpt_dev_reset(struct iocpt_dev *dev) +{ + iocpt_dev_cmd_lif_reset(dev); + (void)iocpt_dev_cmd_wait_check(dev, IONIC_DEVCMD_TIMEOUT); + + iocpt_dev_cmd_reset(dev); + (void)iocpt_dev_cmd_wait_check(dev, IONIC_DEVCMD_TIMEOUT); +} + +int +iocpt_dev_adminq_init(struct iocpt_dev *dev) +{ + struct iocpt_queue *q = &dev->adminq->q; + struct iocpt_q_init_comp comp; + uint32_t retries = 5; + int err; + +retry_adminq_init: + iocpt_dev_cmd_adminq_init(dev); + + err = iocpt_dev_cmd_wait_check(dev, IONIC_DEVCMD_TIMEOUT); + if (err == -EAGAIN && retries > 0) { + retries--; + rte_delay_us_block(IONIC_DEVCMD_RETRY_WAIT_US); + goto retry_adminq_init; + } + if (err != 0) + return err; + + iocpt_dev_cmd_comp(dev, &comp); + + q->hw_type = comp.hw_type; + q->hw_index = rte_le_to_cpu_32(comp.hw_index); + q->db = iocpt_db_map(dev, q); + + IOCPT_PRINT(DEBUG, "adminq->hw_type %d", q->hw_type); + IOCPT_PRINT(DEBUG, "adminq->hw_index %d", q->hw_index); + IOCPT_PRINT(DEBUG, "adminq->db %p", q->db); + + dev->adminq->flags |= IOCPT_Q_F_INITED; + + return 0; +} + +/* Admin_cmd interface */ + +static bool +iocpt_adminq_service(struct iocpt_cq *cq, uint16_t cq_desc_index, + void *cb_arg __rte_unused) +{ + struct iocpt_admin_comp *cq_desc_base = cq->base; + struct iocpt_admin_comp *cq_desc = &cq_desc_base[cq_desc_index]; + struct iocpt_admin_q *adminq = + container_of(cq, struct iocpt_admin_q, cq); + struct iocpt_queue *q = &adminq->q; + struct iocpt_admin_ctx *ctx; + uint16_t curr_q_tail_idx; + uint16_t stop_index; + void **info; + + if (!iocpt_color_match(cq_desc->color, cq->done_color)) + return false; + + stop_index = rte_le_to_cpu_16(cq_desc->comp_index); + + do { + info = IOCPT_INFO_PTR(q, q->tail_idx); + + ctx = info[0]; + if (ctx != NULL) { + memcpy(&ctx->comp, cq_desc, sizeof(*cq_desc)); + ctx->pending_work = false; /* done */ + } + + curr_q_tail_idx = q->tail_idx; + q->tail_idx = Q_NEXT_TO_SRVC(q, 1); + } while (curr_q_tail_idx != stop_index); + + return true; +} + +/** iocpt_adminq_post - Post an admin command. + * @dev: Handle to dev. + * @cmd_ctx: Api admin command context. + * + * Return: zero or negative error status. + */ +static int +iocpt_adminq_post(struct iocpt_dev *dev, struct iocpt_admin_ctx *ctx) +{ + struct iocpt_queue *q = &dev->adminq->q; + struct iocpt_admin_cmd *q_desc_base = q->base; + struct iocpt_admin_cmd *q_desc; + void **info; + int err = 0; + + rte_spinlock_lock(&dev->adminq_lock); + + if (iocpt_q_space_avail(q) < 1) { + err = -ENOSPC; + goto err_out; + } + + q_desc = &q_desc_base[q->head_idx]; + + memcpy(q_desc, &ctx->cmd, sizeof(ctx->cmd)); + + info = IOCPT_INFO_PTR(q, q->head_idx); + info[0] = ctx; + + q->head_idx = Q_NEXT_TO_POST(q, 1); + + /* Ring doorbell */ + iocpt_q_flush(q); + +err_out: + rte_spinlock_unlock(&dev->adminq_lock); + + return err; +} + +static int +iocpt_adminq_wait_for_completion(struct iocpt_dev *dev, + struct iocpt_admin_ctx *ctx, unsigned long max_wait) +{ + struct iocpt_queue *q = &dev->adminq->q; + unsigned long step_usec = IONIC_DEVCMD_CHECK_PERIOD_US; + unsigned long step_deadline; + unsigned long max_wait_usec = max_wait * 1000000L; + unsigned long elapsed_usec = 0; + int budget = 8; + uint16_t idx; + void **info; + + step_deadline = IONIC_ADMINQ_WDOG_MS * 1000 / step_usec; + + while (ctx->pending_work && elapsed_usec < max_wait_usec) { + /* + * Locking here as adminq is served inline and could be + * called from multiple places + */ + rte_spinlock_lock(&dev->adminq_service_lock); + + iocpt_cq_service(&dev->adminq->cq, budget, + iocpt_adminq_service, NULL); + + /* + * Ring the doorbell again if work is pending after step_usec. + */ + if (ctx->pending_work && !step_deadline) { + step_deadline = IONIC_ADMINQ_WDOG_MS * + 1000 / step_usec; + + rte_spinlock_lock(&dev->adminq_lock); + idx = Q_NEXT_TO_POST(q, -1); + info = IOCPT_INFO_PTR(q, idx); + if (info[0] == ctx) + iocpt_q_flush(q); + rte_spinlock_unlock(&dev->adminq_lock); + } + + rte_spinlock_unlock(&dev->adminq_service_lock); + + rte_delay_us_block(step_usec); + elapsed_usec += step_usec; + step_deadline--; + } + + return (!ctx->pending_work); +} + +static int +iocpt_adminq_check_err(struct iocpt_admin_ctx *ctx, bool timeout) +{ + const char *name; + const char *status; + + name = iocpt_opcode_to_str(ctx->cmd.cmd.opcode); + + if (ctx->comp.comp.status == IOCPT_RC_EAGAIN) { + IOCPT_PRINT(DEBUG, "%s (%d) returned EAGAIN (%d)", + name, ctx->cmd.cmd.opcode, + ctx->comp.comp.status); + return -EAGAIN; + } + if (ctx->comp.comp.status != 0 || timeout) { + status = iocpt_error_to_str(ctx->comp.comp.status); + IOCPT_PRINT(ERR, "%s (%d) failed: %s (%d)", + name, + ctx->cmd.cmd.opcode, + timeout ? "TIMEOUT" : status, + timeout ? -1 : ctx->comp.comp.status); + return -EIO; + } + + if (ctx->cmd.cmd.opcode != IOCPT_CMD_SESS_CONTROL) { + IOCPT_PRINT(DEBUG, "%s (%d) succeeded", + name, ctx->cmd.cmd.opcode); + } + + return 0; +} + +int +iocpt_adminq_post_wait(struct iocpt_dev *dev, struct iocpt_admin_ctx *ctx) +{ + bool done; + int err; + + if (ctx->cmd.cmd.opcode != IOCPT_CMD_SESS_CONTROL) { + IOCPT_PRINT(DEBUG, "Sending %s (%d) via the admin queue", + iocpt_opcode_to_str(ctx->cmd.cmd.opcode), + ctx->cmd.cmd.opcode); + } + + err = iocpt_adminq_post(dev, ctx); + if (err != 0) { + IOCPT_PRINT(ERR, "Failure posting %d to the admin queue (%d)", + ctx->cmd.cmd.opcode, err); + return err; + } + + done = iocpt_adminq_wait_for_completion(dev, ctx, + IONIC_DEVCMD_TIMEOUT); + + return iocpt_adminq_check_err(ctx, !done /* timed out */); +} diff --git a/drivers/crypto/ionic/ionic_crypto_main.c b/drivers/crypto/ionic/ionic_crypto_main.c new file mode 100644 index 0000000000..0ca97bd1fd --- /dev/null +++ b/drivers/crypto/ionic/ionic_crypto_main.c @@ -0,0 +1,40 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright 2021-2024 Advanced Micro Devices, Inc. + */ + +#include + +#include +#include +#include + +#include "ionic_crypto.h" + +int iocpt_logtype; + +uint32_t +iocpt_cq_service(struct iocpt_cq *cq, uint32_t work_to_do, + iocpt_cq_cb cb, void *cb_arg) +{ + uint32_t work_done = 0; + + if (work_to_do == 0) + return 0; + + while (cb(cq, cq->tail_idx, cb_arg)) { + cq->tail_idx = Q_NEXT_TO_SRVC(cq, 1); + if (cq->tail_idx == 0) + cq->done_color = !cq->done_color; + + if (++work_done == work_to_do) + break; + } + + return work_done; +} + +struct ionic_doorbell * +iocpt_db_map(struct iocpt_dev *dev, struct iocpt_queue *q) +{ + return dev->db_pages + q->hw_type; +} diff --git a/drivers/crypto/ionic/meson.build b/drivers/crypto/ionic/meson.build new file mode 100644 index 0000000000..6eaef41196 --- /dev/null +++ b/drivers/crypto/ionic/meson.build @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: BSD-3-Clause +# Copyright 2021-2024 Advanced Micro Devices, Inc. + +deps += ['common_ionic'] + +sources = files( + 'ionic_crypto_cmds.c', + 'ionic_crypto_main.c', +) +name = 'ionic_crypto' + +includes += include_directories('../../common/ionic') diff --git a/drivers/crypto/meson.build b/drivers/crypto/meson.build index ee5377deff..e799861bb6 100644 --- a/drivers/crypto/meson.build +++ b/drivers/crypto/meson.build @@ -10,6 +10,7 @@ drivers = [ 'cnxk', 'dpaa_sec', 'dpaa2_sec', + 'ionic', 'ipsec_mb', 'mlx5', 'mvsam',