From patchwork Fri Sep 17 15:42:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Conor Walsh X-Patchwork-Id: 99231 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id E448FA0C43; Fri, 17 Sep 2021 17:42:50 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 23A2F41181; Fri, 17 Sep 2021 17:42:40 +0200 (CEST) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mails.dpdk.org (Postfix) with ESMTP id 0D87941173 for ; Fri, 17 Sep 2021 17:42:37 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10110"; a="222872682" X-IronPort-AV: E=Sophos;i="5.85,301,1624345200"; d="scan'208";a="222872682" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Sep 2021 08:42:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,301,1624345200"; d="scan'208";a="509992067" Received: from silpixa00401160.ir.intel.com ([10.55.128.248]) by fmsmga008.fm.intel.com with ESMTP; 17 Sep 2021 08:42:35 -0700 From: Conor Walsh To: bruce.richardson@intel.com, fengchengwen@huawei.com, jerinj@marvell.com, kevin.laatz@intel.com Cc: dev@dpdk.org, Conor Walsh Date: Fri, 17 Sep 2021 15:42:19 +0000 Message-Id: <20210917154227.737554-4-conor.walsh@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210917154227.737554-1-conor.walsh@intel.com> References: <20210827172550.1522362-1-conor.walsh@intel.com> <20210917154227.737554-1-conor.walsh@intel.com> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v4 03/11] dma/ioat: add datapath structures X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add data structures required for the data path of IOAT devices. Signed-off-by: Conor Walsh Signed-off-by: Bruce Richardson Reviewed-by: Kevin Laatz --- drivers/dma/ioat/ioat_dmadev.c | 61 ++++++++- drivers/dma/ioat/ioat_hw_defs.h | 214 ++++++++++++++++++++++++++++++++ 2 files changed, 274 insertions(+), 1 deletion(-) diff --git a/drivers/dma/ioat/ioat_dmadev.c b/drivers/dma/ioat/ioat_dmadev.c index b815d30bcf..370be3ec72 100644 --- a/drivers/dma/ioat/ioat_dmadev.c +++ b/drivers/dma/ioat/ioat_dmadev.c @@ -15,11 +15,70 @@ RTE_LOG_REGISTER_DEFAULT(ioat_pmd_logtype, INFO); #define IOAT_PMD_NAME dmadev_ioat #define IOAT_PMD_NAME_STR RTE_STR(IOAT_PMD_NAME) +/* Dump DMA device info. */ +static int +ioat_dev_dump(const struct rte_dma_dev *dev, FILE *f) +{ + struct ioat_dmadev *ioat = dev->dev_private; + uint64_t chansts_masked = ioat->regs->chansts & IOAT_CHANSTS_STATUS; + uint32_t chanerr = ioat->regs->chanerr; + uint64_t mask = (ioat->qcfg.nb_desc - 1); + fprintf(f, "========= IOAT =========\n"); + fprintf(f, " Channel status: %s [0x%"PRIx64"]\n", + chansts_readable[chansts_masked], chansts_masked); + fprintf(f, " ChainADDR: 0x%"PRIu64"\n", ioat->regs->chainaddr); + if (chanerr == 0) { + fprintf(f, " No Channel Errors\n"); + } else { + fprintf(f, " ChanERR: 0x%"PRIu32"\n", chanerr); + if (chanerr & IOAT_CHANERR_INVALID_SRC_ADDR_MASK) + fprintf(f, " Invalid Source Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_DST_ADDR_MASK) + fprintf(f, " Invalid Destination Address\n"); + if (chanerr & IOAT_CHANERR_INVALID_LENGTH_MASK) + fprintf(f, " Invalid Descriptor Length\n"); + if (chanerr & IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK) + fprintf(f, " Descriptor Read Error\n"); + if ((chanerr & ~(IOAT_CHANERR_INVALID_SRC_ADDR_MASK | + IOAT_CHANERR_INVALID_DST_ADDR_MASK | + IOAT_CHANERR_INVALID_LENGTH_MASK | + IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK)) != 0) + fprintf(f, " Unknown Error(s)\n"); + } + fprintf(f, "== Private Data ==\n"); + fprintf(f, " Config: { ring_size: %u }\n", ioat->qcfg.nb_desc); + fprintf(f, " Status: 0x%"PRIx64"\n", ioat->status); + fprintf(f, " Status IOVA: 0x%"PRIx64"\n", ioat->status_addr); + fprintf(f, " Status ADDR: %p\n", &ioat->status); + fprintf(f, " Ring IOVA: 0x%"PRIx64"\n", ioat->ring_addr); + fprintf(f, " Ring ADDR: 0x%"PRIx64"\n", ioat->desc_ring[0].next-64); + fprintf(f, " Next write: %"PRIu16"\n", ioat->next_write); + fprintf(f, " Next read: %"PRIu16"\n", ioat->next_read); + struct ioat_dma_hw_desc *desc_ring = &ioat->desc_ring[(ioat->next_write - 1) & mask]; + fprintf(f, " Last Descriptor Written {\n"); + fprintf(f, " Size: %"PRIu32"\n", desc_ring->size); + fprintf(f, " Control: 0x%"PRIx32"\n", desc_ring->u.control_raw); + fprintf(f, " Src: 0x%"PRIx64"\n", desc_ring->src_addr); + fprintf(f, " Dest: 0x%"PRIx64"\n", desc_ring->dest_addr); + fprintf(f, " Next: 0x%"PRIx64"\n", desc_ring->next); + fprintf(f, " }\n"); + fprintf(f, " Next Descriptor {\n"); + fprintf(f, " Size: %"PRIu32"\n", ioat->desc_ring[ioat->next_read & mask].size); + fprintf(f, " Src: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].src_addr); + fprintf(f, " Dest: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].dest_addr); + fprintf(f, " Next: 0x%"PRIx64"\n", ioat->desc_ring[ioat->next_read & mask].next); + fprintf(f, " }\n"); + + return 0; +} + /* Create a DMA device. */ static int ioat_dmadev_create(const char *name, struct rte_pci_device *dev) { - static const struct rte_dma_dev_ops ioat_dmadev_ops = { }; + static const struct rte_dma_dev_ops ioat_dmadev_ops = { + .dev_dump = ioat_dev_dump, + }; struct rte_dma_dev *dmadev = NULL; struct ioat_dmadev *ioat = NULL; diff --git a/drivers/dma/ioat/ioat_hw_defs.h b/drivers/dma/ioat/ioat_hw_defs.h index 73bdf548b3..1f47209e60 100644 --- a/drivers/dma/ioat/ioat_hw_defs.h +++ b/drivers/dma/ioat/ioat_hw_defs.h @@ -43,6 +43,14 @@ extern "C" { #define IOAT_CHANCTRL_ERR_COMPLETION_EN 0x0004 #define IOAT_CHANCTRL_INT_REARM 0x0001 +/* DMA Channel Capabilities */ +#define IOAT_DMACAP_PB (1 << 0) +#define IOAT_DMACAP_DCA (1 << 4) +#define IOAT_DMACAP_BFILL (1 << 6) +#define IOAT_DMACAP_XOR (1 << 8) +#define IOAT_DMACAP_PQ (1 << 9) +#define IOAT_DMACAP_DMA_DIF (1 << 10) + struct ioat_registers { uint8_t chancnt; uint8_t xfercap; @@ -71,8 +79,214 @@ struct ioat_registers { #define IOAT_CHANCMD_RESET 0x20 #define IOAT_CHANCMD_SUSPEND 0x04 +#define IOAT_CHANSTS_STATUS 0x7ULL +#define IOAT_CHANSTS_ACTIVE 0x0 +#define IOAT_CHANSTS_IDLE 0x1 +#define IOAT_CHANSTS_SUSPENDED 0x2 +#define IOAT_CHANSTS_HALTED 0x3 +#define IOAT_CHANSTS_ARMED 0x4 + +#define IOAT_CHANERR_INVALID_SRC_ADDR_MASK (1 << 0) +#define IOAT_CHANERR_INVALID_DST_ADDR_MASK (1 << 1) +#define IOAT_CHANERR_DESCRIPTOR_READ_ERROR_MASK (1 << 8) +#define IOAT_CHANERR_INVALID_LENGTH_MASK (1 << 10) + +const char *chansts_readable[] = { + "ACTIVE", /* 0x0 */ + "IDLE", /* 0x1 */ + "SUSPENDED", /* 0x2 */ + "HALTED", /* 0x3 */ + "ARMED" /* 0x4 */ +}; + +#define IOAT_CHANSTS_UNAFFILIATED_ERROR 0x8ULL +#define IOAT_CHANSTS_SOFT_ERROR 0x10ULL + +#define IOAT_CHANSTS_COMPLETED_DESCRIPTOR_MASK (~0x3FULL) + #define IOAT_CHANCMP_ALIGN 8 /* CHANCMP address must be 64-bit aligned */ +struct ioat_dma_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t null: 1; + uint32_t src_page_break: 1; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_COPY 0x00 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t reserved2; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_fill_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t reserved: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t reserved2: 2; + uint32_t dest_page_break: 1; + uint32_t bundle: 1; + uint32_t reserved3: 15; +#define IOAT_OP_FILL 0x01 + uint32_t op: 8; + } control; + } u; + uint64_t src_data; + uint64_t dest_addr; + uint64_t next; + uint64_t reserved; + uint64_t next_dest_addr; + uint64_t user1; + uint64_t user2; +}; + +struct ioat_xor_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t reserved: 13; +#define IOAT_OP_XOR 0x87 +#define IOAT_OP_XOR_VAL 0x88 + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t dest_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint64_t src_addr4; + uint64_t src_addr5; +}; + +struct ioat_xor_ext_hw_desc { + uint64_t src_addr6; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t next; + uint64_t reserved[4]; +}; + +struct ioat_pq_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_count: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 11; +#define IOAT_OP_PQ 0x89 +#define IOAT_OP_PQ_VAL 0x8a + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t src_addr3; + uint8_t coef[8]; + uint64_t q_addr; +}; + +struct ioat_pq_ext_hw_desc { + uint64_t src_addr4; + uint64_t src_addr5; + uint64_t src_addr6; + uint64_t next; + uint64_t src_addr7; + uint64_t src_addr8; + uint64_t reserved[2]; +}; + +struct ioat_pq_update_hw_desc { + uint32_t size; + union { + uint32_t control_raw; + struct { + uint32_t int_enable: 1; + uint32_t src_snoop_disable: 1; + uint32_t dest_snoop_disable: 1; + uint32_t completion_update: 1; + uint32_t fence: 1; + uint32_t src_cnt: 3; + uint32_t bundle: 1; + uint32_t dest_dca: 1; + uint32_t hint: 1; + uint32_t p_disable: 1; + uint32_t q_disable: 1; + uint32_t reserved: 3; + uint32_t coef: 8; +#define IOAT_OP_PQ_UP 0x8b + uint32_t op: 8; + } control; + } u; + uint64_t src_addr; + uint64_t p_addr; + uint64_t next; + uint64_t src_addr2; + uint64_t p_src; + uint64_t q_src; + uint64_t q_addr; +}; + +union ioat_hw_desc { + struct ioat_dma_hw_desc dma; + struct ioat_fill_hw_desc fill; + struct ioat_xor_hw_desc xor_desc; + struct ioat_xor_ext_hw_desc xor_ext; + struct ioat_pq_hw_desc pq; + struct ioat_pq_ext_hw_desc pq_ext; + struct ioat_pq_update_hw_desc pq_update; +}; + +#define GENSTS_DEV_STATE_MASK 0x03 +#define CMDSTATUS_ACTIVE_SHIFT 31 +#define CMDSTATUS_ACTIVE_MASK (1 << 31) +#define CMDSTATUS_ERR_MASK 0xFF + #ifdef __cplusplus } #endif