From patchwork Fri Mar 31 02:06:16 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "John Daley (johndale)" X-Patchwork-Id: 22978 X-Patchwork-Delegate: ferruh.yigit@amd.com Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id B469A2C31; Fri, 31 Mar 2017 04:06:42 +0200 (CEST) Received: from alln-iport-5.cisco.com (alln-iport-5.cisco.com [173.37.142.92]) by dpdk.org (Postfix) with ESMTP id 74AB22B93 for ; Fri, 31 Mar 2017 04:06:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=17506; q=dns/txt; s=iport; t=1490925987; x=1492135587; h=from:to:cc:subject:date:message-id:in-reply-to: references; bh=TtaDlmxW3IDBX0RBVavDeRILAkl4fg6obVeZsoo2Kfk=; b=Xwk60LUjuwzRTJvUUa4asb+diTbvhUh80To3RfP4sZfTMGJZGY0LefAB 2ZgnIBl9cfnMEiC1rgw/EifzFx5ptREEIp6MmtUWvR3Rhz4orVOi8nYkn CwcdI5zHnp50dxOPN9j3g6k9HVQzbjmV1RzcjOhQlKA5+XJg68JtD+T0p M=; X-IronPort-AV: E=Sophos;i="5.36,250,1486425600"; d="scan'208";a="403662379" Received: from alln-core-11.cisco.com ([173.36.13.133]) by alln-iport-5.cisco.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 31 Mar 2017 02:06:26 +0000 Received: from cisco.com (savbu-usnic-a.cisco.com [10.193.184.48]) by alln-core-11.cisco.com (8.14.5/8.14.5) with ESMTP id v2V26QqD020917; Fri, 31 Mar 2017 02:06:26 GMT Received: by cisco.com (Postfix, from userid 392789) id 0BA003FAAF14; Thu, 30 Mar 2017 19:06:26 -0700 (PDT) From: John Daley To: ferruh.yigit@intel.com, john.mcnamara@intel.com Cc: dev@dpdk.org, John Daley Date: Thu, 30 Mar 2017 19:06:16 -0700 Message-Id: <20170331020622.25498-2-johndale@cisco.com> X-Mailer: git-send-email 2.12.0 In-Reply-To: <20170331020622.25498-1-johndale@cisco.com> References: <20170330212838.31291-1-johndale@cisco.com> <20170331020622.25498-1-johndale@cisco.com> Subject: [dpdk-dev] [PATCH v2 1/7] net/enic: bring NIC interface functions up to date X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Update the base functions for the Cisco VIC. These files are mostly common with other VIC drivers so are left alone is as much as possilbe. Includes in a new filter/action interface which is needed for Generic Flow API PMD support. Update FDIR code to use the new interface. Signed-off-by: John Daley Reviewed-by: Nelson Escobar --- drivers/net/enic/base/cq_enet_desc.h | 13 +++ drivers/net/enic/base/vnic_dev.c | 162 ++++++++++++++++++++++++++++------- drivers/net/enic/base/vnic_dev.h | 5 +- drivers/net/enic/base/vnic_devcmd.h | 81 +++++++++++++++++- drivers/net/enic/enic_clsf.c | 16 ++-- 5 files changed, 238 insertions(+), 39 deletions(-) diff --git a/drivers/net/enic/base/cq_enet_desc.h b/drivers/net/enic/base/cq_enet_desc.h index f9822a450..e8410563a 100644 --- a/drivers/net/enic/base/cq_enet_desc.h +++ b/drivers/net/enic/base/cq_enet_desc.h @@ -71,6 +71,19 @@ struct cq_enet_rq_desc { u8 type_color; }; +/* Completion queue descriptor: Ethernet receive queue, 16B */ +struct cq_enet_rq_clsf_desc { + __le16 completed_index_flags; + __le16 q_number_rss_type_flags; + __le16 filter_id; + __le16 lif; + __le16 bytes_written_flags; + __le16 vlan; + __le16 checksum_fcoe; + u8 flags; + u8 type_color; +}; + #define CQ_ENET_RQ_DESC_FLAGS_INGRESS_PORT (0x1 << 12) #define CQ_ENET_RQ_DESC_FLAGS_FCOE (0x1 << 13) #define CQ_ENET_RQ_DESC_FLAGS_EOP (0x1 << 14) diff --git a/drivers/net/enic/base/vnic_dev.c b/drivers/net/enic/base/vnic_dev.c index 84e4840af..7b3aed31a 100644 --- a/drivers/net/enic/base/vnic_dev.c +++ b/drivers/net/enic/base/vnic_dev.c @@ -387,17 +387,24 @@ static int _vnic_dev_cmd(struct vnic_dev *vdev, enum vnic_devcmd_cmd cmd, static int vnic_dev_cmd_proxy(struct vnic_dev *vdev, enum vnic_devcmd_cmd proxy_cmd, enum vnic_devcmd_cmd cmd, - u64 *a0, u64 *a1, int wait) + u64 *args, int nargs, int wait) { u32 status; int err; + /* + * Proxy command consumes 2 arguments. One for proxy index, + * the other is for command to be proxied + */ + if (nargs > VNIC_DEVCMD_NARGS - 2) { + pr_err("number of args %d exceeds the maximum\n", nargs); + return -EINVAL; + } memset(vdev->args, 0, sizeof(vdev->args)); vdev->args[0] = vdev->proxy_index; vdev->args[1] = cmd; - vdev->args[2] = *a0; - vdev->args[3] = *a1; + memcpy(&vdev->args[2], args, nargs * sizeof(args[0])); err = _vnic_dev_cmd(vdev, proxy_cmd, wait); if (err) @@ -412,24 +419,26 @@ static int vnic_dev_cmd_proxy(struct vnic_dev *vdev, return err; } - *a0 = vdev->args[1]; - *a1 = vdev->args[2]; + memcpy(args, &vdev->args[1], nargs * sizeof(args[0])); return 0; } static int vnic_dev_cmd_no_proxy(struct vnic_dev *vdev, - enum vnic_devcmd_cmd cmd, u64 *a0, u64 *a1, int wait) + enum vnic_devcmd_cmd cmd, u64 *args, int nargs, int wait) { int err; - vdev->args[0] = *a0; - vdev->args[1] = *a1; + if (nargs > VNIC_DEVCMD_NARGS) { + pr_err("number of args %d exceeds the maximum\n", nargs); + return -EINVAL; + } + memset(vdev->args, 0, sizeof(vdev->args)); + memcpy(vdev->args, args, nargs * sizeof(args[0])); err = _vnic_dev_cmd(vdev, cmd, wait); - *a0 = vdev->args[0]; - *a1 = vdev->args[1]; + memcpy(args, vdev->args, nargs * sizeof(args[0])); return err; } @@ -455,24 +464,64 @@ void vnic_dev_cmd_proxy_end(struct vnic_dev *vdev) int vnic_dev_cmd(struct vnic_dev *vdev, enum vnic_devcmd_cmd cmd, u64 *a0, u64 *a1, int wait) { + u64 args[2]; + int err; + + args[0] = *a0; + args[1] = *a1; memset(vdev->args, 0, sizeof(vdev->args)); switch (vdev->proxy) { case PROXY_BY_INDEX: + err = vnic_dev_cmd_proxy(vdev, CMD_PROXY_BY_INDEX, cmd, + args, ARRAY_SIZE(args), wait); + break; + case PROXY_BY_BDF: + err = vnic_dev_cmd_proxy(vdev, CMD_PROXY_BY_BDF, cmd, + args, ARRAY_SIZE(args), wait); + break; + case PROXY_NONE: + default: + err = vnic_dev_cmd_no_proxy(vdev, cmd, args, 2, wait); + break; + } + + if (err == 0) { + *a0 = args[0]; + *a1 = args[1]; + } + + return err; +} + +int vnic_dev_cmd_args(struct vnic_dev *vdev, enum vnic_devcmd_cmd cmd, + u64 *args, int nargs, int wait) +{ + switch (vdev->proxy) { + case PROXY_BY_INDEX: return vnic_dev_cmd_proxy(vdev, CMD_PROXY_BY_INDEX, cmd, - a0, a1, wait); + args, nargs, wait); case PROXY_BY_BDF: return vnic_dev_cmd_proxy(vdev, CMD_PROXY_BY_BDF, cmd, - a0, a1, wait); + args, nargs, wait); case PROXY_NONE: default: - return vnic_dev_cmd_no_proxy(vdev, cmd, a0, a1, wait); + return vnic_dev_cmd_no_proxy(vdev, cmd, args, nargs, wait); } } +static int vnic_dev_advanced_filters_cap(struct vnic_dev *vdev, u64 *args, + int nargs) +{ + memset(args, 0, nargs * sizeof(*args)); + args[0] = CMD_ADD_ADV_FILTER; + args[1] = FILTER_CAP_MODE_V1_FLAG; + return vnic_dev_cmd_args(vdev, CMD_CAPABILITY, args, nargs, 1000); +} + int vnic_dev_capable_adv_filters(struct vnic_dev *vdev) { - u64 a0 = (u32)CMD_ADD_ADV_FILTER, a1 = 0; + u64 a0 = CMD_ADD_ADV_FILTER, a1 = 0; int wait = 1000; int err; @@ -482,7 +531,65 @@ int vnic_dev_capable_adv_filters(struct vnic_dev *vdev) return (a1 >= (u32)FILTER_DPDK_1); } -static int vnic_dev_capable(struct vnic_dev *vdev, enum vnic_devcmd_cmd cmd) +/* Determine the "best" filtering mode VIC is capaible of. Returns one of 3 + * value or 0 on error: + * FILTER_DPDK_1- advanced filters availabile + * FILTER_USNIC_IP_FLAG - advanced filters but with the restriction that + * the IP layer must explicitly specified. I.e. cannot have a UDP + * filter that matches both IPv4 and IPv6. + * FILTER_IPV4_5TUPLE - fallback if either of the 2 above aren't available. + * all other filter types are not available. + * Retrun true in filter_tags if supported + */ +int vnic_dev_capable_filter_mode(struct vnic_dev *vdev, u32 *mode, + u8 *filter_tags) +{ + u64 args[4]; + int err; + u32 max_level = 0; + + err = vnic_dev_advanced_filters_cap(vdev, args, 4); + + /* determine if filter tags are available */ + if (err) + *filter_tags = 0; + if ((args[2] == FILTER_CAP_MODE_V1) && + (args[3] & FILTER_ACTION_FILTER_ID_FLAG)) + *filter_tags = 1; + else + *filter_tags = 0; + + if (err || ((args[0] == 1) && (args[1] == 0))) { + /* Adv filter Command not supported or adv filters available but + * not enabled. Try the normal filter capability command. + */ + args[0] = CMD_ADD_FILTER; + args[1] = 0; + err = vnic_dev_cmd_args(vdev, CMD_CAPABILITY, args, 2, 1000); + if (err) + return err; + max_level = args[1]; + goto parse_max_level; + } else if (args[2] == FILTER_CAP_MODE_V1) { + /* parse filter capability mask in args[1] */ + if (args[1] & FILTER_DPDK_1_FLAG) + *mode = FILTER_DPDK_1; + else if (args[1] & FILTER_USNIC_IP_FLAG) + *mode = FILTER_USNIC_IP; + else if (args[1] & FILTER_IPV4_5TUPLE_FLAG) + *mode = FILTER_IPV4_5TUPLE; + return 0; + } + max_level = args[1]; +parse_max_level: + if (max_level >= (u32)FILTER_USNIC_IP) + *mode = FILTER_USNIC_IP; + else + *mode = FILTER_IPV4_5TUPLE; + return 0; +} + +int vnic_dev_capable(struct vnic_dev *vdev, enum vnic_devcmd_cmd cmd) { u64 a0 = (u32)cmd, a1 = 0; int wait = 1000; @@ -1017,32 +1124,30 @@ int vnic_dev_set_mac_addr(struct vnic_dev *vdev, u8 *mac_addr) * In case of DEL filter, the caller passes the RQ number. Return * value is irrelevant. * @data: filter data + * @action: action data */ int vnic_dev_classifier(struct vnic_dev *vdev, u8 cmd, u16 *entry, - struct filter_v2 *data) + struct filter_v2 *data, struct filter_action_v2 *action_v2) { u64 a0, a1; int wait = 1000; dma_addr_t tlv_pa; int ret = -EINVAL; struct filter_tlv *tlv, *tlv_va; - struct filter_action *action; u64 tlv_size; - u32 filter_size; + u32 filter_size, action_size; static unsigned int unique_id; char z_name[RTE_MEMZONE_NAMESIZE]; enum vnic_devcmd_cmd dev_cmd; - if (cmd == CLSF_ADD) { - if (data->type == FILTER_DPDK_1) - dev_cmd = CMD_ADD_ADV_FILTER; - else - dev_cmd = CMD_ADD_FILTER; + dev_cmd = (data->type >= FILTER_DPDK_1) ? + CMD_ADD_ADV_FILTER : CMD_ADD_FILTER; filter_size = vnic_filter_size(data); - tlv_size = filter_size + - sizeof(struct filter_action) + + action_size = vnic_action_size(action_v2); + + tlv_size = filter_size + action_size + 2*sizeof(struct filter_tlv); snprintf((char *)z_name, sizeof(z_name), "vnic_clsf_%d", unique_id++); @@ -1063,11 +1168,8 @@ int vnic_dev_classifier(struct vnic_dev *vdev, u8 cmd, u16 *entry, filter_size); tlv->type = CLSF_TLV_ACTION; - tlv->length = sizeof(struct filter_action); - action = (struct filter_action *)&tlv->val; - action->type = FILTER_ACTION_RQ_STEERING; - action->u.rq_idx = *entry; - + tlv->length = action_size; + memcpy(&tlv->val, (void *)action_v2, action_size); ret = vnic_dev_cmd(vdev, dev_cmd, &a0, &a1, wait); *entry = (u16)a0; vdev->free_consistent(vdev->priv, tlv_size, tlv_va, tlv_pa); diff --git a/drivers/net/enic/base/vnic_dev.h b/drivers/net/enic/base/vnic_dev.h index 06ebd4cea..9a9e6917a 100644 --- a/drivers/net/enic/base/vnic_dev.h +++ b/drivers/net/enic/base/vnic_dev.h @@ -135,6 +135,9 @@ void vnic_dev_cmd_proxy_end(struct vnic_dev *vdev); int vnic_dev_fw_info(struct vnic_dev *vdev, struct vnic_devcmd_fw_info **fw_info); int vnic_dev_capable_adv_filters(struct vnic_dev *vdev); +int vnic_dev_capable(struct vnic_dev *vdev, enum vnic_devcmd_cmd cmd); +int vnic_dev_capable_filter_mode(struct vnic_dev *vdev, u32 *mode, + u8 *filter_tags); int vnic_dev_asic_info(struct vnic_dev *vdev, u16 *asic_type, u16 *asic_rev); int vnic_dev_spec(struct vnic_dev *vdev, unsigned int offset, size_t size, void *value); @@ -202,7 +205,7 @@ int vnic_dev_enable2_done(struct vnic_dev *vdev, int *status); int vnic_dev_deinit_done(struct vnic_dev *vdev, int *status); int vnic_dev_set_mac_addr(struct vnic_dev *vdev, u8 *mac_addr); int vnic_dev_classifier(struct vnic_dev *vdev, u8 cmd, u16 *entry, - struct filter_v2 *data); + struct filter_v2 *data, struct filter_action_v2 *action_v2); #ifdef ENIC_VXLAN int vnic_dev_overlay_offload_enable_disable(struct vnic_dev *vdev, u8 overlay, u8 config); diff --git a/drivers/net/enic/base/vnic_devcmd.h b/drivers/net/enic/base/vnic_devcmd.h index 785fd6fdf..05d87b919 100644 --- a/drivers/net/enic/base/vnic_devcmd.h +++ b/drivers/net/enic/base/vnic_devcmd.h @@ -92,6 +92,8 @@ #define _CMD_VTYPE(cmd) (((cmd) >> _CMD_VTYPESHIFT) & _CMD_VTYPEMASK) #define _CMD_N(cmd) (((cmd) >> _CMD_NSHIFT) & _CMD_NMASK) +#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0])) + enum vnic_devcmd_cmd { CMD_NONE = _CMDC(_CMD_DIR_NONE, _CMD_VTYPE_NONE, 0), @@ -598,12 +600,29 @@ enum vnic_devcmd_cmd { * out: (u32) a0=filter identifier * * Capability query: - * out: (u64) a0= 1 if capabliity query supported - * (u64) a1= MAX filter type supported + * in: (u64) a1= supported filter capability exchange modes + * out: (u64) a0= 1 if capability query supported + * if (u64) a1 = 0: a1 = MAX filter type supported + * if (u64) a1 & FILTER_CAP_MODE_V1_FLAG: + * a1 = bitmask of supported filters + * a2 = FILTER_CAP_MODE_V1 + * a3 = bitmask of supported actions */ CMD_ADD_ADV_FILTER = _CMDC(_CMD_DIR_RW, _CMD_VTYPE_ENET, 77), }; +/* Modes for exchanging advanced filter capabilities. The modes supported by + * the driver are passed in the CMD_ADD_ADV_FILTER capability command and the + * mode selected is returned. + * V0: the maximum filter type supported is returned + * V1: bitmasks of supported filters and actions are returned + */ +enum filter_cap_mode { + FILTER_CAP_MODE_V0 = 0, /* Must always be 0 for legacy drivers */ + FILTER_CAP_MODE_V1 = 1, +}; +#define FILTER_CAP_MODE_V1_FLAG (1 << FILTER_CAP_MODE_V1) + /* CMD_ENABLE2 flags */ #define CMD_ENABLE2_STANDBY 0x0 #define CMD_ENABLE2_ACTIVE 0x1 @@ -837,6 +856,7 @@ struct filter_generic_1 { /* Specifies the filter_action type. */ enum { FILTER_ACTION_RQ_STEERING = 0, + FILTER_ACTION_V2 = 1, FILTER_ACTION_MAX }; @@ -847,6 +867,22 @@ struct filter_action { } u; } __attribute__((packed)); +#define FILTER_ACTION_RQ_STEERING_FLAG (1 << 0) +#define FILTER_ACTION_FILTER_ID_FLAG (1 << 1) +#define FILTER_ACTION_V2_ALL (FILTER_ACTION_RQ_STEERING_FLAG \ + | FILTER_ACTION_FILTER_ID_FLAG) + +/* Version 2 of filter action must be a strict extension of struct filter_action + * where the first fields exactly match in size and meaning. + */ +struct filter_action_v2 { + u32 type; + u32 rq_idx; + u32 flags; /* use FILTER_ACTION_XXX_FLAG defines */ + u16 filter_id; + u_int8_t reserved[32]; /* for future expansion */ +} __attribute__((packed)); + /* Specifies the filter type. */ enum filter_type { FILTER_USNIC_ID = 0, @@ -859,6 +895,21 @@ enum filter_type { FILTER_MAX }; +#define FILTER_USNIC_ID_FLAG (1 << FILTER_USNIC_ID) +#define FILTER_IPV4_5TUPLE_FLAG (1 << FILTER_IPV4_5TUPLE) +#define FILTER_MAC_VLAN_FLAG (1 << FILTER_MAC_VLAN) +#define FILTER_VLAN_IP_3TUPLE_FLAG (1 << FILTER_VLAN_IP_3TUPLE) +#define FILTER_NVGRE_VMQ_FLAG (1 << FILTER_NVGRE_VMQ) +#define FILTER_USNIC_IP_FLAG (1 << FILTER_USNIC_IP) +#define FILTER_DPDK_1_FLAG (1 << FILTER_DPDK_1) +#define FILTER_V1_ALL (FILTER_USNIC_ID_FLAG | \ + FILTER_IPV4_5TUPLE_FLAG | \ + FILTER_MAC_VLAN_FLAG | \ + FILTER_VLAN_IP_3TUPLE_FLAG | \ + FILTER_NVGRE_VMQ_FLAG | \ + FILTER_USNIC_IP_FLAG | \ + FILTER_DPDK_1_FLAG) + struct filter { u32 type; union { @@ -903,7 +954,7 @@ struct filter_tlv { /* Data for CMD_ADD_FILTER is 2 TLV and filter + action structs */ #define FILTER_MAX_BUF_SIZE 100 #define FILTER_V2_MAX_BUF_SIZE (sizeof(struct filter_v2) + \ - sizeof(struct filter_action) + \ + sizeof(struct filter_action_v2) + \ (2 * sizeof(struct filter_tlv))) /* @@ -949,6 +1000,30 @@ enum { }; /* + * Get the action structure size given action type. To be "future-proof," + * drivers should use this instead of "sizeof (struct filter_action_v2)" + * when computing length for TLV. + */ +static inline u_int32_t +vnic_action_size(struct filter_action_v2 *fap) +{ + u_int32_t size; + + switch (fap->type) { + case FILTER_ACTION_RQ_STEERING: + size = sizeof(struct filter_action); + break; + case FILTER_ACTION_V2: + size = sizeof(struct filter_action_v2); + break; + default: + size = sizeof(struct filter_action); + break; + } + return size; +} + +/* * Writing cmd register causes STAT_BUSY to get set in status register. * When cmd completes, STAT_BUSY will be cleared. * diff --git a/drivers/net/enic/enic_clsf.c b/drivers/net/enic/enic_clsf.c index 487f80449..9e94afdf9 100644 --- a/drivers/net/enic/enic_clsf.c +++ b/drivers/net/enic/enic_clsf.c @@ -345,7 +345,7 @@ int enic_fdir_del_fltr(struct enic *enic, struct rte_eth_fdir_filter *params) /* Delete the filter */ vnic_dev_classifier(enic->vdev, CLSF_DEL, - &key->fltr_id, NULL); + &key->fltr_id, NULL, NULL); rte_free(key); enic->fdir.nodes[pos] = NULL; enic->fdir.stats.free++; @@ -365,8 +365,10 @@ int enic_fdir_add_fltr(struct enic *enic, struct rte_eth_fdir_filter *params) u32 flowtype_supported; u16 flex_bytes; u16 queue; + struct filter_action_v2 action; memset(&fltr, 0, sizeof(fltr)); + memset(&action, 0, sizeof(action)); flowtype_supported = enic->fdir.types_mask & (1 << params->input.flow_type); @@ -439,7 +441,7 @@ int enic_fdir_add_fltr(struct enic *enic, struct rte_eth_fdir_filter *params) * Delete the filter and add the modified one later */ vnic_dev_classifier(enic->vdev, CLSF_DEL, - &key->fltr_id, NULL); + &key->fltr_id, NULL, NULL); enic->fdir.stats.free++; } @@ -451,8 +453,11 @@ int enic_fdir_add_fltr(struct enic *enic, struct rte_eth_fdir_filter *params) enic->fdir.copy_fltr_fn(&fltr, ¶ms->input, &enic->rte_dev->data->dev_conf.fdir_conf.mask); + action.type = FILTER_ACTION_RQ_STEERING; + action.rq_idx = queue; - if (!vnic_dev_classifier(enic->vdev, CLSF_ADD, &queue, &fltr)) { + if (!vnic_dev_classifier(enic->vdev, CLSF_ADD, &queue, &fltr, + &action)) { key->fltr_id = queue; } else { dev_err(enic, "Add classifier entry failed\n"); @@ -462,7 +467,8 @@ int enic_fdir_add_fltr(struct enic *enic, struct rte_eth_fdir_filter *params) } if (do_free) - vnic_dev_classifier(enic->vdev, CLSF_DEL, &old_fltr_id, NULL); + vnic_dev_classifier(enic->vdev, CLSF_DEL, &old_fltr_id, NULL, + NULL); else{ enic->fdir.stats.free--; enic->fdir.stats.add++; @@ -488,7 +494,7 @@ void enic_clsf_destroy(struct enic *enic) key = enic->fdir.nodes[index]; if (key) { vnic_dev_classifier(enic->vdev, CLSF_DEL, - &key->fltr_id, NULL); + &key->fltr_id, NULL, NULL); rte_free(key); enic->fdir.nodes[index] = NULL; }