[v3,1/1] lib/cryptodev: multi-process IPC request handler

  This patch add a function to support queue-pair configuration
request to allow the primary or secondary process to setup/free the
queue-pair via IPC handler. Add in queue pair in-used by process id
array in rte_cryptodev_data for pid tracking.

Signed-off-by: Kai Ji <kai.ji@intel.com>
---
v3:
- addin missing free function for qp_in_use_by_pid

v2:
- code rework
---
 lib/cryptodev/cryptodev_pmd.h |  3 +-
 lib/cryptodev/rte_cryptodev.c | 92 +++++++++++++++++++++++++++++++++++
 lib/cryptodev/rte_cryptodev.h | 37 ++++++++++++++
 lib/cryptodev/version.map     |  2 +
 4 files changed, 133 insertions(+), 1 deletion(-)
  

Hi Kai,

Some small comments below.

> -----Original Message-----
> From: Kai Ji <kai.ji@intel.com>
> Sent: Sunday 2 October 2022 23:45
> To: dev@dpdk.org
> Cc: gakhil@marvell.com; Ji, Kai <kai.ji@intel.com>
> Subject: [dpdk-dev v3 1/1] lib/cryptodev: multi-process IPC request handler
> 
> This patch add a function to support queue-pair configuration request to allow
> the primary or secondary process to setup/free the queue-pair via IPC handler.
> Add in queue pair in-used by process id array in rte_cryptodev_data for pid
> tracking.
> 
> Signed-off-by: Kai Ji <kai.ji@intel.com>
<snip>
> +static int
> +rte_cryptodev_ipc_request(const struct rte_mp_msg *mp_msg, const void
> +*peer) {
> +	struct rte_mp_msg mp_res;
> +	struct rte_cryptodev_mp_param *res =
> +		(struct rte_cryptodev_mp_param *)mp_res.param;
> +	const struct rte_cryptodev_mp_param *param =
> +		(const struct rte_cryptodev_mp_param *)mp_msg->param;
[CP] 
The "param" name could be improved I think - maybe follow the "res" naming and call this one "msg"
Both are "mp_***->param" so I think it is confusing to have one named "param"

> +
> +	int ret;
> +	struct rte_cryptodev *dev;
> +	uint16_t *qps_in_used_by_pid;
[CP] 
Possible typo -   change to "qp_in_use_by_pid" to match the naming of "dev->data->qp_in_use_by_pid"

<snip>
> +int rte_cryptodev_mp_request_register(void)
> +{
> +	RTE_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
> +	return rte_mp_action_register(CRYPTODEV_MP_REQ,
> +				rte_cryptodev_ipc_request);
> +}
> +
> +void rte_cryptodev_mp_request_unregister(void)
> +{
> +	RTE_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
> +	rte_mp_action_unregister(CRYPTODEV_MP_REQ);
> +}
> +
[CP] 
The above two functions should have the function return type on its own line.

<snip>
> +/**
> + * Register multi process request IPC handler
> + *
> + * @return
> + *   - 0: Success registered
> + *	 - 1: Failed registration failed
> + *	 - -EINVAL: device was not configured
[CP] 
Indent is off for success result

<snip>

Thanks,
Ciara
  

> This patch add a function to support queue-pair configuration
> request to allow the primary or secondary process to setup/free the
> queue-pair via IPC handler. Add in queue pair in-used by process id
> array in rte_cryptodev_data for pid tracking.
> 
> Signed-off-by: Kai Ji <kai.ji@intel.com>

I had a comment on v1, that you should include all PMD maintainers as well
When sending any major change in library layer.
But I think you have not read the comments made on v1.

> ---
> v3:
> - addin missing free function for qp_in_use_by_pid
> 
> v2:
> - code rework
> ---
>  lib/cryptodev/cryptodev_pmd.h |  3 +-
>  lib/cryptodev/rte_cryptodev.c | 92 +++++++++++++++++++++++++++++++++++
>  lib/cryptodev/rte_cryptodev.h | 37 ++++++++++++++
>  lib/cryptodev/version.map     |  2 +
>  4 files changed, 133 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/cryptodev/cryptodev_pmd.h b/lib/cryptodev/cryptodev_pmd.h
> index 09ba952455..f404604963 100644
> --- a/lib/cryptodev/cryptodev_pmd.h
> +++ b/lib/cryptodev/cryptodev_pmd.h
> @@ -78,7 +78,8 @@ struct rte_cryptodev_data {
>  	void **queue_pairs;
>  	/** Number of device queue pairs. */
>  	uint16_t nb_queue_pairs;
> -
> +	/** Array of process id used for queue pairs **/
> +	uint16_t *qp_in_use_by_pid;
Why array? And if an array, how will its depth will be calculated.

I commented on v1, that the in_use pid can be stored inside queue private data.


>  	/** PMD-specific private data */
>  	void *dev_private;
>  } __rte_cache_aligned;
> diff --git a/lib/cryptodev/rte_cryptodev.c b/lib/cryptodev/rte_cryptodev.c
> index 9e76a1c72d..dab6a37bff 100644
> --- a/lib/cryptodev/rte_cryptodev.c
> +++ b/lib/cryptodev/rte_cryptodev.c
> @@ -49,6 +49,9 @@ struct rte_crypto_fp_ops
> rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
>  /* spinlock for crypto device callbacks */
>  static rte_spinlock_t rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
> 
> +/* crypto queue pair config */
> +#define CRYPTODEV_MP_REQ "cryptodev_mp_request"
> +
>  /**
>   * The user application callback description.
>   *
> @@ -1050,6 +1053,9 @@ rte_cryptodev_pmd_release_device(struct
> rte_cryptodev *cryptodev)
>  			return ret;
>  	}
> 
> +	if (cryptodev->data->qp_in_use_by_pid)
> +		rte_free(cryptodev->data->qp_in_use_by_pid);
> +
>  	ret = rte_cryptodev_data_free(dev_id,
> &cryptodev_globals.data[dev_id]);
>  	if (ret < 0)
>  		return ret;
> @@ -1138,6 +1144,21 @@ rte_cryptodev_queue_pairs_config(struct
> rte_cryptodev *dev, uint16_t nb_qpairs,
> 
>  	}
>  	dev->data->nb_queue_pairs = nb_qpairs;
> +
> +	if (dev->data->qp_in_use_by_pid == NULL) {
> +		dev->data->qp_in_use_by_pid = rte_zmalloc_socket(
> +				"cryptodev->qp_in_use_by_pid",
> +				sizeof(dev->data->qp_in_use_by_pid[0]) *
> +				dev_info.max_nb_queue_pairs,
> +				RTE_CACHE_LINE_SIZE, socket_id);
> +		if (dev->data->qp_in_use_by_pid == NULL) {
> +			CDEV_LOG_ERR("failed to get memory for qp meta
> data, "
> +							"nb_queues %u",
> +							nb_qpairs);
> +			return -(ENOMEM);
> +		}
> +	}
> +
>  	return 0;
>  }
> 
> @@ -1400,6 +1421,77 @@ rte_cryptodev_queue_pair_setup(uint8_t dev_id,
> uint16_t queue_pair_id,
>  			socket_id);
>  }
> 
> +static int
> +rte_cryptodev_ipc_request(const struct rte_mp_msg *mp_msg, const void
> *peer)
> +{
> +	struct rte_mp_msg mp_res;
> +	struct rte_cryptodev_mp_param *res =
> +		(struct rte_cryptodev_mp_param *)mp_res.param;
> +	const struct rte_cryptodev_mp_param *param =
> +		(const struct rte_cryptodev_mp_param *)mp_msg->param;
> +
> +	int ret;
> +	struct rte_cryptodev *dev;
> +	uint16_t *qps_in_used_by_pid;
> +	int dev_id = param->dev_id;
> +	int qp_id = param->qp_id;
> +	struct rte_cryptodev_qp_conf *queue_conf = param->queue_conf;
> +
> +	res->result = -EINVAL;
> +	if (!rte_cryptodev_is_valid_dev(dev_id)) {
> +		CDEV_LOG_ERR("Invalid dev_id=%" PRIu8, dev_id);
> +		goto out;
> +	}
> +
> +	if (!rte_cryptodev_get_qp_status(dev_id, qp_id))
> +		goto out;
> +
> +	dev = &rte_crypto_devices[dev_id];
> +	qps_in_used_by_pid = dev->data->qp_in_use_by_pid;
> +
> +	switch (param->type) {
> +	case RTE_CRYPTODEV_MP_REQ_QP_SET:
> +		ret = rte_cryptodev_queue_pair_setup(dev_id, qp_id,
> +				queue_conf, param->socket_id);
> +		if (!ret)
> +			qps_in_used_by_pid[qp_id] = param->process_id;
> +		res->result = ret;
> +		break;
> +	case RTE_CRYPTODEV_MP_REQ_QP_FREE:
> +		if ((rte_eal_process_type() == RTE_PROC_SECONDARY) &&
> +			(qps_in_used_by_pid[qp_id] != param->process_id)) {
> +			CDEV_LOG_ERR("Unable to release qp_id=%" PRIu8,
> qp_id);
> +			goto out;
> +		}
> +
> +		ret = (*dev->dev_ops->queue_pair_release)(dev, qp_id);
> +		if (!ret)
> +			qps_in_used_by_pid[qp_id] = 0;
> +
> +		res->result = ret;
> +		break;
> +	default:
> +		CDEV_LOG_ERR("invalid mp request type\n");
> +	}
> +
> +out:
> +	ret = rte_mp_reply(&mp_res, peer);
> +	return ret;
> +}
> +
> +int rte_cryptodev_mp_request_register(void)
> +{
> +	RTE_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
> +	return rte_mp_action_register(CRYPTODEV_MP_REQ,
> +				rte_cryptodev_ipc_request);
> +}
> +
> +void rte_cryptodev_mp_request_unregister(void)
> +{
> +	RTE_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
> +	rte_mp_action_unregister(CRYPTODEV_MP_REQ);
> +}
> +

It looks an incomplete patch right now.
- Documentation is missing on how this feature will be used.
- Test cases?
- PMD implementation is key part for this feature to work.

I believe it is better to defer this to next release as we don't have a clarity on how to use this.

>  struct rte_cryptodev_cb *
>  rte_cryptodev_add_enq_callback(uint8_t dev_id,
>  			       uint16_t qp_id,
> diff --git a/lib/cryptodev/rte_cryptodev.h b/lib/cryptodev/rte_cryptodev.h
> index 56f459c6a0..d8cadebd0c 100644
> --- a/lib/cryptodev/rte_cryptodev.h
> +++ b/lib/cryptodev/rte_cryptodev.h
> @@ -539,6 +539,24 @@ enum rte_cryptodev_event_type {
>  	RTE_CRYPTODEV_EVENT_MAX		/**< max value of this enum */
>  };
> 
> +/* Request types for IPC. */
> +enum rte_cryptodev_mp_req_type {
> +	RTE_CRYPTODEV_MP_REQ_NONE,
> +	RTE_CRYPTODEV_MP_REQ_QP_SET,
> +	RTE_CRYPTODEV_MP_REQ_QP_FREE
> +};

Doxygen comments missing.

> +
> +/* Parameters for IPC. */
> +struct rte_cryptodev_mp_param {
> +	enum rte_cryptodev_mp_req_type type;
> +	int dev_id;
> +	int qp_id;
> +	int socket_id;
> +	uint16_t process_id;
> +	struct rte_cryptodev_qp_conf *queue_conf;
> +	int result;
> +};
> +

Doxygen comments???
Writing just "Parameters for IPC" does not make any sense and it is not clear
How it will be used.


>  /** Crypto device queue pair configuration structure. */
>  struct rte_cryptodev_qp_conf {
>  	uint32_t nb_descriptors; /**< Number of descriptors per queue pair */
> @@ -769,6 +787,25 @@ extern int
>  rte_cryptodev_queue_pair_setup(uint8_t dev_id, uint16_t queue_pair_id,
>  		const struct rte_cryptodev_qp_conf *qp_conf, int socket_id);
> 
> +/**
> + * Register multi process request IPC handler
> + *

This comment is not sufficient to explain what this API will do exactly.

> + * @return
> + *   - 0: Success registered
> + *	 - 1: Failed registration failed
> + *	 - -EINVAL: device was not configured
> + */
> +__rte_experimental
> +int
> +rte_cryptodev_mp_request_register(void);
> +
> +/**
> + * Unregister multi process unrequest IPC handler
> + */
> +__rte_experimental
> +void
> +rte_cryptodev_mp_request_unregister(void);
> +
>  /**
>   * Get the status of queue pairs setup on a specific crypto device
>   *
> diff --git a/lib/cryptodev/version.map b/lib/cryptodev/version.map
> index 6d9b3e01a6..4130c39e7c 100644
> --- a/lib/cryptodev/version.map
> +++ b/lib/cryptodev/version.map
> @@ -157,6 +157,8 @@ EXPERIMENTAL {
>  	__rte_cryptodev_trace_sym_session_get_user_data;
>  	__rte_cryptodev_trace_sym_session_set_user_data;
>  	__rte_cryptodev_trace_count;
> +	rte_cryptodev_mp_request_register;
> +	rte_cryptodev_mp_request_unregister;
>  };
> 
>  INTERNAL {
> --
> 2.17.1
  

Please see my comments inline

> -----Original Message-----
> From: Akhil Goyal <gakhil@marvell.com>
> Sent: Tuesday, October 4, 2022 7:13 PM
> To: Ji, Kai <kai.ji@intel.com>; dev@dpdk.org
> Subject: RE: [EXT] [dpdk-dev v3 1/1] lib/cryptodev: multi-process IPC request
> handler
> 
> > This patch add a function to support queue-pair configuration request
> > to allow the primary or secondary process to setup/free the queue-pair
> > via IPC handler. Add in queue pair in-used by process id array in
> > rte_cryptodev_data for pid tracking.
> >
> > Signed-off-by: Kai Ji <kai.ji@intel.com>
> 
> I had a comment on v1, that you should include all PMD maintainers as well
> When sending any major change in library layer.
> But I think you have not read the comments made on v1.
> 
> > ---
> > v3:
> > - addin missing free function for qp_in_use_by_pid
> >
> > v2:
> > - code rework
> > ---
> >  lib/cryptodev/cryptodev_pmd.h |  3 +-  lib/cryptodev/rte_cryptodev.c
> > | 92 +++++++++++++++++++++++++++++++++++
> >  lib/cryptodev/rte_cryptodev.h | 37 ++++++++++++++
> >  lib/cryptodev/version.map     |  2 +
> >  4 files changed, 133 insertions(+), 1 deletion(-)
> >
> > diff --git a/lib/cryptodev/cryptodev_pmd.h
> > b/lib/cryptodev/cryptodev_pmd.h index 09ba952455..f404604963 100644
> > --- a/lib/cryptodev/cryptodev_pmd.h
> > +++ b/lib/cryptodev/cryptodev_pmd.h
> > @@ -78,7 +78,8 @@ struct rte_cryptodev_data {
> >  	void **queue_pairs;
> >  	/** Number of device queue pairs. */
> >  	uint16_t nb_queue_pairs;
> > -
> > +	/** Array of process id used for queue pairs **/
> > +	uint16_t *qp_in_use_by_pid;
> Why array? And if an array, how will its depth will be calculated.
> 
> I commented on v1, that the in_use pid can be stored inside queue private
> data.
[KJ] the depth can be calculated using the max_nb_queue_pairs in dev_info. By stored this information in rte_cryptodev_data struct, we are no longer need to modify every queue private crypto PMDs,  and this pid info is not exposed to userspace neither.
> 
> >  	/** PMD-specific private data */
> >  	void *dev_private;
> >  } __rte_cache_aligned;
> > diff --git a/lib/cryptodev/rte_cryptodev.c
> > b/lib/cryptodev/rte_cryptodev.c index 9e76a1c72d..dab6a37bff 100644
> > --- a/lib/cryptodev/rte_cryptodev.c
> > +++ b/lib/cryptodev/rte_cryptodev.c
> > @@ -49,6 +49,9 @@ struct rte_crypto_fp_ops
> > rte_crypto_fp_ops[RTE_CRYPTO_MAX_DEVS];
> >  /* spinlock for crypto device callbacks */  static rte_spinlock_t
> > rte_cryptodev_cb_lock = RTE_SPINLOCK_INITIALIZER;
> >
> > +/* crypto queue pair config */
> > +#define CRYPTODEV_MP_REQ "cryptodev_mp_request"
> > +
> >  /**
> >   * The user application callback description.
> >   *
> > @@ -1050,6 +1053,9 @@ rte_cryptodev_pmd_release_device(struct
> > rte_cryptodev *cryptodev)
> >  			return ret;
> >  	}
> >
> > +	if (cryptodev->data->qp_in_use_by_pid)
> > +		rte_free(cryptodev->data->qp_in_use_by_pid);
[KJ] this is qp_in_use_by_pid get freed
> > +
> >  	ret = rte_cryptodev_data_free(dev_id,
> > &cryptodev_globals.data[dev_id]);
> >  	if (ret < 0)
> >  		return ret;
> > @@ -1138,6 +1144,21 @@ rte_cryptodev_queue_pairs_config(struct
> > rte_cryptodev *dev, uint16_t nb_qpairs,
> >
> >  	}
> >  	dev->data->nb_queue_pairs = nb_qpairs;
> > +
> > +	if (dev->data->qp_in_use_by_pid == NULL) {
> > +		dev->data->qp_in_use_by_pid = rte_zmalloc_socket(
> > +				"cryptodev->qp_in_use_by_pid",
> > +				sizeof(dev->data->qp_in_use_by_pid[0]) *
> > +				dev_info.max_nb_queue_pairs,
> > +				RTE_CACHE_LINE_SIZE, socket_id);
[KJ] this is qp_in_use_by_pid get malloced
> > +		if (dev->data->qp_in_use_by_pid == NULL) {
> > +			CDEV_LOG_ERR("failed to get memory for qp meta
> > data, "
> > +							"nb_queues %u",
> > +							nb_qpairs);
> > +			return -(ENOMEM);
> > +		}
> > +	}
> > +
> >  	return 0;
> >  }
> >
> > @@ -1400,6 +1421,77 @@ rte_cryptodev_queue_pair_setup(uint8_t
> dev_id,
> > uint16_t queue_pair_id,
> >  			socket_id);
> >  }
> >
> > +static int
> > +rte_cryptodev_ipc_request(const struct rte_mp_msg *mp_msg, const
> void
> > *peer)
> > +{
> > +	struct rte_mp_msg mp_res;
> > +	struct rte_cryptodev_mp_param *res =
> > +		(struct rte_cryptodev_mp_param *)mp_res.param;
> > +	const struct rte_cryptodev_mp_param *param =
> > +		(const struct rte_cryptodev_mp_param *)mp_msg->param;
> > +
> > +	int ret;
> > +	struct rte_cryptodev *dev;
> > +	uint16_t *qps_in_used_by_pid;
> > +	int dev_id = param->dev_id;
> > +	int qp_id = param->qp_id;
> > +	struct rte_cryptodev_qp_conf *queue_conf = param->queue_conf;
> > +
> > +	res->result = -EINVAL;
> > +	if (!rte_cryptodev_is_valid_dev(dev_id)) {
> > +		CDEV_LOG_ERR("Invalid dev_id=%" PRIu8, dev_id);
> > +		goto out;
> > +	}
> > +
> > +	if (!rte_cryptodev_get_qp_status(dev_id, qp_id))
> > +		goto out;
> > +
> > +	dev = &rte_crypto_devices[dev_id];
> > +	qps_in_used_by_pid = dev->data->qp_in_use_by_pid;
> > +
> > +	switch (param->type) {
> > +	case RTE_CRYPTODEV_MP_REQ_QP_SET:
> > +		ret = rte_cryptodev_queue_pair_setup(dev_id, qp_id,
> > +				queue_conf, param->socket_id);
> > +		if (!ret)
> > +			qps_in_used_by_pid[qp_id] = param->process_id;
> > +		res->result = ret;
> > +		break;
> > +	case RTE_CRYPTODEV_MP_REQ_QP_FREE:
> > +		if ((rte_eal_process_type() == RTE_PROC_SECONDARY) &&
> > +			(qps_in_used_by_pid[qp_id] != param->process_id)) {
> > +			CDEV_LOG_ERR("Unable to release qp_id=%" PRIu8,
> > qp_id);
> > +			goto out;
> > +		}
> > +
> > +		ret = (*dev->dev_ops->queue_pair_release)(dev, qp_id);
> > +		if (!ret)
> > +			qps_in_used_by_pid[qp_id] = 0;
> > +
> > +		res->result = ret;
> > +		break;
> > +	default:
> > +		CDEV_LOG_ERR("invalid mp request type\n");
> > +	}
> > +
> > +out:
> > +	ret = rte_mp_reply(&mp_res, peer);
> > +	return ret;
> > +}
> > +
> > +int rte_cryptodev_mp_request_register(void)
> > +{
> > +	RTE_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
> > +	return rte_mp_action_register(CRYPTODEV_MP_REQ,
> > +				rte_cryptodev_ipc_request);
> > +}
> > +
> > +void rte_cryptodev_mp_request_unregister(void)
> > +{
> > +	RTE_ASSERT(rte_eal_process_type() == RTE_PROC_PRIMARY);
> > +	rte_mp_action_unregister(CRYPTODEV_MP_REQ);
> > +}
> > +
> 
> It looks an incomplete patch right now.
> - Documentation is missing on how this feature will be used.
[KJ] v4 updates release notes and commit message for usage of this feature.
> - Test cases?
[KJ]We have a standalone multi-process test case that I can submit but it needs some rework so it probably cant be included in the next revision. I can submit it anyway and we can do a revision of it later, after discussion. Ideally, the test app should be integrate into l2fwd test app. 
> - PMD implementation is key part for this feature to work.
[KJ] My understanding is that this design should not impact to any PMDs but offers an alternative way for secondary process to configure the queue-pair.  Its up to primary process to call the IPC request register function, otherwise no impact to any cryptodev multi-process functionalities. 
> 
> I believe it is better to defer this to next release as we don't have a clarity on
> how to use this.
[KJ] We would still like to get this feature into the release since we have a customer already using a version of this patchset. We will work to address your concerns/questions and resubmit.
>