[v2,2/2] ethdev: introduce the PHY affinity field in Tx queue API
Checks
Commit Message
For the multiple hardware ports connect to a single DPDK port (mhpsdp),
the previous patch introduces the new rte flow item to match the
phy affinity of the received packets.
This patch adds the tx_phy_affinity setting in Tx queue API, the affinity
value reflects packets be sent to which hardware port.
Value 0 is no affinity and traffic will be routed between different
physical ports, if 0 is disabled then try to match on phy_affinity 0
will result in an error.
Adds the new tx_phy_affinity field into the padding hole of rte_eth_txconf
structure, the size of rte_eth_txconf keeps the same. Adds a suppress
type for structure change in the ABI check file.
This patch adds the testpmd command line:
testpmd> port config (port_id) txq (queue_id) phy_affinity (value)
For example, there're two hardware ports 0 and 1 connected to
a single DPDK port (port id 0), and phy_affinity 1 stood for
hardware port 0 and phy_affinity 2 stood for hardware port 1,
used the below command to config tx phy affinity for per Tx Queue:
port config 0 txq 0 phy_affinity 1
port config 0 txq 1 phy_affinity 1
port config 0 txq 2 phy_affinity 2
port config 0 txq 3 phy_affinity 2
These commands config the TxQ index 0 and TxQ index 1 with phy affinity 1,
uses TxQ 0 or TxQ 1 send packets, these packets will be sent from the
hardware port 0, and similar with hardware port 1 if sending packets
with TxQ 2 or TxQ 3.
Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com>
---
app/test-pmd/cmdline.c | 84 +++++++++++++++++++++
devtools/libabigail.abignore | 5 ++
doc/guides/testpmd_app_ug/testpmd_funcs.rst | 13 ++++
lib/ethdev/rte_ethdev.h | 7 ++
4 files changed, 109 insertions(+)
Comments
30/01/2023 18:00, Jiawei Wang:
> --- a/devtools/libabigail.abignore
> +++ b/devtools/libabigail.abignore
> @@ -20,6 +20,11 @@
> [suppress_file]
> soname_regexp = ^librte_.*mlx.*glue\.
>
> +; Ignore fields inserted in middle padding of rte_eth_txconf
> +[suppress_type]
> + name = rte_eth_txconf
> + has_data_member_inserted_between = {offset_after(tx_deferred_start), offset_of(offloads)}
You are adding the exception inside
"Core suppression rules: DO NOT TOUCH".
Please move it at the end in the section
"Temporary exceptions till next major ABI version"
Also the rule does not work.
It should be:
has_data_member_inserted_between = {offset_of(tx_deferred_start), offset_of(offloads)}
On 1/30/23 20:00, Jiawei Wang wrote:
> For the multiple hardware ports connect to a single DPDK port (mhpsdp),
> the previous patch introduces the new rte flow item to match the
> phy affinity of the received packets.
>
> This patch adds the tx_phy_affinity setting in Tx queue API, the affinity
"This patch adds" -> "Add ..."
> value reflects packets be sent to which hardware port.
> Value 0 is no affinity and traffic will be routed between different
> physical ports,
Who will it be routed?
> if 0 is disabled then try to match on phy_affinity 0
> will result in an error.
Why are you talking about matching here?
>
> Adds the new tx_phy_affinity field into the padding hole of rte_eth_txconf
> structure, the size of rte_eth_txconf keeps the same. Adds a suppress
> type for structure change in the ABI check file.
>
> This patch adds the testpmd command line:
> testpmd> port config (port_id) txq (queue_id) phy_affinity (value)
>
> For example, there're two hardware ports 0 and 1 connected to
> a single DPDK port (port id 0), and phy_affinity 1 stood for
> hardware port 0 and phy_affinity 2 stood for hardware port 1,
> used the below command to config tx phy affinity for per Tx Queue:
> port config 0 txq 0 phy_affinity 1
> port config 0 txq 1 phy_affinity 1
> port config 0 txq 2 phy_affinity 2
> port config 0 txq 3 phy_affinity 2
>
> These commands config the TxQ index 0 and TxQ index 1 with phy affinity 1,
> uses TxQ 0 or TxQ 1 send packets, these packets will be sent from the
> hardware port 0, and similar with hardware port 1 if sending packets
> with TxQ 2 or TxQ 3.
Frankly speaking I dislike it. Why do we need to expose it on
generic ethdev layer? IMHO dynamic mbuf field would be a better
solution to control Tx routing to a specific PHY port.
IMHO, we definitely need dev_info information about a number of
physical ports behind. Advertising value greater than 0 should
mean that PMD supports corresponding mbuf dynamic field to
contol ongoing physical port on Tx (or should just reject
packets on prepare which try to specify outgoing phy port
otherwise). In the same way the information may be provided
on Rx.
I'm OK to have 0 as no phy affinity value and greater than
zero as specified phy affinity. I.e. no dynamic flag is
required.
Also I think that order of patches should be different.
We should start from a patch which provides dev_info and
flow API matching and action should be in later patch.
>
> Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com>
[snip]
> 30/01/2023 18:00, Jiawei Wang:
> > --- a/devtools/libabigail.abignore
> > +++ b/devtools/libabigail.abignore
> > @@ -20,6 +20,11 @@
> > [suppress_file]
> > soname_regexp = ^librte_.*mlx.*glue\.
> >
> > +; Ignore fields inserted in middle padding of rte_eth_txconf
> > +[suppress_type]
> > + name = rte_eth_txconf
> > + has_data_member_inserted_between =
> > +{offset_after(tx_deferred_start), offset_of(offloads)}
>
> You are adding the exception inside
> "Core suppression rules: DO NOT TOUCH".
>
> Please move it at the end in the section "Temporary exceptions till next major
> ABI version"
>
OK, will move.
> Also the rule does not work.
> It should be:
> has_data_member_inserted_between = {offset_of(tx_deferred_start),
> offset_of(offloads)}
>
Thanks, Will change it and send with new version.
>
Hi,
> -----Original Message-----
> From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> Subject: Re: [PATCH v2 2/2] ethdev: introduce the PHY affinity field in Tx queue
> API
>
> On 1/30/23 20:00, Jiawei Wang wrote:
> > For the multiple hardware ports connect to a single DPDK port
> > (mhpsdp), the previous patch introduces the new rte flow item to match
> > the phy affinity of the received packets.
> >
> > This patch adds the tx_phy_affinity setting in Tx queue API, the
> > affinity
>
> "This patch adds" -> "Add ..."
>
OK, will change to 'Add the tx_phy_affinity...."
> > value reflects packets be sent to which hardware port.
> > Value 0 is no affinity and traffic will be routed between different
> > physical ports,
>
> Who will it be routed?
>
Assume there's two slave physical port bonded and DPDK attached the bond master bond,
The packets can be sent from first physical port or second physical port, it depends on the PMD
Driver and low level 'routing' selection.
> > if 0 is disabled then try to match on phy_affinity 0 will result in an
> > error.
>
> Why are you talking about matching here?
>
Previous patch we mentioned the same phy affinity can be used to handled the packet on same hardware
Port, so if 0 is no affinity then match it should report error.
> >
> > Adds the new tx_phy_affinity field into the padding hole of
> > rte_eth_txconf structure, the size of rte_eth_txconf keeps the same.
> > Adds a suppress type for structure change in the ABI check file.
> >
> > This patch adds the testpmd command line:
> > testpmd> port config (port_id) txq (queue_id) phy_affinity (value)
> >
> > For example, there're two hardware ports 0 and 1 connected to a single
> > DPDK port (port id 0), and phy_affinity 1 stood for hardware port 0
> > and phy_affinity 2 stood for hardware port 1, used the below command
> > to config tx phy affinity for per Tx Queue:
> > port config 0 txq 0 phy_affinity 1
> > port config 0 txq 1 phy_affinity 1
> > port config 0 txq 2 phy_affinity 2
> > port config 0 txq 3 phy_affinity 2
> >
> > These commands config the TxQ index 0 and TxQ index 1 with phy
> > affinity 1, uses TxQ 0 or TxQ 1 send packets, these packets will be
> > sent from the hardware port 0, and similar with hardware port 1 if
> > sending packets with TxQ 2 or TxQ 3.
>
> Frankly speaking I dislike it. Why do we need to expose it on generic ethdev
> layer? IMHO dynamic mbuf field would be a better solution to control Tx
> routing to a specific PHY port.
>
OK, the phy affinity is not part of packet information(like timestamp).
And second, the phy affinity is Queue layer, that is, the phy affinity value
should keep the same behavior per Queue.
After the TxQ was created, the packets should be sent the same physical port
If using the same TxQ index.
> IMHO, we definitely need dev_info information about a number of physical
> ports behind. Advertising value greater than 0 should mean that PMD supports
> corresponding mbuf dynamic field to contol ongoing physical port on Tx (or
> should just reject packets on prepare which try to specify outgoing phy port
> otherwise). In the same way the information may be provided on Rx.
>
See above, I think phy affinity is Queue level not for each packet.
> I'm OK to have 0 as no phy affinity value and greater than zero as specified phy
> affinity. I.e. no dynamic flag is required.
>
Thanks for agreement.
> Also I think that order of patches should be different.
> We should start from a patch which provides dev_info and flow API matching
> and action should be in later patch.
>
OK.
> >
> > Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com>
>
> [snip]
On 2/1/23 18:50, Jiawei(Jonny) Wang wrote:
>> -----Original Message-----
>> From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
>> Subject: Re: [PATCH v2 2/2] ethdev: introduce the PHY affinity field in Tx queue
>> API
>>
>> On 1/30/23 20:00, Jiawei Wang wrote:
>>> Adds the new tx_phy_affinity field into the padding hole of
>>> rte_eth_txconf structure, the size of rte_eth_txconf keeps the same.
>>> Adds a suppress type for structure change in the ABI check file.
>>>
>>> This patch adds the testpmd command line:
>>> testpmd> port config (port_id) txq (queue_id) phy_affinity (value)
>>>
>>> For example, there're two hardware ports 0 and 1 connected to a single
>>> DPDK port (port id 0), and phy_affinity 1 stood for hardware port 0
>>> and phy_affinity 2 stood for hardware port 1, used the below command
>>> to config tx phy affinity for per Tx Queue:
>>> port config 0 txq 0 phy_affinity 1
>>> port config 0 txq 1 phy_affinity 1
>>> port config 0 txq 2 phy_affinity 2
>>> port config 0 txq 3 phy_affinity 2
>>>
>>> These commands config the TxQ index 0 and TxQ index 1 with phy
>>> affinity 1, uses TxQ 0 or TxQ 1 send packets, these packets will be
>>> sent from the hardware port 0, and similar with hardware port 1 if
>>> sending packets with TxQ 2 or TxQ 3.
>>
>> Frankly speaking I dislike it. Why do we need to expose it on generic ethdev
>> layer? IMHO dynamic mbuf field would be a better solution to control Tx
>> routing to a specific PHY port.
>>
>
> OK, the phy affinity is not part of packet information(like timestamp).
Why? port_id is a packet information. Why phy_subport_id is not
a packet information.
> And second, the phy affinity is Queue layer, that is, the phy affinity value
> should keep the same behavior per Queue.
> After the TxQ was created, the packets should be sent the same physical port
> If using the same TxQ index.
Why are these queues should be visible to DPDK application?
Nobody denies you to create many HW queues behind one ethdev
queue. Of course, there questions related to descriptor status
API in this case, but IMHO it would be better than exposing
these details to an application level.
>
>> IMHO, we definitely need dev_info information about a number of physical
>> ports behind. Advertising value greater than 0 should mean that PMD supports
>> corresponding mbuf dynamic field to contol ongoing physical port on Tx (or
>> should just reject packets on prepare which try to specify outgoing phy port
>> otherwise). In the same way the information may be provided on Rx.
>>
>
> See above, I think phy affinity is Queue level not for each packet.
>
>> I'm OK to have 0 as no phy affinity value and greater than zero as specified phy
>> affinity. I.e. no dynamic flag is required.
>>
>
> Thanks for agreement.
>
>> Also I think that order of patches should be different.
>> We should start from a patch which provides dev_info and flow API matching
>> and action should be in later patch.
>>
>
> OK.
>
>>>
>>> Signed-off-by: Jiawei Wang <jiaweiw@nvidia.com>
>>
>> [snip]
>
02/02/2023 10:28, Andrew Rybchenko:
> On 2/1/23 18:50, Jiawei(Jonny) Wang wrote:
> > From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
> >> On 1/30/23 20:00, Jiawei Wang wrote:
> >>> Adds the new tx_phy_affinity field into the padding hole of
> >>> rte_eth_txconf structure, the size of rte_eth_txconf keeps the same.
> >>> Adds a suppress type for structure change in the ABI check file.
> >>>
> >>> This patch adds the testpmd command line:
> >>> testpmd> port config (port_id) txq (queue_id) phy_affinity (value)
> >>>
> >>> For example, there're two hardware ports 0 and 1 connected to a single
> >>> DPDK port (port id 0), and phy_affinity 1 stood for hardware port 0
> >>> and phy_affinity 2 stood for hardware port 1, used the below command
> >>> to config tx phy affinity for per Tx Queue:
> >>> port config 0 txq 0 phy_affinity 1
> >>> port config 0 txq 1 phy_affinity 1
> >>> port config 0 txq 2 phy_affinity 2
> >>> port config 0 txq 3 phy_affinity 2
> >>>
> >>> These commands config the TxQ index 0 and TxQ index 1 with phy
> >>> affinity 1, uses TxQ 0 or TxQ 1 send packets, these packets will be
> >>> sent from the hardware port 0, and similar with hardware port 1 if
> >>> sending packets with TxQ 2 or TxQ 3.
> >>
> >> Frankly speaking I dislike it. Why do we need to expose it on generic ethdev
> >> layer? IMHO dynamic mbuf field would be a better solution to control Tx
> >> routing to a specific PHY port.
The design of this patch is to map a queue of the front device
with an underlying port.
This design may be applicable to several situations,
including DPDK bonding PMD, or Linux bonding connected to a PMD.
The default 0, meaning the queue is not mapped to anything (no change).
If the affinity is higher than 0, then the queue can be configured as desired.
Then if an application wants to send a packet to a specific underlying port,
it just has to send to the right queue.
Functionnaly, mapping the queue, or setting the port in mbuf (your proposal)
are the same.
The advantages of the queue mapping are:
- faster to use a queue than filling mbuf field
- optimization can be done at queue setup
[...]
> Why are these queues should be visible to DPDK application?
> Nobody denies you to create many HW queues behind one ethdev
> queue. Of course, there questions related to descriptor status
> API in this case, but IMHO it would be better than exposing
> these details to an application level.
Why not mapping the queues if application requires these details?
> >> IMHO, we definitely need dev_info information about a number of physical
> >> ports behind.
Yes dev_info would be needed.
> >> Advertising value greater than 0 should mean that PMD supports
> >> corresponding mbuf dynamic field to contol ongoing physical port on Tx (or
> >> should just reject packets on prepare which try to specify outgoing phy port
> >> otherwise). In the same way the information may be provided on Rx.
> >
> > See above, I think phy affinity is Queue level not for each packet.
> >
> >> I'm OK to have 0 as no phy affinity value and greater than zero as specified phy
> >> affinity. I.e. no dynamic flag is required.
> >
> > Thanks for agreement.
> >
> >> Also I think that order of patches should be different.
> >> We should start from a patch which provides dev_info and flow API matching
> >> and action should be in later patch.
> >
> > OK.
@@ -764,6 +764,10 @@ static void cmd_help_long_parsed(void *parsed_result,
"port cleanup (port_id) txq (queue_id) (free_cnt)\n"
" Cleanup txq mbufs for a specific Tx queue\n\n"
+
+ "port config (port_id) txq (queue_id) phy_affinity (value)\n"
+ " Set the physical affinity value "
+ "on a specific Tx queue\n\n"
);
}
@@ -12621,6 +12625,85 @@ static cmdline_parse_inst_t cmd_show_port_flow_transfer_proxy = {
}
};
+/* *** configure port txq phy_affinity value *** */
+struct cmd_config_tx_phy_affinity {
+ cmdline_fixed_string_t port;
+ cmdline_fixed_string_t config;
+ portid_t portid;
+ cmdline_fixed_string_t txq;
+ uint16_t qid;
+ cmdline_fixed_string_t phy_affinity;
+ uint16_t value;
+};
+
+static void
+cmd_config_tx_phy_affinity_parsed(void *parsed_result,
+ __rte_unused struct cmdline *cl,
+ __rte_unused void *data)
+{
+ struct cmd_config_tx_phy_affinity *res = parsed_result;
+ struct rte_port *port;
+
+ if (port_id_is_invalid(res->portid, ENABLED_WARN))
+ return;
+
+ if (res->portid == (portid_t)RTE_PORT_ALL) {
+ printf("Invalid port id\n");
+ return;
+ }
+
+ port = &ports[res->portid];
+
+ if (strcmp(res->txq, "txq")) {
+ printf("Unknown parameter\n");
+ return;
+ }
+ if (tx_queue_id_is_invalid(res->qid))
+ return;
+
+ port->txq[res->qid].conf.tx_phy_affinity = res->value;
+
+ cmd_reconfig_device_queue(res->portid, 0, 1);
+}
+
+cmdline_parse_token_string_t cmd_config_tx_phy_affinity_port =
+ TOKEN_STRING_INITIALIZER(struct cmd_config_tx_phy_affinity,
+ port, "port");
+cmdline_parse_token_string_t cmd_config_tx_phy_affinity_config =
+ TOKEN_STRING_INITIALIZER(struct cmd_config_tx_phy_affinity,
+ config, "config");
+cmdline_parse_token_num_t cmd_config_tx_phy_affinity_portid =
+ TOKEN_NUM_INITIALIZER(struct cmd_config_tx_phy_affinity,
+ portid, RTE_UINT16);
+cmdline_parse_token_string_t cmd_config_tx_phy_affinity_txq =
+ TOKEN_STRING_INITIALIZER(struct cmd_config_tx_phy_affinity,
+ txq, "txq");
+cmdline_parse_token_num_t cmd_config_tx_phy_affinity_qid =
+ TOKEN_NUM_INITIALIZER(struct cmd_config_tx_phy_affinity,
+ qid, RTE_UINT16);
+cmdline_parse_token_string_t cmd_config_tx_phy_affinity_hwport =
+ TOKEN_STRING_INITIALIZER(struct cmd_config_tx_phy_affinity,
+ phy_affinity, "phy_affinity");
+cmdline_parse_token_num_t cmd_config_tx_phy_affinity_value =
+ TOKEN_NUM_INITIALIZER(struct cmd_config_tx_phy_affinity,
+ value, RTE_UINT16);
+
+static cmdline_parse_inst_t cmd_config_tx_phy_affinity = {
+ .f = cmd_config_tx_phy_affinity_parsed,
+ .data = (void *)0,
+ .help_str = "port config <port_id> txq <queue_id> phy_affinity <value>",
+ .tokens = {
+ (void *)&cmd_config_tx_phy_affinity_port,
+ (void *)&cmd_config_tx_phy_affinity_config,
+ (void *)&cmd_config_tx_phy_affinity_portid,
+ (void *)&cmd_config_tx_phy_affinity_txq,
+ (void *)&cmd_config_tx_phy_affinity_qid,
+ (void *)&cmd_config_tx_phy_affinity_hwport,
+ (void *)&cmd_config_tx_phy_affinity_value,
+ NULL,
+ },
+};
+
/* ******************************************************************************** */
/* list of instructions */
@@ -12851,6 +12934,7 @@ static cmdline_parse_ctx_t builtin_ctx[] = {
(cmdline_parse_inst_t *)&cmd_show_capability,
(cmdline_parse_inst_t *)&cmd_set_flex_is_pattern,
(cmdline_parse_inst_t *)&cmd_set_flex_spec_pattern,
+ (cmdline_parse_inst_t *)&cmd_config_tx_phy_affinity,
NULL,
};
@@ -20,6 +20,11 @@
[suppress_file]
soname_regexp = ^librte_.*mlx.*glue\.
+; Ignore fields inserted in middle padding of rte_eth_txconf
+[suppress_type]
+ name = rte_eth_txconf
+ has_data_member_inserted_between = {offset_after(tx_deferred_start), offset_of(offloads)}
+
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
; Experimental APIs exceptions ;
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@@ -1605,6 +1605,19 @@ Enable or disable a per queue Tx offloading only on a specific Tx queue::
This command should be run when the port is stopped, or else it will fail.
+config per queue Tx physical affinity
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Configure a per queue physical affinity value only on a specific Tx queue::
+
+ testpmd> port (port_id) txq (queue_id) phy_affinity (value)
+
+* ``phy_affinity``: reflects packet can be sent to which hardware port.
+ uses it on multiple hardware ports connect to
+ a single DPDK port (mhpsdp).
+
+This command should be run when the port is stopped, or else it will fail.
+
Config VXLAN Encap outer layers
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -1138,6 +1138,13 @@ struct rte_eth_txconf {
less free descriptors than this value. */
uint8_t tx_deferred_start; /**< Do not start queue with rte_eth_dev_start(). */
+ /**
+ * Physical affinity to be set.
+ * Value 0 is no affinity and traffic could be routed between different
+ * physical ports, if 0 is disabled then try to match on phy_affinity 0 will
+ * result in an error.
+ */
+ uint8_t tx_phy_affinity;
/**
* Per-queue Tx offloads to be set using RTE_ETH_TX_OFFLOAD_* flags.
* Only offloads set on tx_queue_offload_capa or tx_offload_capa