net/ixgbe: initialize max_rx_pkt_len if rlpml_set_vf fails

  It seems that if the call to ixgbevf_rlpml_set_vf fails,
we will not initialize dev_conf.rxmode.max_rx_pkt_len correctly anymore.

This happens with a 82599EB NIC and a VMware ESXI 6.0 setup,
and is causing VF the ports to fail to initialize

We see the following error:
ixgbevf_dev_rx_init(): Set max packet length to 1518 failed.

Investigating over DPDK 19.11, it seems that the call still fails,
but it doesn't exit prematurely, and max_rx_pkt_len is initialized in
the respective case.

On the ESXi server, we seem to have the following driver
net-ixgbe: 3.7.13.7.14iov-20vmw.600.0.0.2494585

It seems that the behavior related to MTU setting has changed
since the following commit:

commit c77866a16904 ("net/ixgbe: detect failed VF MTU set")

We would like to still be able to support older setups if possible,
as we might have customers running ESXi 6.0 or 6.5, and these seem
to have an older version of the PF driver as default.

Signed-off-by: Tudor Cornea <tudor.cornea@gmail.com>
---
 drivers/net/ixgbe/ixgbe_ethdev.c | 5 +++--
 drivers/net/ixgbe/ixgbe_rxtx.c   | 1 -
 2 files changed, 3 insertions(+), 3 deletions(-)
  

Some of our customers use ESXi 6.0 or 6.5 servers, which could have older
versions of the PF ixgbe driver.
It seems that with a more recent version of the PMD driver, we are not able
to initialize 82599EB ports correctly.
This scenario seems to have worked with DPDK 19.11.

Would it be possible to print a warning, while still allowing the driver to
initialize the ports ?

I was also thinking about the return code of ixgbevf_dev_set_mtu.
Do you think it would be more appropriate to return ENOTSUP or ENOSYS
instead of EINVAL ?

As a user, calling 'rte_eth_dev_mtu_set', I would expect an error like
EINVAL to suggest to me that
the mtu value which I provided is incorrect [1]. The 82599 NIC, however has
some particularities related to mtu,
which could cause the operation to fail. I was thinking that EINVAL might
not be most descriptive error.

[1] https://doc.dpdk.org/api/rte__ethdev_8h.html

Thanks,
Tudor

On Fri, 15 Oct 2021 at 17:06, Tudor Cornea <tudor.cornea@gmail.com> wrote:

> It seems that if the call to ixgbevf_rlpml_set_vf fails,
> we will not initialize dev_conf.rxmode.max_rx_pkt_len correctly anymore.
>
> This happens with a 82599EB NIC and a VMware ESXI 6.0 setup,
> and is causing VF the ports to fail to initialize
>
> We see the following error:
> ixgbevf_dev_rx_init(): Set max packet length to 1518 failed.
>
> Investigating over DPDK 19.11, it seems that the call still fails,
> but it doesn't exit prematurely, and max_rx_pkt_len is initialized in
> the respective case.
>
> On the ESXi server, we seem to have the following driver
> net-ixgbe: 3.7.13.7.14iov-20vmw.600.0.0.2494585
>
> It seems that the behavior related to MTU setting has changed
> since the following commit:
>
> commit c77866a16904 ("net/ixgbe: detect failed VF MTU set")
>
> We would like to still be able to support older setups if possible,
> as we might have customers running ESXi 6.0 or 6.5, and these seem
> to have an older version of the PF driver as default.
>
> Signed-off-by: Tudor Cornea <tudor.cornea@gmail.com>
> ---
>  drivers/net/ixgbe/ixgbe_ethdev.c | 5 +++--
>  drivers/net/ixgbe/ixgbe_rxtx.c   | 1 -
>  2 files changed, 3 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
> b/drivers/net/ixgbe/ixgbe_ethdev.c
> index 4dbe049..4301dfd 100644
> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> @@ -6369,6 +6369,9 @@ ixgbevf_dev_set_mtu(struct rte_eth_dev *dev,
> uint16_t mtu)
>                 return -EINVAL;
>         }
>
> +       /* update max frame size */
> +       dev->data->dev_conf.rxmode.max_rx_pkt_len = max_frame;
> +
>         /*
>          * When supported by the underlying PF driver, use the
> IXGBE_VF_SET_MTU
>          * request of the version 2.0 of the mailbox API.
> @@ -6381,8 +6384,6 @@ ixgbevf_dev_set_mtu(struct rte_eth_dev *dev,
> uint16_t mtu)
>         if (ixgbevf_rlpml_set_vf(hw, max_frame))
>                 return -EINVAL;
>
> -       /* update max frame size */
> -       dev->data->dev_conf.rxmode.max_rx_pkt_len = max_frame;
>         return 0;
>  }
>
> diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c
> b/drivers/net/ixgbe/ixgbe_rxtx.c
> index 0ac89cb..02d9809 100644
> --- a/drivers/net/ixgbe/ixgbe_rxtx.c
> +++ b/drivers/net/ixgbe/ixgbe_rxtx.c
> @@ -5677,7 +5677,6 @@ ixgbevf_dev_rx_init(struct rte_eth_dev *dev)
>             (uint16_t)dev->data->dev_conf.rxmode.max_rx_pkt_len)) {
>                 PMD_INIT_LOG(ERR, "Set max packet length to %d failed.",
>                              dev->data->dev_conf.rxmode.max_rx_pkt_len);
> -               return -EINVAL;
>         }
>
>         /*
> --
> 2.7.4
>
>
  

On 10/15/2021 3:20 PM, Tudor Cornea wrote:
> Some of our customers use ESXi 6.0 or 6.5 servers, which could have older
> versions of the PF ixgbe driver.
> It seems that with a more recent version of the PMD driver, we are not able
> to initialize 82599EB ports correctly.
> This scenario seems to have worked with DPDK 19.11.
> 
> Would it be possible to print a warning, while still allowing the driver to
> initialize the ports ?
> 
> I was also thinking about the return code of ixgbevf_dev_set_mtu.
> Do you think it would be more appropriate to return ENOTSUP or ENOSYS
> instead of EINVAL ?
> 
> As a user, calling 'rte_eth_dev_mtu_set', I would expect an error like
> EINVAL to suggest to me that
> the mtu value which I provided is incorrect [1]. The 82599 NIC, however has
> some particularities related to mtu,
> which could cause the operation to fail. I was thinking that EINVAL might
> not be most descriptive error.
> 
> [1] https://doc.dpdk.org/api/rte__ethdev_8h.html
> 
> Thanks,
> Tudor

Hi Tudor,

Can you please check if the patch is still after 'max_rx_pkt_len' related changes
in next-net?

> 
> On Fri, 15 Oct 2021 at 17:06, Tudor Cornea <tudor.cornea@gmail.com> wrote:
> 
>> It seems that if the call to ixgbevf_rlpml_set_vf fails,
>> we will not initialize dev_conf.rxmode.max_rx_pkt_len correctly anymore.
>>
>> This happens with a 82599EB NIC and a VMware ESXI 6.0 setup,
>> and is causing VF the ports to fail to initialize
>>
>> We see the following error:
>> ixgbevf_dev_rx_init(): Set max packet length to 1518 failed.
>>
>> Investigating over DPDK 19.11, it seems that the call still fails,
>> but it doesn't exit prematurely, and max_rx_pkt_len is initialized in
>> the respective case.
>>
>> On the ESXi server, we seem to have the following driver
>> net-ixgbe: 3.7.13.7.14iov-20vmw.600.0.0.2494585
>>
>> It seems that the behavior related to MTU setting has changed
>> since the following commit:
>>
>> commit c77866a16904 ("net/ixgbe: detect failed VF MTU set")
>>
>> We would like to still be able to support older setups if possible,
>> as we might have customers running ESXi 6.0 or 6.5, and these seem
>> to have an older version of the PF driver as default.
>>
>> Signed-off-by: Tudor Cornea <tudor.cornea@gmail.com>
>> ---
>>   drivers/net/ixgbe/ixgbe_ethdev.c | 5 +++--
>>   drivers/net/ixgbe/ixgbe_rxtx.c   | 1 -
>>   2 files changed, 3 insertions(+), 3 deletions(-)
>>
>> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
>> b/drivers/net/ixgbe/ixgbe_ethdev.c
>> index 4dbe049..4301dfd 100644
>> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
>> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
>> @@ -6369,6 +6369,9 @@ ixgbevf_dev_set_mtu(struct rte_eth_dev *dev,
>> uint16_t mtu)
>>                  return -EINVAL;
>>          }
>>
>> +       /* update max frame size */
>> +       dev->data->dev_conf.rxmode.max_rx_pkt_len = max_frame;
>> +
>>          /*
>>           * When supported by the underlying PF driver, use the
>> IXGBE_VF_SET_MTU
>>           * request of the version 2.0 of the mailbox API.
>> @@ -6381,8 +6384,6 @@ ixgbevf_dev_set_mtu(struct rte_eth_dev *dev,
>> uint16_t mtu)
>>          if (ixgbevf_rlpml_set_vf(hw, max_frame))
>>                  return -EINVAL;
>>
>> -       /* update max frame size */
>> -       dev->data->dev_conf.rxmode.max_rx_pkt_len = max_frame;
>>          return 0;
>>   }
>>
>> diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c
>> b/drivers/net/ixgbe/ixgbe_rxtx.c
>> index 0ac89cb..02d9809 100644
>> --- a/drivers/net/ixgbe/ixgbe_rxtx.c
>> +++ b/drivers/net/ixgbe/ixgbe_rxtx.c
>> @@ -5677,7 +5677,6 @@ ixgbevf_dev_rx_init(struct rte_eth_dev *dev)
>>              (uint16_t)dev->data->dev_conf.rxmode.max_rx_pkt_len)) {
>>                  PMD_INIT_LOG(ERR, "Set max packet length to %d failed.",
>>                               dev->data->dev_conf.rxmode.max_rx_pkt_len);
>> -               return -EINVAL;
>>          }
>>
>>          /*
>> --
>> 2.7.4
>>
>>
  

> -----Original Message-----
> From: Yigit, Ferruh <ferruh.yigit@intel.com>
> Sent: Tuesday, October 19, 2021 8:58 PM
> To: Tudor Cornea <tudor.cornea@gmail.com>; Zhang, Qi Z
> <qi.z.zhang@intel.com>
> Cc: Zhang, AlvinX <alvinx.zhang@intel.com>; Wang, Haiyue
> <haiyue.wang@intel.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH] net/ixgbe: initialize max_rx_pkt_len if
> rlpml_set_vf fails
> 
> On 10/15/2021 3:20 PM, Tudor Cornea wrote:
> > Some of our customers use ESXi 6.0 or 6.5 servers, which could have
> > older versions of the PF ixgbe driver.
> > It seems that with a more recent version of the PMD driver, we are not
> > able to initialize 82599EB ports correctly.
> > This scenario seems to have worked with DPDK 19.11.
> >
> > Would it be possible to print a warning, while still allowing the
> > driver to initialize the ports ?

There is a scenario that we initialize the port successful, with no any error, but we cannot get any packets.
So keep printing error or warning and still allowing the driver to initialize the port may be the best solution.

> >
> > I was also thinking about the return code of ixgbevf_dev_set_mtu.
> > Do you think it would be more appropriate to return ENOTSUP or ENOSYS
> > instead of EINVAL ?

If ixgbevf_dev_set_mtu fails, we have no way to know it because of invalid value or not supported?
ixgbevf_dev_set_mtu returns one the these values: 0 or IXGBE_ERR_MBX

> >
> > As a user, calling 'rte_eth_dev_mtu_set', I would expect an error like
> > EINVAL to suggest to me that the mtu value which I provided is
> > incorrect [1]. The 82599 NIC, however has some particularities related
> > to mtu, which could cause the operation to fail. I was thinking that
> > EINVAL might not be most descriptive error.
> >
> > [1] https://doc.dpdk.org/api/rte__ethdev_8h.html
> >
> > Thanks,
> > Tudor
> 
> Hi Tudor,
> 
> Can you please check if the patch is still after 'max_rx_pkt_len' related changes
> in next-net?
> 
> >
> > On Fri, 15 Oct 2021 at 17:06, Tudor Cornea <tudor.cornea@gmail.com>
> wrote:
> >
> >> It seems that if the call to ixgbevf_rlpml_set_vf fails, we will not
> >> initialize dev_conf.rxmode.max_rx_pkt_len correctly anymore.
> >>
> >> This happens with a 82599EB NIC and a VMware ESXI 6.0 setup, and is
> >> causing VF the ports to fail to initialize
> >>
> >> We see the following error:
> >> ixgbevf_dev_rx_init(): Set max packet length to 1518 failed.
> >>
> >> Investigating over DPDK 19.11, it seems that the call still fails,
> >> but it doesn't exit prematurely, and max_rx_pkt_len is initialized in
> >> the respective case.
> >>
> >> On the ESXi server, we seem to have the following driver
> >> net-ixgbe: 3.7.13.7.14iov-20vmw.600.0.0.2494585
> >>
> >> It seems that the behavior related to MTU setting has changed since
> >> the following commit:
> >>
> >> commit c77866a16904 ("net/ixgbe: detect failed VF MTU set")
> >>
> >> We would like to still be able to support older setups if possible,
> >> as we might have customers running ESXi 6.0 or 6.5, and these seem to
> >> have an older version of the PF driver as default.
> >>
> >> Signed-off-by: Tudor Cornea <tudor.cornea@gmail.com>
> >> ---
> >>   drivers/net/ixgbe/ixgbe_ethdev.c | 5 +++--
> >>   drivers/net/ixgbe/ixgbe_rxtx.c   | 1 -
> >>   2 files changed, 3 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
> >> b/drivers/net/ixgbe/ixgbe_ethdev.c
> >> index 4dbe049..4301dfd 100644
> >> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> >> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> >> @@ -6369,6 +6369,9 @@ ixgbevf_dev_set_mtu(struct rte_eth_dev *dev,
> >> uint16_t mtu)
> >>                  return -EINVAL;
> >>          }
> >>
> >> +       /* update max frame size */
> >> +       dev->data->dev_conf.rxmode.max_rx_pkt_len = max_frame;
> >> +
> >>          /*
> >>           * When supported by the underlying PF driver, use the
> >> IXGBE_VF_SET_MTU
> >>           * request of the version 2.0 of the mailbox API.
> >> @@ -6381,8 +6384,6 @@ ixgbevf_dev_set_mtu(struct rte_eth_dev *dev,
> >> uint16_t mtu)
> >>          if (ixgbevf_rlpml_set_vf(hw, max_frame))
> >>                  return -EINVAL;
> >>
> >> -       /* update max frame size */
> >> -       dev->data->dev_conf.rxmode.max_rx_pkt_len = max_frame;
> >>          return 0;
> >>   }
> >>
> >> diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c
> >> b/drivers/net/ixgbe/ixgbe_rxtx.c index 0ac89cb..02d9809 100644
> >> --- a/drivers/net/ixgbe/ixgbe_rxtx.c
> >> +++ b/drivers/net/ixgbe/ixgbe_rxtx.c
> >> @@ -5677,7 +5677,6 @@ ixgbevf_dev_rx_init(struct rte_eth_dev *dev)
> >>              (uint16_t)dev->data->dev_conf.rxmode.max_rx_pkt_len)) {
> >>                  PMD_INIT_LOG(ERR, "Set max packet length to %d
> failed.",
> >>
> dev->data->dev_conf.rxmode.max_rx_pkt_len);
> >> -               return -EINVAL;
> >>          }
> >>
> >>          /*
> >> --
> >> 2.7.4
> >>
> >>
  

Hi Ferruh,

I have tested with the dpdk-next-net branch.
It includes the commit 'ethdev: fix max Rx packet length'

Setup:
Hypervisor: VMware ESXi 6.0.0
PF ixgbe Driver: 3.7.13.7 (default PF driver installed with ESXi 6.0 and
6.5)
NIC: Intel 82599
Guest OS : Ubuntu 20.04

It seems that with the recent changes testpmd still can't initialize the
ports.

EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
EAL: Probe PCI driver: net_ixgbe_vf (8086:10ed) device: 0000:0b:00.0
(socket 0)
EAL: Probe PCI driver: net_ixgbe_vf (8086:10ed) device: 0000:13:00.0
(socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
previous number of forwarding ports 2 - changed to number of configured
ports 1
Error picking flow transfer proxy for port 0: Function not implemented -
ignore
Error picking flow transfer proxy for port 1: Function not implemented -
ignore
testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port
will pair with itself.

Configuring Port 0 (socket 0)
ixgbevf_dev_rx_init(): Set max packet length to 1518 failed.
ixgbevf_dev_start(): Unable to initialize RX hardware (-22)
Fail to start port 0: Invalid argument
Configuring Port 1 (socket 0)
ixgbevf_dev_rx_init(): Set max packet length to 1518 failed.
ixgbevf_dev_start(): Unable to initialize RX hardware (-22)
Fail to start port 1: Invalid argument
Please stop the ports first
Done
Error during enabling promiscuous mode for port 0: Operation not supported
- ignore
Error during enabling promiscuous mode for port 1: Operation not supported
- ignore
testpmd> start tx_first
Not all ports were started

I think failing to set set 'max_rx_pkt_len' after ixgbevf_rlpml_set_vf()
call failed in ixgbevf_dev_set_mtu(), might have been one half of the
problem.
The other one is in ixgbevf_dev_rx_init(). The init function seems to
return prematurely, and not initialize the ports.

With the below patch (on top of net-next branch), I seem to be able to
initialize the ports correctly, and send packets using testpmd.

diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c b/drivers/net/ixgbe/ixgbe_rxtx.c
index b263dfe1d5..fdd057c789 100644
--- a/drivers/net/ixgbe/ixgbe_rxtx.c
+++ b/drivers/net/ixgbe/ixgbe_rxtx.c
@@ -5676,7 +5676,6 @@ ixgbevf_dev_rx_init(struct rte_eth_dev *dev)
        if (ixgbevf_rlpml_set_vf(hw, frame_size) != 0) {
                PMD_INIT_LOG(ERR, "Set max packet length to %d failed.",
                             frame_size);
-               return -EINVAL;
        }

        /*

Alvin, would it be acceptable to not return -EINVAL in this case while
still printing an error, so that we can still debug mtu issues on 82599 ?

If I recall correctly, the NIC has issues when the MTU of VFs is bigger
than the MTU of the PFs. On my setup, however the MTUs have default values
(1500).
Should I rebase the patch on top of net-next ?

Thanks,
Tudor

On Wed, 20 Oct 2021 at 06:08, Zhang, AlvinX <alvinx.zhang@intel.com> wrote:

> > -----Original Message-----
> > From: Yigit, Ferruh <ferruh.yigit@intel.com>
> > Sent: Tuesday, October 19, 2021 8:58 PM
> > To: Tudor Cornea <tudor.cornea@gmail.com>; Zhang, Qi Z
> > <qi.z.zhang@intel.com>
> > Cc: Zhang, AlvinX <alvinx.zhang@intel.com>; Wang, Haiyue
> > <haiyue.wang@intel.com>; dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH] net/ixgbe: initialize max_rx_pkt_len if
> > rlpml_set_vf fails
> >
> > On 10/15/2021 3:20 PM, Tudor Cornea wrote:
> > > Some of our customers use ESXi 6.0 or 6.5 servers, which could have
> > > older versions of the PF ixgbe driver.
> > > It seems that with a more recent version of the PMD driver, we are not
> > > able to initialize 82599EB ports correctly.
> > > This scenario seems to have worked with DPDK 19.11.
> > >
> > > Would it be possible to print a warning, while still allowing the
> > > driver to initialize the ports ?
>
> There is a scenario that we initialize the port successful, with no any
> error, but we cannot get any packets.
> So keep printing error or warning and still allowing the driver to
> initialize the port may be the best solution.
>
> > >
> > > I was also thinking about the return code of ixgbevf_dev_set_mtu.
> > > Do you think it would be more appropriate to return ENOTSUP or ENOSYS
> > > instead of EINVAL ?
>
> If ixgbevf_dev_set_mtu fails, we have no way to know it because of invalid
> value or not supported?
> ixgbevf_dev_set_mtu returns one the these values: 0 or IXGBE_ERR_MBX
>
> > >
> > > As a user, calling 'rte_eth_dev_mtu_set', I would expect an error like
> > > EINVAL to suggest to me that the mtu value which I provided is
> > > incorrect [1]. The 82599 NIC, however has some particularities related
> > > to mtu, which could cause the operation to fail. I was thinking that
> > > EINVAL might not be most descriptive error.
> > >
> > > [1] https://doc.dpdk.org/api/rte__ethdev_8h.html
> > >
> > > Thanks,
> > > Tudor
> >
> > Hi Tudor,
> >
> > Can you please check if the patch is still after 'max_rx_pkt_len'
> related changes
> > in next-net?
> >
> > >
> > > On Fri, 15 Oct 2021 at 17:06, Tudor Cornea <tudor.cornea@gmail.com>
> > wrote:
> > >
> > >> It seems that if the call to ixgbevf_rlpml_set_vf fails, we will not
> > >> initialize dev_conf.rxmode.max_rx_pkt_len correctly anymore.
> > >>
> > >> This happens with a 82599EB NIC and a VMware ESXI 6.0 setup, and is
> > >> causing VF the ports to fail to initialize
> > >>
> > >> We see the following error:
> > >> ixgbevf_dev_rx_init(): Set max packet length to 1518 failed.
> > >>
> > >> Investigating over DPDK 19.11, it seems that the call still fails,
> > >> but it doesn't exit prematurely, and max_rx_pkt_len is initialized in
> > >> the respective case.
> > >>
> > >> On the ESXi server, we seem to have the following driver
> > >> net-ixgbe: 3.7.13.7.14iov-20vmw.600.0.0.2494585
> > >>
> > >> It seems that the behavior related to MTU setting has changed since
> > >> the following commit:
> > >>
> > >> commit c77866a16904 ("net/ixgbe: detect failed VF MTU set")
> > >>
> > >> We would like to still be able to support older setups if possible,
> > >> as we might have customers running ESXi 6.0 or 6.5, and these seem to
> > >> have an older version of the PF driver as default.
> > >>
> > >> Signed-off-by: Tudor Cornea <tudor.cornea@gmail.com>
> > >> ---
> > >>   drivers/net/ixgbe/ixgbe_ethdev.c | 5 +++--
> > >>   drivers/net/ixgbe/ixgbe_rxtx.c   | 1 -
> > >>   2 files changed, 3 insertions(+), 3 deletions(-)
> > >>
> > >> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
> > >> b/drivers/net/ixgbe/ixgbe_ethdev.c
> > >> index 4dbe049..4301dfd 100644
> > >> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> > >> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> > >> @@ -6369,6 +6369,9 @@ ixgbevf_dev_set_mtu(struct rte_eth_dev *dev,
> > >> uint16_t mtu)
> > >>                  return -EINVAL;
> > >>          }
> > >>
> > >> +       /* update max frame size */
> > >> +       dev->data->dev_conf.rxmode.max_rx_pkt_len = max_frame;
> > >> +
> > >>          /*
> > >>           * When supported by the underlying PF driver, use the
> > >> IXGBE_VF_SET_MTU
> > >>           * request of the version 2.0 of the mailbox API.
> > >> @@ -6381,8 +6384,6 @@ ixgbevf_dev_set_mtu(struct rte_eth_dev *dev,
> > >> uint16_t mtu)
> > >>          if (ixgbevf_rlpml_set_vf(hw, max_frame))
> > >>                  return -EINVAL;
> > >>
> > >> -       /* update max frame size */
> > >> -       dev->data->dev_conf.rxmode.max_rx_pkt_len = max_frame;
> > >>          return 0;
> > >>   }
> > >>
> > >> diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c
> > >> b/drivers/net/ixgbe/ixgbe_rxtx.c index 0ac89cb..02d9809 100644
> > >> --- a/drivers/net/ixgbe/ixgbe_rxtx.c
> > >> +++ b/drivers/net/ixgbe/ixgbe_rxtx.c
> > >> @@ -5677,7 +5677,6 @@ ixgbevf_dev_rx_init(struct rte_eth_dev *dev)
> > >>              (uint16_t)dev->data->dev_conf.rxmode.max_rx_pkt_len)) {
> > >>                  PMD_INIT_LOG(ERR, "Set max packet length to %d
> > failed.",
> > >>
> > dev->data->dev_conf.rxmode.max_rx_pkt_len);
> > >> -               return -EINVAL;
> > >>          }
> > >>
> > >>          /*
> > >> --
> > >> 2.7.4
> > >>
> > >>
>
>
  

since this is not a plain text mail, have to reply on top. ;-)

By checking the kernel implementation, this fix makes sense, and finally it works.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c#n2015

Just give an error message:

        if (ret)
               dev_err(&adapter->pdev->dev,
                       "Failed to set MTU at %d\n", netdev->mtu);

BR,
Haiyue

From: Tudor Cornea <tudor.cornea@gmail.com>
Sent: Wednesday, October 20, 2021 15:13
To: Zhang, AlvinX <alvinx.zhang@intel.com>
Cc: Yigit, Ferruh <ferruh.yigit@intel.com>; Zhang, Qi Z <qi.z.zhang@intel.com>; Wang, Haiyue <haiyue.wang@intel.com>; dev@dpdk.org
Subject: Re: [dpdk-dev] [PATCH] net/ixgbe: initialize max_rx_pkt_len if rlpml_set_vf fails

Hi Ferruh,

I have tested with the dpdk-next-net branch.
It includes the commit 'ethdev: fix max Rx packet length'

Setup:
Hypervisor: VMware ESXi 6.0.0
PF ixgbe Driver: 3.7.13.7 (default PF driver installed with ESXi 6.0 and 6.5)
NIC: Intel 82599
Guest OS : Ubuntu 20.04

It seems that with the recent changes testpmd still can't initialize the ports.

EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
EAL: Probe PCI driver: net_ixgbe_vf (8086:10ed) device: 0000:0b:00.0 (socket 0)
EAL: Probe PCI driver: net_ixgbe_vf (8086:10ed) device: 0000:13:00.0 (socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
previous number of forwarding ports 2 - changed to number of configured ports 1
Error picking flow transfer proxy for port 0: Function not implemented - ignore
Error picking flow transfer proxy for port 1: Function not implemented - ignore
testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port will pair with itself.

Configuring Port 0 (socket 0)
ixgbevf_dev_rx_init(): Set max packet length to 1518 failed.
ixgbevf_dev_start(): Unable to initialize RX hardware (-22)
Fail to start port 0: Invalid argument
Configuring Port 1 (socket 0)
ixgbevf_dev_rx_init(): Set max packet length to 1518 failed.
ixgbevf_dev_start(): Unable to initialize RX hardware (-22)
Fail to start port 1: Invalid argument
Please stop the ports first
Done
Error during enabling promiscuous mode for port 0: Operation not supported - ignore
Error during enabling promiscuous mode for port 1: Operation not supported - ignore
testpmd> start tx_first
Not all ports were started

I think failing to set set 'max_rx_pkt_len' after ixgbevf_rlpml_set_vf() call failed in ixgbevf_dev_set_mtu(), might have been one half of the problem.
The other one is in ixgbevf_dev_rx_init(). The init function seems to return prematurely, and not initialize the ports.

With the below patch (on top of net-next branch), I seem to be able to initialize the ports correctly, and send packets using testpmd.

diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c b/drivers/net/ixgbe/ixgbe_rxtx.c
index b263dfe1d5..fdd057c789 100644
--- a/drivers/net/ixgbe/ixgbe_rxtx.c
+++ b/drivers/net/ixgbe/ixgbe_rxtx.c
@@ -5676,7 +5676,6 @@ ixgbevf_dev_rx_init(struct rte_eth_dev *dev)
        if (ixgbevf_rlpml_set_vf(hw, frame_size) != 0) {
                PMD_INIT_LOG(ERR, "Set max packet length to %d failed.",
                             frame_size);
-               return -EINVAL;
        }

        /*

Alvin, would it be acceptable to not return -EINVAL in this case while still printing an error, so that we can still debug mtu issues on 82599 ?

If I recall correctly, the NIC has issues when the MTU of VFs is bigger than the MTU of the PFs. On my setup, however the MTUs have default values (1500).
Should I rebase the patch on top of net-next ?

Thanks,
Tudor

On Wed, 20 Oct 2021 at 06:08, Zhang, AlvinX <alvinx.zhang@intel.com<mailto:alvinx.zhang@intel.com>> wrote:
> -----Original Message-----
> From: Yigit, Ferruh <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
> Sent: Tuesday, October 19, 2021 8:58 PM
> To: Tudor Cornea <tudor.cornea@gmail.com<mailto:tudor.cornea@gmail.com>>; Zhang, Qi Z
> <qi.z.zhang@intel.com<mailto:qi.z.zhang@intel.com>>
> Cc: Zhang, AlvinX <alvinx.zhang@intel.com<mailto:alvinx.zhang@intel.com>>; Wang, Haiyue
> <haiyue.wang@intel.com<mailto:haiyue.wang@intel.com>>; dev@dpdk.org<mailto:dev@dpdk.org>
> Subject: Re: [dpdk-dev] [PATCH] net/ixgbe: initialize max_rx_pkt_len if
> rlpml_set_vf fails
>
> On 10/15/2021 3:20 PM, Tudor Cornea wrote:
> > Some of our customers use ESXi 6.0 or 6.5 servers, which could have
> > older versions of the PF ixgbe driver.
> > It seems that with a more recent version of the PMD driver, we are not
> > able to initialize 82599EB ports correctly.
> > This scenario seems to have worked with DPDK 19.11.
> >
> > Would it be possible to print a warning, while still allowing the
> > driver to initialize the ports ?

There is a scenario that we initialize the port successful, with no any error, but we cannot get any packets.
So keep printing error or warning and still allowing the driver to initialize the port may be the best solution.

> >
> > I was also thinking about the return code of ixgbevf_dev_set_mtu.
> > Do you think it would be more appropriate to return ENOTSUP or ENOSYS
> > instead of EINVAL ?

If ixgbevf_dev_set_mtu fails, we have no way to know it because of invalid value or not supported?
ixgbevf_dev_set_mtu returns one the these values: 0 or IXGBE_ERR_MBX

> >
> > As a user, calling 'rte_eth_dev_mtu_set', I would expect an error like
> > EINVAL to suggest to me that the mtu value which I provided is
> > incorrect [1]. The 82599 NIC, however has some particularities related
> > to mtu, which could cause the operation to fail. I was thinking that
> > EINVAL might not be most descriptive error.
> >
> > [1] https://doc.dpdk.org/api/rte__ethdev_8h.html
> >
> > Thanks,
> > Tudor
>
> Hi Tudor,
>
> Can you please check if the patch is still after 'max_rx_pkt_len' related changes
> in next-net?
>
> >
> > On Fri, 15 Oct 2021 at 17:06, Tudor Cornea <tudor.cornea@gmail.com<mailto:tudor.cornea@gmail.com>>
> wrote:
> >
> >> It seems that if the call to ixgbevf_rlpml_set_vf fails, we will not
> >> initialize dev_conf.rxmode.max_rx_pkt_len correctly anymore.
> >>
> >> This happens with a 82599EB NIC and a VMware ESXI 6.0 setup, and is
> >> causing VF the ports to fail to initialize
> >>
> >> We see the following error:
> >> ixgbevf_dev_rx_init(): Set max packet length to 1518 failed.
> >>
> >> Investigating over DPDK 19.11, it seems that the call still fails,
> >> but it doesn't exit prematurely, and max_rx_pkt_len is initialized in
> >> the respective case.
> >>
> >> On the ESXi server, we seem to have the following driver
> >> net-ixgbe: 3.7.13.7.14iov-20vmw.600.0.0.2494585
> >>
> >> It seems that the behavior related to MTU setting has changed since
> >> the following commit:
> >>
> >> commit c77866a16904 ("net/ixgbe: detect failed VF MTU set")
> >>
> >> We would like to still be able to support older setups if possible,
> >> as we might have customers running ESXi 6.0 or 6.5, and these seem to
> >> have an older version of the PF driver as default.
> >>
> >> Signed-off-by: Tudor Cornea <tudor.cornea@gmail.com<mailto:tudor.cornea@gmail.com>>
> >> ---
> >>   drivers/net/ixgbe/ixgbe_ethdev.c | 5 +++--
> >>   drivers/net/ixgbe/ixgbe_rxtx.c   | 1 -
> >>   2 files changed, 3 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
> >> b/drivers/net/ixgbe/ixgbe_ethdev.c
> >> index 4dbe049..4301dfd 100644
> >> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> >> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> >> @@ -6369,6 +6369,9 @@ ixgbevf_dev_set_mtu(struct rte_eth_dev *dev,
> >> uint16_t mtu)
> >>                  return -EINVAL;
> >>          }
> >>
> >> +       /* update max frame size */
> >> +       dev->data->dev_conf.rxmode.max_rx_pkt_len = max_frame;
> >> +
> >>          /*
> >>           * When supported by the underlying PF driver, use the
> >> IXGBE_VF_SET_MTU
> >>           * request of the version 2.0 of the mailbox API.
> >> @@ -6381,8 +6384,6 @@ ixgbevf_dev_set_mtu(struct rte_eth_dev *dev,
> >> uint16_t mtu)
> >>          if (ixgbevf_rlpml_set_vf(hw, max_frame))
> >>                  return -EINVAL;
> >>
> >> -       /* update max frame size */
> >> -       dev->data->dev_conf.rxmode.max_rx_pkt_len = max_frame;
> >>          return 0;
> >>   }
> >>
> >> diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c
> >> b/drivers/net/ixgbe/ixgbe_rxtx.c index 0ac89cb..02d9809 100644
> >> --- a/drivers/net/ixgbe/ixgbe_rxtx.c
> >> +++ b/drivers/net/ixgbe/ixgbe_rxtx.c
> >> @@ -5677,7 +5677,6 @@ ixgbevf_dev_rx_init(struct rte_eth_dev *dev)
> >>              (uint16_t)dev->data->dev_conf.rxmode.max_rx_pkt_len)) {
> >>                  PMD_INIT_LOG(ERR, "Set max packet length to %d
> failed.",
> >>
> dev->data->dev_conf.rxmode.max_rx_pkt_len);
> >> -               return -EINVAL;
> >>          }
> >>
> >>          /*
> >> --
> >> 2.7.4
> >>
> >>