diff mbox series

Skip vfio in the scenario of non-privileged mode

Message ID 20250117072847.2741-1-ming.1.yang@nokia-sbell.com (mailing list archive)
State Superseded
Delegated to: Thomas Monjalon
Headers
Series Skip vfio in the scenario of non-privileged mode |

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/loongarch-compilation success Compilation OK
ci/loongarch-unit-testing success Unit Testing PASS
ci/github-robot: build success github build: passed
ci/iol-intel-Performance success Performance Testing PASS
ci/iol-mellanox-Performance success Performance Testing PASS
ci/iol-broadcom-Performance success Performance Testing PASS
ci/iol-marvell-Functional success Functional Testing PASS
ci/Intel-compilation success Compilation OK
ci/intel-Testing success Testing PASS
ci/iol-intel-Functional success Functional Testing PASS
ci/intel-Functional success Functional PASS
ci/iol-abi-testing success Testing PASS
ci/iol-unit-amd64-testing success Testing PASS
ci/iol-unit-arm64-testing success Testing PASS
ci/iol-compile-arm64-testing success Testing PASS
ci/iol-sample-apps-testing success Testing PASS
ci/iol-compile-amd64-testing success Testing PASS

Commit Message

Yang Ming Jan. 17, 2025, 7:28 a.m. UTC 
DPDK detect vfio container according the existence of vfio
module. But for container with non-privileged mode, there is
possibility that no VFIO_DIR(/dev/vfio) mapping from host to
container when host have both Intel NIC and Mellanox NIC but
this conntainer only allocate VFs from Mellanox NIC.
In this case, vfio kernel module has already been loaded from
the host.
This scenario will cause the error log occurs in DPDK primary
process as below:
'EAL:   cannot open VFIO container, error 2 (No such file or
directory)'
'EAL: VFIO support could not be initialized'
Because `rte_vfio_enable()` call `rte_vfio_get_container_fd()`
to execute `vfio_container_fd = open(VFIO_CONTAINER_PATH,
O_RDWR);` but VFIO_CONTAINER_PATH(/dev/vfio/vfio) doesn't exist
in this container.
This scenario will also lead to the delay of DPDK secondary
process because `default_vfio_cfg->vfio_enabled = 0` and
`default_vfio_cfg->vfio_container_fd = -1`, socket error will
be set in DPDK primary process when it sync this info to
the secondary process.
This patch use to skip this kind of useless detection for this
scenario.

Signed-off-by: Yang Ming <ming.1.yang@nokia-sbell.com>
---
 lib/eal/linux/eal_vfio.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

Comments

Stephen Hemminger Jan. 17, 2025, 4:47 p.m. UTC | #1 
On Fri, 17 Jan 2025 15:28:47 +0800
Yang Ming <ming.1.yang@nokia-sbell.com> wrote:

> DPDK detect vfio container according the existence of vfio
> module. But for container with non-privileged mode, there is
> possibility that no VFIO_DIR(/dev/vfio) mapping from host to
> container when host have both Intel NIC and Mellanox NIC but
> this conntainer only allocate VFs from Mellanox NIC.
> In this case, vfio kernel module has already been loaded from
> the host.
> This scenario will cause the error log occurs in DPDK primary
> process as below:
> 'EAL:   cannot open VFIO container, error 2 (No such file or
> directory)'
> 'EAL: VFIO support could not be initialized'
> Because `rte_vfio_enable()` call `rte_vfio_get_container_fd()`
> to execute `vfio_container_fd = open(VFIO_CONTAINER_PATH,
> O_RDWR);` but VFIO_CONTAINER_PATH(/dev/vfio/vfio) doesn't exist
> in this container.
> This scenario will also lead to the delay of DPDK secondary
> process because `default_vfio_cfg->vfio_enabled = 0` and
> `default_vfio_cfg->vfio_container_fd = -1`, socket error will
> be set in DPDK primary process when it sync this info to
> the secondary process.
> This patch use to skip this kind of useless detection for this
> scenario.
> 
> Signed-off-by: Yang Ming <ming.1.yang@nokia-sbell.com>
> ---
>  lib/eal/linux/eal_vfio.c | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
> index 7132e24cba..1679d29263 100644
> --- a/lib/eal/linux/eal_vfio.c
> +++ b/lib/eal/linux/eal_vfio.c
> @@ -7,6 +7,7 @@
>  #include <fcntl.h>
>  #include <unistd.h>
>  #include <sys/ioctl.h>
> +#include <dirent.h>
>  
>  #include <rte_errno.h>
>  #include <rte_log.h>
> @@ -1083,6 +1084,7 @@ rte_vfio_enable(const char *modname)
>  	/* initialize group list */
>  	int i, j;
>  	int vfio_available;
> +	DIR *dir;
>  	const struct internal_config *internal_conf =
>  		eal_get_internal_configuration();
>  
> @@ -1119,6 +1121,15 @@ rte_vfio_enable(const char *modname)
>  		return 0;
>  	}
>  
> +	/* return 0 if VFIO directory not exist for container with non-privileged mode */
> +	dir = opendir(VFIO_DIR);
> +	if (dir == NULL) {
> +		EAL_LOG(DEBUG,
> +			"VFIO directory not exist, skipping VFIO support...");
> +		return 0;
> +	}
> +	closedir(dir);

You need to test the non-container cases.
If vfio is loaded /dev/vfio is a character device (not a directory)

Also looks suspicious that VFIO_DIR is defined but never used currently.
Yang Ming Jan. 22, 2025, 8:15 a.m. UTC | #2 
On 2025/1/18 00:47, Stephen Hemminger wrote:
> Caution: This is an external email. Please be very careful when clicking links or opening attachments. See http://nok.it/nsb for additional information.
>
> On Fri, 17 Jan 2025 15:28:47 +0800
> Yang Ming <ming.1.yang@nokia-sbell.com> wrote:
>
>> DPDK detect vfio container according the existence of vfio
>> module. But for container with non-privileged mode, there is
>> possibility that no VFIO_DIR(/dev/vfio) mapping from host to
>> container when host have both Intel NIC and Mellanox NIC but
>> this conntainer only allocate VFs from Mellanox NIC.
>> In this case, vfio kernel module has already been loaded from
>> the host.
>> This scenario will cause the error log occurs in DPDK primary
>> process as below:
>> 'EAL:   cannot open VFIO container, error 2 (No such file or
>> directory)'
>> 'EAL: VFIO support could not be initialized'
>> Because `rte_vfio_enable()` call `rte_vfio_get_container_fd()`
>> to execute `vfio_container_fd = open(VFIO_CONTAINER_PATH,
>> O_RDWR);` but VFIO_CONTAINER_PATH(/dev/vfio/vfio) doesn't exist
>> in this container.
>> This scenario will also lead to the delay of DPDK secondary
>> process because `default_vfio_cfg->vfio_enabled = 0` and
>> `default_vfio_cfg->vfio_container_fd = -1`, socket error will
>> be set in DPDK primary process when it sync this info to
>> the secondary process.
>> This patch use to skip this kind of useless detection for this
>> scenario.
>>
>> Signed-off-by: Yang Ming <ming.1.yang@nokia-sbell.com>
>> ---
>>   lib/eal/linux/eal_vfio.c | 11 +++++++++++
>>   1 file changed, 11 insertions(+)
>>
>> diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
>> index 7132e24cba..1679d29263 100644
>> --- a/lib/eal/linux/eal_vfio.c
>> +++ b/lib/eal/linux/eal_vfio.c
>> @@ -7,6 +7,7 @@
>>   #include <fcntl.h>
>>   #include <unistd.h>
>>   #include <sys/ioctl.h>
>> +#include <dirent.h>
>>   
>>   #include <rte_errno.h>
>>   #include <rte_log.h>
>> @@ -1083,6 +1084,7 @@ rte_vfio_enable(const char *modname)
>>   	/* initialize group list */
>>   	int i, j;
>>   	int vfio_available;
>> +	DIR *dir;
>>   	const struct internal_config *internal_conf =
>>   		eal_get_internal_configuration();
>>   
>> @@ -1119,6 +1121,15 @@ rte_vfio_enable(const char *modname)
>>   		return 0;
>>   	}
>>   
>> +	/* return 0 if VFIO directory not exist for container with non-privileged mode */
>> +	dir = opendir(VFIO_DIR);
>> +	if (dir == NULL) {
>> +		EAL_LOG(DEBUG,
>> +			"VFIO directory not exist, skipping VFIO support...");
>> +		return 0;
>> +	}
>> +	closedir(dir);
> You need to test the non-container cases.
> If vfio is loaded /dev/vfio is a character device (not a directory)
>
> Also looks suspicious that VFIO_DIR is defined but never used currently.
>
Hi Stephen,
For non-container test, /dev/vfio/vfio will be character device, not 
/dev/vfio.
Here is the command result on my testing environment with Intel NIC.

[root@computer-1 testuser]# ls -l /dev/vfio
total 0
crw-rw-rw-. 1 root root 10, 196 Jan 22 01:50 vfio
[root@computer-1 testuser]# dpdk-devbind.py -b vfio-pci 0000:04:10.2
[root@computer-1 testuser]# ls -l /dev/vfio
total 0
crw-------. 1 root root 239,   0 Jan 22 01:52 59
crw-rw-rw-. 1 root root  10, 196 Jan 22 01:50 vfio
[root@computer-1 testuser]# dpdk-devbind.py -b ixgbevf 0000:04:10.2
[root@computer-1 testuser]# ls -l /dev/vfio
total 0
crw-rw-rw-. 1 root root 10, 196 Jan 22 01:50 vfio

Can you confirm your test scenario?
Yang Ming Feb. 26, 2025, 3:10 a.m. UTC | #3 
On 2025/1/22 16:15, Yang Ming wrote:
> Hi Stephen

Hi Stephen,

Could you please confirm the comment above?

Brs,

Yang Ming
Stephen Hemminger Feb. 26, 2025, 1:45 p.m. UTC | #4 
On Wed, 22 Jan 2025 16:15:03 +0800
Yang Ming <ming.1.yang@nokia-sbell.com> wrote:

> On 2025/1/18 00:47, Stephen Hemminger wrote:
> > Caution: This is an external email. Please be very careful when clicking links or opening attachments. See http://nok.it/nsb for additional information.
> >
> > On Fri, 17 Jan 2025 15:28:47 +0800
> > Yang Ming <ming.1.yang@nokia-sbell.com> wrote:
> >  
> >> DPDK detect vfio container according the existence of vfio
> >> module. But for container with non-privileged mode, there is
> >> possibility that no VFIO_DIR(/dev/vfio) mapping from host to
> >> container when host have both Intel NIC and Mellanox NIC but
> >> this conntainer only allocate VFs from Mellanox NIC.
> >> In this case, vfio kernel module has already been loaded from
> >> the host.
> >> This scenario will cause the error log occurs in DPDK primary
> >> process as below:
> >> 'EAL:   cannot open VFIO container, error 2 (No such file or
> >> directory)'
> >> 'EAL: VFIO support could not be initialized'
> >> Because `rte_vfio_enable()` call `rte_vfio_get_container_fd()`
> >> to execute `vfio_container_fd = open(VFIO_CONTAINER_PATH,
> >> O_RDWR);` but VFIO_CONTAINER_PATH(/dev/vfio/vfio) doesn't exist
> >> in this container.
> >> This scenario will also lead to the delay of DPDK secondary
> >> process because `default_vfio_cfg->vfio_enabled = 0` and
> >> `default_vfio_cfg->vfio_container_fd = -1`, socket error will
> >> be set in DPDK primary process when it sync this info to
> >> the secondary process.
> >> This patch use to skip this kind of useless detection for this
> >> scenario.
> >>
> >> Signed-off-by: Yang Ming <ming.1.yang@nokia-sbell.com>
> >> ---
> >>   lib/eal/linux/eal_vfio.c | 11 +++++++++++
> >>   1 file changed, 11 insertions(+)
> >>
> >> diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
> >> index 7132e24cba..1679d29263 100644
> >> --- a/lib/eal/linux/eal_vfio.c
> >> +++ b/lib/eal/linux/eal_vfio.c
> >> @@ -7,6 +7,7 @@
> >>   #include <fcntl.h>
> >>   #include <unistd.h>
> >>   #include <sys/ioctl.h>
> >> +#include <dirent.h>
> >>   
> >>   #include <rte_errno.h>
> >>   #include <rte_log.h>
> >> @@ -1083,6 +1084,7 @@ rte_vfio_enable(const char *modname)
> >>   	/* initialize group list */
> >>   	int i, j;
> >>   	int vfio_available;
> >> +	DIR *dir;
> >>   	const struct internal_config *internal_conf =
> >>   		eal_get_internal_configuration();
> >>   
> >> @@ -1119,6 +1121,15 @@ rte_vfio_enable(const char *modname)
> >>   		return 0;
> >>   	}
> >>   
> >> +	/* return 0 if VFIO directory not exist for container with non-privileged mode */
> >> +	dir = opendir(VFIO_DIR);
> >> +	if (dir == NULL) {
> >> +		EAL_LOG(DEBUG,
> >> +			"VFIO directory not exist, skipping VFIO support...");
> >> +		return 0;
> >> +	}
> >> +	closedir(dir);  
> > You need to test the non-container cases.
> > If vfio is loaded /dev/vfio is a character device (not a directory)
> >
> > Also looks suspicious that VFIO_DIR is defined but never used currently.
> >  
> Hi Stephen,
> For non-container test, /dev/vfio/vfio will be character device, not 
> /dev/vfio.
> Here is the command result on my testing environment with Intel NIC.
> 
> [root@computer-1 testuser]# ls -l /dev/vfio
> total 0
> crw-rw-rw-. 1 root root 10, 196 Jan 22 01:50 vfio
> [root@computer-1 testuser]# dpdk-devbind.py -b vfio-pci 0000:04:10.2
> [root@computer-1 testuser]# ls -l /dev/vfio
> total 0
> crw-------. 1 root root 239,   0 Jan 22 01:52 59
> crw-rw-rw-. 1 root root  10, 196 Jan 22 01:50 vfio
> [root@computer-1 testuser]# dpdk-devbind.py -b ixgbevf 0000:04:10.2
> [root@computer-1 testuser]# ls -l /dev/vfio
> total 0
> crw-rw-rw-. 1 root root 10, 196 Jan 22 01:50 vfio
> 
> Can you confirm your test scenario?
> 
> 

When vfio-pci is loaded but no device bound:
$ ls -l /dev/vfio
total 0
crw-rw-rw- 1 root root 10, 196 Feb 26 05:39 vfio

After binding device
$ ls -l /dev/vfio
total 0
crw------- 1 root root 511,   0 Feb 26 05:42 15
crw-rw-rw- 1 root root  10, 196 Feb 26 05:39 vfio

So testing for /dev/vfio is good indication that module is loaded.
Not sure what I was thinking earlier.
Yang Ming Feb. 28, 2025, 5:23 a.m. UTC | #5 
On 2025/2/26 21:45, Stephen Hemminger wrote:
> Caution: This is an external email. Please be very careful when clicking links or opening attachments. See http://nok.it/nsb for additional information.
>
> On Wed, 22 Jan 2025 16:15:03 +0800
> Yang Ming <ming.1.yang@nokia-sbell.com> wrote:
>
>> On 2025/1/18 00:47, Stephen Hemminger wrote:
>>> Caution: This is an external email. Please be very careful when clicking links or opening attachments. See http://nok.it/nsb for additional information.
>>>
>>> On Fri, 17 Jan 2025 15:28:47 +0800
>>> Yang Ming <ming.1.yang@nokia-sbell.com> wrote:
>>>   
>>>> DPDK detect vfio container according the existence of vfio
>>>> module. But for container with non-privileged mode, there is
>>>> possibility that no VFIO_DIR(/dev/vfio) mapping from host to
>>>> container when host have both Intel NIC and Mellanox NIC but
>>>> this conntainer only allocate VFs from Mellanox NIC.
>>>> In this case, vfio kernel module has already been loaded from
>>>> the host.
>>>> This scenario will cause the error log occurs in DPDK primary
>>>> process as below:
>>>> 'EAL:   cannot open VFIO container, error 2 (No such file or
>>>> directory)'
>>>> 'EAL: VFIO support could not be initialized'
>>>> Because `rte_vfio_enable()` call `rte_vfio_get_container_fd()`
>>>> to execute `vfio_container_fd = open(VFIO_CONTAINER_PATH,
>>>> O_RDWR);` but VFIO_CONTAINER_PATH(/dev/vfio/vfio) doesn't exist
>>>> in this container.
>>>> This scenario will also lead to the delay of DPDK secondary
>>>> process because `default_vfio_cfg->vfio_enabled = 0` and
>>>> `default_vfio_cfg->vfio_container_fd = -1`, socket error will
>>>> be set in DPDK primary process when it sync this info to
>>>> the secondary process.
>>>> This patch use to skip this kind of useless detection for this
>>>> scenario.
>>>>
>>>> Signed-off-by: Yang Ming <ming.1.yang@nokia-sbell.com>
>>>> ---
>>>>    lib/eal/linux/eal_vfio.c | 11 +++++++++++
>>>>    1 file changed, 11 insertions(+)
>>>>
>>>> diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
>>>> index 7132e24cba..1679d29263 100644
>>>> --- a/lib/eal/linux/eal_vfio.c
>>>> +++ b/lib/eal/linux/eal_vfio.c
>>>> @@ -7,6 +7,7 @@
>>>>    #include <fcntl.h>
>>>>    #include <unistd.h>
>>>>    #include <sys/ioctl.h>
>>>> +#include <dirent.h>
>>>>    
>>>>    #include <rte_errno.h>
>>>>    #include <rte_log.h>
>>>> @@ -1083,6 +1084,7 @@ rte_vfio_enable(const char *modname)
>>>>    	/* initialize group list */
>>>>    	int i, j;
>>>>    	int vfio_available;
>>>> +	DIR *dir;
>>>>    	const struct internal_config *internal_conf =
>>>>    		eal_get_internal_configuration();
>>>>    
>>>> @@ -1119,6 +1121,15 @@ rte_vfio_enable(const char *modname)
>>>>    		return 0;
>>>>    	}
>>>>    
>>>> +	/* return 0 if VFIO directory not exist for container with non-privileged mode */
>>>> +	dir = opendir(VFIO_DIR);
>>>> +	if (dir == NULL) {
>>>> +		EAL_LOG(DEBUG,
>>>> +			"VFIO directory not exist, skipping VFIO support...");
>>>> +		return 0;
>>>> +	}
>>>> +	closedir(dir);
>>> You need to test the non-container cases.
>>> If vfio is loaded /dev/vfio is a character device (not a directory)
>>>
>>> Also looks suspicious that VFIO_DIR is defined but never used currently.
>>>   
>> Hi Stephen,
>> For non-container test, /dev/vfio/vfio will be character device, not
>> /dev/vfio.
>> Here is the command result on my testing environment with Intel NIC.
>>
>> [root@computer-1 testuser]# ls -l /dev/vfio
>> total 0
>> crw-rw-rw-. 1 root root 10, 196 Jan 22 01:50 vfio
>> [root@computer-1 testuser]# dpdk-devbind.py -b vfio-pci 0000:04:10.2
>> [root@computer-1 testuser]# ls -l /dev/vfio
>> total 0
>> crw-------. 1 root root 239,   0 Jan 22 01:52 59
>> crw-rw-rw-. 1 root root  10, 196 Jan 22 01:50 vfio
>> [root@computer-1 testuser]# dpdk-devbind.py -b ixgbevf 0000:04:10.2
>> [root@computer-1 testuser]# ls -l /dev/vfio
>> total 0
>> crw-rw-rw-. 1 root root 10, 196 Jan 22 01:50 vfio
>>
>> Can you confirm your test scenario?
>>
>>
> When vfio-pci is loaded but no device bound:
> $ ls -l /dev/vfio
> total 0
> crw-rw-rw- 1 root root 10, 196 Feb 26 05:39 vfio
>
> After binding device
> $ ls -l /dev/vfio
> total 0
> crw------- 1 root root 511,   0 Feb 26 05:42 15
> crw-rw-rw- 1 root root  10, 196 Feb 26 05:39 vfio
>
> So testing for /dev/vfio is good indication that module is loaded.
> Not sure what I was thinking earlier.
>
>
>
>
Hi Stephen,

Thank you very much for your explanation. It's very clear.
Can you help to accept this patch, or we need more comments?

Brs,
Yang Ming
diff mbox series

Patch

diff --git a/lib/eal/linux/eal_vfio.c b/lib/eal/linux/eal_vfio.c
index 7132e24cba..1679d29263 100644
--- a/lib/eal/linux/eal_vfio.c
+++ b/lib/eal/linux/eal_vfio.c
@@ -7,6 +7,7 @@ 
 #include <fcntl.h>
 #include <unistd.h>
 #include <sys/ioctl.h>
+#include <dirent.h>
 
 #include <rte_errno.h>
 #include <rte_log.h>
@@ -1083,6 +1084,7 @@  rte_vfio_enable(const char *modname)
 	/* initialize group list */
 	int i, j;
 	int vfio_available;
+	DIR *dir;
 	const struct internal_config *internal_conf =
 		eal_get_internal_configuration();
 
@@ -1119,6 +1121,15 @@  rte_vfio_enable(const char *modname)
 		return 0;
 	}
 
+	/* return 0 if VFIO directory not exist for container with non-privileged mode */
+	dir = opendir(VFIO_DIR);
+	if (dir == NULL) {
+		EAL_LOG(DEBUG,
+			"VFIO directory not exist, skipping VFIO support...");
+		return 0;
+	}
+	closedir(dir);
+
 	if (internal_conf->process_type == RTE_PROC_PRIMARY) {
 		if (vfio_mp_sync_setup() == -1) {
 			default_vfio_cfg->vfio_container_fd = -1;