[dpdk-dev,0/4] virtio support for container

  Hello,

On 1/12/2016 11:11 PM, Amit Tomer wrote:
> Hello,
>
>> In vhost-switch, it judges if a virtio device is ready for processing after
>> receiving
>> a pkt from virtio device. So you'd better construct a pkt, and send it out
>> firstly
>> in l2fwd.
> I tried to ping the socket interface from host for the same purpose
> but it didn't work.
>
> Could you please suggest some other approach for achieving same(how
> pkt can be sent out to l2fwd)?
>
> Also, before trying this, I have verified that vhost-switch is working
> ok with testpmd .
>
> Thanks,
> Amit.

You can use below patch for l2fwd to send out an arp packet when it gets 
started.

+#endif
+
+
  /* Send the burst of packets on an output interface */
  static int
  l2fwd_send_burst(struct lcore_queue_conf *qconf, unsigned n, uint8_t port)
@@ -281,6 +330,9 @@ l2fwd_main_loop(void)
          portid = qconf->rx_port_list[i];
          RTE_LOG(INFO, L2FWD, " -- lcoreid=%u portid=%u\n", lcore_id,
              portid);
+#ifdef SEND_ARP
+        dpdk_send_arp(portid, l2fwd_pktmbuf_pool);
+#endif
      }

      while (1) {
  

Hello,

>
> You can use below patch for l2fwd to send out an arp packet when it gets
> started.

I tried to send out arp packet using this patch but buffer allocation
for arp packets itself gets failed:

 m = rte_pktmbuf_alloc(mp);

Return a NULL Value.

Thanks,
Amit.
  

Hi Amit,

On 1/13/2016 11:00 PM, Amit Tomer wrote:
> Hello,
>
>> You can use below patch for l2fwd to send out an arp packet when it gets
>> started.
> I tried to send out arp packet using this patch but buffer allocation
> for arp packets itself gets failed:
>
>   m = rte_pktmbuf_alloc(mp);
>
> Return a NULL Value.

Can you send out how you start this l2fwd program?

Thanks,
Jianfeng


>
> Thanks,
> Amit.
  

Hello,

> Can you send out how you start this l2fwd program?

This is how, I run l2fwd program.

CMD ["/usr/src/dpdk/examples/l2fwd/build/l2fwd", "-c", "0x3", "-n",
"4","--no-pci",
,"--no-huge","--vdev=eth_cvio0,queue_num=256,rx=1,tx=1,cq=0,path=/usr/src/dpdk/usvhost",
"--", "-p", "0x1"]

I tried passing "-m 1024" to it but It causes l2fwd killed even before
it could connect to usvhost socket.

Do I need to create Hugepages from Inside Docker container to make use
of Hugepages?

Thanks,
Amit.
  

Hi Amit,

On 1/14/2016 5:34 PM, Amit Tomer wrote:
> Hello,
>
>> Can you send out how you start this l2fwd program?
> This is how, I run l2fwd program.
>
> CMD ["/usr/src/dpdk/examples/l2fwd/build/l2fwd", "-c", "0x3", "-n",
> "4","--no-pci",
> ,"--no-huge","--vdev=eth_cvio0,queue_num=256,rx=1,tx=1,cq=0,path=/usr/src/dpdk/usvhost",
> "--", "-p", "0x1"]

In this way, you can only get 64M memory. I believe it's too small to 
create a l2fwd_pktmbuf_pool in l2fwd.

> I tried passing "-m 1024" to it but It causes l2fwd killed even before
> it could connect to usvhost socket.

In my patch, when --no-huge is specified, I change previous anonymous 
mmap into file-baked memory in /dev/shm. And usually, Docker mounts a 
64MB-size tmpfs there, so you cannot use -m 1024. If you want to do 
that, use -v to substitute the 64MB tmpfs with a bigger tmpfs.


>
> Do I need to create Hugepages from Inside Docker container to make use
> of Hugepages?

Not necessary. But if you want to use hugepages inside Docker, use -v 
option to map a hugetlbfs into containers.

Most importantly, you indeed uncover a bug here. Current implementation 
cannot work with tmpfs, because it lacks ftruncate() between open() and 
mmap(). It turns out that although mmap() succeeds, the memory cannot be 
touched. However, this is not a problem for hugetlbfs. I don't why they 
differ like that way. In all, if you want to use no-huge, please add 
ftruncate(), I'll fix this in next version.

Thanks,
Jianfeng

>
> Thanks,
> Amit.
  

Hello,

>
> Not necessary. But if you want to use hugepages inside Docker, use -v option
> to map a hugetlbfs into containers.

I modified Docker command line in order to make use of Hugetlbfs:

CMD ["/usr/src/dpdk/examples/l2fwd/build/l2fwd", "-c", "0x3", "-n",
"4","--no-pci", "--socket-mem","512",
"--vdev=eth_cvio0,queue_num=256,rx=1,tx=1,cq=0,path=/var/run/usvhost",
"--", "-p", "0x1"]

Then, I run docker :

 docker run -i -t --privileged  -v /dev/hugepages:/dev/hugepages  -v
/home/ubuntu/backup/usvhost:/var/run/usvhost  l6

But this is what I see:

EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 48 lcore(s)
EAL: Setting up physically contiguous memory...
EAL: Failed to find phys addr for 2 MB pages
PANIC in rte_eal_init():
Cannot init memory
1: [/usr/src/dpdk/examples/l2fwd/build/l2fwd(rte_dump_stack+0x20) [0x48ea78]]

This is from Host:

# mount | grep hugetlbfs
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
none on /dev/hugepages type hugetlbfs (rw,relatime)

 #cat /proc/meminfo | grep Huge
AnonHugePages:    548864 kB
HugePages_Total:    4096
HugePages_Free:     1024
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB

What is it, I'm doing wrong here?

Thanks,
Amit
  

Hi Amit,

On 1/14/2016 8:03 PM, Amit Tomer wrote:
> Hello,
>
>> Not necessary. But if you want to use hugepages inside Docker, use -v option
>> to map a hugetlbfs into containers.
> I modified Docker command line in order to make use of Hugetlbfs:
>
> CMD ["/usr/src/dpdk/examples/l2fwd/build/l2fwd", "-c", "0x3", "-n",
> "4","--no-pci", "--socket-mem","512",
> "--vdev=eth_cvio0,queue_num=256,rx=1,tx=1,cq=0,path=/var/run/usvhost",
> "--", "-p", "0x1"]



For this case, please use --single-file option because it creates much 
more than 8 fds, which can be handled by vhost-user sendmsg().


>
> Then, I run docker :
>
>   docker run -i -t --privileged  -v /dev/hugepages:/dev/hugepages  -v
> /home/ubuntu/backup/usvhost:/var/run/usvhost  l6
>
> But this is what I see:
>
> EAL: Support maximum 128 logical core(s) by configuration.
> EAL: Detected 48 lcore(s)
> EAL: Setting up physically contiguous memory...
> EAL: Failed to find phys addr for 2 MB pages
> PANIC in rte_eal_init():
> Cannot init memory
> 1: [/usr/src/dpdk/examples/l2fwd/build/l2fwd(rte_dump_stack+0x20) [0x48ea78]]

 From the log, it's caused by that it still cannot open 
/proc/self/pagemap. But it's strange that you already specify --privileged).

Thanks,
Jianfeng

>
> This is from Host:
>
> # mount | grep hugetlbfs
> hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
> none on /dev/hugepages type hugetlbfs (rw,relatime)
>
>   #cat /proc/meminfo | grep Huge
> AnonHugePages:    548864 kB
> HugePages_Total:    4096
> HugePages_Free:     1024
> HugePages_Rsvd:        0
> HugePages_Surp:        0
> Hugepagesize:       2048 kB
>
> What is it, I'm doing wrong here?
>
> Thanks,
> Amit
  

Hello,

> For this case, please use --single-file option because it creates much more
> than 8 fds, which can be handled by vhost-user sendmsg().

Thanks, I'm able to verify it by sending ARP packet from container to
host on arm64. But sometimes, I do see following message while running
l2fwd in container(pointed by Rich).

EAL: Master lcore 0 is ready (tid=8a7a3000;cpuset=[0])
EAL: lcore 1 is ready (tid=89cdf050;cpuset=[1])
Notice: odd number of ports in portmask.
Lcore 0: RX port 0
Initializing port 0... PANIC in kick_all_vq():
TUNSETVNETHDRSZ failed: Inappropriate ioctl for device

How it could be avoided?

Thanks,
Amit.