[0/8] eBPF arm64 JIT support

  From: Jerin Jacob <jerinj@marvell.com>

Added eBPF arm64 JIT support to improve the eBPF program performance
on arm64.

dpdk.org/examples/bpf/t1.c application shows around 50% improvement
on OCTEON TX2 platform in JIT vs interpreter mode.

Verified the implementation using existing bpf_autotest application.

# echo "bpf_autotest" | sudo ./build/app/test -c 0x3

RTE>>bpf_autotest
run_test(test_store1) start
run_test(test_store2) start
run_test(test_load1) start
run_test(test_ldimm1) start
run_test(test_mul1) start
run_test(test_shift1) start
run_test(test_jump1) start
run_test(test_alu1) start
run_test(test_bele1) start
run_test(test_xadd1) start
run_test(test_div1) start
bpf_exec(0xffffa37c0000): division by 0 at pc: 0x68;
run_test(test_call1) start
run_test(test_call2) start
run_test(test_call3) start
Test OK

Jerin Jacob (8):
  bpf/arm64: add build infrastructure
  bpf/arm64: add prologue and epilogue
  bpf/arm64: add basic arithmetic operations
  bpf/arm64: add logical operations
  bpf/arm64: add byte swap operations
  bpf/arm64: add load and store operations
  bpf/arm64: add atomic-exchange-and-add operation
  bpf/arm64: add branch operation

 MAINTAINERS                            |    1 +
 doc/guides/prog_guide/bpf_lib.rst      |    2 +-
 doc/guides/rel_notes/release_19_11.rst |    5 +
 lib/librte_bpf/Makefile                |    2 +
 lib/librte_bpf/bpf.c                   |    4 +-
 lib/librte_bpf/bpf_impl.h              |    3 +-
 lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++
 lib/librte_bpf/meson.build             |    2 +
 8 files changed, 1466 insertions(+), 4 deletions(-)
 create mode 100644 lib/librte_bpf/bpf_jit_arm64.c
  

> 
> Added eBPF arm64 JIT support to improve the eBPF program performance
> on arm64.
> 
> dpdk.org/examples/bpf/t1.c application shows around 50% improvement
> on OCTEON TX2 platform in JIT vs interpreter mode.
> 
> Verified the implementation using existing bpf_autotest application.
> 
> # echo "bpf_autotest" | sudo ./build/app/test -c 0x3
> 
> RTE>>bpf_autotest
> run_test(test_store1) start
> run_test(test_store2) start
> run_test(test_load1) start
> run_test(test_ldimm1) start
> run_test(test_mul1) start
> run_test(test_shift1) start
> run_test(test_jump1) start
> run_test(test_alu1) start
> run_test(test_bele1) start
> run_test(test_xadd1) start
> run_test(test_div1) start
> bpf_exec(0xffffa37c0000): division by 0 at pc: 0x68;
> run_test(test_call1) start
> run_test(test_call2) start
> run_test(test_call3) start
> Test OK
> 
> Jerin Jacob (8):
>   bpf/arm64: add build infrastructure
>   bpf/arm64: add prologue and epilogue
>   bpf/arm64: add basic arithmetic operations
>   bpf/arm64: add logical operations
>   bpf/arm64: add byte swap operations
>   bpf/arm64: add load and store operations
>   bpf/arm64: add atomic-exchange-and-add operation
>   bpf/arm64: add branch operation
> 
>  MAINTAINERS                            |    1 +
>  doc/guides/prog_guide/bpf_lib.rst      |    2 +-
>  doc/guides/rel_notes/release_19_11.rst |    5 +
>  lib/librte_bpf/Makefile                |    2 +
>  lib/librte_bpf/bpf.c                   |    4 +-
>  lib/librte_bpf/bpf_impl.h              |    3 +-
>  lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++
>  lib/librte_bpf/meson.build             |    2 +
>  8 files changed, 1466 insertions(+), 4 deletions(-)
>  create mode 100644 lib/librte_bpf/bpf_jit_arm64.c
> 
> --

Series Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>

> 2.23.0
  

03/09/2019 12:59, jerinj@marvell.com:
> Added eBPF arm64 JIT support to improve the eBPF program performance
> on arm64.
> 
>  lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++

I am concerned about duplicating the BPF JIT effort in DPDK and Linux.
Could we try to pull the Linux JIT?
Is the license the only issue?

After a quick discussion, it seems the Linux authors are OK to arrange
their JIT code for sharing with userspace projects.
  

On Thu, Oct 3, 2019 at 6:21 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> 03/09/2019 12:59, jerinj@marvell.com:
> > Added eBPF arm64 JIT support to improve the eBPF program performance
> > on arm64.
> >
> >  lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++
>
> I am concerned about duplicating the BPF JIT effort in DPDK and Linux.
> Could we try to pull the Linux JIT?
> Is the license the only issue?

That's one issue.

>
> After a quick discussion, it seems the Linux authors are OK to arrange
> their JIT code for sharing with userspace projects.

I did a clean room implementation considering some optimization for
DPDK etc(Like if stack is not used then don't push stack etc)
and wherever Linux can be improved, I have submitted the patch also to
Linux as well.(Some more pending as well)

https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e4a2f8373cd332

And Linux has a framework for instruction generation for debugging
etc. So We can not copy and paste the code
from Linux as is.

My view to keep a different code base optimize for DPDK use cases and
library requirements(for example, tail call is not supported in DPDK).
For arm64/x86 case the code is done so it is not worth sync with
Linux. For new architecture, it can be if possible.

Konstantin,
Your thoughts?

>
>
  

Hi everyone,

> 
> On Thu, Oct 3, 2019 at 6:21 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > 03/09/2019 12:59, jerinj@marvell.com:
> > > Added eBPF arm64 JIT support to improve the eBPF program performance
> > > on arm64.
> > >
> > >  lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++
> >
> > I am concerned about duplicating the BPF JIT effort in DPDK and Linux.
> > Could we try to pull the Linux JIT?
> > Is the license the only issue?
> 
> That's one issue.
> 
> >
> > After a quick discussion, it seems the Linux authors are OK to arrange
> > their JIT code for sharing with userspace projects.
> 
> I did a clean room implementation considering some optimization for
> DPDK etc(Like if stack is not used then don't push stack etc)
> and wherever Linux can be improved, I have submitted the patch also to
> Linux as well.(Some more pending as well)
> 
> https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e4a2f8373cd332
> 
> And Linux has a framework for instruction generation for debugging
> etc. So We can not copy and paste the code
> from Linux as is.
> 
> My view to keep a different code base optimize for DPDK use cases and
> library requirements(for example, tail call is not supported in DPDK).
> For arm64/x86 case the code is done so it is not worth sync with
> Linux. For new architecture, it can be if possible.
> 
> Konstantin,
> Your thoughts?
> 

My thought would be that if we have JIT eBPF compiler already in DPDK
for one arch (x86) there is absolutely no reason why we shouldn't allow it for different arch (arm).
About having a common code-base with Linux eBPF JITs implementation -
I think it is a very good idea,
but I don’t' think it could be achieved without significant effort.
DPDK and Linux JIT code-generators differ quite a bit.
So my suggestion - let's go ahead and integrate Jerin patch into 19.11,
meanwhile start talking with linux guys how common JIT code-base could be achieved. 
Konstantin
  

Adding Arm JIT and Kernel experts

> -----Original Message-----
> From: Ananyev, Konstantin <konstantin.ananyev@intel.com>
> Sent: Thursday, October 3, 2019 10:06 AM
> To: Jerin Jacob <jerinjacobk@gmail.com>; thomas@monjalon.net
> Cc: jerinj@marvell.com; dpdk-dev <dev@dpdk.org>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; Gavin Hu (Arm Technology China)
> <Gavin.Hu@arm.com>
> Subject: RE: [dpdk-dev] [PATCH 0/8] eBPF arm64 JIT support
> 
> Hi everyone,
> 
> >
> > On Thu, Oct 3, 2019 at 6:21 PM Thomas Monjalon <thomas@monjalon.net>
> wrote:
> > >
> > > 03/09/2019 12:59, jerinj@marvell.com:
> > > > Added eBPF arm64 JIT support to improve the eBPF program
> > > > performance on arm64.
> > > >
> > > >  lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++
> > >
> > > I am concerned about duplicating the BPF JIT effort in DPDK and Linux.
> > > Could we try to pull the Linux JIT?
> > > Is the license the only issue?
> >
> > That's one issue.
> >
> > >
> > > After a quick discussion, it seems the Linux authors are OK to
> > > arrange their JIT code for sharing with userspace projects.
> >
> > I did a clean room implementation considering some optimization for
> > DPDK etc(Like if stack is not used then don't push stack etc) and
> > wherever Linux can be improved, I have submitted the patch also to
> > Linux as well.(Some more pending as well)
> >
> >
> https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e
> 4a
> > 2f8373cd332
> >
> > And Linux has a framework for instruction generation for debugging
> > etc. So We can not copy and paste the code from Linux as is.
> >
> > My view to keep a different code base optimize for DPDK use cases and
> > library requirements(for example, tail call is not supported in DPDK).
> > For arm64/x86 case the code is done so it is not worth sync with
> > Linux. For new architecture, it can be if possible.
> >
> > Konstantin,
> > Your thoughts?
> >
> 
> My thought would be that if we have JIT eBPF compiler already in DPDK for
> one arch (x86) there is absolutely no reason why we shouldn't allow it for
> different arch (arm).
> About having a common code-base with Linux eBPF JITs implementation - I
> think it is a very good idea, but I don’t' think it could be achieved without
> significant effort.
> DPDK and Linux JIT code-generators differ quite a bit.
> So my suggestion - let's go ahead and integrate Jerin patch into 19.11,
> meanwhile start talking with linux guys how common JIT code-base could be
> achieved.
> Konstantin
> 
> 
>
  

On Fri, Oct 04, 2019 at 05:55:18AM +0100, Honnappa Nagarahalli wrote:
> Adding Arm JIT and Kernel experts

Hi Honnappa,
I'd recommend also reaching out the BPF maintainers:
BPF JIT for ARM64
M:	Daniel Borkmann <daniel@iogearbox.net>
M:	Alexei Starovoitov <ast@kernel.org>
M:	Zi Shen Lim <zlim.lnx@gmail.com>
L:	netdev@vger.kernel.org
L:	bpf@vger.kernel.org
S:	Supported
F:	arch/arm64/net/

As they will have much better knowledge of the state of play and will be
better able to advise.

Cheers,
  

04/10/2019 11:54, Steve Capper:
> I'd recommend also reaching out the BPF maintainers:
> BPF JIT for ARM64
> M:	Daniel Borkmann <daniel@iogearbox.net>
> M:	Alexei Starovoitov <ast@kernel.org>
> M:	Zi Shen Lim <zlim.lnx@gmail.com>
> L:	netdev@vger.kernel.org
> L:	bpf@vger.kernel.org
> S:	Supported
> F:	arch/arm64/net/
> 
> As they will have much better knowledge of the state of play and will be
> better able to advise.

As far as I know Alexei and Daniel are OK with the idea.
But better to let them reply here.

I suggest we think about a way to package the kernel BPF JIT
for userspace usage (not only DPDK) as a library.
I don't understand why the DPDK JIT should be different
or optimized differently.
The only real issue I see is the need for a dual licensing BSD-GPL.
  

On 10/4/19 12:53 PM, Thomas Monjalon wrote:
> 04/10/2019 11:54, Steve Capper:
>> I'd recommend also reaching out the BPF maintainers:
>> BPF JIT for ARM64
>> M:	Daniel Borkmann <daniel@iogearbox.net>
>> M:	Alexei Starovoitov <ast@kernel.org>
>> M:	Zi Shen Lim <zlim.lnx@gmail.com>
>> L:	netdev@vger.kernel.org
>> L:	bpf@vger.kernel.org
>> S:	Supported
>> F:	arch/arm64/net/
>>
>> As they will have much better knowledge of the state of play and will be
>> better able to advise.
> 
> As far as I know Alexei and Daniel are OK with the idea.
> But better to let them reply here.
> 
> I suggest we think about a way to package the kernel BPF JIT
> for userspace usage (not only DPDK) as a library.
> I don't understand why the DPDK JIT should be different
> or optimized differently.

That would be great indeed as both projects would benefit from a shared
JIT instead of reimplementing everything twice. I never looked into DPDK
too much, but I presume the idea would be as well to take the LLVM (or
bpf-gcc) generated object file and load it into a BPF 'engine' that sits
in user space on top of DPDK? Presumably loader could be libbpf here as
well since it already knows how to parse the ELF, perform the relocations
etc. The only difference would be that you have a different context and
different helpers? Is that the goal eventually?

> The only real issue I see is the need for a dual licensing BSD-GPL.

This might be one avenue if all kernel JIT contributors would be on board.
Another option I'm wondering could be to extend the bpf() syscall in order
to pass down a description of context and helper mappings e.g. via BTF and
let everything go through the verifier in the kernel the usual way (I presume
one goal might be that you want to assure that the generated BPF code passes
the safety checks before running the prog), then have it JITed and extract
the generated image in order to use it from user space. Kernel would have
to make sure it never actually allows attaching this program in the kernel.
Generated opcodes can already be retrieved today (see below). Such infra
could potentially help bpf-gcc folks as well as they expressed desire to
have some sort of a simulator for their gcc BPF test suite.. and it would
allow for consistent behavior of the BPF runtime. Just a thought.

# bpftool prog
2: cgroup_skb  tag 7be49e3934a125ba  gpl
         loaded_at 2019-10-03T12:53:11+0200  uid 0
         xlated 296B  jited 229B  memlock 4096B  map_ids 2,3
[...]

# bpftool prog dump xlated id 2
    0: (bf) r6 = r1
    1: (69) r7 = *(u16 *)(r6 +192)
    2: (b4) w8 = 0
    3: (55) if r7 != 0x8 goto pc+14
    4: (bf) r1 = r6
    5: (b4) w2 = 16
    6: (bf) r3 = r10
    7: (07) r3 += -4
    8: (b4) w4 = 4
    9: (85) call bpf_skb_load_bytes#7484768
   10: (18) r1 = map[id:2]
   12: (bf) r2 = r10
   13: (07) r2 += -8
   14: (62) *(u32 *)(r2 +0) = 32
   15: (85) call trie_lookup_elem#90800
   16: (15) if r0 == 0x0 goto pc+1
   17: (44) w8 |= 2
   18: (55) if r7 != 0xdd86 goto pc+14
   19: (bf) r1 = r6
   20: (b4) w2 = 24
   21: (bf) r3 = r10
   22: (07) r3 += -16
   23: (b4) w4 = 16
   24: (85) call bpf_skb_load_bytes#7484768
   25: (18) r1 = map[id:3]
   27: (bf) r2 = r10
   28: (07) r2 += -20
   29: (62) *(u32 *)(r2 +0) = 128
   30: (85) call trie_lookup_elem#90800
   31: (15) if r0 == 0x0 goto pc+1
   32: (44) w8 |= 2
   33: (b7) r0 = 1
   34: (55) if r8 != 0x2 goto pc+1
   35: (b7) r0 = 0
   36: (95) exit

# bpftool prog dump jited id 2 opcodes
    0:   push   %rbp
         55
    1:   mov    %rsp,%rbp
         48 89 e5
    4:   sub    $0x40,%rsp
         48 81 ec 40 00 00 00
    b:   sub    $0x28,%rbp
         48 83 ed 28
    f:   mov    %rbx,0x0(%rbp)
         48 89 5d 00
   13:   mov    %r13,0x8(%rbp)
         4c 89 6d 08
   17:   mov    %r14,0x10(%rbp)
         4c 89 75 10
   1b:   mov    %r15,0x18(%rbp)
         4c 89 7d 18
   1f:   xor    %eax,%eax
         31 c0
   21:   mov    %rax,0x20(%rbp)
         48 89 45 20
   25:   mov    %rdi,%rbx
         48 89 fb
   28:   movzwq 0xc0(%rbx),%r13
         4c 0f b7 ab c0 00 00 00
   30:   xor    %r14d,%r14d
         45 31 f6
   33:   cmp    $0x8,%r13
         49 83 fd 08
   37:   jne    0x0000000000000079
         75 40
   39:   mov    %rbx,%rdi
         48 89 df
   3c:   mov    $0x10,%esi
         be 10 00 00 00
[...]
   cb:   jne    0x00000000000000cf
         75 02
   cd:   xor    %eax,%eax
         31 c0
   cf:   mov    0x0(%rbp),%rbx
         48 8b 5d 00
   d3:   mov    0x8(%rbp),%r13
         4c 8b 6d 08
   d7:   mov    0x10(%rbp),%r14
         4c 8b 75 10
   db:   mov    0x18(%rbp),%r15
         4c 8b 7d 18
   df:   add    $0x28,%rbp
         48 83 c5 28
   e3:   leaveq
         c9
   e4:   retq
         c3

Thanks,
Daniel
  

On Fri, Oct 4, 2019 at 7:39 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
>
> On 10/4/19 12:53 PM, Thomas Monjalon wrote:
> > 04/10/2019 11:54, Steve Capper:
> >> I'd recommend also reaching out the BPF maintainers:
> >> BPF JIT for ARM64
> >> M:   Daniel Borkmann <daniel@iogearbox.net>
> >> M:   Alexei Starovoitov <ast@kernel.org>
> >> M:   Zi Shen Lim <zlim.lnx@gmail.com>
> >> L:   netdev@vger.kernel.org
> >> L:   bpf@vger.kernel.org
> >> S:   Supported
> >> F:   arch/arm64/net/
> >>
> >> As they will have much better knowledge of the state of play and will be
> >> better able to advise.
> >
> > As far as I know Alexei and Daniel are OK with the idea.
> > But better to let them reply here.
> >
> > I suggest we think about a way to package the kernel BPF JIT
> > for userspace usage (not only DPDK) as a library.
> > I don't understand why the DPDK JIT should be different
> > or optimized differently.
>
> That would be great indeed as both projects would benefit from a shared
> JIT instead of reimplementing everything twice. I never looked into DPDK
> too much, but I presume the idea would be as well to take the LLVM (or
> bpf-gcc) generated object file and load it into a BPF 'engine' that sits
> in user space on top of DPDK? Presumably loader could be libbpf here as
> well since it already knows how to parse the ELF, perform the relocations
> etc. The only difference would be that you have a different context and
> different helpers? Is that the goal eventually?
>
> > The only real issue I see is the need for a dual licensing BSD-GPL.
>
> This might be one avenue if all kernel JIT contributors would be on board.
> Another option I'm wondering could be to extend the bpf() syscall in order
> to pass down a description of context and helper mappings e.g. via BTF and
> let everything go through the verifier in the kernel the usual way (I presume
> one goal might be that you want to assure that the generated BPF code passes
> the safety checks before running the prog), then have it JITed and extract
> the generated image in order to use it from user space. Kernel would have
> to make sure it never actually allows attaching this program in the kernel.
> Generated opcodes can already be retrieved today (see below). Such infra
> could potentially help bpf-gcc folks as well as they expressed desire to
> have some sort of a simulator for their gcc BPF test suite.. and it would
> allow for consistent behavior of the BPF runtime. Just a thought.

This idea looks good. This can remove the verifier code also from DPDK.
 A couple of downsides I can think of,

# We may need to extend the kernel verifier to understand the user-space address
and its symbols for CALL and MEM access operations.
# DPDK supports FreeBSD and Windows OS as well
# Need a different treatment for old Linux kernels.




>
>
  

On Thu, Oct 3, 2019 at 8:35 PM Ananyev, Konstantin
<konstantin.ananyev@intel.com> wrote:
>
> Hi everyone,
>
> >
> > On Thu, Oct 3, 2019 at 6:21 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > >
> > > 03/09/2019 12:59, jerinj@marvell.com:
> > > > Added eBPF arm64 JIT support to improve the eBPF program performance
> > > > on arm64.
> > > >
> > > >  lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++
> > >
> > > I am concerned about duplicating the BPF JIT effort in DPDK and Linux.
> > > Could we try to pull the Linux JIT?
> > > Is the license the only issue?
> >
> > That's one issue.
> >
> > >
> > > After a quick discussion, it seems the Linux authors are OK to arrange
> > > their JIT code for sharing with userspace projects.
> >
> > I did a clean room implementation considering some optimization for
> > DPDK etc(Like if stack is not used then don't push stack etc)
> > and wherever Linux can be improved, I have submitted the patch also to
> > Linux as well.(Some more pending as well)
> >
> > https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e4a2f8373cd332
> >
> > And Linux has a framework for instruction generation for debugging
> > etc. So We can not copy and paste the code
> > from Linux as is.
> >
> > My view to keep a different code base optimize for DPDK use cases and
> > library requirements(for example, tail call is not supported in DPDK).
> > For arm64/x86 case the code is done so it is not worth sync with
> > Linux. For new architecture, it can be if possible.
> >
> > Konstantin,
> > Your thoughts?
> >
>
> My thought would be that if we have JIT eBPF compiler already in DPDK
> for one arch (x86) there is absolutely no reason why we shouldn't allow it for different arch (arm).
> About having a common code-base with Linux eBPF JITs implementation -
> I think it is a very good idea,
> but I don’t' think it could be achieved without significant effort.
> DPDK and Linux JIT code-generators differ quite a bit.
> So my suggestion - let's go ahead and integrate Jerin patch into 19.11,
> meanwhile start talking with linux guys how common JIT code-base could be achieved.

I agree with Konstantin here.

Thomas,

Just confirm the following:

While we continue to have 'advanced' discussion on avoiding code duplication etc
and it will take a couple of months to converge(if at all it happens)

Just to be clear, I assume, you are OK to merge this code for 19.11(If
no more technical comment on the patch).

I am only afraid of, our typical last-minute surprise pattern and
followed by back and forth open ended discussions.

i.e

# Code submitted before the proposal window
# Gets ACK from Maintainer
# New non-technical concerns start just before RC1










> Konstantin
>
>
>
>
  

On 10/4/19 4:43 PM, Jerin Jacob wrote:
> On Fri, Oct 4, 2019 at 7:39 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
>> On 10/4/19 12:53 PM, Thomas Monjalon wrote:
>>> 04/10/2019 11:54, Steve Capper:
>>>> I'd recommend also reaching out the BPF maintainers:
>>>> BPF JIT for ARM64
>>>> M:   Daniel Borkmann <daniel@iogearbox.net>
>>>> M:   Alexei Starovoitov <ast@kernel.org>
>>>> M:   Zi Shen Lim <zlim.lnx@gmail.com>
>>>> L:   netdev@vger.kernel.org
>>>> L:   bpf@vger.kernel.org
>>>> S:   Supported
>>>> F:   arch/arm64/net/
>>>>
>>>> As they will have much better knowledge of the state of play and will be
>>>> better able to advise.
>>>
>>> As far as I know Alexei and Daniel are OK with the idea.
>>> But better to let them reply here.
>>>
>>> I suggest we think about a way to package the kernel BPF JIT
>>> for userspace usage (not only DPDK) as a library.
>>> I don't understand why the DPDK JIT should be different
>>> or optimized differently.
>>
>> That would be great indeed as both projects would benefit from a shared
>> JIT instead of reimplementing everything twice. I never looked into DPDK
>> too much, but I presume the idea would be as well to take the LLVM (or
>> bpf-gcc) generated object file and load it into a BPF 'engine' that sits
>> in user space on top of DPDK? Presumably loader could be libbpf here as
>> well since it already knows how to parse the ELF, perform the relocations
>> etc. The only difference would be that you have a different context and
>> different helpers? Is that the goal eventually?
>>
>>> The only real issue I see is the need for a dual licensing BSD-GPL.
>>
>> This might be one avenue if all kernel JIT contributors would be on board.
>> Another option I'm wondering could be to extend the bpf() syscall in order
>> to pass down a description of context and helper mappings e.g. via BTF and
>> let everything go through the verifier in the kernel the usual way (I presume
>> one goal might be that you want to assure that the generated BPF code passes
>> the safety checks before running the prog), then have it JITed and extract
>> the generated image in order to use it from user space. Kernel would have
>> to make sure it never actually allows attaching this program in the kernel.
>> Generated opcodes can already be retrieved today (see below). Such infra
>> could potentially help bpf-gcc folks as well as they expressed desire to
>> have some sort of a simulator for their gcc BPF test suite.. and it would
>> allow for consistent behavior of the BPF runtime. Just a thought.
> 
> This idea looks good. This can remove the verifier code also from DPDK.

Right, definitely makes sense to have consolidation also on this one as well
aside from the JITs, and pushing to the kernel and receiving back the JITed
image seems quite nice and would be generic to open up many other use cases
outside of networking. From app pov, it's just an implementation detail where
to get that BPF opcode image from anyway.

>   A couple of downsides I can think of,
> 
> # We may need to extend the kernel verifier to understand the user-space address
> and its symbols for CALL and MEM access operations.

Yep, that part would be needed, potentially BTF could be of help here as well
for the description of the user space runtime environment like context, helpers
etc, so that JIT knows how to handle this.

> # DPDK supports FreeBSD and Windows OS as well

I'm not too familiar with the state on BPF for the latter two, but afaik FreeBSD
at least had some effort to implement a BPF runtime into their kernel as well,
so similar interface could be provided, but presumably as starting point vast
majority of DPDK users are running Linux underneath anyway?

> # Need a different treatment for old Linux kernels.

Maybe, though I have little insight from DPDK angle here. Wrt BPF and kernel from
what we see major cloud providers usually offer quite recent kernels as well as
most mainstream distros that are run there, but again, environments where DPDK is
typically deployed may differ (?), so cannot really comment.

Thanks,
Daniel
  

> >>
> >> This might be one avenue if all kernel JIT contributors would be on board.
> >> Another option I'm wondering could be to extend the bpf() syscall in order
> >> to pass down a description of context and helper mappings e.g. via BTF and
> >> let everything go through the verifier in the kernel the usual way (I presume
> >> one goal might be that you want to assure that the generated BPF code passes
> >> the safety checks before running the prog), then have it JITed and extract
> >> the generated image in order to use it from user space. Kernel would have
> >> to make sure it never actually allows attaching this program in the kernel.
> >> Generated opcodes can already be retrieved today (see below). Such infra
> >> could potentially help bpf-gcc folks as well as they expressed desire to
> >> have some sort of a simulator for their gcc BPF test suite.. and it would
> >> allow for consistent behavior of the BPF runtime. Just a thought.
> >
> > This idea looks good. This can remove the verifier code also from DPDK.
>
> Right, definitely makes sense to have consolidation also on this one as well
> aside from the JITs, and pushing to the kernel and receiving back the JITed
> image seems quite nice and would be generic to open up many other use cases
> outside of networking. From app pov, it's just an implementation detail where
> to get that BPF opcode image from anyway.
>
> >   A couple of downsides I can think of,
> >
> > # We may need to extend the kernel verifier to understand the user-space address
> > and its symbols for CALL and MEM access operations.
>
> Yep, that part would be needed, potentially BTF could be of help here as well
> for the description of the user space runtime environment like context, helpers
> etc, so that JIT knows how to handle this.

Though, We can not conclude the following non-technical aspects in
this forum like
 # Dual license Linux Kernel BPF code as GPL-BSD as a separate library.
#  DPDK BPF support for FreeBSD and Windows OS, Treatment for Other OS?
# Need a different treatment for old Linux kernels.
This would call for immediate DPDK release to follow the existing semantics.

Yes, For the long term, Using the Kernel JITed EBPF program for
userspace will be helpful. At least DPDK can use it in the future.
A couple of other things to consider when someone does this
# https://github.com/iovisor/ubpf can also benefit from this.
# We need to think about how to support tail call in userspace
# It is possible to have burst mode support in userspace as an
improvement (dealing with 4 packets at the time with SIMD).
Dealing SIMD in kernel space will be an issue or dealing with such
improvement in general.
# Kernel verifier and dealing with address has a lot of security
requirements that may not apply for userspace, and therefore some of
the
optimization specific to userspace needs to consider some way
# Based on my understanding the Linux and DPDK JITed code, following
optimizations may need/have a different path

a) Userspace JIT has to deal with a 64bit address space. Kernel BPF
code can assume the Kernel virtual address space range and optimize.
b) I see, Kernel JIT always makes stack size as non zero even though
BPF applications are not using the stack. I am not sure why it
is that(Could be some security issue). There could be some
optimization not to push the stack pointer in the prologue if EBPF
program does
not use stack
c) In DPDK JIT compiler, In this first pass, we are checking the
actual registers really in use and based on that we are crafting
prologue and
epilogue at runtime to improve the performance.  It could be just
implementation detail, but not sure why Linux kernel is not doing such
optimization.

Konstantin is the author DPDK EBPF support, I just added arm64 JIT
support. Maybe he has more data for this direction.

>
> > # DPDK supports FreeBSD and Windows OS as well
>
> I'm not too familiar with the state on BPF for the latter two, but afaik FreeBSD
> at least had some effort to implement a BPF runtime into their kernel as well,
> so similar interface could be provided, but presumably as starting point vast
> majority of DPDK users are running Linux underneath anyway?
>
> > # Need a different treatment for old Linux kernels.
>
> Maybe, though I have little insight from DPDK angle here. Wrt BPF and kernel from
> what we see major cloud providers usually offer quite recent kernels as well as
> most mainstream distros that are run there, but again, environments where DPDK is
> typically deployed may differ (?), so cannot really comment.







>
> Thanks,
> Daniel
  

Hi everyone,

 
> > >>
> > >> This might be one avenue if all kernel JIT contributors would be on board.
> > >> Another option I'm wondering could be to extend the bpf() syscall in order
> > >> to pass down a description of context and helper mappings e.g. via BTF and
> > >> let everything go through the verifier in the kernel the usual way (I presume
> > >> one goal might be that you want to assure that the generated BPF code passes
> > >> the safety checks before running the prog), then have it JITed and extract
> > >> the generated image in order to use it from user space. Kernel would have
> > >> to make sure it never actually allows attaching this program in the kernel.
> > >> Generated opcodes can already be retrieved today (see below). Such infra
> > >> could potentially help bpf-gcc folks as well as they expressed desire to
> > >> have some sort of a simulator for their gcc BPF test suite.. and it would
> > >> allow for consistent behavior of the BPF runtime. Just a thought.
> > >
> > > This idea looks good. This can remove the verifier code also from DPDK.

Yes, from one side that idea looks very tempting,
As I understand in that case we wouldn't need to worry about licensing,
plus we getting JIT, verifier and might be even cBPF support for free...
As the downside, as Jerin also outlined below - no support for other OSes,
plus in future to get new feature users might need to upgrade to latest kernel.

Just exploring the alternate approach - if we put away for now this licensing hussle,
how difficult you think it would be to make current bpf kernel code sort of
platform independent entity that could be build both as part of kernel and as
standalone user-space lib? 
Again would it be useful for current eBPF users to have an ability to build/run verifier/jit
in user-space too (might be easier to debug, faster prototyping, etc.),
or just extra pain for maintainers? 

Konstantin

> >
> > Right, definitely makes sense to have consolidation also on this one as well
> > aside from the JITs, and pushing to the kernel and receiving back the JITed
> > image seems quite nice and would be generic to open up many other use cases
> > outside of networking. From app pov, it's just an implementation detail where
> > to get that BPF opcode image from anyway.
> >
> > >   A couple of downsides I can think of,
> > >
> > > # We may need to extend the kernel verifier to understand the user-space address
> > > and its symbols for CALL and MEM access operations.
> >
> > Yep, that part would be needed, potentially BTF could be of help here as well
> > for the description of the user space runtime environment like context, helpers
> > etc, so that JIT knows how to handle this.
> 
> Though, We can not conclude the following non-technical aspects in
> this forum like
>  # Dual license Linux Kernel BPF code as GPL-BSD as a separate library.
> #  DPDK BPF support for FreeBSD and Windows OS, Treatment for Other OS?
> # Need a different treatment for old Linux kernels.
> This would call for immediate DPDK release to follow the existing semantics.
> 
> Yes, For the long term, Using the Kernel JITed EBPF program for
> userspace will be helpful. At least DPDK can use it in the future.
> A couple of other things to consider when someone does this
> # https://github.com/iovisor/ubpf can also benefit from this.
> # We need to think about how to support tail call in userspace
> # It is possible to have burst mode support in userspace as an
> improvement (dealing with 4 packets at the time with SIMD).
> Dealing SIMD in kernel space will be an issue or dealing with such
> improvement in general.
> # Kernel verifier and dealing with address has a lot of security
> requirements that may not apply for userspace, and therefore some of
> the
> optimization specific to userspace needs to consider some way
> # Based on my understanding the Linux and DPDK JITed code, following
> optimizations may need/have a different path
> 
> a) Userspace JIT has to deal with a 64bit address space. Kernel BPF
> code can assume the Kernel virtual address space range and optimize.
> b) I see, Kernel JIT always makes stack size as non zero even though
> BPF applications are not using the stack. I am not sure why it
> is that(Could be some security issue). There could be some
> optimization not to push the stack pointer in the prologue if EBPF
> program does
> not use stack
> c) In DPDK JIT compiler, In this first pass, we are checking the
> actual registers really in use and based on that we are crafting
> prologue and
> epilogue at runtime to improve the performance.  It could be just
> implementation detail, but not sure why Linux kernel is not doing such
> optimization.
> 
> Konstantin is the author DPDK EBPF support, I just added arm64 JIT
> support. Maybe he has more data for this direction.
> 
> >
> > > # DPDK supports FreeBSD and Windows OS as well
> >
> > I'm not too familiar with the state on BPF for the latter two, but afaik FreeBSD
> > at least had some effort to implement a BPF runtime into their kernel as well,
> > so similar interface could be provided, but presumably as starting point vast
> > majority of DPDK users are running Linux underneath anyway?
> >
> > > # Need a different treatment for old Linux kernels.
> >
> > Maybe, though I have little insight from DPDK angle here. Wrt BPF and kernel from
> > what we see major cloud providers usually offer quite recent kernels as well as
> > most mainstream distros that are run there, but again, environments where DPDK is
> > typically deployed may differ (?), so cannot really comment.
> 
> 
> 
> 
> 
> 
> 
> >
> > Thanks,
> > Daniel
  

04/10/2019 17:39, Jerin Jacob:
> On Thu, Oct 3, 2019 at 8:35 PM Ananyev, Konstantin
> <konstantin.ananyev@intel.com> wrote:
> >
> > Hi everyone,
> >
> > >
> > > On Thu, Oct 3, 2019 at 6:21 PM Thomas Monjalon <thomas@monjalon.net> wrote:
> > > >
> > > > 03/09/2019 12:59, jerinj@marvell.com:
> > > > > Added eBPF arm64 JIT support to improve the eBPF program performance
> > > > > on arm64.
> > > > >
> > > > >  lib/librte_bpf/bpf_jit_arm64.c         | 1451 ++++++++++++++++++++++++
> > > >
> > > > I am concerned about duplicating the BPF JIT effort in DPDK and Linux.
> > > > Could we try to pull the Linux JIT?
> > > > Is the license the only issue?
> > >
> > > That's one issue.
> > >
> > > >
> > > > After a quick discussion, it seems the Linux authors are OK to arrange
> > > > their JIT code for sharing with userspace projects.
> > >
> > > I did a clean room implementation considering some optimization for
> > > DPDK etc(Like if stack is not used then don't push stack etc)
> > > and wherever Linux can be improved, I have submitted the patch also to
> > > Linux as well.(Some more pending as well)
> > >
> > > https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e4a2f8373cd332
> > >
> > > And Linux has a framework for instruction generation for debugging
> > > etc. So We can not copy and paste the code
> > > from Linux as is.
> > >
> > > My view to keep a different code base optimize for DPDK use cases and
> > > library requirements(for example, tail call is not supported in DPDK).
> > > For arm64/x86 case the code is done so it is not worth sync with
> > > Linux. For new architecture, it can be if possible.
> > >
> > > Konstantin,
> > > Your thoughts?
> > >
> >
> > My thought would be that if we have JIT eBPF compiler already in DPDK
> > for one arch (x86) there is absolutely no reason why we shouldn't allow it for different arch (arm).
> > About having a common code-base with Linux eBPF JITs implementation -
> > I think it is a very good idea,
> > but I don’t' think it could be achieved without significant effort.
> > DPDK and Linux JIT code-generators differ quite a bit.
> > So my suggestion - let's go ahead and integrate Jerin patch into 19.11,
> > meanwhile start talking with linux guys how common JIT code-base could be achieved.
> 
> I agree with Konstantin here.
> 
> Thomas,
> 
> Just confirm the following:
> 
> While we continue to have 'advanced' discussion on avoiding code duplication etc
> and it will take a couple of months to converge(if at all it happens)
> 
> Just to be clear, I assume, you are OK to merge this code for 19.11(If
> no more technical comment on the patch).
> 
> I am only afraid of, our typical last-minute surprise pattern and
> followed by back and forth open ended discussions.
> 
> i.e
> 
> # Code submitted before the proposal window
> # Gets ACK from Maintainer
> # New non-technical concerns start just before RC1

I hope you are not against discussing the real good questions,
even if they come a month after the first submission.

I don't care merging such patch in 19.11,
but I would have preferred such questions were open
when introducing this new library (for x86).

About your urge of having this code merged,
please can you explain what is your usage?
  

On Mon, 7 Oct, 2019, 6:03 PM Thomas Monjalon, <thomas@monjalon.net> wrote:

> 04/10/2019 17:39, Jerin Jacob:
> > On Thu, Oct 3, 2019 at 8:35 PM Ananyev, Konstantin
> > <konstantin.ananyev@intel.com> wrote:
> > >
> > > Hi everyone,
> > >
> > > >
> > > > On Thu, Oct 3, 2019 at 6:21 PM Thomas Monjalon <thomas@monjalon.net>
> wrote:
> > > > >
> > > > > 03/09/2019 12:59, jerinj@marvell.com:
> > > > > > Added eBPF arm64 JIT support to improve the eBPF program
> performance
> > > > > > on arm64.
> > > > > >
> > > > > >  lib/librte_bpf/bpf_jit_arm64.c         | 1451
> ++++++++++++++++++++++++
> > > > >
> > > > > I am concerned about duplicating the BPF JIT effort in DPDK and
> Linux.
> > > > > Could we try to pull the Linux JIT?
> > > > > Is the license the only issue?
> > > >
> > > > That's one issue.
> > > >
> > > > >
> > > > > After a quick discussion, it seems the Linux authors are OK to
> arrange
> > > > > their JIT code for sharing with userspace projects.
> > > >
> > > > I did a clean room implementation considering some optimization for
> > > > DPDK etc(Like if stack is not used then don't push stack etc)
> > > > and wherever Linux can be improved, I have submitted the patch also
> to
> > > > Linux as well.(Some more pending as well)
> > > >
> > > >
> https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e4a2f8373cd332
> > > >
> > > > And Linux has a framework for instruction generation for debugging
> > > > etc. So We can not copy and paste the code
> > > > from Linux as is.
> > > >
> > > > My view to keep a different code base optimize for DPDK use cases and
> > > > library requirements(for example, tail call is not supported in
> DPDK).
> > > > For arm64/x86 case the code is done so it is not worth sync with
> > > > Linux. For new architecture, it can be if possible.
> > > >
> > > > Konstantin,
> > > > Your thoughts?
> > > >
> > >
> > > My thought would be that if we have JIT eBPF compiler already in DPDK
> > > for one arch (x86) there is absolutely no reason why we shouldn't
> allow it for different arch (arm).
> > > About having a common code-base with Linux eBPF JITs implementation -
> > > I think it is a very good idea,
> > > but I don’t' think it could be achieved without significant effort.
> > > DPDK and Linux JIT code-generators differ quite a bit.
> > > So my suggestion - let's go ahead and integrate Jerin patch into 19.11,
> > > meanwhile start talking with linux guys how common JIT code-base could
> be achieved.
> >
> > I agree with Konstantin here.
> >
> > Thomas,
> >
> > Just confirm the following:
> >
> > While we continue to have 'advanced' discussion on avoiding code
> duplication etc
> > and it will take a couple of months to converge(if at all it happens)
> >
> > Just to be clear, I assume, you are OK to merge this code for 19.11(If
> > no more technical comment on the patch).
> >
> > I am only afraid of, our typical last-minute surprise pattern and
> > followed by back and forth open ended discussions.
> >
> > i.e
> >
> > # Code submitted before the proposal window
> > # Gets ACK from Maintainer
> > # New non-technical concerns start just before RC1
>
> I hope you are not against discussing the real good questions,
> even if they come a month after the first submission.
>

I am not against discussing the technical data about the 'patch' and review
it. If there is a review with respect to content of the patch it is very
good, I am happy to address it. Stuff like I don't have any control (
changing the licence) etc, I have am not comfortable to take  in last
minute. I have already shared the eBPF ARM64 JIT support in roadmap a month
ago before implementing it. No question asked that time. Spend a almost
month to add support for it and It is not a simple C code. Now I am not
comfortable in asking the fundamental questions like why this eBPF it self
is required and code duplication  ( code was duplicated when x86 support
has been added) and therefore stall the patch at this point of time, where
this library and x86 support added a year back.


>
>
>
> I don't care merging such patch in 19.11,
> but I would have preferred such questions were open
> when introducing this new library (for x86).
>

Konstantin added enough data on ml this when this library gets added on
reply to different users.


> About your urge of having this code merged,
> please can you explain what is your usage?
>

As an ARM64 maintainter, I would  like to fix any disparity in terms of the
features with respect to x86 and I have been doing for last 3 years. If
some one using eBPF on x86, I want to make sure it run in similar
"performance" on arm64 on architecture perspective.



>
>
  

07/10/2019 15:00, Jerin Jacob:
> On Mon, 7 Oct, 2019, 6:03 PM Thomas Monjalon wrote:
> > 04/10/2019 17:39, Jerin Jacob:
> > > On Thu, Oct 3, 2019 at 8:35 PM Ananyev, Konstantin wrote:
> > > > > On Thu, Oct 3, 2019 at 6:21 PM Thomas Monjalon wrote:
> > > > > > 03/09/2019 12:59, jerinj@marvell.com:
> > > > > > > Added eBPF arm64 JIT support to improve the eBPF
> > > > > > > program performance on arm64.
> > > > > > >
> > > > > > >  lib/librte_bpf/bpf_jit_arm64.c
> > > > > > >  | 1451 ++++++++++++++++++++++++
> > > > > >
> > > > > > I am concerned about duplicating the BPF JIT effort in DPDK and Linux.
> > > > > > Could we try to pull the Linux JIT?
> > > > > > Is the license the only issue?
> > > > >
> > > > > That's one issue.
> > > > >
> > > > > > After a quick discussion, it seems the Linux authors are OK to
> > > > > > arrange their JIT code for sharing with userspace projects.
> > > > >
> > > > > I did a clean room implementation considering some optimization for
> > > > > DPDK etc(Like if stack is not used then don't push stack etc)
> > > > > and wherever Linux can be improved, I have submitted the patch also
> > > > > to Linux as well.(Some more pending as well)
> > > > > https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e4a2f8373cd332
> > > > >
> > > > > And Linux has a framework for instruction generation for debugging
> > > > > etc. So We can not copy and paste the code from Linux as is.
> > > > >
> > > > > My view to keep a different code base optimize for DPDK use cases and
> > > > > library requirements(for example, tail call is not supported in DPDK).
> > > > > For arm64/x86 case the code is done so it is not worth sync with
> > > > > Linux. For new architecture, it can be if possible.
> > > > >
> > > > > Konstantin,
> > > > > Your thoughts?
> > > > >
> > > >
> > > > My thought would be that if we have JIT eBPF compiler already in DPDK
> > > > for one arch (x86) there is absolutely no reason why we shouldn't
> > > > allow it for different arch (arm).
> > > > About having a common code-base with Linux eBPF JITs implementation -
> > > > I think it is a very good idea,
> > > > but I don’t' think it could be achieved without significant effort.
> > > > DPDK and Linux JIT code-generators differ quite a bit.
> > > > So my suggestion - let's go ahead and integrate Jerin patch into 19.11,
> > > > meanwhile start talking with linux guys how common JIT code-base could
> > > > be achieved.
> > >
> > > I agree with Konstantin here.
> > >
> > > Thomas,
> > >
> > > Just confirm the following:
> > >
> > > While we continue to have 'advanced' discussion on avoiding code
> > > duplication etc and it will take a couple of months to converge
> > > (if at all it happens)
> > > Just to be clear, I assume, you are OK to merge this code for 19.11
> > > (If no more technical comment on the patch).
> > >
> > > I am only afraid of, our typical last-minute surprise pattern and
> > > followed by back and forth open ended discussions.
> > >
> > > i.e
> > >
> > > # Code submitted before the proposal window
> > > # Gets ACK from Maintainer
> > > # New non-technical concerns start just before RC1
> >
> > I hope you are not against discussing the real good questions,
> > even if they come a month after the first submission.
> 
> I am not against discussing the technical data about the 'patch' and review
> it. If there is a review with respect to content of the patch it is very
> good, I am happy to address it. Stuff like I don't have any control (
> changing the licence) etc, I have am not comfortable to take  in last
> minute. I have already shared the eBPF ARM64 JIT support in roadmap a month
> ago before implementing it. No question asked that time. Spend a almost
> month to add support for it and It is not a simple C code. Now I am not
> comfortable in asking the fundamental questions like why this eBPF it self
> is required and code duplication  ( code was duplicated when x86 support
> has been added) and therefore stall the patch at this point of time, where
> this library and x86 support added a year back.

I really don't like this reaction.
First, I never said this discussion was blocking the patch.
Second, why am I the only one asking such obvious questions
as not duplicating work?

> > I don't care merging such patch in 19.11,
> > but I would have preferred such questions were open
> > when introducing this new library (for x86).

You Jerin and Konstantin should have answered these questions
a long time ago before starting such development.
Is it so hard to require a bit of thoughts before starting something new?

> Konstantin added enough data on ml this when this library gets added on
> reply to different users.

Really? which data?

> > About your urge of having this code merged,
> > please can you explain what is your usage?
> 
> As an ARM64 maintainter, I would  like to fix any disparity in terms of the
> features with respect to x86 and I have been doing for last 3 years. If
> some one using eBPF on x86, I want to make sure it run in similar
> "performance" on arm64 on architecture perspective.

So we are debating about a library which is probably not used by anybody.
That's not how I plan to spend my time on DPDK.

Sorry Jerin, I really like working with you,
but I think you forward too much pressure here,
instead of quietly discussing the future of DPDK.

Please forget the deadline (we will agree on merging anyway)
and let's restart from the beginning by answering simple questions:
- what are the use cases of BPF in DPDK?
- how much we'll benefit from sharing code with Linux?
- what can we lose in a single JIT implementation?
  

On Mon, 7 Oct, 2019, 11:35 PM Thomas Monjalon, <thomas@monjalon.net> wrote:

> 07/10/2019 15:00, Jerin Jacob:
> > On Mon, 7 Oct, 2019, 6:03 PM Thomas Monjalon wrote:
> > > 04/10/2019 17:39, Jerin Jacob:
> > > > On Thu, Oct 3, 2019 at 8:35 PM Ananyev, Konstantin wrote:
> > > > > > On Thu, Oct 3, 2019 at 6:21 PM Thomas Monjalon wrote:
> > > > > > > 03/09/2019 12:59, jerinj@marvell.com:
> > > > > > > > Added eBPF arm64 JIT support to improve the eBPF
> > > > > > > > program performance on arm64.
> > > > > > > >
> > > > > > > >  lib/librte_bpf/bpf_jit_arm64.c
> > > > > > > >  | 1451 ++++++++++++++++++++++++
> > > > > > >
> > > > > > > I am concerned about duplicating the BPF JIT effort in DPDK
> and Linux.
> > > > > > > Could we try to pull the Linux JIT?
> > > > > > > Is the license the only issue?
> > > > > >
> > > > > > That's one issue.
> > > > > >
> > > > > > > After a quick discussion, it seems the Linux authors are OK to
> > > > > > > arrange their JIT code for sharing with userspace projects.
> > > > > >
> > > > > > I did a clean room implementation considering some optimization
> for
> > > > > > DPDK etc(Like if stack is not used then don't push stack etc)
> > > > > > and wherever Linux can be improved, I have submitted the patch
> also
> > > > > > to Linux as well.(Some more pending as well)
> > > > > >
> https://github.com/torvalds/linux/commit/504792e07a44844f24e9d79913e4a2f8373cd332
> > > > > >
> > > > > > And Linux has a framework for instruction generation for
> debugging
> > > > > > etc. So We can not copy and paste the code from Linux as is.
> > > > > >
> > > > > > My view to keep a different code base optimize for DPDK use
> cases and
> > > > > > library requirements(for example, tail call is not supported in
> DPDK).
> > > > > > For arm64/x86 case the code is done so it is not worth sync with
> > > > > > Linux. For new architecture, it can be if possible.
> > > > > >
> > > > > > Konstantin,
> > > > > > Your thoughts?
> > > > > >
> > > > >
> > > > > My thought would be that if we have JIT eBPF compiler already in
> DPDK
> > > > > for one arch (x86) there is absolutely no reason why we shouldn't
> > > > > allow it for different arch (arm).
> > > > > About having a common code-base with Linux eBPF JITs
> implementation -
> > > > > I think it is a very good idea,
> > > > > but I don’t' think it could be achieved without significant effort.
> > > > > DPDK and Linux JIT code-generators differ quite a bit.
> > > > > So my suggestion - let's go ahead and integrate Jerin patch into
> 19.11,
> > > > > meanwhile start talking with linux guys how common JIT code-base
> could
> > > > > be achieved.
> > > >
> > > > I agree with Konstantin here.
> > > >
> > > > Thomas,
> > > >
> > > > Just confirm the following:
> > > >
> > > > While we continue to have 'advanced' discussion on avoiding code
> > > > duplication etc and it will take a couple of months to converge
> > > > (if at all it happens)
> > > > Just to be clear, I assume, you are OK to merge this code for 19.11
> > > > (If no more technical comment on the patch).
> > > >
> > > > I am only afraid of, our typical last-minute surprise pattern and
> > > > followed by back and forth open ended discussions.
> > > >
> > > > i.e
> > > >
> > > > # Code submitted before the proposal window
> > > > # Gets ACK from Maintainer
> > > > # New non-technical concerns start just before RC1
> > >
> > > I hope you are not against discussing the real good questions,
> > > even if they come a month after the first submission.
> >
> > I am not against discussing the technical data about the 'patch' and
> review
> > it. If there is a review with respect to content of the patch it is very
> > good, I am happy to address it. Stuff like I don't have any control (
> > changing the licence) etc, I have am not comfortable to take  in last
> > minute. I have already shared the eBPF ARM64 JIT support in roadmap a
> month
> > ago before implementing it. No question asked that time. Spend a almost
> > month to add support for it and It is not a simple C code. Now I am not
> > comfortable in asking the fundamental questions like why this eBPF it
> self
> > is required and code duplication  ( code was duplicated when x86 support
> > has been added) and therefore stall the patch at this point of time,
> where
> > this library and x86 support added a year back.
>
> I really don't like this reaction.
>

If it hurts you in some way then I am sorry about that.

First, I never said this discussion was blocking the patch.
>

You said you have concern with this patch. Sorry,
I am not sure how to interpret that and if I don't jump in it will be
stalled for sure. That's my experience. Sorry if you dis agree.


Second, why am I the only one asking such obvious questions
> as not duplicating work?
>

Some things it does not converge at all. Especially relicecing some code
from linux. There are a lot developers(even me) are involved in that code
base. Why would everyone agree? The list would include a recent RISCV JIT
contributer from gmail.com as example.

Duplication the semantics some times gives the morecontrol. We already did
that for rte_flow, rcu etc. I have mentioned the performance reason as well
for JIT in the other thread.



>
> > > I don't care merging such patch in 19.11,
> > > but I would have preferred such questions were open
> > > when introducing this new library (for x86).
>
> You Jerin and Konstantin should have answered these questions
> a long time ago before starting such development.
> Is it so hard to require a bit of thoughts before starting something new?
>

For me, I don't see any better approach to have user space eBPF to support
all OS in DPDK.


> > Konstantin added enough data on ml this when this library gets added on
> > reply to different users.
>
> Really? which data?
>

I am talking about the discussion with
niterome developer.I don't have exact email thread, probably Konstantin may
have


>
> > > About your urge of having this code merged,
> > > please can you explain what is your usage?
> >
> > As an ARM64 maintainter, I would  like to fix any disparity in terms of
> the
> > features with respect to x86 and I have been doing for last 3 years. If
> > some one using eBPF on x86, I want to make sure it run in similar
> > "performance" on arm64 on architecture perspective.
>
> So we are debating about a library which is probably not used by anybody.
> That's not how I plan to spend my time on DPDK.
>

How do anyone know that the library is not used by anyone in community if
it is part of dpdk.org and a customer asked does arm64 has JIT support too.

If something needs to be dynamically controlled then eBPF can be used,
couple of use cases

# packet filtering
# debugging
# function call tracing
# There are some Lua JIT based dataplane implementations. Which can be
replaced with eBPF with JIT.




>
>
> Sorry Jerin, I really like working with you,
>

Mee too.

but I think you forward too much pressure here,
> instead of quietly discussing the future of DPDK.
>
> Please forget the deadline (we will agree on merging anyway)
>

Ok.

and let's restart from the beginning by answering simple questions:
> - what are the use cases of BPF in DPDK?
>

I meantioned what I know,

- how much we'll benefit from sharing code with Linux?
>

I have mentioned some of the performance constraint in the other thread.
Moreover I don't believe it is not easy task for Linux eBPF to run as
userspace and I not sure who is going to do that

- what can we lose in a single JIT implementation?
>

Sorry, I didn't understood this question?




>
>
  

07/10/2019 21:29, Jerin Jacob:
> On Mon, 7 Oct, 2019, 11:35 PM Thomas Monjalon, <thomas@monjalon.net> wrote:
[...] 
> let's restart from the beginning by answering simple questions:
> > - what are the use cases of BPF in DPDK?
> 
> If something needs to be dynamically controlled then eBPF can be used,
> couple of use cases
> 
> # packet filtering
> # debugging
> # function call tracing
> # There are some Lua JIT based dataplane implementations. Which can be
> replaced with eBPF with JIT.
> 
> - how much we'll benefit from sharing code with Linux?
> 
> I have mentioned some of the performance constraint in the other thread.
> Moreover I don't believe it is not easy task for Linux eBPF to run as
> userspace and I not sure who is going to do that

I was asking the benefits here:
- sharing optimizations in both projects
- get verifier support
What else?

> - what can we lose in a single JIT implementation?
> 
> Sorry, I didn't understood this question?

I mean what are the drawbacks of using a Linux implementation?
How performance constraints are differents, etc?

Note: as a lot of people, I don't really know BPF,
so these are real questions to help understanding the challenge.
  

On Tue, 8 Oct, 2019, 1:45 AM Thomas Monjalon, <thomas@monjalon.net> wrote:

> 07/10/2019 21:29, Jerin Jacob:
> > On Mon, 7 Oct, 2019, 11:35 PM Thomas Monjalon, <thomas@monjalon.net>
> wrote:
> [...]
> > let's restart from the beginning by answering simple questions:
> > > - what are the use cases of BPF in DPDK?
> >
> > If something needs to be dynamically controlled then eBPF can be used,
> > couple of use cases
> >
> > # packet filtering
> > # debugging
> > # function call tracing
> > # There are some Lua JIT based dataplane implementations. Which can be
> > replaced with eBPF with JIT.
> >
> > - how much we'll benefit from sharing code with Linux?
> >
> > I have mentioned some of the performance constraint in the other thread.
> > Moreover I don't believe it is not easy task for Linux eBPF to run as
> > userspace and I not sure who is going to do that
>
> I was asking the benefits here:
> - sharing optimizations in both projects
>

Yes. But even if it is different code base it is possible to share the
optimization.

- get verifier support
>

Verifier support already available in the library.

What else?
>

I see only avoiding code duplication and getting new feature like cBPF.


> > - what can we lose in a single JIT implementation?
> >
> > Sorry, I didn't understood this question?
>
> I mean what are the drawbacks of using a Linux implementation?
> How performance constraints are differents, etc?
>

Mention the details in the below thread. Waiting for feedback from Kernel
maintainer.

http://mails.dpdk.org/archives/dev/2019-October/146004.html

http://mails.dpdk.org/archives/dev/2019-October/146063.html


>
>
> Note: as a lot of people, I don't really know BPF,
> so these are real questions to help understanding the challenge.


>
>
  

24/09/2019 19:03, Ananyev, Konstantin:
> > Jerin Jacob (8):
> >   bpf/arm64: add build infrastructure
> >   bpf/arm64: add prologue and epilogue
> >   bpf/arm64: add basic arithmetic operations
> >   bpf/arm64: add logical operations
> >   bpf/arm64: add byte swap operations
> >   bpf/arm64: add load and store operations
> >   bpf/arm64: add atomic-exchange-and-add operation
> >   bpf/arm64: add branch operation
> > 
> 
> Series Acked-by: Konstantin Ananyev <konstantin.ananyev@intel.com>

Applied

Please continue the discussions started with the Linux maintainers
in order to try reducing the code duplication between the two projects.
Thanks
  

On Sat, Oct 5, 2019 at 8:09 PM Jerin Jacob <jerinjacobk@gmail.com> wrote:
>
> > >>
> > >> This might be one avenue if all kernel JIT contributors would be on board.
> > >> Another option I'm wondering could be to extend the bpf() syscall in order
> > >> to pass down a description of context and helper mappings e.g. via BTF and
> > >> let everything go through the verifier in the kernel the usual way (I presume
> > >> one goal might be that you want to assure that the generated BPF code passes
> > >> the safety checks before running the prog), then have it JITed and extract
> > >> the generated image in order to use it from user space. Kernel would have
> > >> to make sure it never actually allows attaching this program in the kernel.
> > >> Generated opcodes can already be retrieved today (see below). Such infra
> > >> could potentially help bpf-gcc folks as well as they expressed desire to
> > >> have some sort of a simulator for their gcc BPF test suite.. and it would
> > >> allow for consistent behavior of the BPF runtime. Just a thought.
> > >
> > > This idea looks good. This can remove the verifier code also from DPDK.
> >
> > Right, definitely makes sense to have consolidation also on this one as well
> > aside from the JITs, and pushing to the kernel and receiving back the JITed
> > image seems quite nice and would be generic to open up many other use cases
> > outside of networking. From app pov, it's just an implementation detail where
> > to get that BPF opcode image from anyway.
> >
> > >   A couple of downsides I can think of,
> > >
> > > # We may need to extend the kernel verifier to understand the user-space address
> > > and its symbols for CALL and MEM access operations.
> >
> > Yep, that part would be needed, potentially BTF could be of help here as well
> > for the description of the user space runtime environment like context, helpers
> > etc, so that JIT knows how to handle this.
>
> Though, We can not conclude the following non-technical aspects in
> this forum like
>  # Dual license Linux Kernel BPF code as GPL-BSD as a separate library.
> #  DPDK BPF support for FreeBSD and Windows OS, Treatment for Other OS?
> # Need a different treatment for old Linux kernels.
> This would call for immediate DPDK release to follow the existing semantics.
>
> Yes, For the long term, Using the Kernel JITed EBPF program for
> userspace will be helpful. At least DPDK can use it in the future.
> A couple of other things to consider when someone does this
> # https://github.com/iovisor/ubpf can also benefit from this.
> # We need to think about how to support tail call in userspace
> # It is possible to have burst mode support in userspace as an
> improvement (dealing with 4 packets at the time with SIMD).
> Dealing SIMD in kernel space will be an issue or dealing with such
> improvement in general.
> # Kernel verifier and dealing with address has a lot of security
> requirements that may not apply for userspace, and therefore some of
> the
> optimization specific to userspace needs to consider some way
> # Based on my understanding the Linux and DPDK JITed code, following
> optimizations may need/have a different path
>
> a) Userspace JIT has to deal with a 64bit address space. Kernel BPF
> code can assume the Kernel virtual address space range and optimize.
> b) I see, Kernel JIT always makes stack size as non zero even though
> BPF applications are not using the stack. I am not sure why it
> is that(Could be some security issue). There could be some
> optimization not to push the stack pointer in the prologue if EBPF
> program does
> not use stack
> c) In DPDK JIT compiler, In this first pass, we are checking the
> actual registers really in use and based on that we are crafting
> prologue and
> epilogue at runtime to improve the performance.  It could be just
> implementation detail, but not sure why Linux kernel is not doing such
> optimization.


Just pinging back to see if anyone interested in the common library.
Answers to the above questions and mainly this below thead will make
us forward progress for common library if there is still interest in
collaboration.

http://mails.dpdk.org/archives/dev/2019-October/146063.html
  

Hi guys,
 
> On Fri, Oct 4, 2019 at 7:39 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
> >
> > On 10/4/19 12:53 PM, Thomas Monjalon wrote:
> > > 04/10/2019 11:54, Steve Capper:
> > >> I'd recommend also reaching out the BPF maintainers:
> > >> BPF JIT for ARM64
> > >> M:   Daniel Borkmann <daniel@iogearbox.net>
> > >> M:   Alexei Starovoitov <ast@kernel.org>
> > >> M:   Zi Shen Lim <zlim.lnx@gmail.com>
> > >> L:   netdev@vger.kernel.org
> > >> L:   bpf@vger.kernel.org
> > >> S:   Supported
> > >> F:   arch/arm64/net/
> > >>
> > >> As they will have much better knowledge of the state of play and will be
> > >> better able to advise.
> > >
> > > As far as I know Alexei and Daniel are OK with the idea.
> > > But better to let them reply here.
> > >
> > > I suggest we think about a way to package the kernel BPF JIT
> > > for userspace usage (not only DPDK) as a library.
> > > I don't understand why the DPDK JIT should be different
> > > or optimized differently.
> >
> > That would be great indeed as both projects would benefit from a shared
> > JIT instead of reimplementing everything twice. I never looked into DPDK
> > too much, but I presume the idea would be as well to take the LLVM (or
> > bpf-gcc) generated object file and load it into a BPF 'engine' that sits
> > in user space on top of DPDK? Presumably loader could be libbpf here as
> > well since it already knows how to parse the ELF, perform the relocations
> > etc. The only difference would be that you have a different context and
> > different helpers? Is that the goal eventually?
> >
> > > The only real issue I see is the need for a dual licensing BSD-GPL.
> >
> > This might be one avenue if all kernel JIT contributors would be on board.
> > Another option I'm wondering could be to extend the bpf() syscall in order
> > to pass down a description of context and helper mappings e.g. via BTF and
> > let everything go through the verifier in the kernel the usual way (I presume
> > one goal might be that you want to assure that the generated BPF code passes
> > the safety checks before running the prog), then have it JITed and extract
> > the generated image in order to use it from user space. Kernel would have
> > to make sure it never actually allows attaching this program in the kernel.
> > Generated opcodes can already be retrieved today (see below). Such infra
> > could potentially help bpf-gcc folks as well as they expressed desire to
> > have some sort of a simulator for their gcc BPF test suite.. and it would
> > allow for consistent behavior of the BPF runtime. Just a thought.
> 
> This idea looks good. This can remove the verifier code also from DPDK.
>  A couple of downsides I can think of,
> 
> # We may need to extend the kernel verifier to understand the user-space address
> and its symbols for CALL and MEM access operations.
> # DPDK supports FreeBSD and Windows OS as well
> # Need a different treatment for old Linux kernels.

Seems like discussion died out eventually.
Pinging to check is there still any interest on that subject
from kernel community.
Thanks
Konstantin