Message ID | 20220407214707.29730-1-ohilyard@iol.unh.edu (mailing list archive) |
---|---|
Headers |
Return-Path: <dev-bounces@dpdk.org> X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 95DA7A050C; Thu, 7 Apr 2022 23:47:29 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3473E4068B; Thu, 7 Apr 2022 23:47:29 +0200 (CEST) Received: from mail-oa1-f98.google.com (mail-oa1-f98.google.com [209.85.160.98]) by mails.dpdk.org (Postfix) with ESMTP id 7743040689 for <dev@dpdk.org>; Thu, 7 Apr 2022 23:47:28 +0200 (CEST) Received: by mail-oa1-f98.google.com with SMTP id 586e51a60fabf-df22f50e0cso7872337fac.3 for <dev@dpdk.org>; Thu, 07 Apr 2022 14:47:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iol.unh.edu; s=unh-iol; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=49Qsuo66QudCF9+O3yvnpNnqmRwjSC89Ze0SeGDuum8=; b=Hh/BBYr3BtXQtf3gndLDdyzZ+g96MwX39CPLEAGC/U/xN0HLe7DOfj8HGw23loNlPo jyry/1if/C7oNSvqMnhrSOf+Kvh8MCeMXn3TKqD5CLMtTkjs1sz9MqKF8sV8Znm5BlkK AkCX8lsRUxMZUjCAjxPPptazriUKN+Yskn/7s= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=49Qsuo66QudCF9+O3yvnpNnqmRwjSC89Ze0SeGDuum8=; b=jGZXgLGrMXzCRwKhRnpxCF7CH73iEAVXcjVKuQxA60mE5HZjIvEzev+8PnTVdF1+/s g8G0cSUbTYfkPMW1rJd2zj9iiHCkU1LGypKugkHN0DI+plcUuaW3qu9CxJLj0ENrmSJV nWi20l+ch/tFLdufdeAF/1ldZKP8VGNj+ShMbtOgfEc5Fzv7FyusiFS50X/YL8Tx7mlc j2Yx3xPmtCx7vFsWctBIgRiFfNgqjdkc+WOh16zwKYDD2CQ4GNkoHtU5PsY201BDQXpl yILqGbhTHPh9SG0xYrp69DVD0hTPU1cuyBJuXSM+0QRNpgNBj8xPhqrMmFXAquQGOjKR Z7TQ== X-Gm-Message-State: AOAM530xlJvOlKJsZt04dBF5qhFnD87P99uApoHVY5B+7TRTt3wgBHAz 8489Q16k39cCDOVk7iG0BmmD655IT7JJLHyINvFRs0mhuBluLUYnj7xuRru1PW2IzA== X-Google-Smtp-Source: ABdhPJwH5ZuhDs+bbAGpMu1cywnAQwLkSyw3ClHylnzUPCGLMB3UUfnPDW+qWLlvPGojWAFL+HA0rznalteT X-Received: by 2002:a05:6870:d20a:b0:e2:182d:47c7 with SMTP id g10-20020a056870d20a00b000e2182d47c7mr7466299oac.244.1649368047769; Thu, 07 Apr 2022 14:47:27 -0700 (PDT) Received: from postal.iol.unh.edu (postal.iol.unh.edu. [2606:4100:3880:1234::84]) by smtp-relay.gmail.com with ESMTPS id g3-20020a056870e1c300b000da371afbe1sm1379586oab.41.2022.04.07.14.47.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 07 Apr 2022 14:47:27 -0700 (PDT) X-Relaying-Domain: iol.unh.edu Received: from iol.unh.edu (unknown [IPv6:2606:4100:3880:1257::105d]) by postal.iol.unh.edu (Postfix) with ESMTP id 310CB6051838; Thu, 7 Apr 2022 17:47:27 -0400 (EDT) From: ohilyard@iol.unh.edu To: dev@dpdk.org Cc: Honnappa.Nagarahalli@arm.com, thomas@monjalon.net, Owen Hilyard <ohilyard@iol.unh.edu> Subject: [PATCH v1 0/4] [RFC] Testpmd RPC API Date: Thu, 7 Apr 2022 17:47:04 -0400 Message-Id: <20220407214707.29730-1-ohilyard@iol.unh.edu> X-Mailer: git-send-email 2.30.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions <dev.dpdk.org> List-Unsubscribe: <https://mails.dpdk.org/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://mails.dpdk.org/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <https://mails.dpdk.org/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> Errors-To: dev-bounces@dpdk.org |
Series | Testpmd RPC API | |
Message
Owen Hilyard
April 7, 2022, 9:47 p.m. UTC
From: Owen Hilyard <ohilyard@iol.unh.edu>
Currently, DTS uses Testpmd for most of its testing. This has been successful in reducing the need to create more test apps, but it has a few drawbacks. First, if some part of DPDK is not exposed via Testpmd or one of the example applications, for the purposes of DTS it is not testable. This is a situation I’d like to avoid. However, adding new functionality to Testpmd is labor-intensive. Testpmd currently uses a hand-written LL(1) parser (https://en.wikipedia.org/wiki/LL_parser) to parse command line options. This makes adding new functionality difficult since the parser is stored as a series of several thousand line long lookup tables. To look at it another way, 64% of the 52238 lines in Testpmd are related to command line input in some way. The command line interface of testpmd also presents several challenges for the underlying implementation, since it requires that everything a user might want to reference is identified via something that is reasonable to ask a user to type. As of right now, this is handled via either strings or integers. This can be handled by creating a global registry for objects, but it is still extra work that I think can be avoided. In addition, this leads to more places where things can go wrong.
This is what DTS running a single command in testpmd looks like right now:
https://drive.google.com/file/d/1hvTcjfVdh8-I3CUNoq6bx82EuNQSK6qW/view?usp=sharing
This approach has a number of disadvantages. First, it requires assembling all commands as strings inside of the test suite and sending them through a full round trip of SSH. This means that any non-trivial command, such as creating an RTE flow, will involve a lot of string templating. This normally wouldn’t be a big issue, except that some of the test suites are designed to hundreds of commands over the course of a test, paying the cost of an SSH round trip for each. Once Testpmd has the commands, it will then call the appropriate functions inside of DPDK, and then print out all of the state to standard out. All of this is sent back to DTS, where the author of the test case then needs to handle all possible outputs of Trex, often by either declaring the presence of a single word or short phrase in the output as meaning success or failure. In my opinion, this is something that is perfectly fine for humans to interact with, but it causes a lot of issues with automation due to its inherent inflexibility and the less-than-ideal methods of information transfer. This is why I am proposing the creation of an automation-oriented pmd, with a focus on exposing as much.
https://drive.google.com/file/d/1wj4-RnFPVERCzM8b68VJswAOEI9cg-X8/view?usp=sharing
That diagram is a high-level overview of the design, which explicitly excludes implementation details. However, it already has some benefits. First, making DPDK do something is a normal method call, instead of needing to format things into a string. This provides a much better interface for people working in both DTS and DPDK. Second, the ability to return structured data means that there won’t be parsers on both sides of communication anymore. Structured data also allows much more verbosity, since it is no longer an interface designed for humans. If a test case author needs to return the bytes of every received packet back to DTS for comparison with the expected value, they can. If you need to return a pointer for DTS to use later, that becomes reasonable. Simply moving to shuffling structured data around and using RPC already provides a lot of benefits.
The next obvious question would be what to use for the implementation. The initial attempt was made using Python on both sides and the standard library xmlrpc module. The RPC aspect of this approach worked very well, with the ability to send arbitrary python objects back and forth between DTS and app. However, having Python interacting with DPDK has a few issues. First, DPDK is generally very multi-threaded and the most common implementation of Python, CPython, does not have concurrency. It has something known as the global interpretr lock, which is a global mutex. This makes it very difficult to interact with blocking, multi-threaded code. The other issue is that I was not able to find a binding generator that I feel would be sufficient for DPDK. Many generators assumed sizeof(int) == 4 or had other portability issues such as assuming GCC or Clang as a C compiler. Others focused on some subset of C, meaning they would throw errors on alignment annotations.
Given this, I decided to look for cross-language RPC libraries. Although libraries exist for performing xmlrpc in C, they generally appeared quite difficult to use and required a lot of manual work. The next best option was gRPC. gRPC allows using a simple language, protobuf, with a language extension for rpc. It provides code generation to make it easy to use multiple languages together, since it was developed to make polyglot microservice interaction easier. The only drawback is that it considers C++ good enough for C support. In this case, I was able to easily integrate DPDK with C++, so that isn’t much of a concern. I used C++17 in the attached patches, but the minimum requirements are C++11. If there is concern about modern C++ causing too much mental overhead, a “C with classes” subset of C++ could easily be used. I also added an on-by-default option to use a C++ compiler, allowing anyone who does not have a C++ compiler available to them to turn off everything that uses C++. This disables the application written for this RFC.
One of the major benefits of gRPC is the asynchronous API. This allows streaming data on both sides of an RPC call. This allows streaming logs back to DTS, streaming large amounts of data from low-memory systems back to DTS for processing, and would allow DTS to easily feed DPDK data, ending the test quickly on a failure. Currently, due to the overhead of sending data to Testpmd, it is common to just send all of the commands over and run everything since that will be much faster when the test passes, but it can cost a lot of time in the event of a failure. There are also optional security features for requiring authentication before allowing code execution. I think that a discussion on whether using them for DTS is necessary is warranted, although I personally think that it’s not worth the effort given the type of environment this sample application is expected to run in.
For this RFC, I ported test-acl because it was mostly self-contained and was something I could run on my laptop. It should be fairly easy to see how you would expand this proof of concept to cover more of DPDK, and I think that most of the functions currently used in testpmd could be ported over to this approach, saving a lot of development time. However, I would like to see some more interest before I take on a task like that. This will require a lot of work on the DTS side to implement, but it will make it much easier to add new features to DTS.
Owen Hilyard (4):
app/test-pmd-api: Add C++ Compiler
app/test-pmd-api: Add POC with gRPC deps
app/test-pmd-api: Add protobuf file
app/test-pmd-api: Implementation files for the API
app/meson.build | 17 +
app/test-pmd-api/api.proto | 12 +
app/test-pmd-api/api_impl.cc | 1160 ++++++++++++++++++++++++++++++++++
app/test-pmd-api/api_impl.h | 10 +
app/test-pmd-api/main.c | 11 +
app/test-pmd-api/meson.build | 96 +++
meson.build | 3 +
meson_options.txt | 2 +
8 files changed, 1311 insertions(+)
create mode 100644 app/test-pmd-api/api.proto
create mode 100644 app/test-pmd-api/api_impl.cc
create mode 100644 app/test-pmd-api/api_impl.h
create mode 100644 app/test-pmd-api/main.c
create mode 100644 app/test-pmd-api/meson.build
Comments
On Fri, Apr 8, 2022 at 3:17 AM <ohilyard@iol.unh.edu> wrote: > > From: Owen Hilyard <ohilyard@iol.unh.edu> > > Currently, DTS uses Testpmd for most of its testing. This has been successful in reducing the need to create more test apps, but it has a few drawbacks. First, if some part of DPDK is not exposed via Testpmd or one of the example applications, for the purposes of DTS it is not testable. This is a situation I’d like to avoid. However, adding new functionality to Testpmd is labor-intensive. Testpmd currently uses a hand-written LL(1) parser (https://en.wikipedia.org/wiki/LL_parser) to parse command line options. This makes adding new functionality difficult since the parser is stored as a series of several thousand line long lookup tables. To look at it another way, 64% of the 52238 lines in Testpmd are related to command line input in some way. The command line interface of testpmd also presents several challenges for the underlying implementation, since it requires that everything a user might want to reference is identified via something that is reasonable to ask a user to type. As of right now, this is handled via either strings or integers. This can be handled by creating a global registry for objects, but it is still extra work that I think can be avoided. In addition, this leads to more places where things can go wrong. > > This is what DTS running a single command in testpmd looks like right now: > https://drive.google.com/file/d/1hvTcjfVdh8-I3CUNoq6bx82EuNQSK6qW/view?usp=sharing > > This approach has a number of disadvantages. First, it requires assembling all commands as strings inside of the test suite and sending them through a full round trip of SSH. This means that any non-trivial command, such as creating an RTE flow, will involve a lot of string templating. This normally wouldn’t be a big issue, except that some of the test suites are designed to hundreds of commands over the course of a test, paying the cost of an SSH round trip for each. Once Testpmd has the commands, it will then call the appropriate functions inside of DPDK, and then print out all of the state to standard out. All of this is sent back to DTS, where the author of the test case then needs to handle all possible outputs of Trex, often by either declaring the presence of a single word or short phrase in the output as meaning success or failure. In my opinion, this is something that is perfectly fine for humans to interact with, but it causes a lot of issues with automation due to its inherent inflexibility and the less-than-ideal methods of information transfer. This is why I am proposing the creation of an automation-oriented pmd, with a focus on exposing as much. > > https://drive.google.com/file/d/1wj4-RnFPVERCzM8b68VJswAOEI9cg-X8/view?usp=sharing > > That diagram is a high-level overview of the design, which explicitly excludes implementation details. However, it already has some benefits. First, making DPDK do something is a normal method call, instead of needing to format things into a string. This provides a much better interface for people working in both DTS and DPDK. Second, the ability to return structured data means that there won’t be parsers on both sides of communication anymore. Structured data also allows much more verbosity, since it is no longer an interface designed for humans. If a test case author needs to return the bytes of every received packet back to DTS for comparison with the expected value, they can. If you need to return a pointer for DTS to use later, that becomes reasonable. Simply moving to shuffling structured data around and using RPC already provides a lot of benefits. > The next obvious question would be what to use for the implementation. The initial attempt was made using Python on both sides and the standard library xmlrpc module. The RPC aspect of this approach worked very well, with the ability to send arbitrary python objects back and forth between DTS and app. However, having Python interacting with DPDK has a few issues. First, DPDK is generally very multi-threaded and the most common implementation of Python, CPython, does not have concurrency. It has something known as the global interpretr lock, which is a global mutex. This makes it very difficult to interact with blocking, multi-threaded code. The other issue is that I was not able to find a binding generator that I feel would be sufficient for DPDK. Many generators assumed sizeof(int) == 4 or had other portability issues such as assuming GCC or Clang as a C compiler. Others focused on some subset of C, meaning they would throw errors on alignment annotations. > Given this, I decided to look for cross-language RPC libraries. Although libraries exist for performing xmlrpc in C, they generally appeared quite difficult to use and required a lot of manual work. The next best option was gRPC. gRPC allows using a simple language, protobuf, with a language extension for rpc. It provides code generation to make it easy to use multiple languages together, since it was developed to make polyglot microservice interaction easier. The only drawback is that it considers C++ good enough for C support. In this case, I was able to easily integrate DPDK with C++, so that isn’t much of a concern. I used C++17 in the attached patches, but the minimum requirements are C++11. If there is concern about modern C++ causing too much mental overhead, a “C with classes” subset of C++ could easily be used. I also added an on-by-default option to use a C++ compiler, allowing anyone who does not have a C++ compiler available to them to turn off everything that uses C++. This disables the application written for this RFC. > One of the major benefits of gRPC is the asynchronous API. This allows streaming data on both sides of an RPC call. This allows streaming logs back to DTS, streaming large amounts of data from low-memory systems back to DTS for processing, and would allow DTS to easily feed DPDK data, ending the test quickly on a failure. Currently, due to the overhead of sending data to Testpmd, it is common to just send all of the commands over and run everything since that will be much faster when the test passes, but it can cost a lot of time in the event of a failure. There are also optional security features for requiring authentication before allowing code execution. I think that a discussion on whether using them for DTS is necessary is warranted, although I personally think that it’s not worth the effort given the type of environment this sample application is expected to run in. > For this RFC, I ported test-acl because it was mostly self-contained and was something I could run on my laptop. It should be fairly easy to see how you would expand this proof of concept to cover more of DPDK, and I think that most of the functions currently used in testpmd could be ported over to this approach, saving a lot of development time. However, I would like to see some more interest before I take on a task like that. This will require a lot of work on the DTS side to implement, but it will make it much easier to add new features to DTS. Thanks, Owen for POC. In my view, May using this scheme is probably over-engineered. The reason for thinking so is -Now that, Test code is also part of DPDK, Exposing as services may not be required. -Now in DPDK, we have two types of existing test cases to verify the API -- Noninteractive - These test cases, can simply run over ssh with bash invocation and return the test from a remote PC, -- Interactive - Testpmd one, I believe, Feeding stdin programmatically would suffice to test all the combinations. -We need to add all test cases in this model and we need to maintain two sets of programs.(Traditional tests and gRPC model-based tests). Just my 2c. > > Owen Hilyard (4): > app/test-pmd-api: Add C++ Compiler > app/test-pmd-api: Add POC with gRPC deps > app/test-pmd-api: Add protobuf file > app/test-pmd-api: Implementation files for the API > > app/meson.build | 17 + > app/test-pmd-api/api.proto | 12 + > app/test-pmd-api/api_impl.cc | 1160 ++++++++++++++++++++++++++++++++++ > app/test-pmd-api/api_impl.h | 10 + > app/test-pmd-api/main.c | 11 + > app/test-pmd-api/meson.build | 96 +++ > meson.build | 3 + > meson_options.txt | 2 + > 8 files changed, 1311 insertions(+) > create mode 100644 app/test-pmd-api/api.proto > create mode 100644 app/test-pmd-api/api_impl.cc > create mode 100644 app/test-pmd-api/api_impl.h > create mode 100644 app/test-pmd-api/main.c > create mode 100644 app/test-pmd-api/meson.build > > -- > 2.30.2 >
> > scheme is probably over-engineered I tried my hardest to keep this as simple as possible. The requirements imposed by DTS being a distributed system in Python restricted what I could do a lot. Needing to be compatible with DPDK's license also got rid of a lot of options. Binding generators are made for simple projects, and DPDK is not a simple project. There were some other options related to choice in the RPC framework, but very few RPC protocols seem to work well with C and be usable from Python, which is why I ended up using C++ with gRPC. Most of the code in api_impl.cc is taken from /app/test-acl/main.c, and the new part is mostly the C++ class at the bottom. Overall, this proposal comes out to ~100 lines of new C++, 9 lines of C, 12 lines of gRPC Protobuf and 100 lines of Meson. gRPC may be able to do a lot more than I need it to for the proof of concept, but many of the features that are not used, like bi-directional streaming, become very useful in writing more complicated tests. Overall, this solution is probably more capable than we need it to be, but I think that those extra capabilities don't come at a very large cost. > Now that, Test code is also part of DPDK. > DTS is pure python. I tried to use FFI to call directly into DPDK from Python and then use xmlrpc from the python standard library. As mentioned in the writeup, I couldn't find a binding generator that would properly handle DPDK's allocators, which made it so that anything passed to DPDK using python was allocated using the system malloc. I don't think it is wise to attempt to programmatically re-write the generated code to allow for custom allocators. The original reason for needing to have DTS and DPDK in the same repository was so that tests could be committed and run alongside the feature patch. Interactive - Testpmd one, I believe, Feeding stdin programmatically would > suffice to test all the combinations. > One of the issues this is trying to address is that human-readable strings are a poor way to pass complex information between two programs. DTS is a distributed system, and it can have up to 3 physical servers involved in any given test. This means that it's not stdin via a pipe, it's an entire SSH session. This adds a noticeable amount of overhead when trying to send and verify the result of sending 1,000+ packets, since the lack of structured output means each packet must be checked before the next can be sent. This might be solvable by adding a structured output mode to testpmd, but that would involve committing to writing output twice for every function in testpmd forever. We need to add all test cases in this model and we need to maintain two > sets of programs.(Traditional tests and gRPC model-based tests). > Assuming by traditional tests you mean the unit tests run by Meson, I would argue that we are already maintaining 2 kinds of tests. The unit tests, and the python-based DTS tests. My intention is to create a thin wrapper around DPDK that would be exposed via gRPC, like you see here, and use that as midware. Then, we would have two front-ends. Testpmd, which takes text and then calls midware as it does now, and the gRPC frontend, which parses messages from the RPC server and runs the midware. This would enable testpmd to still be used to sanity check a DPDK installation, but we would not need to continually expand Testpmd. The primary issue is that, right now, anything not included in Testpmd is not really testable by DTS. This includes portions of the RTE Flow API, which was part of my reason for proposing this. The RTE Flow API would, in my estimation, if fully implemented into Testpmd, probably add at least another 10,000 lines of code. As mentioned in my proposal, Testpmd already does more parsing and lexing than it does interaction with DPDK by line count. Also, since I am proposing making this a separate application, we would be able to gradually migrate the tests inside of DTS. This would have no effect on anything except for Testpmd, the new application and the addition of a flag to toggle the use of a C++ compiler. I'm not sure exactly what you mean by gRPC model-based tests. gRPC uses classes to model services, but for this usecase we are essentially using it to transfer function arguments across the internet and then pass the return value back. Any RPC framework would function similarly if I ignored the restrictions of which languages to use, and the choice is not important to how tests are conducted. Put another way, how you write a test for DTS will not change much if you are using this or testpmd, it's just how you transfer data and get it back that I want to change.
On Mon, Apr 11, 2022 at 11:19 PM Owen Hilyard <ohilyard@iol.unh.edu> wrote: >> >> scheme is probably over-engineered > > > I tried my hardest to keep this as simple as possible. The requirements imposed by DTS being a distributed system in Python restricted what I could do a lot. Needing to be compatible with DPDK's license also got rid of a lot of options. Binding generators are made for simple projects, and DPDK is not a simple project. There were some other options related to choice in the RPC framework, but very few RPC protocols seem to work well with C and be usable from Python, which is why I ended up using C++ with gRPC. Most of the code in api_impl.cc is taken from /app/test-acl/main.c, and the new part is mostly the C++ class at the bottom. Overall, this proposal comes out to ~100 lines of new C++, 9 lines of C, 12 lines of gRPC Protobuf and 100 lines of Meson. gRPC may be able to do a lot more than I need it to for the proof of concept, but many of the features that are not used, like bi-directional streaming, become very useful in writing more complicated tests. Overall, this solution is probably more capable than we need it to be, but I think that those extra capabilities don't come at a very large cost. Now it is clear, I was carried away with the POC test application and I was not knowing existing DTS tests are based on python Is below a fair summary? 1) DPDK has interactive test cases and no interactive test cases. For The interactive test case like testpmd, I agree that we can enable RPC service via gRPC server in C++ as and client in Python, and something along the lines of exposing the existing test-pmd command line function as service to avoid command line parsing and reuse the existing python test suite. If so, I think, gRPC service would be along with existing testpmd functions, like start_packet_forwarding(). Also, We don't need to rewrite the existing testpmd, Instead, RPC service, we can add in existing app/test-pmd/ and hook to existing core testpmd functions to bypass the command-line parsing in C and control from python client as needed as service. Also, I agree that pulling in gRPC C++ server boilerplate and hooking to C functions is a good idea as it is the best C-based RPC scheme available today. 2)I think, DPDK has only one interactive test case which is testpmd, Remaining test cases are non-interactive, non-interactive test cases can simply run over ssh with passwordless login. Right? Do we need gRPC for that? Will the following scheme suffice? If not, How you are planning to do noninteractive test cases? i.e a)Copy test to target b) result=`ssh username@IP /path/to/testapp/in/target` I think, key should be leveraging existing test cases as much as possible and make easy for developers to add new test cases. >> >> Now that, Test code is also part of DPDK. > > > DTS is pure python. I tried to use FFI to call directly into DPDK from Python and then use xmlrpc from the python standard library. As mentioned in the writeup, I couldn't find a binding generator that would properly handle DPDK's allocators, which made it so that anything passed to DPDK using python was allocated using the system malloc. I don't think it is wise to attempt to programmatically re-write the generated code to allow for custom allocators. The original reason for needing to have DTS and DPDK in the same repository was so that tests could be committed and run alongside the feature patch. > >> Interactive - Testpmd one, I believe, Feeding stdin programmatically would suffice to test all the combinations. > > > One of the issues this is trying to address is that human-readable strings are a poor way to pass complex information between two programs. DTS is a distributed system, and it can have up to 3 physical servers involved in any given test. This means that it's not stdin via a pipe, it's an entire SSH session. This adds a noticeable amount of overhead when trying to send and verify the result of sending 1,000+ packets, since the lack of structured output means each packet must be checked before the next can be sent. This might be solvable by adding a structured output mode to testpmd, but that would involve committing to writing output twice for every function in testpmd forever. > >> We need to add all test cases in this model and we need to maintain two sets of programs.(Traditional tests and gRPC model-based tests). > > > Assuming by traditional tests you mean the unit tests run by Meson, I would argue that we are already maintaining 2 kinds of tests. The unit tests, and the python-based DTS tests. My intention is to create a thin wrapper around DPDK that would be exposed via gRPC, like you see here, and use that as midware. Then, we would have two front-ends. Testpmd, which takes text and then calls midware as it does now, and the gRPC frontend, which parses messages from the RPC server and runs the midware. This would enable testpmd to still be used to sanity check a DPDK installation, but we would not need to continually expand Testpmd. The primary issue is that, right now, anything not included in Testpmd is not really testable by DTS. This includes portions of the RTE Flow API, which was part of my reason for proposing this. The RTE Flow API would, in my estimation, if fully implemented into Testpmd, probably add at least another 10,000 lines of code. As mentioned in my proposal, Testpmd already does more parsing and lexing than it does interaction with DPDK by line count. Also, since I am proposing making this a separate application, we would be able to gradually migrate the tests inside of DTS. This would have no effect on anything except for Testpmd, the new application and the addition of a flag to toggle the use of a C++ compiler. > > I'm not sure exactly what you mean by gRPC model-based tests. gRPC uses classes to model services, but for this usecase we are essentially using it to transfer function arguments across the internet and then pass the return value back. Any RPC framework would function similarly if I ignored the restrictions of which languages to use, and the choice is not important to how tests are conducted. Put another way, how you write a test for DTS will not change much if you are using this or testpmd, it's just how you transfer data and get it back that I want to change.
> > If so, I think, gRPC service would be along with existing > testpmd functions, like start_packet_forwarding(). It was my intention to re-use existing functions. I used the ACL tests as an example because they are more self-contained then Testpmd, which made creating the proof of concept much easier. Also, We don't need to rewrite the existing testpmd, Instead, RPC service, > we can add in existing app/test-pmd/ > The reason that I split out the services is that there doesn't seem to be a way to produce multiple binaries without re-writing that section of the build system. I wanted to avoid the hard requirement of having a C++ compiler available in order to be able to use testpmd, since that may affect what platforms Testpmd can run on and I want to avoid this being any kind of breaking change. If we decide to go the route of putting it all in a single application, we would need to conditionally enable the gRPC service at build time. Meson's current lack of support for conditionally detecting compilers causes issues here. I think, DPDK has only one interactive test case which is testpmd, > Could you point me to that test case? Either invocation or source is ok. I can't see anything that would lead me to assume use of testpmd in "meson test --list". To my knowledge, all of the test cases that use testpmd are in DTS. If there is a test that uses testpmd but is not part of DTS, I think it would be a candidate for moving into DTS assuming it's not a unit test. How you are planning to do noninteractive test cases? I'm not planning to make any change to unit testing, you can read more about how testing is currently conducted here: https://www.dpdk.org/blog/2021/07/05/dpdk-testing-approaches/ If there is a unit test that involves testpmd, there are two possibilities. 1. If we are making a separate application for Testpmd with the gRPC api, then nothing changes except for possibly changing where some of the testpmd source lives in order to enable code reuse between the two applications. 2. If gRPC is being added to Testpmd, then the unit test should still function as it previously did if I do any necessary refactoring as correctly. I think, key should be leveraging existing test cases as much as possible > and make easy for developers to add new test cases. That is part of the reason why I want to be able to do this. Adding a new test in DTS is very easy if the functionality needed already exists in Testpmd. If the functionality does not exist, then adding the test becomes difficult, due to the required modifications to the Testpmd lexer and parser to accommodate the new command. My plan is to leave unit testing in C, but help make it easier to expose C functions to Python for integration testing. This gives us the best of both worlds in terms of access to DPDK and the ability to use a high-level language to write the tests. On Tue, Apr 12, 2022 at 2:07 AM Jerin Jacob <jerinjacobk@gmail.com> wrote: > On Mon, Apr 11, 2022 at 11:19 PM Owen Hilyard <ohilyard@iol.unh.edu> > wrote: > >> > >> scheme is probably over-engineered > > > > > > I tried my hardest to keep this as simple as possible. The requirements > imposed by DTS being a distributed system in Python restricted what I could > do a lot. Needing to be compatible with DPDK's license also got rid of a > lot of options. Binding generators are made for simple projects, and DPDK > is not a simple project. There were some other options related to choice in > the RPC framework, but very few RPC protocols seem to work well with C and > be usable from Python, which is why I ended up using C++ with gRPC. Most of > the code in api_impl.cc is taken from /app/test-acl/main.c, and the new > part is mostly the C++ class at the bottom. Overall, this proposal comes > out to ~100 lines of new C++, 9 lines of C, 12 lines of gRPC Protobuf and > 100 lines of Meson. gRPC may be able to do a lot more than I need it to for > the proof of concept, but many of the features that are not used, like > bi-directional streaming, become very useful in writing more complicated > tests. Overall, this solution is probably more capable than we need it to > be, but I think that those extra capabilities don't come at a very large > cost. > > > Now it is clear, I was carried away with the POC test application and > I was not knowing existing DTS tests are based on python > > Is below a fair summary? > > 1) DPDK has interactive test cases and no interactive test cases. > > For The interactive test case like testpmd, I agree that we can enable > RPC service via gRPC server in C++ as and client in Python, and > something along the lines of exposing the existing test-pmd command > line function as service > to avoid command line parsing and reuse the existing python test suite. > > If so, I think, gRPC service would be along with existing testpmd > functions, like start_packet_forwarding(). Also, We don't need to > rewrite the existing testpmd, > Instead, RPC service, we can add in existing app/test-pmd/ and hook to > existing core testpmd functions to bypass the command-line parsing in > C and control from python client as needed as service. > > Also, I agree that pulling in gRPC C++ server boilerplate and hooking > to C functions is a good idea as it is the best C-based RPC scheme > available today. > > 2)I think, DPDK has only one interactive test case which is testpmd, > Remaining test cases are non-interactive, non-interactive test cases > can simply run over ssh with passwordless login. Right? > Do we need gRPC for that? Will the following scheme suffice? If not, > How you are planning to do noninteractive test cases? > i.e > a)Copy test to target > b) result=`ssh username@IP /path/to/testapp/in/target` > > I think, key should be leveraging existing test cases as much as > possible and make easy for developers to add new test cases. > > > >> > >> Now that, Test code is also part of DPDK. > > > > > > DTS is pure python. I tried to use FFI to call directly into DPDK from > Python and then use xmlrpc from the python standard library. As mentioned > in the writeup, I couldn't find a binding generator that would properly > handle DPDK's allocators, which made it so that anything passed to DPDK > using python was allocated using the system malloc. I don't think it is > wise to attempt to programmatically re-write the generated code to allow > for custom allocators. The original reason for needing to have DTS and DPDK > in the same repository was so that tests could be committed and run > alongside the feature patch. > > > >> Interactive - Testpmd one, I believe, Feeding stdin programmatically > would suffice to test all the combinations. > > > > > > One of the issues this is trying to address is that human-readable > strings are a poor way to pass complex information between two programs. > DTS is a distributed system, and it can have up to 3 physical servers > involved in any given test. This means that it's not stdin via a pipe, it's > an entire SSH session. This adds a noticeable amount of overhead when > trying to send and verify the result of sending 1,000+ packets, since the > lack of structured output means each packet must be checked before the next > can be sent. This might be solvable by adding a structured output mode to > testpmd, but that would involve committing to writing output twice for > every function in testpmd forever. > > > >> We need to add all test cases in this model and we need to maintain two > sets of programs.(Traditional tests and gRPC model-based tests). > > > > > > Assuming by traditional tests you mean the unit tests run by Meson, I > would argue that we are already maintaining 2 kinds of tests. The unit > tests, and the python-based DTS tests. My intention is to create a thin > wrapper around DPDK that would be exposed via gRPC, like you see here, and > use that as midware. Then, we would have two front-ends. Testpmd, which > takes text and then calls midware as it does now, and the gRPC frontend, > which parses messages from the RPC server and runs the midware. This would > enable testpmd to still be used to sanity check a DPDK installation, but we > would not need to continually expand Testpmd. The primary issue is that, > right now, anything not included in Testpmd is not really testable by DTS. > This includes portions of the RTE Flow API, which was part of my reason for > proposing this. The RTE Flow API would, in my estimation, if fully > implemented into Testpmd, probably add at least another 10,000 lines of > code. As mentioned in my proposal, Testpmd already does more parsing and > lexing than it does interaction with DPDK by line count. Also, since I am > proposing making this a separate application, we would be able to gradually > migrate the tests inside of DTS. This would have no effect on anything > except for Testpmd, the new application and the addition of a flag to > toggle the use of a C++ compiler. > > > > I'm not sure exactly what you mean by gRPC model-based tests. gRPC uses > classes to model services, but for this usecase we are essentially using it > to transfer function arguments across the internet and then pass the return > value back. Any RPC framework would function similarly if I ignored the > restrictions of which languages to use, and the choice is not important to > how tests are conducted. Put another way, how you write a test for DTS will > not change much if you are using this or testpmd, it's just how you > transfer data and get it back that I want to change. >
Hi everyone, First of all thanks Owen for stepping forward with this RFC. Few thoughts on this subject below. Konstantin > -----Original Message----- > From: Ananyev, Konstantin <konstantin.ananyev@intel.com> > Sent: Thursday, April 14, 2022 12:59 PM > To: Ananyev, Konstantin <konstantin.ananyev@intel.com> > Subject: FW: [PATCH v1 0/4] [RFC] Testpmd RPC API > > > > From: Owen Hilyard <ohilyard@iol.unh.edu> > Sent: Wednesday, April 13, 2022 1:47 PM > To: Jerin Jacob <jerinjacobk@gmail.com> > Cc: dpdk-dev <dev@dpdk.org>; Honnappa Nagarahalli <Honnappa.Nagarahalli@arm.com>; Thomas Monjalon <thomas@monjalon.net> > Subject: Re: [PATCH v1 0/4] [RFC] Testpmd RPC API > > If so, I think, gRPC service would be along with existing testpmd functions, like start_packet_forwarding(). > > It was my intention to re-use existing functions. I used the ACL tests as an example because they are more self-contained then Testpmd, > which made creating the proof of concept much easier. > > Also, We don't need to rewrite the existing testpmd, Instead, RPC service, we can add in existing app/test-pmd/ > > The reason that I split out the services is that there doesn't seem to be a way to produce multiple binaries without re-writing that section of > the build system. I wanted to avoid the hard requirement of having a C++ compiler available in order to be able to use testpmd, since that > may affect what platforms Testpmd can run on and I want to avoid this being any kind of breaking change. If we decide to go the route of > putting it all in a single application, we would need to conditionally enable the gRPC service at build time. Meson's current lack of support > for conditionally detecting compilers causes issues here. > > I think, DPDK has only one interactive test case which is testpmd, > > Could you point me to that test case? Either invocation or source is ok. I can't see anything that would lead me to assume use of testpmd in > "meson test --list". To my knowledge, all of the test cases that use testpmd are in DTS. If there is a test that uses testpmd but is not part of > DTS, I think it would be a candidate for moving into DTS assuming it's not a unit test. > > How you are planning to do noninteractive test cases? > > I'm not planning to make any change to unit testing, you can read more about how testing is currently conducted > here: https://www.dpdk.org/blog/2021/07/05/dpdk-testing-approaches/ > > If there is a unit test that involves testpmd, there are two possibilities. > 1. If we are making a separate application for Testpmd with the gRPC api, then nothing changes except for possibly changing where some > of the testpmd source lives in order to enable code reuse between the two applications. > 2. If gRPC is being added to Testpmd, then the unit test should still function as it previously did if I do any necessary refactoring as correctly. > > I think, key should be leveraging existing test cases as much as possible and make easy for developers to add new test cases. > > That is part of the reason why I want to be able to do this. Adding a new test in DTS is very easy if the functionality needed already exists in > Testpmd. If the functionality does not exist, then adding the test becomes difficult, due to the required modifications to the Testpmd lexer > and parser to accommodate the new command. My plan is to leave unit testing in C, but help make it easier to expose C functions to Python > for integration testing. This gives us the best of both worlds in terms of access to DPDK and the ability to use a high-level language to write > the tests. > > On Tue, Apr 12, 2022 at 2:07 AM Jerin Jacob <mailto:jerinjacobk@gmail.com> wrote: > On Mon, Apr 11, 2022 at 11:19 PM Owen Hilyard <mailto:ohilyard@iol.unh.edu> wrote: > >> > >> scheme is probably over-engineered > > > > > > I tried my hardest to keep this as simple as possible. The requirements imposed by DTS being a distributed system in Python restricted > what I could do a lot. Needing to be compatible with DPDK's license also got rid of a lot of options. Binding generators are made for simple > projects, and DPDK is not a simple project. There were some other options related to choice in the RPC framework, but very few RPC > protocols seem to work well with C and be usable from Python, which is why I ended up using C++ with gRPC. Most of the code in > api_impl.cc is taken from /app/test-acl/main.c, and the new part is mostly the C++ class at the bottom. Overall, this proposal comes out to > ~100 lines of new C++, 9 lines of C, 12 lines of gRPC Protobuf and 100 lines of Meson. gRPC may be able to do a lot more than I need it to > for the proof of concept, but many of the features that are not used, like bi-directional streaming, become very useful in writing more > complicated tests. Overall, this solution is probably more capable than we need it to be, but I think that those extra capabilities don't come > at a very large cost. > > > Now it is clear, I was carried away with the POC test application and > I was not knowing existing DTS tests are based on python > > Is below a fair summary? > > 1) DPDK has interactive test cases and no interactive test cases. > > For The interactive test case like testpmd, I agree that we can enable > RPC service via gRPC server in C++ as and client in Python, and > something along the lines of exposing the existing test-pmd command > line function as service > to avoid command line parsing and reuse the existing python test suite. > > If so, I think, gRPC service would be along with existing testpmd > functions, like start_packet_forwarding(). Also, We don't need to > rewrite the existing testpmd, > Instead, RPC service, we can add in existing app/test-pmd/ and hook to > existing core testpmd functions to bypass the command-line parsing in > C and control from python client as needed as service. > > Also, I agree that pulling in gRPC C++ server boilerplate and hooking > to C functions is a good idea as it is the best C-based RPC scheme > available today. > > 2)I think, DPDK has only one interactive test case which is testpmd, > Remaining test cases are non-interactive, non-interactive test cases > can simply run over ssh with passwordless login. Right? > Do we need gRPC for that? Will the following scheme suffice? If not, > How you are planning to do noninteractive test cases? > i.e > a)Copy test to target > b) result=`ssh username@IP /path/to/testapp/in/target` > > I think, key should be leveraging existing test cases as much as > possible and make easy for developers to add new test cases. > > > >> > >> Now that, Test code is also part of DPDK. > > > > > > DTS is pure python. I tried to use FFI to call directly into DPDK from Python and then use xmlrpc from the python standard library. As > mentioned in the writeup, I couldn't find a binding generator that would properly handle DPDK's allocators, which made it so that anything > passed to DPDK using python was allocated using the system malloc. I don't think it is wise to attempt to programmatically re-write the > generated code to allow for custom allocators. The original reason for needing to have DTS and DPDK in the same repository was so that > tests could be committed and run alongside the feature patch. > > > >> Interactive - Testpmd one, I believe, Feeding stdin programmatically would suffice to test all the combinations. > > > > > > One of the issues this is trying to address is that human-readable strings are a poor way to pass complex information between two > programs. DTS is a distributed system, and it can have up to 3 physical servers involved in any given test. This means that it's not stdin via a > pipe, it's an entire SSH session. This adds a noticeable amount of overhead when trying to send and verify the result of sending 1,000+ > packets, since the lack of structured output means each packet must be checked before the next can be sent. This might be solvable by > adding a structured output mode to testpmd, but that would involve committing to writing output twice for every function in testpmd > forever. > > > >> We need to add all test cases in this model and we need to maintain two sets of programs.(Traditional tests and gRPC model-based > tests). > > > > > > Assuming by traditional tests you mean the unit tests run by Meson, I would argue that we are already maintaining 2 kinds of tests. The > unit tests, and the python-based DTS tests. My intention is to create a thin wrapper around DPDK that would be exposed via gRPC, like you > see here, and use that as midware. Then, we would have two front-ends. Testpmd, which takes text and then calls midware as it does now, > and the gRPC frontend, which parses messages from the RPC server and runs the midware. This would enable testpmd to still be used to > sanity check a DPDK installation, but we would not need to continually expand Testpmd. The primary issue is that, right now, anything not > included in Testpmd is not really testable by DTS. This includes portions of the RTE Flow API, which was part of my reason for proposing this. > The RTE Flow API would, in my estimation, if fully implemented into Testpmd, probably add at least another 10,000 lines of code. As > mentioned in my proposal, Testpmd already does more parsing and lexing than it does interaction with DPDK by line count. Also, since I am > proposing making this a separate application, we would be able to gradually migrate the tests inside of DTS. This would have no effect on > anything except for Testpmd, the new application and the addition of a flag to toggle the use of a C++ compiler. > > > > I'm not sure exactly what you mean by gRPC model-based tests. gRPC uses classes to model services, but for this usecase we are > essentially using it to transfer function arguments across the internet and then pass the return value back. Any RPC framework would > function similarly if I ignored the restrictions of which languages to use, and the choice is not important to how tests are conducted. Put > another way, how you write a test for DTS will not change much if you are using this or testpmd, it's just how you transfer data and get it > back that I want to change. - In general I think it is a good idea to adding gRPC binding to testpmd to expose/improve testing automation. Though I think we shouldn’t remove existing CLI interface. Ideally I’d like to have both – CLI and gRPC for all commands. Don’t know how realistic is that, but at least for major commands - port/queue configure, start/stop, etc. - Conditional compilation (new meson flag or so) is probably good enough for this case. - About RFC itself - I understand that you choose testacl for simplicity, but in fact, it is a standalone application that has not much common with testpmd itself and the problems that you mentioned: interactive commands, parameter and results parsing, etc. Would it be possible to try implement something more realistic with testpmd itself, like simple test-pmd port/queue configure, start, result collection, etc.? To get a better idea how it is going to work and how complicated it would be.
> > Though I think we shouldn’t remove existing CLI interface. > I agree, it's a very useful debugging tool for validating environments. I think having two "frontends", the CLI and API, which both consume one "backend" testpmd library would be the easiest way to go about doing that while keeping long-term maintenance low. Conditional compilation (new meson flag or so) is probably good enough for > this case. > One of the changes I made was an on-by-default meson flag to enable C++ compilation. If that flag is on, and all dependencies are present, then the application will be built. Would it be possible to try implement something more realistic with testpmd > itself I would consider it a "phase 2" version of this RFC. The hard part was getting gRPC working inside of Meson, which is why I picked a simple app to port. If this RFC moves forward, I can look at porting the functionality needed for the nic single core performance test ( http://git.dpdk.org/tools/dts/tree/test_plans/nic_single_core_perf_test_plan.rst ). On Thu, Apr 14, 2022 at 8:08 AM Ananyev, Konstantin < konstantin.ananyev@intel.com> wrote: > > Hi everyone, > > First of all thanks Owen for stepping forward with this RFC. > Few thoughts on this subject below. > Konstantin > > > -----Original Message----- > > From: Ananyev, Konstantin <konstantin.ananyev@intel.com> > > Sent: Thursday, April 14, 2022 12:59 PM > > To: Ananyev, Konstantin <konstantin.ananyev@intel.com> > > Subject: FW: [PATCH v1 0/4] [RFC] Testpmd RPC API > > > > > > > > From: Owen Hilyard <ohilyard@iol.unh.edu> > > Sent: Wednesday, April 13, 2022 1:47 PM > > To: Jerin Jacob <jerinjacobk@gmail.com> > > Cc: dpdk-dev <dev@dpdk.org>; Honnappa Nagarahalli < > Honnappa.Nagarahalli@arm.com>; Thomas Monjalon <thomas@monjalon.net> > > Subject: Re: [PATCH v1 0/4] [RFC] Testpmd RPC API > > > > If so, I think, gRPC service would be along with existing testpmd > functions, like start_packet_forwarding(). > > > > It was my intention to re-use existing functions. I used the ACL tests > as an example because they are more self-contained then Testpmd, > > which made creating the proof of concept much easier. > > > > Also, We don't need to rewrite the existing testpmd, Instead, RPC > service, we can add in existing app/test-pmd/ > > > > The reason that I split out the services is that there doesn't seem to > be a way to produce multiple binaries without re-writing that section of > > the build system. I wanted to avoid the hard requirement of having a C++ > compiler available in order to be able to use testpmd, since that > > may affect what platforms Testpmd can run on and I want to avoid this > being any kind of breaking change. If we decide to go the route of > > putting it all in a single application, we would need to conditionally > enable the gRPC service at build time. Meson's current lack of support > > for conditionally detecting compilers causes issues here. > > > > I think, DPDK has only one interactive test case which is testpmd, > > > > Could you point me to that test case? Either invocation or source is ok. > I can't see anything that would lead me to assume use of testpmd in > > "meson test --list". To my knowledge, all of the test cases that use > testpmd are in DTS. If there is a test that uses testpmd but is not part of > > DTS, I think it would be a candidate for moving into DTS assuming it's > not a unit test. > > > > How you are planning to do noninteractive test cases? > > > > I'm not planning to make any change to unit testing, you can read more > about how testing is currently conducted > > here: https://www.dpdk.org/blog/2021/07/05/dpdk-testing-approaches/ > > > > If there is a unit test that involves testpmd, there are two > possibilities. > > 1. If we are making a separate application for Testpmd with the gRPC > api, then nothing changes except for possibly changing where some > > of the testpmd source lives in order to enable code reuse between the > two applications. > > 2. If gRPC is being added to Testpmd, then the unit test should still > function as it previously did if I do any necessary refactoring as > correctly. > > > > I think, key should be leveraging existing test cases as much as > possible and make easy for developers to add new test cases. > > > > That is part of the reason why I want to be able to do this. Adding a > new test in DTS is very easy if the functionality needed already exists in > > Testpmd. If the functionality does not exist, then adding the test > becomes difficult, due to the required modifications to the Testpmd lexer > > and parser to accommodate the new command. My plan is to leave unit > testing in C, but help make it easier to expose C functions to Python > > for integration testing. This gives us the best of both worlds in terms > of access to DPDK and the ability to use a high-level language to write > > the tests. > > > > On Tue, Apr 12, 2022 at 2:07 AM Jerin Jacob <mailto: > jerinjacobk@gmail.com> wrote: > > On Mon, Apr 11, 2022 at 11:19 PM Owen Hilyard <mailto: > ohilyard@iol.unh.edu> wrote: > > >> > > >> scheme is probably over-engineered > > > > > > > > > I tried my hardest to keep this as simple as possible. The > requirements imposed by DTS being a distributed system in Python restricted > > what I could do a lot. Needing to be compatible with DPDK's license also > got rid of a lot of options. Binding generators are made for simple > > projects, and DPDK is not a simple project. There were some other > options related to choice in the RPC framework, but very few RPC > > protocols seem to work well with C and be usable from Python, which is > why I ended up using C++ with gRPC. Most of the code in > > api_impl.cc is taken from /app/test-acl/main.c, and the new part is > mostly the C++ class at the bottom. Overall, this proposal comes out to > > ~100 lines of new C++, 9 lines of C, 12 lines of gRPC Protobuf and 100 > lines of Meson. gRPC may be able to do a lot more than I need it to > > for the proof of concept, but many of the features that are not used, > like bi-directional streaming, become very useful in writing more > > complicated tests. Overall, this solution is probably more capable than > we need it to be, but I think that those extra capabilities don't come > > at a very large cost. > > > > > > Now it is clear, I was carried away with the POC test application and > > I was not knowing existing DTS tests are based on python > > > > Is below a fair summary? > > > > 1) DPDK has interactive test cases and no interactive test cases. > > > > For The interactive test case like testpmd, I agree that we can enable > > RPC service via gRPC server in C++ as and client in Python, and > > something along the lines of exposing the existing test-pmd command > > line function as service > > to avoid command line parsing and reuse the existing python test suite. > > > > If so, I think, gRPC service would be along with existing testpmd > > functions, like start_packet_forwarding(). Also, We don't need to > > rewrite the existing testpmd, > > Instead, RPC service, we can add in existing app/test-pmd/ and hook to > > existing core testpmd functions to bypass the command-line parsing in > > C and control from python client as needed as service. > > > > Also, I agree that pulling in gRPC C++ server boilerplate and hooking > > to C functions is a good idea as it is the best C-based RPC scheme > > available today. > > > > 2)I think, DPDK has only one interactive test case which is testpmd, > > Remaining test cases are non-interactive, non-interactive test cases > > can simply run over ssh with passwordless login. Right? > > Do we need gRPC for that? Will the following scheme suffice? If not, > > How you are planning to do noninteractive test cases? > > i.e > > a)Copy test to target > > b) result=`ssh username@IP /path/to/testapp/in/target` > > > > I think, key should be leveraging existing test cases as much as > > possible and make easy for developers to add new test cases. > > > > > > >> > > >> Now that, Test code is also part of DPDK. > > > > > > > > > DTS is pure python. I tried to use FFI to call directly into DPDK from > Python and then use xmlrpc from the python standard library. As > > mentioned in the writeup, I couldn't find a binding generator that would > properly handle DPDK's allocators, which made it so that anything > > passed to DPDK using python was allocated using the system malloc. I > don't think it is wise to attempt to programmatically re-write the > > generated code to allow for custom allocators. The original reason for > needing to have DTS and DPDK in the same repository was so that > > tests could be committed and run alongside the feature patch. > > > > > >> Interactive - Testpmd one, I believe, Feeding stdin programmatically > would suffice to test all the combinations. > > > > > > > > > One of the issues this is trying to address is that human-readable > strings are a poor way to pass complex information between two > > programs. DTS is a distributed system, and it can have up to 3 physical > servers involved in any given test. This means that it's not stdin via a > > pipe, it's an entire SSH session. This adds a noticeable amount of > overhead when trying to send and verify the result of sending 1,000+ > > packets, since the lack of structured output means each packet must be > checked before the next can be sent. This might be solvable by > > adding a structured output mode to testpmd, but that would involve > committing to writing output twice for every function in testpmd > > forever. > > > > > >> We need to add all test cases in this model and we need to maintain > two sets of programs.(Traditional tests and gRPC model-based > > tests). > > > > > > > > > Assuming by traditional tests you mean the unit tests run by Meson, I > would argue that we are already maintaining 2 kinds of tests. The > > unit tests, and the python-based DTS tests. My intention is to create a > thin wrapper around DPDK that would be exposed via gRPC, like you > > see here, and use that as midware. Then, we would have two front-ends. > Testpmd, which takes text and then calls midware as it does now, > > and the gRPC frontend, which parses messages from the RPC server and > runs the midware. This would enable testpmd to still be used to > > sanity check a DPDK installation, but we would not need to continually > expand Testpmd. The primary issue is that, right now, anything not > > included in Testpmd is not really testable by DTS. This includes > portions of the RTE Flow API, which was part of my reason for proposing > this. > > The RTE Flow API would, in my estimation, if fully implemented into > Testpmd, probably add at least another 10,000 lines of code. As > > mentioned in my proposal, Testpmd already does more parsing and lexing > than it does interaction with DPDK by line count. Also, since I am > > proposing making this a separate application, we would be able to > gradually migrate the tests inside of DTS. This would have no effect on > > anything except for Testpmd, the new application and the addition of a > flag to toggle the use of a C++ compiler. > > > > > > I'm not sure exactly what you mean by gRPC model-based tests. gRPC > uses classes to model services, but for this usecase we are > > essentially using it to transfer function arguments across the internet > and then pass the return value back. Any RPC framework would > > function similarly if I ignored the restrictions of which languages to > use, and the choice is not important to how tests are conducted. Put > > another way, how you write a test for DTS will not change much if you > are using this or testpmd, it's just how you transfer data and get it > > back that I want to change. > > - In general I think it is a good idea to adding gRPC binding to testpmd > to expose/improve testing automation. > Though I think we shouldn’t remove existing CLI interface. > Ideally I’d like to have both – CLI and gRPC for all commands. > Don’t know how realistic is that, but at least for major commands - > port/queue configure, start/stop, etc. > - Conditional compilation (new meson flag or so) is probably good enough > for this case. > - About RFC itself - I understand that you choose testacl for simplicity, > but in fact, it is a standalone application > that has not much common with testpmd itself and the problems that you > mentioned: > interactive commands, parameter and results parsing, etc. > Would it be possible to try implement something more realistic with > testpmd itself, > like simple test-pmd port/queue configure, start, result collection, > etc.? > To get a better idea how it is going to work and how complicated it > would be. > > >