[RFC,v1,00/15] merge DTS core files to DPDK

  These are the basic libraries that other libraries depend on. There's
also the basic framework functionality related to test execution.

Juraj Linkeš (15):
  dts: merge DTS dep/tclclient.tgz to DPDK
  dts: merge DTS dep/tgen.tgz to DPDK
  dts: merge DTS dts to DPDK
  dts: merge DTS framework/__init__.py to DPDK
  dts: merge DTS framework/asan_test.py to DPDK
  dts: merge DTS framework/checkCase.py to DPDK
  dts: merge DTS framework/dts.py to DPDK
  dts: merge DTS framework/exception.py to DPDK
  dts: merge DTS framework/logger.py to DPDK
  dts: merge DTS framework/packet.py to DPDK
  dts: merge DTS framework/project_dpdk.py to DPDK
  dts: merge DTS framework/serializer.py to DPDK
  dts: merge DTS framework/utils.py to DPDK
  dts: merge DTS main.py to DPDK
  dts: merge DTS version.py to DPDK

 dts/dep/tclclient.tgz         |  Bin 0 -> 199327 bytes
 dts/dep/tgen.tgz              |  Bin 0 -> 134392 bytes
 dts/dts                       |    9 +
 dts/framework/__init__.py     |   30 +
 dts/framework/asan_test.py    |  439 +++++++++++
 dts/framework/checkCase.py    |  214 ++++++
 dts/framework/dts.py          |  721 ++++++++++++++++++
 dts/framework/exception.py    |  148 ++++
 dts/framework/logger.py       |  474 ++++++++++++
 dts/framework/packet.py       | 1292 +++++++++++++++++++++++++++++++++
 dts/framework/project_dpdk.py |  627 ++++++++++++++++
 dts/framework/serializer.py   |  115 +++
 dts/framework/utils.py        |  353 +++++++++
 dts/main.py                   |  236 ++++++
 dts/version.py                |   21 +
 15 files changed, 4679 insertions(+)
 create mode 100644 dts/dep/tclclient.tgz
 create mode 100644 dts/dep/tgen.tgz
 create mode 100755 dts/dts
 create mode 100644 dts/framework/__init__.py
 create mode 100644 dts/framework/asan_test.py
 create mode 100644 dts/framework/checkCase.py
 create mode 100644 dts/framework/dts.py
 create mode 100644 dts/framework/exception.py
 create mode 100644 dts/framework/logger.py
 create mode 100644 dts/framework/packet.py
 create mode 100644 dts/framework/project_dpdk.py
 create mode 100644 dts/framework/serializer.py
 create mode 100644 dts/framework/utils.py
 create mode 100755 dts/main.py
 create mode 100755 dts/version.py
  

On Wed, Apr 6, 2022 at 8:26 PM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
>
> These are the basic libraries that other libraries depend on. There's
> also the basic framework functionality related to test execution.
>
> Juraj Linkeš (15):
>   dts: merge DTS dep/tclclient.tgz to DPDK
>   dts: merge DTS dep/tgen.tgz to DPDK
>   dts: merge DTS dts to DPDK
>   dts: merge DTS framework/__init__.py to DPDK
>   dts: merge DTS framework/asan_test.py to DPDK
>   dts: merge DTS framework/checkCase.py to DPDK
>   dts: merge DTS framework/dts.py to DPDK
>   dts: merge DTS framework/exception.py to DPDK
>   dts: merge DTS framework/logger.py to DPDK
>   dts: merge DTS framework/packet.py to DPDK
>   dts: merge DTS framework/project_dpdk.py to DPDK
>   dts: merge DTS framework/serializer.py to DPDK
>   dts: merge DTS framework/utils.py to DPDK
>   dts: merge DTS main.py to DPDK
>   dts: merge DTS version.py to DPDK

merge->import

>
>  dts/dep/tclclient.tgz         |  Bin 0 -> 199327 bytes
>  dts/dep/tgen.tgz              |  Bin 0 -> 134392 bytes

Some top level comments:
- I think, we should not check in binary files.
- git commit comment should much more than "dts: merge DTS xxxx to DPDK" where
the commit log should have details on check in.
-Add the documentation from the first patch and update the
documentation per patch
based on the content.
  

07/04/2022 07:04, Jerin Jacob:
> On Wed, Apr 6, 2022 at 8:26 PM Juraj Linkeš <juraj.linkes@pantheon.tech> wrote:
> >
> > These are the basic libraries that other libraries depend on. There's
> > also the basic framework functionality related to test execution.
> >
> > Juraj Linkeš (15):
> >   dts: merge DTS dep/tclclient.tgz to DPDK
> >   dts: merge DTS dep/tgen.tgz to DPDK
> >   dts: merge DTS dts to DPDK
> >   dts: merge DTS framework/__init__.py to DPDK
> >   dts: merge DTS framework/asan_test.py to DPDK
> >   dts: merge DTS framework/checkCase.py to DPDK
> >   dts: merge DTS framework/dts.py to DPDK
> >   dts: merge DTS framework/exception.py to DPDK
> >   dts: merge DTS framework/logger.py to DPDK
> >   dts: merge DTS framework/packet.py to DPDK
> >   dts: merge DTS framework/project_dpdk.py to DPDK
> >   dts: merge DTS framework/serializer.py to DPDK
> >   dts: merge DTS framework/utils.py to DPDK
> >   dts: merge DTS main.py to DPDK
> >   dts: merge DTS version.py to DPDK
> 
> merge->import
> 
> >
> >  dts/dep/tclclient.tgz         |  Bin 0 -> 199327 bytes
> >  dts/dep/tgen.tgz              |  Bin 0 -> 134392 bytes
> 
> Some top level comments:
> - I think, we should not check in binary files.

+1

> - git commit comment should much more than "dts: merge DTS xxxx to DPDK" where
> the commit log should have details on check in.

+1

> -Add the documentation from the first patch and update the
> documentation per patch
> based on the content.

+1

More comments:

- Please don't send so many patches, it looks like spam.
- Please let's start small with the very minimal code
to run a dummy test.
- Split by file does not make sense

The process is going to be very long.
The techboard said in the past that we must have a very careful review
of an import piece by piece. So please be patient.
Thank you
  

> -----Original Message-----
> From: Thomas Monjalon <thomas@monjalon.net>
> Sent: Thursday, April 7, 2022 9:34 AM
> To: Juraj Linkeš <juraj.linkes@pantheon.tech>
> Cc: Jerin Jacob <jerinjacobk@gmail.com>; David Marchand
> <david.marchand@redhat.com>; Honnappa Nagarahalli
> <Honnappa.Nagarahalli@arm.com>; ohilyard@iol.unh.edu; Tu, Lijuan
> <lijuan.tu@intel.com>; dpdk-dev <dev@dpdk.org>
> Subject: Re: [RFC PATCH v1 00/15] merge DTS core files to DPDK
> 
> 07/04/2022 07:04, Jerin Jacob:
> > On Wed, Apr 6, 2022 at 8:26 PM Juraj Linkeš <juraj.linkes@pantheon.tech>
> wrote:
> > >

First of all, thanks for the feedback. We've already done a bit of a pre-review and this is where we decided we want the feedback from the rest of the community. We understand it's going to take time which is why we wanted to get feedback sooner than later.

> > > These are the basic libraries that other libraries depend on.
> > > There's also the basic framework functionality related to test execution.
> > >
> > > Juraj Linkeš (15):
> > >   dts: merge DTS dep/tclclient.tgz to DPDK
> > >   dts: merge DTS dep/tgen.tgz to DPDK
> > >   dts: merge DTS dts to DPDK
> > >   dts: merge DTS framework/__init__.py to DPDK
> > >   dts: merge DTS framework/asan_test.py to DPDK
> > >   dts: merge DTS framework/checkCase.py to DPDK
> > >   dts: merge DTS framework/dts.py to DPDK
> > >   dts: merge DTS framework/exception.py to DPDK
> > >   dts: merge DTS framework/logger.py to DPDK
> > >   dts: merge DTS framework/packet.py to DPDK
> > >   dts: merge DTS framework/project_dpdk.py to DPDK
> > >   dts: merge DTS framework/serializer.py to DPDK
> > >   dts: merge DTS framework/utils.py to DPDK
> > >   dts: merge DTS main.py to DPDK
> > >   dts: merge DTS version.py to DPDK
> >
> > merge->import
> >

Ack. I'll keep this in mind when composing commit messages in the future.

> > >
> > >  dts/dep/tclclient.tgz         |  Bin 0 -> 199327 bytes
> > >  dts/dep/tgen.tgz              |  Bin 0 -> 134392 bytes
> >
> > Some top level comments:
> > - I think, we should not check in binary files.
> 
> +1
> 

Thanks, we'll make sure to make this a requirement - it makes sense to us as well.

> > - git commit comment should much more than "dts: merge DTS xxxx to
> > DPDK" where the commit log should have details on check in.
> 
> +1

Right, this is part of the broader question of how exactly are we going to move the whole thing. More below.

> 
> > -Add the documentation from the first patch and update the
> > documentation per patch based on the content.
> 
> +1
> 
> More comments:
> 
> - Please don't send so many patches, it looks like spam.
> - Please let's start small with the very minimal code to run a dummy test.
> - Split by file does not make sense

This is very useful. We started with splitting up DTS into logical chunks, but it's not quite what you have in mind. We'll move things around and send just the first part (containing just a dummy test (is a smoke test ok?) with the necessary code). Now that we have a better understanding of the first step we'll produce a more complete patch set, with proper commit messages and files in each commit.

> 
> The process is going to be very long.
> The techboard said in the past that we must have a very careful review of an
> import piece by piece. So please be patient.
> Thank you
> 
>
  

<snip>

> >
> > 07/04/2022 07:04, Jerin Jacob:
> > > On Wed, Apr 6, 2022 at 8:26 PM Juraj Linkeš
> > > <juraj.linkes@pantheon.tech>
> > wrote:
> > > >
> 
> First of all, thanks for the feedback. We've already done a bit of a pre-review
> and this is where we decided we want the feedback from the rest of the
> community. We understand it's going to take time which is why we wanted to
> get feedback sooner than later.
> 
> > > > These are the basic libraries that other libraries depend on.
> > > > There's also the basic framework functionality related to test execution.
> > > >
> > > > Juraj Linkeš (15):
> > > >   dts: merge DTS dep/tclclient.tgz to DPDK
> > > >   dts: merge DTS dep/tgen.tgz to DPDK
> > > >   dts: merge DTS dts to DPDK
> > > >   dts: merge DTS framework/__init__.py to DPDK
> > > >   dts: merge DTS framework/asan_test.py to DPDK
> > > >   dts: merge DTS framework/checkCase.py to DPDK
> > > >   dts: merge DTS framework/dts.py to DPDK
> > > >   dts: merge DTS framework/exception.py to DPDK
> > > >   dts: merge DTS framework/logger.py to DPDK
> > > >   dts: merge DTS framework/packet.py to DPDK
> > > >   dts: merge DTS framework/project_dpdk.py to DPDK
> > > >   dts: merge DTS framework/serializer.py to DPDK
> > > >   dts: merge DTS framework/utils.py to DPDK
> > > >   dts: merge DTS main.py to DPDK
> > > >   dts: merge DTS version.py to DPDK
> > >
> > > merge->import
> > >
> 
> Ack. I'll keep this in mind when composing commit messages in the future.
> 
> > > >
> > > >  dts/dep/tclclient.tgz         |  Bin 0 -> 199327 bytes
> > > >  dts/dep/tgen.tgz              |  Bin 0 -> 134392 bytes
> > >
> > > Some top level comments:
> > > - I think, we should not check in binary files.
> >
> > +1
> >
> 
> Thanks, we'll make sure to make this a requirement - it makes sense to us as
> well.
Should we move these to the external repo where the GPL file would be stored? I do not expect these files to change frequently. But the external repo also needs to be tagged (along with tagging the DPDK/DTS).

> 
> > > - git commit comment should much more than "dts: merge DTS xxxx to
> > > DPDK" where the commit log should have details on check in.
> >
> > +1
> 
> Right, this is part of the broader question of how exactly are we going to
> move the whole thing. More below.
> 
> >
> > > -Add the documentation from the first patch and update the
> > > documentation per patch based on the content.
> >
> > +1
> >
> > More comments:
> >
> > - Please don't send so many patches, it looks like spam.
> > - Please let's start small with the very minimal code to run a dummy test.
> > - Split by file does not make sense
> 
> This is very useful. We started with splitting up DTS into logical chunks, but it's
> not quite what you have in mind. We'll move things around and send just the
> first part (containing just a dummy test (is a smoke test ok?) with the
> necessary code). Now that we have a better understanding of the first step
> we'll produce a more complete patch set, with proper commit messages and
> files in each commit.
I think bringing in this big chunk of code is a challenge. We need to discuss more on how to get this into DPDK. From that perspective:

1) the current set of patches are just the framework and documentation, no test suites yet. I am not sure if there are further parts that can be stripped out.

2) agree, the split by files is too much. We have tried to group the files into some logical components. So, would it be fine if we remove the commit by file, but still keep the logical component split? i.e. we will squash the files in a series into a single commit and then combine the logical splits as commits in a single series.

3) In my private discussions with David Marchand, he expressed interest in getting the git log history. The current review process will not help in this regard. Is this a must? If yes, are there any known methods to do this?

> 
> >
> > The process is going to be very long.
> > The techboard said in the past that we must have a very careful review
> > of an import piece by piece. So please be patient.
Agree, we need to give time for review.
I would prefer to set some timeline so that we are working accordingly. Currently, we are working with the goal of merging this in 22.07 (assuming the reviews are coming through quick enough). But, we can align on a different timeline.

> > Thank you
> >
> >
  

>
>  3) In my private discussions with David Marchand, he expressed interest
> in getting the git log history. The current review process will not help in
> this regard. Is this a must? If yes, are there any known methods to do this?


The git docs point to https://github.com/newren/git-filter-repo/, since
apparently git filter-branch is full of footguns.  From there, I would
follow an example from Ruby,
https://gist.github.com/x-yuri/9890ab1079cf4357d6f269d073fd9731, which was
merging two reasonably-sized tooling repos. Instead of using git merge, we
would be using git rebase. I did it locally to try it out, and it does have
to replay the ENTIRE git history of dts. This means whoever does it may
want to grab a server and do the rebase in /run. This would also mean that
we would have to bypass the mailing list, since sending out more than 32000
emails doesn't seem reasonable. Possibly just a patch with the script to
run to do the rebase instead of putting all the patches on the mailing list?

Doing this would also probably mean all of our work would be in the DTS
repo until we are ready to merge. That might make it harder to review
changes to DTS in small parts.
  

<snip>

> 
> On Wed, Apr 6, 2022 at 8:26 PM Juraj Linkeš <juraj.linkes@pantheon.tech>
> wrote:
> >
> > These are the basic libraries that other libraries depend on. There's
> > also the basic framework functionality related to test execution.
> >
> > Juraj Linkeš (15):
> >   dts: merge DTS dep/tclclient.tgz to DPDK
> >   dts: merge DTS dep/tgen.tgz to DPDK
> >   dts: merge DTS dts to DPDK
> >   dts: merge DTS framework/__init__.py to DPDK
> >   dts: merge DTS framework/asan_test.py to DPDK
> >   dts: merge DTS framework/checkCase.py to DPDK
> >   dts: merge DTS framework/dts.py to DPDK
> >   dts: merge DTS framework/exception.py to DPDK
> >   dts: merge DTS framework/logger.py to DPDK
> >   dts: merge DTS framework/packet.py to DPDK
> >   dts: merge DTS framework/project_dpdk.py to DPDK
> >   dts: merge DTS framework/serializer.py to DPDK
> >   dts: merge DTS framework/utils.py to DPDK
> >   dts: merge DTS main.py to DPDK
> >   dts: merge DTS version.py to DPDK
> 
> merge->import
> 
> >
> >  dts/dep/tclclient.tgz         |  Bin 0 -> 199327 bytes
> >  dts/dep/tgen.tgz              |  Bin 0 -> 134392 bytes
> 
> Some top level comments:
> - I think, we should not check in binary files.
> - git commit comment should much more than "dts: merge DTS xxxx to DPDK"
> where the commit log should have details on check in.
> -Add the documentation from the first patch and update the documentation
> per patch based on the content.
The documentation is existing documentation. It has evolved based on how the code has evolved. I do not think it is possible to split that documentation now and bring it in on a per patch basis. The best way would be to maintain the history from git.

Some documentation is also getting updated, but this is one time overall update.