[v2] devtools: add tracepoint check in checkpatch

Message ID 20221012114525.12220-1-adwivedi@marvell.com (mailing list archive)
State Superseded, archived
Delegated to: Thomas Monjalon
Headers
Series [v2] devtools: add tracepoint check in checkpatch |

Checks

Context Check Description
ci/checkpatch success coding style OK
ci/Intel-compilation success Compilation OK
ci/intel-Testing success Testing PASS
ci/github-robot: build success github build: passed
ci/iol-mellanox-Performance success Performance Testing PASS
ci/iol-x86_64-unit-testing success Testing PASS
ci/iol-x86_64-compile-testing success Testing PASS
ci/iol-aarch64-unit-testing success Testing PASS
ci/iol-aarch64-compile-testing success Testing PASS
ci/iol-intel-Functional success Functional Testing PASS
ci/iol-intel-Performance success Performance Testing PASS

Commit Message

Ankur Dwivedi Oct. 12, 2022, 11:45 a.m. UTC
  This patch adds a check in checkpatch tool, to check if a
tracepoint is present in any new function added in cryptodev
library.

It uses the existing build_map_changes() function and version.map
file to create a list of newly added functions. The definition of
these functions are checked if they contain tracepoint.
The checkpatch return error if the tracepoint is not present.

For functions for which trace is not needed, they can be
appended to devtools/trace-skiplist.txt file. The above tracepoint
check will be skipped for them.

Signed-off-by: Ankur Dwivedi <adwivedi@marvell.com>
---
v2:
 - Add check for parent directory.

 devtools/check-symbol-change.sh | 76 +-------------------------------
 devtools/check-tracepoint.py    | 52 ++++++++++++++++++++++
 devtools/check-tracepoint.sh    | 66 ++++++++++++++++++++++++++++
 devtools/checkpatches.sh        |  9 ++++
 devtools/common-func.sh         | 77 +++++++++++++++++++++++++++++++++
 devtools/trace-skiplist.txt     |  0
 6 files changed, 206 insertions(+), 74 deletions(-)
 create mode 100755 devtools/check-tracepoint.py
 create mode 100755 devtools/check-tracepoint.sh
 create mode 100644 devtools/common-func.sh
 create mode 100644 devtools/trace-skiplist.txt
  

Comments

Thomas Monjalon Oct. 12, 2022, 1:08 p.m. UTC | #1
12/10/2022 13:45, Ankur Dwivedi:
>  devtools/check-symbol-change.sh | 76 +-------------------------------
>  devtools/check-tracepoint.py    | 52 ++++++++++++++++++++++
>  devtools/check-tracepoint.sh    | 66 ++++++++++++++++++++++++++++
>  devtools/checkpatches.sh        |  9 ++++
>  devtools/common-func.sh         | 77 +++++++++++++++++++++++++++++++++
>  devtools/trace-skiplist.txt     |  0

Before diving into this proposal,
I would like a split of the patch for the rework (and move)
of check-symbol-change.sh alone.

In general I see too many files added:
	check-tracepoint.py and check-tracepoint.sh
common-func.sh is probably a bad name.
  
Ankur Dwivedi Oct. 12, 2022, 3:16 p.m. UTC | #2
>-----Original Message-----
>From: Thomas Monjalon <thomas@monjalon.net>
>Sent: Wednesday, October 12, 2022 6:39 PM
>To: Ankur Dwivedi <adwivedi@marvell.com>
>Cc: dev@dpdk.org; Akhil Goyal <gakhil@marvell.com>;
>royzhang1980@gmail.com; Amit Prakash Shukla <amitprakashs@marvell.com>;
>Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Ankur Dwivedi
><adwivedi@marvell.com>; david.marchand@redhat.com
>Subject: [EXT] Re: [PATCH v2] devtools: add tracepoint check in checkpatch
>
>External Email
>
>----------------------------------------------------------------------
>12/10/2022 13:45, Ankur Dwivedi:
>>  devtools/check-symbol-change.sh | 76 +-------------------------------
>>  devtools/check-tracepoint.py    | 52 ++++++++++++++++++++++
>>  devtools/check-tracepoint.sh    | 66 ++++++++++++++++++++++++++++
>>  devtools/checkpatches.sh        |  9 ++++
>>  devtools/common-func.sh         | 77 +++++++++++++++++++++++++++++++++
>>  devtools/trace-skiplist.txt     |  0
>
>Before diving into this proposal,
>I would like a split of the patch for the rework (and move) of check-symbol-
>change.sh alone.
Will split the patch in next version.
>
>In general I see too many files added:
>	check-tracepoint.py and check-tracepoint.sh common-func.sh is
>probably a bad name.
Regarding common-func.sh name, I am thinking of renaming it to common.sh or helper.sh, considering there may be more common shell routines in future. Otherwise it can be renamed to build-map.sh considering it will contain only build_map_changes() function. 
Please suggest a suitable name if my suggested names are bad.

Will try to combine check-tracepoint.py and check-tracepoint.sh.
>
  
Thomas Monjalon Oct. 12, 2022, 4:19 p.m. UTC | #3
12/10/2022 17:16, Ankur Dwivedi:
> 
> >-----Original Message-----
> >From: Thomas Monjalon <thomas@monjalon.net>
> >Sent: Wednesday, October 12, 2022 6:39 PM
> >To: Ankur Dwivedi <adwivedi@marvell.com>
> >Cc: dev@dpdk.org; Akhil Goyal <gakhil@marvell.com>;
> >royzhang1980@gmail.com; Amit Prakash Shukla <amitprakashs@marvell.com>;
> >Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Ankur Dwivedi
> ><adwivedi@marvell.com>; david.marchand@redhat.com
> >Subject: [EXT] Re: [PATCH v2] devtools: add tracepoint check in checkpatch
> >
> >External Email
> >
> >----------------------------------------------------------------------
> >12/10/2022 13:45, Ankur Dwivedi:
> >>  devtools/check-symbol-change.sh | 76 +-------------------------------
> >>  devtools/check-tracepoint.py    | 52 ++++++++++++++++++++++
> >>  devtools/check-tracepoint.sh    | 66 ++++++++++++++++++++++++++++
> >>  devtools/checkpatches.sh        |  9 ++++
> >>  devtools/common-func.sh         | 77 +++++++++++++++++++++++++++++++++
> >>  devtools/trace-skiplist.txt     |  0
> >
> >Before diving into this proposal,
> >I would like a split of the patch for the rework (and move) of check-symbol-
> >change.sh alone.
> Will split the patch in next version.
> >
> >In general I see too many files added:
> >	check-tracepoint.py and check-tracepoint.sh common-func.sh is
> >probably a bad name.
> Regarding common-func.sh name, I am thinking of renaming it to common.sh or helper.sh, considering there may be more common shell routines in future. Otherwise it can be renamed to build-map.sh considering it will contain only build_map_changes() function. 
> Please suggest a suitable name if my suggested names are bad.
> 
> Will try to combine check-tracepoint.py and check-tracepoint.sh.

There are many very different scripts in devtools.
Having a very common script won't work well.
Please keep scripts specialized.
If you need a script to check symbol, it should include "symbol" in its name.
  

Patch

diff --git a/devtools/check-symbol-change.sh b/devtools/check-symbol-change.sh
index 8992214ac8..4bdd0d727a 100755
--- a/devtools/check-symbol-change.sh
+++ b/devtools/check-symbol-change.sh
@@ -2,80 +2,8 @@ 
 # SPDX-License-Identifier: BSD-3-Clause
 # Copyright(c) 2018 Neil Horman <nhorman@tuxdriver.com>
 
-build_map_changes()
-{
-	local fname="$1"
-	local mapdb="$2"
-
-	cat "$fname" | awk '
-		# Initialize our variables
-		BEGIN {map="";sym="";ar="";sec=""; in_sec=0; in_map=0}
-
-		# Anything that starts with + or -, followed by an a
-		# and ends in the string .map is the name of our map file
-		# This may appear multiple times in a patch if multiple
-		# map files are altered, and all section/symbol names
-		# appearing between a triggering of this rule and the
-		# next trigger of this rule are associated with this file
-		/[-+] [ab]\/.*\.map/ {map=$2; in_map=1; next}
-
-		# The previous rule catches all .map files, anything else
-		# indicates we left the map chunk.
-		/[-+] [ab]\// {in_map=0}
-
-		# Triggering this rule, which starts a line and ends it
-		# with a { identifies a versioned section.  The section name is
-		# the rest of the line with the + and { symbols removed.
-		# Triggering this rule sets in_sec to 1, which actives the
-		# symbol rule below
-		/^.*{/ {
-			gsub("+", "");
-			if (in_map == 1) {
-				sec=$(NF-1); in_sec=1;
-			}
-		}
-
-		# This rule identifies the end of a section, and disables the
-		# symbol rule
-		/.*}/ {in_sec=0}
-
-		# This rule matches on a + followed by any characters except a :
-		# (which denotes a global vs local segment), and ends with a ;.
-		# The semicolon is removed and the symbol is printed with its
-		# association file name and version section, along with an
-		# indicator that the symbol is a new addition.  Note this rule
-		# only works if we have found a version section in the rule
-		# above (hence the in_sec check) And found a map file (the
-		# in_map check).  If we are not in a map chunk, do nothing.  If
-		# we are in a map chunk but not a section chunk, record it as
-		# unknown.
-		/^+[^}].*[^:*];/ {gsub(";","");sym=$2;
-			if (in_map == 1) {
-				if (in_sec == 1) {
-					print map " " sym " " sec " add"
-				} else {
-					print map " " sym " unknown add"
-				}
-			}
-		}
-
-		# This is the same rule as above, but the rule matches on a
-		# leading - rather than a +, denoting that the symbol is being
-		# removed.
-		/^-[^}].*[^:*];/ {gsub(";","");sym=$2;
-			if (in_map == 1) {
-				if (in_sec == 1) {
-					print map " " sym " " sec " del"
-				} else {
-					print map " " sym " unknown del"
-				}
-			}
-		}' > "$mapdb"
-
-		sort -u "$mapdb" > "$mapdb.2"
-		mv -f "$mapdb.2" "$mapdb"
-
-}
+selfdir=$(dirname $(readlink -f $0))
+. $selfdir/common-func.sh
 
 is_stable_section() {
 	[ "$1" != 'EXPERIMENTAL' ] && [ "$1" != 'INTERNAL' ]
diff --git a/devtools/check-tracepoint.py b/devtools/check-tracepoint.py
new file mode 100755
index 0000000000..5ce0934493
--- /dev/null
+++ b/devtools/check-tracepoint.py
@@ -0,0 +1,52 @@ 
+#!/usr/bin/env python3
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (C) 2022 Marvell.
+
+import sys
+
+patch = sys.argv[1]
+fn = sys.argv[2]
+
+with open(patch, 'r') as fr:
+	fstr = fr.read()
+
+def find_fn_def():
+	found = 0
+	tmp = 0
+	idx = 0
+	while found == 0:
+		idx = fstr.find("+"+fn+"(", idx)
+		if (idx != -1):
+			tmp = fstr.find(')', idx)
+			if (fstr[tmp + 1] == ';'):
+				idx = tmp
+				continue
+			else:
+				found = 1
+		else:
+			break
+	return idx
+
+def find_trace(index):
+	fp = fstr.find("{", index)
+	sp = fstr.find("}", fp)
+	fd = fstr[fp:sp]
+
+	i = fd.find("_trace_")
+	if (i != -1):
+		return 0
+	else:
+		return 1
+
+
+def __main():
+	ret=0
+	index = find_fn_def()
+	if (index != -1):
+		# If function definition is present,
+		# check if tracepoint is present
+		ret = find_trace(index)
+	return ret
+
+if __name__ == "__main__":
+	sys.exit(__main())
diff --git a/devtools/check-tracepoint.sh b/devtools/check-tracepoint.sh
new file mode 100755
index 0000000000..dc8f14ae87
--- /dev/null
+++ b/devtools/check-tracepoint.sh
@@ -0,0 +1,66 @@ 
+#!/bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright (C) 2022 Marvell.
+
+selfdir=$(dirname $(readlink -f $0))
+. $selfdir/common-func.sh
+
+# Library for trace check
+libdir="cryptodev"
+
+# Functions for which the trace check can be skipped
+skiplist="$selfdir/trace-skiplist.txt"
+
+check_for_tracepoint()
+{
+	mapdb="$2"
+	ret=0
+
+	while read -r mname symname secname ar; do
+		libp=0
+		skip_sym=0
+		pdir=$(echo $mname | awk 'BEGIN {FS="/"};{print $2}')
+		libname=$(echo $mname | awk 'BEGIN {FS="/"};{print $3}')
+
+		for lib in $libdir; do
+			if [ "$pdir" = "lib" ] && [ $lib = $libname ]; then
+				libp=1
+				break
+			fi
+		done
+
+		for sym in $(cat $skiplist); do
+			if [ $sym = $symname ]; then
+				skip_sym=1
+				break
+			fi
+		done
+
+		if [ $libp -eq 1 ] && [ $skip_sym -eq 0 ] && [ "$ar" = "add" ]; then
+			if [ "$secname" = "EXPERIMENTAL" ]; then
+				# Check if new API is added with tracepoint
+				if ! devtools/check-tracepoint.py $1 $symname; then
+					ret=1
+					echo -n "ERROR: New function $symname is added "
+					echo -n "without a tracepoint. Please add a tracepoint "
+					echo -n "or add the function $symname in "
+					echo "devtools/trace-skiplist.txt to skip this error."
+				fi
+			fi
+		fi
+	done < "$mapdb"
+
+	return $ret
+}
+
+clean_and_exit_on_sig()
+{
+	rm -rf "$mapfile"
+}
+
+trap clean_and_exit_on_sig EXIT
+
+mapfile=$(mktemp -t dpdk.mapdb.XXXXXX)
+
+build_map_changes "$1" "$mapfile"
+check_for_tracepoint "$1" "$mapfile"
diff --git a/devtools/checkpatches.sh b/devtools/checkpatches.sh
index 1f1175c4f1..7392560460 100755
--- a/devtools/checkpatches.sh
+++ b/devtools/checkpatches.sh
@@ -10,6 +10,7 @@ 
 . $(dirname $(readlink -f $0))/load-devel-config
 
 VALIDATE_NEW_API=$(dirname $(readlink -f $0))/check-symbol-change.sh
+VALIDATE_TRACEPOINT=$(dirname $(readlink -f $0))/check-tracepoint.sh
 
 # Enable codespell by default. This can be overwritten from a config file.
 # Codespell can also be enabled by setting DPDK_CHECKPATCH_CODESPELL to a valid path
@@ -352,6 +353,14 @@  check () { # <patch> <commit> <title>
 		ret=1
 	fi
 
+	! $verbose || printf '\nChecking API additions with tracepoint :\n'
+	report=$($VALIDATE_TRACEPOINT "$tmpinput")
+	if [ $? -ne 0 ] ; then
+		$headline_printed || print_headline "$3"
+		printf '%s\n' "$report"
+		ret=1
+	fi
+
 	if [ "$tmpinput" != "$1" ]; then
 		rm -f "$tmpinput"
 		trap - INT
diff --git a/devtools/common-func.sh b/devtools/common-func.sh
new file mode 100644
index 0000000000..c88e949890
--- /dev/null
+++ b/devtools/common-func.sh
@@ -0,0 +1,77 @@ 
+#!/bin/sh
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Neil Horman <nhorman@tuxdriver.com>
+
+build_map_changes()
+{
+	local fname="$1"
+	local mapdb="$2"
+
+	cat "$fname" | awk '
+		# Initialize our variables
+		BEGIN {map="";sym="";ar="";sec=""; in_sec=0; in_map=0}
+
+		# Anything that starts with + or -, followed by an a
+		# and ends in the string .map is the name of our map file
+		# This may appear multiple times in a patch if multiple
+		# map files are altered, and all section/symbol names
+		# appearing between a triggering of this rule and the
+		# next trigger of this rule are associated with this file
+		/[-+] [ab]\/.*\.map/ {map=$2; in_map=1; next}
+
+		# The previous rule catches all .map files, anything else
+		# indicates we left the map chunk.
+		/[-+] [ab]\// {in_map=0}
+
+		# Triggering this rule, which starts a line and ends it
+		# with a { identifies a versioned section.  The section name is
+		# the rest of the line with the + and { symbols removed.
+		# Triggering this rule sets in_sec to 1, which actives the
+		# symbol rule below
+		/^.*{/ {
+			gsub("+", "");
+			if (in_map == 1) {
+				sec=$(NF-1); in_sec=1;
+			}
+		}
+
+		# This rule identifies the end of a section, and disables the
+		# symbol rule
+		/.*}/ {in_sec=0}
+
+		# This rule matches on a + followed by any characters except a :
+		# (which denotes a global vs local segment), and ends with a ;.
+		# The semicolon is removed and the symbol is printed with its
+		# association file name and version section, along with an
+		# indicator that the symbol is a new addition.  Note this rule
+		# only works if we have found a version section in the rule
+		# above (hence the in_sec check) And found a map file (the
+		# in_map check).  If we are not in a map chunk, do nothing.  If
+		# we are in a map chunk but not a section chunk, record it as
+		# unknown.
+		/^+[^}].*[^:*];/ {gsub(";","");sym=$2;
+			if (in_map == 1) {
+				if (in_sec == 1) {
+					print map " " sym " " sec " add"
+				} else {
+					print map " " sym " unknown add"
+				}
+			}
+		}
+
+		# This is the same rule as above, but the rule matches on a
+		# leading - rather than a +, denoting that the symbol is being
+		# removed.
+		/^-[^}].*[^:*];/ {gsub(";","");sym=$2;
+			if (in_map == 1) {
+				if (in_sec == 1) {
+					print map " " sym " " sec " del"
+				} else {
+					print map " " sym " unknown del"
+				}
+			}
+		}' > "$mapdb"
+
+		sort -u "$mapdb" > "$mapdb.2"
+		mv -f "$mapdb.2" "$mapdb"
+}
diff --git a/devtools/trace-skiplist.txt b/devtools/trace-skiplist.txt
new file mode 100644
index 0000000000..e69de29bb2