From patchwork Thu Sep 9 13:48:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ray Kinsella X-Patchwork-Id: 98437 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D9977A0547; Thu, 9 Sep 2021 15:50:32 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id C005641156; Thu, 9 Sep 2021 15:50:29 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id D349A41149 for ; Thu, 9 Sep 2021 15:50:27 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10101"; a="200318415" X-IronPort-AV: E=Sophos;i="5.85,280,1624345200"; d="scan'208";a="200318415" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2021 06:50:27 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,280,1624345200"; d="scan'208";a="580894504" Received: from silpixa00396680.ir.intel.com (HELO silpixa00396680.ger.corp.intel.com) ([10.237.223.54]) by orsmga004.jf.intel.com with ESMTP; 09 Sep 2021 06:50:24 -0700 From: Ray Kinsella To: dev@dpdk.org Cc: bruce.richardson@intel.com, stephen@networkplumber.org, ferruh.yigit@intel.com, thomas@monjalon.net, ktraynor@redhat.com, mdr@ashroe.eu, aconole@redhat.com, roy.fan.zhang@intel.com, arkadiuszx.kusztal@intel.com, gakhil@marvell.com Date: Thu, 9 Sep 2021 14:48:05 +0100 Message-Id: <20210909134808.1585777-2-mdr@ashroe.eu> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210909134808.1585777-1-mdr@ashroe.eu> References: <20210618163659.85933-1-mdr@ashroe.eu> <20210909134808.1585777-1-mdr@ashroe.eu> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v13 1/4] devtools: script to track symbols over releases X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" This script tracks the growth of stable and experimental symbols over releases since v19.11. The script has the ability to count the added symbols between two dpdk releases, and to list experimental symbols present in two dpdk releases (expired symbols). example usages: Count symbols added since v19.11 $ devtools/symbol-tool.py count-symbols Count symbols added since v20.11 $ devtools/symbol-tool.py count-symbols --releases v20.11,v21.05 List experimental symbols present in v20.11 and v21.05 $ devtools/symbol-tool.py list-expired --releases v20.11,v21.05 List experimental symbols in libraries only, present since v19.11 $ devtools/symbol-tool.py list-expired --directory lib Signed-off-by: Ray Kinsella --- devtools/symbol-tool.py | 566 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 566 insertions(+) create mode 100755 devtools/symbol-tool.py diff --git a/devtools/symbol-tool.py b/devtools/symbol-tool.py new file mode 100755 index 0000000000..a77d8b2442 --- /dev/null +++ b/devtools/symbol-tool.py @@ -0,0 +1,566 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2021 Intel Corporation +# pylint: disable=invalid-name +'''Tool to count or list symbols in each DPDK release''' +from pathlib import Path +import sys +import os +import subprocess +import argparse +from argparse import RawTextHelpFormatter +import re +import datetime +try: + from parsley import makeGrammar +except ImportError: + print('This script uses the package Parsley to parse C Mapfiles.\n' + 'This can be installed with \"pip install parsley".') + sys.exit() + +DESCRIPTION = ''' +This script tracks the growth of stable and experimental symbols +over releases since v19.11. The script has the ability to +count the added symbols between two dpdk releases, and to +list experimental symbols present in two dpdk releases +(expired symbols), including the name & email of the original contributor. + +example usages: + +Count symbols added since v19.11 +$ {s} count-symbols + +Count symbols added since v20.11 +$ {s} count-symbols --releases v20.11,v21.05 + +List experimental symbols present in v20.11 and v21.05 +$ {s} list-expired --releases v20.11,v21.05 + +List experimental symbols in libraries only, present since v19.11 +$ {s} list-expired --directory lib +''' + +MAP_GRAMMAR = r""" + +ws = (' ' | '\r' | '\n' | '\t')* + +ABI_VER = ({}) +DPDK_VER = ('DPDK_' ABI_VER) +ABI_NAME = ('INTERNAL' | 'EXPERIMENTAL' | DPDK_VER) +comment = '#' (~'\n' anything)+ '\n' +symbol = (~(';' | '}}' | '#') anything )+:c ';' -> ''.join(c) +global = 'global:' +local = 'local: *;' +symbols = comment* symbol:s ws comment* -> s + +abi = (abi_section+):m -> dict(m) +abi_section = (ws ABI_NAME:e ws '{{' ws global* (~local ws symbols)*:s ws local* ws '}}' ws DPDK_VER* ';' ws) -> (e,s) +""" # noqa: E501 + + +class EnvironException(Exception): + '''Subclass exception for Pylint\'s happiness.''' + + +def get_abi_versions(): + '''Returns a string of possible dpdk abi versions''' + + year = datetime.date.today().year - 2000 + tags = " |".join(['\'{}\''.format(i) + for i in reversed(range(21, year + 1))]) + tags = tags + ' | \'20.0.1\' | \'20.0\' | \'20\'' + + return tags + + +def get_dpdk_releases(): + '''Returns a list of dpdk release tags names since v19.11''' + + year = datetime.date.today().year - 2000 + year_range = "|".join("{}".format(i) for i in range(19, year + 1)) + pattern = re.compile(r'^\"v(' + year_range + r')\.\d{2}\"$') + + cmd = ['git', 'for-each-ref', '--sort=taggerdate', '--format', '"%(tag)"'] + try: + result = subprocess.run(cmd, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=True) + except subprocess.CalledProcessError: + print("Failed to interogate git for release tags") + sys.exit() + + tags = result.stdout.decode('utf-8').split('\n') + + # find the non-rcs between now and v19.11 + tags = [tag.replace('\"', '') + for tag in reversed(tags) + if pattern.match(tag)][:-3] + + return tags + + +def fix_directory_name(path): + '''Prepend librte to the source directory name''' + mapfilepath1 = str(path.parent.name) + mapfilepath2 = str(path.parents[1]) + mapfilepath = mapfilepath2 + '/librte_' + mapfilepath1 + + return mapfilepath + + +def directory_renamed(path, rel): + '''Fix removal of the librte_ from the directory names''' + + mapfilepath = fix_directory_name(path) + tagfile = '{}:{}/{}'.format(rel, mapfilepath, path.name) + + try: + result = subprocess.run(['git', 'show', tagfile], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=True) + except subprocess.CalledProcessError: + result = None + + return result + + +def mapfile_renamed(path, rel): + '''Fix renaming of the map file''' + newfile = None + + result = subprocess.run(['git', 'ls-tree', + rel, str(path.parent) + '/'], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=True) + dentries = result.stdout.decode('utf-8') + dentries = dentries.split('\n') + + # filter entries looking for the map file + dentries = [dentry for dentry in dentries if dentry.endswith('.map')] + if len(dentries) > 1 or len(dentries) == 0: + return None + + dparts = dentries[0].split('/') + newfile = dparts[len(dparts) - 1] + + if newfile is not None: + tagfile = '{}:{}/{}'.format(rel, path.parent, newfile) + + try: + result = subprocess.run(['git', 'show', tagfile], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=True) + except subprocess.CalledProcessError: + result = None + + else: + result = None + + return result + + +def mapfile_and_directory_renamed(path, rel): + '''Fix renaming of the map file & the source directory''' + mapfilepath = Path("{}/{}".format(fix_directory_name(path), path.name)) + + return mapfile_renamed(mapfilepath, rel) + + +FIX_STRATEGIES = [directory_renamed, + mapfile_renamed, + mapfile_and_directory_renamed] + + +def get_symbols(map_parser, release, mapfile_path): + '''Count the symbols for a given release and mapfile''' + abi_sections = {} + + tagfile = '{}:{}'.format(release, mapfile_path) + try: + result = subprocess.run(['git', 'show', tagfile], + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=True) + except subprocess.CalledProcessError: + result = None + + for fix_strategy in FIX_STRATEGIES: + if result is not None: + break + result = fix_strategy(mapfile_path, release) + + if result is not None: + mapfile = result.stdout.decode('utf-8') + abi_sections = map_parser(mapfile).abi() + + return abi_sections + + +def get_terminal_rows(): + '''Find the number of rows in the terminal''' + + try: + return os.get_terminal_size().lines + except IOError: + return 0 + + +class IgnoredSymbols(): # pylint: disable=too-few-public-methods + '''Symbols which are to be ignored for some period''' + + SYMBOL_TOOL_IGNORE = 'devtools/symboltool.abignore' + ignore_regex = [] + __initialized = False + + @staticmethod + def initialize(): + '''initialize once''' + + if IgnoredSymbols.__initialized: + return + IgnoredSymbols.__initialized = True + + if 'DPDK_SYMBOL_TOOL_IGNORE' in os.environ: + IgnoredSymbols.SYMBOL_TOOL_IGNORE = \ + os.environ['DPDK_SYMBOL_TOOL_IGNORE'] + + # if the user specifies an ignore file, we can't find then error. + if not Path(IgnoredSymbols.SYMBOL_TOOL_IGNORE).is_file(): + raise EnvironException('Cannot locate {}\'s ' + 'ignore file'.format(__file__)) + + # if we cannot find the default ignore file, then continue + if not Path(IgnoredSymbols.SYMBOL_TOOL_IGNORE).is_file(): + return + + lines = open(Path(IgnoredSymbols.SYMBOL_TOOL_IGNORE)).readlines() + for line in lines: + + line = line.strip() + + # ignore comments and whitespace + if line.startswith(';') or len(line) == 0: + continue + + IgnoredSymbols.ignore_regex.append(re.compile(line)) + + def __init__(self): + self.initialize() + + def check_ignore(self, symbol): + '''Check symbol against the ignore regexes''' + + for exp in self.ignore_regex: + if exp.search(symbol) is not None: + return True + + return False + + +class SymbolOwner(): + '''Find the symbols original contributors name and email''' + symbol_regex = {} + blame_regex = {'name': r'author\s(.*)', + 'email': r'author-mail\s<(.*)>'} + + def __init__(self, libpath, symbol): + self.libpath = libpath + self.symbol = symbol + + # find variable definitions in C files, and functions in headers. + self.symbol_regex = \ + {'*.c': r'^(?!extern).*' + self.symbol + '[^()]*;', + '*.h': r'__rte_experimental(?:.*\n){0,2}.*' + self.symbol} + + def find_symbol_location(self): + '''Find where the symbol is definited in the source''' + for key in self.symbol_regex: + for path in Path(self.libpath).rglob(key): + file_text = open(path).read() + + # find where the symbol is defined, either preceded by + # rte_experimental tag (functions) + # or followed by a ; (variables) + + exp = self.symbol_regex[key] + pattern = re.compile(exp, re.MULTILINE) + search = pattern.search(file_text) + + if search is not None: + symbol_pos = search.span()[1] + symbol_line = file_text.count('\n', 0, symbol_pos) + 1 + + return [str(path), symbol_line] + return None + + def find_symbol_owner(self): + '''Find the symbols original contributors name and email''' + owners = {} + location = self.find_symbol_location() + + if location is None: + return None + + line = '-L {},{}'.format(location[1], location[1]) + # git blame -p(orcelain) -L(ine) path + args = ['-p', line, location[0]] + + try: + result = subprocess.run(['git', 'blame'] + args, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=True) + except subprocess.CalledProcessError: + return None + + blame = result.stdout.decode('utf-8') + for key in self.blame_regex: + pattern = re.compile(self.blame_regex[key], re.MULTILINE) + match = pattern.search(blame) + + owners[key] = match.groups()[0] + + return owners + + +class SymbolCountOutput(): + '''Format the output to supported formats''' + output_fmt = "" + column_fmt = "" + + def __init__(self, format_output, dpdk_releases): + self.OUTPUT_FORMATS[format_output](self, dpdk_releases) + self.column_titles = ['mapfile'] + dpdk_releases + + self.terminal_rows = get_terminal_rows() + self.row = 0 + + def set_terminal_output(self, dpdk_rel): + '''Set the output format to Tabbed Separated Values''' + + self.output_fmt = '{:<50}' + \ + ''.join(['{:<6}{:<6}'] * (len(dpdk_rel))) + self.column_fmt = '{:50}' + \ + ''.join(['{:<12}'] * (len(dpdk_rel))) + + def set_csv_output(self, dpdk_rel): + '''Set the output format to Comma Separated Values''' + + self.output_fmt = '{},' + \ + ','.join(['{},{}'] * (len(dpdk_rel))) + self.column_fmt = '{},' + \ + ','.join(['{},'] * (len(dpdk_rel))) + + def print_columns(self): + '''Print column rows with release names''' + print(self.column_fmt.format(*self.column_titles)) + self.row += 1 + + def print_row(self, mapfile, symbols): + '''Print row of symbol values''' + mapfile = str(mapfile) + print(self.output_fmt.format(*([mapfile] + symbols))) + self.row += 1 + + if((self.terminal_rows > 0) and + ((self.row % self.terminal_rows) == 0)): + self.print_columns() + + OUTPUT_FORMATS = {None: set_terminal_output, + 'terminal': set_terminal_output, + 'csv': set_csv_output} + + +class ListExpiredOutput(): + '''Format the output to supported formats''' + output_fmt = "" + column_fmt = "" + + def __init__(self, format_output, dpdk_releases): + self.terminal = True + self.OUTPUT_FORMATS[format_output](self, dpdk_releases) + self.column_titles = ['mapfile'] + \ + ['expired (' + ','.join(dpdk_releases) + ')'] + \ + ['contributor name', 'contributor email'] + + def set_terminal_output(self, _): + '''Set the output format to Tabbed Separated Values''' + + self.output_fmt = '{:<50}{:<50}{:<25}{:<25}' + self.column_fmt = '{:50}{:50}{:25}{:25}' + + def set_csv_output(self, _): + '''Set the output format to Comma Separated Values''' + + self.output_fmt = '{},{},{},{}' + self.column_fmt = '{},{},{},{}' + self.terminal = False + + def print_columns(self): + '''Print column rows with release names''' + print(self.column_fmt.format(*self.column_titles)) + + def print_row(self, mapfile, symbols, owner): + '''Print row of symbol values''' + + for symbol in symbols: + mapfile = str(mapfile) + name = owner[symbol]['name'] \ + if owner[symbol] is not None else '' + email = owner[symbol]['email'] \ + if owner[symbol] is not None else '' + + print(self.output_fmt.format(mapfile, symbol, name, email)) + if self.terminal: + mapfile = '' + + OUTPUT_FORMATS = {None: set_terminal_output, + 'terminal': set_terminal_output, + 'csv': set_csv_output} + + +class CountSymbolsAction: + ''' Logic to count symbols added since a give release ''' + IGNORE_SECTIONS = ['EXPERIMENTAL', 'INTERNAL'] + + def __init__(self, mapfile_path, mapfile_parser, format_output): + self.path = mapfile_path + self.parser = mapfile_parser + self.format_output = format_output + self.symbols_count = [] + + def add_mapfile(self, release): + ''' add a version mapfile ''' + symbol_count = experimental_count = 0 + + symbols = get_symbols(self.parser, release, self.path) + + # which versions are present, and we care about + abi_vers = [abi_ver + for abi_ver in symbols + if abi_ver not in self.IGNORE_SECTIONS] + + for abi_ver in abi_vers: + symbol_count += len(symbols[abi_ver]) + + # count experimental symbols + if 'EXPERIMENTAL' in symbols.keys(): + experimental_count = len(symbols['EXPERIMENTAL']) + + self.symbols_count += [symbol_count, experimental_count] + + def __del__(self): + self.format_output.print_row(self.path.parent, self.symbols_count) + + +class ListExpiredAction: + ''' Logic to list expired symbols between two releases ''' + + def __init__(self, mapfile_path, mapfile_parser, format_output): + self.path = mapfile_path + self.parser = mapfile_parser + self.format_output = format_output + self.experimental_symbols = [] + self.ignored_symbols = IgnoredSymbols() + + def add_mapfile(self, release): + ''' add a version mapfile ''' + symbols = get_symbols(self.parser, release, self.path) + + if 'EXPERIMENTAL' in symbols.keys(): + experimental = [exp.strip() for exp in symbols['EXPERIMENTAL']] + + self.experimental_symbols.append(experimental) + + def __del__(self): + if len(self.experimental_symbols) != 2: + return + + tmp = self.experimental_symbols + # find symbols present in both dpdk releases + intersect_syms = [sym for sym in tmp[0] if sym in tmp[1]] + + # remove ignored symbols + intersect_syms = [sym for sym in intersect_syms if not + self.ignored_symbols.check_ignore(sym)] + + # check for empty set + if intersect_syms == []: + return + + sym_owner = {} + for sym in intersect_syms: + sym_owner[sym] = \ + SymbolOwner(self.path.parent, sym).find_symbol_owner() + + self.format_output.print_row(self.path.parent, + intersect_syms, + sym_owner) + + +SRC_DIRECTORIES = 'drivers,lib' + +ACTIONS = {None: CountSymbolsAction, + 'count-symbols': CountSymbolsAction, + 'list-expired': ListExpiredAction} + +ACTION_OUTPUT = {None: SymbolCountOutput, + 'count-symbols': SymbolCountOutput, + 'list-expired': ListExpiredOutput} + + +def main(): + '''Main entry point''' + + dpdk_releases = get_dpdk_releases() + + parser = \ + argparse.ArgumentParser(description=DESCRIPTION.format(s=__file__), + formatter_class=RawTextHelpFormatter) + + parser.add_argument('mode', choices=['count-symbols', 'list-expired']) + parser.add_argument('--format-output', choices=['terminal', 'csv'], + default='terminal') + parser.add_argument('--directory', choices=SRC_DIRECTORIES.split(','), + default=SRC_DIRECTORIES) + parser.add_argument('--releases', + help='2 x comma separated release tags e.g. \'' + + ','.join([dpdk_releases[0], dpdk_releases[-1]]) + + '\'') + args = parser.parse_args() + + if args.releases is not None: + dpdk_releases = args.releases.split(',') + + if args.mode == 'list-expired': + if len(dpdk_releases) < 2: + sys.exit('Please specify two releases to compare ' + 'in \'list-expired\' mode.') + dpdk_releases = [dpdk_releases[0], + dpdk_releases[len(dpdk_releases) - 1]] + + action = ACTIONS[args.mode] + format_output = ACTION_OUTPUT[args.mode](args.format_output, dpdk_releases) + + map_grammar = MAP_GRAMMAR.format(get_abi_versions()) + map_parser = makeGrammar(map_grammar, {}) + + format_output.print_columns() + + for src_dir in args.directory.split(','): + for path in Path(src_dir).rglob('*.map'): + release_action = action(path, map_parser, format_output) + + for release in dpdk_releases: + release_action.add_mapfile(release) + + # all the magic happens in the destructor + del release_action + + +if __name__ == '__main__': + main() From patchwork Thu Sep 9 13:48:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ray Kinsella X-Patchwork-Id: 98438 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 56E49A0547; Thu, 9 Sep 2021 15:50:41 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3E49641149; Thu, 9 Sep 2021 15:50:32 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 8A4C241162 for ; Thu, 9 Sep 2021 15:50:30 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10101"; a="200318423" X-IronPort-AV: E=Sophos;i="5.85,280,1624345200"; d="scan'208";a="200318423" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2021 06:50:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,280,1624345200"; d="scan'208";a="580894512" Received: from silpixa00396680.ir.intel.com (HELO silpixa00396680.ger.corp.intel.com) ([10.237.223.54]) by orsmga004.jf.intel.com with ESMTP; 09 Sep 2021 06:50:27 -0700 From: Ray Kinsella To: dev@dpdk.org Cc: bruce.richardson@intel.com, stephen@networkplumber.org, ferruh.yigit@intel.com, thomas@monjalon.net, ktraynor@redhat.com, mdr@ashroe.eu, aconole@redhat.com, roy.fan.zhang@intel.com, arkadiuszx.kusztal@intel.com, gakhil@marvell.com Date: Thu, 9 Sep 2021 14:48:06 +0100 Message-Id: <20210909134808.1585777-3-mdr@ashroe.eu> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210909134808.1585777-1-mdr@ashroe.eu> References: <20210618163659.85933-1-mdr@ashroe.eu> <20210909134808.1585777-1-mdr@ashroe.eu> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v13 2/4] devtools: script to send notifications of expired symbols X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Use this script with the output of the DPDK symbol tool, to notify maintainers of expired symbols by email. You need to define the environment variable DPDK_GETMAINTAINER_PATH for this tool to work. Use terminal output to review the emails before sending. e.g. $ devtools/symbol-tool.py list-expired --format-output csv \ | DPDK_GETMAINTAINER_PATH=/get_maintainer.pl \ devtools/notify_expired_symbols.py --format-output terminal Then use email output to send the emails to the maintainers. e.g. $ devtools/symbol-tool.py list-expired --format-output csv \ | DPDK_GETMAINTAINER_PATH=/get_maintainer.pl \ devtools/notify_expired_symbols.py --format-output email \ --smtp-server --sender \ --password --cc Signed-off-by: Ray Kinsella --- devtools/notify-symbol-maintainers.py | 302 ++++++++++++++++++++++++++ 1 file changed, 302 insertions(+) create mode 100755 devtools/notify-symbol-maintainers.py diff --git a/devtools/notify-symbol-maintainers.py b/devtools/notify-symbol-maintainers.py new file mode 100755 index 0000000000..edf330f88b --- /dev/null +++ b/devtools/notify-symbol-maintainers.py @@ -0,0 +1,302 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2021 Intel Corporation +# pylint: disable=invalid-name +'''Tool to notify maintainers of expired symbols''' +import os +import smtplib +import ssl +import sys +import subprocess +import argparse +from argparse import RawTextHelpFormatter +import time +from email.message import EmailMessage +from pathlib import Path + +DESCRIPTION = ''' +Use this script with the output of the DPDK symbol tool, to notify maintainers +and contributors of expired symbols by email. You need to define the environment +variable DPDK_GETMAINTAINER_PATH for this tool to work. + +Use terminal output to review the emails before sending. +e.g. +$ devtools/symbol-tool.py list-expired --format-output csv \\ +| DPDK_GETMAINTAINER_PATH=/get_maintainer.pl \\ +{s} --format-output terminal + +Then use email output to send the emails to the maintainers. +e.g. +$ devtools/symbol-tool.py list-expired --format-output csv \\ +| DPDK_GETMAINTAINER_PATH=/get_maintainer.pl \\ +{s} --format-output email \\ +--smtp-server --sender --password \\ +--cc +''' # noqa: E501 + +EMAIL_TEMPLATE = '''Hi there, + +Please note the symbols listed below have expired. In line with the DPDK ABI +policy, they should be scheduled for removal, in the next DPDK release. + +For more information, please see the DPDK ABI Policy, section 3.5.3. +https://doc.dpdk.org/guides/contributing/abi_policy.html + +Thanks, + +The DPDK Symbol Bot + +''' # noqa: E501 + +ABI_POLICY = 'doc/guides/contributing/abi_policy.rst' +DPDK_GMP_ENV_VAR = 'DPDK_GETMAINTAINER_PATH' +MAINTAINERS = 'MAINTAINERS' +get_maintainer = ['devtools/get-maintainer.sh', + '--email', '-f'] + + +class EnvironException(Exception): + '''Subclass exception for Pylint\'s happiness.''' + + +def _die_on_exception(e): + '''Print an exception, and quit''' + + print('Fatal Error: ' + str(e)) + sys.exit() + + +def _check_get_maintainers_env(): + '''Check get maintainers scripts are setup''' + + if not Path(get_maintainer[0]).is_file(): + raise EnvironException('Cannot locate DPDK\'s get maintainers script, ' + ' usually at $' + get_maintainer[0] + '.') + + if DPDK_GMP_ENV_VAR not in os.environ: + raise EnvironException(DPDK_GMP_ENV_VAR + ' is not defined.') + + if not Path(os.environ[DPDK_GMP_ENV_VAR]).is_file(): + raise EnvironException('Cannot locate get maintainers script, usually' + ' at ' + DPDK_GMP_ENV_VAR + '.') + + +def _get_maintainers(libpath): + '''Get the maintainers for given library''' + + try: + _check_get_maintainers_env() + except EnvironException as e: + _die_on_exception(e) + + try: + cmd = get_maintainer + [libpath] + result = subprocess.run(cmd, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=True) + except subprocess.CalledProcessError as e: + _die_on_exception(e) + + if result is None: + return None + + email = result.stdout.decode('utf-8') + if email == '': + return None + + email = list(filter(None, email.split('\n'))) + return email + + +default_maintainers = _get_maintainers(ABI_POLICY) + \ + _get_maintainers(MAINTAINERS) + + +def get_maintainers(libpath): + '''Get the maintainers for given library''' + maintainers = _get_maintainers(libpath) + + if maintainers is None: + maintainers = default_maintainers + + return maintainers + + +def get_message(library, symbols, config): + '''Build email message from symbols, config and maintainers''' + contributors = {} + message = {} + maintainers = get_maintainers(library) + + if maintainers != default_maintainers: + message['CC'] = default_maintainers.copy() + + if 'CC' in config: + message.setdefault('CC', []).append(config['CC']) + + message['Subject'] = 'Expired symbols in {}\n'.format(library) + + body = EMAIL_TEMPLATE + body += '{:<50}{:<25}{:<25}\n'.format('Symbol', 'Contributor', 'Email') + for sym in symbols: + body += ('{:<50}{:<25}{:<25}\n'.format(sym, + symbols[sym]['name'], + symbols[sym]['email'])) + email = symbols[sym]['email'] + contributors[email] = '' + + contributors = list(contributors.keys()) + + message['To'] = maintainers + contributors + message['Body'] = body + + return message + + +class OutputEmail(): + '''Format the output for email''' + + def __init__(self, config): + self.config = config + + self.terminal = OutputTerminal(config) + context = ssl.create_default_context() + + # Try to log in to server and send email + try: + self.server = smtplib.SMTP(config['smtp_server'], 587) + self.server.starttls(context=context) # Secure the connection + self.server.login(config['sender'], config['password']) + except EnvironException as e: + _die_on_exception(e) + + def message(self, message): + '''send email''' + self.terminal.message(message) + + msg = EmailMessage() + msg.set_content(message.pop('Body')) + + for key in message.keys(): + msg[key] = message[key] + + msg['From'] = self.config['sender'] + msg['Reply-To'] = 'no-reply@dpdk.org' + + self.server.send_message(msg) + + time.sleep(1) + + def __del__(self): + self.server.quit() + + +class OutputTerminal(): # pylint: disable=too-few-public-methods + '''Format the output for the terminal''' + + def __init__(self, config): + self.config = config + + def message(self, message): + '''Print email to terminal''' + + terminal = 'To:' + ', '.join(message['To']) + '\n' + if 'sender' in self.config.keys(): + terminal += 'From:' + self.config['sender'] + '\n' + + terminal += 'Reply-To:' + 'no-reply@dpdk.org' + '\n' + + if 'CC' in message: + terminal += 'CC:' + ', '.join(message['CC']) + '\n' + + terminal += 'Subject:' + message['Subject'] + '\n' + terminal += 'Body:' + message['Body'] + '\n' + + print(terminal) + print('-' * 80) + + +def parse_config(args): + '''put the command line args in the right places''' + config = {} + error_msg = None + + outputs = { + None: OutputTerminal, + 'terminal': OutputTerminal, + 'email': OutputEmail + } + + if args.format_output == 'email': + if args.smtp_server is None: + error_msg = 'SMTP server' + else: + config['smtp_server'] = args.smtp_server + + if args.sender is None: + error_msg = 'sender' + else: + config['sender'] = args.sender + + if args.password is None: + error_msg = 'password' + else: + config['password'] = args.password + + if args.cc is not None: + config['CC'] = args.cc + + if error_msg is not None: + print('Please specify a {} for email output'.format(error_msg)) + return None + + config['output'] = outputs[args.format_output] + return config + + +def main(): + '''Main entry point''' + parser = \ + argparse.ArgumentParser(description=DESCRIPTION.format(s=__file__), + formatter_class=RawTextHelpFormatter) + parser.add_argument('--format-output', + choices=['terminal', 'email'], + default='terminal') + parser.add_argument('--smtp-server') + parser.add_argument('--password') + parser.add_argument('--sender') + parser.add_argument('--cc') + + args = parser.parse_args() + config = parse_config(args) + if config is None: + return + + symbols = {} + lastlib = library = '' + + output = config['output'](config) + + for line in sys.stdin: + line = line.rstrip('\n') + + if line.find('mapfile') >= 0: + continue + library, symbol, name, email = line.split(',') + + if library != lastlib: + message = get_message(lastlib, symbols, config) + output.message(message) + symbols = {} + + lastlib = library + symbols[symbol] = {'name': name, 'email': email} + + # print the last library + message = get_message(lastlib, symbols, config) + output.message(message) + + +if __name__ == '__main__': + main() From patchwork Thu Sep 9 13:48:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ray Kinsella X-Patchwork-Id: 98439 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id ACEC1A0547; Thu, 9 Sep 2021 15:50:47 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id CB95941170; Thu, 9 Sep 2021 15:50:34 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id CD5A14116D for ; Thu, 9 Sep 2021 15:50:32 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10101"; a="200318432" X-IronPort-AV: E=Sophos;i="5.85,280,1624345200"; d="scan'208";a="200318432" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2021 06:50:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,280,1624345200"; d="scan'208";a="580894515" Received: from silpixa00396680.ir.intel.com (HELO silpixa00396680.ger.corp.intel.com) ([10.237.223.54]) by orsmga004.jf.intel.com with ESMTP; 09 Sep 2021 06:50:30 -0700 From: Ray Kinsella To: dev@dpdk.org Cc: bruce.richardson@intel.com, stephen@networkplumber.org, ferruh.yigit@intel.com, thomas@monjalon.net, ktraynor@redhat.com, mdr@ashroe.eu, aconole@redhat.com, roy.fan.zhang@intel.com, arkadiuszx.kusztal@intel.com, gakhil@marvell.com Date: Thu, 9 Sep 2021 14:48:07 +0100 Message-Id: <20210909134808.1585777-4-mdr@ashroe.eu> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210909134808.1585777-1-mdr@ashroe.eu> References: <20210618163659.85933-1-mdr@ashroe.eu> <20210909134808.1585777-1-mdr@ashroe.eu> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v13 3/4] maintainers: add new abi scripts X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add new abi management scripts to the MAINTAINERS file. Signed-off-by: Ray Kinsella --- MAINTAINERS | 3 +++ 1 file changed, 3 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 266f5ac1da..ae38af1b85 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -127,8 +127,11 @@ F: devtools/check-abi-version.sh F: devtools/check-symbol-change.sh F: devtools/gen-abi.sh F: devtools/libabigail.abignore +F: devtools/symboltool.abignore F: devtools/update-abi.sh F: devtools/update_version_map_abi.py +F: devtools/notify-symbol-maintainers.py +F: devtools/symbol-tool.py F: buildtools/check-symbols.sh F: buildtools/map-list-symbol.sh F: drivers/*/*/*.map From patchwork Thu Sep 9 13:48:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ray Kinsella X-Patchwork-Id: 98440 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B2F3CA0547; Thu, 9 Sep 2021 15:50:52 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DD73541178; Thu, 9 Sep 2021 15:50:36 +0200 (CEST) Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mails.dpdk.org (Postfix) with ESMTP id 2F15C41175 for ; Thu, 9 Sep 2021 15:50:35 +0200 (CEST) X-IronPort-AV: E=McAfee;i="6200,9189,10101"; a="200318441" X-IronPort-AV: E=Sophos;i="5.85,280,1624345200"; d="scan'208";a="200318441" Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Sep 2021 06:50:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,280,1624345200"; d="scan'208";a="580894518" Received: from silpixa00396680.ir.intel.com (HELO silpixa00396680.ger.corp.intel.com) ([10.237.223.54]) by orsmga004.jf.intel.com with ESMTP; 09 Sep 2021 06:50:32 -0700 From: Ray Kinsella To: dev@dpdk.org Cc: bruce.richardson@intel.com, stephen@networkplumber.org, ferruh.yigit@intel.com, thomas@monjalon.net, ktraynor@redhat.com, mdr@ashroe.eu, aconole@redhat.com, roy.fan.zhang@intel.com, arkadiuszx.kusztal@intel.com, gakhil@marvell.com Date: Thu, 9 Sep 2021 14:48:08 +0100 Message-Id: <20210909134808.1585777-5-mdr@ashroe.eu> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210909134808.1585777-1-mdr@ashroe.eu> References: <20210618163659.85933-1-mdr@ashroe.eu> <20210909134808.1585777-1-mdr@ashroe.eu> MIME-Version: 1.0 Subject: [dpdk-dev] [PATCH v13 4/4] devtools: add asym crypto to symbol-tool ignore X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add asym crypto to the symbol-tool's ignore file. Signed-off-by: Ray Kinsella Acked-by: Akhil Goyal --- devtools/symboltool.abignore | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 devtools/symboltool.abignore diff --git a/devtools/symboltool.abignore b/devtools/symboltool.abignore new file mode 100644 index 0000000000..800c500a82 --- /dev/null +++ b/devtools/symboltool.abignore @@ -0,0 +1,3 @@ +; regex of symbols for the symbol-tool to ignore + +rte_cryptodev_asym_.*