From patchwork Fri Jan 14 16:23:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Morrissey X-Patchwork-Id: 105817 X-Patchwork-Delegate: thomas@monjalon.net Return-Path: X-Original-To: patchwork@inbox.dpdk.org Delivered-To: patchwork@inbox.dpdk.org Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id C8B9EA00C3; Fri, 14 Jan 2022 17:27:07 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9801442780; Fri, 14 Jan 2022 17:26:58 +0100 (CET) Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mails.dpdk.org (Postfix) with ESMTP id E127341163 for ; Fri, 14 Jan 2022 17:26:55 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1642177616; x=1673713616; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LBav9WjVXoaC8qdhePzJwt4fEHVKUj6RGWv6FDYI/8s=; b=cKAXxIzQmwGJI1ssFD+m/W8Egb5pksguwMlwTb5d0wWsRSROANIZmz0U Pf8G/YxDyVoTsTewLuXDCh0YRpM9pHrvjfzQajapRe9vRY10wvk2EyD4n +r8jcxTxbczWKK1/BIrWTdOh53vxQCceZYvgVWnXekUGFS2aLTsuUr/47 hvKFlm1rdSXoz9OQFeFeIibLlr61jnGf5PSZw/XanXannfSsZQ+z0dLRS UymdkGoHgqYwzYBtX5F+O8cKcski0eBCr22XSgukDQ/cNvudOaqLN16qW fwQETp27MMzzXYXiba1BBuuEnOfIvO6ozP1fSCtkkZyHIWVwCaXfR648P Q==; X-IronPort-AV: E=McAfee;i="6200,9189,10226"; a="268642522" X-IronPort-AV: E=Sophos;i="5.88,289,1635231600"; d="scan'208";a="268642522" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 Jan 2022 08:26:53 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.88,289,1635231600"; d="scan'208";a="559523366" Received: from silpixa00401215.ir.intel.com ([10.55.128.96]) by orsmga001.jf.intel.com with ESMTP; 14 Jan 2022 08:26:52 -0800 From: Sean Morrissey To: Cc: dev@dpdk.org, Sean Morrissey , Conor Fogarty , Bruce Richardson Subject: [PATCH v4 01/53] devtools: script to remove unused headers includes Date: Fri, 14 Jan 2022 16:23:17 +0000 Message-Id: <20220114162409.334437-2-sean.morrissey@intel.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220114162409.334437-1-sean.morrissey@intel.com> References: <20211007102557.188739-1-sean.morrissey@intel.com> <20220114162409.334437-1-sean.morrissey@intel.com> MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org This script can be used for removing headers flagged for removal by the include-what-you-use (IWYU) tool. The script has the ability to remove headers from specified sub-directories or dpdk as a whole and tests the build after each removal by calling meson compile. example usages: Remove headers flagged by iwyu_tool output file $ ./devtools/process_iwyu.py iwyu.out -b build Remove headers flagged by iwyu_tool output file from sub-directory $ ./devtools/process_iwyu.py iwyu.out -b build -d lib/kvargs Remove headers directly piped from the iwyu_tool $ iwyu_tool -p build | ./devtools/process_iwyu.py - -b build Signed-off-by: Sean Morrissey Signed-off-by: Conor Fogarty Reviewed-by: Bruce Richardson --- devtools/process_iwyu.py | 109 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100755 devtools/process_iwyu.py diff --git a/devtools/process_iwyu.py b/devtools/process_iwyu.py new file mode 100755 index 0000000000..50f3d4c5c7 --- /dev/null +++ b/devtools/process_iwyu.py @@ -0,0 +1,109 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: BSD-3-Clause +# Copyright(c) 2021 Intel Corporation +# + +import argparse +import fileinput +import sys +from os.path import abspath, relpath, join +from pathlib import Path +from mesonbuild import mesonmain + + +def args_parse(): + "parse arguments and return the argument object back to main" + parser = argparse.ArgumentParser(description="This script can be used to remove includes which are not in use\n") + parser.add_argument('-b', '--build_dir', type=str, default='build', + help="The path to the build directory in which the IWYU tool was used in.") + parser.add_argument('-d', '--sub_dir', type=str, default='', + help="The sub-directory to remove headers from.") + parser.add_argument('file', type=Path, + help="The path to the IWYU log file or output from stdin.") + + return parser.parse_args() + + +def run_meson(args): + "Runs a meson command logging output to process.log" + with open('process_iwyu.log', 'a') as sys.stdout: + ret = mesonmain.run(args, abspath('meson')) + sys.stdout = sys.__stdout__ + return ret + + +def remove_includes(filepath, include, build_dir): + "Attempts to remove include, if it fails then revert to original state" + with open(filepath) as f: + lines = f.readlines() # Read lines when file is opened + + with open(filepath, 'w') as f: + for ln in lines: # Removes the include passed in + if not ln.startswith(include): + f.write(ln) + + # run test build -> call meson on the build folder, meson compile -C build + ret = run_meson(['compile', '-C', build_dir]) + if (ret == 0): # Include is not needed -> build is successful + print('SUCCESS') + else: + # failed, catch the error + # return file to original state + with open(filepath, 'w') as f: + f.writelines(lines) + print('FAILED') + + +def get_build_config(builddir, condition): + "returns contents of rte_build_config.h" + with open(join(builddir, 'rte_build_config.h')) as f: + return [ln for ln in f.readlines() if condition(ln)] + + +def uses_libbsd(builddir): + "return whether the build uses libbsd or not" + return bool(get_build_config(builddir, lambda ln: 'RTE_USE_LIBBSD' in ln)) + + +def process(args): + "process the iwyu output on a set of files" + filepath = None + build_dir = abspath(args.build_dir) + directory = args.sub_dir + + print("Warning: The results of this script may include false positives which are required for different systems", + file=sys.stderr) + + keep_str_fns = uses_libbsd(build_dir) # check for libbsd + if keep_str_fns: + print("Warning: libbsd is present, build will fail to detect incorrect removal of rte_string_fns.h", + file=sys.stderr) + # turn on werror + run_meson(['configure', build_dir, '-Dwerror=true']) + # Use stdin if no iwyu_tool out file given + for line in fileinput.input(args.file): + if 'should remove' in line: + # If the file path in the iwyu_tool output is an absolute path it + # means the file is outside of the dpdk directory, therefore ignore it. + # Also check to see if the file is within the specified sub directory. + filename = line.split()[0] + if (filename != abspath(filename) and + directory in filename): + filepath = relpath(join(build_dir, filename)) + elif line.startswith('-') and filepath: + include = '#include ' + line.split()[2] + print(f"Remove {include} from {filepath} ... ", end='', flush=True) + if keep_str_fns and '' in include: + print('skipped') + continue + remove_includes(filepath, include, build_dir) + else: + filepath = None + + +def main(): + process(args_parse()) + + +if __name__ == '__main__': + main()