Patch Detail
get:
Show a patch.
patch:
Update a patch.
put:
Update a patch.
GET /api/patches/48036/?format=api
{ "id": 48036, "url": "http://patchwork.dpdk.org/api/patches/48036/?format=api", "web_url": "http://patchwork.dpdk.org/project/dpdk/patch/88cced4f601c44c6203b9adb09abacdce0b3a260.1542122595.git.anatoly.burakov@intel.com/", "project": { "id": 1, "url": "http://patchwork.dpdk.org/api/projects/1/?format=api", "name": "DPDK", "link_name": "dpdk", "list_id": "dev.dpdk.org", "list_email": "dev@dpdk.org", "web_url": "http://core.dpdk.org", "scm_url": "git://dpdk.org/dpdk", "webscm_url": "http://git.dpdk.org/dpdk", "list_archive_url": "https://inbox.dpdk.org/dev", "list_archive_url_format": "https://inbox.dpdk.org/dev/{}", "commit_url_format": "" }, "msgid": "<88cced4f601c44c6203b9adb09abacdce0b3a260.1542122595.git.anatoly.burakov@intel.com>", "list_archive_url": "https://inbox.dpdk.org/dev/88cced4f601c44c6203b9adb09abacdce0b3a260.1542122595.git.anatoly.burakov@intel.com", "date": "2018-11-13T15:29:27", "name": "eal: clean up unused files on initialization", "commit_ref": null, "pull_url": null, "state": "superseded", "archived": true, "hash": "e675aaeb8129311e8d15529dcdc4faf1c386d883", "submitter": { "id": 4, "url": "http://patchwork.dpdk.org/api/people/4/?format=api", "name": "Anatoly Burakov", "email": "anatoly.burakov@intel.com" }, "delegate": null, "mbox": "http://patchwork.dpdk.org/project/dpdk/patch/88cced4f601c44c6203b9adb09abacdce0b3a260.1542122595.git.anatoly.burakov@intel.com/mbox/", "series": [ { "id": 2385, "url": "http://patchwork.dpdk.org/api/series/2385/?format=api", "web_url": "http://patchwork.dpdk.org/project/dpdk/list/?series=2385", "date": "2018-11-13T15:29:27", "name": "eal: clean up unused files on initialization", "version": 1, "mbox": "http://patchwork.dpdk.org/series/2385/mbox/" } ], "comments": "http://patchwork.dpdk.org/api/patches/48036/comments/", "check": "fail", "checks": "http://patchwork.dpdk.org/api/patches/48036/checks/", "tags": {}, "related": [], "headers": { "Return-Path": "<dev-bounces@dpdk.org>", "X-Original-To": "patchwork@dpdk.org", "Delivered-To": "patchwork@dpdk.org", "Received": [ "from [92.243.14.124] (localhost [127.0.0.1])\n\tby dpdk.org (Postfix) with ESMTP id 79A282D13;\n\tTue, 13 Nov 2018 16:29:32 +0100 (CET)", "from mga03.intel.com (mga03.intel.com [134.134.136.65])\n\tby dpdk.org (Postfix) with ESMTP id D8C0891;\n\tTue, 13 Nov 2018 16:29:30 +0100 (CET)", "from fmsmga006.fm.intel.com ([10.253.24.20])\n\tby orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;\n\t13 Nov 2018 07:29:29 -0800", "from irvmail001.ir.intel.com ([163.33.26.43])\n\tby fmsmga006.fm.intel.com with ESMTP; 13 Nov 2018 07:29:28 -0800", "from sivswdev01.ir.intel.com (sivswdev01.ir.intel.com\n\t[10.237.217.45])\n\tby irvmail001.ir.intel.com (8.14.3/8.13.6/MailSET/Hub) with ESMTP id\n\twADFTSEx005691; Tue, 13 Nov 2018 15:29:28 GMT", "from sivswdev01.ir.intel.com (localhost [127.0.0.1])\n\tby sivswdev01.ir.intel.com with ESMTP id wADFTSrb005316;\n\tTue, 13 Nov 2018 15:29:28 GMT", "(from aburakov@localhost)\n\tby sivswdev01.ir.intel.com with LOCAL id wADFTR9h005311;\n\tTue, 13 Nov 2018 15:29:27 GMT" ], "X-Amp-Result": "SKIPPED(no attachment in message)", "X-Amp-File-Uploaded": "False", "X-ExtLoop1": "1", "X-IronPort-AV": "E=Sophos;i=\"5.54,499,1534834800\"; d=\"scan'208\";a=\"280735912\"", "From": "Anatoly Burakov <anatoly.burakov@intel.com>", "To": "dev@dpdk.org", "Cc": "Bruce Richardson <bruce.richardson@intel.com>, stable@dpdk.org", "Date": "Tue, 13 Nov 2018 15:29:27 +0000", "Message-Id": "<88cced4f601c44c6203b9adb09abacdce0b3a260.1542122595.git.anatoly.burakov@intel.com>", "X-Mailer": "git-send-email 1.7.0.7", "Subject": "[dpdk-dev] [PATCH] eal: clean up unused files on initialization", "X-BeenThere": "dev@dpdk.org", "X-Mailman-Version": "2.1.15", "Precedence": "list", "List-Id": "DPDK patches and discussions <dev.dpdk.org>", "List-Unsubscribe": "<https://mails.dpdk.org/options/dev>,\n\t<mailto:dev-request@dpdk.org?subject=unsubscribe>", "List-Archive": "<http://mails.dpdk.org/archives/dev/>", "List-Post": "<mailto:dev@dpdk.org>", "List-Help": "<mailto:dev-request@dpdk.org?subject=help>", "List-Subscribe": "<https://mails.dpdk.org/listinfo/dev>,\n\t<mailto:dev-request@dpdk.org?subject=subscribe>", "Errors-To": "dev-bounces@dpdk.org", "Sender": "\"dev\" <dev-bounces@dpdk.org>" }, "content": "When creating process data structures, EAL will create many files\nin EAL runtime directory. Because we allow multiple secondary\nprocesses to run, each secondary process gets their own unique\nfile. With many secondary processes running and exiting on the\nsystem, runtime directory will, over time, create enormous amounts\nof sockets, fbarray files and other stuff that just sits there\nunused because the process that allocated it has died a long time\nago. This may lead to exhaustion of disk (or RAM) space in the\nruntime directory.\n\nFix this by removing every unlocked file at initialization that\nmatches either socket or fbarray naming convention. We cannot be\nsure of any other files, so we'll leave them alone. Also, remove\nsimilar code from mp socket code.\n\nWe do it at the end of init, rather than at the beginning, because\nsecondary process will use primary process' data structures even\nif the primary itself has died, and we don't want to remove those\nbefore we lock them.\n\nBugzilla ID: 106\n\nCc: stable@dpdk.org\n\nReported-by: Vipin Varghese <vipin.varghese@intel.com>\n\nSigned-off-by: Anatoly Burakov <anatoly.burakov@intel.com>\n---\n lib/librte_eal/bsdapp/eal/eal.c | 100 ++++++++++++++++++++++++\n lib/librte_eal/common/eal_common_proc.c | 30 -------\n lib/librte_eal/common/eal_filesystem.h | 3 +\n lib/librte_eal/linuxapp/eal/eal.c | 99 +++++++++++++++++++++++\n 4 files changed, 202 insertions(+), 30 deletions(-)", "diff": "diff --git a/lib/librte_eal/bsdapp/eal/eal.c b/lib/librte_eal/bsdapp/eal/eal.c\nindex 508cbc46f..0c43b1080 100644\n--- a/lib/librte_eal/bsdapp/eal/eal.c\n+++ b/lib/librte_eal/bsdapp/eal/eal.c\n@@ -3,6 +3,8 @@\n * Copyright(c) 2014 6WIND S.A.\n */\n \n+#include <dirent.h>\n+#include <fnmatch.h>\n #include <stdio.h>\n #include <stdlib.h>\n #include <stdint.h>\n@@ -141,6 +143,92 @@ eal_create_runtime_dir(void)\n \treturn 0;\n }\n \n+static int\n+eal_clean_runtime_dir(void)\n+{\n+\tDIR *dir;\n+\tstruct dirent *dirent;\n+\tint dir_fd, fd, lck_result;\n+\tstatic const char * const filters[] = {\n+\t\t\"fbarray_*\",\n+\t\t\"mp_socket_*\"\n+\t};\n+\n+\t/* open directory */\n+\tdir = opendir(runtime_dir);\n+\tif (!dir) {\n+\t\tRTE_LOG(ERR, EAL, \"Unable to open runtime directory %s\\n\",\n+\t\t\t\truntime_dir);\n+\t\tgoto error;\n+\t}\n+\tdir_fd = dirfd(dir);\n+\n+\t/* lock the directory before doing anything, to avoid races */\n+\tif (flock(dir_fd, LOCK_EX) < 0) {\n+\t\tRTE_LOG(ERR, EAL, \"Unable to lock runtime directory %s\\n\",\n+\t\t\truntime_dir);\n+\t\tgoto error;\n+\t}\n+\n+\tdirent = readdir(dir);\n+\tif (!dirent) {\n+\t\tRTE_LOG(ERR, EAL, \"Unable to read runtime directory %s\\n\",\n+\t\t\t\truntime_dir);\n+\t\tgoto error;\n+\t}\n+\n+\twhile (dirent != NULL) {\n+\t\tunsigned int f_idx;\n+\t\tbool skip = true;\n+\n+\t\t/* skip files that don't match the patterns */\n+\t\tfor (f_idx = 0; f_idx < RTE_DIM(filters); f_idx++) {\n+\t\t\tconst char *filter = filters[f_idx];\n+\n+\t\t\tif (fnmatch(filter, dirent->d_name, 0) == 0) {\n+\t\t\t\tskip = false;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t\tif (skip) {\n+\t\t\tdirent = readdir(dir);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* try and lock the file */\n+\t\tfd = openat(dir_fd, dirent->d_name, O_RDONLY);\n+\n+\t\t/* skip to next file */\n+\t\tif (fd == -1) {\n+\t\t\tdirent = readdir(dir);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* non-blocking lock */\n+\t\tlck_result = flock(fd, LOCK_EX | LOCK_NB);\n+\n+\t\t/* if lock succeeds, remove the file */\n+\t\tif (lck_result != -1)\n+\t\t\tunlinkat(dir_fd, dirent->d_name, 0);\n+\t\tclose(fd);\n+\t\tdirent = readdir(dir);\n+\t}\n+\n+\t/* closedir closes dir_fd and drops the lock */\n+\tclosedir(dir);\n+\treturn 0;\n+\n+error:\n+\tif (dir)\n+\t\tclosedir(dir);\n+\n+\tRTE_LOG(ERR, EAL, \"Error while clearing runtime dir: %s\\n\",\n+\t\tstrerror(errno));\n+\n+\treturn -1;\n+}\n+\n+\n const char *\n rte_eal_get_runtime_dir(void)\n {\n@@ -805,6 +893,18 @@ rte_eal_init(int argc, char **argv)\n \t\treturn -1;\n \t}\n \n+\t/*\n+\t * clean up unused files in runtime directory. we do this at the end of\n+\t * init and not at the beginning because we want to clean stuff up\n+\t * whether we are primary or secondary process, but we cannot remove\n+\t * primary process' files because secondary should be able to run even\n+\t * if primary process is dead.\n+\t */\n+\tif (eal_clean_runtime_dir() < 0) {\n+\t\trte_eal_init_alert(\"Cannot clear runtime directory\\n\");\n+\t\treturn -1;\n+\t}\n+\n \trte_eal_mcfg_complete();\n \n \t/* Call each registered callback, if enabled */\ndiff --git a/lib/librte_eal/common/eal_common_proc.c b/lib/librte_eal/common/eal_common_proc.c\nindex 97663d3ba..e66d7fafb 100644\n--- a/lib/librte_eal/common/eal_common_proc.c\n+++ b/lib/librte_eal/common/eal_common_proc.c\n@@ -542,29 +542,6 @@ open_socket_fd(void)\n \treturn mp_fd;\n }\n \n-static int\n-unlink_sockets(const char *filter)\n-{\n-\tint dir_fd;\n-\tDIR *mp_dir;\n-\tstruct dirent *ent;\n-\n-\tmp_dir = opendir(mp_dir_path);\n-\tif (!mp_dir) {\n-\t\tRTE_LOG(ERR, EAL, \"Unable to open directory %s\\n\", mp_dir_path);\n-\t\treturn -1;\n-\t}\n-\tdir_fd = dirfd(mp_dir);\n-\n-\twhile ((ent = readdir(mp_dir))) {\n-\t\tif (fnmatch(filter, ent->d_name, 0) == 0)\n-\t\t\tunlinkat(dir_fd, ent->d_name, 0);\n-\t}\n-\n-\tclosedir(mp_dir);\n-\treturn 0;\n-}\n-\n int\n rte_mp_channel_init(void)\n {\n@@ -603,13 +580,6 @@ rte_mp_channel_init(void)\n \t\treturn -1;\n \t}\n \n-\tif (rte_eal_process_type() == RTE_PROC_PRIMARY &&\n-\t\t\tunlink_sockets(mp_filter)) {\n-\t\tRTE_LOG(ERR, EAL, \"failed to unlink mp sockets\\n\");\n-\t\tclose(dir_fd);\n-\t\treturn -1;\n-\t}\n-\n \tif (open_socket_fd() < 0) {\n \t\tclose(dir_fd);\n \t\treturn -1;\ndiff --git a/lib/librte_eal/common/eal_filesystem.h b/lib/librte_eal/common/eal_filesystem.h\nindex b3e8ae5ea..1528282cf 100644\n--- a/lib/librte_eal/common/eal_filesystem.h\n+++ b/lib/librte_eal/common/eal_filesystem.h\n@@ -25,6 +25,9 @@\n int\n eal_create_runtime_dir(void);\n \n+int\n+eal_clean_runtime_dir(void);\n+\n /* returns runtime dir */\n const char *\n rte_eal_get_runtime_dir(void);\ndiff --git a/lib/librte_eal/linuxapp/eal/eal.c b/lib/librte_eal/linuxapp/eal/eal.c\nindex 361744d40..a1bc02dfa 100644\n--- a/lib/librte_eal/linuxapp/eal/eal.c\n+++ b/lib/librte_eal/linuxapp/eal/eal.c\n@@ -13,7 +13,9 @@\n #include <syslog.h>\n #include <getopt.h>\n #include <sys/file.h>\n+#include <dirent.h>\n #include <fcntl.h>\n+#include <fnmatch.h>\n #include <stddef.h>\n #include <errno.h>\n #include <limits.h>\n@@ -149,6 +151,91 @@ eal_create_runtime_dir(void)\n \treturn 0;\n }\n \n+int\n+eal_clean_runtime_dir(void)\n+{\n+\tDIR *dir;\n+\tstruct dirent *dirent;\n+\tint dir_fd, fd, lck_result;\n+\tstatic const char * const filters[] = {\n+\t\t\"fbarray_*\",\n+\t\t\"mp_socket_*\"\n+\t};\n+\n+\t/* open directory */\n+\tdir = opendir(runtime_dir);\n+\tif (!dir) {\n+\t\tRTE_LOG(ERR, EAL, \"Unable to open runtime directory %s\\n\",\n+\t\t\t\truntime_dir);\n+\t\tgoto error;\n+\t}\n+\tdir_fd = dirfd(dir);\n+\n+\t/* lock the directory before doing anything, to avoid races */\n+\tif (flock(dir_fd, LOCK_EX) < 0) {\n+\t\tRTE_LOG(ERR, EAL, \"Unable to lock runtime directory %s\\n\",\n+\t\t\truntime_dir);\n+\t\tgoto error;\n+\t}\n+\n+\tdirent = readdir(dir);\n+\tif (!dirent) {\n+\t\tRTE_LOG(ERR, EAL, \"Unable to read runtime directory %s\\n\",\n+\t\t\t\truntime_dir);\n+\t\tgoto error;\n+\t}\n+\n+\twhile (dirent != NULL) {\n+\t\tunsigned int f_idx;\n+\t\tbool skip = true;\n+\n+\t\t/* skip files that don't match the patterns */\n+\t\tfor (f_idx = 0; f_idx < RTE_DIM(filters); f_idx++) {\n+\t\t\tconst char *filter = filters[f_idx];\n+\n+\t\t\tif (fnmatch(filter, dirent->d_name, 0) == 0) {\n+\t\t\t\tskip = false;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t\tif (skip) {\n+\t\t\tdirent = readdir(dir);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* try and lock the file */\n+\t\tfd = openat(dir_fd, dirent->d_name, O_RDONLY);\n+\n+\t\t/* skip to next file */\n+\t\tif (fd == -1) {\n+\t\t\tdirent = readdir(dir);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\t/* non-blocking lock */\n+\t\tlck_result = flock(fd, LOCK_EX | LOCK_NB);\n+\n+\t\t/* if lock succeeds, remove the file */\n+\t\tif (lck_result != -1)\n+\t\t\tunlinkat(dir_fd, dirent->d_name, 0);\n+\t\tclose(fd);\n+\t\tdirent = readdir(dir);\n+\t}\n+\n+\t/* closedir closes dir_fd and drops the lock */\n+\tclosedir(dir);\n+\treturn 0;\n+\n+error:\n+\tif (dir)\n+\t\tclosedir(dir);\n+\n+\tRTE_LOG(ERR, EAL, \"Error while clearing runtime dir: %s\\n\",\n+\t\tstrerror(errno));\n+\n+\treturn -1;\n+}\n+\n const char *\n rte_eal_get_runtime_dir(void)\n {\n@@ -1096,6 +1183,18 @@ rte_eal_init(int argc, char **argv)\n \t\treturn -1;\n \t}\n \n+\t/*\n+\t * clean up unused files in runtime directory. we do this at the end of\n+\t * init and not at the beginning because we want to clean stuff up\n+\t * whether we are primary or secondary process, but we cannot remove\n+\t * primary process' files because secondary should be able to run even\n+\t * if primary process is dead.\n+\t */\n+\tif (eal_clean_runtime_dir() < 0) {\n+\t\trte_eal_init_alert(\"Cannot clear runtime directory\\n\");\n+\t\treturn -1;\n+\t}\n+\n \trte_eal_mcfg_complete();\n \n \t/* Call each registered callback, if enabled */\n", "prefixes": [] }