From patchwork Tue Dec 2 09:55:28 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: chixiaobo X-Patchwork-Id: 1726 Return-Path: X-Original-To: patchwork@dpdk.org Delivered-To: patchwork@dpdk.org Received: from [92.243.14.124] (localhost [IPv6:::1]) by dpdk.org (Postfix) with ESMTP id 81D2E32A5; Tue, 2 Dec 2014 10:56:12 +0100 (CET) Received: from demumfd001.nsn-inter.net (demumfd001.nsn-inter.net [93.183.12.32]) by dpdk.org (Postfix) with ESMTP id 36D60DE0 for ; Tue, 2 Dec 2014 10:55:56 +0100 (CET) Received: from demuprx016.emea.nsn-intra.net ([10.150.129.55]) by demumfd001.nsn-inter.net (8.14.3/8.14.3) with ESMTP id sB29tsWw027467 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Tue, 2 Dec 2014 09:55:55 GMT Received: from SGSIHTC002.nsn-intra.net ([10.159.225.19]) by demuprx016.emea.nsn-intra.net (8.12.11.20060308/8.12.11) with ESMTP id sB29tXkC026030 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=FAIL) for ; Tue, 2 Dec 2014 10:55:54 +0100 Received: from SGSIMBX004.nsn-intra.net ([169.254.4.161]) by SGSIHTC002.nsn-intra.net ([10.159.225.19]) with mapi id 14.03.0195.001; Tue, 2 Dec 2014 17:55:29 +0800 From: "Chi, Xiaobo (NSN - CN/Hangzhou)" To: "Chi, Xiaobo (NSN - CN/Hangzhou)" , "dev@dpdk.org" Thread-Topic: [PATCH] add one option memory-only for those secondary PRBs Thread-Index: AQHQDhQcLwXdX9mh20CnxuPUpRKT8Jx8DF1Q Date: Tue, 2 Dec 2014 09:55:28 +0000 Message-ID: References: <1417513268-6040-1-git-send-email-xiaobo.chi@nsn.com> In-Reply-To: <1417513268-6040-1-git-send-email-xiaobo.chi@nsn.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.159.225.116] MIME-Version: 1.0 X-purgate-type: clean X-purgate-Ad: Categorized by eleven eXpurgate (R) http://www.eleven.de X-purgate: clean X-purgate: This mail is considered clean (visit http://www.eleven.de for further information) X-purgate-size: 11921 X-purgate-ID: 151667::1417514155-0000658F-879712D6/0/0 Subject: Re: [dpdk-dev] [PATCH] add one option memory-only for those secondary PRBs X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Hi, The similar functionality of this patch has been sent out for review on Nov.12, email topic is "[dpdk-dev] one lightwight rte_eal_init() for SECONDARY processes which only use sharedmemory". This time, I just change the implementation method according to comments from Bruce Richardson. Background: What we are doing now is port make telecom network element to be cloud based. For one of our product, DPDK is applied not only for fastpath/dataplane processing, but also for Distributed Message eXchange (DMX) between different processes/applications which may located in different VM even different host. for such a DMX system, in one VM, we have one DPDK based dmxdemo (which acts as the PRIMARY) which is in charge of distribute message between different applications, and dozens of applications (act as SECONDARY) to use DPDK based rte_tx_ring/rte_rx_ring/mempool/memzone to send receive messages to dmxdemo. Problem: Here, these DPDK based SECONDARY processes need only the DPDK's hugepage based sharememory mechanism and it's upper libs (such as ring, mempool, etc.), they need not cpu core pinning, iopl privilege changing , pci device, timer, alarm, interrupt, shared_driver_list, core_info, threads for each core, etc. Then, for such kind of SECONDARY processes, the current rte_eal_init() is too heavy. I have seen some others also met similar troubles. Solution: One new EAL initializing argument, --memory-only, is added. It is only for those SECONDARY processes which only want to share memory with primary process. When this argument is defined, users need not define those madentory arguments, such as "-c xx -n xx", due to we don't want to pin such kind of processes to any CPUs. I have tested on my test env, it works. Please kindly help to review and give comments. Thanks. -----Original Message----- From: chixiaobo [mailto:xiaobo.chi@nsn.com] Sent: Tuesday, December 02, 2014 5:41 PM To: dev@dpdk.org Cc: Chi, Xiaobo (NSN - CN/Hangzhou) Subject: [PATCH] add one option memory-only for those secondary PRBs --- lib/librte_eal/common/eal_common_options.c | 18 +++- lib/librte_eal/common/eal_internal_cfg.h | 1 + lib/librte_eal/common/eal_options.h | 4 +- lib/librte_eal/linuxapp/eal/eal.c | 137 +++++++++++++++-------------- 4 files changed, 89 insertions(+), 71 deletions(-) mode change 100644 => 100755 lib/librte_eal/common/eal_common_options.c mode change 100644 => 100755 lib/librte_eal/common/eal_internal_cfg.h mode change 100644 => 100755 lib/librte_eal/common/eal_options.h mode change 100644 => 100755 lib/librte_eal/linuxapp/eal/eal.c diff --git a/lib/librte_eal/common/eal_common_options.c b/lib/librte_eal/common/eal_common_options.c old mode 100644 new mode 100755 index e2810ab..5ab4b87 --- a/lib/librte_eal/common/eal_common_options.c +++ b/lib/librte_eal/common/eal_common_options.c @@ -85,6 +85,7 @@ eal_long_options[] = { {OPT_XEN_DOM0, 0, 0, OPT_XEN_DOM0_NUM}, {OPT_CREATE_UIO_DEV, 1, NULL, OPT_CREATE_UIO_DEV_NUM}, {OPT_VFIO_INTR, 1, NULL, OPT_VFIO_INTR_NUM}, + {OPT_MEMORY_ONLY, 0, NULL, OPT_MEMORY_ONLY_NUM}, {0, 0, 0, 0} }; @@ -126,6 +127,7 @@ eal_reset_internal_config(struct internal_config *internal_cfg) internal_cfg->no_hpet = 1; #endif internal_cfg->vmware_tsc_map = 0; + internal_cfg->memory_only= 0; } /* @@ -454,6 +456,10 @@ eal_parse_common_option(int opt, const char *optarg, conf->process_type = eal_parse_proc_type(optarg); break; + case OPT_MEMORY_ONLY_NUM: + conf->memory_only= 1; + break; + case OPT_MASTER_LCORE_NUM: if (eal_parse_master_lcore(optarg) < 0) { RTE_LOG(ERR, EAL, "invalid parameter for --" @@ -525,9 +531,9 @@ eal_check_common_options(struct internal_config *internal_cfg) { struct rte_config *cfg = rte_eal_get_configuration(); - if (!lcores_parsed) { - RTE_LOG(ERR, EAL, "CPU cores must be enabled with options " - "-c or -l\n"); + if (!lcores_parsed && !(internal_cfg->process_type == RTE_PROC_SECONDARY&& internal_cfg->memory_only) ) { + RTE_LOG(ERR, EAL, "For those processes without memory-only option, CPU cores " + "must be enabled with options -c or -l\n"); return -1; } if (cfg->lcore_role[cfg->master_lcore] != ROLE_RTE) { @@ -545,6 +551,11 @@ eal_check_common_options(struct internal_config *internal_cfg) "specified\n"); return -1; } + if ( internal_cfg->process_type != RTE_PROC_SECONDARY && + internal_cfg->memory_only) { + RTE_LOG(ERR, EAL, "only secondary processes can specify memory-only option.\n"); + return -1; + } if (index(internal_cfg->hugefile_prefix, '%') != NULL) { RTE_LOG(ERR, EAL, "Invalid char, '%%', in --"OPT_FILE_PREFIX" " "option\n"); @@ -590,6 +601,7 @@ eal_common_usage(void) " --"OPT_SYSLOG" : set syslog facility\n" " --"OPT_LOG_LEVEL" : set default log level\n" " --"OPT_PROC_TYPE" : type of this process\n" + " --"OPT_MEMORY_ONLY": only use shared memory, valid only for secondary process." " --"OPT_PCI_BLACKLIST", -b: add a PCI device in black list.\n" " Prevent EAL from using this PCI device. The argument\n" " format is .\n" diff --git a/lib/librte_eal/common/eal_internal_cfg.h b/lib/librte_eal/common/eal_internal_cfg.h old mode 100644 new mode 100755 index aac6abf..68b982c --- a/lib/librte_eal/common/eal_internal_cfg.h +++ b/lib/librte_eal/common/eal_internal_cfg.h @@ -85,6 +85,7 @@ struct internal_config { unsigned num_hugepage_sizes; /**< how many sizes on this system */ struct hugepage_info hugepage_info[MAX_HUGEPAGE_SIZES]; + volatile unsigned memory_only; /**name); - solib->lib_handle = dlopen(solib->name, RTLD_NOW); - if (solib->lib_handle == NULL) - RTE_LOG(WARNING, EAL, "%s\n", dlerror()); - } - - eal_thread_init_master(rte_config.master_lcore); - - RTE_LOG(DEBUG, EAL, "Master core %u is ready (tid=%x)\n", - rte_config.master_lcore, (int)thread_id); - - if (rte_eal_dev_init() < 0) - rte_panic("Cannot init pmd devices\n"); - - RTE_LCORE_FOREACH_SLAVE(i) { - - /* - * create communication pipes between master thread - * and children - */ - if (pipe(lcore_config[i].pipe_master2slave) < 0) - rte_panic("Cannot create pipe\n"); - if (pipe(lcore_config[i].pipe_slave2master) < 0) - rte_panic("Cannot create pipe\n"); - - lcore_config[i].state = WAIT; - - /* create a thread for each lcore */ - ret = pthread_create(&lcore_config[i].thread_id, NULL, - eal_thread_loop, NULL); - if (ret != 0) - rte_panic("Cannot create thread\n"); - } - - /* - * Launch a dummy function on all slave lcores, so that master lcore - * knows they are all ready when this function returns. - */ - rte_eal_mp_remote_launch(sync_func, NULL, SKIP_MASTER); - rte_eal_mp_wait_lcore(); - - /* Probe & Initialize PCI devices */ - if (rte_eal_pci_probe()) - rte_panic("Cannot probe PCI\n"); - + if( !internal_config.memory_only ){ +#ifdef RTE_LIBRTE_IVSHMEM + if (rte_eal_ivshmem_obj_init() < 0) + rte_panic("Cannot init IVSHMEM objects\n"); +#endif + if (rte_eal_alarm_init() < 0) + rte_panic("Cannot init interrupt-handling thread\n"); + + if (rte_eal_intr_init() < 0) + rte_panic("Cannot init interrupt-handling thread\n"); + + if (rte_eal_timer_init() < 0) + rte_panic("Cannot init HPET or TSC timers\n"); + + eal_check_mem_on_local_socket(); + + rte_eal_mcfg_complete(); + + TAILQ_FOREACH(solib, &solib_list, next) { + RTE_LOG(INFO, EAL, "open shared lib %s\n", solib->name); + solib->lib_handle = dlopen(solib->name, RTLD_NOW); + if (solib->lib_handle == NULL) + RTE_LOG(WARNING, EAL, "%s\n", dlerror()); + } + + eal_thread_init_master(rte_config.master_lcore); + + RTE_LOG(DEBUG, EAL, "Master core %u is ready (tid=%x)\n", + rte_config.master_lcore, (int)thread_id); + + if (rte_eal_dev_init() < 0) + rte_panic("Cannot init pmd devices\n"); + + RTE_LCORE_FOREACH_SLAVE(i) { + + /* + * create communication pipes between master thread + * and children + */ + if (pipe(lcore_config[i].pipe_master2slave) < 0) + rte_panic("Cannot create pipe\n"); + if (pipe(lcore_config[i].pipe_slave2master) < 0) + rte_panic("Cannot create pipe\n"); + + lcore_config[i].state = WAIT; + + /* create a thread for each lcore */ + ret = pthread_create(&lcore_config[i].thread_id, NULL, + eal_thread_loop, NULL); + if (ret != 0) + rte_panic("Cannot create thread\n"); + } + + /* + * Launch a dummy function on all slave lcores, so that master lcore + * knows they are all ready when this function returns. + */ + rte_eal_mp_remote_launch(sync_func, NULL, SKIP_MASTER); + rte_eal_mp_wait_lcore(); + + /* Probe & Initialize PCI devices */ + if (rte_eal_pci_probe()) + rte_panic("Cannot probe PCI\n"); + } return fctret; } @@ -859,3 +861,4 @@ int rte_eal_has_hugepages(void) { return ! internal_config.no_hugetlbfs; } +