diff mbox series

[v7,04/12] net/nfp: add initial flower firmware support

Message ID 1660299750-10668-5-git-send-email-chaoyong.he@corigine.com (mailing list archive)
State Changes Requested
Delegated to: Ferruh Yigit
Headers show
Series preparation for the rte_flow offload of nfp PMD | expand

Checks

Context Check Description
ci/checkpatch success coding style OK

Commit Message

Chaoyong He Aug. 12, 2022, 10:22 a.m. UTC
Adds the basic probing infrastructure to support the flower firmware.
This firmware is geared towards offloading OVS and can generally be
found in /lib/firmware/netronome/flower. It is also used by the NFP
kernel driver when OVS offload with TC is desired.

Adds the basic infrastructure needed by the flower firmware to operate.
The firmware requires threads to service both the PF vNIC and the ctrl
vNIC. The PF is responsible for handling any fallback traffic and the
ctrl vNIC is used to communicate OVS flows and flow statistics to and
from the smartNIC. rte_services are used to facilitate this logic.

Adds the cpp service, used for some user tools.

Signed-off-by: Chaoyong He <chaoyong.he@corigine.com>
Signed-off-by: Heinrich Kuhn <heinrich.kuhn@corigine.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund@corigine.com>
---
 drivers/net/nfp/flower/nfp_flower.c | 102 ++++++++++++++++++++++++++++++++++++
 drivers/net/nfp/flower/nfp_flower.h |  22 ++++++++
 drivers/net/nfp/meson.build         |   1 +
 drivers/net/nfp/nfp_common.h        |   1 +
 drivers/net/nfp/nfp_cpp_bridge.c    |  88 ++++++++++++++++++++++++++-----
 drivers/net/nfp/nfp_cpp_bridge.h    |   6 ++-
 drivers/net/nfp/nfp_ethdev.c        |  40 ++++++++++++--
 7 files changed, 241 insertions(+), 19 deletions(-)
 create mode 100644 drivers/net/nfp/flower/nfp_flower.c
 create mode 100644 drivers/net/nfp/flower/nfp_flower.h

Comments

Ferruh Yigit Sept. 5, 2022, 3:39 p.m. UTC | #1
On 8/12/2022 11:22 AM, Chaoyong He wrote:
> Adds the basic probing infrastructure to support the flower firmware.
> This firmware is geared towards offloading OVS and can generally be
> found in /lib/firmware/netronome/flower. It is also used by the NFP
> kernel driver when OVS offload with TC is desired.

'/lib/firmware/netronome/flower' FW is loaded (automatically ?) by 
kernel driver, right?
Is there anything related to the DPDK with the name/path of the FW?


And I wonder if these kind of information should be part of driver 
documentation?

<...>

> @@ -965,6 +968,16 @@
>   			goto hwqueues_cleanup;
>   		}
>   		break;
> +	case NFP_APP_FLOWER_NIC:
> +		PMD_INIT_LOG(INFO, "Initializing Flower");
> +		pci_dev->device.driver = &pci_drv->driver;

Why this assignment is required? Driver shouldn't need to update this 
bus related data struct.
Chaoyong He Sept. 7, 2022, 8:22 a.m. UTC | #2
> -----Original Message-----
> From: Ferruh Yigit <ferruh.yigit@xilinx.com>
> Sent: Monday, September 5, 2022 11:40 PM
> To: Chaoyong He <chaoyong.he@corigine.com>; dev@dpdk.org
> Cc: oss-drivers <oss-drivers@corigine.com>; Niklas Soderlund
> <niklas.soderlund@corigine.com>; Heinrich Kuhn
> <heinrich.kuhn@corigine.com>
> Subject: Re: [PATCH v7 04/12] net/nfp: add initial flower firmware support
> 
> On 8/12/2022 11:22 AM, Chaoyong He wrote:
> > Adds the basic probing infrastructure to support the flower firmware.
> > This firmware is geared towards offloading OVS and can generally be
> > found in /lib/firmware/netronome/flower. It is also used by the NFP
> > kernel driver when OVS offload with TC is desired.
> 
> '/lib/firmware/netronome/flower' FW is loaded (automatically ?) by kernel
> driver, right?
> Is there anything related to the DPDK with the name/path of the FW?
> 
> 
> And I wonder if these kind of information should be part of driver
> documentation?
> 
> <...>
> 
> > @@ -965,6 +968,16 @@
> >   			goto hwqueues_cleanup;
> >   		}
> >   		break;
> > +	case NFP_APP_FLOWER_NIC:
> > +		PMD_INIT_LOG(INFO, "Initializing Flower");
> > +		pci_dev->device.driver = &pci_drv->driver;
> 
> Why this assignment is required? Driver shouldn't need to update this
> bus related data struct.

Following the framework's logic: 
function 'rte_pci_probe_one_driver()' in file drivers/bus/pci/pci_common.c

```
...
ret = dr->probe(dr, dev);    // here call our nfp_pf_pci_probe()
if (ret) {
...
} else {
        dev->device.driver = &dr->driver;
}

```

Here the framework will do the assignment.

But in our logic, if we won't add this assignment, we will run into a coredump:
```
Program terminated with signal SIGSEGV, Segmentation fault.
#0  rte_eth_dev_info_get (port_id=0, dev_info=0x7ffe9c3d3f70) at ../lib/ethdev/rte_ethdev.c:3138
3138            dev_info->driver_name = dev->device->driver->name;
(gdb) bt
#0  rte_eth_dev_info_get (port_id=0, dev_info=0x7ffe9c3d3f70) at ../lib/ethdev/rte_ethdev.c:3138
#1  0x00007fa95128565f in rte_eth_dev_configure (port_id=0, nb_rx_q=1, nb_tx_q=1,
    dev_conf=0x7fa94ce338e0 <port_conf>) at ../lib/ethdev/rte_ethdev.c:1110
#2  0x00007fa94cdf74b7 in nfp_flower_init_ctrl_vnic (hw=0x2001e8530)
    at ../drivers/net/nfp/flower/nfp_flower.c:1035
#3  0x00007fa94cdf7b32 in nfp_init_app_fw_flower (pf_dev=0x2001e8c80)
    at ../drivers/net/nfp/flower/nfp_flower.c:1219
#4  0x00007fa94ce30e33 in nfp_pf_init (pci_dev=0xcd22c0, pci_drv=0x7fa94ce44460 <rte_nfp_net_pf_pmd>)
    at ../drivers/net/nfp/nfp_ethdev.c:976
#5  0x00007fa94ce31429 in nfp_pf_pci_probe (pci_drv=0x7fa94ce44460 <rte_nfp_net_pf_pmd>, dev=0xcd22c0)
    at ../drivers/net/nfp/nfp_ethdev.c:1152
#6  0x00007fa94e8f8f34 in rte_pci_probe_one_driver (dr=0x7fa94ce44460 <rte_nfp_net_pf_pmd>, dev=0xcd22c0)
    at ../drivers/bus/pci/pci_common.c:269
#7  0x00007fa94e8f91c0 in pci_probe_all_drivers (dev=0xcd22c0) at ../drivers/bus/pci/pci_common.c:353
#8  0x00007fa94e8f9244 in pci_probe () at ../drivers/bus/pci/pci_common.c:380
#9  0x00007fa951163c0e in rte_bus_probe () at ../lib/eal/common/eal_common_bus.c:72
#10 0x00007fa951193903 in rte_eal_init (argc=10, argv=0xcc5150) at ../lib/eal/linux/eal.c:1279
#11 0x0000000000601531 in dpdk_init__ (ovs_other_config=0xccefd0) at lib/dpdk.c:466
#12 0x0000000000601913 in dpdk_init (ovs_other_config=0xccefd0) at lib/dpdk.c:545
#13 0x00000000004129ed in bridge_run () at vswitchd/bridge.c:3252
#14 0x0000000000418364 in main (argc=11, argv=0x7ffe9c3d49c8) at vswitchd/ovs-vswitchd.c:129

```

Our nfp card use `control message` to exchange message between PMD and firmware when we use flower firmware.
The control message is in the form of pkt and we use a `ctrl vNIC` ehtdev as the agent to send and receive these pkts.
e.g., if we want to create representor port, the PMD must send the corresponding control message to firmware.
To be able to send and receive pkt, we must do some configure steps to this ethdev firstly.
But the framework has not do the assignment step at this moment.
And this is where the problem comes from.

Logically, if our probe process is successful, assignment again won't import any problem.
If our probe process failed, our logic will undo the assignment and the framework will go as its original logic.
So maybe we can keep the logic now?
Or there exist another way more standard?
Ferruh Yigit Sept. 7, 2022, 1:24 p.m. UTC | #3
On 9/7/2022 9:22 AM, Chaoyong He wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
> 
> 
>> -----Original Message-----
>> From: Ferruh Yigit <ferruh.yigit@xilinx.com>
>> Sent: Monday, September 5, 2022 11:40 PM
>> To: Chaoyong He <chaoyong.he@corigine.com>; dev@dpdk.org
>> Cc: oss-drivers <oss-drivers@corigine.com>; Niklas Soderlund
>> <niklas.soderlund@corigine.com>; Heinrich Kuhn
>> <heinrich.kuhn@corigine.com>
>> Subject: Re: [PATCH v7 04/12] net/nfp: add initial flower firmware support
>>
>> On 8/12/2022 11:22 AM, Chaoyong He wrote:
>>> Adds the basic probing infrastructure to support the flower firmware.
>>> This firmware is geared towards offloading OVS and can generally be
>>> found in /lib/firmware/netronome/flower. It is also used by the NFP
>>> kernel driver when OVS offload with TC is desired.
>>
>> '/lib/firmware/netronome/flower' FW is loaded (automatically ?) by kernel
>> driver, right?
>> Is there anything related to the DPDK with the name/path of the FW?
>>
>>
>> And I wonder if these kind of information should be part of driver
>> documentation?
>>
>> <...>
>>
>>> @@ -965,6 +968,16 @@
>>>                      goto hwqueues_cleanup;
>>>              }
>>>              break;
>>> +   case NFP_APP_FLOWER_NIC:
>>> +           PMD_INIT_LOG(INFO, "Initializing Flower");
>>> +           pci_dev->device.driver = &pci_drv->driver;
>>
>> Why this assignment is required? Driver shouldn't need to update this
>> bus related data struct.
> 
> Following the framework's logic:
> function 'rte_pci_probe_one_driver()' in file drivers/bus/pci/pci_common.c
> 
> ```
> ...
> ret = dr->probe(dr, dev);    // here call our nfp_pf_pci_probe()
> if (ret) {
> ...
> } else {
>          dev->device.driver = &dr->driver;
> }
> 
> ```
> 
> Here the framework will do the assignment.
> 
> But in our logic, if we won't add this assignment, we will run into a coredump:
> ```
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0  rte_eth_dev_info_get (port_id=0, dev_info=0x7ffe9c3d3f70) at ../lib/ethdev/rte_ethdev.c:3138
> 3138            dev_info->driver_name = dev->device->driver->name;
> (gdb) bt
> #0  rte_eth_dev_info_get (port_id=0, dev_info=0x7ffe9c3d3f70) at ../lib/ethdev/rte_ethdev.c:3138
> #1  0x00007fa95128565f in rte_eth_dev_configure (port_id=0, nb_rx_q=1, nb_tx_q=1,
>      dev_conf=0x7fa94ce338e0 <port_conf>) at ../lib/ethdev/rte_ethdev.c:1110
> #2  0x00007fa94cdf74b7 in nfp_flower_init_ctrl_vnic (hw=0x2001e8530)
>      at ../drivers/net/nfp/flower/nfp_flower.c:1035
> #3  0x00007fa94cdf7b32 in nfp_init_app_fw_flower (pf_dev=0x2001e8c80)
>      at ../drivers/net/nfp/flower/nfp_flower.c:1219
> #4  0x00007fa94ce30e33 in nfp_pf_init (pci_dev=0xcd22c0, pci_drv=0x7fa94ce44460 <rte_nfp_net_pf_pmd>)
>      at ../drivers/net/nfp/nfp_ethdev.c:976
> #5  0x00007fa94ce31429 in nfp_pf_pci_probe (pci_drv=0x7fa94ce44460 <rte_nfp_net_pf_pmd>, dev=0xcd22c0)
>      at ../drivers/net/nfp/nfp_ethdev.c:1152
> #6  0x00007fa94e8f8f34 in rte_pci_probe_one_driver (dr=0x7fa94ce44460 <rte_nfp_net_pf_pmd>, dev=0xcd22c0)
>      at ../drivers/bus/pci/pci_common.c:269
> #7  0x00007fa94e8f91c0 in pci_probe_all_drivers (dev=0xcd22c0) at ../drivers/bus/pci/pci_common.c:353
> #8  0x00007fa94e8f9244 in pci_probe () at ../drivers/bus/pci/pci_common.c:380
> #9  0x00007fa951163c0e in rte_bus_probe () at ../lib/eal/common/eal_common_bus.c:72
> #10 0x00007fa951193903 in rte_eal_init (argc=10, argv=0xcc5150) at ../lib/eal/linux/eal.c:1279
> #11 0x0000000000601531 in dpdk_init__ (ovs_other_config=0xccefd0) at lib/dpdk.c:466
> #12 0x0000000000601913 in dpdk_init (ovs_other_config=0xccefd0) at lib/dpdk.c:545
> #13 0x00000000004129ed in bridge_run () at vswitchd/bridge.c:3252
> #14 0x0000000000418364 in main (argc=11, argv=0x7ffe9c3d49c8) at vswitchd/ovs-vswitchd.c:129
> 
> ```
> 
> Our nfp card use `control message` to exchange message between PMD and firmware when we use flower firmware.
> The control message is in the form of pkt and we use a `ctrl vNIC` ehtdev as the agent to send and receive these pkts.
> e.g., if we want to create representor port, the PMD must send the corresponding control message to firmware.
> To be able to send and receive pkt, we must do some configure steps to this ethdev firstly.
> But the framework has not do the assignment step at this moment.
> And this is where the problem comes from.
> 

I am not sure about PMD calling 'rte_eth_dev_configure()', this API is 
intended for application.

And even worse, not sure if it is allowed to call 
'rte_eth_dev_configure()' before driver probe completed successfully.

It looks to me assigning 'pci_dev->device.driver' within the PMD is hack 
to allow it run 'rte_eth_dev_configure()' before probe() completed.
Do you really need to call ''rte_eth_dev_configure()' to prepare ethdev 
for the 'control message' exchange? What exactly needs to be configured 
for this?


> Logically, if our probe process is successful, assignment again won't import any problem.
> If our probe process failed, our logic will undo the assignment and the framework will go as its original logic.
> So maybe we can keep the logic now?
> Or there exist another way more standard?
>
diff mbox series

Patch

diff --git a/drivers/net/nfp/flower/nfp_flower.c b/drivers/net/nfp/flower/nfp_flower.c
new file mode 100644
index 0000000..1e12f49
--- /dev/null
+++ b/drivers/net/nfp/flower/nfp_flower.c
@@ -0,0 +1,102 @@ 
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Corigine, Inc.
+ * All rights reserved.
+ */
+
+#include <rte_common.h>
+#include <ethdev_driver.h>
+#include <rte_service_component.h>
+#include <rte_malloc.h>
+#include <ethdev_pci.h>
+#include <ethdev_driver.h>
+
+#include "../nfp_common.h"
+#include "../nfp_logs.h"
+#include "../nfp_ctrl.h"
+#include "../nfp_cpp_bridge.h"
+#include "nfp_flower.h"
+
+static struct rte_service_spec flower_services[NFP_FLOWER_SERVICE_MAX] = {
+};
+
+static int
+nfp_flower_enable_services(struct nfp_app_flower *app_flower)
+{
+	int i;
+	int ret = 0;
+
+	for (i = 0; i < NFP_FLOWER_SERVICE_MAX; i++) {
+		/* Pass a pointer to the flower app to the service */
+		flower_services[i].callback_userdata = (void *)app_flower;
+
+		/* Register the flower services */
+		ret = rte_service_component_register(&flower_services[i],
+				&app_flower->flower_services_ids[i]);
+		if (ret != 0) {
+			PMD_INIT_LOG(WARNING,
+				"Could not register Flower PF vNIC service");
+			break;
+		}
+
+		PMD_INIT_LOG(INFO, "Flower PF vNIC service registered");
+
+		/* Map them to available service cores*/
+		ret = nfp_map_service(app_flower->flower_services_ids[i]);
+		if (ret != 0)
+			break;
+	}
+
+	return ret;
+}
+
+int
+nfp_init_app_flower(struct nfp_pf_dev *pf_dev)
+{
+	int ret;
+	unsigned int numa_node;
+	struct nfp_net_hw *pf_hw;
+	struct nfp_app_flower *app_flower;
+
+	numa_node = rte_socket_id();
+
+	/* Allocate memory for the Flower app */
+	app_flower = rte_zmalloc_socket("nfp_app_flower", sizeof(*app_flower),
+			RTE_CACHE_LINE_SIZE, numa_node);
+	if (app_flower == NULL) {
+		ret = -ENOMEM;
+		goto done;
+	}
+
+	pf_dev->app_priv = app_flower;
+
+	/* Allocate memory for the PF AND ctrl vNIC here (hence the * 2) */
+	pf_hw = rte_zmalloc_socket("nfp_pf_vnic", 2 * sizeof(struct nfp_net_adapter),
+			RTE_CACHE_LINE_SIZE, numa_node);
+	if (pf_hw == NULL) {
+		ret = -ENOMEM;
+		goto app_cleanup;
+	}
+
+	/* Start up flower services */
+	ret = nfp_flower_enable_services(app_flower);
+	if (ret != 0) {
+		ret = -ESRCH;
+		goto vnic_cleanup;
+	}
+
+	return 0;
+
+vnic_cleanup:
+	rte_free(pf_hw);
+app_cleanup:
+	rte_free(app_flower);
+done:
+	return ret;
+}
+
+int
+nfp_secondary_init_app_flower(__rte_unused struct nfp_cpp *cpp)
+{
+	PMD_INIT_LOG(ERR, "Flower firmware not supported");
+	return -ENOTSUP;
+}
diff --git a/drivers/net/nfp/flower/nfp_flower.h b/drivers/net/nfp/flower/nfp_flower.h
new file mode 100644
index 0000000..4a9b302
--- /dev/null
+++ b/drivers/net/nfp/flower/nfp_flower.h
@@ -0,0 +1,22 @@ 
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2022 Corigine, Inc.
+ * All rights reserved.
+ */
+
+#ifndef _NFP_FLOWER_H_
+#define _NFP_FLOWER_H_
+
+enum nfp_flower_service {
+	NFP_FLOWER_SERVICE_MAX
+};
+
+/* The flower application's private structure */
+struct nfp_app_flower {
+	/* List of rte_service ID's for the flower app */
+	uint32_t flower_services_ids[NFP_FLOWER_SERVICE_MAX];
+};
+
+int nfp_init_app_flower(struct nfp_pf_dev *pf_dev);
+int nfp_secondary_init_app_flower(struct nfp_cpp *cpp);
+
+#endif /* _NFP_FLOWER_H_ */
diff --git a/drivers/net/nfp/meson.build b/drivers/net/nfp/meson.build
index 810f02a..7ae3115 100644
--- a/drivers/net/nfp/meson.build
+++ b/drivers/net/nfp/meson.build
@@ -6,6 +6,7 @@  if not is_linux or not dpdk_conf.get('RTE_ARCH_64')
     reason = 'only supported on 64-bit Linux'
 endif
 sources = files(
+        'flower/nfp_flower.c',
         'nfpcore/nfp_cpp_pcie_ops.c',
         'nfpcore/nfp_nsp.c',
         'nfpcore/nfp_cppcore.c',
diff --git a/drivers/net/nfp/nfp_common.h b/drivers/net/nfp/nfp_common.h
index b99971b..8d721bd 100644
--- a/drivers/net/nfp/nfp_common.h
+++ b/drivers/net/nfp/nfp_common.h
@@ -114,6 +114,7 @@ 
 /* Firmware application ID's */
 enum nfp_app_id {
 	NFP_APP_CORE_NIC               = 0x1,
+	NFP_APP_FLOWER_NIC             = 0x3,
 };
 
 /* nfp_qcp_ptr - Read or Write Pointer of a queue */
diff --git a/drivers/net/nfp/nfp_cpp_bridge.c b/drivers/net/nfp/nfp_cpp_bridge.c
index 0922ea9..ac14029 100644
--- a/drivers/net/nfp/nfp_cpp_bridge.c
+++ b/drivers/net/nfp/nfp_cpp_bridge.c
@@ -28,22 +28,86 @@ 
 static int nfp_cpp_bridge_serve_write(int sockfd, struct nfp_cpp *cpp);
 static int nfp_cpp_bridge_serve_read(int sockfd, struct nfp_cpp *cpp);
 static int nfp_cpp_bridge_serve_ioctl(int sockfd, struct nfp_cpp *cpp);
+static int nfp_cpp_bridge_service_func(void *args);
 
-void nfp_register_cpp_service(struct nfp_cpp *cpp)
+static struct rte_service_spec cpp_service = {
+	.name         = "nfp_cpp_service",
+	.callback     = nfp_cpp_bridge_service_func,
+};
+
+int
+nfp_map_service(uint32_t service_id)
 {
-	uint32_t *cpp_service_id = NULL;
-	struct rte_service_spec service;
+	int32_t ret;
+	uint32_t slcore = 0;
+	int32_t slcore_count;
+	uint8_t service_count;
+	const char *service_name;
+	uint32_t slcore_array[RTE_MAX_LCORE];
+	uint8_t min_service_count = UINT8_MAX;
+
+	slcore_count = rte_service_lcore_list(slcore_array, RTE_MAX_LCORE);
+	if (slcore_count <= 0) {
+		PMD_INIT_LOG(DEBUG, "No service cores found");
+		return -ENOENT;
+	}
+
+	/*
+	 * Find a service core with the least number of services already
+	 * registered to it
+	 */
+	while (slcore_count--) {
+		service_count = rte_service_lcore_count_services(slcore_array[slcore_count]);
+		if (service_count < min_service_count) {
+			slcore = slcore_array[slcore_count];
+			min_service_count = service_count;
+		}
+	}
 
-	memset(&service, 0, sizeof(struct rte_service_spec));
-	snprintf(service.name, sizeof(service.name), "nfp_cpp_service");
-	service.callback = nfp_cpp_bridge_service_func;
-	service.callback_userdata = (void *)cpp;
+	service_name = rte_service_get_name(service_id);
+	PMD_INIT_LOG(INFO, "Mapping service %s to core %u", service_name, slcore);
+	ret = rte_service_map_lcore_set(service_id, slcore, 1);
+	if (ret != 0) {
+		PMD_INIT_LOG(DEBUG, "Could not map flower service");
+		return -ENOENT;
+	}
 
-	if (rte_service_component_register(&service,
-					   cpp_service_id))
-		RTE_LOG(WARNING, PMD, "NFP CPP bridge service register() failed");
+	rte_service_runstate_set(service_id, 1);
+	rte_service_component_runstate_set(service_id, 1);
+	rte_service_lcore_start(slcore);
+	if (rte_service_may_be_active(slcore))
+		RTE_LOG(INFO, PMD, "The service %s is running", service_name);
 	else
-		RTE_LOG(DEBUG, PMD, "NFP CPP bridge service registered");
+		RTE_LOG(INFO, PMD, "The service %s is not running", service_name);
+
+	return 0;
+}
+
+int nfp_enable_cpp_service(struct nfp_cpp *cpp, enum nfp_app_id app_id)
+{
+	int ret = 0;
+	uint32_t id = 0;
+
+	cpp_service.callback_userdata = (void *)cpp;
+
+	/* Register the cpp service */
+	ret = rte_service_component_register(&cpp_service, &id);
+	if (ret != 0) {
+		PMD_INIT_LOG(WARNING, "Could not register nfp cpp service");
+		return -EINVAL;
+	}
+
+	PMD_INIT_LOG(INFO, "NFP cpp service registered");
+
+	/* Map it to available service core*/
+	ret = nfp_map_service(id);
+	if (ret != 0) {
+		PMD_INIT_LOG(DEBUG, "Could not map nfp cpp service");
+		if (app_id == NFP_APP_FLOWER_NIC)
+			return -EINVAL;
+	}
+
+	return 0;
 }
 
 /*
@@ -307,7 +371,7 @@  void nfp_register_cpp_service(struct nfp_cpp *cpp)
  * unaware of the CPP bridge performing the NFP kernel char driver for CPP
  * accesses.
  */
-int32_t
+static int
 nfp_cpp_bridge_service_func(void *args)
 {
 	struct sockaddr address;
diff --git a/drivers/net/nfp/nfp_cpp_bridge.h b/drivers/net/nfp/nfp_cpp_bridge.h
index aea5fdc..dde50d7 100644
--- a/drivers/net/nfp/nfp_cpp_bridge.h
+++ b/drivers/net/nfp/nfp_cpp_bridge.h
@@ -16,6 +16,8 @@ 
 #ifndef _NFP_CPP_BRIDGE_H_
 #define _NFP_CPP_BRIDGE_H_
 
+#include "nfp_common.h"
+
 #define NFP_CPP_MEMIO_BOUNDARY	(1 << 20)
 #define NFP_BRIDGE_OP_READ	20
 #define NFP_BRIDGE_OP_WRITE	30
@@ -24,8 +26,8 @@ 
 #define NFP_IOCTL 'n'
 #define NFP_IOCTL_CPP_IDENTIFICATION _IOW(NFP_IOCTL, 0x8f, uint32_t)
 
-void nfp_register_cpp_service(struct nfp_cpp *cpp);
-int32_t nfp_cpp_bridge_service_func(void *args);
+int nfp_map_service(uint32_t service_id);
+int nfp_enable_cpp_service(struct nfp_cpp *cpp, enum nfp_app_id app_id);
 
 #endif /* _NFP_CPP_BRIDGE_H_ */
 /*
diff --git a/drivers/net/nfp/nfp_ethdev.c b/drivers/net/nfp/nfp_ethdev.c
index a6a7e7e..e097a58 100644
--- a/drivers/net/nfp/nfp_ethdev.c
+++ b/drivers/net/nfp/nfp_ethdev.c
@@ -38,6 +38,8 @@ 
 #include "nfp_ctrl.h"
 #include "nfp_cpp_bridge.h"
 
+#include "flower/nfp_flower.h"
+
 static int
 nfp_net_pf_read_mac(struct nfp_app_nic *app_nic, int port)
 {
@@ -837,7 +839,8 @@ 
 }
 
 static int
-nfp_pf_init(struct rte_pci_device *pci_dev)
+nfp_pf_init(struct rte_pci_device *pci_dev,
+		struct rte_pci_driver *pci_drv)
 {
 	int ret;
 	int err = 0;
@@ -965,6 +968,16 @@ 
 			goto hwqueues_cleanup;
 		}
 		break;
+	case NFP_APP_FLOWER_NIC:
+		PMD_INIT_LOG(INFO, "Initializing Flower");
+		pci_dev->device.driver = &pci_drv->driver;
+		ret = nfp_init_app_flower(pf_dev);
+		if (ret != 0) {
+			PMD_INIT_LOG(ERR, "Could not initialize Flower!");
+			pci_dev->device.driver = NULL;
+			goto hwqueues_cleanup;
+		}
+		break;
 	default:
 		PMD_INIT_LOG(ERR, "Unsupported Firmware loaded");
 		ret = -EINVAL;
@@ -972,7 +985,12 @@ 
 	}
 
 	/* register the CPP bridge service here for primary use */
-	nfp_register_cpp_service(pf_dev->cpp);
+	ret = nfp_enable_cpp_service(pf_dev->cpp, pf_dev->app_id);
+	if (ret != 0) {
+		PMD_INIT_LOG(ERR, "Enable cpp service failed.");
+		ret = -EINVAL;
+		goto hwqueues_cleanup;
+	}
 
 	return 0;
 
@@ -1096,6 +1114,14 @@ 
 			goto sym_tbl_cleanup;
 		}
 		break;
+	case NFP_APP_FLOWER_NIC:
+		PMD_INIT_LOG(INFO, "Initializing Flower");
+		ret = nfp_secondary_init_app_flower(cpp);
+		if (ret != 0) {
+			PMD_INIT_LOG(ERR, "Could not initialize Flower!");
+			goto sym_tbl_cleanup;
+		}
+		break;
 	default:
 		PMD_INIT_LOG(ERR, "Unsupported Firmware loaded");
 		ret = -EINVAL;
@@ -1106,7 +1132,11 @@ 
 		goto sym_tbl_cleanup;
 
 	/* Register the CPP bridge service for the secondary too */
-	nfp_register_cpp_service(cpp);
+	ret = nfp_enable_cpp_service(cpp, app_id);
+	if (ret != 0) {
+		PMD_INIT_LOG(ERR, "Enable cpp service failed.");
+		ret = -EINVAL;
+	}
 
 sym_tbl_cleanup:
 	free(sym_tbl);
@@ -1115,11 +1145,11 @@ 
 }
 
 static int
-nfp_pf_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+nfp_pf_pci_probe(struct rte_pci_driver *pci_drv,
 		struct rte_pci_device *dev)
 {
 	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
-		return nfp_pf_init(dev);
+		return nfp_pf_init(dev, pci_drv);
 	else
 		return nfp_pf_secondary_init(dev);
 }