Oracle SOA Suite 12c Deployment failed- JCA deployment validation errors for 'Adapters/dbadapter_db.jca'

Recently I have come across a strange situation for deployment of a SOA composite 12c (having DbAdapter) in production environment. Below is summary of issue and solution-

Error: 

Deployment of a SOA composite is failing with "Error while validating JCA Reference Binding meta data during composite deployment. : JCA deployment validation errors for 'Adapters/dbadapter_db.jca'

Environment Details

1. SOA Suite: Oracle SOA Suite 12c(12.1.3.0.0)

2. Operating System: Oracle Linux 7

3. Database: Oracle Database 12cR2 (12.2.2.1)

Error Log- soa_server1-diagnostic.log

2019-05-23T17:20:15.480+05:30] [soa_server1] [ERROR] [] [oracle.integration.platform.blocks.deploy] [tid: [ACTIVE].ExecuteThread: '57' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: weblogic] [ecid: d2fb4bbc-5b4e-4d2d-a19a-a19ac0d8be37-00001b17,0:7:2] [APP: soa-infra] [partition-name: DOMAIN] [tenant-name: GLOBAL] [DSID: 0000Mf^XjTr56i05nzp2iZ1StcZn000001] Exception caught, remove composite data:Operation failed - Member(Id=1, Timestamp=2019-05-23 16:56:01.912, Address=10.96.11.163:44871, MachineId=48535, Location=site:uppclonline.com,machine:oimhost2,process:8249,member:soa_server2, Role=soa_cluster):Error while validating JCA Reference Binding meta data during composite deployment. : JCA deployment validation errors for 'Adapters/GetOfficeDetails_db.jca'[[

; .  Please see the server diagnostic logs on soa_server1 for details

]]

This error suggests that the Connection Factory mentioned in the logfile (in this example "eis/DB/adapter_new") does not exist.

However when checking the connection factories configured for the DB Adapter using the WebLogic Admin Console:

Deployments -> DbAdapter -> Configuration -> Outbound Connection Pools -> javax.resource.cci.ConnectionFactory

it is found that the Connection Factory mentioned does exist, and is correctly configured.

Cause

The deployer requires that the JNDI of the connection factory exists on all managed servers. In our case the SOA cluster is span across two nodes soa_server1 & soa_server2. However, there is no common shared storage pointing to /u01/oracle/products/ORACLE_HOME/soa/soa/Plan.xml

If the deployment plan for the DB Adapter (Plan.xml) has not have been propagated to all managed servers in the cluster, then this will not be the case and the deployment will fail.

To check if this is the cause, in WebLogic Administration Console, for each managed server, look under Deployments -> DbAdapter, and check if the Connection Factory (in this example "eis/DB/peterw_new") is visible under both the Monitoring and Testing tabs.

If the Connection Factory is not visible under the Monitoring and Testing tabs of all managed servers, then it is likely that the Deployment Plan for the DB Adapter has not been propagated to all managed servers in the cluster.

Solution

1. First try redeploying the DB Adapter using the deployment plan.

In Admin Console, click the Lock & Edit button and check the box next to DB Adapter in the Deployments . Hit the Update button and then select "Redeploy this application using the following deployment files". Choose the Deployment Plan, and on the next screen hit the Finish button.

2. If that does not resolve the issue, follow these steps :-

2a. Using Adminstration Console, identify the DB Adapter Deployment Plan (by default it is called Plan.xml).

Deployments -> DbAdapter -> Deployment Plan

2b. Check the Plan.xml on each managed server(in our case soa_server1 & soa_server2), and ensure that each copy is identical, and contains the Connection Factory (in this example "eis/DB/adapter_new")

2c. If the DB Adapter Plan.xml is out of sync on any managed server, ensure the DB Adapter Deployment Plan/Plan.xml is either on shared storage, or has been manually copied as per the advice in these documents

References-

Doc ID 1273602.1 - SOA 11g: Propagating Changes in the Deployment Plan (Plan.xml) for an Adapter Deployment in a Cluster.
Doc ID 1324544.1 - How to Make Sure Plan.xml is Updated on All Machines in a Cluster for SOA Adapter Deployment Updates