WSO2 APIM 3.0.0 - Timeout: Pool empty. Unable to fetch a connection in 60 seconds

Recently I have come across a situation where my WSO2 APIM GUI publisher portal was taking too long to load and it was not showing the APIs listing. I have checked the logs available at wso2carbon.log and found error related to JDBC connection. Below is the summary of the issue, cause and solution-

Environment

1. WSO2 APIM 3.0.0
2. DB- MySQL
3. OS- Ubuntu 16.04

Issue

[2020-04-23 10:33:18,496] ERROR {org.wso2.carbon.identity.oauth2.OAuth2Service} - Error occurred while issuing the access token for Client ID : CUskLST4RR0mzQumGw_nq6PRykUa, User ID null, Scope : [] and Grant Type : authorization_code org.wso2.carbon.identity.base.IdentityRuntimeException: org.apache.tomcat.jdbc.pool.PoolExhaustedException: [https-jsse-nio-9443-exec-27] Timeout: Pool empty. Unable to fetch a connection in 60 seconds, none available[size:50; busy:50; idle:0; lastwait:60000].at

org.wso2.carbon.identity.base.IdentityRuntimeException.error(IdentityRuntimeException.java:71)

................................................................................................so on.

Cause

JDBC Connection pool size exhausted as specified in master-datasources.xml

In my case, the path to the file is specified below-

/opt/apim/usr/lib64/wso2/wso2am/3.0.0/repository/conf/datasources/master-datasources.xml

Solution

Since current environment issuing MySQL Database with APIM 3.0.0. By default initial setup of APIM provides following configurations in file master-datasources.xml-

<maxActive>50</maxActive>

<maxWait>60000</maxWait>

Here MaxActive is the maximum number of database connections in use at one time. If a connection is requested while MaxActive other connections are in use, APIM will wait for maxWait milliseconds for a connection to be returned, and will then throw an exception if no connection was made available.

So, just change the values as per requirement either maxActive or maxWait or both. In my case, I changed to the following-

<maxActive>300</maxActive>

<maxWait>60000</maxWait>

Update

In addition to above changes if did not work, kindly refer to do the changes in deployment.toml (../../repository/conf/deployment.toml) file which overrides the parameters once wso2 apim starts-

New parameter= pool_options.maxActive

[database.apim_db]

type = "mysql"

url = "jdbc:mysql://localhost:3306/apimgtdb?autoReconnect=true&amp;useUnicode=true&amp;characterEncoding=UTF-8&amp;allowMultiQueries=true&amp;useSSL=false"

username = "carbon"

password = "carbonpass"

pool_options.maxActive=300

Now restart wso2 apim and verify the same at carbon as-

Config > Datasources > Select datasource from list > Datasource Configuration Parameters > Max Active

That's it and everything now worked fine.

Reference- https://apim.docs.wso2.com/en/3.0.0/reference/config-catalog/