Aparna Karunarathna's Blog

Tuesday, February 27, 2024

How to enable GC logs in Java8

Step-by-step guide

Add the following lines to the integrator.sh file under $JVM_MEM_OPTS \ as below.

`$JVM_MEM_OPTS \`
   `-XX:+HeapDumpOnOutOfMemoryError \`
   `-XX:HeapDumpPath="$CARBON_HOME/repository/logs/heap-dump.hprof"` `\`
   `-XX:+PrintGC \`
   `-XX:+PrintGCDetails \`
   `-XX:+PrintGCDateStamps \`
   `-Xloggc:"$PRODUCT_HOME/logs/gc.log"` `\`
`$JAVA_OPTS \`

It will print the GC logs into gc.log file inside <product_location>/logs/ directory.

Friday, February 23, 2024

If you have encountered an issue of 'Too many open files', you can monitor the open file counts from the following commands.

TID: [-1] [] [202X-0X-XX 10:30:03,426] WARN {org.apache.synapse.transport.passthru.PassThroughHttpSSLListener} - System may be unstable: HTTPS ListeningIOReactor encountered a checked exception : Too many open files {org.apache.synapse.transport.passthru.PassThroughHttpSSLListener} java.io.IOException: Too many open files at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:424

Monitor the open file count and the details for specific processes via the below command.

lsof -a -d ^mem -d ^cwd -d ^rtd -d ^txt -d ^DEL -p > out_with_wso2_pid.txt

Monitor the open file count and the details from all processes running in the VM via the below command.

lsof -a -d ^mem -d ^cwd -d ^rtd -d ^txt -d ^DEL > out_for_all.txt

Thursday, February 8, 2024

How to check Master, Slave database differences

If you observed Master, Slave database difference, please execute the following SQL statements and check the results.

SELECT CONCAT(ROUND(SUM(data_length) / (1024*1024*1024),2),'GB') Data_Size, CONCAT(ROUND(SUM(index_length)/ (1024*1024*1024),2),'GB') Index_Size, CONCAT(ROUND((sum(data_length)+sum(index_length))/(1024*1024*1024),2),'GB') Total_Size  FROM information_schema.TABLES  WHERE TABLE_SCHEMA NOT IN ('information_schema', 'performance_schema', 'sys')  GROUP BY NULL;

SELECT table_schema, table_name, CONCAT(ROUND((data_free / 1024 / 1024),2),'MB') data_free  FROM information_schema.tables  WHERE table_schema NOT IN ('mysql', 'information_schema', 'performance_schema', 'sys')    AND ENGINE LIKE 'InnoDB' AND data_free > 100 * 1024 *  1024;

Thursday, May 11, 2023

How to update a MySQL master-slave setup

Sometimes you'll get a requirement to upgrade the MySQL setup for VA fixes.

Upgrading Steps

Upgrade the Slave DB node
Sync the Master DB to the Slave DB node
Upgrade the Master DB node
Sync the Slave DB to the Master

Slave and Master nodes Upgrading steps
Slave Node

Download required Mysql rpm version
Stop Slave
Store/Save Slave (including Master) Coordinates
Shutdown Slave MySQL Server
Backup Slave Datadir : (If possible create an OS backup of this datadir in case its needed to fall back).
i. Taking DB dump
nohup sh -c 'mysqldump -u root -pxxxxxxxxx --all-databases --single-transaction --quick --lock-tables=false --compress --order-by-primary > /data1 /mysqlBackup/dataDump/masterBack.sql' &
ii. Taking data directory backup
datadir=/var/lib/mysql
iii. Taking config and log back
/etc/my.cnf
log-error=/var/log/mysqld.log
Install new version :
i. RPM Install
Typical command looks like this : "rpm Uvh mysql-community {server,client,common,libs-compat,libs}-*.rpm"
Start Slave Server
Execute "mysql_upgrade" : Typical command is:
mysql_upgrade --user=<user_name> --password --host=<host_name> --port=<port_num> [other_options]
Start Slave again

Master Node

Stop Slave
Store/Save Slave (including Master) Coordinates
Shutdown Master MySQL Server
Backup Master Datadir : (If possible create an OS backup of this datadir in case its needed to fall back)
i. Taking DB dump
nohup sh -c 'mysqldump -u root -pxxxxxxxxx --all-databases --single-transaction --quick --lock-tables=false --compress --order-by-primary > /data1/mysqlBackup/dataDump/masterBack.sql' &
ii. Taking data directory backup
datadir=/var/lib/mysql
iii. Taking config and log back
/etc/my.cnf
log-error=/var/log/mysqld.log
Install new version :
i. RPM Install :
Typical command looks like this : "rpm Uvh mysql-community{server,client,common,libs-compat,libs}-*.rpm"
Start Master Server
Execute "mysql_upgrade" : Typical command is
mysql_upgrade --user=<user_name> --password --host=<host_name> --port=<port_num> [other_options]
Start Slave again
i. Perform validation
ii. via application
iii. Review MySQL Server Error Log file

Tuesday, February 11, 2020

How to upgrade MySQL in single command

Sometimes you might have encountered dependency issue when upgrading MySQL version. In order to prevent the dependency issue, you can upgrade the MySQL db in single command like following.

Download the required Mysql version to be upgraded and go to the rpm file location, then execute the following command.

rpm Uvh mysql-community-{server,client,common,libs-compat,libs}-*.rpm

Friday, July 4, 2014

How to use Jmeter LDAP reuest sampler to populate users to a LDAP

This post describes how to populate users to LDAP from Jmeter LDAP request sampler.

In this scenario I'm going to populate 100000 users to openLDAP.

Now lets configure the Jmeter test plan. First add a LDAP Request sampler and configure the sampler as shown in below. For user name I'm going to add username and post-fix to the username. To do that add a counter config element to get a auto increment number as shown in below screen-shot. Then assigned that variable to end of the username user${no}. I'm giving a same password for all users.

This kind of user population is more useful for test the system behaviours with large users base.

LDAP Request sampler configurations
Username: cn=admin,dc=wso2qa,dc=com
Password: qwert
Server Name:192.168.5.5
Port:389
Entry DN: cn=user${no},ou=people,dc=wso2qa,dc=com
Properties:
cn - user${no}
sn - user${no}
userPassword - user123

Then configure Tread Properties as fallows, depends on your requirement. In this scenario I'm populating 100000 users to openLDAP. Therefore configure the Tread Properties as follows.
Number of treads : 1
Ramp-up period : 1
Loop count : 100000

Then execute the jmeter script.

URL appending issue for REST invocations

Have you experienced the URL appending issue when invoking a WSO2 ESB proxy service as REST ?

In this scenario I'm using a 'JsonReqestProxy' proxy service to write into a file via vfs protocol and JsonReqestProxy is exposed as API in WSO2 API Manager. When we do Post REST invocation to write a file, you'll get URL appending error similar to the following issue.

ERROR {org.apache.synapse.core.axis2.Axis2Sender} - Unexpected error during sending message out {org.apache.synapse.core.axis2.Axis2Sender}
org.apache.axis2.AxisFault: Couldn't send the message to file : file:///home/aparna/Release_testing/abc/xyz/response.txt/services/JsonReqestProxy

In the above issue "/services/JsonReqestProxy" path has beed appended to the endpoint path, so therefore ESB cannot find the correct path and it will throw an error.

To resolve URL Appending issue you have to set the following property inside the InSequnce.
<property name="REST_URL_POSTFIX" scope="axis2" action="remove"/>

Likewise, there are several HTTP Transport properties that can be configured in the WSO2 ESB.

Wednesday, June 11, 2014

How to use Jmeter JMS publisher & subscriber with WSO2 Message Broker

This post describes basic steps to how to use Jmeter JMS publisher and subscriber with WSO2 Message Broker.

First copy all the client libraries from WSO2MB_HOME/client-lib/ to the JMETER_HOME/lib/ directory and start the Jmeter.

For WSO2 MB you cannot connect directly like activemq connection(see my previous post), you have to use a myjndi.propertise file. 'myjndi.propertise' file contains the connection details and Queue/Topic details.

myjndi.properties

#  Copyright (c) 2011, WSO2 Inc. (http://wso2.com) All Rights Reserved.
#
#  Licensed under the Apache License, Version 2.0 (the "License");
#  you may not use this file except in compliance with the License.
#  You may obtain a copy of the License at
#
#        http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
#  limitations under the License.
#

# register some connection factories
# connectionfactory.[jndiname] = [ConnectionURL]
connectionfactory.Connectionfactory = amqp://admin:admin@clientID/carbon?brokerlist='tcp://192.168.10.5:5682'

# register some queues in JNDI using the form
# queue.[jndiName] = [physicalName]
queue.TestQ = TestQ

# register some topics in JNDI using the form
# topic.[jndiName] = [physicalName]
topic.TestT = TestT

JMS Publisher (Queues/Topics)

Initial Connection Factory: org.wso2.andes.jndi.PropertiesFileInitialContextFactory
Provider URL: /home/aparna/Blogging/myjndi.properties
Connection Factory: ConnectionFactory
Destination: TestQ ( or TestT - you can add hierarchical topic as TestT.L1T)
JMS Properties: SOAPAction - urn:placeOrder and Content-Type - text/xml
Message: <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Header xmlns:wsa="http://www.w3.org/2005/08/addressing"><wsa:ReplyTo><wsa:Address>http://www.w3.org/2005/08/addressing/none</wsa:Address></wsa:ReplyTo><wsa:MessageID>urn:uuid:3460afc7-dda1-4eed-ba2e-acacda943384</wsa:MessageID><wsa:Action>urn:placeOrder</wsa:Action></soapenv:Header><soapenv:Body><m0:placeOrder xmlns:m0="http://services.samples"><m0:order><m0:price>96.78974610308872</m0:price><m0:quantity>19374</m0:quantity><m0:symbol>IBM</m0:symbol></m0:order></m0:placeOrder></soapenv:Body></soapenv:Envelope>

Queue or Topic (sample request)

JMS Subscriber (Queues/Topics)

Initial Connection Factory: org.wso2.andes.jndi.PropertiesFileInitialContextFactory
Provider URL: /home/aparna/Blogging/myjndi.properties
Connection Factory: Connectionfactory
Destination: TestQ

Queue or Topic (sample request)

Now you can execute publisher & subscriber jmeter scripts. Published messages are getting from Subscriber.

Wednesday, March 12, 2014

How to configure WSO2 BPS worker/manager separated cluster

This blog describes how to configure WSO2 ELB fronted WSO2 BPS worker/manager separated cluster setup

WSO2 BPS worker/manager separated cluster deployment diagram

Pre-requisite:
Download and extract the wso2elb-2.1.0.zip, wso2bps-3.1.0.zip and wso2greg-4.6.0.zip files.

All WSO2 products are shipped embedded H2 registry, for clustering purpose we need to configure with a database. Here I’m using the MySQL 5.5 as database. I’m referred ELB home as ELB_HOME, Greg home as GREG_HOME, BPS manager node as BPS_MANAGER, BPS node 1 as BPS_NODE1 and BPS node 2 as BPS_NODE2.

We will use 3 servers to deploy the clustered setup.
Server 1(192.168.10.1): ELB (defaults ports)
Server 2(192.168.10.2): BPS manager(defaults ports) & BPS Node2 (offset=100)
Server 3(192.168.10.3): BPS Node 1(defaults ports) & Greg (offset=100)

BPS cluster database configuration

Instance Name	Registry Space	Instance Name Registry Space Mysql database name/mounted path
BPS Manager	local	bpsman
	config	greg /_system/BPSConfig
	governance	greg /_system/governance
	user-manager	greg
BPS Node1	local	bpsnode1
	config	greg /_system/BPSConfig
	governance	greg /_system/governance
	user-manager	greg
BPS Node2	local	bpsnode2
	config	greg /_system/BPSConfig
	governance	greg /_system/governance
	user-manager	greg
G-Reg	All	greg

ELB Configurations
1. First will configure the ELB and go to ELB_HOME/repository/conf/loadbalancer.conf and add the following entries under services. Group management port is the ELB listening port.

bpsserver {
 domains{
    wso2.bps.domain {
       tenant_range *;
       group_mgt_port 4005;
       mgt {
               hosts mgt.bps.wso2.com;
       }
       worker {
              hosts wkr.bps.wso2.com;
       }
    }
  }
}

2. Go to <ELB_HOME>/repository/conf/axis2/axis2.xml file and specify the IP address for “localMemberHost” to be exposed to members of the cluster.

<parameter name="localMemberHost">192.168.10.1</parameter>

3. Go to ELB_HOME/bin and start the server
./wso2server.sh

4. Now will create all necessary databases for this setup according to above diagram.

create database regdb; // Registry DB that used to mount to all BPS nodes
create database bpsman; // BPS manager local database
create database bpsnode1; // BPS Node1 local database
create database bpsnode2; // BPS Node2 local database
create database bpsdb; // BPS DB, that will be shared with all BPS
create database bpsuserdb; // User DB, that will be shared with all BPS and Greg Nodes

Greg Configurations
1. Configure the Greg server, we are not going to expose this to outside therefore we are not going to configure any ELB settings here. Go to GREG_HOME/repository/conf/datasources/master-datasources.xml file and comment the original carbon <datasource> section and add following database configurations. (IP, url, username and password should change). In here we are configuring Registry database and user database.

     <datasource>
            <name>WSO2_CARBON_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2CarbonDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                  <url> jdbc:mysql://<IP>:3306/regdb?autoReconnect=true</url>
                    <username>wso2carbon</username>
                    <password>wso2carbon</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
        
     <datasource>
            <name>WSO2_USER_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2UserDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                  <url> jdbc:mysql://<IP>:3306/bpsuserdb?autoReconnect=true</url>
                    <username>wso2carbon</username>
                    <password>wso2carbon</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

2. Configure the user-mgt.xml file, go to GREG_HOME/repository/conf/user-mgt.xml

<Property name="dataSource">jdbc/WSO2UserDB</Property>

3. Configure the registry.xml file, go to GREG_HOME/repository/conf/registry.xml

<dbConfig name="wso2registry">
  <dataSource>jdbc/WSO2CarbonDB</dataSource>
</dbConfig>

4. Configure the offset for greg, to do that go to the GREG_HOME/repository/conf/carbon.xml file and set the offset value to "100".

<Offset>100</Offset>

5. Now copy the mysql database driver to GREG_HOME/repository/components/lib/ directory

BPS Manager Configurations
1. Now configure the BPS manager node, first will start from axis2.xml file.
Go to <BPS_MANAGER>/repository/conf/axis2/axis2.xml and update as follows.

Enable the hazelcast clustering

<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">

Change the membershipScheme to wka

<parameter name="membershipScheme">wka</parameter>

Update the same cluster domain as defined in loadbalancer.conf

<parameter name="domain">wso2.bps.domain</parameter>

Update the server IP and port

<parameter name="localMemberHost">192.168.10.2</parameter>
<parameter name="localMemberPort">4100</parameter>

Change the "subDomain" property to "mgt" as defined in loadbalancer.conf

<parameter name="properties">
<property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
<property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
<property name="subDomain" value="mgt"/>
</parameter>

Update load balancer’s IP or port (group management port) as defined in loadbalancer.conf

<members>
   <member>
      <hostName>192.168.10.1</hostName>
      <port>4005</port>
   </member>
</members>

2. Configure carbon.xml file, go to <BPS_MANAGER>/repository/conf/carbon.xml

Update the HostName and MgtHostName elements defined in loadbalancer.conf

<HostName>wkr.bps.wso2.com</HostName>
<MgtHostName>mgt.bps.wso2.com</MgtHostName>

Update the svn configuration as follows

   <DeploymentSynchronizer>
        <Enabled>true</Enabled>
        <AutoCommit>true</AutoCommit>
        <AutoCheckout>true</AutoCheckout>
        <RepositoryType>svn</RepositoryType>
        <SvnUrl>http://svnrepo.example.com/repos/</SvnUrl>
        <SvnUser>bps</SvnUser>
        <SvnPassword>bps123</SvnPassword>
        <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
   </DeploymentSynchronizer>

3. Next configure catalina-server.xml file, go to /repository/conf/tomcat/catalina-server.xml as follows.

<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
           port="9763"
           proxyPort="8280"
 
<Connector protocol="org.apache.coyote.http11.Http11NioProtocol"
           port="9443"
           proxyPort="8243"

4. Then configure master-datasources.xml file, go to BPS_MANAGER/repository/conf/datasources/master-datasources.xml

   <datasource>
            <name>WSO2_Local_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2LocalDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                  <url> jdbc:mysql://<IP>:3306/bpsman?autoReconnect=true</url>
                    <username>wso2carbon</username>
                    <password>wso2carbon</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
    
    <datasource>
            <name>WSO2_REG_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2RegDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                  <url> jdbc:mysql://<IP>:3306/regdb?autoReconnect=true</url>
                    <username>wso2carbon</username>
                    <password>wso2carbon</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
        
     <datasource>
            <name>WSO2_USER_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2UserDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                  <url>jdbc:mysql://<IP>:3306/bpsuserdb?autoReconnect=true</url>
                    <username>wso2carbon</username>
                    <password>wso2carbon</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

5. Configure the user-mgt.xml file, go to BPS_MANAGER/repository/conf/user-mgt.xml

<Property name="dataSource">jdbc/WSO2UserDB</Property>

6. Configure the registry.xml file, go to BPS_MANAGER/repository/conf/registry.xml

    <dbConfig name="wso2registry">
        <dataSource>jdbc/WSO2LocalDB</dataSource>
    </dbConfig>
    
    <dbConfig name="bpsMountRegistry">
        <dataSource>jdbc/WSO2RegDB</dataSource>
    </dbConfig>

   <remoteInstance url="https://<IP of G-Reg>:<HTTPS port of G-Reg>/registry">
     <id>Mount1</id>
     <dbConfig>bpsMountRegistry</dbConfig>
     <readOnly>false</readOnly>
     <registryRoot>/</registryRoot>
     <cacheId>wso2carbon@jdbc:mysql://<IP>:3306/regdb</cacheId>
   </remoteInstance>

 <mount path="/_system/config" overwrite="true">
     <instanceId>Mount1</instanceId>
     <targetPath>/_system/bpsConfig</targetPath>
 </mount>

 <mount path="/_system/governance" overwrite="true">
     <instanceId>Mount1</instanceId>
     <targetPath>/_system/governance</targetPath>
 </mount>

7. Now configure the BPEL database settings as follows.

synapse.datasources=bpsds
synapse.datasources.icFactory=com.sun.jndi.rmi.registry.RegistryContextFactory
## If following property is present , then assumes that there is an external JNDI provider and will not start a RMI registry
#synapse.datasources.providerUrl=rmi://localhost:2199
synapse.datasources.providerPort=2199

synapse.datasources.bpsds.registry=JNDI
synapse.datasources.bpsds.type=BasicDataSource
synapse.datasources.bpsds.driverClassName=com.mysql.jdbc.Driver
synapse.datasources.bpsds.url= jdbc:mysql://<IP>:3306/bpsdb?autoReconnect=true
synapse.datasources.bpsds.username=wso2carbon
synapse.datasources.bpsds.password=wso2carbon
synapse.datasources.bpsds.validationQuery=SELECT 1
synapse.datasources.bpsds.dsName=bpsds
synapse.datasources.bpsds.maxActive=100
synapse.datasources.bpsds.maxIdle=20
synapse.datasources.bpsds.maxWait=10000

8. Now copy the mysql database driver to BPS_MANAGER/repository/components/lib/ directory and copy the svnClientBundle-1.0.0.jar to BPS_MANAGER/repository/components/dropins/ directory.

BPS Worker Node1 Configurations
Now take a copy of the BPS Manager pack and will change only the worker specific configurations. We need to update axis2.xml, carbon.xml and master-datasources.xml files.

1. Now configure the BPS Worker node1, first will start from axis2.xml file.
Go to BPS_WNODE1/repository/conf/axis2/axis2.xml and update as follows.

Update the server IP and port

<parameter name="localMemberHost">192.168.10.3</parameter>
<parameter name="localMemberPort">4100</parameter>

Change the "subDomain" property to "worker" as defined in loadbalancer.conf

<parameter name="properties">
<property name="backendServerURL" value="https://${hostName}:${httpsPort}/services/"/>
<property name="mgtConsoleURL" value="https://${hostName}:${httpsPort}/"/>
<property name="subDomain" value="worker"/>
</parameter>

2. Configure carbon.xml file, go to BPS_WNODE1/repository/conf/carbon.xml

Update the svn configuration as follows (here we are setting auto commit as false).

   <DeploymentSynchronizer>
        <Enabled>true</Enabled>
        <AutoCommit>false</AutoCommit>
        <AutoCheckout>true</AutoCheckout>
        <RepositoryType>svn</RepositoryType>
        <SvnUrl>http://svnrepo.example.com/repos/</SvnUrl>
        <SvnUser>bps</SvnUser>
        <SvnPassword>bps123</SvnPassword>
        <SvnUrlAppendTenantId>true</SvnUrlAppendTenantId>
   </DeploymentSynchronizer>

3. Configure master-datasources.xml file, go to BPS_WNODE1/repository/conf/datasources/master-datasources.xml

      <datasource>
            <name>WSO2_Local_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2LocalDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                  <url> jdbc:mysql://<IP>:3306/bpsnode1?autoReconnect=true</url>
                    <username>wso2carbon</username>
                    <password>wso2carbon</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
       </datasource>

4. Configure the registry.xml file, go to BPS_WNODE1/repository/conf/registry.xml

Change the readOnly property to “true”

<remoteInstance url="https://<IP of G-Reg>:<HTTPS port of G-Reg>/registry">
     <id>Mount1</id>
     <dbConfig>bpsMountRegistry</dbConfig>
     <readOnly>true</readOnly>
     <registryRoot>/</registryRoot>
     <cacheId>wso2carbon@jdbc:mysql://<IP>:3306/regdb</cacheId>
 </remoteInstance>

BPS Worker Node 2 Configurations
Now take a copy of the BPS Manager pack and will change only the worker specific configurations. We need to update axis2.xml, carbon.xml and master-datasources.xml files.

1. Configure the BPS Worker node1, first will start from axis2.xml file. Go to BPS_WNODE2/repository/conf/axis2/axis2.xml and update as follows.

Update the server IP and port

<parameter name="localMemberHost">192.168.10.2</parameter>
<parameter name="localMemberPort">4200</parameter>

2. Configure carbon.xml file, go to BPS_WNODE2/repository/conf/carbon.xml and set the offset value to "100.

<Offset>100</Offset>

3. Configure master-datasources.xml file, go to BPS_WNODE2/repository/conf/datasources/master-datasources.xm.

        <datasource>
            <name>WSO2_Local_DB</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2LocalDB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                  <url> jdbc:mysql://<IP>:3306/bpsnode2?autoReconnect=true</url>
                    <username>wso2carbon</username>
                    <password>wso2carbon</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>

Now configurations are completed and ELB is already started, we need to start servers one by one for first time.

Start the BPS manager node with -Dsetup, when start the servers with –Dsetup all the necessary tables are getting created in the database.

Now go to BPS_MANAGER/bin directory and start the server with following command.

./wso2server.sh –Dsetup

When server started you will see following kind of logs in the ELB side.

TID: [0] [ELB] [<date>]  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent} -  Member joined [9f29f21e-45e5-41f8-95bf-5a1bacc8e5db]: /192.168.10.2:4100 {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent}
TID: [0] [ELB] [<date>]  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent} -  Member joined [9f29f21e-45e5-41f8-95bf-5a1bacc8e5db]: /192.168.10.2:4100 {org.wso2.carbon.core.clustering.hazelcast.HazelcastGroupManagementAgent}
TID: [0] [ELB] [<date>]  INFO {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils} -  Added member: Host:192.168.10.2, Remote Host:null, Port: 4100, HTTP:8280, HTTPS:8243, Domain: wso2.bps.domain, Sub-domain:mgt, Active:true {org.wso2.carbon.core.clustering.hazelcast.util.MemberUtils}

Next start the BPS Node 1 with –Dsetup & -DworkerNode=true parameters, when start the server with DworkerNode parameter, it’ll remove all the UI bundle from the worker node.

Go to BPS_WNODE1/bin & BPS_WNODE2/bin and start servers.

./wso2server.sh –Dsetup -DworkerNode=true start

When worker nodes are started you'll see above mentioned logs in the ELB side for two worker nodes as well.

Now login to the Management console using admin/admin

https://mgt.bps.wso2.com:8243/carbon

When you logged in you can see the user logging logs in mgt backend console.

TID: [0] [BPS] [2014-01-13 11:07:58,565]  INFO {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil} -  'admin@carbon.super [-1234]' logged in at [2014-01-13 11:07:58,564+0530] {org.wso2.carbon.core.services.util.CarbonAuthenticationUtil}

Then deploy a sample BPEL artifact “CustomerInfo.zip” from the UI and when it successful you can see the deploying logs from mgt, worker consoles.

Management logs

TID: [0] [BPS] [2014-01-13 11:15:15,201]  INFO {org.springframework.context.support.ClassPathXmlApplicationContext} -  Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@417eb61: startup date [Mon Jan 13 11:15:15 GMT+05:30 2014]; root of context hierarchy {org.springframework.context.support.ClassPathXmlApplicationContext}
TID: [0] [BPS] [2014-01-13 11:15:15,203]  INFO {org.springframework.beans.factory.xml.XmlBeanDefinitionReader} -  Loading XML bean definitions from class path resource [beans.xml] {org.springframework.beans.factory.xml.XmlBeanDefinitionReader}
TID: [0] [BPS] [2014-01-13 11:15:15,252]  INFO {org.apache.ode.bpel.compiler.BpelC} -  info: [CompilingProcess] Compiling BPEL process. {org.apache.ode.bpel.compiler.BpelC}
TID: [0] [BPS] [2014-01-13 11:15:17,136]  INFO {org.wso2.carbon.bpel.core.ode.integration.store.TenantProcessStoreImpl} -  Deploying BPEL archive: /home/wso2/4.2.0/BPS320/Mgr/9jan/wso2bps-3.2.0/repository/deployment/server/bpel/CustomerInfo.zip {org.wso2.carbon.bpel.core.ode.integration.store.TenantProcessStoreImpl}
TID: [0] [BPS] [2014-01-13 11:15:17,164]  INFO {org.springframework.context.support.ClassPathXmlApplicationContext} -  Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@7c50b04f: startup date [Mon Jan 13 11:15:17 GMT+05:30 2014]; root of context hierarchy {org.springframework.context.support.ClassPathXmlApplicationContext}
TID: [0] [BPS] [2014-01-13 11:15:17,166]  INFO {org.springframework.beans.factory.xml.XmlBeanDefinitionReader} -  Loading XML bean definitions from class path resource [beans.xml] {org.springframework.beans.factory.xml.XmlBeanDefinitionReader}
TID: [0] [BPS] [2014-01-13 11:15:17,168]  INFO {org.wso2.carbon.bpel.core.ode.integration.store.TenantProcessStoreImpl} -  Starting deployment of processes from directory /home/wso2/4.2.0/BPS320/Mgr/9jan/wso2bps-3.2.0/repository/bpel/-1234/CustomerInfo-6 {org.wso2.carbon.bpel.core.ode.integration.store.TenantProcessStoreImpl}
TID: [0] [BPS] [2014-01-13 11:15:17,168]  INFO {org.springframework.context.support.ClassPathXmlApplicationContext} -  Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@1567524c: startup date [Mon Jan 13 11:15:17 GMT+05:30 2014]; root of context hierarchy {org.springframework.context.support.ClassPathXmlApplicationContext}
TID: [0] [BPS] [2014-01-13 11:15:17,170]  INFO {org.springframework.beans.factory.xml.XmlBeanDefinitionReader} -  Loading XML bean definitions from class path resource [beans.xml] {org.springframework.beans.factory.xml.XmlBeanDefinitionReader}
TID: [0] [BPS] [2014-01-13 11:15:17,206]  INFO {org.apache.ode.bpel.compiler.BpelC} -  info: [CompilingProcess] Compiling BPEL process. {org.apache.ode.bpel.compiler.BpelC}
TID: [0] [BPS] [2014-01-13 11:15:17,393]  INFO {org.wso2.carbon.core.deployment.DeploymentInterceptor} -  Deploying Axis2 service: CustomerInfoService {super-tenant} {org.wso2.carbon.core.deployment.DeploymentInterceptor}
TID: [0] [BPS] [2014-01-13 11:15:17,419]  INFO {org.apache.axis2.deployment.DeploymentEngine} -  Deploying Web service: CustomerInfoService -  {org.apache.axis2.deployment.DeploymentEngine}
TID: [0] [BPS] [2014-01-13 11:15:17,453]  INFO {org.apache.ode.bpel.engine.BpelServerImpl} -  Registered process {http://wso2.org/bps/samples/CustomerInfo}CustomerInfo-6. {org.apache.ode.bpel.engine.BpelServerImpl}
TID: [0] [BPS] [2014-01-13 11:15:21,109]  INFO {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask} -  Sent [SynchronizeRepositoryRequest{tenantId=-1234, tenantDomain='carbon.super', messageId=03bdc35e-89bb-4780-ba43-00f7fbd09e0c}] {org.wso2.carbon.core.deployment.CarbonDeploymentSchedulerTask}

Worker logs

TID: [0] [BPS] [2014-01-13 11:15:21,117]  INFO {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusterMessageListener} -  Received ClusteringMessage: SynchronizeRepositoryRequest{tenantId=-1234, tenantDomain='carbon.super', messageId=03bdc35e-89bb-4780-ba43-00f7fbd09e0c} {org.wso2.carbon.core.clustering.hazelcast.HazelcastClusterMessageListener}
TID: [0] [BPS] [2014-01-13 11:15:21,117]  INFO {org.wso2.carbon.core.deployment.SynchronizeRepositoryRequest} -  Received [SynchronizeRepositoryRequest{tenantId=-1234, tenantDomain='carbon.super', messageId=03bdc35e-89bb-4780-ba43-00f7fbd09e0c}]  {org.wso2.carbon.core.deployment.SynchronizeRepositoryRequest}
TID: [0] [BPS] [2014-01-13 11:15:21,117]  INFO {org.wso2.carbon.core.deployment.SynchronizeRepositoryRequest} -  Updating repo update required parameter {org.wso2.carbon.core.deployment.SynchronizeRepositoryRequest}
TID: [0] [BPS] [2014-01-13 11:15:26,101]  INFO {org.wso2.carbon.bpel.core.ode.integration.store.TenantProcessStoreImpl} -  Deploying BPEL archive: /home/wso2/4.2.0/BPS320/Wrk1/9jan/wso2bps-3.2.0/repository/deployment/server/bpel/CustomerInfo.zip {org.wso2.carbon.bpel.core.ode.integration.store.TenantProcessStoreImpl}
TID: [0] [BPS] [2014-01-13 11:15:26,264]  INFO {org.wso2.carbon.bpel.core.ode.integration.store.TenantProcessStoreImpl} -  Looking for BPEL package in file system for deployment unit CustomerInfo-6 {org.wso2.carbon.bpel.core.ode.integration.store.TenantProcessStoreImpl}
TID: [0] [BPS] [2014-01-13 11:15:26,492]  INFO {org.springframework.context.support.ClassPathXmlApplicationContext} -  Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@261abf16: startup date [Mon Jan 13 11:15:26 GMT+05:30 2014]; root of context hierarchy {org.springframework.context.support.ClassPathXmlApplicationContext}
TID: [0] [BPS] [2014-01-13 11:15:26,495]  INFO {org.springframework.beans.factory.xml.XmlBeanDefinitionReader} -  Loading XML bean definitions from class path resource [beans.xml] {org.springframework.beans.factory.xml.XmlBeanDefinitionReader}
TID: [0] [BPS] [2014-01-13 11:15:26,527]  INFO {org.apache.axis2.deployment.DeploymentEngine} -  Deploying Web service: CustomerInfoService -  {org.apache.axis2.deployment.DeploymentEngine}
TID: [0] [BPS] [2014-01-13 11:15:26,540]  INFO {org.apache.ode.bpel.engine.BpelServerImpl} -  Registered process {http://wso2.org/bps/samples/CustomerInfo}CustomerInfo-6. {org.apache.ode.bpel.engine.BpelServerImpl}

Now you can invoke the BPEL service using tryIt or soapUI.