When you try to start SAS 9.4 Cache Locator services, the services fail and the following error messages are generated:
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sas_gemfire_cache': FactoryBean threw exception on object creation; nested exception is com.gemstone.gemfire.SystemConnectException: Attempt to connect to distributed system timed out
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean
FactoryBeanRegistrySupport.java:149)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean
FactoryBeanRegistrySupport.java:102)
at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance
AbstractBeanFactory.java:1441)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean
AbstractBeanFactory.java:305)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean (AbstractBeanFactory.java:193)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference
(BeanDefinitionValueResolver.java:322)
. . .28 more lines. . .
Caused by: com.gemstone.gemfire.SystemConnectException: Attempt to connect to distributed system timed out
at com.gemstone.org.jgroups.protocols.pbcast.GMS.down(GMS.java:1593
at com.gemstone.org.jgroups.stack.Protocol.passDown(Protocol.java:788)
at com.gemstone.org.jgroups.protocols.FC.down(FC.java:357)
The problem occurs because of a miscommunication between the middle tier and the compute tier. To solve this problem, follow the steps below. There are three sets of steps that you need to implement:
- Steps for the middle-tier machine
- Steps for the compute-tier machines
- Final steps to restart Cache Locator and SASServer services on both middle-tier and compute-tier machines
Important Note: When you upgrade your SAS 9.4 environment to the latest SAS 9.4M6 (TS1M6) maintenance release, any of the following recommended manual implementations of the Java parameter will be lost. During maintenance upgrade, the underlying Pivotal Gemfire version is upgraded to the latest release, and new binary files are added within the same folder structure. However, SAS® Deployment Wizard backs up the previous manual configuration changes in the backup folder, from which you can move the custom Java parameters to the new gemfire configuration file accordingly. The previous configuration remains available in the following Backup folders.
- In UNIX operating environments: SAS-configuration-directory/Lev1/Web/Backup/gemfire.YYYY-MM-DD-HH.MM.SS
- In Microsoft Windows environments: SAS-configuration-directory\Lev1\Web\Backup\gemfire.YYYY-MM-DD-HH.MM.SS
Steps for the Middle-Tier Machine
- Stop the middle-tier services. Check the status of the middle-tier services and make sure that they are stopped correctly.
- Ensure that the parameter shown below is set in the location appropriate to your operating environment.
-Dsas.cache.locators=virtual-DNS-host-name_compute-tier[41415],virtual-DNS-host-name_middle-tier[41415]
In this argument:
- Replace virtual-DNS-host-name_compute-tier with the fully qualified virtual DNS host name of the compute-tier machine. (Or, in some cases, you need to replace it with the fully qualified virtual DNS host name of the machine on which the Web Infrastructure Platform data server is running.)
- Replace virtual-DNS-host-name_middle-tier with the fully qualified virtual DNS host name of the middle-tier machine.
Under UNIX, set the parameter in the setenv.sh script file that resides in the SAS-configuration-directory/Lev1/Web/WebAppServer/SASServer1_1/bin/ directory. Under Windows, set the parameter in the wrapper.conf file that resides in the SAS-configuration-directory/Lev1/Web/WebAppServer/SASServer1_1/conf/ directory. In these files, add the parameter above to the section that is labeled # Java Additional Parameter.
Note: If you have more than one web application server instance (for example, SASServer2_1, SASServer11_1, and so on), make sure that you implement these changes for those web application-server instances as well.
- In addition, add the Java virtual machine (JVM) argument that is shown below to the JVM_OPTS arguments in the location that is appropriate to your operating environment.
-Dgemfire.bind-address=IP-address
In this argument, replace IP-address with the actual IP address of your middle-tier machine.
Under UNIX, add the argument to the setenv.sh script file that resides in the SAS-configuration-directory/Lev1/Web/WebAppServer/SASServer1_1/bin/ directory. Under Windows, add the argument in the wrapper.conf file that resides in the SAS-configuration-directory/Lev1/Web/WebAppServer/SASServer1_1/conf/ directory. In these files, add the argument above to the section that is labeled # Java Additional Parameter.
- Stop the Cache Locator processes on the middle-tier machine. Check the gemfire.log file to make sure that the processes stopped completely.
- Navigate to the GemFire instance directory (for example, SAS-configuration-directory/Lev1/Web/gemfire/instances/ins_41415/). Clean up this directory by removing the following files:
- gemfire.log
- gemfireCommand.log
- locator41415state.dat (or a similar file that is based on your configuration instance)
- Add the additional JVM argument that is shown below to the JAVA_ARGS option in the location that is appropriate to your operating environment.
-Dgemfire.bind-address=IP-address
In this argument, replace IP-address with the actual IP address of your middle-tier machine.
Under UNIX, add the argument on the middle-tier machine to the gemfire-start-locator-sas.sh script file that resides in the SAS-configuration-directory/Lev1/Web/gemfire/instances/ins_41415/. Under Windows, add the argument to the wrapper.conf file that resides in the SAS-configuration-directory/Lev1/Web/gemfire/instances/ins_41415/. In these files, add the argument above to the section that is labeled # Java Additional Parameter.
- Note: This step is for UNIX environments only. On your middle-tier machine, add the following items to the gemfire-locator.sh script file that resides in SAS-configuration-directory/Lev1/Web/gemfire/instances/ins_41415/:
- Add the following bind_address argument to the gemfire-locator.sh script:
bind_address=virtual-DNS-host-name
export bind_address
In this argument, replace virtual-DNS-host-name with the fully qualified virtual DNS host name of your middle-tier machine.
- Add the following argument to the stop) section as well as the restart) section in the gemfire-locator.sh script:
-address=$bind_address
When you add the argument, the stop) and restart) sections should appear as follows:
stop)
$LOCATOR_HOME/bin/gemfire-sas stop-locator -port=$LOCATOR_PORT -dir=$LOCATOR_INSTANCE_HOME
-address=$bind_address -Dgemfire.log-file=$LOCATOR_INSTANCE_HOME/gemfire.log
restart)
$LOCATOR_HOME/bin/gemfire-sas stop-locator -port=$LOCATOR_PORT -dir=$LOCATOR_INSTANCE_HOME
-address=$bind_address -Dgemfire.log-file=$LOCATOR_INSTANCE_HOME/gemfire.log
Steps for the Compute-Tier Machine
- Stop the Cache Locator processes on the compute-tier machine. Check the gemfire.log file to ensure that the processes stop completely.
- Add the JVM argument that is shown below to the JAVA_ARGS variable in the location that is appropriate to your operating environment.
-Dgemfire.bind-address=IP-address
In this argument, replace IP-address with the actual IP address of your compute-tier machine.
Under UNIX, add the argument on the compute-tier machine to the gemfire-start-locator-sas.sh script file that resides in SAS-configuration-directory/Lev1/Web/gemfire/instances/ins_41415/. Under Windows, add the argument to the wrapper.conf file that resides in SAS-configuration-directory/Lev1/Web/gemfire/instances/ins_41415/conf/. In these files, add the parameter below to the section that is labeled # Java Additional Parameter.
- Note: This step is for UNIX environments only. Add the following bind_address argument to the gemfire-locator.sh script file that resides in SAS-configuration-directory/Lev1/Web/gemfire/instances/ins_41415/:
bind_address=virtual-DNS-host-name
export bind_address
Replace virtual-DNS-host-name with the fully qualified virtual DNS host name of your compute-tier machine.
- Also append the following line as an additional argument in the stop) and restart) sections of the gemfire-locator.sh script:
-address=$bind_address
When you add the argument, the stop) and restart) sections should appear as follows:
stop)
$LOCATOR_HOME/bin/gemfire-sas stop-locator -port=$LOCATOR_PORT -dir=$LOCATOR_INSTANCE_HOME
-address=$bind_address -Dgemfire.log-file=$LOCATOR_INSTANCE_HOME/gemfire.log
restart)
$LOCATOR_HOME/bin/gemfire-sas stop-locator -port=$LOCATOR_PORT -dir=$LOCATOR_INSTANCE_HOME
-address=$bind_address -Dgemfire.log-file=$LOCATOR_INSTANCE_HOME/gemfire.log
Final Steps
- Start the Cache Locator on your compute-tier machine. Check the gemfire.log file to make sure that the locator starts successfully.
- Start the Cache Locator on either your middle-tier or web-tier machine. Check the gemfire.log file to make sure that the locator starts successfully.
- Start SASServer1__1. Check the server.log file to make sure that the server starts successfully.