During the time of collection, the Java Virtual Machine (JVM) is paused. If a cache-locator node attempts to check the health of the Pivotal GemFire member that is running within the paused JVM during this time frame, the GemFire member appears to be down because it does not respond. The Cache Locator then acts upon the reported failed state of the GemFire member, and it removes the GemFire member from the distribution list.
When the failure occurs, the following messages occur in the SASLogon.log and server.log files:
SASLogon.log file:
2019-04-22 11:55:28,490 [tomcat-http--4] ERROR com.sas.svcs.security.authentication.manager.AuthenticationManagerImpl - com.sas.svcs.security.authentication.handler.support.ClearpassDecorator threw error authenticating [username: principal=rdt0001$$#$$person=rdt0001]
com.gemstone.gemfire.distributed.DistributedSystemDisconnectedException: GemFire on birdsmp08(30620)<v1>:46556 started at Thu Apr 18 15:40:41 EDT 2019: Message distribution has terminated, caused by com.gemstone.gemfire.ForcedDisconnectException: This member has been forced out of the distributed system. Reason='did not respond to are-you-dead messages'
at com.gemstone.gemfire.distributed.internal.DistributionManager$Stopper.generateCancelledException(DistributionManager.java:781)
at com.gemstone.gemfire.distributed.internal.InternalDistributedSystem$Stopper.generateCancelledException(InternalDistributedSystem.java:974)
at com.gemstone.gemfire.internal.cache.GemFireCacheImpl$Stopper.generateCancelledException(GemFireCacheImpl.java:1298)
at com.gemstone.gemfire.CancelCriterion.checkCancelInProgress(CancelCriterion.java:82)
at com.gemstone.gemfire.internal.cache.LocalRegion.checkRegionDestroyed(LocalRegion.java:7810)
. . .more message lines. . .
Example server.log fragment:
2019-04-22 11:56:10,799 ERROR (tomcat-http--15) [org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/SASLogon].[cas]] Servlet.service() for servlet [cas] in context with path [/SASLogon] threw exception [Request processing failed; nested exception is java.lang.NullPointerException] with root cause
java.lang.NullPointerException
at com.github.inspektr.audit.AuditTrailManagementAspect.executeAuditCode(AuditTrailManagementAspect.java:147)
at com.github.inspektr.audit.AuditTrailManagementAspect.handleAuditTrail(AuditTrailManagementAspect.java:139)
at sun.reflect.GeneratedMethodAccessor1158.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
. . .more message lines. . .
For workarounds to this problem, see SAS KB0036333, "Changing the member time-out setting for the SAS® 9.4 Cache Locator and Pivotal GemFire members."