Saturday 14 March 2015

Diagnosing java.lang.StackOverflowError - MS

StackOverflow  Exceptions can occur when a thread stack continues to grow in size until reaches to the maximum limit.

Symptom for getting the StackOverflow Errors
  A stack overflow can result from:
  --> A deeply nested application.
  --> An infinite loop within an application.
  --> A problem in just-in time (JIT) compiled code.
  --> Applications requiring a larger stack size , especially one's relying on XML,GUI, or java2D classes.
 --> Native method calls.
Stack overflow issues are frequently masked by Out of Memory exceptions.By resolving the memory constraints , the stack overflow can be resolved.

Cause
           When a stack overflow occurs, the amount of stack space required by the program exceeds what is configured for the stack in the Java Virtual Machine(JVM) process , or the native stack size configured by the operating system.
          Some applications require stacks that are larger than the default size .
              for Example- a Graphical - intensive Java program can require a larger stack , which may requir an increase in the stack size to avoid StackOverflow.

Diagnosing the problem
           Look for either Out of Memory messages or java.lang.StackOverflow in the server logs. the process may continue to run after either of these messages are seen.
           If crash did occur ,a javacore should have generated on IBM SDK.You will either see the signal as SIGSEGV,SIGILL,SIGABRT, or SIGBUS, Usually the current thread will indicate the following state.

pending= java.lang.StackOverflowError

Resolution
1) Infinite Recursion- If an application is performing recursion , the maximum stack size can easily be reached and a stack Overflow exception is thrown. The thread stack has a limited size and eventually its space will run out as the thread stack grows without bounds.
            Some traits of recursion:
               --> Large thead stacks that appears to repeat.
               --> An infinite loop that continuously spawns off threads.
               --> Very large XML documents loaded into the document ObjectModel (DOM).
               --> JSP or servlets calling itself (usually by executing forward or include to itself).
               -->  Repeated calls in active functions.

       Increasing the thread stack size allows for larger thread stacks. However if the recursion continues to trigger a stack overflow , the next step is to identify what code is causing the recursion from javacores, thread dumps, or even system core files.
       A Thread stack that indicates it's too large to display can be an indicator of stack overflow. This is especially true if the stack appears to repeat such as recusive method calls.

JIT/HotSpot Compiled Code
                    The JIT/HotSpot compiler (JIT) is designed to speed up the JVM execution times by compiling method calls. This can speed up execution time, but as more aggressive optimizations are used , this can inadvertently cause recursion , resulting in stack overflow or crash. The documents linked below explain how to debug JIT and HotSpot Compiler Issues:

       Deplated Native Stack 
                       It is almost guaranteed that a crash will occur if the native stack runs out of space. System cores can be used to identify long and possible recursively looping native Thread stacks.
                       To resolve , increase the native stack size by adjusting the operating system limit for stack Size to accommodate the recursive native calls  however, Identify the recursive native calls will help determine the root cause of the stack overflow.
                     
       Adjusting the Stack Sizes (XSS and Xmso) options  
                           If this does not work, you can adjust the stack sizes. Doubling the stack size is suggested as a firsted step however every Thread that the JVM creates will consume memory. Be careful to not exhaust your physical and heap memory resources.
                          For every java Thread , there are two stacks that are utilized. One is for Java code for method calls, and the other is for native C Lang Code but on Solaries and HP-UX only the native stack is utilized. these are adjustable and can be raised to provided more room on the stacks to prevent an overflow.
      Maximum Thread Stack Size(-Xss)
                               This parameter  controls the stack size of Java method calls that are non native to track the state of variables. If you find looping code or large stacks that are all calling Java methods and do not make native calls, try raising the stack size by using the generic JVM argument below:
                 -Xss<size>
             where size has the format ,nn[k|m|g|M|G] , for example: -Xss512K
            On Solaries and HP-UX systems which use the HotSpot JVM , there is no distinction between native and thread stack sizes. this is ths only configurable value for all stack sizes however on HP-UX systems , there is another argument used to control stack size of the UX systems,there is another argument used to control stack size of the main method. 
           -XX:MainThreadStackSize=<size>
          while the -Xss controls the stack size of all threads in native memory, -XX: MainThreadStackSize controls the size of the main thread. The main thread's native size will be set to whichever value is higher.
      Initial Thread Stack Size (Xiss)
              This  is for distributed platforms (AIX,LINUX,Windows)
         Adjust the initial thread stack size that the JVM will start with. The default for all distributed platforms is 2KB. In most cases, you will need to change this option
              -Xiss<size>
  Where <size> has the format , nn[k|m|g|K|M|G] , for example  Xiss2k
      Initial Native Stack Size(-Xmso)
                  This is only for distriuted platforms(AIX,Linux,Windows)
              This parameter controls the initial stack size of native (operating system) threads. Java code uses this to process calls made into native libraries,such as JIT or JNI calls. If there is an abundance of calls made on the native stack , adjust the native stack size using this generic JVM argument
          -Xmso<size>
         where  <size> has the format nn[k|m|g|K|M|G] s -Xmso512k

Note: The maximum stack size for the operating system is controlled by ulimit -s on UNIX and LINUX

Default values 
                             Initial Thread Stack  Size        Max Thread stack Size          Native Stack Size

AIX /Linux                         2KB                                   256KB (in 32bit)                     256KB
                                                                                      512KB (in 64bit) 
Windows                             2KB                                  256KB(in 32bit)                      32KB(in 32bit)
                                                                                      512KB(in 64bit)                      256KB(in 64 bit)


Server logs
           These logs my contain references to StackOverflow ,alongside other messages. Thread stacks may accompany these messages , indicating if htere is a recursive call.

                                                         In SystemOut.log
     the following text will be recorded 
[3/14/15 2:20:10: 783 CST] 314 15926 BaseMap    CWOBJ0006W:  An exceptio occured:
            com.ibm.websphere .objectgrid.ObjectGridRuntimeException:
            java.lang.StackOverflowError


                                                        In SystemErr.log
[3/14/15 3:14:15:865 PST] 314 15926 SystemErr   R  caused by: java.lang.StackOverflowError
 at java.util.Hashtable.get(Hashtable.java:462)
 at                                                     com.ibm.ws.webcontainer.srt.SRTServletRequest.getParameter(SRTServletRequest.java:1257)
 at
psdi.webclient.system.session.WebClientSession.applySkin(WebClientSession.java:295)
 at
psdi.webclient.system.controller.AppInstance.render(AppInstance.java:1177)
at
psdi.webclient.system.controller.AppInstance.render(AppInstance.java:1213)


                                                      In Thread Dump(java core)
 In javacore , look for "StackOverflow Error" in the thread name entry:
 3XMTHREADINFO    " ORB.thread.pol : 19" (TID : 10167F78, sys_thread_t: E072BCEO, state:  R, native ID : E06BEBA0) PRIO=5: PENDING=JAVA.LANG.sTACKoVERFLOWeRROR


                                                     In System Dump
Look for this message near the end of a very large stack. This we has 1700 recursively called entries.

Java stack frames dump is too long:
<analyzerror>

Friday 13 March 2015

java.lang.ClassNotFoundException DSRA8000E

While trying to Use the Test datasource connection button,you receive a java.lang.ClassNotFoundException

Reason 1: This error occur due to Required jar file is not available in the configured location.,make sure you have provided the Jar file location in JDBC provider.

Reason 2:in JDBC provider configured location placing multiple versions of Jar files cause the WebSphere to pick wrong jar file , so avoid using multiple version in same location.

Reason 3 : human Error is People place jar in location but they configure with variables forget to configure variable value.

Reason 4: accessing user might not have permission on jar file.

Wednesday 11 March 2015

application is accessible throw application server and is not accessible throw mapped webserver.and application accessible throw other webserver to which application is not mapped.

we have found issue reported  application ear's deployed and mapped properly with web(ex abc-8050) and app server(abs_server). but before this is deployed some other application also deployed with same context root and mapped to other its appserver(xyz_server)  not mapped to its webserver(xyz-8051). now our application is internally mapping to other application web server(xyz-8051) and our appserver(abc_server). to fix this web have changed the other application context root and restarted the other appserver(xyz_server). and unmapped and remapped our application with our web(abc-8050) and appserver(abc_server) and restarted our web and appserver.

Monitoring team is getting alerts saying SWAP memory reached the threshold level even 16GB free memory is available on the host ,it is taking more memory from swap memory.

we have coordinated with unix team in setting host level parameters like swappiness value in /etc/sysctl.conf file to use 30 percent of swap memory to use and 70 percent from main memory.

various solutions

 The swap file will then only be used when my RAM usage is around 80 or 90 percent. To change the system swappiness value, open/etc/sysctl.conf as root. Then, change or add this line to the file:

vm.swappiness = 10

Reboot for the change to take effect

$ cat /proc/sys/vm/swappiness

60

meaning that the swap file will be used fairly often if the memory usage is around half of my RAM. You can check your own system's swappiness value by running:the above require reboot. to avoid reboot use below command 

You can also change the value while your system is still running using below

sysctl vm.swappiness=10


Tuesday 10 March 2015

Federating Application Server with Dmgr Profile using addNode Command

C:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01\bin>addNode.bat was.local.com 8879 -conntype soap -profileName AppSrv01 -username wasadmin -password wasadmin


The process carried when you run the addNode Command is
ADMU0116I: Tool information is being logged in file
C:\Progra~1\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\addNode.log
ADMU0128I: Starting tool with the AppSrv01 profile
CWPKI0308I: Adding signer alias “CN=WASCell.local.com, ” to local
keystore “ClientDefaultTrustStore” with the following SHA digest:
BF:9B:69:F2:3A:88:53:A2:D7:6B:7F:8A:02:E5:F7:7A:BE:2D:98:22
CWPKI0308I: Adding signer alias “default” to local keystore
“ClientDefaultTrustStore” with the following SHA digest:
F9:E9:87:BC:ED:CA:B3:64:53:71:F9:1B:13:19:AA:C8:55:2C:88:A0
CWPKI0308I: Adding signer alias “datapower” to local keystore
“ClientDefaultTrustStore” with the following SHA digest:
A9:BA:A4:B5:BC:26:2F:5D:2A:80:93:CA:BA:F4:31:05:F2:54:14:17
ADMU0001I: Begin federation of node AppSrv01node with Deployment Manager at
dmgr01:8879.
ADMU0009I: Successfully connected to Deployment Manager Server: Dmgr01:8879
ADMU0507I: No servers found in configuration under:
C:\Progra~1\IBM\WebSphere\AppServer\profiles\AppSrv01\config/cells/WASCell/nodes/AppSrv01node/servers
ADMU2010I: Stopping all server processes for node AppSrv01node
ADMU0024I: Deleting the old backup directory.
ADMU0015I: Backing up the original cell repository.
ADMU0012I: Creating Node Agent configuration for node: AppSrv01node
ADMU0014I: Adding node Appsrv01node configuration to cell: WASCell
ADMU0016I: Synchronizing configuration between node and cell.
ADMU0018I: Launching Node Agent process for node: AppSrv01node
ADMU0020I: Reading configuration for Node Agent process: nodeagent
ADMU0022I: Node Agent launched. Waiting for initialization status.
ADMU0030I: Node Agent initialization completed successfully. Process id is:
6308
ADMU0505I: Servers found in configuration:
ADMU0506I: Server name: nodeagent
ADMU0300I: The node AppSrv01node was successfully added to the WASCell
cell.
ADMU0306I: Note:
ADMU0302I: Any cell-level documents from the standalone WASCell
configuration have not been migrated to the new cell.
ADMU0307I: You might want to:
ADMU0303I: Update the configuration on the WASCell Deployment
Manager with values from the old cell-level documents.
ADMU0306I: Note:
ADMU0304I: Because -includeapps was not specified, applications installed on
the standalone node were not installed on the new cell.
ADMU0307I: You might want to:
ADMU0305I: Install applications onto the WASCell cell using
wsadmin $AdminApp or the Administrative Console.
ADMU0003I: Node AppSrv01node has been successfully federated.


For UNIX Flavours:

/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/bin>./addNode.sh was.local.com 8879 -conntype soap -profileName AppSrv01 -userName wasadmin -password wasadmin


NOTE: 1.HostName of DMGR is was.local.com
           2.SOAP port of DMGR is 8879
           3. we can use below options with addNode command 
                       [-includebuses] [-startingport <portnumber>] [-portprops
                       <qualified-filename>] [-nodeagentshortname <name>] [-nodegroupname
                      <name>] [-registerservice] [-serviceusername <name>]
                       [-servicepassword <password>] [-coregroupname <name>] [-noagent]
                       [-statusport <port>] [-quiet] [-nowait] [-logfile <filename>]
                       [-replacelog] [-trace] [-username <username>] [-password <pwd>]
                       [-localusername <localusername>] [-localpassword <localpassword>]
                       [-profileName <profile>] [-excludesecuritydomains] [-asExistingNode]
                       [-help]

Creating Deploment manager Profile command line


Creating Deployment manager Profile


Deployment Manager :

A Deployment Manager is "a server that manages operations for a logical group or cell of other
servers," the deployment manager is:... the central administration point of a cell that consists of multiple nodes and node groups in a distributed server configuration. ... The deployment manager uses the node agent to manage the application servers within one node. A deployment manager provides Overview of IBM WebSphere Application Terminology for IBM WebSphere Application Server Administrators 3management capability for multiple federated nodes and can manage nodes that span multiple systems and platforms. A node can only be managed by a single deployment manager and must be federated to the cell of that deployment manager. The configuration and application files for all nodes in the cell are centralized into a master configuration repository.This centralized repository is managed by the deployment manager and
synchronized with local copies that are held on each of the nodes.


For Windows:


AppServer_root/bin>manageprofiles 


-create -profileName Dmgr01 


-profilePath "C:\Program Files\IBM\WebSphere\AppServer\profiles\Dmgr01" 


-templatePath "C:\Program Files\IBM\WebSphere\AppServer\profileTemplates\dmgr" 


-nodeName Dmgr01node 


-cellName WASCell 


-hostName was.local.com


-enableAdminSecurity true

-adminUserName wasadmin

-adminPassword wasadmin




verfication:

check Dmgr01_create.log is created under C:\Program Files\IBM\WebSphere\AppServer\logs\manageprofiles\


find for <message>Returning with return code: INSTCONFSUCCESS</message>in Dmgr01_create.log file




Now Start the server 


C:\Program Files\IBM\WebSphere\AppServer\profiles\Dmgr01\bin>startManager.bat -username wasadmin -password wasadmin


after PID is generated in generated




opening console


from Browser and give URL


http://was.local.com:9062/ibm/console




For Unix Flavours:


/usr/IBM/WebSphere/AppServer/bin>./manageprofiles.sh \


-create -profileName Dmgr01 \

-profilePath /usr/IBM/WebSphere/AppServer/profiles/Dmgr01 \

-templatePath /usr/WebSphere/AppServer/profileTemplates/dmgr \

-nodeName Dmgr01node  \

-cellName WASCell \

-hostName was.local.com

-enableAdminSecurity true

-adminUserName wasadmin

-adminPassword wasadmin



verfication:

check Dmgr01_create.log is created under /usr/IBM/WebSphere/AppServer/logs/manageprofiles/


find for <message>Returning with return code: INSTCONFSUCCESS</message> in Dmgr01_create.log file




Now Start the server 


/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/bin>./startManager.sh 
-username wasadmin -password wasadmin

after PID is generated in generated




opening console


from Browser and give URL


http://was.local.com:9062/ibm/console



Note:  1.Change your HostName to was.local.com


          2.For Windows: Find Administrative console port (Default port no 9062) from  C:\Program Files\IBM\WebSphere\AppServer\profiles\Dmgr01\logs\AboutThisProfile.txt 

            For Unix Flavours :         /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/AboutThisProfile.txt 

WebSphere Application Server Profile Creation command line

An application server is a Server program in a computer in a Distributed Network that provides the business logic for an application program. The application server is frequently viewed as part of a three-tier application, consisting of a graphical user interface  server, an application (business logic) server, and a Data Base and Transaction server.
The application server is the primary runtime component in all configurations and is where
an application actually executes. All WebSphere Application Server configurations can have
one or more application servers. ... With Network Deployment, you can build a distributed
server environment consisting of multiple application servers maintained from a central
administration point. In a distributed server environment, you can cluster application
servers for workload distribution.

Creating Application Server Profile

For Windows:

AppServer_root/bin>manageprofiles 

-create -profileName AppSrv01 

-profilePath "C:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01" 

-templatePath "C:\Program Files\IBM\WebSphere\AppServer\profileTemplates\default" 

-nodeName AppSrv01node 

-cellName WASCell 

-hostName was.local.com

-enableAdminSecurity true

-adminUserName wasadmin

-adminPassword wasadmin


verfication:

check AppSrv01_create.log is created under C:\Program Files\IBM\WebSphere\AppServer\logs\manageprofiles\

find for <message>Returning with return code: INSTCONFSUCCESS</message> in AppSrv01_create.log file



Now Start the server 

C:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01\bin>startServer server1 -username wasadmin -password wasadmin

after PID is generated in generated



opening console

from Browser and give URL

http://was.local.com:9060/ibm/console



For Unix Flavours:

/usr/IBM/WebSphere/AppServer/bin>./manageprofiles.sh \

-create -profileName AppSrv01 \

-profilePath /usr/IBM/WebSphere/AppServer/profiles/AppSrv01 \

-templatePath /usr/WebSphere/AppServer/profileTemplates/default \

-nodeName AppSrv01node  \

-cellName WASCell \

-hostName was.local.com

-enableAdminSecurity true

-adminUserName  wasadmin

-adminPassword wasadmin


verfication:

check AppSrv01_create.log is created under /usr/IBM/WebSphere/AppServer/logs/manageprofiles/

find for <message>Returning with return code: INSTCONFSUCCESS</message> in AppSrv01_create.log file



Now Start the server 

/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/bin>./startServer server1 -username wasadmin -password wasadmin

after PID is generated in generated



opening console

from Browser and give URL

http://was.local.com:9060/ibm/console


Note:  1.Change your HostName to was.local.com

          2.For Windows: Find Administrative console port (Default port no 9060) from  C:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\AboutThisProfile.txt 
            For Unix Flavours :         /usr/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/AboutThisProfile.txt 




Monday 9 March 2015

Enable Trace during configuration Time and Runtime from Admin Console

Troubleshooting --> Logs and Trace--> select  Server Name you want enable trace


Click on Diagnostic trace

Click  on Change Log Detail Level

Click on all components --> Message and trace level --> select “fine” or  “finer” or  “finest” according to your requirement and click on apply


Save the changes to master Repository and stop the server and start server.




Verification
Now we can fing trace  in logs





Enabling Runtime Trace

Enabling RunTime trace is similar like Configuration time ,but instead of configuration tab we will enable Same Trace String in RunTime . but here difference between Configuration Time and RunTime is RunTime Trace does not require server Restart, as soon as you pass trace string and apply trace logs will be generated, but for trace logs will not be generated until you restart trace enabled server. RunTime Trace setting will not survive in server restart, where as  configuration time traces will not be lost on server restart.

Saturday 7 March 2015

Reason's for JVM is not starting

reasons for JVM's not starting can be multiple

1.Not enough free memory on machine.

2.Space available on the mount is not enough.

3.Not having permisssion on startServer.log file.check the process execution user is the user of startServer.log file and having write permission.

4.Process might be running behind and showing stop status in front end,then kill the process and start Server.

5.Orphaned process id file left in pid file but pid is stoped.

6.JVM log file size might crossed the 2GB size.

7.Sometimes due to JMS draining also Server will not. It will start on purging the messages from MQ server queue of that WAS JVM.

8.Passing the cross functionality JVM arguments also JVM will not .

9.In Rare cases port conflict also make  JVM not to start.

10.On server certificates expiry can also causes the server not to start.

11.Server may not start if coregroup  coordinator is failing.