Friday 25 December 2015

Application is Deployed successfully but new code is not reflecting

There are couple of reasons for this code is not updating

sometimes code deployed is not updated in applications in JVM level ,we can see that time stamp will not be updated in installed apps level,in such a cases we need to restart the nodeagent then newly installed code will be updated.

If still you are not getting the new code reflected then you need to generate the WebServer plugin and propagate the WebServer Plugin.if there is possibility restart WebServers.

now the new code will be reflected.

Note:Ensure you have deployed new ear ,and updated latest jar files in shared libraries.

Friday 18 December 2015

server is not coming up and showing HAManager related error with server(ex:server A) is not in syn with the servers(server b ,server c ,server d) in logs

server is not coming up and showing hamanger related error with server(ex:server A) is not in syn with the server(server b,server c ,server d) in SystemOut.log

This will occur when HA coordinator is not in sync with all the servers in cell.

resolution is restart all the server that are not in sync with hamanager and restart the server b,server c ,server d then start server A.

Note: be carefull if you are dealing with Production and DR servers

Tuesday 15 December 2015

If Node agent is not starting

some times due to unix level patches ip address of host changes to other IP Address which not available during profile installation.so if profile is created with IP Address based constraints we will get IP related  exceptions in SystemOut.log.

If you find IP Address based exception then request unix Team to get back the old IP address host name should be resolved.after getting the IP Address we can restart the Nodeagent.

for getting IP Address of  Unix box use below command

$ifconfig

Saturday 12 December 2015

During Deployment time if we are getting application already existing and not allowing us to complete the deployment successfully

We will get this error stop means application binaries still exist in profiles,this existing binaries are not allowing to deploy an application

To make Deployment sucessfull

1. Stop Dmgr

clear the application related binaries from cus  and blas and applications directories in below location of dmgr and all profiles.

"/opt/IBM/WebSphere/Appserver/profiles/<profilename>/config/cells/<cellname>/

2. Start Dmgr

now Deploy the Application

Application is not Starting After Deplyment in one of the servers in cluster

It happens with some times with WebSphere7 and WebSphere6.1 it is because of Bug in WebSphere Product.we have noticed this with WebSphere7 with fix pack 11(i.e WebSphere7.0.0.11)

To Start Application we need to copy cus and blas folder from Application working host to applcation not starting host.

location of cus , blas folder is under the location

"/opt/IBM/WebSphere/Appserver/profiles/<profilename>/config/cells/<cellname>/

Ex:-
"/opt/IBM/WebSphere/Appserver/profiles/Appserver01/config/cells/localhostCello1"

Saturday 26 September 2015

Error while Testing DataSource Connection

If you are getting Node Related Error and giving Invalid username and password while testing the Data Source connection.

This error is not due to invalid username and password .It occur if do not restart nodeagent and dmgr after creating DataSource.
so Recommendations before Testing newly created Data Source are.

1.restart nodeagent.
2.restart dmgr
3.Now test the DataSource connection.

Even after if you are getting Test Connection Error then problem may be with other things dig into to systemOut,systemErr logs for problem that is causing the issue.

Sunday 6 September 2015

shell script is not executing due to ^M at the end of each line in script file

This is due to unix conversion.we can see this type of ^M symbols when script file is copied from one host to other.

Run the below command

#dos2unix afile.sh afile.sh

after running the above command you will get file without ^M at the end.
Note: try to run this command using root user

Note : if you are getting ^M during execution time of script check provided environment details provided in script file are correct or not.

Tuesday 14 April 2015

Error while Deleting the Profile

$cd   /opt/IBM/WebSphere/AppServer/bin

$./manageprofiles -delete -profile <profile Name>

if it is Still giving error then give profile absolute path of Profile with profilePath

$./manageprofiles -delete -profile AppSrv01 -profilePath  /opt/IBM/WebSphere/AppServer/Profiles/AppSrv01

Note: Here AppSrv01 is profileName , ensure profile is deleted from profiles list ,if it is not get removed then manually delete after running the above command or run option                                            -validateAndUpdateRegistry with mange profile command to remove profile directory that is deleted throw -delete option.
command: $./manageprofiles -validateAndUpdateRegistry

Now check the deleted profile will not be there using -listProfiles option with manageprofiles command.

Comand: $./manageprofiles -listProfiles


Saturday 4 April 2015

Disabling SSL v2 and Allow only SSL v3 and TLS v1 -MS

SSL

              Having SSL is additional layer of security you are adding into Wb Application. However, Default SSL configuration leads to certain valunerabilities and you should consider tweaking those configurations. we require some tool to verify SSL settings. There are many available however,I would use SSL-Scan free tool. You can download from

  http://sourceforge.net/projects/sslscan/ 

Download & Installation

                         Following prerequisites must be installed on server where you wish to use Mod Security with Apache. If any one of these doesn't exist then Mod Security compilation will fail. You may use yum install on Linux flavors to install these packages.
                      --> apache 2.X or higher version
                      --> libapr and libapr-util package
                      --> libcurl package
                      --> libpcre package
                      --> libxml2 package
                      --> liblua package
                      --> mod_unique_id module bundled with Apache web server

                 Now ,let's download the latest stable version of Mod Security 2.9.0 from 

                         http://www.modsecurity.org/download/ 

                    Transfer downloaded file to /opt/apache
          
                 [/opt/apache]# ls -lrt  modsecurity-apache_2.9.0.tar.gz

                 Now Extract modsecurity-apache _2.9.0.tar.gz
                                 
                                      # gunzip -c modsecurity-apache_2.9.0.tar.gz
                                      # tar  -xvf  modsecurity-apche_2.9.0.tar

                 Now Move to Extracted folder modsecurity-apache_2.9.0
                                       
                                      # cd modsecurity-apache_2.9.0

                 Run the configuration script including apxs path to existing Apache
                                          
                                      #./configure -with-apxs=/opt/apache/bin/apxs
                      
                 Now Compile and install with make Script
   
                                     # make
                                     # make install

                 Once installation is done ,we can find mod_security2.so  in modules folder under                                             /opt/apache

                Now We have installed Mod Security module in existing Apache web server

Configuration

                 In order to use Mod security feature with Apche, we have to load mod security module in httpd.conf mod_unique_id module is pre requisite for Mod Security. This module provides an environment variable with a unique identifier for each request, which is tracked and used by Mod Security.
  
                 -->Add following line to load module for Mod Security in httpd.conf and save the configuration                            file
               
                 LoadModule unique_id_module modules/mod_unique_id.so
                 LoadModule security2_module modules/mod_security2.so

                --> Restart  Apache web server

                Mod Security is now installed ,Now we need to install Mod Security core rule to take a full advantage of its features. Latest Core Rule can be downloaded from following link,which is free

                https://github.com/SpiderLabs/owasp-modsecurity-crs/zipball/master


                --> copy downloaded core rule zip to /opt/apche/conf    folder
                --> Unzip core rule file, you should see the extracted folder as SpiderLabs-owasp-                                             modsecurity-crs-0f07cbb
                --> lets make folder name simple .rename the SpiderLabs-owasp-modsecurity-crs- 0f07cbb to crs

                [/opt/apache/conf]# mv SpiderLabs-owasp-modsecurity-crs-0f07cbb/  crs  
                        
                --> now Go to crs  folderand rename modsecurity_crs_10_setup.conf.example to                                           modsecurity_crs_10_setup.conf
                        
                                  [/opt/apache/conf/crs]# mv modsecurity_crs_10_setup.conf.exaple                                                                                                 modsecurity_crs_10_setup.conf

                --> Now ,Let's enable these rules to get it working with Apache web server.
                                          Add the following lines in httpd.conf
                                       
                                                  <IfModule security2_module>
                                                        Include conf/crs/modsecurity_crs_10_setup.conf
                                                        Include conf/crs/base_rules/*.conf
                                                    </IfModule>
                In the above Configuration, we are loading Mod Security main configurationfile                                      modsecurity_crs_10_setup.conf and base rules base_rules/*.conf provided by
        Mod Security Core Rules to protect web applications

                --> Restart apache webserver
      Now we successfully configured Mod Security with Apache. Our Apache Web server is protected with Mod Security web application firewall.


Get Started

                Lets get it started with some of the important configuration in Mod Security to harden &secure web applications. In this section , we will do all configuration modification in /opt/apache/conf/crs/modsecurity_crs_10_setup.conf

               we will refer /opt/apache/conf/crs/modsecurity_crs_10_setup.conf as setup.conf in this section for example purpose. It's important to understand what are the OWASP rules are provided in free.

Types of rules provided by OWASP are.

Base Rules-these rules are heavily tested and probably false alarm ratio is less.

Experimental Rules-these rules are for experimental purpose and you may have high false alarm. It's important to configure , test and implement in UAT before using these in production environment.

Optional Rules- these optional rules may not be suitable for the entire environment. Based on your requirement you may use them. If you are looking for CSRF, User Trackin, Session hijacking, etc. protection then you may consider using optional rules. we have base, optional  and experimental rules after extracting downloaded crs zip file from OWASP download page. These rules configuration file is available in crs/base, crs/optional_rules ans crs/experimental_rules folder.

Some of the Base rules are 

modsecurity_crs_20_protocal_violations.conf:This rule is protecting from protocal vulnerabilities like response splitting , request smuggling, using non-allowed protocal(HTTP 1.0).
modsecurity_crs_21_protocal_anamalies.conf:This is to protect from request, which is missing with Host, Accept, User-Agent in header.
modsecurity_crs_23_protocal_request_limit.conf:This rule has dependency on application specific like request size , upload size, length of parameter, etc.
modsecurity_crs_30_http_policy.conf:This is to configure and protect allowed or disallowed method like CONNECT , TRACE,PUT, DLETE, etc.
modsecurity_crs_35_bad_robots.conf: Detect malicious robots.
modsecurity_crs_40_generic_attacks.conf:This is to protect from OS command injection, remote file inclusion, etc.
modsecurity_crs_41_sql_injection_attacks.conf: This rule is to protect SQL and blindSQL inject request.
modsecurity_crs_41_xss_attacks.conf: Protect from cross site scripting request
modsecurity_crs_42_tight_security.conf: Directory traversal detection and request.
modsecurity_crs_45_trojans.conf:Tthis rule is to detect generic file management output , uploading of http backdoor page, known signature.
modsecurity_crs_47_common_exceptions.conf:This is used as an exception mechanism to remove common false positives that may be encountered suck as Apache internal dummy connection, SSL pinger, etc.

Logging
             Logging is one of the first things to configure so you can have logs created for what Mod Security is doing. there are two types of logging available , Debug &Audit log.

          Debug Log : This is to Duplicate the Apache error , warning and notice message from the error log.
   
          Audit Log  : This is to write the transaction logs that are marked by Mod Security rule, Mod Security gives you flexibility to configure Audit , Debug or both logging. By Default configuration will write both logs. However , you can change based on your requirement. Log is controlled in SecDefultAction directive. Let's look at default logging configuration in setup.conf

          secDefaultAction "phase:1,deny,log"

 To log Debug ,Audit log - use "log" .    
 To log  only audit log - use  "nolog,auditlog"
 To log only debug log - use "log,noauditlog"
You can specify the Audit Log location to be stored which is controlled by SecAuditLog directive. Let's write audit log into /opt/apache/logs/modsec_audit.log by adding as shown below.

         Implementation
          Add SecAuditLog directive in setup.conf and  restart apache web server
       
           SecAuditLog /opt/apache/logs/modsec_audit.log

After restart , you should see modsec_audit.log getting generated in /opt/apache/logs

Enable Rule Engine

By default Engine Rule is Off that means if you don't enable RuleEngine you are not utilizing all the advantage of Mod Security. Rule Engine enabling or disabling is controlled by SecRuleEngine directive
          Implementation
                        Add SecRuleEngin directive in setup.conf and restart apache  web Server

                             SecRuleEngine On

There are three values for SecRuleEngine:
          On- to enable Rule Engine
          Off- to disable Rule Engine
          DetectOnly - enable Rule  Engine but never executes any actions like                                                                                   block,deny,drop,allow,proxy or redirect.

Once Rule Engine is on -Mod Security is ready to protect with some of the common attack types.

Disable SSL v2
SSL v2 has many security flaws and if you are working towards penetration test or PCI compliance then you are expected to close security finding SSL v2. Any SSL v2 communication may be vulnerable to Man-in Middile attack that could allow data tempering or disclosure. let's implement apache web server to accept only latest SSL v3 and reject SSL v2 connection request.

Implementation:

          Go to $Web_Server/conf/extra  folder
          Modify SSL Protocol directive in httpd-ssl.conf as below to accept only SSL v3 and TLS v1

          SSLProtocal -ALL +SSLv3 +TLSv1

verification
         Let's use sslscan utility to validate as below command .change localhost to your actual domain name

           [/opt/apache/conf/extra] # sslscan --no-failed localhost

In output you could see only SSLv3 and TLSv1 ,Which is Safe from SSLv2 vulnerabilities

Disabling or enabling a high availability manager -MS

A unique HAManagerService configuration object exists for every core group member. The enable attribute in this configuration object determines if the high availability manager is enabled or disabled for the corresponding process. When the enable attribute is set to true,the high availability manager is enabled.when the enable attribute is set to false, the high availability manager is disabled. By default ,the high availability manager is enabled. if the setting for the enable attribute is changed, the corresponding process must be restarted before the change goes into effect.You must use the wsadmin tool to disable or enable a high availability manager.
First You must ensure you need to use a high availability manager to manage members of a core group.
to Avoid the trouble : when you disable the high availability manager,make sure that you disable it for all the servers in core group so that all members are disabled. Do not disable the high availability manager for select members of the core group.

Procedure to disable or enable  high availability manager

1.In the administrative console, navigate to the Core Group Service Page for the Process
      
  From Deployment manger ,
              for  Deployment Manager click System Administration -> Deployment manager -> Core group service
            for Node Agent click System Administration -> Node agent-> node_agent-> Core group service
            for an application server click Servers->Server Types-> WebSphere application servers-> server_name -> Core group service

2.If you want to disable the high availability manager for this process, deselect the Enable service at server start up option.

3.If you want to enable the high availability manager for this process, select the Enable service at server start up option.

4. Click on OK and  then click Review.

5.select Synchronize changes with nodes , and then click Save.

6. Restart all of the processes for which you changed the Enable service at server start up property setting.

Now processes start with the high availability manager in the changed state.

Verify                                              
check the Systemout  log file for following messages:

           HMGR005l: The Single Server DCS Core Stack Transport has been started for core group DefaultCoreGroup

The Above message indicates that the high availability manager is disabled because the hihg availability manager communication transport can only establish communication with a single server process.

   HMGR001l: The DCS Core Stack Transport has been started for core group DefaultCoreGroup There are x members.

The above message indicates that the high availability manager is enabled because the high  availability manager communication transport can establish communication with multiple server processes. X indicates the numbers of server processes with which communication is established.

JDBC Connection Problems -MS

You have created the JDBC Provider and created alias and Datasource .You are Testing the Connection

         If you are getting NameNotFoundException  in the SystemOut.log that means there is JNDI lookup problem check Namespace lookup that are mentioned in the Code and config file's as mentioned in Previous post

         If you are getting the SQLException Problem is either userid and password passes in  J2C Authentication alias ,make sure the valid userid and password are passed.

         If you are getting the java.lang.UnsatisfiedLinkError ,
              1.If you are using Type2 or Type3 driver and getting  java.lang.UnsatisfiedLinkError    
                 This means JVM is unable to Load a native libraries that are needed by JDBC driver to
                 Resolve this check DataBase Home and library paths are correct
                 (ex:Oracl_Home,LIBPATH)
              2.connecting user has proper permissions to access Native libraries
              3.32-bit  native libraries cannot be used with 64-bit application server JVM and vice
                 verse.Mismatches between the two can also cause java.lang.UnsatisfiedLinkError

         If  you are getting java.sql.SQLException: [IBM][SSLServer JDBC Driver][SQLServer]Could not find stored procedure 'master..xp_jdbc_open'. The problem is with stored procedure for JTA are not installed on the SQL ServerDatabase.these Stored procedures are required for XA connections.


         If you are typical error messages like
DSRA8040I: java.sql.SQLException: IOException:The Network Adapter could not establish the connection 
DSRA0010E: SQL State=null, Error code=17,002
Then try to connect to the database server using a stand alone java program to determine if you are able to connect successfully with the same properties that you have set in your Data Source outside the WebSphere Application server then Go with other possible Trouble shooting Technics .if it is giving same error out side WebSphere Application Server it is not related to WebSphere Application Server Coordinate with DBA Team to Resolve the Issue.

JNDI and Naming Problems -MS

Troubleshooting naming and JNDI problems using the WebSphere Application and  Deployment Manager.

Steps to help resolve common naming lookup problems
Which methods are you using , corbaloc / corbaname, for direct lookups or Java for indirect lookups.
e.g: WebSphere _AppServer_Root/bin>dumpNameSpace.[sh/bat] -host myhost.mycompany.com -port 901
  OR 
WebSphere _AppServer_Root/bin>dumpNameSpace.[sh/bat] url corbaloc:iiop:myhost.mycompany.com:901

Example

example using corbaloc provider URL and lookup:

env.put(Context.PROVIDER_URL," corbaloc::server.aust.in.ibm.com:9809");
ctx=new InitialContext(env);
Object obj = ctx.lookup("jdbc/PlantsByWebSphereDataSource")

The following is how the object is seen in the dumpNameSpace output:

Install_Root\bin> dumpNameSpace -url corbaloc::server.austin.ibm.com:9809

the output is get is as follows
.......
.......
.......
39  (top) /nodes/server/servers/server1jdbc/petstore/PetStoreDB  PetStoreDB
40  (top) /nodes/server/servers/server1jdbc/petstore/PlantsByWebSpehereDataSource  PLANTSDB
41  (top) /nodes/server/servers/server1jdbc/petstore/CatalogDB  CatalogDB
.......
.......
.......

Now You can able to see the object int he namespace
if  YES:
         -->check for spelling mistakes on lookup code versus what is seen in namespace.
        --> Confirm you are connecting to the namespace of the proper server,see the provider URL
        -->  If Provider URL or ctx.lookup values are pickedup from the properties file,print out the variable values to SystemOut to confirm the correct values are provided.
       --> If This does not resolve the problem , then go to what to do next.
if NO:
         Use the administrative console to verify that:
      --> The server hosting the target resource is started.
      --> The Web module or EJB container, if applicable, hosting the target resource is running.
      -->The JNDI name of the target resource is correct and updated.
      --> If the problem resource is remote , that is , not on the same node as the Name Server node, that the JNDI name is fully qualified, including the host name. This is especially applicable to Network Deployment Configurations.
If Still Problem not Resolved

  If you are using the java:lookup method, you cannot use dumpNameSpace to see the lookup Object. This is because the object is in the application's own namespace.
                   
1.Check your configuration against the following example of a java: namespace lookup. this shows the mapping from the sample code with the resource.xml, the resource .xml with the ibm-resource-bnd.xmi, and finally with the object as seen in the namespace.(where resource =web,application,appclient or ejbjar)

Code sample:
       ctx = new InitialContext();
       Object obj = ctx.lookup("java:comp/env/plantsby/PlantsDS");

NOTE: Mapping is configuring between the resource.xml and ibm-resource-bnd.xmi files.

web.xml:
                <resource-ref id="ResouceRef_1">
                             <res-ref-name>plantsby/PlantsDS</res-ref-name>
                             <res-type>javax.sql.DataSorce</res-type>
                             <res-auth> Container</res-auth>
                 </resource-ref>
ibm-web-bnd.xmi:
                     <resRefBindings xmi:id="ResourceRefBinding_1" jndiName = "jdbc/PlantsByWebSphereData                                   Source">
                      <bindingResourceRef href="WEB-INF/web.xml#ResourceRef_1"/>
                     </resRefBindings>

2. The dumpNameSpace utility can Still be used to show the real object in the namespace as defined in the binding file. This is different from that of the java:'s namespace.
output of dumpNameSpace Output:

.......
.......
.......
39  (top) /nodes/server/servers/server1jdbc/petstore/PetStoreDB  PetStoreDB
40  (top) /nodes/server/servers/server1jdbc/petstore/PlantsByWebSpehereDataSource  PLANTSDB
41  (top) /nodes/server/servers/server1jdbc/petstore/CatalogDB  CatalogDB
.......
.......
.......
If the configuration looks to be setup properly, when compared with the preceding sample, then get a dump of the applications,java: namespace using the wsadmin utility.

Saturday 14 March 2015

Diagnosing java.lang.StackOverflowError - MS

StackOverflow  Exceptions can occur when a thread stack continues to grow in size until reaches to the maximum limit.

Symptom for getting the StackOverflow Errors
  A stack overflow can result from:
  --> A deeply nested application.
  --> An infinite loop within an application.
  --> A problem in just-in time (JIT) compiled code.
  --> Applications requiring a larger stack size , especially one's relying on XML,GUI, or java2D classes.
 --> Native method calls.
Stack overflow issues are frequently masked by Out of Memory exceptions.By resolving the memory constraints , the stack overflow can be resolved.

Cause
           When a stack overflow occurs, the amount of stack space required by the program exceeds what is configured for the stack in the Java Virtual Machine(JVM) process , or the native stack size configured by the operating system.
          Some applications require stacks that are larger than the default size .
              for Example- a Graphical - intensive Java program can require a larger stack , which may requir an increase in the stack size to avoid StackOverflow.

Diagnosing the problem
           Look for either Out of Memory messages or java.lang.StackOverflow in the server logs. the process may continue to run after either of these messages are seen.
           If crash did occur ,a javacore should have generated on IBM SDK.You will either see the signal as SIGSEGV,SIGILL,SIGABRT, or SIGBUS, Usually the current thread will indicate the following state.

pending= java.lang.StackOverflowError

Resolution
1) Infinite Recursion- If an application is performing recursion , the maximum stack size can easily be reached and a stack Overflow exception is thrown. The thread stack has a limited size and eventually its space will run out as the thread stack grows without bounds.
            Some traits of recursion:
               --> Large thead stacks that appears to repeat.
               --> An infinite loop that continuously spawns off threads.
               --> Very large XML documents loaded into the document ObjectModel (DOM).
               --> JSP or servlets calling itself (usually by executing forward or include to itself).
               -->  Repeated calls in active functions.

       Increasing the thread stack size allows for larger thread stacks. However if the recursion continues to trigger a stack overflow , the next step is to identify what code is causing the recursion from javacores, thread dumps, or even system core files.
       A Thread stack that indicates it's too large to display can be an indicator of stack overflow. This is especially true if the stack appears to repeat such as recusive method calls.

JIT/HotSpot Compiled Code
                    The JIT/HotSpot compiler (JIT) is designed to speed up the JVM execution times by compiling method calls. This can speed up execution time, but as more aggressive optimizations are used , this can inadvertently cause recursion , resulting in stack overflow or crash. The documents linked below explain how to debug JIT and HotSpot Compiler Issues:

       Deplated Native Stack 
                       It is almost guaranteed that a crash will occur if the native stack runs out of space. System cores can be used to identify long and possible recursively looping native Thread stacks.
                       To resolve , increase the native stack size by adjusting the operating system limit for stack Size to accommodate the recursive native calls  however, Identify the recursive native calls will help determine the root cause of the stack overflow.
                     
       Adjusting the Stack Sizes (XSS and Xmso) options  
                           If this does not work, you can adjust the stack sizes. Doubling the stack size is suggested as a firsted step however every Thread that the JVM creates will consume memory. Be careful to not exhaust your physical and heap memory resources.
                          For every java Thread , there are two stacks that are utilized. One is for Java code for method calls, and the other is for native C Lang Code but on Solaries and HP-UX only the native stack is utilized. these are adjustable and can be raised to provided more room on the stacks to prevent an overflow.
      Maximum Thread Stack Size(-Xss)
                               This parameter  controls the stack size of Java method calls that are non native to track the state of variables. If you find looping code or large stacks that are all calling Java methods and do not make native calls, try raising the stack size by using the generic JVM argument below:
                 -Xss<size>
             where size has the format ,nn[k|m|g|M|G] , for example: -Xss512K
            On Solaries and HP-UX systems which use the HotSpot JVM , there is no distinction between native and thread stack sizes. this is ths only configurable value for all stack sizes however on HP-UX systems , there is another argument used to control stack size of the UX systems,there is another argument used to control stack size of the main method. 
           -XX:MainThreadStackSize=<size>
          while the -Xss controls the stack size of all threads in native memory, -XX: MainThreadStackSize controls the size of the main thread. The main thread's native size will be set to whichever value is higher.
      Initial Thread Stack Size (Xiss)
              This  is for distributed platforms (AIX,LINUX,Windows)
         Adjust the initial thread stack size that the JVM will start with. The default for all distributed platforms is 2KB. In most cases, you will need to change this option
              -Xiss<size>
  Where <size> has the format , nn[k|m|g|K|M|G] , for example  Xiss2k
      Initial Native Stack Size(-Xmso)
                  This is only for distriuted platforms(AIX,Linux,Windows)
              This parameter controls the initial stack size of native (operating system) threads. Java code uses this to process calls made into native libraries,such as JIT or JNI calls. If there is an abundance of calls made on the native stack , adjust the native stack size using this generic JVM argument
          -Xmso<size>
         where  <size> has the format nn[k|m|g|K|M|G] s -Xmso512k

Note: The maximum stack size for the operating system is controlled by ulimit -s on UNIX and LINUX

Default values 
                             Initial Thread Stack  Size        Max Thread stack Size          Native Stack Size

AIX /Linux                         2KB                                   256KB (in 32bit)                     256KB
                                                                                      512KB (in 64bit) 
Windows                             2KB                                  256KB(in 32bit)                      32KB(in 32bit)
                                                                                      512KB(in 64bit)                      256KB(in 64 bit)


Server logs
           These logs my contain references to StackOverflow ,alongside other messages. Thread stacks may accompany these messages , indicating if htere is a recursive call.

                                                         In SystemOut.log
     the following text will be recorded 
[3/14/15 2:20:10: 783 CST] 314 15926 BaseMap    CWOBJ0006W:  An exceptio occured:
            com.ibm.websphere .objectgrid.ObjectGridRuntimeException:
            java.lang.StackOverflowError


                                                        In SystemErr.log
[3/14/15 3:14:15:865 PST] 314 15926 SystemErr   R  caused by: java.lang.StackOverflowError
 at java.util.Hashtable.get(Hashtable.java:462)
 at                                                     com.ibm.ws.webcontainer.srt.SRTServletRequest.getParameter(SRTServletRequest.java:1257)
 at
psdi.webclient.system.session.WebClientSession.applySkin(WebClientSession.java:295)
 at
psdi.webclient.system.controller.AppInstance.render(AppInstance.java:1177)
at
psdi.webclient.system.controller.AppInstance.render(AppInstance.java:1213)


                                                      In Thread Dump(java core)
 In javacore , look for "StackOverflow Error" in the thread name entry:
 3XMTHREADINFO    " ORB.thread.pol : 19" (TID : 10167F78, sys_thread_t: E072BCEO, state:  R, native ID : E06BEBA0) PRIO=5: PENDING=JAVA.LANG.sTACKoVERFLOWeRROR


                                                     In System Dump
Look for this message near the end of a very large stack. This we has 1700 recursively called entries.

Java stack frames dump is too long:
<analyzerror>

Friday 13 March 2015

java.lang.ClassNotFoundException DSRA8000E

While trying to Use the Test datasource connection button,you receive a java.lang.ClassNotFoundException

Reason 1: This error occur due to Required jar file is not available in the configured location.,make sure you have provided the Jar file location in JDBC provider.

Reason 2:in JDBC provider configured location placing multiple versions of Jar files cause the WebSphere to pick wrong jar file , so avoid using multiple version in same location.

Reason 3 : human Error is People place jar in location but they configure with variables forget to configure variable value.

Reason 4: accessing user might not have permission on jar file.

Wednesday 11 March 2015

application is accessible throw application server and is not accessible throw mapped webserver.and application accessible throw other webserver to which application is not mapped.

we have found issue reported  application ear's deployed and mapped properly with web(ex abc-8050) and app server(abs_server). but before this is deployed some other application also deployed with same context root and mapped to other its appserver(xyz_server)  not mapped to its webserver(xyz-8051). now our application is internally mapping to other application web server(xyz-8051) and our appserver(abc_server). to fix this web have changed the other application context root and restarted the other appserver(xyz_server). and unmapped and remapped our application with our web(abc-8050) and appserver(abc_server) and restarted our web and appserver.

Monitoring team is getting alerts saying SWAP memory reached the threshold level even 16GB free memory is available on the host ,it is taking more memory from swap memory.

we have coordinated with unix team in setting host level parameters like swappiness value in /etc/sysctl.conf file to use 30 percent of swap memory to use and 70 percent from main memory.

various solutions

 The swap file will then only be used when my RAM usage is around 80 or 90 percent. To change the system swappiness value, open/etc/sysctl.conf as root. Then, change or add this line to the file:

vm.swappiness = 10

Reboot for the change to take effect

$ cat /proc/sys/vm/swappiness

60

meaning that the swap file will be used fairly often if the memory usage is around half of my RAM. You can check your own system's swappiness value by running:the above require reboot. to avoid reboot use below command 

You can also change the value while your system is still running using below

sysctl vm.swappiness=10


Tuesday 10 March 2015

Federating Application Server with Dmgr Profile using addNode Command

C:\Program Files\IBM\WebSphere\AppServer\profiles\AppSrv01\bin>addNode.bat was.local.com 8879 -conntype soap -profileName AppSrv01 -username wasadmin -password wasadmin


The process carried when you run the addNode Command is
ADMU0116I: Tool information is being logged in file
C:\Progra~1\IBM\WebSphere\AppServer\profiles\AppSrv01\logs\addNode.log
ADMU0128I: Starting tool with the AppSrv01 profile
CWPKI0308I: Adding signer alias “CN=WASCell.local.com, ” to local
keystore “ClientDefaultTrustStore” with the following SHA digest:
BF:9B:69:F2:3A:88:53:A2:D7:6B:7F:8A:02:E5:F7:7A:BE:2D:98:22
CWPKI0308I: Adding signer alias “default” to local keystore
“ClientDefaultTrustStore” with the following SHA digest:
F9:E9:87:BC:ED:CA:B3:64:53:71:F9:1B:13:19:AA:C8:55:2C:88:A0
CWPKI0308I: Adding signer alias “datapower” to local keystore
“ClientDefaultTrustStore” with the following SHA digest:
A9:BA:A4:B5:BC:26:2F:5D:2A:80:93:CA:BA:F4:31:05:F2:54:14:17
ADMU0001I: Begin federation of node AppSrv01node with Deployment Manager at
dmgr01:8879.
ADMU0009I: Successfully connected to Deployment Manager Server: Dmgr01:8879
ADMU0507I: No servers found in configuration under:
C:\Progra~1\IBM\WebSphere\AppServer\profiles\AppSrv01\config/cells/WASCell/nodes/AppSrv01node/servers
ADMU2010I: Stopping all server processes for node AppSrv01node
ADMU0024I: Deleting the old backup directory.
ADMU0015I: Backing up the original cell repository.
ADMU0012I: Creating Node Agent configuration for node: AppSrv01node
ADMU0014I: Adding node Appsrv01node configuration to cell: WASCell
ADMU0016I: Synchronizing configuration between node and cell.
ADMU0018I: Launching Node Agent process for node: AppSrv01node
ADMU0020I: Reading configuration for Node Agent process: nodeagent
ADMU0022I: Node Agent launched. Waiting for initialization status.
ADMU0030I: Node Agent initialization completed successfully. Process id is:
6308
ADMU0505I: Servers found in configuration:
ADMU0506I: Server name: nodeagent
ADMU0300I: The node AppSrv01node was successfully added to the WASCell
cell.
ADMU0306I: Note:
ADMU0302I: Any cell-level documents from the standalone WASCell
configuration have not been migrated to the new cell.
ADMU0307I: You might want to:
ADMU0303I: Update the configuration on the WASCell Deployment
Manager with values from the old cell-level documents.
ADMU0306I: Note:
ADMU0304I: Because -includeapps was not specified, applications installed on
the standalone node were not installed on the new cell.
ADMU0307I: You might want to:
ADMU0305I: Install applications onto the WASCell cell using
wsadmin $AdminApp or the Administrative Console.
ADMU0003I: Node AppSrv01node has been successfully federated.


For UNIX Flavours:

/usr/IBM/WebSphere/AppServer/profiles/AppSrv01/bin>./addNode.sh was.local.com 8879 -conntype soap -profileName AppSrv01 -userName wasadmin -password wasadmin


NOTE: 1.HostName of DMGR is was.local.com
           2.SOAP port of DMGR is 8879
           3. we can use below options with addNode command 
                       [-includebuses] [-startingport <portnumber>] [-portprops
                       <qualified-filename>] [-nodeagentshortname <name>] [-nodegroupname
                      <name>] [-registerservice] [-serviceusername <name>]
                       [-servicepassword <password>] [-coregroupname <name>] [-noagent]
                       [-statusport <port>] [-quiet] [-nowait] [-logfile <filename>]
                       [-replacelog] [-trace] [-username <username>] [-password <pwd>]
                       [-localusername <localusername>] [-localpassword <localpassword>]
                       [-profileName <profile>] [-excludesecuritydomains] [-asExistingNode]
                       [-help]

Creating Deploment manager Profile command line


Creating Deployment manager Profile


Deployment Manager :

A Deployment Manager is "a server that manages operations for a logical group or cell of other
servers," the deployment manager is:... the central administration point of a cell that consists of multiple nodes and node groups in a distributed server configuration. ... The deployment manager uses the node agent to manage the application servers within one node. A deployment manager provides Overview of IBM WebSphere Application Terminology for IBM WebSphere Application Server Administrators 3management capability for multiple federated nodes and can manage nodes that span multiple systems and platforms. A node can only be managed by a single deployment manager and must be federated to the cell of that deployment manager. The configuration and application files for all nodes in the cell are centralized into a master configuration repository.This centralized repository is managed by the deployment manager and
synchronized with local copies that are held on each of the nodes.


For Windows:


AppServer_root/bin>manageprofiles 


-create -profileName Dmgr01 


-profilePath "C:\Program Files\IBM\WebSphere\AppServer\profiles\Dmgr01" 


-templatePath "C:\Program Files\IBM\WebSphere\AppServer\profileTemplates\dmgr" 


-nodeName Dmgr01node 


-cellName WASCell 


-hostName was.local.com


-enableAdminSecurity true

-adminUserName wasadmin

-adminPassword wasadmin




verfication:

check Dmgr01_create.log is created under C:\Program Files\IBM\WebSphere\AppServer\logs\manageprofiles\


find for <message>Returning with return code: INSTCONFSUCCESS</message>in Dmgr01_create.log file




Now Start the server 


C:\Program Files\IBM\WebSphere\AppServer\profiles\Dmgr01\bin>startManager.bat -username wasadmin -password wasadmin


after PID is generated in generated




opening console


from Browser and give URL


http://was.local.com:9062/ibm/console




For Unix Flavours:


/usr/IBM/WebSphere/AppServer/bin>./manageprofiles.sh \


-create -profileName Dmgr01 \

-profilePath /usr/IBM/WebSphere/AppServer/profiles/Dmgr01 \

-templatePath /usr/WebSphere/AppServer/profileTemplates/dmgr \

-nodeName Dmgr01node  \

-cellName WASCell \

-hostName was.local.com

-enableAdminSecurity true

-adminUserName wasadmin

-adminPassword wasadmin



verfication:

check Dmgr01_create.log is created under /usr/IBM/WebSphere/AppServer/logs/manageprofiles/


find for <message>Returning with return code: INSTCONFSUCCESS</message> in Dmgr01_create.log file




Now Start the server 


/usr/IBM/WebSphere/AppServer/profiles/Dmgr01/bin>./startManager.sh 
-username wasadmin -password wasadmin

after PID is generated in generated




opening console


from Browser and give URL


http://was.local.com:9062/ibm/console



Note:  1.Change your HostName to was.local.com


          2.For Windows: Find Administrative console port (Default port no 9062) from  C:\Program Files\IBM\WebSphere\AppServer\profiles\Dmgr01\logs\AboutThisProfile.txt 

            For Unix Flavours :         /usr/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/AboutThisProfile.txt