Sunday 27 March 2016

ClassNotFound Exception after installation of application | application url is giving 404 exception and giving classNotFound in logs

1.check the parent class first class loader policy is set.

2. make sure all required shared libraries is mapped to application.

3. check all the jar files are available at shared library location.

4. verify class path is set for the application is there is any.




Wednesday 23 March 2016

disk space full alerts on /tmp which is 1GB size and there are no file available or all files available are taking less than 10MB on /tmp.

To free up the space on /tmp we have used below command to identify which process are taking space on /tmp. and restarted the process will free up the space on /tmp.

$lsof|grep -i wasadm|grep -i deleted


the above command lists all the open files and unmapped process using /tmp space and running with wasadm user.

after creating the datasource test connection is successful but when application is accessed that time it is giving error from newly creating server(WAS8.5) and it is not giving any error when accessible from the old server(WAS6.1).

issue is caused due to Microsoft SQL server went to read only mode, after Microsoft SQL server restart everything was working fine.

Tuesday 22 March 2016

Application is not able to access from web seal servers and they were working fine from direct URL's.

issue is with web seal log file reached to 2GB so it stopped the responding .to resolve this we have renamed the log file and restarted the web seal server then application is able to access from the web seal servers as log rotation is made.

Webseal log Rotation

The websead msg log is not controlled from within the webseal conf file, but instead, via the routing file.
--------------------------------------------
The contents of the msg__webseald.log file come from webseal's STDERR, as controlled in the /opt/pdweb/etc/routing file
 FATAL:STDERR:-
ERROR:STDERR:-
WARNING:STDERR:-

The routing file can be modified to redirect these messages to a file, and the log file management can be configured at the same time.
 For example.
FATAL:UTF8FILE.10.10000:/var/pdweb/log/msg__webseald.log
ERROR:UTF8FILE.10.10000:/var/pdweb/log/msg__webseald.log
WARNING:UTF8FILE.10.10000:/var/pdweb/log/msg__webseald.log


Note:log file will be rotated after 10000 entries in any of file Fatal or error or warning log file .and max historical files are 10

In one of the application giving the SSLHandshake error after clicking on one of the tab's in application even all certificates are available in server.

after discussing with application team we understand that after clicking on that tab it will go to other application in other jvm (reporting application residing server)in the same host and generate the report. so we have used the retrieve from port option with reporting server certificate name to get the other report generating server certificate and restarted the error giving server to resolve the error. if reporting server is residing in other host  then we need to add reporting application root and intermediate cert in truststore of error giving application and restart the server is other way to resolve this type of issue .

Monday 21 March 2016

hostname not found and SSLHandshake error

we have renewed VeriSign web server certificate from our end informed to application team for checkout the application ,during checkout it went successful without any issue. we were not sure to which other applications the current application is interacting with. later when people soft servers are bounce after 3months from people soft team, people soft  were seeing sslhandshake error(related to our vip name) from people soft team end along with host not found exception in logs. to solving this we have give verisign intermediate cert (class G4) to them to add intermediate cert  in their server, after adding cert in people soft server and restarting peoplesoft servers the sslhandshake error is fixed.to resolve the host not found we have reached DNS team to resolve from DNC team end, for temporary fix we have added our vip name in host aliases in /etc/resolve.conf and restarted networkmanager service .

webserver is not starting or not stoping and just giving unable to start webserver or unable to stop webserver

when i tried to grep process using ps command some of the web server process are running with root id and pid is generated with root .for solving this issue i have killed all running instances of that particular web server and changed the pid running user and group to correct id and group and started web server started.

grepping the process

in Linux:

$ps -ef|grep -i <webserver_name> | grep -i <webserver_port>

in Solaries

$/usr/ucb/ps -auxwww|grep -i <webserver_name> | grep -i <webserver_port>

killing process:

$kill -9 <pid>

Sunday 20 March 2016

script to install Application

AdminApp.install("/opt/MyApplications/Application1.ear", [-cluster cluster1]')
AdminConfig.save()


Note: here cluster1 is the name of Cluster

PKIX or java.security.cert.CertPathValidatorException or unable to find valid certification path to requested target error

ERROR: PKIX path validation failed | subject/issuer name chaining check failed
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: subject/issuer name chaining check failed
...
...
Caused by: java.security.cert.CertPathValidatorException: subject/issuer name chaining check failed

(OR)

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
                at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
                at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
                at sun.security.validator.Validator.validate(Validator.java:260)
                at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
                at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229)
                at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124)
                at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1351)
                at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:156)
                at sun.security.ssl.Handshaker.processLoop(Handshaker.java:925)
                at sun.security.ssl.Handshaker.process_record(Handshaker.java:860)
                at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1043)
                at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1343)
                at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:728)
                at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
                at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:138)
                at SSLPoke.main(SSLPoke.java:31)
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
                at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:145)
                at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
                at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
                at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
                ... 15 more

Problem: whenever Java Attempts to connect to another application over SSL ex: HTTPS, IMAPS, LDAPS). it will only be able to connect to that application if it can trust it. the way trust is handled in java world is that you have a keystore(typically $JAVA_HOME/lib/security/cacerts), also known as truststore. this contains a list of all known Certificate Authority(CA) certificates. and java will only trust that certificates that are signed by one of those CA's or public certificates that exist in keystore.

This problem is therefore caused by a certificate that is self signed (a CA didnot sign it) or a certificate chain does not exist within the java truststore.java does not trust the certificate and fail to connect to the application.

Rosoultion:
1.Make sure you have imported the public certificate (root)of the target instance into the truststore.
2.Make sure any certificates have been imported into the correct truststore; you may have multiple JRE/JDKs.
importing the Public cert(root cert of server)
Ex: <JAVA_HOME>/bin/keytool -import -alias <server_name> -keystore <JAVA_HOME>/jre/lib/security/cacerts -file public.crt
3.Check to see that the correct truststore is in use. If -Djavax.net.ssl.trustStore has been configured, it will override the location of the default truststore, which will need to be checked. Check if your Anti Virus tool has "SSL Scanning" blocking SSL/TLS. If it does, disable this feature or set exceptions for the target
 addresses.
4.If connecting to a mail server, such as Exchange, ensure authentication allows plain text.
Verify that the target server is configured to serve SSL correctly.

Alternative KeyStore Locations
Java will normally use a system-wide keystore in $JAVA_HOME/jre/lib/security/cacerts, but it is possible to use a different keystore by specifying a parameter,
-Djavax.net.ssl.trustStore=/path/to/keystore, where '/path/to/keystore' is the absolute file path of the alternative keystore.

However, setting this is not recommended because if Java is told to use a custom keystore (eg. containing a self-signed certificate),
then Java will not have access to the root certificates of signing authorities found in $JAVA_HOME/jre/lib/security/cacerts, and
 accessing most CA-signed SSL sites will fail. It is better to add new certificates (eg. self-signed) to the system-wide keystore

configuring the alternate keystore location in websphere application server
Add the trustStore information to WAS via the admin console in the
Generic JVM arguments.

->Application Servers
-> server1
-> Process Definition
-> Java Virtual Machine
in custom properties

javax.net.ssl.trustStore=/ourownLocation/cacerts
javax.net.ssl.trustStorePassword=changeit
javax.net.ssl.trustStoreType=jks



second way to fix the issue by not disclosing the password in console

Password will no longer be displayed on the process list by including the JKS password to a file in the following step.
1. Create the following file.

/usr/jksProps.txt;
=====
-Djavax.net.ssl.trustStore=/usr/IBM/abcapp.jks -Djavax.net.ssl.trustStorePassword=password
=====

2. Add the trustStore information to WAS via the admin console in the
Generic JVM arguments.

->Application Servers
-> server1
-> Process Definition
-> Java Virtual Machine

example;
-Dclient.encoding.override=UTF-8 -Xoptionsfile=/usr/IBM/jksProps.txt

3. Restart JVM