Saturday, May 31, 2014

Tomcat 7.0.52 stalls during startup

Last night, I created a new server instance on HP Cloud and punched a hole in the firewall for port 8080.  I became puzzled when I couldn't reach the Tomcat test page (http://server.ip.address:8080).  I waited for a few minutes and the server still hadn't responded.  I checked the logs and noticed that Tomcat seemed to be stuck at the point of deploying the default ROOT webapp.

May 31, 2014 7:43:15 AM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/usr/share/tomcat7/common/classes], exists: [false], isDirectory: [false], canRead: [false]
May 31, 2014 7:43:15 AM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/usr/share/tomcat7/common], exists: [false], isDirectory: [false], canRead: [false]
May 31, 2014 7:43:16 AM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/usr/share/tomcat7/server/classes], exists: [false], isDirectory: [false], canRead: [false]
May 31, 2014 7:43:16 AM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/usr/share/tomcat7/server], exists: [false], isDirectory: [false], canRead: [false]
May 31, 2014 7:43:16 AM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/usr/share/tomcat7/shared/classes], exists: [false], isDirectory: [false], canRead: [false]
May 31, 2014 7:43:16 AM org.apache.catalina.startup.ClassLoaderFactory validateFile
WARNING: Problem with directory [/usr/share/tomcat7/shared], exists: [false], isDirectory: [false], canRead: [false]
May 31, 2014 7:43:22 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8080"]
May 31, 2014 7:43:22 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 5612 ms
May 31, 2014 7:43:22 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
May 31, 2014 7:43:22 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.52 (Ubuntu)
May 31, 2014 7:43:22 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /var/lib/tomcat7/webapps/ROOT

I was looking for the following line, but it never came.

INFO: Server startup in 3908 ms

I checked and re-checked the firewall settings in HP Cloud - just in case - and they seemed to be correct.  It was 4am EST so I decided to head to bed.  This morning, I woke up to check if Tomcat had eventually started.  I made a request to the Tomcat test page and received the expected response.  

Weird.

I tried to restart Tomcat to see if it was just a network hiccup, but Tomcat continued to stall at the same place in the log file.  So I went back through the log from the night before and noticed the last four lines below:

May 31, 2014 7:43:22 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.52 (Ubuntu)
May 31, 2014 7:43:22 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /var/lib/tomcat7/webapps/ROOT
May 31, 2014 9:03:02 AM org.apache.catalina.util.SessionIdGenerator createSecureRandom
INFO: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [4,772,515] milliseconds.
May 31, 2014 9:03:02 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8080"]
May 31, 2014 9:03:02 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 4780082 ms

Looks like it took 4.8 million ms (about 80 minutes) to create the SecureRandom instance (this is used by Tomcat to generate unique session IDs and other random values).  I haven't looked into why that might be the case, but I found a workaround here (http://wiki.apache.org/tomcat/HowTo/FasterStartUp) that uses a non-blocking entropy source.  The wiki page mentions that the non-blocking entropy source (/dev/urandom) is less secure than the blocking entropy source (/dev/random) because the non-blocking option generates less-random data.  However, given that this is a completely unacceptable startup delay, I figured I could live with the workaround until I've had some time to read up on different entropy sources.  

In any case, I just needed to add the following system property to Tomcat's startup routine.

-Djava.security.egd=file:/dev/./urandom

On Ubuntu, I generally add command-line properties like this to the setenv.sh script that is called by catalina.sh or startup.sh (/usr/share/tomcat7/bin/setenv.sh). That might be different for Windows or Mac, so you may need to look around to figure out the correct place for this setting -- adding it to Tomcat's start script is generally a bad idea. 

export CATALINA_OPTS="-Xms256m -Xmx512m -XX:MaxPermSize=128m -Duser.timezone=EST -Djava.security.egd=file:/dev/./urandom"

I'll try to update this blog entry once I've figured out the best way to resolve this issue for good.

Software versions
Ubuntu 14.04
Tomcat 7.0.52
Java version "1.7.0_55"
OpenJDK Runtime Environment (IcedTea 2.4.7) (7u55-2.4.7-1ubuntu1)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)


Saturday, December 14, 2013

Grails 2.3.2 > IllegalArgumentException: No enum constant org.springframework.http.HttpMethod.

I just encountered a pretty benign error that ended up taking me about an hour to resolve.  Since I didn't find any Q&As about this on stackoverflow.com, I figured I should document it in case anyone else runs into it.

So I wrote a simple Spock test for my AccountController.

@TestFor(AccountController)
@Mock([Account])
class AccountControllerSpec extends Specification {
    void "should return account with given identfier"() {
        when:
            def account = new Account(firstName: "Justin", lastName: "Miranda", age: 35).save(flush:true)
            params.id = account.id            
            controller.show()

        then:
            response.json.id == account.id
            response.json.firstName == account.firstName
            response.json.lastName == account.lastName
    }
}

Next, I wrote a simple controller with just a show method.

class AccountController {
    static allowedMethods = [show:"GET"]
    def show(Account account) {
        if (account == null) {
            render status:404
            return
        }
        render account as JSON
    }
}

Then I ran the unit test and received the following error.

grails> test-app unit: AccountController --verbose --stacktrace
| Running 1 unit test... 1 of 1
ID: 1
| Failure:  should return account with given identfier(com.kinectus.AccountControllerSpec)
|  java.lang.IllegalArgumentException: No enum constant org.springframework.http.HttpMethod.
at java.lang.Enum.valueOf(Enum.java:236)
at org.springframework.http.HttpMethod.valueOf(HttpMethod.java:27)
at org.codehaus.groovy.grails.plugins.web.api.ControllersApi.initializeCommandObject(ControllersApi.java:436)
at com.kinectus.AccountControllerSpec.should return account with given identfier(AccountControllerSpec.groovy:24)

Now that I know the solution, the error message is a little more helpful than I originally thought.  The period (.)  after HttpMethod is not the end of the sentence ... it's actually a hint to the answer.  It's saying No enum constant org.springframework.http.HttpMethod.SOMEVALUE.   It really should have only taken a few seconds to figure this out, but I had just spent a good deal of time working with HttpStatus, so I thought the issue was with HttpStatus for some reason.

Anyway, the solution is to add a request method to the test before calling the show controller action.

@TestFor(AccountController)
@Mock([Account])
class AccountControllerSpec extends Specification {
    void "should return account with given identfier"() {
        when:
            def account = new Account(firstName: "Justin", lastName: "Miranda", age: 35).save(flush:true)
            params.id = account.id
            request.method = "GET"
            controller.show()

        then:
            response.json.id == account.id
            response.json.firstName == account.firstName
            response.json.lastName == account.lastName
    }
}

And that's it.

Thursday, November 14, 2013

How to add a Unity Launcher for Intellij IDEA

This has been annoying me for the past two days.  I couldn't figure out how to edit the Intellij IDEA launcher after upgrading to the latest version of IDEA.  After playing around a bit and getting things working, I figured I'd post this so that I remember for the next upgrade (i.e. IDEA 13 is right around the corner).

Fire up Intellij IDEA

$ ~/apps/idea-IU-129.1359/bin/idea.sh

Create desktop entry (I always forget about this option)

Tools > Create desktop entry ...
(check the "Create entry for all users" checkbox so that the .desktop file is saved to /usr/share/applications)


Update .desktop file when you upgrade to a new version of Intellij IDEA

Different versions of IDEA create launchers with different names, so your file might have been saved as something else.  But for Intellij IDEA 12.1.6 Build 129.1359 the new desktop file is called jetbrains-idea.desktop.

If you ever lose that launcher or need to upgrade to a new version of Intellij IDEA, you can edit
$ sudo gedit /usr/share/applications/jetbrains-idea.desktop

/usr/share/applications/jetbrains-idea.desktop

[Desktop Entry]
Version=1.0
Type=Application
Name=IntelliJ IDEA
Icon=/home/jmiranda/apps/idea-IU-129.1359/bin/idea.png
Exec="/home/jmiranda/apps/idea-IU-129.1359/bin/idea.sh" %f
Comment=Develop with pleasure!
Categories=Development;IDE;
Terminal=false
StartupNotify=true
StartupWMClass=jetbrains-idea

If, for some reason, you click on the launcher and it doesn't launch, you may need to make the .desktop file executable.
$ sudo chmod +x /usr/share/applications/jetbrains-idea.desktop


Monday, November 11, 2013

How to disable a service in Ubuntu ...

$ sudo update-rc.d tomcat7 disable
update-rc.d: warning: tomcat7 start runlevel arguments (none) do not match LSB Default-Start values (2 3 4 5)
update-rc.d: warning: tomcat7 stop runlevel arguments (none) do not match LSB Default-Stop values (0 1 6)
 Disabling system startup links for /etc/init.d/tomcat7 ...
 Removing any system startup links for /etc/init.d/tomcat7 ...
   /etc/rc0.d/K08tomcat7
   /etc/rc1.d/K08tomcat7
   /etc/rc2.d/S92tomcat7
   /etc/rc3.d/S92tomcat7
   /etc/rc4.d/S92tomcat7
   /etc/rc5.d/S92tomcat7
   /etc/rc6.d/K08tomcat7
 Adding system startup for /etc/init.d/tomcat7 ...
   /etc/rc0.d/K08tomcat7 -> ../init.d/tomcat7
   /etc/rc1.d/K08tomcat7 -> ../init.d/tomcat7
   /etc/rc6.d/K08tomcat7 -> ../init.d/tomcat7
   /etc/rc2.d/K08tomcat7 -> ../init.d/tomcat7
   /etc/rc3.d/K08tomcat7 -> ../init.d/tomcat7
   /etc/rc4.d/K08tomcat7 -> ../init.d/tomcat7
   /etc/rc5.d/K08tomcat7 -> ../init.d/tomcat7

Thursday, August 9, 2012

Disabling Ubuntu services (like Jenkins) during startup

sudo sh -c "echo 'manual' > /etc/init/jenkins.override"

Monday, July 23, 2012

Using SSH with hostname tab completion

I've always hated typing the fully qualified hostname for a server that I wanted to SSH into.  I just learned how to configure Ubuntu to use tab completion with the SSH command.  Just run the following command and you can start using tab completion (NOTE: this might require a restart of your terminal session):

$ complete -W "$(echo $(grep '^ssh ' .bash_history | sort -u | sed 's/^ssh //'))" ssh


$ ssh [TAB]

See the following article for more details:
http://www.commandlinefu.com/commands/view/2759/ssh-autocomplete

Update:  I just realized this morning that there's one more thing you need to do to get this working properly.  Somehow you need to source your custom complete commands when you open a new terminal.  I use bash, so I vi'd my .bashrc file and added my custom completion commands.  There's also a directory called /etc/bash_completion.d for these scripts, but I felt a little squeamish putting my custom ones there.


# enable programmable completion features (you don't need to 
enable this, if it's already enabled in /etc/bash.bashrc 
# and /etc/profile sources /etc/bash.bashrc).
if [ -f /etc/bash_completion ] && ! shopt -oq posix; then
    . /etc/bash_completion


    # custom completions
    complete -W "$(echo $(grep '^ssh ' .bash_history | sort -u | sed 's/^ssh //'))" ssh
fi

Monday, March 8, 2010

[SOLVED] RSA host key has changed

I've run into the "RSA host key for has changed" issue is a couple times in the past few months as we have a few servers whose IP addresses have changed.
jmiranda@jmiranda-laptop:~$ ssh www.example.com
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: POSSIBLE DNS SPOOFING DETECTED!
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The RSA host key for www.example.com has changed,
and the key for the corresponding IP address 12.34.56.78
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
aa:bb:cc:dd:ee:ff:11:22:33:44:55:66:77:88:99:00.
Please contact your system administrator.
Add correct host key in /home/jmiranda/.ssh/known_hosts to get rid of this message.
Offending key in /home/jmiranda/.ssh/known_hosts:2
RSA host key for www.example.com has changed and you have requested strict checking.
Host key verification failed.
I've wasted a few minutes Googling the solution each time, so I'm adding a blog entry for it now so that I have a reference to the solution next time it happens.
jmiranda@jmiranda-laptop:~$ ssh-keygen -R www.example.com
/home/jmiranda/.ssh/known_hosts updated.
Original contents retained as /home/jmiranda/.ssh/known_hosts.old