Unexpected Java Runtime Environment Error…
These are the kind of errors I like the most. When you see such an error being thrown when starting ODI agent in a production environment, you can go ahead and cancel dinner. Fortunately, for me, this error was on development server, so I had quite a nice dinner that evening.

I wanted to publish this post for some time now, thinking I will get to the cause of it later, my mistake for thinking in the first place. Now, as all the traces of the errors and the log files are beyond my reach, I decided just to share the information. The agent mentioned in this post is ODI 10g agent. Although, I don’t see a reason why such issue cannot arise with at least the 11g standalone agent.

I don’t know exactly what caused these errors, all I know is what I have done to resolve the situation.

The errors:

# An unexpected error has been detected by Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=<...>, pid=<...>, tid=<...>
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (11.0-b16 mixed mode linux-amd64)
# Problematic frame:
# V  [libjvm.so+...]
## If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug.

I have found a nice discussion about the first error on stackoverflow.com: SIGSEGV, libjvm. At the end, it was a hardware failure according to that discussion.

# An unexpected error has been detected by Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=<...>, pid=<...>, tid=<...>
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (11.0-b16 mixed mode linux-amd64)
# Problematic frame:
# C  [libclntsh.so.10.1+...]
#
# If you would like to submit a bug report, please visit:
# http://java.sun.com/webapps/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug.

While searching for the solution, I have seen different sayings about these kind of errors on the web. Starting from hardware failure and finishing with the wind blowing west.

The first error I have solved by asking to move that virtual machine to another physical host, based on the discussion I mentioned earlier. This had changed the error a little, notice that the problematic frame is different in the second error.

Finally, I came across another saying, which claimed that the second error is caused by a jar file being modified, while the application was still using it. Unfortunately, I don’t remember the link.

Well, it was worth a try, I have copied the entire “drivers” and “lib” directories under $ODI_HOME from another identical server. This indeed solved the issue, the agent came right up.
The most natural thing, of course, was to try and reproduce the problem and I tried, but when trying to replace or delete jars in the “drivers” or “lib” directories – nothing happens and restarting the agent works without any problems.

Well to wrap it up, the first error was solved by moving to another physical host, while the second by replacing jar files. The errors came one after another without any period in between. I would say:

  • Definitely, Unexpected errors 🙂
  • Both errors issued the SIGSEGV signal, so it must be a memory related issue, here is what Wikipedia has to say about that: SIGSEGV is the signal sent to a process when it makes an invalid memory reference.
  • Although, I am not sure right now, how the errors were caused, I hope, I have provided some useful information for whoever is expriencing similar problems and if I find out more about any of this, I will update this post.