Java Crashing in libc.so.6

I must share this horrible experience although it is not directly related to programming. But I hope that article will be helpful for somebody.

I use Java in my browser for practical things like digital signing of requests when communicating with offices and banks. I use it very often and without any serious problems. Of course I had to configure my truststore and keystore and I also have two versions of extension (jre/lib/ext) directory, because different offices and banks use different versions of the IAIK library. But it is another story.

But one nice morning when I tried to connect to a page containing an applet, my Java console window (which I have configured to be always started) flashed and the entire browser crashed. I used Java JDK 1.6.0_04 (32bit on 64bit Linux machine - kernel 2.6.23).

I tried again and again, but with the same result - crash. After a while I found, that there are generated files with name "hs_err_pidXXXX.log" wher XXXX was a PID of the process which crashed. I looked into it - it began with these lines:

#
# An unexpected error has been detected by Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0094a8ac, pid=5587, tid=4108254096
#
# Java VM: Java HotSpot(TM) Client VM (10.0-b19 mixed mode, sharing linux-x86)
# Problematic frame:
# C  [libc.so.6+0x718ac]  memcpy+0x1c
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

And then a lot of stack traces, thread states and so on, which I was not interested.

So it looked like a problem in libc.so.6 ! It was strange, because I used Java in my browser a day before without problems.

I realized that was the time to upgrade my browser, so I tried these steps:

• updating from FireFox 3.0 to FireFox 3.6 - hmm, the Java plugin (libjavaplugin_oji.so) did not worked


• updating from JDK 1.6.0_04 to JDK 1.6.0_21 - hmm, the Java plugin (libjavaplugin_oji.so) still did not worked - using info at FireFox Java Plugin Doc


• Finally I found info about new Java plugin technology here, so Java was able to start in FireFox 3.6, but crashed again - only the Java process, not FireFox, but I needed Java...

I tried to clear all my FireFox user profile (including cache, cookies..) - but Java crashed again.

I tried to clear all my Java user profile data - but the result was the same - crash!

I even simulated crash with Java only - no browser plugin. So now it seemed that problem is not in the plugin or browser at all.

There was still the same info in the hserr_pidXXXX.log - "problematic frame in libc.so.6". Ok, it was the time to look at this.

I found that I use libc-2.7.so. It seemed that problem could be there. OK, I had to try to compile the newest version of it and try to start Java with it (certainly I could not change the core library for the whole system !!)

(Un)Fortunately I could not compile the libc sources.
It was horrible - only reinstalling the whole system was my last chance to bring Java to life again.

I looked into the log file again and tried to decompile the info there:

Stack: [0xf4d9f000,0xf4df0000],  sp=0xf4d6e0f4,  free space=-196k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x718ac]  memcpy+0x1c
C  [libc.so.6+0x59c01]  _IO_getline+0x41
C  [libc.so.6+0x6236b]  fgets_unlocked+0x5b
C  [libnss_files.so.2+0x3045]
C  [libnss_files.so.2+0x382b]  _nss_files_gethostbyname2_r+0xdb

The last line was a little sun in dark. It looked like a problem when translating DNS names to IP addresses using file - /etc/hosts !

I looked into /etc/hosts and I was surprised - there were two lines (records) with duplicating host names:
127.0.0.1 localhost localhost localhost .....
The whole file had 3.3MB in size !! That was the reason that it crashed in the libc - it was an overflow error caught by the system and killing the process !

And why Java only was crashing ? Maybe other applications would crash soon as well. Java was crashing just after the start, because it tried to resolve DNS names stored in the cache - in the cache of loaded applets - that was the reason !!

Uff, I was very happy, that the solution was so simple at the end.

Raw AAC Player for Android

This blog (and project) was written for those developers spending hours of googling the internet to answer a simple question:

How to make Android to play raw AAC files (streams) ?

I've spent many hours by it and I want to save other's time to public what I found.

I tried to:

• invoke MediaPlayer on raw AAC http:// URL (error)


• invoke MediaPlayer on raw AAC file (error in emulator, OK on HTC Desire)


• pass a NetSocket FileHandle to MediaPlayer (error)


• filter the stream using local ServerSocket and:
  
  
  • discard HTTP header (error)
  
  
  • change content-type to several possibilities like audio/mpeg (error)
  
  
  • sync ADTS header (icecast does not do it) (error)
  
  
• split the AAC stream into small file chunks (OK, but not excellent)


• use 3rd party native library (FAAD2) to decode AAC (OK on HTC Desire, working on emulator - but slow)

Using FAAD2 library works on my HTC Desire phone without problems. I tried 32kbit AAC stream served by www.play.cz (the exact URL you can find in the sources). It failed when trying 128kbit AAC stream served by the same server - but it also failed when using mplayer or when I saved it to a file and tried the internal HTC Desire's AAC player.

On the emulator it works, but it is not smooth (performance problems).
I tried Android 1.5 (cupcake) and Android 2.1 (Eclair) versions - both with the same results (working, but not perfect).

So the result is that playing raw AAC on Android devices (er even on the emulator) is possible, but there are several "buts":

• although Android contains AAC decoder, you cannot use it / access it natively


• if you create your own AAC decoder, then as a developer you must obtain a license from the patent holders - and you must pay for it

The sources you can find here:
http://code.google.com/p/aacplayer-android/

---

Updated 2011-05-05
I've added OpenCORE aacdec library/decoder and rewrote both Java and C parts of the player. Now I am able to play streams smoothly even on the emulator (using no more than 60% of my host CPU)!
You can download the sources and compiled shared libraries from the Downloads section (aacplayer-android-r20.zip).

How to compile it ? Just unzip the archive, edit local.properties, copy sample.ant.properties to .ant.properties (oh, yes in rev20 you must change the name of the property "loglevel" to "jni.loglevel") and run Apache Ant (beginners please consult Android SDK and NDK documentation - you need to have installed Apache Ant and GNU Make - Windows users also Cygwin). Or read the README file :-)

---

Updated 2011-05-26
I've created a follow-up project "AACDecoder" which uses only OpenCORE aacdec library and allows using the working library in other 3rd party Android projects.

http://code.google.com/p/aacdecoder-android/

Thank you all for your feedback!

GAE/J - Memcache namespaces and validations

In GAE/J SDK 1.3.4 the method MemcacheService.setNamespace(String ns) was deprecated. Instead of that one should use MemcacheServiceFactory.getMemcacheService(String ns).

When using the new one, the name of the namespace is validated against pattern, but this validation is not done when calling the old (deprecated) method:

  String namespace = "TEST:One";

  MemcacheService msOld = MemcacheServiceFactory.getMemcacheService();
  // deprecation warning by compiler, but OK at runtime:
  msOld.setNamespace( namespace );

  // no warning by compiler, but throws exception at runtime:
  MemcacheService msNew = MemcacheServiceFactory.getMemcacheService( namespace );

The exception is:

  java.lang.IllegalArgumentException:
    Namespace 'TEST:One' does not match pattern '[0-9A-Za-z._-]{0,100}'.

I think that this behaviour should be documented in low-level API at least.

http://groups.google.com/group/google-appengine-java/browse_thread/thread/8b5792d38bf74f28#

Google App Engine Data Viewer - GQL Java

A simple data viewer based on the GQL dynamic parser. The parser needs ANTLR library - antlr-runtime-3.2.jar

Currently this code was tested on local devapp server running GAE SDK 1.3.1

<%@ taglib uri="http://java.sun.com/jsp/jstl/core" prefix="c" %>
<jsp:directive.page language="java" contentType="text/html; charset=UTF-8" isELIgnored="false"/>
<jsp:directive.page import="com.google.appengine.api.datastore.Cursor"/>
<jsp:directive.page import="com.google.appengine.api.datastore.DatastoreServiceFactory"/>
<jsp:directive.page import="com.google.appengine.api.datastore.Entity"/>
<jsp:directive.page import="com.google.appengine.api.datastore.FetchOptions"/>
<jsp:directive.page import="com.google.appengine.api.datastore.QueryResultIterator"/>
<jsp:directive.page import="com.google.appengine.api.datastore.PreparedQuery"/>
<jsp:directive.page import="com.spoledge.audao.parser.gql.GqlDynamic"/>
<jsp:directive.page import="java.util.ArrayList"/>
<jsp:directive.page import="java.util.HashSet"/>
<jsp:directive.page import="java.util.Iterator"/>
<html>
<head>
<title>GQL Console</title>
<style type="text/css">
body { background: #fff; color: #333; font-family: Arial, Verdana, Sans-Serif;}
table { border-collapse: collapse; border: 1px solid #888; width:100%}
td { border-top: 1px solid #888; padding: 2px 10px; font-size: 85%}
tr.header td { font-weigth: bold; background: #555; color: #fff}
tr.even td { background: #fff;}
tr.odd td { background: #ccc;}
</style>
</head>
<body>
<form action="gql.jsp" method="post">
    Enter a GQL query (e.g. "SELECT * FROM MyEntity WHERE propertyName='propertyValue'"):<br/>
    <input type="text" size="96" name="gql" value="<c:out value="${param.gql}" escapeXml="true"/>"/><br/>
    <input type="submit" value="Execute"/>
</form>
<%
    String gql = request.getParameter( "gql" );
    String encCursor = request.getParameter( "cursor" );
    int offset = 0;
    int maxRows = 20;

    try {
        offset = Integer.parseInt( request.getParameter("offset"));
    }
    catch (Exception e) {}

    if (gql != null && gql.length() != 0) {
        GqlDynamic gqld = new GqlDynamic();
        gqld.setDatastoreService( DatastoreServiceFactory.getDatastoreService());

        try {
            PreparedQuery pq = gqld.prepareQuery( gql );
            FetchOptions fo = gqld.getFetchOptions();

            if (encCursor != null) {
                // add +1 to detect more rows:
                FetchOptions foo = FetchOptions.Builder.withLimit( (fo.getLimit() != null ? fo.getLimit() : maxRows)+1);
                foo.cursor( Cursor.fromWebSafeString( encCursor ));

                fo = foo;
            }
            else {
                offset = fo.getOffset();

                if (fo.getLimit() == null) fo.limit( maxRows + 1 );
            }

            QueryResultIterator<Entity> result = pq.asQueryResultIterator( fo );
            Cursor cursor = null;

            // find all property names
            ArrayList<Entity> entities = new ArrayList<Entity>( maxRows );
            HashSet<String> propNameSet = new HashSet<String>();

            while (result.hasNext()) {
                Entity ent = result.next();
                propNameSet.addAll( ent.getProperties().keySet() );

                entities.add( ent );
                if (entities.size() >= maxRows) {
                    if (result.hasNext()) cursor = result.getCursor();
                    break;
                }
            }

            ArrayList<String> propNames = new ArrayList<String>( propNameSet );
            java.util.Collections.sort( propNames,
                new java.util.Comparator<String>() {
                    public int compare( String s1, String s2) {
                        return s1.compareTo( s2 );
                    }
                });

            // render header
            out.print("<table><tr class=\"summary\"><td colspan=\""+ (propNames.size()+1)+"\">");
            if (entities.size() == 0) {
                out.print("No records found");
            }
            else {
                out.print("<b>" + (offset+1) + '-' + (offset+entities.size()) + "</b> ");

                if (cursor != null) {
                    out.print("<a href=\"gql.jsp?gql=" + gql + "&cursor=" + cursor.toWebSafeString()
                        + "&offset=" + (offset+maxRows)
                        + "\">Next</a> " );
                }
                else {
                    out.print( "Next" );
                }
            }
            out.print("</td></tr>");

            if (entities.size() != 0) {
                out.print("<tr class=\"header\">");
                out.print("<td>Id</td>");
                for (String propName : propNames) {
                    out.print("<td>" + propName + "</td>");
                }
                out.print("</tr>");
            }

            // render values
            int i = 0;
            for (Entity ent : entities) {
                out.print("<tr class=\"" + ((++i % 2)==0 ? "odd" : "even" ) +"\">");
                out.print("<td>");
                out.print( ent.getKey().getName() != null ? ent.getKey().getName() : ent.getKey().getId());
                out.print("</td>");
                for (String propName : propNames) {
                    out.print( "<td>" );
                    if (ent.hasProperty( propName )) {
                        Object o = ent.getProperty( propName );
                        if (o == null) {
                            out.print( "null" );
                        }
                        else {
                            pageContext.setAttribute("val", o.toString());
%>
    <c:out value="${val}" escapeXml="true"/>
<%
                        }
                    }
                    out.print( "</td>" );
                }
                out.print("</tr>");
            }
            out.print("</table>");
        }
        catch (Exception e) {
            out.print( "Invalid GQL: " + e );
            e.printStackTrace();
        }
    }
    else {
%>
    Please enter a query.
<%
    }
%>
</body>
</html>

Lists and Nulls in Google App Engine Datastore

Do you know what happens when you store empty List into Google App Engine Datastore ? And do you know what happens if you store a List containing one null value ?

Let's look at a piece of code:

    DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
    ArrayList emptyList = new ArrayList();
    ArrayList hasNullList = new ArrayList();
    hasNullList.add( null );

    Entity ent = new Entity("Test");
    ent.setProperty("nullProp", null);
    ent.setProperty("emptyList", emptyList);
    ent.setProperty("hasNullList", hasNullList);

    System.out.println("ENTITY - before: " + ent);

    Key key = ds.put( ent );
    ent = ds.get( key );

    System.out.println("ENTITY - after: " + ent);

The output of the code is following:

 ENTITY - before: <Entity [Test(no-id-yet)]:
     nullProp = null
     hasNullList = [null]
     emptyList = []
 >
 
 ENTITY - after: <Entity [Test(1)]:
     nullProp = null
     hasNullList = [null]
     emptyList = null
 >

So it means, that GAE does not store an empty list at all ! Instead of an empty list, the property is null.

This is not so big surprise and is not against anything in the official documentation. But the problem is the following:

Can we filter 'null' and '[null]' values separately ?

We will answer quickly - just extend our first example by these lines:

    System.out.println( "count(emptyList==null) = "
        + ds.prepare( 
            new Query("Test").addFilter("emptyList",
                Query.FilterOperator.EQUAL, null)
         ).countEntities());

    System.out.println( "count(hasNullList==null) = "
        + ds.prepare(
            new Query("Test").addFilter("hasNullList",
                Query.FilterOperator.EQUAL, null)
         ).countEntities());

And the output is:

 count(emptyList==null) = 1
 count(hasNullList==null) = 1

This means that it is not possible (this way) to distinguish between null and [null] properties!

So the summary is: do not store nulls into Lists, otherwise you will get into a problem in future.

Large Response in AJAX XMLHttpRequest

Sometimes happens that you need to send large data in the response of the XMLHttpRequest (XHR). For example you need to fetch a sorce code and display it somewhere in the document - exactly as I needed.

The code was as follows:

    if (window.XMLHttpRequest) {
      xhr=new XMLHttpRequest();
    }
    else {
      xhr=new ActiveXObject("Microsoft.XMLHTTP");
    }
    xhr.open("GET","/code/MyCode.java",false);
    xhr.send("");

    xmlDoc=xhr.responseXML;

    // a bug is somewhere here:
    code=xmlDoc.getElementsByTagName('code').childNodes[0].nodeValue;

Unfortunately this worked fine in Internet Explorer (IE). In FireFox (FF) it worked if the size of the code was less than certain limit. The limit in my FF 3.0 was 4096 bytes.

So the solution which works ion both IE and FF was the following:

    cnt = xmlDoc.getElementsByTagName('code')[0].childNodes;

    code = cnt[0].nodeValue;
    for (i=1; i < cnt.length; i++) code += cnt[j].nodeValue;

So you do not need to create chunks on the server, just concatenate what FF split on the client.

XSLT Using Xalan on Google App Engine

The online DAO generator at audao.spoledge.com is running on the Google App Engine. The DAO generator uses XSLT transformation and Xalan is used as the XSLT implementation.

When I tested it locally using the dev appserver, everything worked fine. But after I uploaded it to the real Google App Engine, then strange things started to happen - the XSL transformations were somehow cut, unfinished, simply different than I expected. When I looked into the logs, I found why:

javax.xml.transform.TransformerException: Failed calling setMethod method
 at org.apache.xalan.processor.StylesheetHandler.error(StylesheetHandler.java:907)
 at org.apache.xalan.processor.StylesheetHandler.error(StylesheetHandler.java:950)
 at org.apache.xalan.processor.XSLTAttributeDef.setAttrValue(XSLTAttributeDef.java:1638)
 ...
Caused by: java.lang.IllegalAccessException: Class
com.google.apphosting.runtime.security.shared.intercept.java.lang.reflect.Method_$3
can not access a member of class org.apache.xalan.processor.ProcessorOutputElem with
modifiers "public"
 at sun.reflect.Reflection.ensureMemberAccess(Unknown Source)
 at java.lang.reflect.Method.invoke(Unknown Source)
 at
com.google.apphosting.runtime.security.shared.intercept.java.lang.reflect.Method_$3.run(Method_.java:149)
 at java.security.AccessController.doPrivileged(Native Method)
 at
com.google.apphosting.runtime.security.shared.intercept.java.lang.reflect.Method_.privilegedInvoke(Method_.java:147)
 at
com.google.apphosting.runtime.security.shared.intercept.java.lang.reflect.Method_.invoke(Method_.java:120)
 at org.apache.xalan.processor.XSLTAttributeDef.setAttrValue(XSLTAttributeDef.java:1626)
 ...

The problem was only with the XSL <xsl:output ...> tag. When I removed it, everything was OK. It was a surprise for me, because the reflection used for this tag is also used for other XSL tags and there were no errors there - just only with the <xsl:output...> tag.

Since I relied in Xalan (I have a bad experience using more than one implementation in one project), I had to patch it myself. The patch just calls the implementation Java method directly - without reflection. And it works now ! The patch you can find here (post 21).

StackOverflowError - Including a non-JSP Page

This happened to me when developing an application on Google App Engine (GAE SDK 1.3.1).

I tried to include a non-JSP page using standard <jsp:include page="/resources/news.xml"/> tag. There was no problem when testing on the local dev appserver (I only had to specify that the "news.xml" is a resource file in the appengine-web.xml). But when it was running on the production - real - GAE, I got the following error:

java.lang.StackOverflowError
 at java.lang.String.startsWith(Unknown Source)
 at org.mortbay.jetty.servlet.Dispatcher$IncludeAttributes.setAttribute(Dispatcher.java:475)
 at org.mortbay.jetty.servlet.Dispatcher$IncludeAttributes.removeAttribute(Dispatcher.java:508)
 at org.mortbay.jetty.servlet.Dispatcher$IncludeAttributes.setAttribute(Dispatcher.java:488)
 at org.mortbay.jetty.servlet.Dispatcher$IncludeAttributes.removeAttribute(Dispatcher.java:508)
 at org.mortbay.jetty.servlet.Dispatcher$IncludeAttributes.setAttribute(Dispatcher.java:488)
 at org.mortbay.jetty.servlet.Dispatcher$IncludeAttributes.removeAttribute(Dispatcher.java:508)
 ...

When I renamed the "news.xml" to "news.jsp", then everything worked fine. Is it a bug or feature ?