Wednesday, February 24, 2010

Google App Engine Data Viewer - GQL Java

A simple data viewer based on the GQL dynamic parser. The parser needs ANTLR library - antlr-runtime-3.2.jar

Currently this code was tested on local devapp server running GAE SDK 1.3.1

<%@ taglib uri="" prefix="c" %>
< language="java" contentType="text/html; charset=UTF-8" isELIgnored="false"/>
< import=""/>
< import=""/>
< import=""/>
< import=""/>
< import=""/>
< import=""/>
< import="com.spoledge.audao.parser.gql.GqlDynamic"/>
< import="java.util.ArrayList"/>
< import="java.util.HashSet"/>
< import="java.util.Iterator"/>
<title>GQL Console</title>
<style type="text/css">
body { background: #fff; color: #333; font-family: Arial, Verdana, Sans-Serif;}
table { border-collapse: collapse; border: 1px solid #888; width:100%}
td { border-top: 1px solid #888; padding: 2px 10px; font-size: 85%}
tr.header td { font-weigth: bold; background: #555; color: #fff}
tr.even td { background: #fff;}
tr.odd td { background: #ccc;}
<form action="gql.jsp" method="post">
Enter a GQL query (e.g. "SELECT * FROM MyEntity WHERE propertyName='propertyValue'"):<br/>
<input type="text" size="96" name="gql" value="<c:out value="${param.gql}" escapeXml="true"/>"/><br/>
<input type="submit" value="Execute"/>
String gql = request.getParameter( "gql" );
String encCursor = request.getParameter( "cursor" );
int offset = 0;
int maxRows = 20;

try {
offset = Integer.parseInt( request.getParameter("offset"));
catch (Exception e) {}

if (gql != null && gql.length() != 0) {
GqlDynamic gqld = new GqlDynamic();
gqld.setDatastoreService( DatastoreServiceFactory.getDatastoreService());

try {
PreparedQuery pq = gqld.prepareQuery( gql );
FetchOptions fo = gqld.getFetchOptions();

if (encCursor != null) {
// add +1 to detect more rows:
FetchOptions foo = FetchOptions.Builder.withLimit( (fo.getLimit() != null ? fo.getLimit() : maxRows)+1);
foo.cursor( Cursor.fromWebSafeString( encCursor ));

fo = foo;
else {
offset = fo.getOffset();

if (fo.getLimit() == null) fo.limit( maxRows + 1 );

QueryResultIterator<Entity> result = pq.asQueryResultIterator( fo );
Cursor cursor = null;

// find all property names
ArrayList<Entity> entities = new ArrayList<Entity>( maxRows );
HashSet<String> propNameSet = new HashSet<String>();

while (result.hasNext()) {
Entity ent =;
propNameSet.addAll( ent.getProperties().keySet() );

entities.add( ent );
if (entities.size() >= maxRows) {
if (result.hasNext()) cursor = result.getCursor();

ArrayList<String> propNames = new ArrayList<String>( propNameSet );
java.util.Collections.sort( propNames,
new java.util.Comparator<String>() {
public int compare( String s1, String s2) {
return s1.compareTo( s2 );

// render header
out.print("<table><tr class=\"summary\"><td colspan=\""+ (propNames.size()+1)+"\">");
if (entities.size() == 0) {
out.print("No records found");
else {
out.print("<b>" + (offset+1) + '-' + (offset+entities.size()) + "</b> ");

if (cursor != null) {
out.print("<a href=\"gql.jsp?gql=" + gql + "&cursor=" + cursor.toWebSafeString()
+ "&offset=" + (offset+maxRows)
+ "\">Next</a> " );
else {
out.print( "Next" );

if (entities.size() != 0) {
out.print("<tr class=\"header\">");
for (String propName : propNames) {
out.print("<td>" + propName + "</td>");

// render values
int i = 0;
for (Entity ent : entities) {
out.print("<tr class=\"" + ((++i % 2)==0 ? "odd" : "even" ) +"\">");
out.print( ent.getKey().getName() != null ? ent.getKey().getName() : ent.getKey().getId());
for (String propName : propNames) {
out.print( "<td>" );
if (ent.hasProperty( propName )) {
Object o = ent.getProperty( propName );
if (o == null) {
out.print( "null" );
else {
pageContext.setAttribute("val", o.toString());
<c:out value="${val}" escapeXml="true"/>
out.print( "</td>" );
catch (Exception e) {
out.print( "Invalid GQL: " + e );
else {
Please enter a query.

Friday, February 19, 2010

Lists and Nulls in Google App Engine Datastore

Do you know what happens when you store empty List into Google App Engine Datastore ? And do you know what happens if you store a List containing one null value ?

Let's look at a piece of code:

DatastoreService ds = DatastoreServiceFactory.getDatastoreService();
ArrayList emptyList = new ArrayList();
ArrayList hasNullList = new ArrayList();
hasNullList.add( null );

Entity ent = new Entity("Test");
ent.setProperty("nullProp", null);
ent.setProperty("emptyList", emptyList);
ent.setProperty("hasNullList", hasNullList);

System.out.println("ENTITY - before: " + ent);

Key key = ds.put( ent );
ent = ds.get( key );

System.out.println("ENTITY - after: " + ent);

The output of the code is following:

ENTITY - before: <Entity [Test(no-id-yet)]:
nullProp = null
hasNullList = [null]
emptyList = []

ENTITY - after: <Entity [Test(1)]:
nullProp = null
hasNullList = [null]
emptyList = null

So it means, that GAE does not store an empty list at all ! Instead of an empty list, the property is null.

This is not so big surprise and is not against anything in the official documentation. But the problem is the following:

Can we filter 'null' and '[null]' values separately ?

We will answer quickly - just extend our first example by these lines:

System.out.println( "count(emptyList==null) = "
+ ds.prepare(
new Query("Test").addFilter("emptyList",
Query.FilterOperator.EQUAL, null)

System.out.println( "count(hasNullList==null) = "
+ ds.prepare(
new Query("Test").addFilter("hasNullList",
Query.FilterOperator.EQUAL, null)

And the output is:

count(emptyList==null) = 1
count(hasNullList==null) = 1

This means that it is not possible (this way) to distinguish between null and [null] properties!

So the summary is: do not store nulls into Lists, otherwise you will get into a problem in future.

Wednesday, February 17, 2010

Large Response in AJAX XMLHttpRequest

Sometimes happens that you need to send large data in the response of the XMLHttpRequest (XHR). For example you need to fetch a sorce code and display it somewhere in the document - exactly as I needed.

The code was as follows:

if (window.XMLHttpRequest) {
xhr=new XMLHttpRequest();
else {
xhr=new ActiveXObject("Microsoft.XMLHTTP");


// a bug is somewhere here:

Unfortunately this worked fine in Internet Explorer (IE). In FireFox (FF) it worked if the size of the code was less than certain limit. The limit in my FF 3.0 was 4096 bytes.

So the solution which works ion both IE and FF was the following:

cnt = xmlDoc.getElementsByTagName('code')[0].childNodes;

code = cnt[0].nodeValue;
for (i=1; i < cnt.length; i++) code += cnt[j].nodeValue;

So you do not need to create chunks on the server, just concatenate what FF split on the client.

Monday, February 15, 2010

XSLT Using Xalan on Google App Engine

The online DAO generator at is running on the Google App Engine. The DAO generator uses XSLT transformation and Xalan is used as the XSLT implementation.

When I tested it locally using the dev appserver, everything worked fine. But after I uploaded it to the real Google App Engine, then strange things started to happen - the XSL transformations were somehow cut, unfinished, simply different than I expected. When I looked into the logs, I found why:

javax.xml.transform.TransformerException: Failed calling setMethod method
at org.apache.xalan.processor.StylesheetHandler.error(
at org.apache.xalan.processor.StylesheetHandler.error(
at org.apache.xalan.processor.XSLTAttributeDef.setAttrValue(
Caused by: java.lang.IllegalAccessException: Class$3
can not access a member of class org.apache.xalan.processor.ProcessorOutputElem with
modifiers "public"
at sun.reflect.Reflection.ensureMemberAccess(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at Method)
at org.apache.xalan.processor.XSLTAttributeDef.setAttrValue(

The problem was only with the XSL <xsl:output ...> tag. When I removed it, everything was OK. It was a surprise for me, because the reflection used for this tag is also used for other XSL tags and there were no errors there - just only with the <xsl:output...> tag.

Since I relied in Xalan (I have a bad experience using more than one implementation in one project), I had to patch it myself. The patch just calls the implementation Java method directly - without reflection. And it works now ! The patch you can find here (post 21).

StackOverflowError - Including a non-JSP Page

This happened to me when developing an application on Google App Engine (GAE SDK 1.3.1).

I tried to include a non-JSP page using standard <jsp:include page="/resources/news.xml"/> tag. There was no problem when testing on the local dev appserver (I only had to specify that the "news.xml" is a resource file in the appengine-web.xml). But when it was running on the production - real - GAE, I got the following error:

at java.lang.String.startsWith(Unknown Source)
at org.mortbay.jetty.servlet.Dispatcher$IncludeAttributes.setAttribute(
at org.mortbay.jetty.servlet.Dispatcher$IncludeAttributes.removeAttribute(
at org.mortbay.jetty.servlet.Dispatcher$IncludeAttributes.setAttribute(
at org.mortbay.jetty.servlet.Dispatcher$IncludeAttributes.removeAttribute(
at org.mortbay.jetty.servlet.Dispatcher$IncludeAttributes.setAttribute(
at org.mortbay.jetty.servlet.Dispatcher$IncludeAttributes.removeAttribute(

When I renamed the "news.xml" to "news.jsp", then everything worked fine. Is it a bug or feature ?