CFQueryReader Update: Support for ExtJS 5 and ColdFusion 11

Due to a donation from Bruce Lomasky (thanks Bruce), I've written a new CFQueryReader for ExtJS 5. CFQueryReader is an Open Source project, available on GitHub, that I began years ago to create a proper reader for parsing Adobe's native ColdFusion query JSON return format into something usable by ExtJS data stores. As ExtJS has changed over the years, I have updated the reader to accommodate new versions. But, I don't do much ExtJS development anymore. It's a great front-end app development framework, but I've changed jobs a few times since co-authoring the first books on the library, and just don't work with the technology on a day to day basis anymore. Bruce had a need, and offered to pay for the development of an updated reader. So I dove back in to the ExtJS pool, to help him fulfill his requirements and put a new reader out there.

ExtJS went through a major rewrite between versions 3 and 4, requiring an entirely new direction. While much is backward compatible, in the move from version 4 to version 5, there were some big changes to ExtJS' data packages. Sencha has always done a good job with creating a rich, dynamic framework for web application development. I have always been impressed with their commitment to improving the product, continuously modernizing the framework, and creating more of a "software craftsmanship" approach to web application development. That said they haven't always done such a great job with ensuring an easy upgrade path for developers. Changes to their data package completely broke some of the use cases of CFQueryReader, requiring some refactoring to accomadate.

And that's OK. Sencha's changes to their data packages are some welcome changes, especially for our little plugin. In all of the past revisions of CFQueryReader, we've extended a base Array Reader class, and written an override method of the core readRecords() method. While this worked, it was really kinda kludgy. What we really needed was a way to transform the incoming data packet prior to the reader really processing the object. With ExtJS, we now have that ability to do just that.

ExtJS 5 introduced a new configuration option for data readers: transform. This new configuration attribute takes a function within which you can manipulate the return data of a store's ajax request prior to the reader actually processing that data. This gives some underlying flexibility that wasn't really there before, especially when using Ext.Direct, but for now you really just need to know the basics.

ColdFusion upper cases object keys when it serializes to JSON. If you are manually creating the structure, there are ways to fix that:

view plain print about
1var myStruct = {
2 "key1": "some value,
3 "Key2": "this one is cased differently",
4 "differentKey3": "this one is more different"
5 };

This is fine, and helps a little bit, but ColdFusion creates an object when it serializes a ColdFusion query object, and you can't control it's casing:

view plain print about
1ColdFusionJson = {
3 DATA: [
4 ["some value","this one is cased differently","this one is more different"]
5 ]
6 };

When you build your reader configurations, we no longer worry about this casing, as we will do a case insensitive matching of keys during the pre-process:

view plain print about
1reader: {
2 type: "cfquery",
3 rootProperty: "myQuery",
4 totalProperty: "totalCount"
5 }

NOTE: We will lowercase column names during preprocess, and you should remember that when creating your "dataIndex" attributes of your Model's Field configurations.

CFQueryReader will read ColdFusion query JSON serializations that are:

  1. The root of your return
  2. Nested within a larger return object (struct)
  3. Have been converted with ColdFusion QueryForGrid (not suggested, but supported)

And, because ColdFusion 11 includes the new "struct" type for the queryFormat parameter (even as an argument of your ajax request), CFQueryReader will properly parse that as well.

And, since we can now process the return prior to ExtJS calling it's internal readRecords() method, CFQueryReader no longer extends as it's base class, but the more appropriate instead, allowing for a more native access approach under the hood.

I've made the new plugin available, for now, within the Ext5 branch of the repository. If you have the need, take it for a spin, and let me know if there's anything that might require tweaking.

ColdFusion UI: cfwindow

Not sure why I forgot to blog this, but a few weeks back Ray Camden told everyone that he wasn't going to tell you to stop using ColdFusion UI tags anymore. Instead, he and Adam Cameron started the ColdFusion UI the Right Way project. This project is a collection of articles and demo assets to show people that you can use other methods for displaying these dynamic client-side "widgets" in your applications, without using the CFUI tags.

Why? Well, it was a nice concept, but honestly (and in my continuing opinion) ColdFusion doesn't need to do UI. Yes, cfoutput, and looping over queries is fine, but tags like <cfform>, <cfpod>, and <cfwindow> are the devil's work. There have been long running issues with letting ColdFusion write your HTML, CSS and JavaScript for you, the chief among them being that they're bloated, buggy, limited, and (ultimately) unnecessary. You have a much finer degree of control over your application, and how it functions, when you write it yourself.

The CFUI tags were provided to allow the lowest common denominator (i.e. someone who can't really write code) to crank out something that works. They are not intended for skilled developers who write web sites and applications for a living.

And so, Rey kicked off the project with an article on replacing <cftootip> with the jQueryUI tooltip plugin, and I wrote the next article on replacing <cfwindow> with Bootstrap's modal plugin. I just checked the repository, and now there are also articles for <cfajaxproxy> and <cfdiv>.

The idea of the project is to show you a) that it's possible to do these things without the CFUI tags, and b) show you examples using a variety of libraries, to reiterate that you aren't limited to a single option.

CFQueryReader v2.1: Now with metaData support

The Sencha guys just keep upping the bar, and Ext JS 4.1 is no exception. I've been reading Loiane Groner's Ext JS 4 First Look, to review it, and continually find new, cool stuff. 4.1, however, takes it even further. While upgrading CFQueryReader, I was working with 4.1 RC2. And, while extending the base classes, I came across a new feature that wasn't fully documented yet: adding metaData to a server-side store response for changing configuration on the fly. Sometimes it would be nice to just...change up. Now that 4.1 is fully released, I had to make sure that this worked in CFQueryReader.

So, last night I sat down and hammered out this functionality. It took a lot more than I realised, and I learned a lot more about the Ext JS internal code, but I think CFQueryReader is better for it. Consider the following ColdFusion method:

view plain print about
2 *    FUNCTION getWithMeta
3 *    This function returns the ColdFusion Query object as part of a struct object.
4 *
5 *    @access remote
6 *    @returnType struct
7 *    @output false
8 */

9function getWithMeta(numeric pageIndex = 1, numeric pageSize = 50, string sort = "", string search = "") {
10    var retVal = {"success" = true, "pageIndex" = ARGUMENTS.pageIndex, "pageCount" = 0, "recordCount" = 0, "message" = "", "getEntries" = "", "metaData" = {"root" = "getEntries", "totalProperty" = "recordCount", "successProperty" = "success", "messageProperty" = "message", "idProperty" = "id", "fields" = []}};
11    StructAppend(LOCAL.retVal, GetEntries(argumentCollection: ARGUMENTS), true);
12    var colArr = ListToArray(LOCAL.retVal.getEntries.columnList);
13    LOCAL.retVal.metaData.fields = [
14        {"name" = "id", "type" = "string", "mapping" = JavaCast("int",0)},
15        {"name" = "title", "type" = "string", "mapping" = JavaCast("int",3)},
16        {"name" = "posted", "type" = "date", "mapping" = JavaCast("int",2)},
17        {"name" = "views", "type" = "int", "mapping" = JavaCast("int",1)}
18    ];
19    return LOCAL.retVal;

I reused my getEntries method, to get my query. Here, I'm creating a metaData object, at the root of the return, to define the dataset. Configuration options that I normally define client side (root, totalProperty, etc) I put in to the metaData key. When the response is received by the client, the reader will pass this metaData in to our app, applying this configuration to our reader, store, model, and so on. In the above method, we let the metaData map our columns to fields, rather than doing it client side (CFQueryReader will automatically skip the column mapping if metaData.fields is present in the response.) Our client side store might now look like this:

view plain print about
1Ext.create('', {
2    storeId: 'entryStore',
3    model: 'Entry',
4    remoteSort: true,
5    proxy: {
6        type: 'ajax',
7        url: '/com/cc/Blog/Entries.cfc',
8        extraParams: {
9            returnFormat: 'json',
10            method: 'getWithMeta'
11        },
12        limitParam: 'pageSize',
13        pageParam: 'pageIndex',
14        sortParam: 'sort',
15        reader: {
16            type: 'cfquery'
17        }
18    },
19    autoLoad: true

A full example of this, in action, can be seen on a demo page of the CFQueryReader site. The full source code, of the example, can be found in the CFQueryReader GitHub repository.

ColdFusion Query Column Order: Did You Know?

Now first let me say that I've only seen this behavior on ColdFusion 9.01 with Cumulative Hotfix 2. My ColdFusion 10 VM blew up this morning, and I haven't had 7 or 8 for a while now, so someone else might have to verify this behavior on those platforms for me.

This morning I was testing out a new feature of Ext JS 4.1, and ran into something interesting. One of the reasons that I wrote CFQueryReader, as well as serializeCFJSON, was because of the way that ColdFusion handles it's JSON conversions of it's native query object.

view plain print about
2    "COLUMNS":["ID","NAME","EMAIL"],
3    "DATA":[
4        [1,"Ed Spencer",""],
5        [2,"Abe Elias",""],
6        [3,"Cutter","no@address.giv"]
7    ]

I've written before, about how Adobe's dataset serialization is much trimmer, but many client-side libraries and plugins expect a more standardized format.

view plain print about
2 "users": [
3 {
4 "id": 1,
5 "name": "Ed Spencer",
6 "email": ""
7 },
8 {
9 "id": 2,
10 "name": "Abe Elias",
11 "email": ""
12 }
13 ]

It's not too difficult for us, client-side, to map the column to the proper data position in the record array. In fact, that's basically what CFQueryReader and serializeCFJSON both do. But why is it even necessary? Most of these client-side libraries allow us to write mappings in configurations anyway, so why write something else to do it?

Well, there's two reasons really. First of all, the client-side developer may not have that much insight into the server-side developer's code (they can be different people), and would probably be unaware of API changes. The second reason? Well, if you've used ColdFusion for a while, you might know that there was a time when you could not depend on the column order always being the same. At one point, the columns were returned in alphabetical order. This would make it really hard to figure the position out prior to run time, unless you have intimate knowledge of every change to your API as it happens.

It appears that this isn't as much of a problem today. Kind of. I'll run this query:

view plain print about
3        posted,
4        title
5FROM    tblblogentries
6WHERE 0 = 0";
7LOCAL.q = new Query(sql = LOCAL.sql, datasource = VARIABLES.dsn);
8LOCAL.retVal.getEntries = LOCAL.q.execute().getResult();

If you dump the results to the page, you'll see the following:

Now, server-side, ever query comes with a variable we can use to reference the columns used in a query. I'll use ColdFusion's ArrayToList(LOCAL.q.columnList) method to dump the query's columnList property to the page:

Look closely and you'll see that both appear to be in alphabetical order. The query dump and the column list showed them in alphabetical order. That said, we then look at the JSON return:

Wait a minute? Now the columns appear in the same order that they are in the query. That's a change.Let's change the column order in the query, and see if it changes anything.

view plain print about
2        title,
3        posted,
5FROM    tblblogentries
6WHERE 0 = 0";

Notice that, again, the column order matches that of the order it was used in the query. Running a cfdump of the query, and it's columnList, still show alphabetical order. Why is it different? Why does this matter? Well, I'm going to show you that in a follow-up post (on Ext JS 4.1 and ColdFusion), but in the mean time it's a good thing to know, that the JSON returnFormat will give you query column orders identical to that used in the query. And, as I said above, somebody please verify this for me on other versions of the platform.

JQuery Plugin: serializeCFJSON

Quick note about my serializeCFJSON project out on GitHub. I wrote this quick JQuery plugin to convert ColdFusion's JSON representation of it's native query objects. ColdFusion represents datasets in a trim manner, as an object containing two arrays: One, an array of column names, and the other, an array of arrays, each representing one record of the set. Most representations of recordsets are an array of objects, each record represented as a collection of name/value pairs. ColdFusion's representation is much smaller, removing a lot of unnecessary duplication, but many pre-built frameworks and plugins look for the name/value pair objects.

What the plugin does is convert ColdFusion's representation into the more standard form. Ajax transfer is unaffected, as the trimmer format is still being passed. What we get is some very minor client-side overhead in the creation of a new object. What's more, the plugin recursively searches through a JSON object and converts any ColdFusion query that it finds. So, if you nest your query inside a larger object, the plugin will still convert it for you.

I put up a demo in the Projects menu, at the top of this site. This is a refactor of my Grouped jqGrid example, using the plugin for the data conversion. The full sample code for the demo can be found in the GitHub repository.

Take her out for a spin, and let me know what you think.

Intro to jqGrid Part 6: The Action Column

So far we've covered the base grid configuration, populating the grid with remote data, controlling layout by refining the column model, and given an example of event handling by binding a search form to our grid. This covers most of what the average developer might need to know about working with jqGrid, but I've been promising to tell you how to work with the icons we setup in our action column.


ColdFusion JSON Serialization Changes

Because I've had to search on this three times in the last year, I thought it'd be a good idea to document this here, for myself and others. There were a number of important changes that can trip you up from version to version and, if you know which version the changes came in it can be beneficial.

First is a change that affects JSON output, though it's not specifically about JSON. The ColdFusion 9.01 updater (the current base install, at the time of this post) addressed Bug 82980 by removing ColdFusion Debug Output from direct CFC requests. This was only important if you had debugging output turned on in the Administrator, but caused lots of issues when testing ajax applications in development environments that kept that setting on by default.

Unfortunately, Adobe made some changes to JSON serialization that were not only unpopular, but technically dead wrong. They were trying to resolve the treatment of numbers in JSON serialization. Numbers were being converted to floats (1 became 1.0, 12 became 12.0, and so forth), so Adobe changed that by converting all numbers into strings (11 became "11", 17.24 became "17.24"). This was wrong too, as numbers weren't numeric anymore. At the time it raised quite a stir.

They fixed this (Bug 83638) really quickly, releasingCumulative Hotfix 1 for ColdFusion 9.01, which fixed the issue of integers being converted to float and having them as numeric values (ie: [1,12,true,"this var",17.24]). Hurray! Except....

Enter the improperly titled Cumulative Hotfix 2 for ColdFusion 9.01. I say this because it really isn't 'cumulative'. Most things are straight, from a cumulative standpoint, with the seeming exception of the fix to JSON formatting. If you have not applied CHF 1 prior to installing CHF 2, then you will still have issues with JSON formatting.

Overall, Hotfix 2 is great, fixing many bugs. The install is crazy/scary, and must be followed to the letter to prevent major issues, but it's worth it. As long as you did apply CHF 1 first, that is. Thankfully the hotfix/upgrade install process is much better in ColdFusion 10.

Big thanks to Ray for helping me to realize that I wasn't crazy, and pointing me in the right directions for all of this info. Hopefully this helps someone down the line.

The Joys of Developing for Internet Explorer

Note: Follow the madness here to it's conclusion, to discover yet another "Really? I didn't know that..." IE moment, that may save you heartache and pain.

Is the sarcasm evident in my title? It should be. While Internet Explorer may have been "groundbreaking" when it was released, it has ever been the bain of the web developer's existance. Why, you ask? Because it refuses to adhere to standards, and just operates differently than everyone else. To be fair, Internet Explorer 10 is in the works, and supposedly closes the gap a fair amount (and even wins out in some of the html 5/css 3 support), and 7, 8 and 9 did progressively improve (if slowly). Unfortunately the fact remains that some companies/organizations/governments are stuck on Internet Explorer 6, or maybe even 7. If you are a web developer, and write cross-browser web applications, chances are Internet Explorer has hit you more than once in your career.

It's the dumbest things that hit you too. The most obscure "WTF!?!" moments that drive you crazy. That is a daily experience for me now.


Anatomy of a Shopping Cart: A Usability Study

This little writeup is a usability study of cart layout and process in general. So many apps today are still sporting the 1999 click-and-reload interface, and times have changed. Users are tired of the old way, embracing the new, and if you're behind the times you could be losing clients fast.

Shopping Carts are funny, terrible things. The less intuitive they are, the higher the abandon rate, and yet you have to pack a ton of stuff in there. The trick is to anticipate user workflow, and match that as much as possible, without making the user work for it too much. This is the whole precept behind Don't Make Me Think.

First things first, create a method for detecting JavaScript prior to a user ever coming to the shopping cart, as it redefines the experience for the user entirely. If JavaScript isn't present, then it's either a bot, some non-Class A mobile device, or less than 1% of the desktop user. Surfing the web without JavaScript is even less of a chance than a user surfing without the Flash player. There's just too much of a user experience that is lost without it, and most standard users don't have a clue how to disable it to begin with.

You end up with two separate cart processes, one with JavaScript, and one without. "Two processes?" Well, you write a process with graceful degradation, that can do what needs to be done in either scenario, and with code reusability in mind. This writeup will go over the user with JavaScript enabled.

I've mocked up a simple, yet effective, cart design. This is just a wireframe model, but there's nothing here that can't be done with HTML and CSS. A user without JavaScript would have to step through the stages of placing an order, screen by screen. A user with JavaScript has to step through as well, but through JavaScript and Ajax, this user does most of their work right from the cart.

Below is the mockup, with an explanation of the different stages of process to follow:

This is pretty loose. Balsamiq only allows you to do so much, so color and highlights and icons give things a lot more pizazz, but this should give you a good idea. If you're unfamiliar with Balsamiq, it's a great tool for doing simple "pen on napkin" wireframe layouts during interface design. It's also a great way to draft quick mockups for clients without spending hours of unpaid time, and even more hours on revisions.

From the top down:

The only button at the top is the "Keep Shopping" button. A user has to verify all the details prior to checking out, so the "Checkout" button is at the bottom of the form. The "Clear" button is also at the bottom, giving the user a second chance not to clear out. A second "Keep Shopping" button is at the bottom of the page, to encourage the user to buy more.

Next up is the cart contents. A user, coming to the cart, wants to know what's in it first. This is the user's main focus, so registering for the site or entering credit details and stuff always comes after the cart itself.

There are a few different ways to handle actions on cart items. Here I show the 'Remove' checkbox. If a user hits the box, take the item out. Don't prompt 'Are you sure?' This annoys users. Of course they're sure, or they wouldn't have checked the box. Another way of handling 'Remove' is part of an action link list. There are already two other action links showing (Gift Wrap and Add to Registry), it would be easy to adjust the layout to use all action links instead (Remove at the top, Add to Wishlist, Add to Registry, with Gift Wrap last). Here's a tip though: Any of these actions will ultimately remove the item from the cart.

Being tabular in nature, it's best to use a table here. Unwritten rules of cart usability: the 'Item' is listed first, unless something small, like a checkbox, radio button, or icon precedes it. The cost (Qty * Price) is always last. This allows us to build a columnar layout of the money details. If you have line item discounts, you may show the math here too, to emphasize what the client is getting. In this mockup, it's all shown below.

Next, show the user some love. Discounts, Promos, Gift Cards and Coupons should follow the cart. 1) it makes it easier to calculate the Subtotal and 2) it gives the user a warm fuzzy to watch their Total decrease. If there were discounts on items, pull them out here (if you didn't do it above). Don't make a user refresh the page to get a new Total. Gift Cards and Coupons? Use Ajax to verify their validity and value, and apply it to the tally block right then.

If addresses aren't pre-populated, then the user hasn't logged in or registered or set up addresses yet. Our action depends on which scenario is the case. If the user hasn't logged in or registered, then we show them a login box, with a register button. If they login, we go ahead with the rest of the order process. If they must register, handle that registration in a modal popup and then update your area. If the system allows for anonymous users (those who don't want, or have, to register with the site), or a registered user hasn't setup addresses yet, then we display "add address" links to the display that can load modal forms to collect the necessary details.

You can't tally Tax unless you know you need to collect it. If you don't have an address yet, don't show the Tax line item. If you get an address, and tax does need to be applied, add it in right away, and highlight it, and the Total change, so there's no surprises.

And, once you have an address, then you already know the zip/postal code the order is going to. Once a Shipping Method is selected we can lookup the shipping cost via Ajax. And, if the purchase is a download, then there's no reason to show them the Shipping line item at all. Avoid the "Estimated Shipping" thing, just give them the info once you have it. If need be, show them text ("We need your address and Shipping Method to show Shipping Cost") to help push them through the order process.

Last is the Payment Method. When a user selects the Payment Method, then the input fields for this area should change to the appropriate fields for that method. Don't show them fields for every possible option, just those for the option they've chosen.

That's it. By streamlining your Cart and Checkout process to match user workflow, you should see a lower abandon rate and higher sales. There's a lot of code involved, and both client-side and server-side data validation is a must, for security purposes, but the end gain makes the effort more than worthwhile. What are your suggestions for improving Cart usability? Give me your feedback below.

2011 In Review, and the View for 2012

My, how time flies when you're having fun! It seems like only yesterday that I was welcoming in 2011, and now we're here a year later. So many things have happened in the last year, and rereading that post I see that I missed some things I should've done, but let's take a look in retrospect.

I wrote 27 blog posts in 2011. This is nothing, compared to guys like Ray Camden or Ben Nadel, but for me it was quite a bit, especially when you consider that between March and August I released only one post. Very early in the year, I began a series on creatingmany sites with one codebase. In the process, the series has evolved to contain a fairly detailed primer in ColdFusion application architecture (because of it's importance to this process), has currently spanned 8 separate posts, and was even referenced by Sean Corfield in his great presentations on the same topic. 2012 will see the completion of that CF app discussion, and gradually move it back to the MSOC topic itself, as there is still a ton to talk about there, and a lot of interest in the topic. I also began a series on the jqGrid JQuery plugin. jqGrid is another Data Grid visualization tool (I have now written about three, including Ext JS and DataTables), and is a clear choice for those who must use JQuery. (To be fair, JQueryUI is working on a grid component, but they are still behind the curve, and way behind Sencha.) Finally, one common thread seen in the majority of my posts, is how much I've embraced cfscript. I wrote a lot of things, on a variety of topics, but most of my code examples were pure scripted examples.

Now let's talk about some other departures from the norm for Cutter.

You did not see a lot of content around Ext JS. In fact, I stopped writing Ext JS books. This is not, in any way, a reflection on my feelings for Ext JS. I still believe that Sencha has built one of the best client-side libraries for web application development. In evaluating the overall ROI, I realized that I was writing more for the community than the money, and that my reach was greater through my blog, while giving me flexibility on when and what I deliver from a content standpoint. That said, I didn't have a single project this year that used Ext JS, so had very little time to experiment and write about it. This year, I'm going to expand on a personal project, and get back to some great Ext JS content for my readers.

You, also, did not see me speak at any conferences this past year. Nor at any user group meetings. This wasn't because I didn't want to, but because of some more personal reasons. I'm not going to go in depth here, other than to say that I've had some long standing health issues that required me to have some surgery done on my mouth. (Mark Drew is making a joke right now...) Aside from the fact that this has been very costly (chewing up any conference/travel budget), it also meant that my speech has been affected for a good part of the year. Thankfully this experience is (mostly) over now, and I hope to get back to presenting sometime this year. Any user group looking for a speaker this year, please contact me through the Contact link on this blog.

One group I am hoping to speak to this year is the Northeast Florida CFUG. I have to call Mike back, but he's looking to get things kicked off again, and I want to help it be successful. If you're in or around the Jacksonville area, make sure to keep an eye on the site for upcoming events.

One other thing I'm looking to do is to migrate all of my projects into GitHub. I've been using Git at work, and I am loving it, and I think combining GitHub with RIAForge is a great way to promote the terrific technologies we work with every day. I will make the time, I promise.

This comes to the final discussion of this post, Adobe. I again had the pleasure of being an Adobe Community Professional this past year. Due to my health issues, I didn't get to do everything I would've wanted to this year, but I've tried to be a good supporter. There are some fabulous things coming in ColdFusion Zeus and, by extension, to ColdFusion Builder as well. There has been a lot of hub-bub over Adobe's communications flubs regarding Flash, mobile, and Flex. I've avoided much of the discussion, other than to say "be patient and watch". Flash isn't going away, and neither is Flex. HTML 5 is a beautiful thing, if you aren't developing desktop browser applications (i.e. You're only writing for mobile/tablet development). There, that is my whole contribution to that discussion. Give it a rest.

2012 will be a fantastic year. Set yourself some clear, definable goals. Break them down, step by step, and write the steps down on paper. Each successive step, print out in large letters and place it somewhere where you will see it each and every day. Set yourself up to succeed, and you will. Have a great year, everyone, and I can't wait to hear what you have planned for 2012.

More Entries