Request vs Variables - Which is right?

Last week I stirred the pot a bit, stating that we've been improperly using the VARIABLES scope, and it's hurting our applications. Variables placed within the VARIABLES scope do not appear to have proper Garbage Collection performed on them when we're done with them, except in the case of the scope's usage within a CFC or a custom tag. This brought up some interesting discussion, where Ray Camden, Mike Brunt, and others chimed in.

I, with very good success, have been using REQUEST where most people use the VARIABLES scope. Wait! Hold on! You have to let me finish first before you get bent. What I said was that, in the past, we placed variables into the VARIABLES scope within our base request templates, with no way of clearing those variables at the end of the request (if using Application.cfc). My thought is, if VARIABLES have no true (apparent) mechanism for release, in context of variables scoped this way in a base request, then they don't really belong in the VARIABLES scope, but rather in the REQUEST scope.

This was met with some push back. The first argument was that this breaks encapsulation. I want to begin my response here by saying the first obvious thing that comes to mind: ColdFusion is not OO! Sure, your ColdFusion application can be written in an OO fashion, but ultimately it doesn't have to be. There are even some who seem that think that maybe it shouldn't be. All of that being said, I agree with OO principles, and think that proper encapsulation is very important when developing your applications, especially in terms of portability and reuse. Just because the REQUEST scope can be used directly within your custom tags or CFC's doesn't mean that you should. I would never suggest referencing the REQUEST scope directly within a CFC, with the exception of Application.cfc itself.

So another argument was that the REQUEST scope should only be used "when (and only when) the variable must be accessible to all the elements composing the request...", going further to state that the REQUEST scope should be avoided if at all possible. Why? The rest of the argument once again comes back to encapsulation. My point is to use the REQUEST scope within the context of a request, paying attention to maintain encapsulation. Within that context, doesn't it make sense to use the REQUEST scope for variables that should only exist for the length of a request?

The ColdFusion Jedi piped in to correct me on my timeline of scope introductions to ColdFusion, and we had some back and forth over declaration of scope to clearing of scope. I contended that onApplicationEnd() cleared the APPLICATION scope, that onSessiondEnd() cleared the SESSION scope, and that onRequestEnd() cleared the REQUEST scope. Ray reminded me that Application.cfc was about application process, and that the server handled when to 'clear' variables 'under the hood.' My point back was that something has to tell the server when to de-reference these variables, and that I had always assumed that was triggered by the execution of these methods. Mike Brunt chimed in on this, saying that the application must de-reference the variables prior to the server knowing it can perform Garbage Collection on them (Side Note: he didn't come out and agree with all of my conclusions here, only seeming to agree with the logic applied.)

Possibly the largest argument overall relates to whether the apparent memory leak, when using the VARIABLES scope within a base request, is really a leak or just a byproduct? I've theorized that the intention of the VARIABLES scope was that it was to be used only within CFC's and custom tags, where it does appear to properly get de-referenced. I put out there that the only reason you were able to use VARIABLES within the base request was because custom tags were created from .cfm files as well, so there really isn't any way to restrict it's usage. Ray says that the VARIABLES scope was introduced to provide access (like StructKeyExists()) to the unscoped variables out there in the wild.I think our thoughts of 'best practice' usage should follow these lines: Using the REQUEST scope within the base request, while maintaining encapsulation.

Ultimately, I just don't know. Maybe there's still an Adobe engineer out there who goes back that far, and can remember exactly what's what and why. I do know that, after migrating several high traffic applications to these guidelines I did get a higher return on application performance. I've gotten notes from others, after reading the original post, who have done similar changes to their systems with the same improvements. What other pluses or minuses do you, dear reader, think of when considering this practice of variable scoping? Is it right? Is it wrong? Is it a sin? What do you say?

(You can follow the original comment thread by referencing the post linked below)

Scope Usage and Application Scaling

Mike Brunt has posted an excellent article called Good Practices For Scaling ColdFusion Applications, in which he speaks about the importance of memory management within ColdFusion applications, and calls out certain avenues of scope usage that every CF developer should pay attention too.

Great post Mike! Another 'scope' issue that I've often seen (and abused myself) is the misuse of the VARIABLES scope. Back in the early days of CF, every example you saw showed an unscoped variable, placing it into the VARIABLES scope by default. This was fine, back then, as your view layers primarily consisted of Custom Tags. Use of the VARIABLES scope, within a custom tag, was permissable as those variables were then only available within that tag (or within a nested tag, if the nested tag used the CALLER scope). Once execution of the tag was completed, the memory space of those variables was cleared.

But, since custom tags were made from .cfm templates, there is no constraint within CFML from placing VARIABLES scoped variables within any .cfm template. This, in itself, is bad, as the VARIABLES scope (in the case of a base template, not a custom tag) is not properly cleared from memory space.

What I find people doing (and I've done it myself) is using the VARIABLES scope in place of the REQUEST scope. Memory allocated to REQUEST variables is properly recovered at the end of a request. Although variables in the VARIABLES scope are no longer available (except within a persistent CFC), their memory isn't properly released at the end of request execution. (For more information on this, see Jason Sheedy's post on the ColdFusion Memory Leak, and the accompanying comment thread and other referenced posts.)

A technique that some used, to get around this issue, was to have a StructClear(VARIABLES) statement as the last line of their onRequestEnd.cfm. This would explicitly clear out the VARIABLES scope at the end of a request. But, with the advent of Application.cfc, this no longer became an option (if you use Application.cfc.) If you tried to use that statement within the onRequestEnd() method of Application.cfc, you would only be clearing the VARIABLES scope of the CFC itself, and not the request.

That's actually a good pointer at the true underlying issue: proper scope usage. Use of the VARIABLES scope within a CFC makes variables within that scope only available directly by all methods of that CFC. Use of the VARIABLES scope within a custom tag makes variables within that scope only available within that tag (or a nested tag using the CALLER scope to access them.) Within a custom tag, you can not access variables of the calling template's VARIABLES scope, unless you use the CALLER scope to do so (which should only be in the case of a nested custom tag). You can, however, access REQUEST variables through out the request, from within a custom tag, an include file, or even within a CFC (though I wouldn't suggest doing that one.) By confining the usage of the VARIABLES scope to within custom tags and CFC's, and properly setting request level variables within the REQUEST scope, you could better manage your memory usage and application performance.

I've personally seen major differences in application performance, both through Mike's advice on persistent scope usage, as well as what I've outlined above. Unlike Jason Sheedy though, I'm not very familiar with Java profiling tools, so I'll have to leave it up to someone else to test all of this for the scientific proof. I only know that it appears my applications are running better as I migrate them to this paradigm. What are your thoughts?

*** Post Revision 080609: 2230 hrs *** More revision to my comments than the post itself, but Ray, the ColdFusion Jedi himself, corrected me. According to Ray, the VARIABLES scope wasn't added until CFMX, so my supposition on the 'intention' of the VARIABLES scope was incorrect. You really have to read the comment thread to see the fun we're having;) Hopefully we'll find an answer. Cutter

CFQueryReader v1.2 - Critical Update Supporting ExtJS 3.x

I have updated CFQueryReader, addressing issues that had arisen with new builds of ExtJs 3.x. This new build should cover sporadic issues with loading a new Ext.data.Store. There is also a simple example of using Aaron Conran's DirectCFM Ext.Direct ColdFusion API stack.

The CFQueryReader Example Site has been updated as well. You can update CFQueryReader from RIAForge.

Ext Js 3.0 is Finally Released!

Yes, Ext Js 3.0 has finally arrived! This long awaited update to the popular library has finally hit the download page as a production ready build (though the Release Candidates have been pretty stable as it is). There are many great enhancements to Ext, including an even more consistent underlying model (how could it get more consistent?), and some exciting new data marshalling features.

A quick perusal of the updated Samples & Demos page gives us immediate insight into some of the new features that are available:

There's a lot more that you'll have to dig to see, like improved browser support, a better container model, and (experimental) ARIA support (for accessabiltiy). Some of the greatest enhancements come in the way of the data marshalling capabilities added via the new Ext Direct. With Direct, Ext is providing the remoting specifications so that anyone can write data marshalling services around their favorite backend language. Ext has even published Example Server Side Stacks as a jumping off point to beginning with data marshalling via Direct. [Side Note: Aaron Conran, the team lead on the Ext Js team, is a long time ColdFusion guy, and he wrote the example CF stack.) By configuring your Direct API, you can utilize data readers and writers (they're new!) easily, even passing multiple requests within a single Ajax request. [Another Side Note: CFQueryReader is fully functional with and without Direct.]

One of the nicest features of this release is the backwards compatability. There are little to few changes that most will have to make, to upgrade their applications from 2.x to 3.0. And, it was announced, on a recent User Group Tour stop, that Adobe is including Ext Js 3.0 in ColdFusion 9. This opens up the possability of some very nice, new CFAjax components to come.

All in all a fantastic release. I've had the opportunity to play with 3.0 for a while now, watching the SVN updates daily, and my hat's off to the Ext Js crew for another excellent release.

ColdFusion PreRelease Tour Coming to Nashville

NCFUG CF Tour

Greg Wilson, Adobe Platform Evangelist, is coming to Nashville to give us a sneak peak at ColdFusion 9, Codename: Centaur, and Bolt, the new ColdFusion IDE. According to various posts from cf.Objective, and other tour dates, there appear to be some major enhancements in integration capabilities, Flex and AIR interaction, and more. You don't want to miss out on this special event, giving you the inside scoop!

Click on the link above to register.

Placement of the PagingToolbar on a Grid

This morning I found I was courtesy copied by Ray Camden, on a reply that he was making to a message he had received from his Blog Contact form from Mike Knox. Mike was trying to find out if it was possible to place the PagingToolbar, used in cfgrid, at the top of the grid rather than the bottom. Ray had told Mike that he had Googled it and not found on obvious solution, and that he was CCing me for my feedback.

Like any padawan, it's pretty humbling for me to be asked advice from the Jedi Master. I've been taking Ray's advice on ColdFusion since the 4.x days, so I try to be pretty diligent when he includes me in helping others with their queries.

I've always held that the use of the CF Ajax components are primarily for rapid application prototyping. They are fantastic for putting up some very basic functionality, but when you need more advanced configuration it is then time to dig in, and go straight to Ext JS itself. Mike's question is a prime example of this. We have nearly no control over a cfgrid's configuration prior to render, and the configuration options that we do have (through the cfgrid and cfgridcolumn attributes) barely scratch the surface to what one can do with an Ext Grid. The PagingToolbar is a class object of Ext and, as such, you should be able to add it to any toolbar of a grid: top, bottom, or both. I fired up Eclipse, booted up my CF and Apache, and pulled up the Ext examples from my local copy of the latest 3.0 download. All of the examples that you can see in the Samples & Demos section of the Ext site are given to you in the download of the library, so that you can run them locally and pick them apart. So, my quest was on.

I went to the Paging Grid Example, and examined the source code of the paging.js file. I quickly found the configuration object for the paging toolbar:

view plain print about
1var pagingBar = new Ext.PagingToolbar({
2 pageSize: 25,
3 store: store,
4 displayInfo: true,
5 displayMsg: 'Displaying topics {0} - {1} of {2}',
6 emptyMsg: "No topics to display",
7
8 items:[
9 '-', {
10 pressed: true,
11 enableToggle:true,
12 text: 'Show Preview',
13 cls: 'x-btn-text-icon details',
14 toggleHandler: function(btn, pressed){
15 var view = grid.getView();
16 view.showPreview = pressed;
17 view.refresh();
18 }
19 }]
20 });

The PagingToolbar instance is then applied to the Grid, placing the object in the 'bbar' (bottom bar) attribute:

view plain print about
1// paging bar on the bottom
2 bbar: pagingBar

After seeing this in the online demo, I went to my local (3.0) demo to see if I could change it. The Ext team rewrote the demo for the upcoming 3.0 release, placing the instance initialization directly in the attribute:

view plain print about
1// paging bar on the bottom
2 bbar: new Ext.PagingToolbar({
3 pageSize: 25,
4 store: store,
5 displayInfo: true,
6 displayMsg: 'Displaying topics {0} - {1} of {2}',
7 emptyMsg: "No topics to display",
8 items:[
9 '-', {
10 pressed: true,
11 enableToggle:true,
12 text: 'Show Preview',
13 cls: 'x-btn-text-icon details',
14 toggleHandler: function(btn, pressed){
15 var view = grid.getView();
16 view.showPreview = pressed;
17 view.refresh();
18 }
19 }]
20 })

So, I changed 'bbar' to 'tbar' (top bar) and reloaded the page. Success! The PagingToolbar was now in the top toolbar of the Grid, and fully functional. After reviewing all of this, I decided to go a step further. Could you have two separate PagingToolbar configs on the grid? One at the top and one at the bottom? So, I tried this:

view plain print about
1// paging bar on the bottom
2 bbar: new Ext.PagingToolbar({
3 pageSize: 25,
4 store: store,
5 displayInfo: true,
6 displayMsg: 'Displaying topics {0} - {1} of {2}',
7 emptyMsg: "No topics to display",
8 items:[
9 '-', {
10 pressed: true,
11 enableToggle:true,
12 text: 'Show Preview',
13 cls: 'x-btn-text-icon details',
14 toggleHandler: function(btn, pressed){
15 var view = grid.getView();
16 view.showPreview = pressed;
17 view.refresh();
18 }
19 }]
20 }),
21 // paging bar on the top
22 tbar: new Ext.PagingToolbar({
23 pageSize: 25,
24 store: store,
25 displayInfo: true,
26 displayMsg: 'Displaying topics {0} - {1} of {2}',
27 emptyMsg: "No topics to display",
28 items:[
29 '-', {
30 pressed: true,
31 enableToggle:true,
32 text: 'Show Preview',
33 cls: 'x-btn-text-icon details',
34 toggleHandler: function(btn, pressed){
35 var view = grid.getView();
36 view.showPreview = pressed;
37 view.refresh();
38 }
39 }]
40 })

That's part of what I love about development, I get to play, experiment, have some (geek) fun. After refreshing the page, I now had two toolbars, at both the top and bottom of the page. Both worked perfectly, and stayed in sync during paging. That's what I love about the Jack Slocum and the Ext Team, they think about things like having more than one paging bar and how they would need to stay in sync, and they've already written it in to the library.

And so, on rare occasion, the student has the opportunity to become the master, though I'm sure there's still plenty more I can learn from Master Camden;)

CFQueryReader v1.1

A while back, Justin Carter contacted me about some updates he was making to the ColdExt, a wrapper for the Ext JS library for ColdFusion. He had been looking at my CFQueryReader, as a reader for ColdFusion's JSON data return, and had noticed that there was no support for the 'root' attribute that's associated with most JSON readers in Ext. CFQueryReader is specifically designed for use with Query objects of ColdFusion 8's native JSON return type.

When I first wrote CFQueryReader, I was replacing the CFJsonReader with something better suited to the array structure used in the ColdFusion JSON return of a Query Object, and really did a one-for-one port. So, I took a deeper look at the issue Justin wanted resolved.

I did a deep comparison of Ext's ArrayReader and JsonReader source code, following everything that was being accomplished. What I found was how it used an internal accessor method to pull certain information on demand. I was able to refactor the reader to now support all of the base level meta attributes that are currently available to the ArrayReader: id, root, successProperty, and totalProperty.

In the previous version of CFQueryReader, when defining your fieldsets you were also required to uppercase the entire value of the column names. This is no longer a requirement either, although casing is important in identifying your 'root' and 'totalProperty' attributes.

The download below includes the reader, as well as some sample code showing how to implement it. I also setup these sample in a subdomain to show it working (requires Firefox with Firebug for full effect). There is a fair amount of comment documentation directly within the CFQueryReader.js file. A big shout out to Justin, Adam Bellas, and a few others, for helping me out with the testing.

Ext.Direct: Details on Data Marshalling

One of the core focuses, in the development of Ext JS 3.0, is the marshalling of data services under a centralized location. The intent is to make for a more portable application, where you can have a single touch point for data I/O that could easily be switched from one platform to another.

This is done through the new Ext.Direct package of classes, and many have been waiting on some detailed information on what it is and how to use it. This morning the Ext Blog got an update: Ext JS 3.0 - Remoting for Everyone. This explains Ext.Direct fairly well, even giving you a sample app to learn from, some sample code, links to some pre-constructed routers for several platforms, as well as a link to the Remoting Specification to write your own custom routers. Aaron Conran, Senior Software Architect and Ext Services Team Lead, wrote the ColdFusion Router. That's fitting, as Aaron's a long time ColdFusion guy, having contributed to the ColdBox project back in it's early days.

I'm excited about what Ext.Direct can mean for Ext - ColdFusion based applications, and I'll be deep diving this integration soon. I'll be curious to see how I can integrate CFQueryReader into the mix. I have an update to that which I hope to get out in the next few days.

Ext Conference: Day 1

Day 1 of the Ext JS Conference has been busy. Last night I had dinner and drinks with Jeff, a really nice guy who heads up development for a group writing web based interfaces to hardware appliances. Turns out Jeff is also the guy who introduced JavaScript to Douglas Crockford, who was today's keynote speaker. Douglas made sure to recognize Jeff at the beginning of his keynote, which I thought was a nice nod to the guy who helped give him direction in his career. Douglas ran us through a brief history of hypertext, the evolution of the web, JavaScipt, and the browser wars, before touching on the future of the web, some of the things in the new JavaScript specs being worked on through ECMA, and the fact that none of it would matter for years until the browser manufacturers would implement it. He did give props to Flash, AIR and SilverLight for trying to further the web experience where it has lagged so much over the years. He also spoke about the pitfalls of security in Web 2.0 applications, and the primary culprits of those issues (the browser manufacturers).

Before I go a whole lot further, let me tell you that the conference center, and their service to us, is fantastic. Held at the Ritz-Carlton Orlando, the location is beautiful with it's palm lined drives, a well groomed golf course winding around several 'water traps', and a huge pool with several fountains. The food has been exceptional, with an incredible meal at lunch, complete with salad, brazed chicken, and desert with complete table service (this as part of the conference). No buffet sandwhiches here, though a breakfast buffet would have been nice before that 830 session this morning. My room is about half the size of the one I had at the Hotel Rouge last year for WebManiacs, but it is very comfortable and service keeps it nice. The shower did flood the entire bathroom this morning and the in-room coffee is horribly weak (and small), but everything can't be perfect. I've also carried on my conference tradition of bringing a sinus infection with me, which is keeping me from the after-session activities this evening, but the concierge was able to get me covered with some NyQuil for the evening to try to knock this stuff out of me.

Aaron Conran, fellow ColdFusion developer and one of the original contributors to ColdBox, delivered the session covering what has changed and what's new within the 3.0 release, which is officially released tomorrow by Jack Slocum during his keynote. A great deal of work has been spent to make the upgrade fairly painless, with attention payed to backwards compatability, improvements to overall memory management (always important), and an eye towards keeping the code lean and mean, doing what is necessary to keep code size to a minimum and eliminate any unnecessary bloat. Menus and Toolbars have been converted into true Container objects, and the only significant markup changes appear to be to the Button objects.

There are several new visual components, maintaining the professional feel and consistency that has made Ext JS so compelling. Two that attracted a great deal of attention were the ListView and the Buffered Grid View. Sometimes you want the basic view afforded by a Grid, without having the need for (or the overhead of) things like drag and drop column ordering, sorting, and the like. The ListView gives you a basic grid style layout without all of that overhead, and a very trim DOM footprint as well, providing your view without unnecessary code bloat and browser memory overhead. The Buffered Grid View is a full Grid implementation that allows the browser to only render DOM for records in actual view, plus a few extra to make smooth transitions on scrolling, that automatically adds and removes DOM items when scrolling through your recordsets. The memory management improvements with this, and other changes, are very nice to see. Like the new .mon(), or 'managed on', methods that allow you to bind events in a temporary fashion, getting a proper cleanup from browser memory when that component gets destroyed. This was actually a method of Ext prior to it's 1.0 release, and has been heavily requested for this return.

Being the server-side guy that I've been for the last decade, the important new components came to the Ext.data package, with the addition of the Ext.data.Direct and Ext.data.DataWriter classes. Direct provides the ability to marshall your remote service call definitions into one container, calling your Ajax data requests from a centralized location with the ability to make multiple commands through a single Ajax request as well as call listeners on specific actions performed by Direct. Chris Scott did a presentation on Direct this afternoon, explaining how the developer has to create a server side Router to which Direct can attach it's requests, and showing us all a nice and very simple example (even if it was written in Ruby). The Ext Team provides a Reference for development of Routers in a number of server-side languages, including ColdFusion, PHP, Ruby, and .NET. The DataWriter class, covered in more depth in a future session, does for POST requests what the DataReader does for your GET requests, allowing a developer to define your routing of requests to send data back to the server. The combination of these classes with the existing classes of the Ext.data package really improves upon the power of using Ext with any server-side technology.

I was able to see the presentation on the Direct class because of a snafu with a scheduling change of the presentation of the Ext.air package. Originally these two sessions were the one conflict in my schedule, but they moved the Ext.air presentation session up a time slot (a change I didn't know of until after the fact) which allowed me the opportunity to sit in on Chris's session. Luckily the presentations are being recorded and are supposed to be available online after the conference. I'm glad they're doing this, since I had to leave during my final session of the day, on Ext.Core, due to the sinus issues that progressively worsened through the day. The Ext.air session is one I am very looking forward to watching.

The members of the Ext Team are a great group of guys. Abe, again, congratulated me on the book, and invited me to sit in on the Industry Experts panel tomorrow. I've had several people say they wished they had brought their copies of the book so Shea and I could have signed them for them. One attendee asked why PackT didn't have a booth setup somewhere, which is when I first noticed that their don't appear to be any outside sponsors for the conference. Ext appears to have financed the conference solely through the registration of the 200+ attending, which is pretty impressive considering the facility and it's services. On another note, ColdFusion has been publicly mentioned in several presentations, and with so many different server-side programmers represented here it is refreshing to hear so many comment that they should give ColdFusion a hard look.

And with that, I'm going to pack it in for the evening and pray that the NyQuil does it's job to kick the junk out of my head. Tomorrow looks to be an exciting day, with Jack's keynote releasing Ext 3.0, sessions on User Experience Design, Theming, and Performance Optimization. I'll be posting updates to Twitter and my Facebook profile throughout the day. If you're here, make sure to come up and say hi.

CF 8 and Ext: Presenting to Phoenix CFUG

March 25th, at 6:30pm MST (9:30 EDT, 8:30 CDT [Phoenix doesn't do Daylight Savings]), I will be presenting to the Phoenix ColdFusion User Group my presentation on ColdFusion 8 Ajax and Extension with Ext JS. For those not in the Phoenix area, you can attend over Adobe Connect.

Previous Entries / More Entries