Raj's Reflections

EMML Best practices

2009-04-07T10:49:00.000-07:00

EMML is a DSL (domain-specific language) for creating data-mashup services. Here are my thoughts on some of best-practices to follow for architecting EMML-based Mashup data services.

EMML Virtual Services
A re-usable services layer consisting of EMML service wrappers for complex web services & other data-sources. This helps create high-level facades for complex web-service interfaces. It allows helps in loosely-coupling Mashup Services with native Web/Data services. This layer can also help with service infrastructure policies i.e., rate-limiting, data-caching, service versioning, etc.

EMML Utility Services
Utility service layer contain reusable, non-business logic services. This avoids occurunces of common redundant logic in higher-level Mashup Services layer.
Common candidates for common utility services are
Persistence Service
Web clipping service
Domain-specific authentication services [e.g. Salesforce / Google authentication]
Data validation/transformation service
Notification service
Logging service

EMML Mashup Services
EMML Mashup services contain core data-mashup logic. Data originating from down-stream service layers are combined in interesting ways. Data produced by Mashup services are typically consumed and rendered by Mashlets / Widgets. Since it is a composable service layer, Mashup services also consume other Mashup services. Depending on the complexity of mashup logic, these services may be created in eclipse-based Mashup Studio or visually-composed using Wires.

EMML offers a rich-set of declarative primitives for filtering, merging, joining, annotating, sorting, transforming data. There is a rich-set of XPath functions available for use in these declarative EMML statements. Category of functions include comprehensive string, date, number, aggregate functions. Refer http://www.w3.org/TR/xpath-functions/ for the full list.
All these functions are available for use in EMML statements. In addition to these standard functions, Presto provides ability to create Domain-specific custom functions to XPath library. Once added, all these custom functions can be accessed similar to standard XPath functions. This provides a powerful mechanism to create a library of re-usable logic.

EMML Macros
Macros provide power to extend EMML language with domain-specific vocabulary. These are eseentially re-usable EMML snippets that contain finer-grained logic than EMML utility services. Additionally, Macros may be made available in Wires for performing user actions. Thus Macros enrich Wires-based visual-style of programming.
Macros can be created at global or package scope. This provides for Macro reuse across and within business organizations.

XPath functions

Most needs for Java-style imperative logic with Mashup services can be eliminated by using standard XPath functions in EMML statements. In addition custom-XPath functions specific to business domain can be created and added to Presto XPath Library. Subsequently these domain-specific functions can be used as any standard XPath functions in EMML statements. This provides a very powerful mechanism to encapsulate & re-use logic across Mashup Services. Custom XPath functions are written in Java.

EMML Refactoring
During development life-cycle, it can typically be observed that programming artifacts(EMML in this instance) tend to grow larger. At certain point, they become incomprehensible and tedious to maintain. It is essential to refactor such large EMML scripts into more modular, reusable EMML services. In a refactoring process extracted artifacts may include Utility EMML services, EMML Macros or Custom XPath functions.

EMML Policy
By setting appropriate configurations, it is feasible to disable usage of SQL, directinvoke or scripting logic in EMML scripts. This is a good way to constraint arbitrary usage of specific resources. This can typically be enforced for EMML scripts in Mashup Services layer. Any required complex logic in Mashup Services layer can be addressed by leveraging EMML Macros, Utility Services or XPath functions.

EMML Parallel
Parallel construct in EMML provides mechanism for executing coarse-grained parallel tasks. The EMML concurrency model must be viewed as message-passing concurrency model, rather than shared-state, lock-based model. So, typically parallel tasks must be used for performing isolated tasks that involve no sharing of data between them. For example they may be used to invoke multiple parallel services and subsequently merge returned data from multiple services. Importantly avoid usage of a same,shared variable across Parallel tasks.

EMML namespaces
A frequent, common error encountered is not accounting for namespace specification in XPath expressions when used in various EMML statements. Hence, ensure specification of appropriate namespace prefix/uri in xpath expressions. If you choose to ignore namespaces, a ‘*’ wildcard namespace may be used in XPath expression.

A wildcard namespace in filter expression may be specified as follows :

Note the usage of ‘*” wildcard for namespace prefix in the above xpath expression.

EMML ‘nanny’ scripts
EMML offers a rich declarative vocabulary for doing data-mashups. However, it is easy/natural for Java developers to fall-back on imperative programming style for coding data mashup logic. They need to be informed/educated about EMML language features and XPath function library available to them. Scripting logic should be considered only as the last option. Even so, it would be ideal to create EMML Service wrappers that contains such scripting logic. The higher-level EMML mashup services can then reuse the scripting logic
In addition, ‘nanny’ EMML scripts can be written to monitor the usage of Scripting logic in every EMML mashup script. Since, EMML is encoded in XML this EMML-on-EMML approach is feasible.

Such ‘nanny’ scripts would traverse EMML scripts to enforce sample policy constraints i.e.,
check if script, sql tags are being used.
check if any dis-allowed Java Interfaces/classes are being consumed in script
check if any dis-allowed endpoints are being accessed using directinvoke
check if any dis-allowed databases are being accessed using datasource

EMML Error Handling
Service invocations to external entities are fraught with reliability issues related to communication failure or poor providers QoS metrics. Hence, its best to be prudent about error handling / fault-tolerance.
When invoking a service using invoke or directinvoke, specify a timeout attribute. This ensures that the service invocation does not wait forever, and is aborted beyond a certain time period. Upon invocation error, it would be best to fail-over to any semantically equivalent alternate service. Typically, this fail-over service may be hosted in-house with previously cached or fall-back data. EMML offers primitives like onerror=continue/abort; faultcode, faultexception variables to handle Error processing.

EMML Testing
All Mashup Services are exposed using REST or JUMP (HTTP + JSON) interface. Any HTTP Client-based Load-Testing tool can be used for automated, regression and load-testing Mashup Services.

BI vs Mashups Analogy

2009-02-11T13:07:00.000-08:00

"The difference between BI and Mashups is the same as the windows in your
car - they are similar yet they serve different purpose. BI, like your side
and rear windows, provides a lot of info about what has passed - but you
would never drive by looking only out the sides and rear. Mashups, like
your windshield, give you real-time info so as to make timely decisions
based on what is happening now. All of the info combined allows you to make
the necessary corrections instead of making misinformed choices based only
on the past. See, they are not competitive, they are complementary!"

- Luis Derechin, JackBe

Yahoo BOSS + Twitter

2009-02-09T13:34:00.000-08:00

Aggregating search results from Yahoo & Twitter Search API:

Difficult decisions ?

2009-01-28T09:48:00.000-08:00

"When you have a difficult and important decision to make, flip a coin. Not because flipping a coin is a good way to make important decisions, but because in the moments while it is spinning in the air you will know which way you want it to land." - Anonymous

Embarking on a new journey

2008-12-25T19:29:00.000-08:00

Google Reader Subscription API

2008-12-20T01:49:00.000-08:00

Sample script for accessing Google Reader subscription feeds programmatically using our Mashup Domain Language.

http://gist.github.com/38275

Here's the sample response
http://gist.github.com/38277

Pretty slick huh ?!

Great Milestone!

2008-10-22T00:38:00.000-07:00

Yay! we released our development edition today! Presto developer edition is a free, open & pragmatic mashup development platform. It provides Ajax-based UI tools for high-level rapid-mashup development and a sophisticated mashup language (EMML) for developing advanced mashups.
And I am the creator of Presto's Mashup DSL :) Sweet!

My Workspace - Loving it!

2008-10-22T00:26:00.000-07:00

Now, that's Dedication!

2008-10-15T17:07:00.001-07:00

here

How-to keep iPhone CAPS Lock On

2008-10-08T00:30:00.000-07:00

Settings > General > Keyboard > Enable Caps Lock [ON]

and double-tap Shift to leave Caps Lock on and type ALL CAPS words like AAPL :) I am surprised Apple did not turn this on by default.

'Statistically Improbable Phrases' for advertising Mashups

2008-09-15T13:56:00.000-07:00

Amazon has a cool feature called "Statistically Improbable Phrases".
It basically mines for unique & repeatable phrase occurrences in a book and provides buyers a different (unique) perspective about the book.

For example, Given a Book about Virtual Machines

SIP for this book are
runtime constant pool, constant pool index, stack organization, literal pool, replace return address, local variable array, entire virtual machine, alt rules, virtual machine instructions, suspend instruction, current stack frame, virtual machine code, process descriptor, global event queue, termination flag, initialization code, new stack frame, top stack element, threaded code, instruction pointer, runtime representation, static chain, machine organization, register machine, control stack

Another way is to look at them are as auto-discovered Tags (i.e. mined from related contents)

A similar feature would be very useful in Presto Mashup Platform. SIPs can be mined and created for various Services & Mashups registered in Presto. The Mashup / Service content to be mined would include its associated data & meta-data. These SIPs would help give users a unique perspective of Mashups & Services registered in Presto.

Sketches

2008-08-27T18:00:00.000-07:00

I am a visual kind of person - sketches help clarify my thoughts and comprehend complex subject matter. OmniGraffle is my favorite sketching tool and can't live without it. To my surprise I ran into Graffletopia a vibrant stencils community for OmniGraffle.