Wednesday, November 7, 2012

Why Sears Is Going All-In On Hadoop

Sears pushes the cutting edge with some big data techniques, while trying to sell its big data services. Can emerging tech drive change in old-school companies?

By Doug Henschen InformationWeek
October 31, 2012 08:00 AM
For complete article, click this link:

Like many retailers, Sears Holdings, the parent of Sears and Kmart, is trying to get closer to its customers. At Sears' scale, that requires big-time data analysis capabilities, but three years ago, Sears' IT wasn't really up to the task.

"We wanted to personalize marketing campaigns, coupons, and offers down to the individual customer, but our legacy systems were incapable of supporting that," says Phil Shelley, Sears' executive VP and CTO, in a meeting with InformationWeek editors and his team at company headquarters in suburban Chicago. 

Fast And Agile
Sears' process for analyzing marketing campaigns for loyalty club members used to take six weeks on mainframe, Teradata, and SAS servers. The new process running on Hadoop can be completed weekly, Shelley says. For certain online and mobile commerce scenarios, Sears can now perform daily analyses. What's more, targeting is more granular, in some cases down to the individual customer. Whereas the old models made use of 10% of available data, the new models run on 100%.

"The Holy Grail in data warehousing has always been to have all your data in one place so you can do big models on large data sets, but that hasn't been feasible either economically or in terms of technical capabilities," Shelley says, noting that Sears previously kept data anywhere from 90 days to two years. "With Hadoop we can keep everything, which is crucial because we don't want to archive or delete meaningful data."

Sears says it has surpassed its initial target to reduce mainframe costs by $500,000 per year, while also delivering "at least 20, sometimes 50, up to 100 times better performance on batch times," Shelley says. Eliminating all of the mainframes in use would enable it to save "tens of millions" of dollars, he says.

The MetaScale Mission
Shelley is still CTO of Sears, but if his portrayal of all the things Hadoop can do sounds a bit rosy, keep in mind that he's also now CEO of MetaScale, a division that Sears is hoping will make money from the company's specialized big data expertise.

The rarest commodity that MetaScale offers is Sears' experience in bringing mainframe data into the Hadoop world. Old-school Cobol programmers at Sears were initially Hadoop skeptics, Shelley says, but many turned out to be eager and highly skilled adopters of the Pig language for running MapReduce jobs on Hadoop. Tasks that required 3,000 to 5,000 lines of Cobol can be reproduced with a few hundred lines of Pig, he says. The company learned how to load data from IMS (mainframe) databases into Hadoop and bring result sets back into mainframe apps. That's not trivial work because it involves a variety of compressed data format transformations, and packing and unpacking of data.

Would-be MetaScale customers in other industries will face different challenges as they consider embracing Hadoop. Could quick analytical access to an entire decade of medical record data change how doctors diagnose and treat patients? Could faster processing spot financial services fraud more effectively? Companies are focused on choosing and building out the next-generation platforms that will handle those big data jobs. Will Hadoop be that platform, and will Hadoop help turn MetaScale into a successful pioneer? That's a story that has yet to unfold.

Special note to Blacks Gone Geek readers. I'm featuring this article because I'm working at Sears where I first heard about Metascale, a Sears company. MetaScale provides best-in-class, technology-managed services and solutions to enterprises that are looking to unlock the potential in their data without the time, cost and complexity associated with traditional big data initiatives. Let me know if you want to connect with the folks at Metascale.

Milt Haynes, Founder and Chief Knowledge Officer
Blacks Gone Geek

Join The Blacks Gone Geek Community

No comments:

Post a Comment