What are the sounds of the city saying?

IBM develops analytics and classification technology to provide data for a new kind of multimedia-based search engine

Editor’s note: This posting was authored by Zvi Kons, researcher in the Speech Technologies group at IBM Research - Haifa

When you walk down a busy street, do you ever notice the sounds that surround you? People, traffic, music; city sounds are often like the foreign language the couple next to you at the cafĂ© is speaking—background chatter. That city buzz, though, together with related visual images, has the potential to generate a continuous stream of information that can indicate real-time dynamics of the city.

To gather, process, analyze and ultimately separate useful sound from white noise, my team at IBM Research-Haifa is working on new technology for searchable audio analysis as part of the EU-funded project called SMART (Search Engine for Multimedia Environment Generated Content).

We’re developing algorithms and an engine to analyze those city sounds, extracting information that can be cross referenced with video images to generate real-time content. Our research on audio classification is an integral aspect of a new kind of internet search engine that could provide locally oriented, readily available and informative content with practical applications. 

Capturing the sights and sounds of city streets to gain insight

Our team collected data from two locations in Santander, Spain. Because the municipality is a partner in the SMART project, they offered to support the technical aspects of the infrastructure needed and are helping test the technology. Cameras and microphones set up in the town square and market area provided continuous audio and visual data of normal daily activity for one month, collecting more than 1,000 hours of data. We analyzed the sounds to note various types of activities, and to identify patterns and anomalies, like peak hours for busy crowds in the market square, traffic, and special events.  

Santander city square

Visual representation of weekly audio from the city square

The audio from the video above and others produced this diagram that shows a visual representation of the weekly crowd activity level; blue for low activity, red for high activity.

Another sample detected a day with unusual crowd noise, music, and applause. By cross-referencing with video footage from nearby street cameras, it turned out to be from a protest rally on a nearby street, which could be important information for analyzing any immediate security risk, or the need to send a news team to report on a developing story.

Listen to the mid-day rally as it passed on the top right corner of the frame:

Applying audio analytics

The sounds of privacy
To address potential privacy and legal issues, the SMART team used wide angles and low resolution for the video cameras. The microphones were placed at a distance to pick up crowd noise rather than intelligible speech or individual conversations.
The idea behind SMART’s new multi-media-based search engine is the incorporation of information gleaned from the environment. We can use data from city sounds and video images, as well as social media like tweets, to identify events and situations in real-time and make that information available online. The sounds of the city can help identify a drunken brawl, a spontaneous demonstration, a musical event, or an accident during rush hour. This kind of readily available information could be valuable for security systems, municipal and media use, and helpful knowledge for city residents.

Our research highlights the enormous potential of easily accessible information in our physical surroundings. The technology to use that information has exciting and practical applications for smart cities, with innovative ways to interpret sounds and images.


The IBM Mainframe: The Machine that Keeps Making History

by Dr. John E. Kelly III, Senior Vice President, Solutions Portfolio and Research

In December, I had the opportunity to take part in a tribute to one of the great architects of the IBM mainframe, Erich Bloch.

In 1985, President Ronald Reagan presented Bloch and two of his IBM colleagues, Fred Brooks and Bob Evans, with the first U.S. National Medal of Technology and Innovation. The award cited "their contributions to the IBM System/360, a computer system and technologies that revolutionized the data processing industry."

Jim Collins, author of Good to Great, ranks the S/360 as one of the all-time top three business accomplishments, along with Ford’s Model T and Boeing’s first jetliner, the 707. That’s pretty exclusive company.

This S/360 evolved over time into what we know today as “z System,” now 50 years young. Today, IBM launched the latest, most powerful iteration of the mainframe: the z13. This remarkable technology continues to power the platforms upon which entire industries and much of our global economy depend.

What sets the z13 apart? The same thing that differentiated its predecessors from the competition: the world-class IBM research and development inside every machine. There is no greater font of innovation in our industry.

That’s critical, because clients today require systems of insight. They must bring together massive amounts of data — customer data, enterprise data, mobile, social, streaming and genomic data — and analyze all of it to provide insights we’ve never had before. And they must do it securely.

IBM Research contributed significantly to achieving these goals in the z13, especially in the architecture and design of the processor and memory-cache systems, optimizing the technology for performance gains. Research results also factored into the high-security crypto devices inside, strengthening hardware protection.

We are now entering a new era of cognitive computing, in which computers will increasingly analyze problems in more human-like ways. And so it’s fair to ask: will the mainframe still be relevant in this context?

The answer is a resounding “yes.” In fact, it may be one of very few systems that can make this leap. The perfect combination to perform this task is mainframes and cloud, sometimes attached, sometimes one and the same where the mainframe is the cloud. 

As part of this transformation, we are bringing the computation to the core data. z13 is designed to speed the mobile transaction experience by allowing organizations to conduct analytics on the mainframe without the need to offload data to other systems. z13 "native analytics" can handle billions of transactions per day while analyzing data in real-time. It delivers faster, more personalized services with new levels of fraud protection. This is a competitive differentiator made possible by IBMers in Research.

We’re already starting to infuse cognitive sensing technologies inside the machines. We’ll continue to embed our most advanced learning machine technology into the mainframe, not only to gain greater efficiencies, but also to extract better and deeper insights for our clients from the tremendous amounts of data they possess. 
All of this innovation is accelerating on z13. In just the past year, IBM has received more than 500 patents on mainframe technology. There are companies that would be thrilled to have 500 patents in their entire portfolio.

This is innovation with a purpose. We listened to clients from retail to healthcare, from global finance to transportation, from energy to government.  They told us what they needed to drive their industries forward and to serve their customers and constituents better. We took that feedback and brought together the brightest minds thousands of IBMers who are passionate about solving the world’s greatest challenges to design a system that is unmatched.

Every new generation of the IBM mainframe is state-of-the-art technology that has the power to transform industries and society. The z13 is the latest product of this remarkable legacy.

The revolution continues. 

Tune in to the Livestream event on Wednesday, January 14, from 2 – 4:30 p.m. ET (US) when IBM will share a whole new generation of IBM z Systems; and watch the new z Systems movie: New Possibilities (beginning 11:30 a.m. ET on January 14) to find out how today's IBM mainframe is built for the needs of today’s digital economy.


Validating a (hack-free) web experience

Patent verifies information flow downgraders  

Hacks, phishing, and other malicious cyberattacks all happen due to bugs in a software program’s code. Because no one can manually check the millions of lines of code behind a mobile app or a website, developers use verification software to check their work. And even then, sometimes the analysis that traditionally trusts that defenses installed by the developer were implemented correctly, misses vulnerabilities. 

The patent process

What I see most often is someone with a great idea, but who doesn't think of it in terms of a patentable idea. As a Master Inventor, I want to help my colleagues move those ideas to patents. – Marco Pistoia

It’s also about point of view. Many file patents that describe what they have done. But what’s more valuable is the abstract, creative use of the idea. – Omer Tripp
This is where IBM Master Inventors Omer Tripp, PhD and Marco Pistoia, PhD focused patent #8,635,602: Verification of information-flow downgraders. Filed in 2010, the invention aims to close the loop on code that – because of poorly implemented defenses – compromises sensitive data, such as bank account numbers, and passwords. 

“I would say the patent is more valuable today (it was issued in 2014) because of the explosion in cloud and mobile technologies, which necessitate defenses that are more challenging to implement correctly than ever before. We’re all accessing websites that require sensitive information about ourselves, and sometimes the software asking for that information is vulnerable,” said Omer, who has filed 174 patent applications and been issued 64. In 2012, Omer earned more than 1 percent of IBM’s entire patent total of 6,478.

Omer and Marco set out four years ago to verify what software applications claimed to be secure “downgrader” code (the part of a software program that sanitizes or validates untrusted input to a website, or obfuscates and declassifies confidential data before its release). By developing a way to check a website’s information flow downgrader, they found that live, implemented code still had security holes. “Our tool simulates what developers have a hard time testing for, such as ‘double encoded’ input, or other odd combinations of validation routines,” said Marco, who has 148 patent filings and 72 patent issuances to his name.

Their ISSTA 2011 paper, Path-and index-sensitive string analysis based on monadic second-order logic demonstrated vulnerabilities on several open source websites, earning ACM SIGSOFT’s Distinguished Paper Award. 

Trust and verify

Downgraders take input that’s not trusted, like erroneous details entered into a website login form, and help make it trusted. They sanitize the information by getting rid of certain unintended, and potentially malicious, characters and substrings. But they can be tricked by recursive nesting of the payload, and other clever tricks. The invention detects when a downgrader incorrectly allows (or rejects) accurate input – and can be integrated into standard analysis tools.

"Developers do validate that their software works as it should. But what’s often left unchecked are the inputs to the software – how the average person checks Twitter, or applies for a bank account,” Marco said. “We can now analyze those inputs to make sure the defenses a developer puts in place work, or alternatively, have errors.”  

“Now, we want to connect this invention with others we’re working on in this area, namely tools that automatically fix broken defenses. This would help developers check their code, and their downgraders,” Omer said.

More about IBM's 2014 patent leadership

Patent powers a cleaner commute

What if a patent that could make your car aware of the environment to help improve your city’s air quality? 

by James Kozloski, Computational and Applied Neuroscience, Research Staff Member, IBM Research 

Imagine you’re driving into New York City. It’s a summer day. Sunny, hot… and the smog is bad. The city knows the pollution is a problem, and wants your help to abate it. So, as you drive through the EZPass toll on the Triborough Bridge, you – or more accurately, your hybrid car – gets a message asking “Can the City of New York switch your powertrain to ‘all electric power’ while you’re visiting?” If you say “yes,” your toll fee will be discounted. 

That’s the concept behind patent 8,781,668: create a location-based vehicle powertrain regulation system.  

I had the idea while riding around Bangkok. Many of the taxi drivers can switch their fuel sources from standard petrol (regular unleaded gas) to cheaper liquefied petroleum (such as propane or butane). Why not have software automatically make a switch based on environmental conditions? That’s essentially what the Thai cabbies were doing manually to save a few baht. The system we invented does it to save on automobile emissions based on location.

James Kozloski on a family vacation in Bangkok, Thailand
 A partnership between your car and the environment

You and I don’t necessarily know the ozone rating or smog levels as we commute to and from work. But sensors do. Sensors that send data to weather stations, and city officials. If our cars, in turn, have sensors that picked up this data, and could communicate with these third parties, our car’s powertrain could adjust its emissions while driving. 

Currently, only plug-in hybrid electric vehicles (PHEVs) can make this kind of auto-powertrain switch (unless you count taxicabs from Thailand). But think about driving around in a congested city. You’re probably not driving in a way to notice the difference between a gas-powered versus all electric engine. So, with your permission, why not let the environment – via a sensor – tell your car to stay on “electric” while you’re in city limits?

Today’s hybrids work only to maximize the vehicle’s performance based on what the driver does. This invention just adds external considerations to the vehicle’s efficiency calculations. This means retrofitting is doable by adding a sensor or two, and updating the onboard software to respond to their measurements. And in the patent’s current form, the driver would still have to agree to allow any of these outside adjustments. (So don’t worry, nothing will take over your car!) Just passing through via the highway? Say “no” to the all-electric switch at the toll booth. 

Every PHEV and city it drives through is a customer

In attempting to determine the patent’s potential impact, we calculated that about 30 percent of the cars on the road need to use alternative fuels to make a significant dent in city smog levels. While the hybrid market share currently hovers around 3 percent, PHEVs continue to grow, and J.D. Power predicts that one-third of all vehicles will use some kind of alternative fuel by 2025 (17.5% being PHEVs). And some cities may reach a practical number much sooner. As civic and other government environmental agencies get a clearer picture of emission levels, this patent could deliver a small, but significant, way citizens can help improve a pollution problem.

IBM’s own “commuter pain index” shows that bad traffic isn’t going away. So while the patent doesn’t solve a lousy commute, it can help make it cleaner.

Podcast: James Kozloski explains patent #7,781,668

More about IBM's 2014 patent leadership

Turning mobile security upside down

Hackers won’t know which way your passcode points with this patent 

By James Kozloski, Computational and Applied Neuroscience scientist, IBM Research 

How many numbers long is your smartphone’s log in passcode? The standard four digits? (If you don’t use a login password, you should!) Because my work email and calendar are on my phone, I have to enter an eight-digit code. Not a big deal until I’m trying to pay for coffee, and the barista asks me to rescan my bar-code, so I have to re-enter those eight digits again (and sometimes again) – with angry caffeine-deprived patrons in line behind me.
My ideas for patents are often completely different and separate from my day-to-day job in studying neuroscience. I actually spend most of my time with healthcare clients working on neuro-degenerative diseases, deep brain stimulation, and pharmaceuticals, trying to understand the relationship between brain tissue and the functions that the brain implements, such as behavior selection.

I think of patents this way: what are the systems and methods around us that could be improved to solve a problem I’ve identified? So, even if I don't have the specific technical expertise, I can explain how something should work, and then seek out a colleague with the expertise to bring something to a patentable stage.

In commiserating about smartphone passcode follies with a colleague, we realized that alpha-numeric digits for a passcode could be reduced if coupled with an orientation, since adding orientation would actually change the probability that any one digit is actually the correct digit. Now, the question becomes: is the "2" right side up? Or is the "2" at a 90 degree angle? Just one digit has multiple possibilities. 

You could say that patent 8,832,823: “User access control based on handheld device orientation” came out of wanting to access my phone (and pay for coffee) faster!

Secure disorientation 

Think of device orientation like a keyboard’s “shift” key. It gives the device a new set of bits to access without needing a new physical key or character to enter. But the orientation precision needed is no greater than what’s needed to rotate a device’s screen. Easier to manipulate than a sticky “shift” key, but still difficult to guess – even if you password is “password.”

More options. Fewer keystrokes!

Turning your phone (or any mobile device with an accelerometer) as you enter a passcode is just one level of improved security. The patent also takes into account how to store the digits separately from the orientations. This means that even if your passcode is stolen (say from an online hack), your locally-stored orientations would prevent remote access. And vice versa, if your phone is stolen, there’s almost no chance the thief could replicate your “digit + orientation” passcode entry. 

Connecting a device’s local orientation, or accele-metric component, with alphanumeric codes stored by a remote web-based service (such as a bank or online store) means any website that requires a password can use this invention. Then locally, your phone’s OS can determine the orientation of portrait or landscape for a key press (regardless of character identity), and verify that the key press order – plus orientation order – is valid. And just as different passcodes give you access to different websites, different orientation passcodes could give you access to different parts of your phone. 

So, hopefully during a not-too-distant future stop for coffee, when I’m armed with a two-key orientation passcode – that only accesses a digital payment option (and not my work email) – I won’t drop my phone when I have to turn it upside down!


Top IBM Research Stories of 2014

A Guinness World Record. A new kind of plastic. Oh, and a chip modeled after the human brain. Here are some of our most popular stories in 2014. It’s been a busy year! Thank you for reading, sharing, and commenting on them. But most of all, thank you for being curious about pushing the limits of science and technology.  

Help Set a Guinness World Record

IBM scientists partnered with National Geographic Kids to set a Guinness World Records title for the world's smallest magazine cover.

Ballet or mathematics? Most ordinary eight year olds girls would probably choose ballet, but Maria Dubovitskaya was anything but an ordinary eight year old.

One day, after ballet lessons in the Moscow suburb of Domodedovo, Maria’s parents were running a little late. She heard other children, mostly boys her age, clacking away on IBM 286 PC keyboards in the classroom next door. Peeking through a crack in the door Maria was overcome with curiosity.

When scientists succeed at IBM Research, they tend to stay. Robert Dennard, the inventor of the DRAM, for instance, had been at IBM for 56 years when he retired earlier this year. But there are researchers at the opposite end of the seniority spectrum who are already making their marks—on IBM and the world.

Project Lucy: Confronting Africa’s Grand Challenges, with Watson's Help

Amid the chaos of civil war, Abdigani Diriye’s family fled Somalia in a rush when he was just five years old. Diriye and his sister escaped to London in the care of a 19-year-old aunt; his father flew to Sweden; and his mother made her way through the battle zone to Kenya.

Every year, top U.S. men’s college basketball teams enter a month-long tournament for a chance to be crowned champion. And it always stirs up fan and pundit predictions to pick potential winners of all 63 games. 

To really give the pot a stir this year, Berkshire Hathaway, Quicken Loans and Yahoo Sports teamed up to create the Billion Dollar Bracket Challenge. Choose every match up correctly and win $1B.

(download a "Bracketology" infographic)

Graphene Circuit Ready for Wireless

Graphene, one of the world’s thinnest electronic nanomaterials, has long held the promise as a wonder material in everything from flexible touchscreens to super-fast circuits. It’s that interest in semiconductors that led my team to build the world’s most advanced, fully functional integrated circuit made of wafer-scale graphene – 10,000 times better performance than previously reported efforts.

Tilted view scanning electron microscopic (SEM) image revealing the integration of key
components in IC with enlarged view showing the advanced gate structure
of the graphene field-effect transistors (GFET). Inset image shows crosssectional
SEM of embedded T-shaped gate. Scale bar, 500 nm.

“We need to talk.” Generally not something you want to hear from your PhD advisor. But when it was followed by, “Have you heard of HBO?”, then-Stanford student, now-IBM Watson engineer Vinith Misra was intrigued (and as a film enthusiast, a little amused). 

His advisor, Dr. Tsachy Weissman, had been contacted by the technical advisor for HBO’s Silicon Valley in 2013 to help the show’s creators develop fictionalized compression algorithms, and he wanted to bring Vinith on board.

Theseus was a great hero in Greek mythology known for qualities such as strength, courage and wisdom. 

Therefore it’s no surprise that a team of Greek IBM scientists in Zurich and Professor Theodore Antonakopoulos and his team from the University of Patras, Greece, borrowed his name as a codeword for a groundbreaking new memory technology, which combines flash with phase change memory (PCM) on a PCI-e card. Initial tests have clocked 12x and 275x improvements — and that’s no myth.

Tanuja Ganu grew up in a small town in India about 400 kilometers south of Mumbai, where – like much of the country – energy outages happen all the time. 

“The voltage was often so low that the lights were dim and the refrigerator would burn out.

“I studied for exams by candlelight, and endured summers without working fans. To deal with this as children, we learned to time-shift critical things we needed electricity for – like cooking and cleaning," Tanuja said.

See What's Cooking in the Cognitive Kitchen

Raise your hand if you have interests outside of your day job. Probably most everyone, I imagine. Now, how often do they serendipitously collide? Probably not that often, right? But that’s what happened for me two years ago when I applied my computer science skills to my love of the culinary arts as part of IBM’s Cognitive Cooking project.

Seven nanometers. That’s slightly longer than the 2.5 nm circumference of a DNA double helix. And 100,000 times smaller than the width of a human hair. It’s also the miniscule dimension that future semiconductors must push past to keep up with today’s computing demands being put on cloud, big data and cognitive systems.

What is a cognitive chip? The latest SyNAPSE chip, introduced in August of 2014, has the potential to transform mobility by spurring innovation around an entirely new class of applications with sensory capabilities at incredibly low power levels. This is enabled by a revolutionary new technology design inspired by the human brain.