Magic Leap and its augmented reality patents

Magic Leap is a Florida company, backed by Google, who have published some unusual patent applications about using googles to achieve a state of augmented reality, if not nirvana. They believe it is a wholly new approach.

I will admit that I had never heard of them when I accidentally came across a discussion of a brand-new patent application by them. It was from The Verge, and was titled See the beautiful, nightmarish patent illustrations for a Google-funded augmented reality device. They are indeed weird, and suggests that the invention is versatile in its use. It seems that you will always be in another reality -- a dystopia, perhaps, as the article comments. Commands can be found on your hand, or you can see data on your shopping cart. It seems it can do just about anything to entertain or instruct you. Here's one of the drawings.

And here's another.

The US patent application, US20150016777, was published on the 15 January 2015. Clicking on "Images" at the top of the link to Planar waveguide apparatus with diffraction element(s) and system employing same will take you to the full patent specification. It is 60 pages long, with the 14 claims to protection (which will be assessed by patent examiners) at the end. Interestingly, claims 15 to 27 are noted as "cancelled" and were not published.

That is from the official US Patent and Trademark Office website. I normally link to the Espacenet website, which links to patent documents from numerous countries, but at present the actual PDF is not available there.

On the same day was published the corresponding World or PCT patent application, WO2015006784 is available at the PCT's own site, PatentScope. It is an A2 document, which tells us that it was published without a search report listing relevant prior art that might mean it would not be allowed protection. An A1 would mean it is published with a search report. The A3 search report will at some stage be published on the website as an additional document.

These World or PCT search reports are more valuable than the US reports (which are anyway only published on the granted patents, not the applications), as they spell out which are cited as  X or Y documents (done before, or obvious improvements) against parts of the application.

WO2015006784 is a link to where the A3 will eventually be published on the Espacenet website.

This is a list of the (at the time of writing, six) US patent documents in the name of Magic Leap. All, at present, are only applications (you can tell as they are preceded with the year). Here's one illustration, from their System and method for augmented and virtual reality.

Rony Abovitz, the CEO, president and founder of the company, is named as an inventor on three of them. At present Espacenet lists 27 US patent records in Rony Abovitz' name. He is a busy man -- he has sold a company, MAKO Surgical, for $1.65 billion. He founded it in 2004. Business Inside UK has an interesting article on his life.

Google and other venture capitalists funded the company with $542 million in October 2014.

Apparently (I missed it) there was a lot of speculation about what Magic Leap were planning to do. An interesting article by Gizmodo is called How Magic Leap is secretly creating a new alternate reality, published in November 2014. Another, from YouRift, was published in December: Will Magic Leap kill the Oculus Rift ? This is a reference to a company in much the same field, Oculus VR. They were purchased by Facebook in March 2014 for $2 billion in cash and stock. Their are responsible for the World application Perception based predictive tracking for head mounted displays.

It remains to be seen if this invention will take off or if will turn out to be a damp squib, like Google Glass, which was recently taken off the market for more work. Augmented reality using goggles does seem to be the current fad.

Magic Leap has a company website. Their slogan is, It's time to bring magic back into the world, and they are currently looking for wizards to work for them.

Google's Android Wear wristwatch

Gizmag has a story, Google unveils the Android Wear platform: Google Now on your wrist. There is a Google official blog post from the 18 March about their new smart watch, Android Wear. It asks for apps to be developed for the watch. Here's one of that post's two videos, showing individuals enjoying using the watch to get information.

So it's not just Google Glass (which I posted about in May 2013). Others have mentioned the patents, but I've done my own research to identify four relevant patents owned by Google.

First there was an Israeli invention, by Modu, which dates as far back as 2005. In 2011 Google purchased their patent portfolio for $4.9 million, which explains why the owner of the rights is given as Google in their Wireless telecommunication device and uses thereof.

The next three are all by inventors Gossweiler and Miller. In 2011 there was applied for what was granted, in October 2012, Smart watch including flip-up display. Here are a couple of its drawings.

Next to be granted US protection, in June 2013, is Smart-watch with user interface features. Here is its main drawing.

Then in January 2014, but originally applied for in 2008, is Gesture-based small device input. The patent makes interesting reading with its talk of a virtual mouse pad. Below is its main drawing.

And here is another drawing from the same.

Time will tell (excuse the pun) on how well this device will sell.

Nest Labs' patents

Google has announced a takeover of Palo Alto-based Nest Labs, paying $3.2 billion.

It was only founded in 2010, by Tony Fadell, who had been Senior Vice President of Apple's iPod Division. While building an energy-efficient house he was frustrated by the limitations of thermostats. The company started, almost inevitably, in a garage.

This is yet another indication that Google is expanding into high-tech areas. Nest Labs is a designer of thermostats with a difference: the Nest Learning Thermostat, a smart device that can perform a variety of functions, to help create an intelligent or smart house.

In December 2013, for example, they were granted the US patent Dynamic distributed-sensor thermostat network for forecasting external events. Here is one of its drawings.

Here is another drawing from the same patent, showing the whole house wired up.

Much of their work is based on assessing occupancy in rooms to reduce energy usage, as in Occupancy pattern detection, estimation and prediction, illustrated below.

This is a list of the US granted patents published in their name, with the two at the top published only the 14 January, yesterday, and hence not yet available on the Espacenet database. Eleven were published in 2012. There is an overlap with the US patents applications published from 2012.

The company does not restrict itself to thermostats. Their smoke detector is discussed in a New Yorker article by Matt Buchanan, Can smart design make you love their smoke detector ?

This is a list of granted US patents by Tony Fadell.

Below is a video of a (half hour)  interview with Fadell.

Google surging ahead with patenting

MIT's Technology Review has a very interesting article on Google's enormous efforts to take out patents, Google's growing patent stockpile by Antonio Regalado.

As an example, back in May I posted on The Google Glass patents. I found 8 World patent applications containing the word "wearable".

Now there are no fewer than 15 Google World patent applications with the word "wearable" in the title. These include Wearable device with input and output structures, as illustrated below.

If only all patent applicants used such predicable language in their titles !

We are only a few days into December, but already there are 18 World patent applications, all published on the 5 December, as listed here. This is phenomenal, and as the article says the company looks like it will become one of the top patenting companies. There were 43 in all of November -- and 573 in all of 2013.

This compares with 743 by Microsoft for 2013, and 67 by Amazon.

As the article points out, the patent documents cover a wide variety of topics. Turning to the 1134 US utility patents granted to Google so far in 2013, we find such titles as Disambiguation of spoke proper names; System and method of identifying advertisement in images; Bicycle directions; and Inferring the gender of a face in an image, which is illustrated below.

Most are, however, to do with linguistic problems, as might be expected from a company which originally began searching text.

Sony's invention for wig sensors

Sony has had a US patent application published on the 21 November for its Wearable computing device, which is for a wig containing sensors. The idea is that the wig hides the sensors, and gives plenty of scope for placing the sensors. One of the drawing pages is shown below.

The BBC website has a useful article on "SmartWig". As it points out, the idea of wearable sensors is likely to become a big growth area in technology. Rather than having a device in your hand or in your pocket, it's easier to simply wear it.

Time will show which technology wins out -- maybe several will survive as competitors. The obvious one is Google's glasses, published as Wearable device with input and output structures. Watches are the other main area so far. Until the technology settles down, perhaps, it will be hard for anyone trying to identify relevant patents to find them -- not every relevant patent document will have the word "wearable" teamed with words like "sensor" or "device" or "computing".

This is a list of (mostly relevant) World patent applications whose titles combine wearable with sensor or computing etc. That is a very crude search, and no doubt many more are out there.

I did not trace a European patent application for the same invention, which apparently has not yet been published. I expect this to be published soon. The advantage of seeing it is the ability to review the search report which is usually available with it, listing prior art.

Snapchat®, a patented app

Snapchat® is an app that has become very popular in just two years. It deletes any messages received on a mobile phone within 1 to 10 seconds (the sender controls the length) of its being viewed. This can include videos, images and texts. It does so by spotting that eye contact has occurred. after which the timer kicks in.

Its demographic was originally teenagers, and 80% of its usage is in the USA. It is handy for rude or obscene messages (“sexting”). “Selfies”, self-portraits, are also popular. So, good when teachers or other adults are around. The idea reminds me of the Mission Impossible TV series’ openings, where a tape recording self-destructed after instructions were heard.

The logo used by the company on the app is of a ghost with a face. Oddly, the US registered trade mark is of the ghost without a face – apparently they redesigned it later, in which case they should have applied for the variant. Here is registration 4573338, claimed to be first used in June 2011.

Here is the logo as commonly used.

The two inventors devised the app as a project at Stanford University where Spiegel was a product design major. When he explained it in April 2011 before his fellow students, they disliked the fact that the messages would be deleted. Nevertheless he persisted, and launched the app that September from his father’s living room. Venture capital was raised and the company worked on sorting out technical issues rather than branding, or trying to make money from it.

I am puzzled by the fact that the granted patent for the invention, Single mode media visual capture, was only applied for in August 2012. It should have been applied for before it was marketed,as patent applications are supposed to cover new concepts. It was swiftly granted, in April 2013. It is one of the rare apps that to be patented. Here are two of the drawing pages from the patent.

In November 2013 Facebook’s offer of $3 billion for the company was rejected. As the company apparently has no revenue this is brave, and makes me wonder about conditions imposed by the backers -- it seems that they did not insist that a good offer must be accepted.

Problems are that an app called SnapHack has appeared which enables the recipients to store Snapchat messages, while Forbes magazine claims that it is not too difficult to actually retrieve supposedly deleted messages. In addition, a fellow ex-student has claimed that they took the idea from him and has filed a lawsuit, as explained by an interesting article by TechCrunch.

Modular robot toys

The BBC has an interesting story Build-your-own robot construction kits unveiled, though I did wonder why the principle had to be restricted to toys.

Modular Robotics, based in Boulder, Colorado, is responsible for the World patent application Modular Kinematic Construction Kit and also the older Modular Robotics , which has the drawing shown below.

The idea is mixing cubes to form new structures. Many toys follow that theme, but here magnetic balls act as joints and hinges, and internal flywheels enable movement, or wheels can be attached. No software needs to be written to operate them, and a Bluetooth module means that a smartphone or tablet can control them.

Each cubelet will cost about $25 and up. The video below shows how it works.

Shipping in January or February is envisaged by the company.

MIT has also been devising cube-shaped robots, as explained in an article from MITNews. They are called M-blocks, and look rather similar.

If the price can be got down I can see such cubes as a brilliant entry point for robotics classes in schools.

They also reminded me of the World patent application Programmable materials, applied for in 1994 by Joseph Michael of London, as illustrated below.

Cubes are used in all three concepts. It is the obvious shape: they neatly stack together, and are compact, but more importantly they mean that the surfaces are adjacent to each other, allowing all sorts of interactions without using wifi. I love the way such toys make the imagination soar by enabling students to "mix and match" to create new shapes and ideas.

The Adidas watch that checks your pulse: patent classification problems

There has been a lot of publicity about wearable technology that diagnoses you -- checking your heartbeat or pulse and so on.

When I first heard of this idea I thought of shirts incorporating technology, but most of the interest is in wristwatches. An example is in Adidas' new sensor for athletes, so that training programmes can be planned, and there is a BBC news story on it, Adidas Micoach smartwatch has heart-rate sensor

Now, supposing you wanted to identify inventions by Adidas in that field. You could ask a patent database for Adidas plus keywords such as watch. Better would be to take the patent classification A61B5/024, "Detecting, measuring, or recording pulse rate or heart rate."

This is an IPC -- it will be used by patent offices when they publish patent specifications. A61B5/02438 is a CPC which is a more precise class, not used on the specifications but added to those published by leading Western nations plus the World PCT system. It is for portable devices worn by the patient (or any user, really). It might seem obvious to use that class combined with the company name but it is easy to miss material, so while any detailed search would certainly use it, a second search should always be used with the broader class plus keywords.

Let's call using A61B5/024 for Adidas Search 1, and A61B5/02438 Search 2.

It is still necessary to decide which search box in Espacenet you use: CPC or IPC. The results below give first CPC and then IPC for Search 1 (Search 2 doesn't have an IPC option, as it is not a valid class).

Search 1 gives in CPC 4179 hits, with 2 by Adidas; in IPC 7127 with 5 by Adidas.

Search 2 gives in CPC 3129 hits, with 4 by Adidas.

How can this be ? The apparently broad search found fewer hits by Adidas as CPC, but more with the IPC than the apparently more precise CPC.

Besides the fact that the company may be working in related fields, and double-counting when more than one entry appears for the same invention, a problem is that using simply A61B5/024 says that only that class is wanted. Any specification classified by a more detailed class instead is missed.

Espacenet deals with this problem by allowing anyone searching the classification to enter the class in the search box which automatically selects the subordinate classes. They can then be copied to the search form. This results in a CPC search which we can call Search 3.

Search 3 gives in CPC 12,925 hits, with 7 by Adidas.

These include the intriguingly named Method and system for extracting cardiac parameters from plethysmographic signals, illustrated below.

Six of the seven are for similar coats that do the monitoring etc., while the seventh is not obviously for a watch -- it is a "portable electronic unit" which monitors congestive heart failure patients, and hence obviously not athletes.

So which is the best approach ? If asked to search for inventions for the concept by Adidas, I would use broad classes in both CPC and IPC (as the company has relatively few inventions). I would use the "In my patents list" ability (below the title in the bibliographic format) to list patent specifications by Adidas in the general field for later retrieval. I would also use the descriptions and the search reports at the end of European or World publications to see if they mentioned related material by Adidas.

I also searched the very broad A61B together with timepiece, watch and wristwatch (and their plurals) for Adidas and got zero hits (but beware their clever use of "portable electronic unit", which includes most watches).

It does seem that Adidas has not yet published the details -- hardly surprising, as it takes 18 months for a new specification to be published.

Sensors in clothing

The Daily Telegraph published an article today with the title UK researchers win patent for wearable smart sensors. It sounds like a nice idea, from Liverpool John Moores University. I have for a long time thought that wearing sensors is going to be an important advance in telemedicine. There are just two things wrong with the article.

The first is that there is no mention of the patent document's number, or a link to it, to help those interested in learning more. Nobody would write a review of a book or film without mentioning the title, after all.

The second is that the university hasn't actually got a patent. All that has happened is that a patent application was published on the 11 September as Microwave monitoring using an electrically conductive textile. Granting a patent is a second stage. Maybe the usage by the UK Intellectual Property Office calling it a "milestone patent" confused the issue in their press release on the invention.

The invention provides continual monitoring of the body using non-invasive techniques. This area is so important that the Cooperative Patent Classification (CPC) has provided an area for classifying diagnostic sensors mounted on clothing at A61B/6804. Patent documents on that subject can be found by ticking the box next to the required classification and then on Find Patents (on the left). This gives nearly 3,000 hits.

Better, perhaps, is to click on Copy to Search Form when additional fields can be added, such as keywords, company names, or patent authorities to narrow down the hits found. A problem with the CPC is that often the classes are added many months after publication, while the less detailed International Patent Classification on which it is based is available on publication day.

In this particular publication, very unusually, the CPC is already available, perhaps to mark the fact that the publication is GB2500000 in a series that began with 2000001 in 1978.

Subject searching in a sophisticated way is complicated and it is easy to make mistakes, so I always recommend using experts to help, such as the librarians in the Patlib UK network.

Apple's watch invention

The TechCrunch website has just announced that Apple has had a US patent granted for a disappearing bezel for an electronic watch to enable a bigger screen to be visible. Published today, it's US 8447114.

The problem with putting screens on tiny devices is that it's hard to see anything. Any ability to enlarge the screen is highly beneficial for the user -- and for those who wish to communicate with them.

But what do you call it ? In December 2012 Apple applied for the Taiwan trade mark iWatch in Class 9, which includes software, as quoted in an article on the MacRumors website.

If they want a US trade mark for that trade mark, the problem is that in August 2012 OMG Electronics applied for that trade mark with 85703706 in Class 9 -- but failed, inexplicably, to say it was for watches as well as various listed devices.

Earlier still, in July 2007 Platinum International Holdings applied for 77236688 for iWatch, again in Class 9, but mentioned "cell phones watch". This has not yet been registered, which is odd after six years.

This could be important, as listing goods is sometimes vital when the identical trade mark co-exists in the same class, on the grounds that they can't be confused. The earliest company to file gets the trade mark. You can quote a "priority", when you first filed it within 6 months in another country, to get an earlier filing date.

Forbes Magazine has an article on the subject which cites the Taiwan data.

The Google Glass patents

I t's been fun having a look for the patents associated with Google Glass, the spectacles that enable its wearers to interact with the Web and, above all, to see projections of information.

Google Glass itself is the subject of a US trade mark application which, oddly enough, was published 18 June 2013 for opposition -- several weeks in the future, as I write.

The key term seems to be "wearable" although it is possible that some new technology involved does not depend on a wearable frame and therefore may be hard to find.

There are seven US Design patents for the look of variant models, all with the title "Wearable display device" (although one adds the word "frame", and one "section").

An article on the CNET website, published 21 February, suggests that US20130044042A, is highly relevant. The article is called Google Glass patent application gets really technical. I prefer to link to the corresponding World patent application, Wearable device with input and output structures. One of its drawings is given below.

However, three other patent applications were published in March or April and there are in all eight World patent applications by Google containing the word "Wearable".

None of these have been granted rights as B documents. One advantage of using the World documents rather than the corresponding US applications (listed here) is that search reports listing prior art found by patent office officials is often available.

These are either on A1 documents (at the end) or in separate A3 documents. The USA only publishes search reports on its granted patents, where a brief summary is given on the front page.

[24 May -- the New Yorker website has an interesting article on forerunners, Glass before Google, which links to a copy of a patent by Morton Heilig, applied for in 1957]

[7 Dec -- see my article Google surging ahead with patenting for update on wearable technology]