Research for the Future of Humanity: Dr. Kitano Hiroaki | TMC Talks Vol.15
Watch the full movie here
What is Sony CSL?
I would like to share with you some of the research activities Sony Computer Science Laboratories (Sony CSL) and Sony AI are conducting at this moment under the ground theme of the 'Research for the Future of Humanity.'
To begin with, I would like to introduce a little bit about Sony CSL, and what kind of organization Sony CSL is.
As you may know, Sony is a conglomerate with multiple industry sectors primarily centered on entertainment like games, music, and pictures. We also have electronics and image sensing, which is a major semiconductor business, and domestically we have a financial sector as well, which has now turned into Sony Group corporations.
Sony CSL is directly under the HQ and is an independent research lab working on what may seem blue-sky research, which is not directly attached to any of the business units, but we are looking broadly into the future of humanity. We expect that we will tackle some of the most important issues for humanity. Some of the research results and technologies will be built with scientific insight and have the potential to be transformed into Sony's business as well as into the industry in general. We hope we can contribute to the future of humanity as well.
Our priority is to perform high-quality research and deploy it in the real world. It will benefit the future of humanity and we hope that some of the approved work will give feedback to Sony's businesses in the coming years.
Sony CSL was established in 1988 in Tokyo, which is the main base for us. In 1996, the Paris office was created as well, and in the last year (2020) we established the office in Kyoto. All of the offices are operated under the same principle: we want to contribute to the future of mankind.
Of course, our topic varies, and we would not want to have a predefined topic. For example, we recruit very interesting people, what we call 'crazy people', expecting that they know what they are supposed to do, as there will be individual curiosity, individual missions, and a sense of responsibility to define what research should be carried out, how it should be operated, and how it should be deployed in the real world.
Sony CSL's motto, which is the most important thing, is 'Act Beyond Borders'
Usually, research labs do their research and they hand the results over to someone else to be deployed in the real world, but in the complex environment, we cannot guarantee if the handed-over research results would really be able to be deployed or not. The only way that you can make sure that the research results can really be fed back to society, creating the future of humanity, is to actually 'Do It Yourself.' So it is very important to be able to do not just the research but the deployment as well.
'Beyond Borders' is very obvious because the problems we are facing are so complicated. Thus, we have to go beyond the borders of academic disciplines, whether these are research or business or social deployment, or national borders. Borders are just the artifacts of human beings, so we have to go beyond any kind of border. That is the most important guiding principle driving Sony CSL.
We are not a huge lab, we have only 30 to 40 researchers in Tokyo, Kyoto, and Paris. You can see what kind of people we are talking about. Each individual has their own agenda and is expecting to contribute to the transformation of this world into a better place.
Looking back at the history of Sony CSL, in the first 10 years we really needed to focus on the traditional elements of computer science like DOS (Distributed Operating Systems), network interface, and AR (Augmenting Reality), and some of these have actually moved into Sony's products. For example, DOS was translated into one of the key technologies for the OS of PlayStation. As for AR, Dr. Rekimoto Jun is the founding father of AR and things got started by him.
Then we expanded our research beyond computer science into 'computer science and human beings.' It is focused more on biomedical research, brain science, and economy as well. The computer science part has become more interactive, like involving visual computing and interactive music. It has expanded more into the scientific area and more into the human-computer interaction area as well.
In the third decade, we recognized the importance of the global agenda, such as energy issues, global climate issues, healthcare issues, and biodiversity issues, etc., so we have a huge chunk of research going on in relation to global-agenda issues.
We have increasing importance in the AI, data analysis, and cybernetic intelligence areas, so we are helping the Sony group to be able to utilize data and AI capability group-wise. We are providing data analytics services and education services to the entire Sony group. Those are the research areas and are not the kind of things for management to decide the importance of. Those are the things that individual researchers consider important. The key is to find the individual who can help us to change the world and who has the desire to do so, because these researchers are much more knowledgeable than I. I trust them and I trust that these researchers will be able to cut through the issues and transform the world into a better place.
The most recent addition to Sony CSL is the Kyoto Lab. Dr. Rekimoto, known to be the father of AR, is currently the director there. The lab has a very interesting initiative. What we are trying to do in Kyoto is research the essence of high human value. It could be space, material, or culture. We try to find out what provides us with wealth in terms of value. It may be cultural or emotional issues rather than financial issues, but of course, the financial issues are important as well. We try to see happiness and wealth in a much broader sense in harmony with culture and nature. That is what we are trying to achieve.
The research lab is still small with like 3 or 4 people, but we try to understand what is going on and connect with the local community in Kyoto. We try to incorporate our technology and insight into craftsmanship, art, and culture.
Some of the research going on is in collaboration with the local community, like the Urasenke tea ceremony society, because they underpin the cultural basis for Kyoto and try to get new technology and new insight. We can learn from them.
Robotics and Prosthetics
Let me talk about a specific topic, something relevant to the Olympics and Paralympics.
Dr. Endo Ken, one of the researchers on our team in Tokyo, is a robotics researcher and at the same time, he found prosthetic-device research extremely important because his close friend actually lost a leg and he realized he couldn't really help his friend very much with robotics technology.
He has been working with me in creating a biped robot; a humanoid that has two legs. Even with the robot walking technology, however, he could not really help his friend who had his leg amputated. He was shocked at that fact, and so he transformed his research topic into the prosthetic device. Finally, he moved to the MIT media lab and worked on the prosthetic device.
Dr. Endo has also created a small company called Xiborg, providing the running blade for athletes, and he is also working with a clinic in India to provide low-cost prosthetic devices in that country in collaboration with Japan. One of his goals is to have athletes with his product win a gold medal in the Paralympics. We are probably going to see this come true in a month, and that record might be better than the Olympic Games record. He believes the perceptions of people with disabilities will be dramatically changed when someone in the Paralympics runs much faster than someone in the Olympics. I think the whole concept will change and he will get some success. He said the blade was used by one of the athletes in the Japanese Paralympic team for the 400-m relay, and he got a bronze medal. He is working with multiple athletes so that his blades will be used at the Paralympics next month.
He has been doing very detailed physiological simulations and analytics to see how athletes need to be trained, how to design the blade using composite materials, and how to adopt a new design structure that is scientifically tuned. He found out that athletes with blades need to be trained in a dramatically different way from the athletes in Olympic games because all the mechanical dynamics are very different.
When non-amputees wear the blade, the movement of the upper torso muscles is very different. Dr. Endo came up with not only the blade but with the way athletes need to be trained based on the specific blade design. It is a really interesting scientific activity to help the athletes. What we have learned should be able to be translated for runners who have had their legs amputated; 'Blade for All.' We will possibly transfer that research into the general robotic prosthetic device or the non-robotic prosthetic device.
Hopefully, this experience will be able to help a lot of people by providing the ability to walk and run. One of the challenges faced by Dr. Endo was providing legs for Mr. Ototake Hirotada. Mr. Ototake is a very famous figure in Japan who has had both his arms and legs amputated. Providing 'both legs' is very challenging because if the legs are amputated under the knee, the device is more controllable, but for Mr. Ototake, whose legs were amputated above the knees, the device is very difficult to control. Eventually, Mr. Ototake trained himself with the prosthetic device on both legs and walked by himself. Through this achievement, I think this technology will be applied to general rehabilitation devices as well.
with Dr. Endo's technology will be able to achieve. What he expects is something like a gold medal and more importantly, to achieve a better record than the Olympians in the future. We can transform perceptions, and this I think is the real goal that we want to achieve.
The Blu-ray on the ISS for High-Speed Communication
Now, moving to the global-agenda issues, we would like to briefly share with you three topics: one is "Space", the second is "Ecosystem", and the third is "Energy."
In relation to space, we created the low-earth-orbit satellite, an inter-satellite communication system using the Blu-ray disk technologies.
We have Blu-ray pickup technology, which is extremely precise and fast-moving tracking technology. Some of the engineers not only track the Blu-ray or SA-CD pitch, but also try to track the satellite in space; we empower the laser with all the actuators. Two years ago, we launched the device to the International Space Station.
What we are trying to achieve is this. This is a 10x10 centimeter small device. It is lightweight but has a laser.
Rather than radio waves, the laser has high energy efficiency and a high level of security. It can be used for communication as well. We are trying to implement high-speed broadband communication in between the low-earth-orbit satellites and ground stations, and more so we can create comprehensive high-performance broadband network connections around the globe. We had successfully tested the communication between the ISS and ground stations. This was most difficult because of atmospheric dispersion. Then we are going to try to achieve low-earth-orbit satellite communication to complete the proof of concept, verify that the technology works, and finally, verify that it would be able to run for the business.
This is something we are trying to cover because the laser is more secure and more energy-efficient. It probably runs much faster than the traditional radio wave-based satellite communication system.
This is a slide showing our journey, with how things started with the Blu-ray disk technologies and how they got to the ISS. We can achieve communication for low-earth-orbit satellites, as well as stratosphere platforms. I think this is a very interesting project to carry out.
'Synecoculture' Enriches Nature and People
Now, let me talk about the second topic, related to Synecoculture™.
It is a very interesting project, trying to achieve maximum biodiversity while achieving stable and high-yield agriculture around the world. This idea is advocated by Dr. Funabashi Masatoshi. He noticed that monoculture, which is more like industrial farming, appears to be very prevalent, but the reality is that up to 80 percent of basic commodities are from small- to medium-sized farming, and this represents 90 percent of the farming area in the world, so he thought, why not put the science there.
That is really the critical part because industrial farming is capital intensive and, using a specific definition, is therefore very productive. At the same time, however, it reduces biodiversity a great deal and that puts stress on the soil and brings other problems.
Then we thought, why don't we put the science into backyard farming and try to design the ecosystem with multiple crops growing together and meet the natural development of the ecological system. Like an actual ecosystem, farms will become more productive with multiple harvests, and biodiversity will increase even beyond the natural environment. Dr. Funabashi is trying to design such a system.
When we think about the design for that, we need the science behind it. We cannot just put random plants and crops in the area, so we have to prepare a huge biodiversity database and understand which terrain is good, what kind of plants have to be planted in what combination, what kind of insects are eating the plants, and what kind of predators are eating the insects. Overall, we will try to design a stable and dynamic ecosystem. The micro ecosystem would be able to constantly monitor the situation and help farmers make decisions. In traditional farming, you basically have one type of crop in isolation. You probably need to make 4 to 5 major decisions, like which one to plant, when to plant, when to add fertilizers and chemicals, when to harvest, and things like that. But if you have 100 different crops in the same plot then the complexity is enormous. You cannot really do it yourself, and we need that experience sharing, we need the help of science and we need the help of technology, and that is something we are trying to achieve.
We have a huge database, with sensor systems to provide the proper advice to the farmers who are working on multiple crops and plants together, like 100 to 200 different kinds in one place.
This is a farm in Burkina Faso, and we have used it for three years to test Synecoculture. The outcome is very interesting. It was almost desertified land, but we turned it into an extremely fertile and highly biodiverse area. The agriculture yield is stable and remarkably productive.
If a family has a very specific, reasonable farming ground, the monthly agricultural output is equivalent to 1000 euros (per person on 500 sq.m). With a family of three on 1500sq.m, that would be 3000 euros, which is the average income level of ASEAN countries.
We made quite a substantial achievement in Burkina Faso. We established a research and training institution in Sub Saharan Africa to promote Synecoculture in the region. We are hoping that Synecoculture will be able to help them to recover and increase biodiversity and provide a high agricultural output in a way adaptive to climate change. Hopefully, we can help them to solve their social issues and also environmental issues at the same time.
We also founded a company led by Dr. Funabashi to commercially promote this development. Of course, we want to be sustainable, and 'sustainable' in business as well, so that we can actually help people to grow their businesses.
We will try to create a scheme for the business so that we can scale this idea onto the global scene.
A Stable Energy System with Fully Renewable Energy
The third topic is the Energy System.
What we are trying to do is to use a Micro-Grid system.
We started with public viewing for the 2010 FIFA world cup in Ghana using renewable energy with support from JICA. We brought the system back to Okinawa, the southern islands of Japan, and have had quite a successful operation there for five years, providing stable renewable energy.
This is a picture of photovoltaic panels and lithium-ion batteries. Each battery is connected by the bus line so that energy can be exchanged when houses are short of electricity. The house with the relative surplus of energy can send energy through the private bus lines. This is the outline of energy exchange in a Micro-Grid.
They autonomously exchange energy to provide a better balance and the most efficient use of the facilities.
When Photovoltaic panels start generating electricity from around 11 a.m., usually the batteries become saturated because they generate more electricity than they discharge. Then they can send surplus electricity to other half-full batteries that need electricity.
This chart is very interesting. The vertical axis shows what kind of or what size battery you need, and the horizontal axis is the self-sufficiency rate. If you want to have a self-sufficiency rate of 96 percent, you need a battery of 50-kilowatt-hour size, which is substantial. With this energy exchange, you can reduce equipment costs down to 20 kilowatt-hours and achieve the same 96% self-sufficiency.
The energy exchange would be able to reduce the cost of the infrastructure while maintaining this high self-sufficiency of the renewable energy source. And by using such a system, we can reduce the load for the traditional grid system as well. If you are struggling to get energy in the residential areas, you can go renewable and reduce the burden for the main grid by using both the Micro-Grid and the current infrastructure as well.
As this technology is very important, we decided to go open source. One of the reasons is that, because of COVID-19, many countries had a lockdown, and the outcome of the initial 3 months of lockdown was really shocking.
With that kind of slowing economy, we couldn't really reverse the trend of global climate change. We just went back to around a 10-year level in terms of CO2 discharge. Then we realized that although we could try to grow a business by ourselves, that would probably not be fast enough, so we should disclose the technology to make it open source, so that anyone interested in it would be able to take our technology and run with it.
We work with the LF Energy, one of projects of Linux Foundation and created a new project called "Hyphae Project"., which is one of the International projects we are working on based on our technology, with multiple stakeholders. We will be able to further develop this project and they will be able to take it into their business as well.
I know this is really a global issue. Given that the energy infrastructure is there, the agricultural idea is there, and the satellite communication idea is there, if we combined all three, we would be able to help people to achieve highly biodiverse agriculture, clean energy, and communications, and we would be able to provide all operations necessary for agriculture and energy control, as well as education and medicine on a global scale. That is what we are trying to achieve in the global-agenda area.
Technology to Enhance Creativity
Of course, Sony is a company that works with creators, so creativity is extremely important for us as well. We have many projects regarding creativity.
I don't think we can go through every one of these projects, but one that we find interesting is Dr. Furuya Shinichi's project on musical performance.
Dr. Furuya himself was a concert pianist but he retired because he got injured with muscle and physiological issues. He found out that proper training prevents musicians from having muscle injuries or neurological issues. He came up with a device for training to have pianists experience the precise finger and hand movements. The device can record, playback, and provide very specific feedback for how the individual should be trained.
With this very interesting device and others, we created the Music Excellence Project, a kind of musical academy. There, young and very talented pianists are enrolled and trained using Dr. Furuya's method with training devices. The technology provides us with the know-how and the data of the top-level pianists' performance. We can pass on the data to the young pianists so that they can understand very quickly what kind of finger or hand movement is required for such high-level performance.
The method provides the skillset with interesting new technologies, so they can really focus on how they express the music.
The music basically requires you to have the skill; you have to have the expressions in your mind. Now we can help them to learn the skills much quicker, much more accurately, and we can provide very precise feedback so that they can focus more on the expression part of the performance.
There is another very interesting invention by Dr. Furuya. We talked about a project for the players, but this is for the composers. An AI-assisted music production app Flow Machines was initially launched in Paris and moved to become part of the project between Tokyo and Paris. It demonstrates how AI can help people to compose their music.
Though this is not automatic music generation, we actually started initially with the auto-generation of music to see how things work. We have a machine-learning system to learn all the music and particularly the tunes to create the music in a specific niche.
For example, the AI created Beatles-style music and put it on YouTube. It became very popular, but at the same time, we thought, this won't help the creators. Creators don't want to have a song like that of the Beatles. They want to have their own original songs. They are looking into new kinds of music. Starting from there, we altered the technology to create music of the specific style of each creator.
In this diagram, you have a lot of scores, sound sources, and machine learning to understand what kind of score or sound makes a specific kind of music. You will understand the nature of machine learning and music generation through the machine.
Of course, creators want to have their own tunes, so we put them into the digital music composing system, what we call DAW (Digital Audio Workstations). Instead of generating the Beatles, the composers can register their own phrases and come up with specific instructions with those pieces. They have such a kind of possibility to explore.
Rather than a computer generating music automatically, the system works with the composers interactively. Even if composers have specific intentions, the AI helps the composers to match their specific style, and suggests what could be a possible choice, and then the composers think "Yes! this is interesting! I haven't thought about it, but this is good, so let's take it." Not just taking it as it is, they tweak it and then feed it back into the AI system, and then the AI says "this is tweaked, then what's next? (there are three options)" and the composers keep on going. It is more like a dialogue-based interactive partner, which can help the composers to be more creative.
This system is embedded into DAW as a plugin and serves as backup.
We already have a few professional high-quality tracks.
Sony music entertainment Japan has a lo-fi beat channel and the music from here is composed by the Flow Machines.
We have had some professional musicians create their new tracks with the Flow Machines. Some music pieces of the top creators are made using the system so it becomes one of the de facto tools for the musicians to use.
We are very happy to be able to help them to become more productive and more creative, exploring a new kind of music space and a new kind of music.
This is a broad lineup of the technology we are putting forward and we have a lot of interesting AI-based tools as well. Those are the artists who are working with us and using the tool for their tracks.
Sony AI Tries Gastronomy
There is a lot more to Sony CSL, but let me turn to Sony AI, which is another organization I am managing.
Sony AI was created two years ago to become an organization for AI research and AI deployment.
We have 4 major topics, "AI for Gaming", "AI for Imaging & Sensing ", "AI for Gastronomy", and "AI for Ethics", for responsible AI.
I will briefly introduce one of the topics, Gastronomy. This is the flagship project.
Gastronomy is very interesting because it is an obvious part of entertainment. It is very creative in terms of the choice of food, and the choice of cooking and ingredients. It impacts the wellness and sustainability of people, so this is the area we are really interested in exploring a lot.
There are 2 things we are trying to do. One is AI-Powered Recipe Creation. For example, we are asking how we can help chefs to create a new recipe, because there are more constraints because they have to make a choice of proper ingredients, in terms of traceability and the health impact, etc. At the same time, the cuisine has to be tasty and creative.
As there are a lot of constraints, how can we actually help the chef to be able to create a new recipe? On the physical side, how can we assist chefs in cooking, plating, or cutting, and all kinds of things? How can we enable them to do more interesting and creative cooking? Or in the case of a labor shortage, how can we actually help them?
This is a very new kind of project because Sony is new to the gastronomy area, so we are learning a lot from top level chefs.
Some of the early outcomes of our AI engine will help with pairings, such as between something like chocolate and nori (Seaweed). Some other ingredients might actually well go together because some of their chemicals share flavors. There are chemicals that have the same flavor common to chocolate and nori or blueberry.
We are going into the molecular gastronomy level and will try to find out what the impact of the combination on the sensations is. Of course, we all have receptive structures. We can see how the molecule attaches to the specific receptor and impacts our sensations. We are trying to come up with all kinds of combinations and to suggest to chefs a certain combination, and if that might work, chefs will try to look into that. This is the combination of AI and molecular gastronomy.
As for the second thing we are trying to do, we are trying to help chefs with plating. For example, if you are trying to prepare a plate for one person, you can do this simply, but if the plates are for eight or so persons, and if the plating is extremely time sensitive, like if you have to complete the plates in 2 minutes, and if the plating is particularly complicated, it is a very difficult thing to do.
With the robotics systems, you would be able to plate up many dishes with precision very quickly. That would be of help, and it could be artistic as well. It is quite a challenge to achieve such a high level of fine dining, and to be always conscious about sustainability and being healthy. How can we achieve this?
This is a challenge because we are new, so we are carrying out interviews with top-level chefs around the world. There are 18 professional chefs, sommeliers, and patisseries with 24 Michelin stars altogether. We are learning a lot from their insights.
Advising us in particular are Mr. Yoneda Hajime and Mr. François Chartier, and we have a very close ongoing collaboration with them. We are trying to build an extended network of top-level experts to be able to work together and learn a lot from them.
Overall
That is what is going on within Sony AI and Sony CSL. We have interesting things going on like Dr. Endo's project for the Paralympics, the project for the global sustainability agenda, and we also have the project for creativity and the project for gastronomy. All combine to enhance the creativity of human beings, trying to solve the issues in the world, to tackle the global agenda. I hope we can contribute to a better future.
Dr. Kitano Hiroaki
